Method and device for elucidating a vehicle's vehicle surroundings and vehicle

Abstract

The invention relates to a method for elucidating a vehicle environment (106) of a vehicle (100). The method comprises the steps of providing a representation of the vehicle environment (106); constructing relationship information using a first position of a first object (115) in the representation and a second position of a second object (116) in the representation, wherein the relationship information represents a spatial relationship between the first object (115) and the second object (116) in the vehicle environment (106); and using the relationship information for elucidating the vehicle environment (106).

Description

Technical Field

[0001] This invention relates to methods and apparatus for elucidating the environment surrounding a vehicle. The subject of this invention is also a computer program. Background Technology

[0002] Systems used for driver assistance systems (FAS) typically employ sensors with different physical properties to detect the vehicle's status. The sensor information is processed in the following way to enable driver assistance functions, such as longitudinal control of the vehicle, also known as "ACC" (Adaptive Cruise Control). Summary of the Invention

[0003] Against this backdrop, utilizing the proposed solution, a method for elucidating the vehicle's surrounding environment is proposed, along with a device for using this method, a vehicle, and finally, a corresponding computer program. The invention also proposes advantageous extensions and improvements.

[0004] The use of the location of objects in the vehicle's image (Abbild) to construct relational information—which represents the spatial relationships between objects in the vehicle's surrounding environment—enables the import of relative object descriptions for implementing driver assistance functions and / or automated driving functions and / or autonomous driving functions.

[0005] An image can refer to a sensor-near image of a scene. An image can be understood as a description of the driving conditions around the vehicle. An image can, for example, refer to a scene shown in a center-perspective image or a bird's-eye view. An image can show a diagram of a scene from the perspective of a video sensor.

[0006] The imagery may in particular relate to images from a vehicle’s video sensors, which, according to one embodiment of the concept presented herein, analyze relative object descriptions and object relationships from the images. These object descriptions and object relationships are combined when internally elucidated, and the results of the elucidation and / or the relationships between objects are made available at an external interface to the vehicle’s driver assistance systems and / or driving functions.

[0007] A method for elucidating the vehicle's surrounding environment is proposed, wherein the method comprises the following steps:

[0008] Provides an image of the vehicle's surroundings;

[0009] Relationship information is constructed using the first position of a first object in the image and the second position of a second object in the image, wherein the relationship information represents the spatial relationship between the first object and the second object in the environment surrounding the vehicle.

[0010] The aforementioned relational information is used to clarify the environment surrounding the vehicle.

[0011] The method can be implemented in software or hardware, or in a hybrid form consisting of software and hardware—for example, in a control device.

[0012] The vehicle's surrounding environment can refer to the vehicle's surroundings in relation to driver assistance functions used in the vehicle. The vehicle's surrounding environment can include infrastructure and / or living or inanimate objects in the relative vicinity of the vehicle. The vehicle's surrounding environment can vary depending on whether the vehicle is moving in a rural or urban environment, and can also vary depending on the type of road the vehicle is currently traveling on. Optical sensors can be part of a video camera assigned to the vehicle's driver assistance system, positioned behind the vehicle's windshield. The image can show a sensor-near image of a scene in the vehicle's surrounding environment. The image can be, for example, a center-perspective image. The image can also show a bird's-eye view. The image can represent an image generated by optical sensors aligned with the vehicle's direction of travel. The image can correspondingly involve digital video images. Depending on the camera's position, the image can depict segments of the vehicle's surrounding environment in front of the vehicle in the direction of travel, particularly road segments in front of the vehicle in the direction of travel and the infrastructure areas adjacent to said road segments, and objects located within said segments.

[0013] Objects may include, for example, traffic participants (e.g., vehicles, pedestrians, or cyclists), the drivable area of ​​the scene (so-called open space), lanes of the driving lane, traffic infrastructure, side edges (e.g., guardrails and guide posts, green stripes, etc.), vegetation, buildings, sky, etc. Objects can be understood in the appropriate context as real objects in the actual environment surrounding the vehicle or virtual objects of real objects in a digital image representing the environment surrounding the vehicle. The position of an object can represent the position of the object depicted by pixels in the digital image. The spatial relationship between a first object and a second object can represent, for example, the spacing between objects, the distance between at least one object and the vehicle, the direction and / or speed of movement of at least one object, or the occlusion of one object by the other. This relationship information can be obtained using the number and size of pixels in the image, indicating the relationship between the first and second objects, such as the spacing between the objects.

[0014] To advantageously track at least one object in the environment surrounding the vehicle, the steps of the method can be repeated.

[0015] The concept presented herein describes an interface for a video sensor in a preferred embodiment, the interface being optimally matched to the characteristics of the sensor and directly utilizing the physical characteristics of the video sensor. This avoids or supplements performance degradation caused by sensor information conversion (e.g., from a frontal view to a grid pattern (mesh pattern)) with additional direct video information. Here, information directly derived from the physical characteristics of the sensor can supplement the interface, for example, in a bird's-eye view projection. This method expands the range of information that can be extracted from the video sensor and used. This reduces the number of other sensors used for implementing driver assistance functions in the future. This results in significant advantages in the sensor market, allowing for the fullest utilization of the potential of each sensor. The concept presented herein increases the range of action and / or the availability of detected objects in driving conditions.

[0016] According to one embodiment, the relationship information can be used in the usage steps for lateral steering functions such as lane keeping, and / or for longitudinal vehicle adjustment. Here, the longitudinal adjustment also includes, for example, vehicle following. This allows for particularly robust implementation of the lane keeping function.

[0017] Advantageously, the method includes a step of assigning a first object and / or a second object to an object category. Accordingly, the object category can also be used in the usage step to clarify the vehicle's surrounding environment. According to the described embodiment of the method, the vehicle's surrounding environment can be clarified advantageously with low computational cost.

[0018] According to another embodiment, the method may include a step of constructing occlusion information. The occlusion information may represent a first object in the image being occluded by a second object. Accordingly, the occlusion information may also be used in the usage step to clarify the environment surrounding the vehicle. The occlusion information can be readily used to determine the spatial relationship of the first object relative to the vehicle.

[0019] In the step of constructing the relationship, for example, another relationship information can be constructed using another first position of another first object in the image and another second position of another second object in the image. Here, the other relationship information can represent the spatial relationship between another first object and another second object in the vehicle's surrounding environment. Accordingly, in the usage step, the other relationship information can also be used to clarify the vehicle's surrounding environment. The other relationship information can be easily used to verify the relationship information.

[0020] According to a particular implementation, the steps may include generating a relationship value between the relationship information and the other relationship information. This allows for the clarification of the vehicle's surrounding environment when using the relationship value.

[0021] The described implementation of the method enables the beneficial propagation of the imagery of the vehicle's surrounding environment to distant locations.

[0022] Here, for example, the relationship value can be used in the usage step to follow another vehicle.

[0023] Advantageously, the method includes a step of determining the orientation (Peilung) of one or the other object relative to the optical sensor. Accordingly, this orientation can also be used in the usage step to clarify the environment surrounding the vehicle. Thus, the polarity detection principle of the optical sensor can be advantageously used in its direct application to clarify the environment surrounding the vehicle.

[0024] The proposed solution also implements an apparatus configured to perform, operate, or implement the steps of the proposed method within a corresponding device. The task based on the invention can also be solved quickly and efficiently through an embodiment of the invention in the form of an apparatus.

[0025] To this end, the device may have at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface to a sensor or actuator for reading sensor signals from the sensor or for outputting data signals or control signals to the actuator, and / or at least one communication interface for reading or outputting data, the data being embedded in a communication protocol. The computing unit may be, for example, a signal processor, microprocessor, ASSP (Application-Specific Standard Product), SOC (System-on-a-Chip), etc., wherein the storage unit may be a flash memory, EPROM, or magnetic storage unit. Data may also be stored in a cloud distributed across vehicles. The communication interface may be configured for wirelessly and / or wiredly reading or outputting data, wherein a communication interface capable of reading or outputting wired data can electrically or optically read data from or output data to a corresponding data transmission line.

[0026] In this application, "device" can be understood as an electrical device that processes sensor signals and outputs control signals and / or data signals based on the sensor signals. The device may have an interface, which may be constructed in hardware and / or software. In a hardware construction, the interface may, for example, be part of a so-called system ASIC and / or system ASSP, which contains the most distinct functions of the device. Alternatively, the interface may be a standalone integrated circuit or at least partially composed of discrete devices. In a software construction, the interface may be a software module, which may exist, for example, on a microprocessor alongside other software modules.

[0027] In an advantageous configuration, the device constructs relational information by utilizing the positions of objects in a digital image of the vehicle's surrounding environment. For this purpose, the device can invoke video signals from video sensors installed in the vehicle.

[0028] A vehicle having an optical sensor, a driver assistance system, and the device described above is also proposed. The device is electrically connected to the optical sensor and / or the driver assistance system.

[0029] Driver assistance systems can provide lane keeping assist or functions such as autonomous driving or automatic braking.

[0030] It is also advantageous to have a computer program product or computer program having program code, which can be stored on a machine-readable medium, such as semiconductor memory, hard disk memory or optical memory, and is used, especially when the program product or program is run on a computer or device, to perform, implement and / or manipulate the steps of the method according to any one of the embodiments described above. Attached Figure Description

[0031] Embodiments of the proposed solution are shown in the accompanying drawings and further described below. The drawings show:

[0032] Figure 1 A vehicle having a driver assistance system according to one embodiment;

[0033] Figure 2 Block diagram of an apparatus for illustrating the vehicle's surrounding environment according to one embodiment.

[0034] Figure 3 A flowchart of a method for illustrating the vehicle's surrounding environment according to one embodiment.

[0035] Figure 4 A schematic diagram of the central perspective of a highway scene for constructing relational information according to one embodiment.

[0036] Figure 5 A schematic diagram of the central perspective of an urban scene for constructing relational information according to one embodiment.

[0037] Figure 6 A schematic diagram of another central perspective of an urban scene for constructing relational information according to one embodiment.

[0038] In the following description of advantageous embodiments of the invention, the same or similar reference numerals are used for elements shown in different figures and having similar functions, wherein the elements are not described repeatedly. Detailed Implementation

[0039] Figure 1 Vehicle 100 is shown in a top view. For Figure 1 In the embodiment shown, vehicle 100 relates to a passenger car. Vehicle 100 may also relate to other road-related vehicles, such as trucks or motorcycles.

[0040] The example vehicle 100 has multiple video cameras mounted at different locations on the vehicle. A first video camera 102 is mounted behind the windshield of the vehicle 100 and faces the surrounding environment 106 of the vehicle 100 in the direction of travel 104, in front of the vehicle 100. A second video camera 108 is mounted in the right-side rearview mirror of the vehicle 100 and faces the surrounding environment 106 of the vehicle 100 in the direction of travel 104, in front of the vehicle 100. A third video camera 110 is mounted in the left-side rearview mirror of the vehicle 100 and faces the surrounding environment 106 of the vehicle 100 in the direction of travel 104, in front of the vehicle 100. A fourth video camera 112 is mounted in the rear area of ​​the vehicle 100 and faces the surrounding environment 106 of the vehicle 100 in the direction of travel 104, in front of the vehicle 100.

[0041] The vehicle 100 also includes a driver assistance system 114, which is electrically connected, for example, to video cameras 102, 108, 110, and 112 via the vehicle 100's CAN bus. Optical or video sensors of the video cameras 102, 108, 110, and 112 generate video signals of the vehicle's surrounding environment 106 visible to the video cameras. These video signals are processed into digital images of the detected vehicle's surrounding environment 106 in image processing devices associated with the video cameras 102, 108, 110, and 112.

[0042] The image information is provided to the driver assistance system 114 via a suitable interface through subsequent and / or functions included in the video cameras 102, 108, 110, 112 to enable driver assistance functions.

[0043] According to Figure 1 The diagram shows video cameras 102, 108, 110, and 112 using their respective image sensors. The system detects the condition of vehicle 100 and identifies objects or other information relevant to driver assistance system functions. For example, in... Figure 1 The diagram shows a first object 115 and a second object 116 in the vehicle's surrounding environment 106, where the first object and the second object are... Figure 1 The exemplary driving situation shown is located within the detection area of ​​the first video camera 102.

[0044] For example, the optical sensor 118 of the first video camera 102 detects a road segment and two objects 115, 116 located in front of the vehicle 100 in the driving direction 104 during the driving condition of the vehicle 100, and provides corresponding image signals. Using these image signals, the image processing device of the first video camera 102 generates a central perspective digital image of the detected surrounding environment 106, in which the infrastructure elements of the detected segment of the vehicle's surrounding environment 106 and the objects 115, 116 are planarly displayed. Here, the central perspective view is merely an exemplary choice. For example, a bird's-eye view view could also be implemented.

[0045] According to the concept presented herein, vehicle 100 is equipped with at least one device 120 for clarifying the vehicle's surrounding environment 106. Device 120 is electrically connected to a first video camera 102 and configured to construct at least one relational information representing the spatial relationships between real objects 115, 116 in a central perspective image of the vehicle's surrounding environment 106 generated by the video camera 102, and to use the relational information to clarify the vehicle's surrounding environment 106.

[0046] exist Figure 1 In the illustrated embodiment, device 120 is part of a first video camera 102, for example, in the form of an interface or interface of video camera 102, the device being optimally matched to the characteristics of sensor 118 and directly utilizing the physical characteristics of the video sensor.

[0047] According to one embodiment, the video cameras 108, 110, and 112, which are typically installed in the vehicle 100, are also equipped with their own devices 120.

[0048] In an alternative variant, device 120 is mounted remotely within vehicle 100 from video camera 102 and is electrically coupled to video camera 102, for example, via a CAN bus. Device 120 may also be part of driver assistance system 114 and configured to centrally process digital images of all available cameras 102, 108, 110, 112 to clarify relational information about the vehicle's surrounding environment 106. In addition to the digital images, descriptions of the vehicle's surrounding environment from each camera 102, 108, 110, 112 may also be transmitted to central unit 114.

[0049] The task of device 120 is to provide relative object descriptions and object relationships based on the kontellation of objects 115, 116 detected in the vehicle's surrounding environment 106 in the center perspective image of the planar image sensor of video camera 102.

[0050] The camera system for driver assistance designed based on the concept presented herein has high performance while maintaining low system complexity and low system cost.

[0051] According to the proposed concept, the driver assistance functions of the driver assistance system 114 can implement an improved adjustment strategy based on the relative description of the video sensor 118, which enhances the quality of each function. For example, the (relative) collision time can be adjusted more robustly compared to absolute coordinates determined indirectly from video material. This derives behavioral patterns advantageous to the video-based driver assistance system 114. For example, for following in a driving lane, the relative distance to the vehicle ahead is adjusted relative to the lane direction. For driving in an open area, guidance is provided based on the relative distance to side boundaries such as guardrails.

[0052] To support scene clarification, information from the digital map stored in vehicle 100 can also be advantageously used. If the road conditions do not change on the current road segment, for example, driving on a highway without entering or exiting, the situation can be relatively reliably assumed to be the same and constant. Here, the vicinity and distant areas of camera 102 should have a high degree of similarity in the road infrastructure. This relative description of the scene content can be advantageously used by driver assistance functions optimized for this interface specification.

[0053] According to one embodiment, a radar sensor may be attached to or replace at least several of the video cameras 102, 108, 110, and 112 installed in the vehicle 100. The radar sensor detects the backscattered signals from objects 115 and 116 and measures the distance between objects 115 and 116. Other sensors, such as optical radar or ultrasonic sensors, are also conceivable in this case.

[0054] The concept of relative object description proposed here for use in a novel video sensor interface for driver assistance systems can also be implemented in conjunction with driver assistance, where information from different sensors in the vehicle driver assistance system is merged to optimize functional performance. Before merging, this information is converted to a unified, mergeable diagram. Here, a Cartesian, absolute diagram of the vehicle's situation is frequently used. This diagrammatic method is called a grid diagram (mesh diagram). The unified modeling of the scene results in sensors, based on conversions to a mergeable form, often no longer being able to provide their data and information in their local coordinate system.

[0055] If the information from the video sensor 118 is converted to a grid diagram, then the scene is viewed from the perspective view, rotated almost 90° from the original viewing direction—from which the scene was initially received—and viewed from above. The relationships between objects 115 and 116, initially contained in the image of the scene, are lost here. Here, the functionality on the “classic” grid diagram utilizes, for example, the video sensor 118 only suboptimally. According to the proposed scheme here, this can be compensated for by adding relational information based on the perspective view.

[0056] The sensor installation of vehicle 100 can be advantageously simplified using the concept presented herein. This allows for a reduction to just one radar sensor and one video sensor. It is also conceivable that complex driving functions can be implemented on a single main sensor; here, the video sensor 118 is primarily provided because it has the largest bandwidth and can provide complete information about the driving situation.

[0057] Figure 2 A block diagram illustrating one embodiment of a device 120 for clarifying the vehicle's surrounding environment is shown. The device 120 has a construction device 200 and a usage device 202.

[0058] A central perspective image 204 of the vehicle's surrounding environment, generated by the image sensor, is provided to the image device 200 via a suitable interface—for example, to the image sensor of a video camera in the vehicle. The construction device 200 is configured to construct relational information 206 using a first position of a first object in the image and a second position of a second object in the image, and to provide the relational information to the user device 202.

[0059] The device 202 processes relational information 206 representing the spatial relationship between a first object and a second object in the vehicle's surrounding environment in order to clarify the vehicle's surrounding environment in light of necessary driver assistance functions.

[0060] According to one embodiment of device 120, while constructing device 200, another relationship information 208 is constructed using another first position of another first object in the image and another second position of another second object in the image, the relationship information representing the spatial relationship between another first object and another second object in the vehicle's surrounding environment. The other relationship information 208 is also processed in device 202 to further clarify the vehicle's surrounding environment, taking into account necessary driver assistance functions.

[0061] The result information 210, in a form suitable for processing, provides, for example, the vehicle's driver assistance system with a clarification of the vehicle's surrounding environment as performed in the use of device 202, which, when using the result information, controls, for example, the vehicle's lane-keeping function.

[0062] According to one embodiment of device 120, construction device 200 is also configured to assign objects shown in the digital image to one or more object categories. Accordingly, the vehicle's surrounding environment is also elucidated in use device 202 when the object classification is employed.

[0063] Alternatively or additionally, the constructing device 200 may also construct occlusion information 212 in cases where one object is occluded by another object in a central perspective image and provide said occlusion information to the using device 202. Accordingly, the using device 202 also clarifies the vehicle's surrounding environment when using the occlusion information 212.

[0064] Here, the number of relationships between objects is not limited to, for example, relationships 206, 208, 212, and 214. Any number of relationships of type 206, 208, 212, and 214, or other meaningful relationships, can exist, which will not be discussed in detail here.

[0065] Based on the principle of polarity measurement of optical sensors, according to another embodiment, the orientation 214 of at least one object detected by the sensor and shown in the image is determined in the imaging device 200 and provided to the using device 202. The using device 202 uses the orientation 214 to clarify the environment around the vehicle, especially in terms of moving objects that may intersect the vehicle's path, such as pedestrians or cyclists.

[0066] Figure 3 A flowchart illustrating one embodiment of a method 300 for elucidating the vehicle's surrounding environment is shown. This can be achieved by... Figure 1 and Figure 2 The method 300 shown herein is for implementing an apparatus for clarifying the environment surrounding a vehicle.

[0067] In step 302, a central perspective image of the vehicle's surrounding environment is provided, the image having been generated by an optical sensor aligned with the vehicle's direction of travel. In step 304, at least one relational information representing spatial relationships between objects in the image is constructed. In step 306, the relational information is used, for example, in a driver assistance device of the vehicle or in a computing unit of a camera assigned to the optical sensor, to clarify the vehicle's surrounding environment.

[0068] For tracking or tracing functions of moving objects in the environment surrounding the vehicle, steps 302, 304, and 306 can be repeated.

[0069] If other information is generated when using 306 or when clarifying the constructed information 304, the other information can also be used as the constructed information 304 in method 300.

[0070] To intuitively illustrate the concept of relative object description proposed here as the basis for driver assistance functions, Figure 4 An exemplary center-view digital image 400 of the environment surrounding a vehicle is schematically shown. The scene depicted here is a typical highway scene, as seen and imaged by the image sensor of a video camera mounted behind the windshield of the exemplary vehicle—the vehicle itself. The vehicle is hypothetically located... Figure 4 The schematic perspective image 400 shown is in front of and in front of... Figure 4 The line 402 represents the position of the vehicle relative to the central longitudinal axis of the driving scene shown in the digital image 400.

[0071] exist Figure 4 In the central perspective image 400, a highway scene as a driving situation is schematically illustrated. The scene consists of an open space 404, in which lanes are marked by markers 406. The side boundaries of the driving lanes formed by the open space 404 are left by side markers 408. A guardrail 410 is arranged next to the open space 404. A truck 412 is traveling in front of the vehicle. The truck 412 is traveling in lane 414 of the driving lane 404, which is also used by the vehicle. Markers 406, side markers 408, guardrails 410, truck 412, and lane 414 constitute the objects in image 400.

[0072] The following implements a new relative description of the scene shown in the digital image 400. The lateral distance 416 is measured from the centerline 402 representing the vehicle itself to the right lane boundary 408. The loaded vehicle 412 travels ahead with a lateral distance 418 to the right lane boundary 408. The lateral distance 418 is measured from another line 420 representing the centerline of the loaded vehicle 412 to the lane boundary 408.

[0073] Based on the concept presented herein, the relationships between objects 406, 408, 410, 412, and 414 shown in the central perspective image 400 serve as the basis for elucidating the vehicle's surrounding environment. Here, these relationships describe the relationships between objects 406, 408, 410, 412, and 414 in image 400, such as "adjacent relationships." Figure 4 In this context, the adjacency is defined by another vehicle 412 traveling in front of the vehicle along the guardrail 410. This relative relationship is similar to the interpretation of a situation by a person and offers great potential for understanding a scene or driving situation. Therefore, a possible approach for the vehicle's assistance functions is to accurately follow the vehicle 412 and the guardrail 410. If, for example, the vehicle 412 cannot be well identified ahead, the vehicle follows the guardrail 410.

[0074] According to one embodiment, a device installed in the vehicle for determining the vehicle's surrounding environment first determines relationship information using a side distance 416 and then determines another relationship information using a side distance 418. Using both the relationship information and the other relationship information, the device then generates a relationship value that provides a clarification of the vehicle's surrounding environment as shown in the image 400. This clarification can be used in the vehicle's driver assistance system for following a load-bearing vehicle 412.

[0075] The vehicle can thus functionally follow the vehicle 412 ahead by maintaining a lateral distance 416 from the same marker 408, without having to measure the vehicle 412 ahead precisely and absolutely—for example, via a Cartesian coordinate system. While driving, one only needs to ensure that the vehicle does not collide with the vehicle 412 ahead upon approach. This advantageously reduces the requirements for identifying the vehicle 412 ahead. Instead, it is observed whether the relative values ​​derived from the distances 416, 418 to the common object 408 remain constant.

[0076] The object description used in the proposed scheme is more robust than other description methods because it tracks structures directly visible to the camera or uses direct measurement parameters, such as the dimensions of objects 406, 408, 410, 412, and 414 in image 400, or the spacing between objects 406, 408, 410, 412, and 414. In addition to improved robustness, it also increases the effective range of the video sensor.

[0077] Alternative reference objects for clarifying the vehicle's surroundings can be provided by side guardrails 410, to which the loaded vehicle 412 has a relative distance 422. The vehicle itself also has a corresponding distance 424 to the same guardrail 410. Another exemplary alternative shows the lateral distance 426 from the loaded vehicle 412 to marker 406. The relationship information between objects 406, 408, 410, 412, and 414 can be determined and used as the basis for clarifying the surroundings.

[0078] Relationship information can also be determined within objects 406, 408, 410, 412, and 414. Thus, the middle lane 414 has a first measured width 428 in the surrounding area. If the following premise is set: the lane width does not change in the distant area within the current scene, then the second measured width 430 in the distant area is approximately consistent with the value 428. Robustness and working range are also improved here by using the internal relationships. The measured value 430 can also be used to measure other objects in the surrounding environment at the location of the measured value 430, with the camera in the vehicle itself having already obtained metric values ​​for the measured value.

[0079] Figure 4 The example shown, illustrating a relative description of traffic conditions on a possible highway scenario, visually demonstrates the proposed solution: to clarify the image content of the vehicle's surroundings (image 400), the camera within the vehicle itself maintains the stated viewpoint and generates a holistic and complete description of the situation. Projection to a bird's-eye view is not implemented initially to obtain added value from the information and avoid omissions and errors based on conversion techniques. Instead, information is obtained directly from the video sensors for maximum accuracy and usability.

[0080] In particular, the polarity characteristics of the measurement principles of video cameras are considered. Thus, the observation of objects 406, 408, 410, 412, and 414, under a defined angle and orientation, exhibits robust and direct measurement parameters. Here, based on the camera's polarity, the accuracy and resolution of measurements of spacing and object size are higher in the nearby region than in the distant region. Dimensional changes can also be measured directly from the object's surface and used to calculate, for example, time-to-contact (TTC) values. "Time-to-contact" describes the time elapsed after which the vehicle will collide with the specifically observed object.

[0081] If objects 406, 408, 410, 412, and 414 are identified and, if necessary, classified, they can be tracked in the planar graphic signal of the video sensor in the sense of so-called tracking. Regions with a uniform tracking vector—i.e., a light "stream" resembling an image sequence—show additional markings for objects located in the environment surrounding the vehicle, especially moving objects such as the heavy-duty vehicle 412.

[0082] Mark 408 Figure 4 In the exemplary image 400 shown, a common relative reference object is constructed for objects 402 and 412. The association between the area near the vehicle and the distant area is achieved through this common relative reference object 408.

[0083] In the vicinity of the camera, reliable information for measuring the detected objects 406, 408, 410, 412, and 414 can often be additionally obtained. This reliable information forms the basis for a Cartesian description of the scene by measuring lateral (horizontal) and forward (vertical) spacing and is a prerequisite for the scene's bird's-eye view and grid diagram. Three-dimensional, or 3D, measurement of the scene is achieved in the vicinity. This measurement can be appropriately propagated to distant areas of the scene through the aforementioned relationship. This strategy can also be used in human vision.

[0084] As an example, lane 406, detected optically, is referred to as an exemplary object, whose width does not change drastically as it transitions from a nearby area to a distant area. Based on this width, for example, it becomes possible to estimate or propagate the width of other vehicles in the distant area—such as truck 412. In the distant area, the two-dimensional (2D) characteristics in the perspective image 400 are primarily analyzed and processed. The device used to elucidate the vehicle's surroundings is configured to complete a scene elucidation of the current driving situation from a combination of all available image characteristics, thereby achieving high completeness.

[0085] Image characteristics available under the proposed scheme include, for example: relative object relationships (e.g., adjacency and direction of motion) of objects 406, 408, 410, 412, and 414; propagation of object relationships in the scene—e.g., on guardrail 410; special object descriptions for the near (3D) and far (2D) regions of the camera, where relationships can be extracted or propagated between information in the near and far regions of the scene; and the use of direct video sensor or camera parameters relative to adjacency; the planar appearance of the object; orientation; TTC; polarity characteristics of the video sensor, etc.

[0086] In order to clarify the relative object description presented herein as the basis for driver assistance functions, Figure 5 An exemplary center-view digital image 500 of the vehicle's surroundings in a typical urban environment is schematically shown. This illustrates a possible urban scene, as seen and imaged by the image sensor of a video camera mounted behind the windshield of the exemplary vehicle. The vehicle itself is also hypothetically positioned here. Figure 5 The front of the schematic stereoscopic image 500 shown.

[0087] exist Figure 5 In an example of a traffic scene depicted in a central perspective image 500 of the vehicle's surroundings, as shown, a person 502 enters a lane 414 from a sidewalk 504, which is located within a traffic lane 404. A curb 506 exists between the lane 414 and the sidewalk 504. A structure 508 stands to the right of the sidewalk 504. The person or pedestrian 502 enters the road behind another parked vehicle 412 and enters the crosswalk or zebra crossing 510.

[0088] Also there Figure 5 In the urban scene shown, to clarify the environment around the vehicle, the image sensor or camera imager of the video camera installed in the vehicle makes full use of the planar detection of the situation during image detection. The device installed in the vehicle to clarify the environment around the vehicle establishes the position of pedestrian 502 in the digital image 500 relative to the position of vehicle 412, so as to construct relational information using the distance 512 between pedestrian 502 and vehicle 412.

[0089] The relational information describes the adjacency relationship between objects 502 and 412. Additionally, the device determines occlusion information representing that pedestrian 502 is partially occluded 513 by vehicle 412. From this relation, for example, the geometric boundary conditions of the scene in image 500 can be derived. Thus, the longitudinal position 514 behind the end of vehicle 412's longitudinal position 516—or the distance in front of its own vehicle—can be determined. The analytical processing describing the object occlusion relationship 513 supports the metrical specification of the scene. Functions built upon this can thus achieve higher performance, such as higher recognition rates with lower false recognition rates.

[0090] Another relational information in the digital image 500 is determined by the distance 518 between person 502 and curbstone 506. (Complementary) The content describes another relationship information based on the distance from person 502 to the center of the lane. According to an embodiment, this relationship information can support each other and / or reduce measurement errors. Measurement of distance 518 can also be supported by occlusion information.

[0091] In the digital image 500, a person 502 moves across a crosswalk, or zebra crossing 510. A positional assignment relationship 522 between the pedestrian 502 and the object 510 is then generated, specifically a "Befindet-sich-auf" relationship 522. Using the "Befindet-sich-auf" relationship 522, the device can also determine relationship information. This relationship information can be used in multiple ways to clarify the vehicle's surroundings. On one hand, the classification of the crosswalk 510 can support the object assumption of the person 502, and vice versa. On the other hand, the (probable) intention of the pedestrian 502—to cross lane 414 via the crosswalk 510—can be derived from the combination of the object 502 classified as a pedestrian and the object 510 classified as a crosswalk. Similarly, occlusion information based on occlusion relationship 513 can be considered for clarifying the surroundings, etc.

[0092] Thus, firstly, the described adjacency relationships simplify the clarification process in the video sensor and then provide crucial information for driving functions built upon relative information to the camera's external interface. Similarly, regions connected at locations in image 500—e.g., sidewalk 504—can be considered for clarification. In particular, this clarification can propagate from the vicinity of the video sensor to distant regions. Besides the relationships described, which can be derived in a scene at a single moment or in images within a sequence of images, the dynamics of object motion can be considered for the description of relationships.

[0093] Figure 6Another exemplary central perspective digital image 600 of the vehicle's surroundings in a typical urban environment is schematically shown, created by a video camera of a vehicle moving within the surrounding environment. Primarily shown here... Figure 5 A city scene without parked vehicles.

[0094] Figure 6 The illustration in the diagram illustrates the polarity measurement of the object—in this pedestrian 502—based on the concept presented herein, using the polar coordinates of the pedestrian 502 in the image 600.

[0095] The device used to clarify the environment around the vehicle employs direct measurement parameters, obtained by a video sensor. The polarity distance 602 describes the distance between the vehicle and the person 502. This distance is measured at a suitable angular measure (Winkelmaβ) with a polarity angle 604 relative to the vehicle's centerline 402. Based on these distance and angle values, the device determines the orientation of the pedestrian 502 relative to the optical sensors in the vehicle and uses this orientation to clarify the environment around the vehicle.

[0096] According to one embodiment, the device for clarifying the surrounding environment also uses the extension scale 606 of person 502 in the lateral and / or longitudinal directions and the object height 608 of person 502. These spatial extension scales 606, 608 in the digital image 600 are measured in image pixels and / or metric dimensions and / or angular measurements.

[0097] In the tracking or tracing function of the proposed concept, changes in orientation, object size 608, or object extension scale 606 can be measured by tracking or tracing an object—such as pedestrian 502. A temporally changing orientation indicates that the vehicle will not collide with object 502; conversely, a temporally unchanged orientation indicates that a collision with object 502 is highly probable.

[0098] The changing orientation between objects can also be used to calculate relative distances or for object depth estimation. This depth estimation can be used as further support for clarifying the scene. Here, the disparity of different objects can be considered.

[0099] according to Figures 1 to 6The proposed techniques for elucidating the environment surrounding a vehicle offer the possibility of bottom-up or reverse analysis of scenes shown in central perspective digital images 204, 400, 500, and 600. To determine suitable relational information for implementing the concepts presented herein, spacing relationships, adjacency relationships, relationships between common reference objects, occlusion relationships, positional distribution relationships (e.g., "above," "below"), complementarity relationships, and propagation relationships (e.g., between nearby and distant areas of the scene in digital images 400, 500, and 600). Other relationships may also be envisioned.

[0100] The relationships are directly moderated in images 204, 400, 500, and 600 and / or referenced to other suitable coordinate systems. Motion relationships are added, describing relative changes on objects in the scene and / or relative changes between objects in the scene. Exemplary motion relationships are the tracking of directional changes opposite to constant orientation, size changes, relative directional changes (parallax) between objects, and relationship changes; adjacency relationships can, for example, be altered and from which object behavior is derived. Examples for this include vehicles leaving or approaching ahead, vehicles overtaking or being overtaken, oncoming vehicles, and vehicles obstructing other traffic participants.

[0101] Analysis can be supplemented by bottom-up analysis, which is advantageously complemented by top-down or forward analysis. Abstract information about the scene is derived from its connections and / or relationships. For example, objects adjacent to each other in digital images 204, 400, 500, 600 are often in the same context as other objects in the scene. The completeness of the scene's elucidation can also be achieved by identifying and addressing "gaps" in the description and / or scene elucidation. Furthermore, the completeness of the scene elucidation can be envisioned along with the reduction of un-elucidated areas of the scene. According to one embodiment, similar movement patterns of objects can be identified and considered, such as vehicles in a convoy, a group of pedestrians, etc. The behavior of objects can be identified, or the relationships connecting objects can be identified and / or the intentions of objects can be determined. Behavioral pattern or intention recognition can also be used to improve the effective scope of identification and / or the usability of object information.

[0102] According to another embodiment, it is also conceivable to identify the macroscopic structure of the scene. Here, for example, knowledge about the main direction of the road and / or the proximity of the vanishing point is obtained. The (main) limitations of the drivable road surface are identified, and elements of the lateral structure such as guardrails, guide posts, tree-lined avenues, etc., are determined.

[0103] The process of illustrating the exemplary scenario can be advantageously divided into two phases using relational support. The first phase involves a reverse analysis of the scenario, driven, for example, by data from sensor information. The second phase implements a top-down analysis of the scenario, driven by content and meaning from the scenario's perspective.

[0104] The concept of relative object description proposed herein, as a novel video sensor interface for driver assistance systems, is suitable for realizing high-performance automatic and / or (partially) automated driving functions. Scenarios that can be improved with the proposed measures are crucial in the roadmap for autonomous driving. The proposed scheme is advantageous in contexts where, for example, the high availability of longitudinal and lateral vehicle control exists.

[0105] If an embodiment includes an "and / or" connection between a first feature and a second feature, then this should be interpreted such that the embodiment has both the first feature and the second feature according to one implementation, and either only the first feature or only the second feature according to another implementation.

Claims

1. A method (300) for elucidating the vehicle's (106) surrounding environment, wherein, The method (300) comprises the following steps: Provides (302) an image (204; 400; 500; 600) of the vehicle's surrounding environment (106); Using the image (204; The first position of the first object (115) in (400; 500; 600) and the image (204; In the case of the second position of the second object (116) in (400; 500; 600), relationship information (206) is constructed (304), wherein the relationship information (206) represents the spatial relationship between the first object (115) and the second object (116) in the vehicle surrounding environment (106); The relationship information (206) described in (306) is used to clarify the vehicle's surrounding environment (106). The image is a central perspective image. In the construction (304) step, another relationship information (208) is constructed using another first position of another first object in the image (204; 400; 500; 600) and another second position of another second object in the image (204; 400; 500; 600). This other relationship information (208) represents the spatial relationship between the other first object and the other second object in the vehicle's surrounding environment (106). In the use (306) step, the other relationship information (208) is also used to clarify the vehicle's surrounding environment (106). The use (306) step includes generating a relationship value between the relationship information (206) and the other relationship information (208) in order to clarify the vehicle's surrounding environment (106) when using the relationship value. In the use (306) step, the relational information is used in the vehicle's driver assistance system to clarify the vehicle's surrounding environment, or the result information is used to provide the vehicle's driver assistance system with the result of clarifying the vehicle's surrounding environment.

2. The method (300) according to claim 1, wherein, In the use (306) step, the relationship information (206) is used for the lateral steering function and / or longitudinal adjustment of the vehicle (100).

3. The method (300) according to claim 1 or 2, comprising the step of assigning the first object (115) and / or the second object (116) to an object category, wherein, In the use (306) step, the object category is also used to clarify the vehicle’s surrounding environment (106).

4. The method (300) according to claim 1 or 2, comprising the step of constructing occlusion information (212), the occlusion information representing that the first object (115) in the image (204; 400; 500; 600) is occluded by the second object (116), wherein, In the use (306) step, the occlusion information (212) is also used to clarify the vehicle’s surrounding environment (106).

5. The method (300) according to claim 1 or 2, wherein, In the use (306) step, the relationship value is used to follow another vehicle (412).

6. The method (300) according to claim 1 or 2, comprising the step of determining the orientation (214) of one or the other of the objects (115, 116) relative to the vehicle (100) by an optical sensor (118), wherein, In the use (306) step, the orientation (214) is also used to clarify the environment around the vehicle (106).

7. An apparatus (120) for elucidating the vehicle surrounding environment (106) of a vehicle (100), configured to implement the steps of the method (300) according to any one of the preceding claims in a corresponding unit.

8. A vehicle (100) comprising an optical sensor (118), a driver assistance system (114), and the device (120) according to claim 7, wherein, The device (120) is electrically connected to the optical sensor (118) and / or the driver assistance system (114).

9. A machine-readable storage medium having a computer program stored thereon, the computer program being configured to implement the method (300) according to any one of claims 1 to 6.

Images