Method and assistance system for controlling a motor vehicle based on object and area recognition and correspondingly configured motor vehicle
By utilizing two independent environmental sensors for object and area recognition and implementing a consistency check, the method improves the safety and reliability of automated vehicle control, addressing the issue of inconsistent sensor data and reducing the risk of safety-critical errors.
Patent Information
- Application Number
- PCT/DE2024/100285
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-14
AI Technical Summary
Current automated vehicle control systems face inaccuracies in environmental detection, leading to potential safety-critical situations due to inconsistent results from single environmental sensors, which can result in incorrect recognition of road users and passable areas.
The method employs two independent environmental sensors operating on different principles to record data, with object recognition and area detection performed using distinct methodologies, and a consistency check is implemented to ensure accurate control decisions, initiating safety measures if results are inconsistent.
This approach enhances the safety and reliability of partially automated vehicle control by reducing the risk of incorrect decisions based on single sensor errors, ensuring consistent recognition of road users and passable areas, thereby preventing potentially hazardous situations.
Smart Images

Figure DE2024100285_14112024_PF_FP_ABST
Abstract
Description
[0001] Method and assistance system for controlling a motor vehicle based on object and area recognition and correspondingly equipped motor vehicle
[0002] The present invention lies in the field of automotive engineering and relates to a method and an assistance system for controlling a motor vehicle in at least partially automated operation. The invention further relates to a correspondingly configured motor vehicle.
[0003] Today, efforts are being made to automate the operation of motor vehicles. This typically involves sensors detecting the vehicle's surroundings in order to automatically guide the vehicle correctly along the roadway and avoid collisions. This process must become more precise and reliable the more responsibility is assigned to an automated vehicle control system or the higher the degree of automation. However, such environmental detection is not yet completely accurate and reliable in all situations, so there is still a need for further improvements.
[0004] As one approach, DE 11 2019 000 048 T5 addresses the determination of a drivable free space for autonomous vehicles. It provides a method in which recorded sensor data is applied to a neural network. This calculates a group of boundary points represented by the sensor data, which correspond to a boundary separating the drivable free space from a non-drivable space in a vehicle's environment. Locations in the environment are then determined that correspond to the group of boundary points. Furthermore, class designations are calculated for boundary points of the group of boundary points. Finally, an operation for steering the vehicle through the environment is executed depending on the locations and the class designations.
[0005] As a further approach, DE 102021 129 825 A1 deals with object detection and collision avoidance using a neural network. It provides a circuit for using a neural network to identify an object with which a vehicle may collide based on a difference between the size of an image of the object detected at a first time and a size of an image of the object detected at a subsequent time.
[0006] As a further approach, DE 102021 119 871 A1 describes a method for operating an automated driving function in a motor vehicle. In this method, individual objects in the vehicle's environment are detected from sensor data using an object classifier. A relationship classifier then detects a pairwise object relationship for some of the individual objects. Using an aggregation module, corresponding relationship data is aggregated across several consecutive individual images of the environment to form aggregation data that describe aggregated object relationships. This aggregation data is then provided for object tracking and / or trajectory planning.
[0007] The object of the present invention is to enable a particularly safe and reliable at least partially automated ferry operation of a motor vehicle.
[0008] This object is achieved by the subject matter of the main claim and the subsidiary claims or the independent patent claims. Further possible embodiments of the invention are disclosed in the subclaims, the description and the figures. Features, advantages and possible embodiments that are set out in the description for one of the subject matter of the independent claims are to be regarded at least analogously as features, advantages and possible embodiments of the respective subject matter of the other independent claims as well as of any possible combination of the subject matter of the independent claims, optionally in conjunction with one or more of the subclaims. The method according to the invention can be used to control a motor vehicle in at least partially automated operation. This can therefore be driving operation of the motor vehicle in at least autonomy level 2, for example according to SAE J3016.The method according to the invention comprises a plurality of method steps that can be carried out automatically, for example by an assistance system set up to carry out the method or for use in the method. In one method step of the method according to the invention, environmental data that characterizes, for example represents or maps, a respective environment of the motor vehicle is recorded by means of at least one first environmental sensor and at least one second environmental sensor. The first environmental sensor and the second environmental sensor operate independently of one another and according to different functional principles. The environmental data therefore originate from at least two different data sources or comprise at least two different types of environmental data. The recorded environmental data can each be detected or received by the assistance system.
[0009] In a further method step of the method according to the invention, object detection is performed for an area of the environment relevant for an upcoming automated driving decision for a planned driving maneuver of the motor vehicle, i.e., for example, for a corresponding driving decision of a system for the at least partially automated control of the motor vehicle. This object detection is performed based on environmental data from the first environmental sensor, i.e., accordingly, using a first methodology for detecting other road users in the relevant area that are relevant for the driving decision. The driving decision can, for example, concern or include whether, when, or how the planned driving maneuver of the motor vehicle should be carried out.This makes it possible, for example, to decide whether the motor vehicle should continue directly along its route, i.e., continue moving, or whether it should first be stopped. Depending on the situation, controlling the motor vehicle according to the planned driving maneuver can therefore mean or include, for example, automatically continuing directly in a planned manner or temporarily slowing down or stopping the motor vehicle, for example, to yield right of way to another detected road user, i.e., allowing them to pass first. A corresponding planned automated stopping can then be part of the driving maneuver or part of the intended execution of the driving maneuver.Another road user, i.e. a road user different from the motor vehicle, may be relevant for the driving decision, for example, if he has the right of way with respect to the motor vehicle or if he would or could be obstructed by the motor vehicle when carrying out the driving maneuver or if he would or could collide with it.
[0010] Furthermore, area detection is performed for the area of the environment relevant for the upcoming automated driving decision for the planned driving maneuver. This area detection is performed based on environmental data from the second environmental sensor, i.e., in particular, using a second methodology that differs from the first methodology, to detect a clear, drivable area in the relevant area or to detect the relevant area as a clear, drivable area.
[0011] If there is another road user in the relevant area, they can be detected based on the environmental data from the first environmental sensor, i.e. using the first method. If, on the other hand, there is no other road user in the relevant area, a clear, drivable area can be detected there based on the environmental data from the second environmental sensor, i.e. using the second method. In practice, however, false detections can occur. For example, an actually present other road user may not be detected, i.e. be overlooked. Likewise, a clear, drivable area may be detected incorrectly, even though there is actually another road user there, or no clear, drivable area may be detected even though one is present.To achieve the highest possible safety and robustness in the at least partially automated control of the motor vehicle, the motor vehicle should not be guided or moved into a potentially safety-critical situation based on such a false detection. However, detecting false detections based on sensor data from a single environmental sensor using this environmental data itself can be problematic.
[0012] According to the invention, in a further process step of the inventive
[0013] Therefore, the driving decision is made depending on whether the object detection and the area detection deliver consistent results with each other or with each other. If this is the case, i.e., if the object detection and area detection deliver consistent results, the motor vehicle is controlled at least partially automatically according to the consistent results in accordance with the planned driving maneuver. Otherwise, i.e., if the object detection and area detection deliver inconsistent, for example, contradictory, results, a specified or predefined safety measure or safety reaction is automatically initiated or executed to avoid a collision with another road user who may be present in the relevant area or to avoid another safety-critical situation.This safety measure or safety reaction can, for example, mean or include the motor vehicle being automatically stopped, or control of the motor vehicle being transferred to the driver, or confirmation from the driver to carry out the planned driving maneuver, in particular to continue moving the motor vehicle, and / or the like. Controlling the motor vehicle in accordance with the planned driving maneuver when the results of object detection and area detection are consistent with one another can, in particular, mean that the motor vehicle is controlled automatically and without consulting the driver. In this case, the motor vehicle can, for example, be set in motion again automatically after stopping and without prior confirmation from the driver in order to carry out or complete the planned driving maneuver.However, after the vehicle has been stopped as a safety reaction, the vehicle would not be put into automatic motion again, or not without prior confirmation from the driver.
[0014] In the present case, a consistency check is proposed based on different sensors or on different sensor or environmental data originating from different sensors, each of which is carried out for a specific spatial area. This means that the motor vehicle can only be controlled automatically, in particular moved further, if the relevant area is correctly detected or if both the object detection and the area detection are simultaneously faulty in such a way that they nevertheless deliver consistent results. The latter is extremely unlikely due to the different functional principles of the two environmental sensors. Thus, the present invention can provide additional safeguards for the driving decision, thus increasing safety during at least partially automated control of the motor vehicle and thus also in traffic overall.
[0015] An area relevant for the driving decision or its position or location can generally be known, for example, based on digital map data and the current position of the motor vehicle. For example, intersections, junctions, lanes or lane courses and / or the like can be entered as such in the map data and, if appropriate, can be classified as a relevant area. In the at least partially automated operation of the motor vehicle, for example, in an automatically generated environment model or a detection list or the like, not only objects detected by sensors, such as other road users or obstacles or the like, but also areas or traffic spaces explicitly recognized as such that are freely drivable, i.e. areas or traffic spaces suitable and intended for driving, can be entered.It can then be checked, for example, whether both another road user and a clear, drivable area were detected for the same spatial area or at the same location or position, or whether only another road user or only a clear, drivable area was detected there, or whether neither another road user nor a clear, drivable area was detected. Based on this, it can then be checked whether there is consistent environmental detection for these areas or whether there are any inconsistencies or conflicts. This can be done for all areas, locations or positions that are classified as relevant or potentially relevant for the respective upcoming driving decision, i.e. an respective upcoming driving maneuver. Which areas or locations are relevant in each case can be determined in the respective environment, for example from road orLane layouts, applicable right-of-way rules, expected or measured speeds of other road users, the speed of the motor vehicle, the planned route of the motor vehicle and / or the like.
[0016] The present invention also relates to an assistance system for a motor vehicle for the at least partially automated control of the motor vehicle or a motor vehicle. The assistance system according to the invention has an input interface for acquiring environmental data from at least two different environmental sensors. The assistance system further comprises a data processing device for processing the environmental data and for generating a control signal for controlling the motor vehicle equipped with the assistance system or a device of this motor vehicle depending on the environmental data or a respective result of the processing of the environmental data. The assistance system further has an output interface for outputting the respective control signal. The input interface and the output interface can be separate interfaces or combined or integrated into a common interface.The input interface and the output interface can each be implemented entirely or partially in hardware and / or software.
[0017] The assistance system according to the invention is configured for the method according to the invention, i.e., for use in the method according to the invention. For this purpose, the data processing device can comprise a processing device, i.e., for example, a microchip, microprocessor, microcontroller, or the like, and a computer-readable data memory coupled thereto. A corresponding operating or computer program can be stored in this data memory, which encodes or implements the method steps, measures, or sequences, or corresponding control instructions, described in connection with the method according to the invention and intended for execution in the assistance system or by means of the assistance system. This operating or computer program can then be executable by means of the processing device in order to execute the corresponding method or to effect its execution.The assistance system according to the invention can therefore, in particular, be the assistance system mentioned in connection with the method according to the invention or correspond to it. The assistance system according to the invention can, for example, be designed as a vehicle control unit or as a module of a vehicle control unit or the like.
[0018] In one possible embodiment of the present invention, the assistance system is configured to determine, as the area of the environment relevant for the respective driving decision, an area into which the motor vehicle is to enter, for example according to automated trajectory or driving maneuver planning or a current navigation route or the like, or in which another road user with right of way over the motor vehicle - if any - would or should be located. A corresponding area into which the motor vehicle is to enter can, for example, be a central area of an intersection to be driven through or crossed, or a part of a road currently being used by the motor vehicle that leads past an intersection, or a section of a lane adjacent to a lane currently being used by the motor vehicle, or the like.An area in which another road user with right-of-way over the motor vehicle would or should be located can, for example, be an area of an intersection adjacent to the road currently being traveled by the motor vehicle, or a part of a road that crosses the road currently being traveled by the motor vehicle and is superior to it in the right-of-way hierarchy, and / or similar. The size, length, or extent of the relevant area can, for example, depend on the speeds of other road users expected or present there, whereby the relevant area can be correspondingly larger at higher speeds.
[0019] The embodiment of the present invention proposed here allows relevant areas that are particularly critical for a large number of driving maneuvers and potentially dangerous situations in practice to be defined and recognized particularly easily and reliably and to be taken into account for vehicle control.
[0020] In a further possible embodiment of the present invention, the assistance system is configured to classify the results of the object detection and the area detection as being consistent with one another, to make the driving decision for automatically controlling the motor vehicle in accordance with the planned driving maneuver, and to generate and output the corresponding control signal if or when no other road user was detected in the relevant area during object detection and the relevant area was detected as a free, drivable area during area detection, or if another road user was detected in the relevant area during object detection and the relevant area was not detected as a free, drivable area during area detection.This can reduce the probability that the motor vehicle will be automatically moved on because another road user who is actually relevant was mistakenly overlooked by or during object recognition, or was classified as irrelevant. This can be achieved in particular by the necessary positive recognition of the clear, drivable area. This is the case because such a clear, drivable area is typically not recognized, or not recognized with sufficient reliability or confidence (e.g., above a predefined threshold), if another road user is located there, even if this user is not recognized as such. If the other road user has been recognized and the relevant area has therefore consistently not been recognized as a clear area, this can result in a correspondingly improved orA particularly reliable database is available for the decision that the motor vehicle must be stopped automatically. In this case, this automated stopping can be carried out based on the correspondingly improved or confirmed by consistency knowledge of the presence of the other road user, rather than as part of a safety response based on excessive uncertainty. Therefore, the motor vehicle can then continue to be controlled automatically, for example, without involving its driver.
[0021] In a further possible embodiment of the present invention, the assistance system is configured to classify the results of the object detection and the area detection as inconsistent with one another, to make the driving decision to execute the safety measure, and to generate and output the corresponding control signal if or when another road user was detected in the relevant area during object detection and the relevant area was detected as a clear, drivable area during area detection. Since a relevant other road user and a clear, drivable area cannot be present at a given point at the same time, the criterion proposed here detects a corresponding detection conflict, misdetection, or misinterpretation particularly reliably.This can be particularly the case if both the object detection and the area detection individually demonstrate or indicate a sufficient detection probability or detection confidence. By initiating or executing the safety measure in this case, a particularly high level of safety can be achieved. For example, there is no need to make a potentially error-prone or risky decision as to whether the object detection or the area detection should be classified as reliable and used as the basis for the driving decision.
[0022] In a further possible embodiment of the present invention, the
[0023] The assistance system is set up to classify the results of the object detection and the area detection as inconsistent with one another, to make the driving decision to carry out the safety measure and to generate and output the corresponding control signal if no other road user was detected in the relevant area during object detection and the relevant area was not detected as a free, drivable area during area detection. A relevant area can always be defined or regarded as an area that is fundamentally drivable, i.e. in which another road user relevant for the respective driving decision could be located. Accordingly, there must be an incorrect detection in either the object detection or the area detection, and thus an inconsistency, if neither another road user nor a free, drivable area is detected in the relevant area.Accordingly, such an inconsistency can be detected even if the results of object detection and area detection do not directly contradict each other, for example, without further a priori assumptions regarding the relevant area, or do not appear to directly contradict each other. The embodiment of the present invention proposed here allows even such seemingly conflict-free or contradictory situations to be detected and handled in a particularly reliable manner.
[0024] In a further possible embodiment of the present invention, the assistance system is configured to perform object detection based on environmental data from active environmental scanning, in particular by means of radar or lidar, of the respective environment or the respective relevant area. The environmental data originating from the first environmental sensor can therefore be or include, in particular, radar echoes or lidar reflection points. Accordingly, a radar sensor or a lidar sensor, for example, can be used as the first environmental sensor. Such active environmental scanning, i.e. based on the transmission of signals and the detection of corresponding echoes or reflections, can enable particularly reliable and precise object detection, for example, at least substantially independent of ambient brightness.This enables correspondingly reliable automated control or guidance of the motor vehicle, and excessively frequent initiation or execution of the specified safety measure can be avoided. This can be seen, for example, in comparison to object recognition or the like, in particular based exclusively on camera images. In a further possible embodiment of the present invention, the assistance system is configured to perform area recognition based on a semantic segmentation of a camera image of the respective environment or the respective relevant area, captured as part of the environmental data.A free, drivable surface typically forms a contiguous, uniform area which is of a significant size, i.e., at a distance relevant for an impending driving maneuver of the motor vehicle, typically comprises a large number of pixels of the corresponding camera image. Such structures can be recognized particularly precisely, reliably, and robustly on the basis of semantic segmentation, in particular pixel-by-pixel segmentation. In particular, such drivable surfaces can thus be recognized as such reliably, precisely, and robustly even if, for example, no structural features are present there which would lead to a reliable reflection of signals emitted from an active environmental scanning system. Ultimately, this makes it possible overall to use the method according to the invention to enable particularly reliable automated control of the motor vehicle, wherein, in particular, excessive initiation orImplementing the specified safety measure can be avoided.
[0025] In a further possible embodiment of the present invention, the assistance system is designed to generate and output the control signal in the event of mutually inconsistent results of object detection and area detection in such a way that, as a predetermined safety measure, the motor vehicle is automatically stopped and / or confirmation is requested or interrogated from the driver to steer, in particular to continue moving, the motor vehicle in accordance with the planned driving maneuver. The former can contribute to improved safety regardless of the driver's level of attention, since stationary traffic is typically associated with a lower safety risk than moving traffic, whereas the latter - at least if the driver has provided confirmation to steer or continue moving the motor vehicle - can contribute to improved safety.: This allows the motor vehicle to continue moving - at the same time a high level of safety and driving comfort as well as a particularly high level of efficiency in traffic situations or traffic flow can be achieved, since the motor vehicle then does not have to stop - possibly unnecessarily. However, the control signal can also cause the motor vehicle to slow down. This can also increase safety and give the driver more time to record and correctly assess the respective traffic situation and answer or react correctly to the query. Furthermore, it can be provided that if the driver does not respond to the query or denies the confirmation, i.e. rejects the automated control or further movement of the motor vehicle in accordance with the planned driving maneuver, the motor vehicle is automatically stopped by the control signal or a further control signal then generated.This can also contribute to safety and comfort, as the driver then does not have to, for example, stop the vehicle manually, which could take more time than stopping the vehicle automatically.
[0026] The present invention also relates to a motor vehicle that is configured for at least partially automated operation. The motor vehicle according to the invention comprises an environmental sensor system with at least a first environmental sensor and a second environmental sensor, which operate independently of one another and according to different functional principles. The motor vehicle according to the invention is equipped with the assistance system according to the invention and / or is configured to carry out the method according to the invention, in particular automatically. The motor vehicle according to the invention can therefore be or correspond to the motor vehicle mentioned in connection with the method according to the invention and / or in connection with the assistance system according to the invention. The motor vehicle according to the invention can therefore have some or all of the corresponding properties and / or features.The motor vehicle according to the invention can, for example, comprise a radar device and / or lidar device as well as a camera as part of the environmental sensor system. Likewise, the motor vehicle according to the invention can, for example, comprise a device or interface for outputting a query to the driver and for detecting a corresponding response or reaction from the driver.
[0027] Further features of the invention may emerge from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as the features and combinations of features shown below in the description of the figures and / or in the figures alone, can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the invention. The drawing shows:
[0028] Fig. 1 is a partial schematic overview of a traffic scene to illustrate a method for safely controlling a motor vehicle in an at least partially automated ferry operation; and
[0029] Fig. 2 shows an exemplary schematic flow chart for a corresponding method for controlling the motor vehicle.
[0030] Fig. 1 shows a partially schematic overview (not to scale) of a traffic scene with a road 1 and an intersection 2 that leads into road 1. In the situation depicted here, a motor vehicle 3 is traveling on road 1 in a direction of travel 4, indicated here by an arrow, toward the intersection 2. The next driving maneuver of motor vehicle 3 may be to follow road 1 past the intersection 2. By way of example, this is a situation in which the "right-before-left" rule applies. Thus, another vehicle approaching the intersection 2 and wishing to turn into road 1 from there would have right of way over motor vehicle 3.
[0031] For automated operation of motor vehicle 3 or automated execution of the planned driving maneuver, this must be taken into account and it must be reliably detected whether another vehicle is approaching at junction 2.
[0032] In previous approaches, the focus has been on asking or determining whether an object, such as another vehicle or another road user or the like, is located at a corresponding location, here for example in a schematically indicated relevance area 5, to which the motor vehicle 3 must react, and which is therefore relevant for a driving maneuver or a corresponding driving decision of the motor vehicle 3. The relevance area 5 can be limited in its size or extent, in particular into the junction 2, as indicated here. This is the case because another vehicle that is closer at the junction 2 but further away would not reach the road 1 before the motor vehicle 3 has passed the junction 2.Such a different vehicle would then not be relevant for the corresponding driving maneuver, in this case, for the driver's decision as to whether motor vehicle 3 should continue driving or stop. An example of a simpler automated vehicle control system is automatic distance control (ACC, Adaptive Cruise Control), in which an object is automatically detected or identified as a vehicle in front and then always reacts to precisely this object with appropriate control.
[0033] In the present example of a right-of-way situation, motor vehicle 3 may have to stop if another object or road user with right-of-way is detected in the relevant area 5. If, however, no such other road user is present there, motor vehicle 3 can or should then independently drive past the junction 2 in the direction of travel 4. In such a situation, misrecognition, i.e., failure to see another road user with right-of-way who is actually present in the relevant area 5, could lead to a potentially dangerous situation. Since an automated control of motor vehicle 3 must therefore assume more responsibility for handling or managing such situations, there is also a requirement for additional protection against corresponding detection errors.The described situation, in which no other road user with right of way or relevant information is mistakenly detected, even though it actually exists, and motor vehicle 3 is therefore automatically moved on, must be avoided for safety reasons. This is problematic if motor vehicle 3 or its automated control system relies solely on the criterion of whether or not a corresponding object has been detected, since a corresponding detection error then poses a safety risk, as the detection error can lead to incorrect, safety-critical behavior of motor vehicle 3.
[0034] In order to counteract this problem, the motor vehicle 3 in the present case is equipped with a correspondingly adapted or expanded assistance system 6. The motor vehicle 3 also has an environmental sensor system with a first environmental sensor 7, which is designed here as a radar or lidar, for example, and a second environmental sensor 8, which is designed here as a camera, for example. The environmental sensors 7, 8 operate independently of one another based on different functional principles. The environmental sensors 7, 8 can be coupled to the assistance system 6 via an interface 9, for example, via an on-board electrical system of the motor vehicle 3. The assistance system 6 can then record corresponding sensor or environmental data from the first environmental sensor 7 and the second environmental sensor 8. In order to process this sensor or environmental data,In order to receive environmental data and to generate a control signal as a function thereof, the assistance system 6 has, as indicated schematically here, a processor 10 and a computer-readable data memory 11.
[0035] To illustrate or explain the functionality of the assistance system 6, Fig. 2 shows an exemplary schematic flow chart 16 for a corresponding method for controlling the motor vehicle 3 by the assistance system 6 or by means of or using the assistance system 6. In a method step S1, the aforementioned environmental data are recorded and / or acquired by the assistance system 6, for example via the interface 9. In the present case, environmental data characterizing or depicting, for example, or at least the relevant area 5 are recorded or marked by the assistance system 6 for further processing.
[0036] In a method step S2a, object detection can then be carried out based on the environmental data of the first environmental sensor 7 for the relevance area 5. Depending on the situation, for example, an external vehicle 12 can be detected in the relevance area 5, or no such external vehicle 12 can be detected. The external vehicle 12 that is detected can be relevant for further movement, i.e., the upcoming planned driving maneuver of the motor vehicle 3 and thus a corresponding driving decision, since or when it approaches road 1 at the junction 2 in a direction of travel 13 of the external vehicle, indicated here by an arrow. If, on the other hand, the external vehicle 12 were to move away from road 1 in the opposite direction, for example, it could possibly be classified as irrelevant for the driving decision or the further movement of the motor vehicle 3, even though it may still be within the relevance area 5.
[0037] For example, in parallel thereto, in a method step S2b, a semantic segmentation can be carried out based on the environmental data of the second environmental sensor 8 or, if this has already been carried out, for example, by the second environmental sensor 8 itself, an area detection can be carried out to recognize the relevance area 5 as a drivable area 14. Depending on the situation, it can therefore be recognized that the relevance area 5 is completely or at least substantially or at least to a predetermined minimum extent or up to a maximum of a part that is smaller than a predetermined maximum size, occupied or filled by the free drivable area 14 or not. The possibly predetermined minimum portion can, for example, be so large in each case that any remaining portion of the relevance area 5 would be significantly smaller than a possible other road user.Likewise, the maximum size, which may be specified, can, for example, comprise or represent an area or a number of pixels that is(are) significantly smaller than that of another road user, such as the other vehicle 12, that may be present in the relevance area 5. This can improve the robustness of the method against misrecognition or misclassification of individual pixels. In principle, during semantic segmentation, each pixel, i.e., each image point of a camera image captured as part of the environmental data of the second environmental sensor 8, can be assigned the class of the structure of which this pixel forms or represents a part. Such possible classes can, for example, include "road surface" or "drivable area" or "motor vehicle" or the like.
[0038] In a method step S3, it is checked whether the results of the object detection in method step S2a and the area detection in method step S2b are consistent with one another. Based on this, the driving decision for the planned driving maneuver of the motor vehicle 3 is then made and a corresponding control signal is generated. This allows the above-mentioned problem of conventional approaches, for example those based only on object detection by means of the first environmental sensor 7, to be solved by applying or incorporating semantic segmentation or the corresponding area detection. Here, the criterion used for the driving decision is not only whether a relevant other vehicle 12 was detected in the relevance area 5. Rather, an additional criterion used or incorporated is whether the relevance area 5 was at least essentially completely recognized as a free, drivable area 14 or not.This can reduce the risk of incorrect behavior by the motor vehicle 3 due to a recognition error. If, for example, the check in method step S3 shows that no other vehicle 12 was detected in the relevance area 5 in a mutually consistent manner during the object recognition in method step S2a and, moreover, the relevance area 5 was recognized as a free, drivable area 14 during the area recognition in method step S2b, the driving decision to continue moving the motor vehicle 3 can then be made. Accordingly, the assistance system 6 can then generate a corresponding control signal and, in a method step S4a, the motor vehicle 3 can be moved further in an automated manner, in this case, continue driving or being guided past the junction 2 in the direction of travel 4 along the road 1. Likewise, the results of the object recognition and the area recognition can be recognized or recognized as being consistent with one another.be classified if the other vehicle 12 was detected in the relevant area 5 and the relevant area 5 was not detected as a free area 14. In this case, the control signal can be generated such that the motor vehicle 3 also automatically carries out the planned driving maneuver in method step S4a, but first brakes or stops before the junction 2 until the other vehicle 12 has turned into the street 1, and only then is it guided past the junction 2.
[0039] On the other hand, an inconsistency can be detected in method step S3 if, as part of the object detection in the relevance area 5, the relevant other vehicle 12 was detected and, moreover, as part of the area detection, the relevance area 5 was detected as a free, drivable area 14. Likewise, an inconsistency can be detected in method step S3 if, as part of the object detection in the relevance area 5, no relevant other vehicle 12 was detected and, as part of the area detection, the relevance area 5 was not detected as a free, drivable area 14. In these cases, a correspondingly different control signal can be generated by the assistance system 6. This allows a predefined safety reaction or safety measure to be initiated and executed in a method step S4b.In this case, for example, the assistance system 6 can obtain confirmation from the driver 15 of the motor vehicle 3 to continue moving or the motor vehicle 3 can be automatically stopped before reaching the relevant area 5, in this case before reaching the junction 2.
[0040] In situations such as the traffic scene depicted here with junction 2, or even more complex intersection situations, the complexity and thus also the requirements, for example, for correct object detection or environmental recognition, increase considerably compared to a pure following situation in which motor vehicle 3, for example, simply drives straight ahead and automatically regulates its distance from a vehicle in front. In such more complex situations, there may be one or more other road users who may be moving in different or changing directions relative to the vehicle 3 itself, and who may need to be taken into account.In such situations, the behavior, i.e. the automatic control of the motor vehicle 3, can be improved compared to conventional approaches if, as described, in addition to the detection of objects, such as any other vehicle 12 that may be present, and their mapping, i.e. their placement or localization in a digitized map or in a digital environment model or the like, areas or road sections known, for example, from a corresponding digital map of the respective environment are additionally evaluated or taken into account with regard to the criterion of recognition as a free, drivable area 14. A corresponding geometry of the respective environment, here for example the positions and courses of the road 1 and the junction 2, can be known from the respective predetermined digital map orso that, based on this, the relevance area 5 to be considered for the next or upcoming driving decision can be determined in each situation. Likewise, such relevance areas 5 can, for example, already be contained or specified in the digital map, i.e., corresponding predefined map data. This allows the assistance system 6 or the automated control system of the motor vehicle 3 to reliably recognize, resolve, or handle corresponding situations in a particularly automated manner.
[0041] Another exemplary situation in which the described method can be applied can include a lane change as an impending or planned driving maneuver of the motor vehicle 3. For example, the motor vehicle 3 in the present case is traveling in a first lane 17 on road 1 and is to change from this to an adjacent second lane 18 on road 1. For this purpose, a lane change relevance area 19, indicated schematically here, can be taken into account. For example, there must be a sufficiently large gap or a sufficiently large free area into which the motor vehicle 3 can enter, i.e. change, while maintaining the necessary safety distances both from a vehicle traveling ahead and from a vehicle traveling behind in the second lane 18.In addition, it must be considered whether another vehicle is approaching from behind in the direction of travel 4 on the second lane 18 at a correspondingly high speed, which would be forced to brake impermissibly if the motor vehicle 3 were to change lanes into the second lane 18 and is therefore relevant to the lane change, i.e., the corresponding driving maneuver or the corresponding driving decision of the motor vehicle 3. For this purpose, the lane change relevance area 19 can extend further to the rear of the motor vehicle 3 than is only partially shown here, i.e., opposite to the direction of travel 4.
[0042] It can be particularly challenging for automated environmental recognition to recognize whether or not the lane change from the first lane 17 to the second lane 18 can be carried out safely if no gap bordered by other vehicles on either side is detected or is present, but only or at most one other vehicle is detected or present that borders the space or area available for the lane change maneuver to the front, i.e. in the direction of travel 4. In such a situation, if the corresponding area is clear to the rear or against the direction of travel 4, it can be problematic if a corresponding driving decision to carry out the lane change maneuver would be based solely on the fact that the object detection system has not detected a relevant object there.This can also be circumvented or resolved in this case by performing area detection for the lane change relevance area 19. The lane change maneuver can then only be executed if no other relevant road user has been detected in the lane change relevance area 19 and, in addition, the
[0043] Lane change relevance area 19, in particular based on a corresponding semantic segmentation, was recognized as a drivable free area or road surface that is sufficiently large for the lane change maneuver.
[0044] The area relevant in a specific situation or for a specific driving maneuver or a specific driving decision can be determined not only based on corresponding map data, but also, for example, based on environmental detection. In the example of a lane change maneuver, the lane change relevance area 19 can be determined, for example, based on automatic detection of lane markings and / or road boundaries and / or the like. Overall, the examples described demonstrate how, for example, the use of semantic segmentation for area detection and independent object detection, for example in yield-right situations, can achieve increased traffic safety with at least partially automated vehicle guidance.
[0045] List of reference symbols
[0046] 1 Street
[0047] 2 Junction
[0048] 3 Motor vehicle
[0049] 4 Direction of travel
[0050] 5 Relevance area
[0051] 6 Assistance system
[0052] 7 first environmental sensor
[0053] 8 second environmental sensor
[0054] 9 Interface
[0055] 10 processor
[0056] 11 Data storage
[0057] 12 third-party vehicle
[0058] 13 Direction of other vehicle travel
[0059] 14 drivable areas
[0060] 15 drivers
[0061] 16 Schedule
[0062] 17 first lane
[0063] 18 second lane
[0064] 19 Lane change relevance area
Claims
Patent claims 1. Method (16) for controlling a motor vehicle (3) in an at least partially automated operation, wherein automatically - by means of at least one first environmental sensor (7) and one second environmental sensor (8), which operate independently of one another and according to different functional principles, environmental data are recorded which characterise a respective environment of the motor vehicle (3), - for an area (5, 19) of the environment relevant for an upcoming automated driving decision for a planned driving maneuver of the motor vehicle (3) - based on environmental data from the first environmental sensor (7), an object detection is carried out to detect other road users (12) relevant for the driving decision, and - based on environmental data from the second environmental sensor (8), an area detection is carried out to detect the relevant area (5, 19) as a free, drivable area (14), and - the driving decision is made depending on whether the object detection and the area detection deliver consistent results, wherein if this is the case, the motor vehicle (3) is automatically controlled according to the planned driving maneuver and otherwise a predetermined safety measure is carried out to avoid a collision with another road user (12) possibly present in the relevant area (5, 19).
2. Assistance system (6) for a motor vehicle (3) for the at least partially automated control thereof, comprising an input interface (9) for detecting environmental data from at least two different environmental sensors (7, 8), a data processing device (10, 11) for processing the environmental data and for generating a control signal as a function of the environmental data, and an output interface (9) for outputting the control signal, wherein the assistance system (6) is set up for the method (16) according to claim 1.
3. Assistance system (6) according to claim 2, characterized in that the assistance system (6) is set up to determine, as the area (5, 19) relevant for the respective driving decision, an area (5, 19) into which the motor vehicle (3) is to enter or in which another road user (12) having right of way over the motor vehicle (3) would be located.
4. Assistance system (6) according to claim 2 or 3, characterized in that the assistance system (6) is designed to classify the results of the object recognition and the area recognition as consistent with each other, to make the driving decision for controlling the motor vehicle (3) according to the planned driving maneuver and to output the corresponding control signal if - during object detection in the relevant area (5, 19) no other road user (12) was detected and during area detection the relevant area (5, 19) was detected as a free drivable area (14), or - during object detection in the relevant area (5, 19) another road user (12) was detected and during area detection the relevant area (5, 19) was not detected as a free drivable area (14).
5. Assistance system (6) according to one of claims 2 to 4, characterized in that the assistance system (6) is set up to classify the results of the object detection and the area detection as inconsistent with one another, to make the driving decision to carry out the safety measure and to output the corresponding control signal if, during the object detection in the relevant area (5, 19), another road user (12) was detected and, during the area detection, the relevant area (5, 19) was detected as a free drivable area (14).
6. Assistance system (6) according to one of claims 2 to 5, characterized in that the assistance system (6) is configured to classify the results of the object detection and the area detection as inconsistent with each other, to make the driving decision to carry out the safety measure and to output the corresponding control signal if, during object detection, no other road user (12) was detected in the relevant area (5, 19) and, during area detection, the relevant area (5, 19) was not detected as a free, drivable area (14).
7. Assistance system (6) according to one of claims 2 to 6, characterized in that the assistance system (6) is designed to carry out the object recognition based on environmental data from an active environmental scanning, in particular by means of radar or lidar.
8. Assistance system (6) according to one of claims 2 to 7, characterized in that the assistance system (6) is configured to carry out the area recognition based on a semantic segmentation of a camera image of the respective environment captured as part of the environmental data.
9. Assistance system (6) according to one of claims 2 to 8, characterized in that the assistance system (6) is designed to generate the control signal as a control signal for automatically stopping the motor vehicle (3) and / or for requesting confirmation for controlling, in particular for moving on, the motor vehicle (3) in accordance with the planned driving maneuver of its driver (15) in the event of inconsistent results of the object detection and the area detection.
10. Motor vehicle (3) which is designed for at least partially automated operation and has an environmental sensor system with at least one first environmental sensor (7) and one second environmental sensor (8) which operate independently of one another and according to different functional principles, wherein the motor vehicle (3) has an assistance system (6) according to one of claims 2 to 9 and / or is arranged to carry out the method (16) according to claim 1.
Citation Information
Patent Citations
Method and processor circuit for operating an automated driving function with object classifier in a motor vehicle, as well as motor vehicle
DE102021119871A1
OBJECT DETECTION AND COLLISION AVOIDANCE USING A NEURAL NETWORK
DE102021129825A1
DEFINITION OF A DRIVABLE CENTER FOR AUTONOMOUS VEHICLES
DE112019000048T5
Situation-dependent definition of observation areas for at least partially autonomously operated motor vehicles
DE102020213514A1
Obstacle Detection by Object Fusion for Automobile
FR3056530A1
