Method for detecting and classifying objects in road traffic
By combining neural networks and symbol monitoring algorithms for redundancy detection, the vulnerability of single-sensor modalities and neural networks to attacks is solved, achieving high reliability and safety in the detection and classification of road traffic signal systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ROBERT BOSCH GMBH
- Filing Date
- 2021-07-26
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, single sensor modalities are insufficient for reliably detecting and classifying road traffic signal systems. Neural networks are vulnerable to attacks and their decision-making processes are opaque, leading to safety and reliability issues.
Detection and classification are performed by combining neural networks and redundant symbol monitoring algorithms. The symbol monitoring algorithm is based on explicit training examples, independent of the neural network, and performs consistency checks to ensure the accuracy of the results. The additional symbol monitoring algorithm also prevents attacks.
It improves the reliability of detection and classification in road traffic, prevents neural network attacks, and ensures the safety and reliability of autonomous vehicles.
Smart Images
Figure CN116075867B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for detecting and classifying at least one object in road traffic. Background Technology
[0002] Within the framework of controlling autonomous vehicles, the detection of traffic signal systems is a safety-related aspect. For example, a erroneously detected signal requiring the autonomous vehicle to stop could lead to sudden braking within the control framework, potentially causing a rear-end collision. Conversely, the signal might go unrecognized, potentially leading to a traffic violation and also an accident. For this reason, a method for detecting and classifying traffic signal systems should be designed to prevent accidental or inappropriate driving maneuvers. The corresponding detection system must also be protected against misuse and attacks.
[0003] Typically, object detection can be broken down into involving various sensors, where the relatively higher reliability of a second sensor can compensate for the lower reliability of the first. However, this is not possible when detecting traffic signal systems in road traffic because only one sensor mode—the camera—can capture the colors of the traffic signal system's signal generator by providing a color image of the system. For this reason, a minimum level of integrity must be ensured for the detection of traffic signal systems via cameras. This can be achieved, for example, by performing detection in a redundant manner.
[0004] Neural networks are known to be used for detecting and classifying traffic signal systems in camera images. The advantages of neural networks are that they provide a high probability of object recognition and produce only a small number of false positives. However, neural networks also have disadvantages. For example, neural networks are a technology whose characteristics have not been fully traced. The decision-making process of neural networks, for example, cannot be analyzed and explained. Therefore, it is unclear what prompts the neural network to make certain decisions. Even if a neural network can be trained and tested with the help of a comprehensive dataset, it may still behave unexpectedly due to rare or atypical traffic scenes or conditions not fully represented in the training and testing datasets, or due to noise and / or artifacts in the images.
[0005] Furthermore, neural networks are vulnerable to attacks using artificial patterns that can cause false detections and / or classifications. Such patterns can, for example, be printed on films and affixed to the windshields of motor vehicles and / or to the signal generators of traffic signal systems. However, these patterns can also be generated virtually and overlaid on camera images. Summary of the Invention
[0006] The object of the present invention is to: describe an improved method for detecting and classifying at least one object in road traffic; provide a system configured to perform the method; provide a computer program comprising instructions that, when executed by a computer, cause the computer to perform the method; and a machine-readable storage medium having the computer program stored thereon. The object is achieved by: a method for detecting and classifying at least one object in road traffic having the features of the respective independent claims; a system configured to perform the method having the features of the respective independent claims; a computer program having the features of the respective independent claims, comprising instructions that, when executed by a computer, cause the computer to perform the method; and a machine-readable storage medium having the features of the respective independent claims having the computer program stored thereon. Advantageous extensions are described in the dependent claims.
[0007] A method for detecting and classifying at least one object in road traffic includes the following steps: First, sensor data from sensors is provided. At least one object is detected and classified based on the sensor data using a neural network. Additionally, the object is detected and classified based on the sensor data using a redundant symbol monitoring algorithm. The consistency between the neural network and the symbol monitoring algorithm in their detection and classification of the object is checked.
[0008] The object can be, for example, a traffic signal system with signal generators, provided for traffic management in road traffic. However, the object can also be a sign, a motor vehicle, a pedestrian, or a cyclist. This symbol monitoring algorithm is based on symbolic machine learning methods. That is, it is a classic and rule-based algorithm. In this symbol monitoring algorithm, trained examples are explicitly represented, meaning the decision-making process is traceable. In contrast, neural networks are based on non-symbolic / subsymbolic methods, where trained examples are implicitly represented.
[0009] Advantageously, this method is characterized by redundant detection and classification of the object. A neural network represents the first detection and classification path, while a symbolic monitoring algorithm represents the second detection and classification path. Here, the second detection and classification path is formed independently of the first detection path. Because the neural network is additionally monitored, the object can be reliably detected and classified even if the neural network is susceptible to false detection and / or misclassification. Furthermore, the second detection path can prevent attacks using artificial patterns designed to deceive the neural network that have not been trained on it. Advantageously, this method can contribute to road traffic safety due to its high reliability.
[0010] In one implementation, prior to the consistency check, an additional symbol monitoring algorithm checks whether the object detected and classified by the neural network possesses predefined characteristics. Advantageously, the results of the neural network are thus additionally checked. This enables the control of autonomous motor vehicles using the detection and classification results of the neural network in relatively non-dangerous traffic conditions. The additional symbol monitoring algorithm is designed to examine the output data of the neural network, which contains the results of the neural network's detection and classification of the object. This output data contains representations of the detected and classified object, which are checked by the additional symbol monitoring algorithm for predefined characteristics. Here, the additional symbol monitoring algorithm may, for example, check whether the object's geometry, size, and / or color are reasonably represented within these output data. Alternatively or additionally, the color distribution may also be checked for reasonable specifications.
[0011] In one implementation, the symbol monitoring algorithm and / or the additional symbol monitoring algorithm includes multiple symbol sub-algorithms or multiple additional symbol sub-algorithms. This could be, for example, a bright spot detector and / or a circle detector and / or a symbol classifier.
[0012] In one implementation, sensor data is provided at a predefined frequency and consistency checks are performed at predefined time intervals. This method can thus be advantageously adapted to traffic conditions. For example, the consistency checks can be performed at different time intervals depending on traffic conditions. For instance, near intersections or in emergency situations, shorter time intervals may be required for consistency checks.
[0013] In one implementation, the detection and classification results of the neural network and symbolic monitoring algorithm are stored in memory. Within the framework of an additional method step, statistics are created regarding the results of multiple consistency checks over a predefined time period. These temporal statistics can advantageously improve the reliability of the detection and / or classification of the object, and thereby contribute to improved safety in road traffic. Thus, the first and second detection and classification paths need not be designed to have the same performance. One detection and classification path may be superior (e.g., the first detection and classification path with a neural network) and primarily achieve the usability and high comfort behavior of the autonomous vehicle, while the other detection and classification path can validate more critical decisions. In one implementation, the statistics include information about the frequency with which the object is detected by the neural network and symbolic monitoring algorithm, and the frequency with which the neural network's results are validated by the symbolic monitoring algorithm in terms of detection and classification.
[0014] In one implementation, an object is considered detected and classified if a predefined number of consistent results exist. In one implementation, the predefined number of consistent events is determined based on the object's distance from the sensor. In principle, the closer the object is to the sensor, the more likely it is to be correctly detected and classified. For critical driving maneuvers, a certain number of consistent results are required on both detection and classification paths. On the other hand, for comfortable driving maneuvers, a single detection and classification path is sufficient to provide reliable results. The idea behind this method is that critical driving maneuvers often need to be introduced in areas close to objects, such as traffic signal systems. In areas close to the object, both detection and classification paths can identify the object with a high probability. Therefore, this process prevents the inherently different detection probabilities / performance of the two detection and classification paths from degrading the overall recognition and classification performance, for example, due to comparison failures at a greater distance from the object.
[0015] In one implementation, the sensor is a camera that provides images as sensor data. Advantageously, the camera can capture the color of the object and its surrounding environment, thereby capturing, for example, the status of a traffic signal system. However, it can also detect and classify traffic signs, vehicles, pedestrians, cyclists, or other traffic participants or objects.
[0016] In one implementation, sensors are an integral part of the autonomous motor vehicle. In another implementation, within the framework of automatic control of the autonomous motor vehicle, objects considered to be detected and classified are taken into account. In this case, the vehicle is operated based on this classification. For example, lateral and / or longitudinal control may be based on this classification.
[0017] A system is configured to perform a method for detecting and classifying at least one object. A computer program includes instructions that, when executed by a computer, perform the method for detecting and classifying at least one object. The computer program is stored on a machine-readable storage medium. Attached Figure Description
[0018] The above features and advantages of the present invention become clearer and easier to understand in conjunction with the following description of embodiments, which are illustrated in more detail with reference to the illustrative drawings. Wherein:
[0019] Figure 1 A method for detecting and classifying at least one object in road traffic is shown;
[0020] Figure 2 It shows that it is set up to execute Figure 1 The system of methods; and
[0021] Figure 3 A machine-readable storage medium is shown, on which a computer program is stored, the computer program including instructions that, when executed by a computer, cause the computer to perform... Figure 1 The method. Detailed Implementation
[0022] Figure 1 A method 10 for detecting and classifying at least one object in road traffic is illustrated schematically.
[0023] Within the framework of the first method step 11 of method 10, sensor data from a sensor is provided. For example, a camera can be used as the sensor, providing sensor data in the form of images. The camera can, for example, be designed to generate images in the visible electromagnetic spectrum. Alternatively, the camera can also be designed as an infrared camera. However, it is not necessarily required to use a camera as the sensor. The sensor may also be, for example, a LiDAR (Light Detection and Ranging) system. The sensor may, for example, be a component of an autonomous vehicle. However, the sensor may also be a component of a conventional vehicle.
[0024] In the second method step 12 of method 10, at least one object is detected and classified using a neural network based on sensor data. For example, images acquired during the movement of an autonomous motor vehicle can be provided to the neural network. The neural network can be designed, for example, as a deep neural network (DNN), i.e., a neural network with a large number of layers of neurons for processing. The neural network can also be, for example, a convolutional neural network (CNN). The neural network is trained to detect and classify objects. For example, the neural network can be trained to detect and classify traffic signal systems provided for traffic management in road traffic. The neural network can be trained to detect and classify traffic signs and / or other objects such as motor vehicles or buildings. Training the neural network may also include learning to detect and classify pedestrians and / or cyclists. Depending on which objects should be detected and classified within the framework of method 10, a suitable neural network can be used. The neural network represents the first detection and classification path.
[0025] As a result of detection and classification, neural networks generate output data containing representations of the detected and classified objects. For example, a neural network can be designed so that detected objects are enclosed in bounding boxes. Neural networks can also be designed to detect and classify colors in sensor data. Thus, a neural network can, for example, identify the state of a traffic signal system. Information about the state of a traffic signal system can then be used within the framework of automatic control of autonomous vehicles.
[0026] In a third method step 13, which can be executed concurrently with the second method step 12 of method 10, the at least one object is additionally and redundantly detected and classified based on these sensor data using a symbolic monitoring algorithm. Unlike neural networks based on non-symbolic and / or sub-symbolic learning methods in which trained examples are implicitly represented, the symbolic monitoring algorithm is based on symbolic methods. In this case, the trained examples are explicitly represented. Thus, the decision-making process of the symbolic monitoring algorithm can be traced and the reasons for the decision-making process can be named. This is not the case for neural networks. The symbolic monitoring algorithm represents a redundant second detection and classification path.
[0027] In the fourth method step 14 of method 10, it is checked whether the neural network and the symbol monitoring algorithm provide consistent results regarding the detection and classification of the object. If so, the object is considered to have been detected and classified. In this case, information about the object can be used, for example, within the framework of automatic control of autonomous motor vehicles. For example, the state of a traffic signal system can be detected, classified, and used for automatic control. For example, if a predefined number of consistent results exist, the object can be considered to have been detected and classified. Since the neural network is additionally monitored by means of the symbol monitoring algorithm, the object can be reliably detected and classified. Advantageously, this method can contribute to safety in road traffic due to its high reliability.
[0028] In an optional fifth method step 15 of method 10, prior to the consistency check, an additional symbol monitoring algorithm is used to check whether the objects detected and classified by the neural network possess predefined characteristics. This additional symbol monitoring algorithm is also based on a symbolic method and is designed in a classical rule-based manner. The additional symbol monitoring algorithm is designed to check the representation of the detected and classified objects in the output data of the neural network. Here, the additional symbol monitoring algorithm may, for example, check whether the object's geometry, size, and / or color are reasonably represented within the output data. Alternatively or additionally, it may also check whether the color distribution, for example, the color distribution in a traffic signal system, conforms to reasonable specifications. If the object does not possess the predefined characteristics, in one embodiment of method 10, the detection and classification of the object is marked as unreliable.
[0029] The symbol monitoring algorithm and / or the additional symbol monitoring algorithm may include multiple symbol sub-algorithms or multiple additional symbol sub-algorithms. For example, these symbol monitoring algorithms may include a bright spot detector. The bright spot detector is designed to detect and / or classify bright pixels in front of a dark background within an image. Alternatively or additionally, these symbol monitoring algorithms may include a circular detector. Bright spot detectors and circular detectors are, for example, suitable for detecting and classifying signal generators in traffic signal systems in road traffic. Another symbol sub-algorithm may, for example, be designed to count the number of colored pixels within the signal generator in order to compare the signal state identified by the neural network with the state derived from the multiple colored pixels.
[0030] Method 10 can be implemented, for example, such that the sensor data is provided at a predefined frequency. The sensor data could be provided, for example, at a frequency of 15 Hz. However, this is not mandatory. The sensor data can be provided at any frequency for the detection and classification of at least one object. Additionally, the consistency check, or multiple consistency checks, can be performed at predefined time intervals, such as every 40 seconds. However, consistency checks can also be performed at other time intervals. Here, these time intervals can be periodic or irregular. The time interval for the consistency check can be predefined, for example, based on the distance of the object from the sensor.
[0031] The detection and classification results of the neural network and the symbol monitoring algorithm can be stored in memory. In an optional sixth method step 16 of method 10, statistics are created regarding the results of multiple consistency checks within a predefined time period. This process can also be called "merge and track" because the detection and classification results of the neural network and the symbol monitoring algorithm are correlated with each other and recorded within a certain time period. The statistics may, for example, include information about the frequency with which the object is detected by the neural network and the symbol monitoring algorithm, and the frequency with which the results of the neural network are verified by the symbol monitoring algorithm in terms of detection and classification. For example, in this case, if a predefined number of consistent results exist, the object can also be considered to have been detected and classified. In an optional seventh method step 17, it can be shown whether the object is considered to have been detected and / or classified.
[0032] Objects considered to be detected and classified can be considered within the framework of automatic control of an autonomous vehicle. For example, a traffic signal system that is red at the time of detection may be detected. In response, the autonomous vehicle may be stopped. In order to respond appropriately to traffic conditions in principle, in one embodiment of method 10, a predefined number of consistent events can be specified based on the distance of the object from the sensor. The probability of the object being correctly detected and classified increases as the distance decreases. If method 10 is used within the framework of automatic control of an autonomous vehicle, a certain number of consistent results are required on two detection and classification paths for critical driving maneuvers. On the other hand, for comfortable driving maneuvers, one detection and classification path is sufficient to provide reliable results. Critical driving maneuvers must be introduced periodically in areas close to traffic signal systems. In areas close to the object, both detection and classification paths can identify the object with a high probability. Therefore, this process prevents the inherently different detection probabilities / performance of the two detection and classification paths from degrading the overall recognition and classification performance, for example, due to comparison failures at a distance from the object.
[0033] Figure 2 System 20 is schematically shown, and the system is configured to perform Figure 1 Method 10. The system 20 has a sensor 21. The sensor 21 is designed to generate sensor data and provide this sensor data to a neural network 22 and a symbol monitoring algorithm 23. The symbol monitoring algorithm 23 may include a bright spot detector 24, a circular detector 25, and / or a symbol classifier 26.
[0034] The system 20 also includes an additional symbol monitoring algorithm 27. However, this additional symbol monitoring algorithm is optional and can be omitted. The system 20 may also include a memory 28. To check whether the neural network and the symbol monitoring algorithm provide consistent results regarding the detection and classification of the object, the system 20 includes a comparison device 29. The comparison device 29 can be designed to generate statistics on the results of multiple consistency checks over a predefined time period. Alternatively, in Figure 2 Additional evaluation devices, not shown, can be designed to generate this statistic. Comparison device 29, or alternatively, in... Figure 2 The display device, not shown, can be designed such that if, for example, a predefined number of consistent results exist within the framework of the statistics, the object is marked as detected and classified.
[0035] The system 20 may, for example, be a component of a motor vehicle 40, such as an autonomous motor vehicle 40. Here, all aspects of the system 20 are... Figure 2 All the elements shown can be components of the motor vehicle 40. However, this is not mandatory. The system 20 may, for example, be only partially a component of the motor vehicle 40. For example, it is possible that only the sensor 21 of the system 20 is a component of the motor vehicle 40.
[0036] Figure 3 A machine-readable storage medium 30 is schematically shown, on which a computer program 31 is stored, the computer program including instructions 32 that, when executed by a computer, cause the computer to perform... Figure 1 The method.
Claims
1. A method (10) for detecting and classifying at least one object in road traffic, the method comprising the following method steps (11, 12, 13, 14): - Provide sensor data for sensor (21); - Using a neural network (22), at least one object is detected and classified based on the sensor data, wherein, The neural network (22) is based on a non-symbolic / subsymbolic method, in which trained examples are implicitly represented; - Using a symbolic monitoring algorithm (23), the object is detected and classified based on the sensor data, wherein the symbolic monitoring algorithm (23) is a classic and rule-based algorithm with a traceable decision-making process, wherein the symbolic monitoring algorithm (23) is based on a symbolic method, wherein in the symbolic monitoring algorithm (23), trained examples are explicitly represented; - Check whether the neural network (22) and the symbol monitoring algorithm (23) provide consistent results regarding the detection and classification of the object.
2. The method (10) according to claim 1. Prior to the consistency check, an additional symbol monitoring algorithm (27) checks whether the objects detected and classified by the neural network (22) have predefined characteristics.
3. The method (10) according to any one of the preceding claims. The symbol monitoring algorithm (23) and / or the additional symbol monitoring algorithm (27) include multiple symbol sub-algorithms (24, 25, 26) or multiple additional symbol sub-algorithms.
4. The method (10) according to any one of the preceding claims. The sensor data is provided at a predefined frequency and a consistency check is performed at predefined time intervals.
5. The method (10) according to any one of the preceding claims. The detection and classification results of the neural network (22) and the symbol monitoring algorithm (23) are stored in the memory (28). The method (10) includes the following additional method steps (16): - Create statistics on the results of multiple consistency checks over a predefined time period.
6. The method (10) according to claim 5. The statistics include information about the frequency with which the object is detected by the neural network (22) and the symbol monitoring algorithm (23), and the frequency with which the results of the neural network (22) are verified by the symbol monitoring algorithm (23) in terms of detection and classification.
7. The method (10) according to any one of the preceding claims. If a predefined number of consistent results exist, the object is considered to have been detected and classified.
8. The method (10) according to claim 7. The predefined number of consistent events is determined based on the distance of the object from the sensor (21).
9. The method (10) according to any one of the preceding claims. The sensor (21) mentioned therein is a camera, which provides images as sensor data.
10. The method (10) according to any one of the preceding claims. The sensor (21) is a component of the autonomous motor vehicle (40).
11. The method (10) according to claim 10. Within the framework of the automatic control of the autonomous motor vehicle (40), objects that are considered to be detected and classified are taken into account.
12. A system (20) configured to perform the method (10) according to any one of claims 1 to 11.
13. A computer program product comprising a computer program (31) including instructions (32) which, when executed by a computer, cause the computer to perform the method (10) according to any one of claims 1 to 11.
14. A machine-readable storage medium (30) having a computer program (31) stored thereon, the computer program including instructions (32) which, when executed by a computer, cause the computer to perform the method (10) according to any one of claims 1 to 11.