Method for operating a support system for a motor vehicle, computer program product and support system

Abstract

The invention relates to a method for operating a support system (2) for a motor vehicle (1), comprising the steps of: detecting the environment (4) of the motor vehicle (1) by means of a detection device (5) of the support system (2); determining the presence of a traffic sign (3) in the detected environment (4) by means of an electronic computing device (6) of the support system (2); classifying the detected traffic sign (3) by means of the electronic computing device (6); if the traffic sign (3) is classified, outputting a virtual traffic sign (3) associated with the detected traffic sign (3) on an output device (7) of the support system (2); or if the traffic sign (3) is not classified, outputting an image of the traffic sign (3) generated by the detection device (5) on the output device (7).Furthermore, the invention relates to a computer program product and a support system (2).

Description

The following invention relates to a method for operating a support system for a motor vehicle according to claim 1. The invention further relates to a corresponding computer program product, a computer-readable storage medium and a support system. Traffic sign recognition for cameras in motor vehicles is an important application in the field of, for example, assisted and automated driving. A detection device, such as a camera, recognizes and classifies traffic signs to provide the driver or passengers with relevant information or to enable the vehicle to steer itself autonomously according to traffic regulations. However, there are some challenges associated with this traffic sign recognition. Recognition accuracy can be affected by factors such as rain, snow, fog, or strong sunlight. Problems can also occur at dusk or in darkness. Worn, damaged, or dirty traffic signs can impair recognition accuracy. Furthermore, reflections on wet asphalt or shadows can prevent traffic signs from being correctly identified. The wide variety of traffic signs with different designs, shapes, and sizes also complicates recognition. Regional variations in sign designs can also occur. When multiple traffic signs are captured in a single image, it can be difficult to classify the individual signs and determine their relevance to the vehicle. Additionally, basic elements such as trees, buildings, or other vehicles can prevent traffic signs from being correctly identified.The speed and dynamics of the vehicle also play a crucial role. The camera must be able to recognize traffic signs at high speeds and in rapidly changing situations. Therefore, traffic sign recognition must function reliably under various conditions and in different locations to ensure road safety. To overcome this challenge, it is already known in the state of the art that advanced image processing and artificial intelligence methods, such as deep learning and convolutional neural networks (CNNs), are used. Continuous improvement of the algorithms and adaptation to different environments and situations optimizes the accuracy and reliability of traffic sign recognition. US Patent 2023 / 186637 A1 discloses systems and methods for determining the inference quality of a deep neural network (DNN) without requiring ground truth information for use in driver-assisted vehicles, including receiving an image frame from a source; applying a normal inference DNN model to the image frame to generate a first inference with a first bounding box using a normal inference DNN model; applying a deep inference DNN model to multiple filtered versions of the image frame to generate multiple deep inferences with multiple bounding boxes; comparing the multiple bounding boxes to identify a cluster condition of the multiple bounding boxes; and determining an inference quality of the image frame of the normal inference DNN model as a function of the cluster condition. The object of the present invention is to provide a method, a corresponding computer program product, a corresponding computer-readable storage medium and a support system by means of which an improved operation of the support system can be realized. This problem is solved by a method, a corresponding computer program product, a corresponding computer-readable storage medium, and a corresponding support system according to the independent claims. Advantageous embodiments are specified in the dependent claims. One aspect of the invention relates to a method for operating a support system for a motor vehicle. The environment of the motor vehicle is detected by means of a detection device of the support system. The presence of a traffic sign in the detected environment is determined by means of an electronic computing unit of the support system. The detected traffic sign is classified by means of the electronic computing unit. If the traffic sign is classified, a virtual traffic sign associated with the detected traffic sign is displayed on an output device, for example, on a screen of the motor vehicle or the support system. If the traffic sign is not classified, an image of the traffic sign generated by the detection device is displayed on the output device. In particular, it is therefore proposed that if, for example, the electronic computing device is unable to classify the traffic sign (hereinafter referred to as a road sign) accordingly, because it is dirty or outdated and therefore illegible, a corresponding image captured by the detection device can be displayed on the screen. The user can thus be informed that the support system was unable to perform a classification, but can still obtain information from the displayed image and, for example, independently take appropriate steps to manage the traffic situation according to the traffic sign. Especially since traffic sign recognition (TSR) is now used in a large number of motor vehicles, its proper functioning is of great importance. A front camera, for example, is used to process the images, identify the traffic signs, and classify them into predefined categories. The information is then displayed to the driver or user of the vehicle. However, if, for example, new traffic signs are introduced in a country, the camera may not be able to classify them correctly. Furthermore, an overly complex sign, such as one consisting of several characters, can lead to reduced reliability of the traffic sign recognition and produce errors in the output or even no output at all. In particular, the display of traffic signs in motor vehicles, provided the corresponding traffic sign has been classified, serves to provide the driver or user with relevant information about traffic signs and rules while driving. By integrating traffic sign displays into, for example, driver assistance systems or in-car displays such as head-up displays, drivers can be better informed and thus participate more safely in road traffic. The display of traffic signs and road signs can be achieved in various ways, for example, in a head-up display, where information such as speed and navigation instructions can be projected directly into the driver's field of vision.Displaying traffic signs in the head-up display (HUD) can help keep the driver's attention on the road while simultaneously providing relevant traffic information. The so-called in-dash display, often used for navigation systems or multimedia displays, can show traffic signs detected by cameras. This display can be helpful if the driver has missed a sign or to provide additional information about rules and restrictions. Furthermore, voice prompts can be used, allowing integrated voice assistants to recognize the corresponding traffic signs and provide the driver with the relevant information verbally. This enables the driver to keep their eyes on the road while remaining informed about traffic regulations.Even with certain assistance systems, i.e., in the at least partially automated operation of the motor vehicle, assistance systems such as lane keeping or speed limit assistants can use the traffic sign display to support the driver in complying with traffic rules. In other words, the invention provides that instead of relying solely on predefined classes or classifications, the responsibility can also be transferred to the vehicle user. Even if the camera cannot directly classify a traffic sign, the information that it is an actual traffic sign can still be available. Instead of performing the classification and showing the vehicle user a pre-rendered image, i.e., the virtual traffic sign, the camera can transmit the actual captured image of the real traffic sign and leave the decision regarding the appropriate action to the driver. In particular, this has the advantage of making the process very flexible, as the vehicle user can be involved and is not solely dependent on software updates. Furthermore, more complex scenarios can be handled, even if a correct classification by an algorithm is not directly possible. Traffic signs and traffic signals are used synonymously in the following text and refer to the same signs / symbols used to regulate and direct traffic. However, in technical terminology, a distinction is often made between traffic signs and traffic signals, with traffic signs being a specific subset of traffic signals. Traffic signals encompass not only traffic signs but also traffic control devices such as traffic lights, traffic management devices such as guardrails, and road markings such as lane markings. A traffic signal is a sign or symbol that serves to inform road users such as drivers, cyclists, and pedestrians about rules, hazards, restrictions, instructions, and information related to road traffic. Traffic signals are determined by the relevant authorities and are generally standardized to ensure uniform and easily understandable symbols and colors.This includes, for example, regulatory signs, warning signs, directional signs, information signs, and supplementary signs. Traffic signs are an important tool for making road traffic safe, orderly, and efficient. Their uniform design and standardization allow road users to react quickly and easily to rules and information. According to an advantageous embodiment, a camera is used as a detection device to capture the surroundings. This camera is particularly relevant when it is already installed in the vehicle, for example, as part of a vehicle assistance system. In other words, the assistance system can utilize cameras already installed in other systems, such as driver assistance systems. This allows the process to be implemented with fewer components. Furthermore, the camera can capture the surroundings and, in particular, display an image of the traffic sign instead of a virtual image. Thus, the process is highly functional and can be implemented with few components. It has also proven advantageous to conduct driver observation and / or driving behavior observation when a recognized traffic sign is not classified. For example, it is possible to observe how the driver reacts to the traffic sign and whether it has any influence on the driver. Furthermore, driving behavior observation can also be carried out. If, for example, a sign indicating a city exit is not recognized by the system because, for instance, the age of the traffic sign precludes classification, but the driver slowed down to, say, 50 km / h after this sign and then accelerated to 100 km / h after the next sign, this can be interpreted as an indication that this is not the beginning of the city, but rather its end. Thus, a classification can be carried out retrospectively based on driving behavior. It has also proven advantageous to classify traffic signs based on driver observation and / or behavioral observation. This could, for example, allow for the future classification of this or a similar traffic sign. This would improve the support system and enable a more effective classification of traffic signs in the future. In a further advantageous embodiment, input from a vehicle occupant regarding an unclassified traffic sign is recorded, and a classification is performed based on this input. For example, the image of the traffic sign can be displayed on a central display, which can also be touch-sensitive. After the unclassification, the user or an occupant can then, for example, annotate the traffic sign. They can indicate, for instance, that it is a speed limit sign or another type of traffic sign. Furthermore, additional information can also be entered. Thus, annotations of the unclassified traffic sign can be performed retrospectively, allowing for adjustments to be made for future traffic sign recognition. In particular, while the recognition algorithms of front cameras are trained with enormous amounts of data, a multitude of real-world traffic scenarios remain unaccounted for, leading to inaccurate perceptions of the situation by the front camera. This advantageous design allows the front camera's perception to be improved through continuous training while driving the production vehicle. The training data is continuously recorded by the front camera, and the annotations are provided by the occupant. It is further advantageous if a machine learning algorithm for traffic sign recognition is trained based on driver observation and / or driving behavior observation and / or input. For example, the electronic computing device can include the machine learning algorithm, such as an artificial intelligence, in particular a deep neural network or a convolutional neural network. These artificial intelligences can have corresponding traffic sign models to enable traffic sign recognition. These models can, for example, be pre-trained and already present on the electronic computing device when the vehicle is put into operation. These models have already been trained with a large amount of training data; however, the training data cannot represent all real-world scenarios.The training of the machine learning algorithm with user input, driver observation, or driving behavior observation can thus be actively continued and adapted during the operation of the motor vehicle, enabling improved recognition of traffic signs during vehicle operation. It has also proven advantageous to generate training data for the machine learning algorithm based on driver observation and / or driving behavior observation and / or input, and to transmit this training data to an external electronic computing unit. Thus, it is possible to use the training data, which further develops the vehicle's internal machine learning algorithm, to also transmit it to a higher-level electronic computing unit. This training data can then be made available to the higher-level electronic computing unit, allowing future models to be trained accordingly. This enables a general improvement of the models. Furthermore, it has proven advantageous to suppress future classifications in the event of a misclassification and / or incorrect driving behavior. For example, if the user or occupant intentionally makes a misclassification and / or deliberately exhibits incorrect driving behavior, this can be easily identified by a decrease in the trustworthiness of the machine learning algorithm. If this occurs, the corresponding adaptation or adjustment can be suppressed. In particular, no training data is generated and cannot be transmitted to the vehicle's external electronic computing unit. This can be observed, in particular, by a decrease in overall model performance, which can be detected early by checking whether the annotation and the current model output show any correlation.As mentioned previously, the generation of training data will then no longer be offered or will be aborted. The overall model performance is determined using a test dataset stored in the vehicle, which can be updated as soon as new scenarios need to be tested. This prevents intentional misuse of the assistance system. It has also proven advantageous to generate training data for the machine learning algorithm only after a predetermined number of traffic sign classifications has been exceeded. This means, for example, that the training dataset is not adjusted immediately after the first adjustment of the training dataset or the number of classifications, but only after several classifications. This helps prevent misuse and results in more robust traffic sign recognition. In a further advantageous embodiment, it is provided that, in the case of partial classification of a traffic sign, only the unclassified part is displayed as an image. Particularly since, for example, traffic signs can have numerous sub-signs, it is possible that traffic signs are partially recognized and partially not. It is now provided that recognized traffic signs can still be rendered and displayed as virtual traffic signs, while the unclassified part can be displayed as an image. Thus, the recognized traffic signs can still be used for appropriate control or information display for the driver. The loss of information is therefore minimal. It has also proven advantageous if a control signal for a vehicle function unit is generated when a traffic sign is classified. For example, further information about the traffic sign itself can be displayed on a driver's display or output as a voice message. Furthermore, a longitudinal or lateral acceleration system of the vehicle can be controlled accordingly, provided the vehicle is equipped with a corresponding assistance system that allows at least partially automated operation. For example, the vehicle's speed can be automatically adjusted according to the detected speed. Furthermore, it has proven advantageous to generate a warning signal for a vehicle occupant if a traffic sign is not classified. For example, an audible or visual warning signal can be generated. The driver or occupant is thus informed that the assistance system was unable to classify the corresponding traffic sign. This alerts the user that they must, for example, perform a self-classification. The presented method is, in particular, a computer-implemented method. Therefore, a further aspect of the invention relates to a computer program product with program code means which, when the program code means are executed by the electronic computing device, cause a method according to the preceding aspect to be carried out. Furthermore, the invention therefore also relates to a computer-readable storage medium with at least the computer program product according to the preceding aspect. A further aspect of the invention relates to a support system for a motor vehicle, comprising at least one detection device, an electronic computing device, and an output device, wherein the support system is configured to carry out a method according to the preceding aspect. In particular, the method is carried out by means of the support system. The invention also relates to a motor vehicle with a support system according to the preceding aspect. The motor vehicle can, in particular, be at least partially automated or fully automated. Advantageous embodiments of the process are to be regarded as advantageous embodiments of the computer program product, the computer-readable storage medium, the support system, and the motor vehicle. The support system and the motor vehicle possess tangible features to enable the execution of the corresponding process steps. For the recognition of the traffic sign, a suitable object recognition algorithm can be used in particular. Within the scope of this disclosure, an object recognition algorithm can be understood as a computer algorithm that is capable of identifying and locating one or more objects within a provided input data set, for example, an input image, for example, by defining corresponding boundary boxes or regions of interest (ROIs) and, in particular, by assigning a corresponding object class to each of the boundary boxes, wherein the object classes can be selected from a predefined set of object classes.The assignment of an object class to a bounding box can be understood as providing a corresponding confidence value or probability that the object identified within the bounding box belongs to the corresponding object class. For example, the algorithm can provide such a confidence value or probability for each object class within a given bounding box. The object class assignment might involve selecting or providing the object class with the highest confidence value or probability. Alternatively, the algorithm can simply define the bounding boxes without assigning a corresponding object class. In the present disclosure, a computing unit / electronic computing device can be understood, for example, as a data processing device with processing circuits. A computing unit can therefore perform arithmetic operations to process data. These arithmetic operations can also include indexed access to a data structure, such as a lookup table (LUT). A computing unit may, in particular, comprise one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems on a chip (SoCs). The computing unit may also include one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also comprise a physical or virtual cluster of computers or other units of the aforementioned type. A processing unit can also include one or more hardware and / or software interfaces and / or one or more memory units. A memory unit can be implemented as volatile data storage, for example as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data storage, for example as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, or ferromagnetic random access memory (FRAM).a magnetoresistive random access memory, MRAM (magnetoresistive random access memory), or a phase-change random access memory, PCRAM (phase-change random access memory). Further features of the invention are evident from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as those subsequently mentioned in the description of the figures and / or shown in the figures alone, are not only usable in the combinations specified, but also in other combinations without departing from the scope of the invention. Thus, embodiments that are not explicitly shown and explained in the figures, but which can be derived and generated from the explained embodiments by separate combinations of features, are also to be considered as encompassed and disclosed by the invention. Embodiments and combinations of features that do not exhibit all the features of an originally formulated independent claim are also to be considered disclosed.Furthermore, embodiments and combinations of features, in particular those set out above, are to be considered disclosed which go beyond or deviate from the combinations of features set out in the cross-references of the claims. Figure 1 shows a schematic top view of an embodiment of a motor vehicle with an embodiment of a support system; and Figure 2 shows a further schematic top view of another embodiment of a motor vehicle with a further embodiment of the support system. In the figures, identical or functionally equivalent elements are provided with the same reference symbols. Fig. 1 shows a schematic top view of an embodiment of a motor vehicle 1. The motor vehicle 1 has at least one assistance system 2. The assistance system 2 is designed, in particular, to recognize a traffic sign 3 in the vicinity 4 of the motor vehicle 1. For this purpose, the assistance system 2 has at least one detection device 5, in particular a camera, an electronic computing device 6, and an output device 7. In the present embodiment, the output device 7 is designed, in particular, as an infotainment interface and is located, for example, in a center console of the motor vehicle 1. In the present embodiment, a user 8, for example, a driver of the motor vehicle 1, is also shown.For example, the user 8 can also make corresponding inputs via the output device 7, which is also designed to be touch-sensitive, or at least visually perceive the symbols displayed on the output device 7. According to one embodiment of the procedure, the environment 4 is detected by the detection device 5. The presence of a traffic sign 3 in the detected environment 4 is determined by the electronic computing device 6. The detected traffic sign 3 is then classified by the electronic computing device 6. If the traffic sign 3 could be classified, a traffic sign assigned to the detected traffic sign 3 is virtually displayed on the output device 7. However, if the traffic sign 3 could not be classified, an image of the traffic sign 3 generated by the detection device 5 is displayed on the output device 7. In particular, it is thus intended that, instead of relying solely on predefined classifications by the electronic computing unit 6, the responsibility can be at least partially transferred to the user 8. Even if the detection unit 5 cannot classify the traffic sign 3, the information that it is indeed a traffic sign 3 can still be available. Instead of performing the classification and showing the user 8 a pre-rendered image at the output unit 7, the detection unit 5 can forward the actual image of the real traffic sign 3 and leave the decision about an action to the user 8. As an example, imagine that a new traffic sign 3 is introduced in a country, and the algorithms within the detection device 5 or the electronic computing device 6 are unable to correctly recognize and classify this newly introduced traffic sign 3. Nevertheless, the electronic computing device 6 can be used to extract the traffic sign 3 from the image, and the image can be displayed to the user 8 on the output device 7. The user 8 can then decide whether the sign is relevant to them. Alternatively, a list of the most recent traffic sign 3 can also be displayed. Another example is a traffic sign 3, which contains text on a supplementary sign that cannot be correctly interpreted by the algorithms of the electronic computing device 6. As mentioned above, the information can nevertheless be forwarded to user 8 and interpreted by him. Furthermore, it may also be provided that, if traffic sign 3 is classified, a control signal is generated for a functional unit 9 of the motor vehicle 1, for example, for the longitudinal acceleration device and / or lateral acceleration device of the motor vehicle 1. Furthermore, if traffic sign 3 is not classified, a warning signal may be generated, for example, by means of a warning device 10 of the motor vehicle 1 for the occupant or user 8 of the motor vehicle 1. Fig. 2 shows another schematic embodiment according to the support system 2. In the present embodiment, it is shown in particular that the electronic computing device 6 can, for example, include a machine learning algorithm 11. The machine learning algorithm 11 can, for example, be provided in the form of a deep neural network or a convolutional neural network. In particular, Figure 2 illustrates, for example, that if the detected traffic sign 3 is not classified, driver observation and / or driving behavior observation can be performed. Based on this observation, traffic sign 3 can then be classified. Furthermore, input from an occupant or user 8 of the motor vehicle 1 regarding the unclassified traffic sign 3 can be recorded, and classification can be performed based on this input. It can also be provided that the machine learning algorithm 11 for traffic sign recognition is trained based on the driver observation and / or driving behavior observation and / or the input.It may also be provided that, based on driver observation and / or driving behavior observation and / or input, training data for the machine learning algorithm are generated and the training data 12 are transmitted to an electronic computing device 13 external to the motor vehicle. Furthermore, it can be provided that in the event of a misclassification and / or incorrect driving behavior, a future classification is suppressed. Likewise, training data 12 for the machine learning algorithm 11 can only be generated once a predetermined number of classifications of traffic sign 3 has been exceeded. It can also be provided that, in the case of a partial classification of traffic sign 3, only the unclassified part is output as an image. The example in Fig. 2 shows in particular that continuous training of the machine learning algorithm 11 makes it possible to constantly improve the computer vision model of, for example, the front camera, since many new scenes and special cases can be considered as training data 12. The user 8 or, for example, the passenger provide real-time comments for these scenarios, which are recorded during daily journeys. These so-called annotations are used by the continuous training algorithm as ground truth. If the image processing model is improved by the input of user 8 or the passenger, the user can, for example, receive "awards" for vehicle 1. However, if the model performance decreases, which can be detected early by checking whether the annotation and the current model output show no correlation, continuous training is no longer offered. The overall model performance is determined using the test dataset stored in vehicle 1, which can be updated as soon as new scenarios need to be tested. In an example scenario, vehicle 1 passes a town exit sign on its daily commute. Because the crossed line on the sign is already slightly blurred, the front camera perceives the sign as a town entrance sign. Such town exit signs with a blurred line are almost nonexistent in the original training dataset 12. With continuous camera recognition training, user 8, for example, correctly identifies the town exit scene every time it passes the sign. After several sessions, the front camera model is retrained to correctly recognize the town exit sign with the blurred line. Now, all other similar scenarios with blurred town exit signs are also correctly perceived, leading to increased satisfaction for user 8. This improved computer vision model could also be made available for other vehicles to enhance their perception performance. QUOTES INCLUDED IN THE DESCRIPTION This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature US 2023 / 186637 A1

[0005]

Claims

Method for operating a support system (2) for a motor vehicle (1), comprising the steps of: - detecting an environment (4) of the motor vehicle (1) by means of a detection device (5) of the support system (2); - determining the presence of a traffic sign (3) in the detected environment (4) by means of an electronic computing device (6) of the support system (2); - classifying the detected traffic sign (3) by means of the electronic computing device (6); - if the traffic sign (3) is classified, outputting a virtual traffic sign (3) associated with the detected traffic sign (3) on an output device (7) of the support system (2); or - if the traffic sign (3) is not classified, outputting an image of the traffic sign (3) generated by the detection device (5) on the output device (7). Method according to claim 1, characterized in that the environment (4) is detected by means of a camera as a detection device (5). Method according to claim 1 or 2, characterized in that, in the event of non-classification of the recognized traffic sign (3), a driver observation and / or a driving behavior observation is carried out. Method according to claim 3, characterized in that a classification of the traffic sign (3) is carried out on the basis of driver observation and / or driving behavior observation. Method according to one of the preceding claims, characterized in that an input from an occupant of the motor vehicle (1) to the unclassified traffic sign (3) is recorded and a classification is carried out on the basis of the input. Method according to one of claims 3 to 5, characterized in that a machine learning algorithm (11) for traffic sign recognition is trained on the basis of driver observation and / or driving behavior observation and / or input. Method according to claim 6, characterized in that training data (12) for the machine learning algorithm (11) are generated on the basis of driver observation and / or driving behavior observation and / or input and the training data (12) are transmitted to an electronic computing device (13) external to the motor vehicle. Method according to one of claims 3 to 7, characterized in that in the event of a misclassification and / or incorrect driver behavior by the user (8), a future classification is suppressed. Method according to one of claims 3 to 8, characterized in that training data (12) for the machine learning algorithm (11) are generated only when a predetermined number of classifications of the traffic sign (3) is exceeded. Method according to one of the preceding claims, characterized in that in the case of a partial classification of the traffic sign (3) only the unclassified part is output as an image. Method according to one of the preceding claims, characterized in that a control signal for a functional unit (9) of the motor vehicle (1) is generated during a classification of the traffic sign (3). Method according to one of the preceding claims, characterized in that in the event of a non-classification of the traffic sign (3) a warning signal is generated for an occupant of the motor vehicle (1). Computer program product with program code means which cause an electronic computing device (6) to perform a method according to one of claims 1 to 12 when the program code means are processed by the electronic computing device (6). Computer-readable storage medium comprising at least the computer program product according to claim 13. Support system (2) for a motor vehicle (1), comprising at least one detection device (5), one electronic computing device (6) and one output device (7), wherein the support system (2) is configured to perform a method according to one of claims 1 to 12.

Images