Method for determining and archiving problematic image sections for subsequent checking of an image evaluation system of a vehicle, device and vehicle for use in the method, and computer program

Abstract

Method for determining and archiving problematic image sections (14) for subsequent checking of an image evaluation system of a vehicle (10), wherein it is checked whether an object recognition for a moving object (12) by the image evaluation system was carried out correctly in terms of time or location, wherein, if deviations beyond a tolerance limit are detected during the temporal or spatial analysis, it is determined which image sections (14) are to be identified and archived for more detailed examination, wherein the images are temporarily stored in an allocated memory area of a memory (60) in the vehicle (10) managed as a ring buffer, wherein the step of determining the problematic image sections (14) to be archived includes that at the time of detection of the moving object (12), a current speed and acceleration of the moving object (12) is determined from several successive images using various sensor data of the vehicle (10) and a trajectory of the moving object (12) is estimated, wherein, with the aid of the estimated trajectory and in conjunction with an estimation of the vehicle's own motion (10), a backward calculation is carried out to determine a number of problematic images to be used to check why the moving object (12) could not be detected, wherein for the number of determined problematic images, respective image sections (14) are determined for points in time in the past, within which the not yet detected moving object (12) would have had to be located at that point in time, wherein a size of a respective image section (14) is influenced by parameters of the moving object (12) and their uncertainties, possible alternatives in the calculated trajectory of the moving object (12) and known sensor inaccuracies, and wherein the determined problematic image sections (14) are transmitted from the ring buffer of the memory (60) via a communication module (160) to an external archiving location (240).

Description

[0001] The invention relates to the technical field, from driver assistance systems to autonomous vehicles. In particular, the invention relates to a method for capturing image material for testing image analysis systems. The invention also relates to a device and a vehicle for use in the method, as well as a computer program.

[0002] Intensive work is currently underway on technologies that will eventually enable autonomous driving. A first approach is the introduction of various driver assistance systems that relieve the driver of certain tasks. Examples of driver assistance systems include blind spot warning systems, emergency braking systems, parking assistants, turning assistants, lane keeping assistants, cruise control assistants, etc. A further stage of development could involve integrating several assistance systems. This does not completely relieve the driver of their responsibilities, but rather ensures that the driver can take control of the vehicle at any time. The driver then also performs monitoring functions.

[0003] In the near future, it can therefore be assumed that, through the use of newer technologies (vehicle-to-vehicle communication, use of databases, backend connectivity, cloud services, server deployment, vehicle sensors, etc.), comprehensive information about objects (especially vehicles) in the visible and hidden / invisible surroundings of the vehicle will be available on the system side. In the area of vehicle sensors, the following components in particular are mentioned, which enable environmental monitoring: RADAR devices corresponding to radio detection and ranging, LIDAR devices corresponding to light detection and ranging, primarily for distance detection / warning, and cameras with corresponding image processing for object detection. Ultrasound and infrared sensors are also mentioned. This data about the environment can thus be used as the basis for system-based driving recommendations, warnings, etc.For example, indications / warnings about the direction (possibly into the driver's own trajectory) of another nearby vehicle's intention to turn are conceivable. Traffic sign recognition is also mentioned as a use case that is important for determining the legal framework.

[0004] Vehicle-to-vehicle communication also plays an important role in autonomous driving. Mobile communication systems such as LTE (Long Term Evolution) or 5G have now been developed that also support vehicle-to-vehicle communication. As an alternative, systems based on Wi-Fi technology are available for direct vehicle communication, particularly the Wi-Fi p system. The term "autonomous driving" is sometimes used differently in the literature.

[0005] To clarify this term, the following explanatory note is presented here. Autonomous driving (sometimes also called automatic driving, automated driving, or piloted driving) refers to the movement of vehicles, mobile robots, and driverless transport systems that behave largely autonomously. There are various levels of the term autonomous driving. At certain levels, autonomous driving is also referred to when a driver is still in the vehicle, who may only monitor the automated driving process. In Europe, the various transport ministries (in Germany, the Federal Highway Research Institute was involved) have collaborated and defined the following levels of autonomy. • Level 0: “Driver only”, the driver drives, steers, accelerates, brakes, etc. • Level 1: Certain assistance systems help with vehicle operation (including a distance control system - Automatic Cruise Control ACC). • Level 2: Partial automation. Automatic parking, lane keeping, general longitudinal guidance, acceleration, braking, etc. are handled by the assistance systems (including traffic jam assistant). • Level 3: High automation. The driver does not need to constantly monitor the system. The vehicle performs functions such as activating the turn signal, changing lanes, and keeping in lane independently. The driver can attend to other tasks but, if necessary, will be prompted by the system to take over within a pre-warning period. This form of autonomy is technically feasible on highways. Legislators are working toward approving Level 3 vehicles. The legal framework for this has already been established. • Level 4: Full automation. The system permanently assumes control of the vehicle. If the system can no longer handle the driving tasks, the driver can be asked to take over. • Level 5: No driver required. No human intervention is required other than setting the destination and starting the system.

[0006] Automated driving functions from level 3 onwards relieve the driver of responsibility for controlling the vehicle. The VDA has published a similar classification of the different levels of autonomy, which can also be used.

[0007] Due to the current development towards higher levels of autonomy, where many vehicles are still controlled by the driver, it can be assumed that corresponding additional information can already be used for manually driven vehicles in the medium term and not only in the long term for highly automated systems.

[0008] For driver-vehicle interaction, the question arises as to how this information can be presented in a way that creates real added value for the human driver and allows them to quickly and intuitively locate the information provided. The following solutions in this area are already known from the state of the art.

[0009] One future vision in the automotive industry is to be able to display virtual elements on the windshield of one's own vehicle, offering the driver a number of advantages. This technology uses so-called "augmented reality" (AR) or "mixed reality" (MR) technology. The corresponding German term "erweiterten Realität" or "gemischten Realität" is less common. In this case, the real environment is enriched with virtual elements. This has several advantages: There is no need to look down at displays other than the windshield, as much relevant information appears when looking through the windshield. This means the driver does not have to look away from the road. Furthermore, the precise positioning of the virtual elements in the real environment is likely to reduce the cognitive effort on the part of the driver, as there is no need to interpret a graphic on a separate display.Added value can also be created with regard to automated driving. In this regard, reference is made to the article "3D-FRC: Depiction of the future road course in the Head-Up Display" by CA.

[0010] Wiesner, M. Ruf D. Sirim and G. Klinker in 2017 IEEE International Symposium on Mixed and Augmented Reality, in which these advantages are explained in more detail.

[0011] Given the current limited technological resources, it can be assumed that fully displayable windshields will not be found in vehicles in the medium term. Head-up displays are currently used in vehicles. These also have the advantage that the HUD image appears closer to the real environment. These displays are essentially projection units that project an image onto the windshield. However, from the driver's perspective, this image is located a few to 15 meters in front of the vehicle, depending on the module's design.

[0012] The "image" is composed as follows: It's less a virtual display and more a kind of "keyhole" into the virtual world. The virtual environment is theoretically superimposed over the real world and contains the virtual objects that support and inform the driver while driving. The limited display area of the HUD means that only a section of it can be seen. You look through the HUD's display area at the section of the virtual world. Since this virtual environment complements the real environment, this is also referred to as "mixed reality."

[0013] A major advantage of previously known "augmented reality" (AR) displays is that they can be presented directly within or as part of the environment. Relatively obvious examples usually relate to the field of navigation. While classic navigation displays (in conventional HUDs) usually show schematic representations (e.g., an arrow pointing at a right angle to indicate that a right turn should be made at the next opportunity), AR displays offer significantly more effective options. Because the displays can be presented as "part of the environment," extremely quick and intuitive interpretations for the user are possible. The driver, as well as a higher level of automatic driving function, must be able to rely on the correct initial detection of objects. If additional information is derived from this, it should be displayed at the correct location in the image.

[0014] Therefore, it is necessary to demonstrate the correct functioning of the image capture systems, at least in test mode. However, this can also be planned for normal operation. Public transport is highly complex, and so many moving road users can be involved that obscuration can occur. Other problems include visibility restrictions due to various weather conditions, such as fog, rain, snow, etc., backlighting, darkness, and other factors.

[0015] A vehicle-based system for traffic sign recognition is known from US 2016 / 0170414 A1. It uses environmental sensors such as LIDAR sensors, RADAR sensors, and cameras. The vehicle's position is also recorded via GPS. The detected traffic signs are reported externally along with their position and entered into a database.

[0016] US 2017 / 0069206 A1 discloses a vehicle-based system for verifying traffic sign recognition. It also uses environmental sensors such as LIDAR sensors, RADAR sensors, and cameras. The detected traffic signs are also reported externally along with their positions and entered into a database. To detect false detections, various verification measures are proposed, including manual visual inspections by qualified personnel, visual inspections through crowdsourcing, computer-assisted analysis, and statistical analysis.

[0017] US 2018 / 0260639 A1 discloses a vehicle-based system and method for traffic sign recognition. The recognized traffic signs are stored with their position in an image processing unit.

[0018] From US 2008 / 0239078 A1 it is known to display detected objects in the video images of a camera used in a vehicle for environmental monitoring with additional information derived from other sensors in the vehicle.

[0019] US 2018 / 0012082 A1 discloses a method for image analysis, in which a sequence of images is captured by a camera mounted on a vehicle. The image analysis system includes an object detection module. A bounding box is defined, which serves as a border around a detected object for highlighting.

[0020] US 2018 / 0201227 A1 discloses a vehicle environment monitoring system in which the vehicle environment is monitored by a camera. The vehicle also includes a display unit for displaying the video images recorded by the camera. An image evaluation unit is provided that monitors the image data for the presence of moving external objects in the environment. Furthermore, a controller is programmed to determine a threat assessment value based on the conditions near the vehicle and to upload image data to an external server if the threat assessment value is greater than a first threshold.

[0021] US 2018 / 0349785 A1 discloses a method for in-situ perception in an autonomously driving vehicle. Several types of sensor data are continuously collected via several types of sensors mounted on the vehicle. The sensor data provide information about the vehicle's surroundings. Elements surrounding the vehicle are tracked using the sensor data based on a model. For a tracked element, cross-temporal validation is performed by determining an estimated label for the element and retrieving previous labels corresponding to the element with corresponding previous timestamps. The estimated label is then assigned to the element when it is validated based on previous labels.

[0022] The known solutions suffer from various disadvantages. This was recognized in the context of the invention. The problem with currently known image acquisition methods for use in driver assistance systems is that their correct function is not, or only inadequately, verified during operation. Therefore, incorrect detections or even misclassifications of objects are possible.

[0023] There is therefore a need for further improvements in the verification of vehicle-based image acquisition systems, particularly for verification procedures that enable subsequent verification in the event of false detections.

[0024] DE 10 2019 206 147 A1 demonstrates ways in which, with the help of a ring buffer for images recorded by environmental sensors and the consideration of vehicle odometry data, only certain image sections can be considered for data recording / transmission to a backend in order to reduce the required data volume. However, this patent application only considers static objects. Especially with regard to autonomous driving, it will also be necessary to detect moving objects in the recorded images. The corresponding image analysis algorithms, however, are even more error-prone. In order to improve the algorithms, subsequent verifications are also required in the event of false detections. Therefore, there is also a need for further improvements in the verification of vehicle-based image acquisition systems.

[0025] The invention aims to find such an approach. At the same time, the effort required for archiving the image data, as a prerequisite for subsequent verification, should be kept to a minimum. This object is achieved by a method for acquiring image material for verifying image evaluation systems according to claim 1, a device and a vehicle for use in the method according to claims 7 and 8, and a computer program according to claim 9.

[0026] The dependent claims contain advantageous developments and improvements of the invention according to the following description of these measures.

[0027] According to one embodiment of the invention, in a method for capturing and archiving image material for checking image analysis systems of a vehicle, it is checked whether an object was detected by the image analysis system correctly in terms of time or location. If so, no image data need to be retained for subsequent analysis. However, if deviations in time or location that lie beyond a tolerance limit are detected during the temporal or spatial analysis, a step is taken to determine which images or image sections should be archived for more detailed review, and these determined images or image sections are then archived. In order to be able to do this for dynamic objects as well, it is necessary to calculate or obtain a trajectory for the moving object for which the deviation was detected.In the future, vehicles equipped with automatic driving functions will calculate a trajectory for their own movement. They can then transmit this calculated trajectory to surrounding vehicles using direct vehicle communication. This allows the observer vehicle to estimate the positions where the object should have been before it was detected by the object recognition system. Using backward calculations, the images and image sections relevant for subsequent analysis are then determined. The invention also provides for the image sections determined in this way to be extracted from the recorded images and archived only to the extent necessary. This is done using the vehicle's odometry and position data, as well as the trajectory of the relevant dynamic object. The dynamic object is usually another road user.In order to be able to retrieve subsequently determined images or image sections, a ring buffer for image data is required, the size of which (i.e. maximum storage duration) may be dimensioned depending on the function and, if applicable, the driving speed.

[0028] The method offers the advantage that an image analysis system that provides safety-relevant data can be tested during operation, while requiring minimal memory requirements for the test. The method is particularly advantageous for testing image analysis systems that are still in the development stage. However, the method can also be used advantageously in series products. This is advantageous for making subsequent improvements to the installed image analysis system or for detecting a camera or sensor misalignment that requires vehicle maintenance.

[0029] According to the invention, a backward calculation is carried out using the trajectory in order to calculate a number of images or image sections in which the moving object could presumably be visible, even though the object recognition could not detect the moving object in them.

[0030] The images or image sections determined through the reverse calculation are sent to an external archiving center. This has the advantage that only the problematic images need to be archived. Archiving at an external location has the advantage that the images do not have to be archived in the vehicle first. The evaluation will be carried out at the external location by experts anyway, who will then also be able to improve the image analysis system.

[0031] It is also advantageous to use an object recognition algorithm for image analysis. To verify whether the object recognition occurred correctly in terms of time or location, the distance from the detected object is determined. A standard recognition distance is defined, which indicates the distance at which the object recognition algorithm should provide object recognition. If a deviation is detected, the images or image sections can then be archived.

[0032] The invention can be used particularly advantageously for testing image analysis systems in vehicles. These are now equipped with imaging sensors for detecting the surroundings, such as cameras, LIDAR, or RADAR sensors. These sensors detect traffic signs, vehicles ahead, and other road users, intersections, turning points, potholes, etc. Such vehicles are also equipped with position detection systems. The determined position of the vehicle can be used to check whether there is an error in image recognition. In the simplest case, the vehicle detects the position of the detected object, even if it passes by the location of the object. This can then be used to calculate the distance between the vehicle and the object. In another embodiment, the position of the object can be taken from a highly accurate map. In a further variant, the position can be estimated based on the position of the vehicle.

[0033] Satellite navigation and / or odometry are suitable for determining the position data of the vehicle and the object. A generic term for such satellite navigation systems is GNSS, which stands for Global Navigation Satellite System. Existing satellite navigation systems include the Global Positioning System (GPS), Galileo, GLONASS (Globalnaja navigatzionnaja sputnikowaja sistema), and Beidou.

[0034] Another advantageous option is to transmit the images temporarily stored in the vehicle to an external location once the vehicle has returned to the owner's home. Modern vehicles are equipped with Wi-Fi. If this is then logged into the vehicle owner's private Wi-Fi network, the archived image data can be transmitted at a high data rate. Compared to direct transmission via a cellular communication system, to which the vehicle is logged while driving, this has the advantage of lower costs for the vehicle owner and less strain on the cellular network. Its capacity is utilized by a variety of other applications.

[0035] Alternatively, additional images or image sections can be archived to expand the verification options. It is advantageous to archive the images or image sections captured from the standard detection distance to the object detection distance at high quality, while the other images or other image sections are archived at lower quality. This minimizes the storage and transmission costs for archiving additional images or image sections.

[0036] Another advantageous variant is to determine the size of the image sections to be archived and / or the recording period of the images / image sections to be archived depending on one or more of the following environmental parameters: • Accuracy of the positioning of the vehicle and / or object • Time, especially with day / night distinction • Weather conditions • Traffic situation • Road surface condition.

[0037] This is always advantageous when environmental conditions are not always the same. In practice, environmental conditions are constantly changing. If the same number of images or image sections are always archived, it could easily happen that important scenes are missed when environmental conditions change, making it impossible to find the cause of the problem.

[0038] For a device for use in the method according to one of the preceding claims, it is advantageous if the device has an image generation device, a computing device, and a storage device. The computing device is designed such that it can perform object recognition in the images supplied by the image generation device. The computing device then also has the task of determining whether the object recognition was carried out correctly in terms of time or location. Furthermore, it is advantageous if the computing device is designed to calculate a trajectory of the moving object and, depending on the calculated trajectory, to determine the images or image sections to be archived. By using the trajectory, the images or image sections to be archived can be better narrowed down, thereby reducing the effort required for archiving.

[0039] In a particularly advantageous variant, the device additionally comprises a communication module, and the communication module is designed to send the acquired images or image sections to an external archiving location upon receipt of the command to archive these acquired images or image sections. The external archiving location can be a data center of the manufacturer of the image evaluation system or the vehicle.

[0040] A further variant is proposed, wherein the device has a storage device, and the storage device is designed to store the acquired images or image sections upon receipt of the command to archive these acquired images or image sections. In this variant, archiving takes place in the vehicle until the data is read out. As described above, the readout could take place in a workshop, or the data could be transferred to the external archiving location via the home WLAN network upon return. For the temporary storage of the data on the vehicle, it is advantageous if the storage device is designed as a ring buffer, in which the oldest previously stored images or image sections are overwritten by the new images or image sections in the event of a memory overflow.

[0041] A video camera, a LIDAR, or a RADAR sensor can advantageously be used as the image generation device. For the wireless communication interface, it is advantageous to use an interface according to at least one of the WLAN communication systems according to a standard of the IEEE 802.11 standard family or an LTE or 5G mobile radio communication system according to a 3GPP standard.

[0042] For a vehicle for use in the method, it is advantageous if the vehicle is equipped with a device corresponding to the proposed device.

[0043] For a computer program that is executed in a computing device in order to carry out the steps for capturing image material according to the method according to the invention, the corresponding advantages apply as described for the method according to the invention.

[0044] Embodiments of the invention are illustrated in the drawings and are explained in more detail below with reference to the figures.

[0045] They show: Fig. 1 the typical cockpit of a vehicle; Fig. 2 a schematic diagram of the various communication options provided in a vehicle; Fig. 3 a block diagram of the vehicle’s on-board electronics; Fig. 4 a first representation of a driving situation to explain the problems involved in checking the function of an image evaluation system of the vehicle; Fig. 5 a second representation of the driving situation, the second representation showing the driving situation at an earlier point in time; and Fig. 6 a flowchart for a program for acquiring image material for checking the function of the image evaluation system.

[0046] The present description illustrates the principles of the inventive disclosure. It is thus understood that those skilled in the art will be able to devise various arrangements that, although not explicitly described herein, embody principles of the inventive disclosure and are also intended to be protected within their scope.

[0047] Fig. Figure 1 shows the typical cockpit of a vehicle 10. A passenger car is depicted. However, any other vehicle could also be considered as vehicle 10. Examples of other vehicles include buses, commercial vehicles, particularly trucks, agricultural machinery, construction machinery, rail vehicles, etc. The invention could generally be used in land vehicles, rail vehicles, watercraft, and aircraft.

[0048] The cockpit features two display units of an infotainment system. These are a touchscreen 30 located in the center console and the instrument cluster 110, which is mounted in the dashboard. While driving, the center console is not in the driver's field of vision. Therefore, additional information is not displayed on the display unit 30.

[0049] The touch-sensitive screen 30 serves in particular to operate functions of the vehicle 10. For example, it can be used to control a radio, a navigation system, playback of stored music tracks and / or an air conditioning system, other electronic devices, or other comfort functions or applications of the vehicle 10. This is often referred to collectively as an "infotainment system." In motor vehicles, especially cars, an infotainment system refers to the combination of a car radio, navigation system, hands-free system, driver assistance systems, and other functions in a central control unit. The term infotainment is a portmanteau of the words "information" and "entertainment."The touch-sensitive screen 30 ("touchscreen") is primarily used to operate the infotainment system. This screen 30 can be easily viewed and operated, particularly by a driver of the vehicle 10, but also by a passenger of the vehicle 10. Mechanical control elements, such as buttons, rotary controls, or combinations thereof, such as push-button controls, can also be arranged below the screen 30 in an input unit 50. Steering wheel operation of parts of the infotainment system is typically also possible. This unit is not shown separately but is considered part of the input unit 50.

[0050] Fig. 1 also shows a head-up display 20. The head-up display 20 is mounted in the vehicle 10, from the driver's perspective, behind the instrument cluster 110 in the dashboard area. It is an image projection unit. By projecting onto the windshield, additional information is displayed in the driver's field of vision. This additional information appears as if it were projected onto a projection surface 21 located 7 - 15 m in front of the vehicle 10. However, the real world remains visible through this projection surface 21. With the displayed additional information, a virtual environment is essentially created. The virtual environment is theoretically superimposed over the real world and contains the virtual objects that support and inform the driver while driving. However, it is only projected onto part of the windshield, so that the additional information cannot be arranged arbitrarily in the driver's field of vision.This type of representation is also known as “augmented reality”.

[0051] Fig. Figure 2 shows a system architecture for vehicle communication via mobile radio. The vehicles 10 are equipped with an on-board communication module 160 with a corresponding antenna unit, allowing them to participate in the various types of vehicle-to-vehicle communication, V2V and V2X. Fig. 2 shows that the vehicle 10 can communicate with the mobile radio base station 210 of a mobile radio provider.

[0052] Such a base station 210 may be an eNodeB base station of an LTE (Long Term Evolution) mobile operator. The base station 210 and the corresponding equipment are part of a mobile communications network with a plurality of mobile radio cells, each cell being served by a base station 210.

[0053] The base station 210 is positioned near a main road on which the vehicles 10 travel. In LTE terminology, a mobile terminal corresponds to user equipment (UE), which enables a user to access network services by connecting to the UTRAN or the Evolved UTRAN via the radio interface. Typically, such user equipment corresponds to a smartphone. Such mobile terminals are used by the passengers in the vehicles 10. In addition, the vehicles 10 are each equipped with an on-board communication module 160. This on-board communication module 160 corresponds to an LTE communication module, with which the vehicle 10 can receive mobile data (downlink) and transmit such data in the uplink direction. This on-board communication module 160 can further be equipped with a WLAN p module in order to participate in an ad hoc V2X communication mode. V2V and V2X communication is also supported by the new 5th generation of mobile radio systems. The corresponding radio interface is referred to as the PC5 interface. In the LTE mobile radio communication system, the Evolved UMTS Terrestrial Radio Access network (E-UTRAN) of LTE consists of several eNodeBs that provide the E-UTRA user plane (PDCP / RLC / MAC / PHY) and the control plane (RRC). The eNodeBs are interconnected via the so-called X2 interface. The eNodeBs are also connected to the EPC (Evolved Packet Core) 200 via the so-called S1 interface.

[0054] From this general architecture, Fig. 2, that the base station 210 is connected to the EPC 200 via the S1 interface, and the EPC 200 is connected to the Internet 220. A backend server 240, to which the vehicles 10 can send and receive messages, is also connected to the Internet 220. In the case under consideration here, the backend server 240 can be located in a data center of the vehicle manufacturer. Finally, a road infrastructure station 230 is also shown. This can be illustrated, for example, by a roadside unit, often referred to in technical jargon as a Road Side Unit (RSU) 230. To simplify the implementation, it is assumed that all components have been assigned an Internet address, typically in the form of an IPv6 address, so that the packets transporting messages between the components can be routed accordingly. The various interfaces mentioned are standardized.In this regard, reference is made to the relevant LTE specifications that have been published.

[0055] Fig. Figure 3 shows a schematic block diagram of the on-board electronics, which also includes the infotainment system of the vehicle 10. The infotainment system is operated by the touch-sensitive display unit 30, a computing device 40, an input unit 50, and a memory 60. The display unit 30 includes both a display surface for displaying variable graphic information and a user interface (touch-sensitive layer) arranged above the display surface for inputting commands by a user. It can be designed as an LCD touchscreen display.

[0056] The display unit 30 is connected to the computing device 40 via a data line 70. The data line can be designed according to the LVDS standard, corresponding to Low Voltage Differential Signaling. The display unit 30 receives control data for controlling the display surface of the touchscreen 30 from the computing device 40 via the data line 70. Control data for the entered commands are also transmitted from the touchscreen 30 to the computing device 40 via the data line 70. Reference number 50 designates the input unit. Associated with this unit are the aforementioned control elements, such as buttons, rotary controls, slide controls, or rotary push buttons, with which the operator can make inputs via the menu navigation. Input is generally understood to mean selecting a selected menu option, changing a parameter, switching a function on or off, etc.

[0057] The memory device 60 is connected to the computing device 40 via a data line 80. A pictogram and / or symbol directory is stored in the memory 60 containing the pictograms and / or symbols for the possible display of additional information.

[0058] The other parts of the infotainment system, camera 150, radio 140, navigation device 130, telephone 120, and instrument cluster 110, are connected to the device for operating the infotainment system via data bus 100. The high-speed variant of the CAN bus according to ISO Standard 11898-2 can be used as data bus 100. Alternatively, a bus system based on Ethernet technology, such as BroadR-Reach, could also be used. Bus systems in which data is transmitted via fiber optic cables can also be used. Examples include the MOST bus (Media Oriented System Transport) or the D2B bus (Domestic Digital Bus). A vehicle measuring unit 170 is also connected to data bus 100. This vehicle measuring unit 170 is used to record the movement of the vehicle, in particular to record the acceleration of the vehicle. It can be designed as a conventional IMU unit, corresponding to an Inertial Measurement Unit.An IMU unit typically contains acceleration sensors and angular rate sensors such as a laser gyroscope or a magnetometer gyroscope. The vehicle measurement unit 170 can be considered part of the odometry of the vehicle 10. This also includes the wheel speed sensors.

[0059] It should also be mentioned here that the camera 150 can be designed as a conventional video camera. In this case, it records 25 full frames per second, which corresponds to 50 half frames per second in the interlace recording mode. Alternatively, a special camera can be used that records more frames per second to increase the accuracy of object detection for faster-moving objects, or that records light in a spectrum other than the visible spectrum. Multiple cameras can be used for environmental monitoring. In addition, the previously mentioned RADAR or LIDAR systems 152 and 154 are also used in addition or alternatively to perform or expand environmental monitoring. For wireless communication inside and outside, the vehicle 10 is equipped with the communication module 160, as already mentioned.

[0060] The camera 150 is primarily used for object detection. Typical objects to be detected are traffic signs, vehicles driving ahead, surrounding vehicles, and other road users, intersections, turning points, potholes, etc. If an object with potential significance is detected, information can be output via the infotainment system. Typically, symbols for the detected traffic signs are displayed, for example. It can also be a warning message if the object poses a hazard. The information is projected directly into the driver's field of vision via the HUD 20. An example of an object posing a hazard is the case where the image analysis of the images supplied by the camera 150 has shown that a vehicle is approaching an intersection from the right, towards which the vehicle itself is moving. The image analysis takes place in the computing unit 40.Known object detection algorithms can be used for this purpose. As a result, a hazard symbol is displayed at the vehicle's position. The display should be positioned in such a way that the hazard symbol does not obscure the vehicle, as otherwise the driver would not be able to clearly identify the hazard.

[0061] The object detection algorithms are processed by the computing unit 40. The number of images that can be analyzed per second depends on the performance of this computing unit. Typically, an automated driving function is divided into different phases: In the perception phase, data from various environmental sensors is processed and consolidated on a high-performance platform. Furthermore, the vehicle must be located on a highly accurate digital map based on its GNSS position. The data is used to generate a three-dimensional environmental model and provides information on dynamic and static objects as well as open spaces around the vehicle (object list). Under certain conditions, the accuracy of the GNSS position is insufficient. Therefore, the odometry data in the vehicle is also used to improve the accuracy of the positioning.

[0062] Reference number 181 designates an engine control unit. Reference number 182 corresponds to an ESP control unit, and reference number 183 designates a transmission control unit. Additional control units, such as an additional vehicle dynamics control unit (for vehicles with electrically adjustable dampers), an airbag control unit, etc., may be present in the vehicle. Networking of such control units, all of which fall into the powertrain category, is typically achieved using the CAN (Controller Area Network) bus system 104, which is standardized as an ISO standard, usually as ISO 11898-1. Various sensors 171 to 173 in the motor vehicle, which are no longer intended to be connected to individual control units, are also intended to be connected to the bus system 104, and their sensor data is transmitted via the bus to the individual control units.Examples of sensors in a motor vehicle include wheel speed sensors, steering angle sensors, acceleration sensors, yaw rate sensors, tire pressure sensors, distance sensors, knock sensors, air quality sensors, etc. In particular, the wheel speed sensors and steering angle sensors are part of the vehicle's odometry. The acceleration sensors and yaw rate sensors can also be connected directly to the vehicle measuring unit 170.

[0063] The Fig. 4 and Fig. Figure 5 shows the basic functionality of an image analysis system as used in vehicles. It shows the camera image captured by the front camera 150 of vehicle 10.

[0064] In Fig. Figure 4 depicts a situation in which a vehicle is approaching a priority road. Only shortly before turning onto the priority road is a crossing vehicle 12 with right of way detected by evaluating the camera images and / or data from radar and lidar sensors 154, 152. The autonomous vehicle 10 must therefore brake sharply. The need for particularly harsh braking creates an unpleasant situation for the vehicle occupants. Such braking maneuvers should be avoided, especially for autonomously driving vehicles, in order to give the vehicle occupants a sense of safety. At least during the test phase, this incident is classified as undesirable by the on-board electronics. The same can also occur during normal operation, i.e., after the test phase. There are several possibilities for this. A first possibility is that this classification occurs when a specified acceleration value is exceeded.Another possibility is to perform classification when a negative expression from the occupants or driver is observed by the driver / occupant state detection system. Another possibility is to perform classification when a specified minimum distance to object detection (in this case, the vehicle approaching from the right) has been exceeded.

[0065] In order to reduce the amount of image data to be saved, only those image sections from the ring buffer in which the object may have been located should be saved / transferred.

[0066] The Fig. Figure 5 shows a camera image of the same driving situation at an earlier point in time. Vehicle 10 approaching from the right is even further away from the intersection. Fig. 5 even shows a camera image from a time when the object detection system had not yet detected the vehicle as an approaching vehicle. This can have various causes. The vehicle is still too far away, and the object detection algorithm could not yet detect the vehicle with the available image information. The vehicle is obscured by other objects (e.g., trees or bushes).

[0067] The reason why the approaching vehicle was detected so late will be determined through subsequent analysis. For this purpose, the recorded camera images and, if applicable, the images recorded by other imaging sensors will be identified and archived for subsequent analysis. The amount of data to be archived will be reduced as much as possible.

[0068] How this happens is explained below with the help of the Fig. 6 explained. Fig.Figure 6 shows a flowchart for a program that implements the verification function. Reference number 310 denotes the program start. The program is always started when a dangerous situation has been detected in which a dynamic object was detected too late. In program step 312, the trajectory for the dynamic object detected too late is estimated. This occurs as follows: At the time of object detection, the current speed and acceleration of the detected object are determined from several consecutive video images using the various sensor data from the observer vehicle 10. With this, and if necessary, using the route geometry known from the map of the navigation system 130, a trajectory of the detected object is estimated.In conjunction with the vehicle's own motion estimation, an image section can be determined from the recorded image data for points in time in the past, within which the object (not yet detected at that time) would have been located. This image section is also referred to as the bounding box. It can be assumed that the bounding box will be larger at earlier points in time than at points in time not so far in the past. The size of the bounding box is influenced by the object parameters (e.g. size) at the time of detection, but also by their uncertainties, possible alternatives in the object trajectory, known sensor errors / inaccuracies and empirically determined factors.

[0069] In program step 314, the estimated trajectory is used for a backward calculation to determine a number of problematic images, which are to be used to check why the approaching vehicle 12 could not be detected in them. The distance between the observer vehicle 10 and the approaching vehicle 12 is also taken into account. If the distance exceeds a predetermined limit, no further past images need to be archived. For the various environmental observation sensors, the distances up to which the desired objects can still be detected are known. The program then continues with program step 316. In program step 316, the number of problematic images is calculated.

[0070] In program step 316, the respective bounding boxes for the number of problematic images are calculated. This is also done taking the estimated trajectory into account.

[0071] In another variant, steps 314 and / or 316 also determine whether non-recognition may have been caused by the relevant object possibly being covered by other, possibly correctly detected objects (other vehicles, houses / walls / noise barriers known from navigation data, trees, bushes, forests). In this case, the images may not be transmitted. The period of problematic images may be limited by the fact that the presumed object trajectory suggests that the relevant object was not yet in the camera's range at an earlier point in time.

[0072] In program step 318, the problematic image sections are then transferred from the ring buffer of the storage device 60 to the communication module 160, and the communication module 160 sends this image data via mobile radio to the backend server 240. There, they are archived. The archived images are later evaluated in the data center by experts or by machine using artificial intelligence. The purpose of the check is to identify potential problems with the image evaluation system. The results of the checks can be used to improve the evaluation algorithms. Ideally, the improved evaluation algorithm can be transmitted back to the vehicle 10 via OTA (over-the-air) download and installed there. However, it may also be the case that the check reveals a systematic error due to a misalignment of the camera 150.In this case, a message could be sent to vehicle 10 to inform the driver to visit the workshop. The program ends in program step 320.

[0073] The number and quality of images to be archived can be influenced by various factors. In addition to the cases already presented, the following influencing factors are particularly important: Weather conditions

[0074] In adverse weather conditions, visibility is severely limited. This can be so severe that object detection is no longer possible. However, the other environmental detection sensors, RADAR sensors and LIDAR sensors, can provide better results in this case. In any case, it can be planned that the number of images or image sections to be archived is limited in adverse weather conditions because the corresponding objects cannot be detected from the standard detection distance. Position accuracy

[0075] The accuracy of positioning based on GNSS and odometry signals can also be weather-dependent. However, it can also depend on other influencing factors. One example is the environment in which the vehicle is traveling. In cities, heavily built-up areas can limit satellite signal reception. This can also be the case during cross-country journeys. Satellite signals can be weakened in forests. In mountains, tectonics can also lead to poor reception. In such cases, it is recommended that more images or image sections be recorded. This makes it more likely that the relevant route sections were recorded despite the positioning inaccuracy. Road surface condition

[0076] Road surface conditions can also have a similar impact. Cobblestones are subject to strong vibrations, which can result in blurred images. When the road surface is slippery due to ice, snow, or rain, the traction control system estimates the friction coefficient. If the slip is sufficiently large, the odometry data is no longer as reliable, and this influence should be taken into account, just as it is for the influence of positioning inaccuracies. time

[0077] Obviously, image quality will vary significantly depending on the time of day. At least a distinction should be made between day and night. At night, data from radar or lidar sensors should be recorded rather than camera image data. Traffic conditions

[0078] Here, a distinction can be made between whether the vehicle is traveling in city traffic, on the highway, on a country road, etc. In city traffic, particularly accurate detection of traffic signs is crucial. In this case, the number of recorded images can be increased. In traffic jams on the highway, however, the number of recorded images can be reduced.

[0079] The images or image sections 14 to be archived are temporarily stored in the vehicle 10. The memory 60 can be used for this purpose. In practical implementation, a ring buffer is set up in the memory 60. This is managed in such a way that the newly recorded images are written one after the other to the free memory area of the ring buffer. When the end of the free memory area is reached, the already written part of the ring buffer is overwritten. By managing the allocated memory area as a ring buffer, the oldest part of the memory is always overwritten. The images or image sections 14 can be stored uncompressed or compressed. However, it is recommended to use a lossless compression method to prevent any relevant image content from being lost. The FFmpeg codec is cited as an example of a lossless compression method.The image data is saved in a corresponding file format. Various data container formats are possible here. Examples include the ADTF format, which was developed for the automotive sector, and the TIFF format. Other container formats for saving image and sound data are MPEG, Ogg, Audio Video Interleave, DIVX, Quicktime, Matroska, etc. If the images are transferred to the backend server 240 while driving, it may be sufficient to design the ring buffer for a recording time of 20 seconds. A separate storage device can also be provided in the vehicle for archiving the images. A removable USB hard drive, for example, is suitable for this purpose. This can be connected to a computer used to evaluate the archived images.

[0080] All examples and conditional language mentioned herein are to be understood without limitation to such specifically recited examples. For example, it will be appreciated by those skilled in the art that the block diagram shown herein represents a conceptual view of an exemplary circuit arrangement. Similarly, it will be appreciated that a shown flowchart, state transition diagram, pseudocode, and the like represent various variations for representing processes that can be substantially stored in computer-readable media and thus executed by a computer or processor. The object recited in the claims can also expressly be a person.

[0081] It should be understood that the proposed method and associated devices can be implemented in various forms of hardware, software, firmware, special-purpose processors, or a combination thereof. Special-purpose processors may include application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs), and / or field-programmable gate arrays (FPGAs). Preferably, the proposed method and device are implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. Typically, this is a machine based on a computer platform that includes hardware such as one or more central processing units (CPUs), random access memory (RAM), and one or more input / output (I / O) interfaces.An operating system is also typically installed on the computer platform. The various processes and functions described here may be part of the application program or a part executed by the operating system.

[0082] The disclosure is not limited to the embodiments described herein. There is room for various adaptations and modifications that a person skilled in the art would consider based on their technical knowledge and within the scope of the disclosure.

[0083] The invention is explained in more detail in the exemplary embodiments using the example of its use in vehicles. Reference is also made here to the possible use in aircraft and helicopters, for example, during landing maneuvers or search missions, etc.

[0084] The invention can also be used in remote-controlled devices such as drones and robots, where image analysis is highly important. Other possible applications include smartphones, tablet computers, personal assistants, and data glasses. List of reference symbols 10 Observer Vehicle 12 approaching vehicle 14 relevant image section 20 Head-Up Display 30 touch-sensitive display unit 40 computing unit 50 input unit 60 storage units 70 Data line to the display unit 80 Data line to the storage unit 90 Data line to the input unit 100 data bus 110 instrument cluster 120 Telephone 130 navigation device 140 Radio 142 Gateway 144 on-board diagnostic interface 150 Camera 160 communication module 170 vehicle measuring unit 171 Sensor 1 172 Sensor 2 173 Sensor 3 180 Control unit for automatic driving function 181 Engine control unit 182 ESP control unit 183 Transmission control unit 200 Evolved Packet Core 210 base station 220 Internet 230 road infrastructure station 240 backend servers 310 - 320 different program steps of a computer program

Claims

[1] Method for determining and archiving problematic image sections (14) for the subsequent checking of an image evaluation system of a vehicle (10), wherein it is checked whether an object recognition for a moving object (12) by the image evaluation system was carried out correctly in terms of time or location, wherein, if deviations beyond a tolerance limit are detected during the temporal or spatial analysis, it is determined which image sections (14) are to be identified and archived for more detailed examination, wherein the images are temporarily stored in an allocated memory area of a memory (60) in the vehicle (10) managed as a ring buffer, wherein the step of determining the problematic image sections (14) to be archived includes that at the time of detection of the moving object (12), a current speed and acceleration of the moving object (12) is determined from several successive images using various sensor data of the vehicle (10) and a trajectory of the moving object (12) is estimated, wherein, with the aid of the estimated trajectory and in conjunction with an estimation of the vehicle's own motion (10), a backward calculation is carried out to determine a number of problematic images to be used to check why the moving object (12) could not be detected, wherein for the number of determined problematic images, respective image sections (14) are determined for points in time in the past, within which the not yet detected moving object (12) would have had to be located at that point in time, wherein a size of a respective image section (14) is influenced by parameters of the moving object (12) and their uncertainties, possible alternatives in the calculated trajectory of the moving object (12) and known sensor inaccuracies, and wherein the determined problematic image sections (14) are transmitted from the ring buffer of the memory (60) via a communication module (160) to an external archiving location (240). [2] Method according to claim 1, wherein an object recognition algorithm is used for image evaluation, and wherein, in order to check whether object recognition was carried out correctly, it is checked at which distance from the recognized object (12) the object recognition was carried out, wherein a standard recognition distance is defined which indicates from which distance the object recognition algorithm should provide object recognition. [3] Method according to claim 2, wherein the image evaluation system for detecting deviations in the object detection evaluates position data of the vehicle (10) and position data of the detected object (12) to determine a distance between the vehicle (10) and the object (12). [4] Method according to one of the preceding claims, wherein vehicles equipped with automatic driving function themselves calculate a trajectory for the vehicle's own movement, and wherein this calculated trajectory is transmitted to surrounding vehicles with the aid of direct vehicle communication, so that the positions are estimated in the vehicle (10) where the object (12) should have been located when it had not yet been detected by the object detection. [5] Method according to one of the preceding claims, wherein a separate memory is provided in the vehicle on which the problematic image sections (14) are archived, and wherein the separate memory is removable for evaluating the archived problematic image sections (14). [6] Method according to one of the preceding claims, wherein further images or image sections (14) are archived, and wherein the problematic image sections (14) determined with the aid of the trajectory are archived with high quality and the further images or image sections are archived with a lower quality. [7] Apparatus comprising an image generating device, a computing device (40) and a storage device (60), wherein the computing device (40) is designed to carry out a method according to one of claims 1 to 6. [8] Vehicle (10) for use in the method according to any one of claims 1 to 6, wherein the vehicle (10) is equipped with a device according to claim 7. [9] Computer program, wherein the computer program is designed to carry out the steps of the method according to one of claims 1 to 6 when executed in a computing device (40).

Images