METHOD FOR CAPTURING IMAGE MATERIAL FOR VERIFYING IMAGE ANALYSIS SYSTEMS, DEVICE AND VEHICLE FOR USE IN THE METHOD, AS WELL AS COMPUTER PROGRAM

Description

[0001] The invention relates to the technical field of driver assistance systems up to and including autonomous vehicles. The invention relates in particular to a method for acquiring image data for the verification of image processing systems. The invention also relates to a device and a vehicle for use in the method, as well as a computer program.

[0002] Currently, intensive work is underway on technologies that will eventually enable autonomous driving. A first approach is the introduction of various driver assistance systems that relieve the driver of certain tasks. Examples of driver assistance systems include blind spot monitoring, emergency braking, parking assist, turning assist, lane keeping assist, cruise control, and so on. A further stage of development could involve combining several assistance systems. This would not completely relieve the driver of their duties, but rather ensure that the driver can take control of the vehicle at any time. The driver would then also perform monitoring functions.

[0003] Therefore, in the near future, it can be assumed that the use of newer technologies (vehicle-to-vehicle communication, databases, backend connectivity, cloud services, server deployment, vehicle sensors, etc.) will make comprehensive information about objects (especially vehicles) in the visible and hidden / invisible environment of the vehicle available. In the area of ​​vehicle sensors, the following components are particularly relevant for environmental monitoring: radar devices (Radio Detection and Ranging), lidar devices (Light Detection and Ranging), primarily for distance detection / warning, and cameras with corresponding image processing for object recognition. Ultrasound and infrared sensors are also mentioned.

[0004] This environmental data can thus be used as a basis for system-based driving recommendations, warnings, etc. For example, displays / warnings are conceivable indicating in which direction (possibly in the driver's own trajectory) another, surrounding vehicle intends to turn. Traffic sign recognition is also mentioned as an application, which is important for determining the legal framework.

[0005] Vehicle-to-vehicle communication plays a crucial role in autonomous driving. Mobile communication systems such as LTE (Long Term Evolution) and 5G have been developed that support vehicle-to-vehicle communication. Alternatively, WLAN-based systems are available for direct vehicle communication, particularly the WLANp system. The term "autonomous driving" is used somewhat differently in the literature.

[0006] To clarify this term, the following explanation is provided. Autonomous driving (sometimes also called automatic driving, automated driving, or piloted driving) refers to the movement of vehicles, mobile robots, and driverless transport systems that behave largely autonomously. There are different levels of autonomous driving. At certain levels, the term "autonomous driving" is still used even if a driver is present in the vehicle, who may only be responsible for monitoring the automated driving process. In Europe, the various transport ministries (in Germany, the Federal Highway Research Institute was involved) have collaborated and defined the following levels of autonomy. Level 0: "Driver only," the driver drives, steers, accelerates, brakes, etc. Level 1: Certain driver assistance systems help with vehicle operation (including adaptive cruise control - ACC). Level 2: Partial automation. Features such as automatic parking, lane keeping, general longitudinal control, acceleration, braking, etc., are handled by the assistance systems (including traffic jam assist). Level 3: High automation. The driver does not need to constantly monitor the system. The vehicle independently performs functions such as activating the turn signals, changing lanes, and maintaining lane position. The driver can attend to other tasks but will be prompted by the system to take over driving if necessary, within a reasonable warning time. This form of autonomy is technically feasible on highways. Legislators are working towards approving Level 3 vehicles. The legal framework for this has already been established. Level 4: Full automation.The system permanently takes over vehicle control. If the system can no longer handle the driving tasks, the driver may be prompted to take over. Level 5: No driver required. Apart from setting the destination and starting the system, no human intervention is necessary.

[0007] Automated driving functions from level 3 onwards relieve the driver of responsibility for controlling the vehicle. The German Association of the Automotive Industry (VDA) has published a similar classification of the different levels of autonomy, which can also be used.

[0008] Due to the current trend towards higher levels of autonomy, where many vehicles are still controlled by the driver, it can be assumed that corresponding additional information can be used in the medium term for manually driven vehicles and not only in the long term for highly automated systems.

[0009] For driver-vehicle interaction, the question arises as to how this information can be presented in a way that creates genuine added value for the human driver and allows them to quickly and intuitively locate the information provided. The following solutions in this area are already known from the state of the art.

[0010] A future vision in the automotive industry is the ability to project virtual elements onto the windshield of one's own vehicle, offering the driver several advantages. This utilizes so-called "Augmented Reality" (AR) or "Mixed Reality" (MR) technology. The corresponding German terms "erweiterte Realität" (extended reality) and "gemischte Realität" (mixed reality) are less common. In this process, the real environment is enhanced with virtual elements. This has several advantages: Looking down at displays other than the windshield becomes unnecessary, as much relevant information appears when looking through the windshield. Thus, the driver doesn't have to take their eyes off the road. Furthermore, the precise positioning of the virtual elements within the real environment likely reduces the cognitive effort required from the driver, as there is no need to interpret graphics on a separate display.With regard to automated driving, added value can also be generated. In this respect, reference is made to the article "3D-FRC: Depiction of the future road course in the Head-Up Display" by CA Wiesner, M. Ruf, D. Sirim and G. Klinker in the 2017 IEEE International Symposium on Mixed and Augmented Reality, in which these advantages are explained in more detail.

[0011] Given the current limitations of technological advancements, it's unlikely that fully functional windshields will be common in vehicles in the medium term. Currently, head-up displays (HUDs) are used in vehicles. These have the advantage of projecting an image closer to the driver's real-world surroundings. These displays are essentially projection units that project an image onto the windshield. However, depending on the module's design, this image is positioned a few to 15 meters in front of the vehicle from the driver's perspective.

[0012] The "image" is composed as follows: It's less of a virtual display and more of a "keyhole" into the virtual world. The virtual environment is theoretically overlaid on the real world and contains the virtual objects that support and inform the driver. The limited display area of ​​the HUD means that only a portion of it can be seen. You're essentially looking through the HUD's display area at this section of the virtual world. Because this virtual environment complements the real world, it's also referred to as "mixed reality."

[0013] A major advantage of existing augmented reality (AR) displays is their ability to integrate information directly into the surrounding environment. Common examples relate to navigation. While traditional navigation displays (in conventional head-up displays) typically show schematic representations (e.g., a right-angled arrow indicating a right turn), AR displays offer significantly more effective solutions. Because the displays can be integrated into the environment, users can interpret information quickly and intuitively. Drivers, as well as more advanced automated driving systems, must be able to rely on the initial, accurate detection of objects. If additional information is derived from this data, it should be displayed in the appropriate location within the image.

[0014] Therefore, it is necessary to demonstrate the correct functioning of the image capture systems, at least during testing. This can also be implemented during normal operation. Public transport is highly complex, and so many moving road users can be involved that obstructions can occur. Other problems include limitations in visibility due to various weather conditions such as fog, rain, snow, etc., backlighting, darkness, and other factors.

[0015] A vehicle-based traffic sign recognition system is known from US patent 2016 / 0170414 A1. This system uses environmental sensors such as LiDAR, radar, and cameras. The vehicle's position is also recorded via GPS. The recognized traffic signs and their positions are transmitted externally and entered into a database.

[0016] US patent 2017 / 0069206 A1 discloses a vehicle-based system for verifying traffic sign recognition. This system also utilizes environmental sensors such as LiDAR, radar, and cameras. The recognized traffic signs and their positions are transmitted externally and entered into a database. To detect false recognitions, various verification methods are proposed, including manual visual inspections by qualified personnel, crowdsourcing visual inspections, computer-aided analysis, and statistical analysis.

[0017] US patent 2018 / 0260639 A1 discloses a vehicle-based system and a method for traffic sign recognition. The recognized traffic signs are stored with their position in an image processing unit.

[0018] From US 2008 / 0239078 A1 it is known to display detected objects in the video images of a camera used in a vehicle for environmental monitoring with additional information derived from other sensors in the vehicle.

[0019] US patent 2018 / 0012082 A1 discloses a method for image analysis in which a sequence of images is captured by a camera mounted on a vehicle. An object recognition module is included in the image evaluation system. A bounding box is defined, which serves as a border around a recognized object for highlighting purposes.

[0020] US patent 2018 / 0201227 A1 discloses a vehicle environment monitoring system in which the vehicle's surroundings are monitored by a camera. The vehicle also includes a display unit for showing the video images captured by the camera. An image evaluation unit is provided that monitors the image data for the presence of moving external objects in the environment. Furthermore, a controller is programmed to determine a threat assessment value based on the conditions near the vehicle and uploads image data to an external server if the threat assessment value exceeds a first threshold.

[0021] From the article by J. Liu et al.: "Rewind to track: Parallelized apprenticeship learning with backward tracklets", 2017 IEEE International Conference on Multimedia and Expo (ICME), 2017, pp. 433-438, a multi-object tracker is known that incorporates future observations to process offline data. Tracking multiple objects is modeled by considering each object as an agent following a Markov decision process. Reversing the underlying video sequence generates backward tracklets.

[0022] The known solutions have several disadvantages. This was recognized within the scope of the invention. With currently known image acquisition methods for use in driver assistance systems, the problem is that their correct function during operation is not, or only insufficiently, verified. Therefore, false detections or even misclassifications of objects are possible.

[0023] Therefore, there is a need for further improvements in the verification of vehicle-based image acquisition systems, especially for verification procedures that allow for subsequent verification in case of false detections.

[0024] The invention aims to find such an approach. It also seeks to minimize the effort required for archiving image data as a prerequisite for subsequent verification. This objective is achieved by a method for acquiring image material for verifying image evaluation systems according to claim 1, a device and a vehicle according to claims 7 and 12, and a computer program according to claim 13.

[0025] The dependent claims include advantageous further developments and improvements of the invention in accordance with the following description of these measures.

[0026] In another patent application by the applicant, ways are shown how, with the help of a ring buffer for recorded images from environmental sensors and the consideration of odometry data of the vehicle, only certain image sections can be considered for data recording / transmission to a backend in order to reduce the required amount of data.

[0027] According to one embodiment of the invention, a method for acquiring image material for the verification of image evaluation systems comprises the following steps: Recording images captured by a vehicle's image generation system in a memory; performing object recognition using an image processing system; determining the distance between the vehicle and an object at which the object was first correctly recognized; establishing that the determined distance deviates from a standard recognition distance for object recognition beyond a tolerance limit; identifying those images or image sections that were captured between the determined distance and the standard recognition distance; and archiving the identified images or image sections.

[0028] This method offers the advantage that an image processing system providing safety-relevant data can be checked during operation, with minimal storage requirements for the check. The method is particularly advantageous for testing image processing systems still under development. However, it can also be used effectively in series production vehicles. This is beneficial for implementing subsequent improvements to the installed image processing system or for determining whether a camera or sensor is misaligned, thus necessitating vehicle maintenance.

[0029] If the distance at which the object was actually detected deviates from the standard detection distance, according to the invention those images or image sections that were taken from the standard detection distance to the actual object detection distance are archived for more precise verification.

[0030] This solution can be particularly advantageous for static objects, such as traffic signs. However, with regard to autonomous driving, it will also be necessary to recognize moving objects in the recorded images. The corresponding image analysis algorithms are still more prone to errors. To improve these algorithms, subsequent reviews are also required in the event of incorrect detection. Therefore, there is also a further need for improvements in the verification of vehicle-based image acquisition systems.

[0031] To enable this to be performed for dynamic objects as well, one embodiment of the invention calculates or obtains a trajectory for the moving object in which the deviation was detected. In the future, vehicles equipped with automatic driving functions will themselves calculate a trajectory for their own movement. They can then transmit this calculated trajectory to surrounding vehicles using direct vehicle communication. This allows the observing vehicle to estimate the positions where the object should have been when it was not yet detected by the object recognition system. Through reverse calculation, the images and image sections relevant for subsequent analysis are then determined. The invention further provides that the image sections determined in this way are extracted from the recorded images and archived only to the extent necessary.This is done using the vehicle's odometry and position data, as well as the trajectory of the relevant dynamic object. The dynamic object is usually another road user. To be able to retrieve subsequently determined images or image sections, a ring buffer for image data is required, the size of which (i.e., maximum storage duration) may depend on the function and possibly on the vehicle's speed.

[0032] It is advantageous if, with the help of the trajectory, a number of images or image sections are determined in which the moving object was presumably visible, even though the object recognition could not detect the moving object in them.

[0033] In an enhanced embodiment, an initial analysis of the problematic image data can be performed directly in the vehicle. This could, for example, involve determining whether temporary obscuration of the object by other objects could explain the false detection. This occurs frequently in public road traffic, for instance, when buses or trucks obscure road signs. In such cases, the false detection would be explainable, and no data would be archived, or only a reduced data set (e.g., only a few seconds before the first successful object detection) would be stored.

[0034] The invention can be particularly advantageous for testing image processing systems in vehicles. These vehicles are now equipped with imaging environmental sensors such as cameras, LiDAR, or radar sensors. These sensors detect traffic signs, vehicles ahead and other road users, intersections, turning points, potholes, etc. Such vehicles are also equipped with position detection systems. The determined position of the vehicle can be used to check for deviations in image recognition. In the simplest case, the vehicle detects the position of the detected object, even if it passes by the object's location. The distance between the vehicle and the object can then be calculated. In another embodiment, the object's position can be taken from a high-precision map. In a further variant, the position can be estimated based on the vehicle's position.

[0035] Satellite navigation and / or odometry are suitable for determining the position data of the vehicle and the object. A general term for such satellite navigation systems is GNSS, which stands for Global Navigation Satellite System. Existing satellite navigation systems include the Global Positioning System (GPS), Galileo, GLONASS (Globalnaya Navigatsionnaya Sputnikovaya Sistema), and BeiDou.

[0036] In one embodiment, the images or image excerpts to be archived are sent to an external archiving facility. This has the advantage that only the problematic images need to be archived. Archiving at an external location also eliminates the need to archive the images in the vehicle itself. The evaluation is intended to be carried out at the external location by experts who will then also be able to improve the image evaluation system.

[0037] In another option, the images or image excerpts to be archived are stored in a memory unit located in the vehicle. This requires the vehicle to be equipped with the necessary storage space. The archived images can later be retrieved by experts, for example, during a service appointment. Another advantageous option is to transfer the images temporarily stored in the vehicle to an external location once the vehicle is back at the owner's residence. Modern vehicles are equipped with Wi-Fi. If this connection is established within the vehicle owner's private Wi-Fi network, the archived image data can be transferred at a high data rate. Compared to direct transmission via a mobile communication system to which the vehicle is connected while driving, this has the advantage of lower costs for the vehicle owner and less strain on the mobile network.Its capacity is utilized by a variety of other applications.

[0038] Alternatively, it can be stipulated that additional images or image sections be archived to expand the verification possibilities. It is advantageous if the images or image sections captured from the standard recognition distance to the object recognition distance are archived at high quality, while the additional images or other image sections are archived at lower quality. This keeps the storage and transmission costs for archiving additional images or image sections to a minimum.

[0039] In addition, there is another advantageous option to determine the size of the image sections to be archived and / or the recording period of the images / image sections to be archived depending on one or more of the following environmental parameters: Accuracy of the vehicle and / or object position determination, time of day, especially with day / night differentiation, weather conditions, traffic conditions, road surface conditions.

[0040] This is always advantageous when environmental conditions are not constant. In practice, environmental conditions are almost always changing. If the same number of images or image sections are always archived, it could easily happen that important scenes are missed under changing environmental conditions, and thus the causes of the problems cannot be found.

[0041] For a device to be used in the method according to the invention, it is advantageous if the device comprises an image generation unit, a computing unit, and a storage unit. The storage unit is designed to record images supplied by the image generation unit. The computing unit is designed as follows: to perform object recognition on the images supplied by the image generation device; to determine a distance at which an object was first correctly recognized; to ascertain that the determined distance deviates from a standard recognition distance for object recognition beyond a tolerance limit; to identify those images or image sections that were captured between the determined distance and the standard recognition distance; and to issue a command to archive the identified images or image sections.

[0042] Furthermore, it is advantageous if the computing system is designed to calculate the trajectory of the moving object and, based on this trajectory, determine which images or image sections should be archived. By applying the trajectory, the images or image sections to be archived can be more precisely defined, thereby reducing the archiving effort.

[0043] In a particularly advantageous embodiment, the device additionally includes a communication module, which is designed to send the acquired images or image sections to an external archiving location upon receiving the command to archive these acquired images or image sections. The external archiving location can be a data center of the manufacturer of the image processing system or the vehicle.

[0044] In addition, a further variant is proposed in which the device includes a storage unit designed to save the captured images or image excerpts upon receiving the command to archive them. In this variant, archiving takes place in the vehicle until the data is read out. As described above, the data could be read out in a workshop, or upon returning to the vehicle, it could be transferred to the external archiving location via the home Wi-Fi network.

[0045] For the vehicle-side temporary storage of data, it is advantageous if the storage device is designed as a ring buffer, in which, in the event of a buffer overflow, the oldest previously stored images or image sections are overwritten by the new images or image sections.

[0046] Advantageously, a video camera or a LiDAR or radar sensor can be used as the image generation device. For the wireless communication interface, it is advantageous to use an interface according to at least one of the following communication systems: WLAN according to a standard of the IEEE 802.11 standard family, or an LTE or 5G mobile communication system according to a 3GPP standard.

[0047] For a vehicle to be used in the process, it is advantageous if the vehicle is equipped with a device that corresponds to the proposed device.

[0048] For a computer program that is executed in a computing device to perform the steps for capturing image material according to the inventive method, the corresponding advantages apply as described for the inventive method.

[0049] Exemplary embodiments of the invention are shown in the drawings and are explained in more detail below with reference to the figures.

[0050] They show: Fig. 1 the typical cockpit of a vehicle; Fig. 2 a schematic representation of the various communication options provided in a vehicle for external communication; Fig. 3 a block diagram of the vehicle's on-board electronics; Fig. 4 a first representation of a driving situation to illustrate the problem of verifying the function of the vehicle's image processing system; Fig. 5 a second representation of the driving situation, the second representation showing the driving situation at an earlier time; and Fig. 6 a flowchart for a program for acquiring image data for verifying the function of the image processing system in the first driving situation; Fig. 7 a first representation of a second driving situation to illustrate the problem of verifying the function of the vehicle's image processing system; Fig.8. A second representation of the second driving situation, wherein the second representation shows the second driving situation at an earlier time; and Fig. 9. A flowchart for a program for acquiring image data for verifying the function of the image evaluation system during the second driving situation; Fig. 10. A representation of the variation in image quality for archiving image data depending on the environmental conditions; and Fig. 11. A representation of the variation in recording duration for archiving image data depending on the environmental conditions.

[0051] The present description illustrates the principles of the inventive disclosure. It is therefore understood that those skilled in the art will be able to design various arrangements which, although not explicitly described here, embody principles of the inventive disclosure and which are also intended to be protected in their scope.

[0052] Fig. 1 Figure 10 shows the typical cockpit of a vehicle. A passenger car is depicted. However, any other vehicle could also be considered as vehicle 10. Examples of other vehicles include: buses, commercial vehicles, especially trucks, agricultural machinery, construction machinery, rail vehicles, etc. The invention could generally be used in land vehicles, rail vehicles, watercraft, and aircraft.

[0053] The cockpit features two display units of an infotainment system: a touchscreen 30, located in the center console, and the instrument cluster 110, which is mounted in the dashboard. While driving, the center console is not within the driver's field of vision. Therefore, additional information is not displayed on the touchscreen 30 while driving.

[0054] The touchscreen 30 is used primarily for operating vehicle functions 10. For example, it can be used to control a radio, a navigation system, playback of stored music, and / or air conditioning, other electronic devices, or other comfort functions or applications of the vehicle 10. This is often referred to collectively as an "infotainment system." In motor vehicles, especially passenger cars, an infotainment system refers to the integration of the car radio, navigation system, hands-free system, driver assistance systems, and other functions into a central control unit. The term "infotainment" is a portmanteau word, composed of the words "information" and "entertainment."The infotainment system is primarily operated via the touchscreen 30, which is easily visible and operable by both the driver and passenger of the vehicle 10. Below the touchscreen 30, mechanical controls, such as buttons, rotary knobs, or combinations thereof, like rotary push-button controls, may be arranged in an input unit 50. Typically, parts of the infotainment system can also be operated via the steering wheel. This unit is not shown separately but is considered part of the input unit 50.

[0055] Fig. 1Figure 10 also shows a head-up display 20. The head-up display 20 is located in the vehicle 10, behind the instrument cluster 110 in the dashboard area, from the driver's perspective. It is an image projection unit. Additional information is projected onto the windshield and displayed in the driver's field of vision. This additional information appears as if it were projected onto a projection surface 21 at a distance of 7–15 m in front of the vehicle 10. However, the real world remains visible through this projection surface 21. The displayed additional information essentially creates a virtual environment. The virtual environment is theoretically superimposed on the real world and contains virtual objects that support and inform the driver while driving. However, the projection is only onto a portion of the windshield, so the additional information cannot be arbitrarily positioned within the driver's field of vision.This type of display is also known as "augmented reality".

[0056] Fig. 2 This shows a system architecture for vehicle communication via mobile communications. The vehicles 10 are equipped with an on-board communication module 160 with a corresponding antenna unit, enabling them to participate in the various types of vehicle communication V2V and V2X. Fig. 1 shows that vehicle 10 can communicate with the mobile phone base station 210 of a mobile phone provider.

[0057] Such a Base Station 210 can be an eNodeB base station of an LTE (Long Term Evolution) mobile network operator. The Base Station 210 and the associated equipment are part of a mobile communications network with a multitude of mobile cells, each cell being served by a Base Station 210.

[0058] Base station 210 is positioned near a main road on which vehicles 10 travel. In LTE terminology, a mobile device corresponds to user equipment (UE), which allows a user to access network services by connecting to the UTRAN or the Evolved-UTRAN via the radio interface. Typically, such user equipment corresponds to a smartphone. These mobile devices are used by the passengers in vehicles 10. Additionally, each vehicle 10 is equipped with an on-board communication module 160. This on-board communication module 160 is an LTE communication module that allows the vehicle 10 to receive mobile data (downlink) and transmit such data upwards (uplink). This on-board communication module 160 can also be equipped with a WLAN p module to participate in an ad-hoc V2X communication mode.

[0059] V2V and V2X communication is also supported by the new 5th generation of mobile communication systems. The corresponding radio interface is referred to as the PC5 interface. Regarding the LTE mobile communication system, the Evolved UMTS Terrestrial Radio Access Network (E-UTRAN) consists of several eNodeBs that provide the E-UTRA user layer (PDCP / RLC / MAC / PHY) and the control layer (RRC). The eNodeBs are interconnected via the X2 interface. The eNodeBs are also connected to the EPC (Evolved Packet Core) 200 via the S1 interface.

[0060] This general architecture shows Fig. 2The diagram shows that base station 210 is connected to EPC 200 via the S1 interface, and EPC 200 is connected to Internet 300. A backend server 320, to which vehicles can send and receive messages, is also connected to Internet 300. In this case, backend server 320 can be located in a data center of the vehicle manufacturer. Finally, a road infrastructure station 310 is also shown. This can be represented, for example, by a roadside unit, often referred to as a Road Side Unit (RSU) 310. To simplify the implementation, it is assumed that all components have been assigned an Internet address, typically in the form of an IPv6 address, so that the packets carrying messages between the components can be routed accordingly. The various interfaces mentioned are standardized.Reference is made to the relevant LTE specifications, which have been published.

[0061] Fig. 3 Figure 1 schematically shows a block diagram of the vehicle's electronics 200, which also includes the vehicle's infotainment system 10. The infotainment system is operated by a touch-sensitive display unit 30, a computing unit 40, an input unit 50, and a memory unit 60. The display unit 30 comprises both a display area for showing variable graphical information and a user interface (touch-sensitive layer) arranged above the display area for entering commands by a user. It can be designed as an LCD touchscreen display.

[0062] The display unit 30 is connected to the computer unit 40 via a data line 70. The data line can be designed according to the LVDS standard, corresponding to Low Voltage Differential Signaling. The display unit 30 receives control data from the computer unit 40 via data line 70 to control the display area of ​​the touchscreen 30. Control data for the entered commands from the touchscreen 30 to the computer unit 40 is also transmitted via data line 70. The input unit is designated by the reference number 50. It includes the aforementioned operating elements such as buttons, rotary knobs, sliders, or rotary push-buttons, which the operator uses to make inputs via the menu navigation. Input generally refers to selecting a menu option, changing a parameter, switching a function on and off, etc.

[0063] The storage unit 60 is connected to the computing unit 40 via a data line 80. The storage unit 60 contains a pictogram directory and / or symbol directory with pictograms and / or symbols for the possible display of additional information.

[0064] The remaining components of the infotainment system—camera 150, radio 140, navigation system 130, telephone 120, and instrument cluster 110—are connected to the infotainment system control unit via data bus 100. The high-speed variant of the CAN bus according to ISO standard 11898-2 is suitable for data bus 100. Alternatively, a bus system based on Ethernet technology, such as BroadR-Reach, could be used. Bus systems that transmit data via fiber optic cables are also possible. Examples include the MOST bus (Media Oriented System Transport) and the D2B bus (Domestic Digital Bus). A vehicle measurement unit 170 is also connected to data bus 100. This vehicle measurement unit 170 is used to detect the vehicle's movement, particularly its acceleration. It can be configured as a conventional IMU (Inertial Measurement Unit).An IMU unit typically contains accelerometers and yaw rate sensors such as a laser gyroscope or a magnetometer gyroscope. The vehicle measurement unit 170 can be considered part of the vehicle 10's odometry system. This also includes the wheel speed sensors.

[0065] It is also mentioned here that camera 150 can be configured as a conventional video camera. In this case, it records 25 full frames per second, which corresponds to 50 half frames per second in interlaced recording mode. Alternatively, a special camera can be used that records more frames per second to increase the accuracy of object detection for faster-moving objects, or that records light in a spectrum other than the visible spectrum. Several cameras can be used for environmental monitoring. In addition, the previously mentioned radar or lidar systems 152 and 154 will also be used, either additionally or alternatively, to perform or expand environmental monitoring. For wireless communication both internally and externally, vehicle 10 is equipped with communication module 160, as already mentioned.

[0066] Camera 150 is primarily used for object recognition. Typical objects to be recognized include traffic signs, vehicles ahead, surrounding vehicles, parked vehicles, other road users, intersections, turning points, potholes, etc. When an object with potential significance is detected, information can be displayed via the infotainment system. Typically, for example, symbols for the detected traffic signs are shown. It can also be a warning message if the object poses a hazard. The information is projected directly into the driver's field of vision via the HUD 20. As an example of an object posing a hazard, the image analysis from Camera 150 indicates that a vehicle is approaching an intersection from the right, towards which the vehicle itself is moving. This image analysis takes place in the processing unit 40.This can be achieved using established object recognition algorithms. As a result, a hazard symbol is displayed at the vehicle's location. The display should be positioned so that the hazard symbol does not obscure the vehicle, otherwise the driver cannot accurately identify the hazard.

[0067] The object recognition algorithms are processed by processing unit 40. The number of images that can be analyzed per second depends on the performance of this processing unit. An automated driving function typically goes through different phases: In the perception phase, data from various environmental sensors are processed and combined on a high-performance platform. The vehicle must also be located on a highly accurate digital map based on its GNSS position. The data is used to generate a three-dimensional environmental model and provides information on dynamic and static objects as well as open areas around the vehicle (object list). Under certain conditions, the accuracy of the GNSS position is insufficient. Therefore, odometry data from within the vehicle is also used to improve the accuracy of the position determination.

[0068] Reference number 181 designates an engine control unit. Reference number 182 corresponds to an ESP control unit, and reference number 183 designates a transmission control unit. Other control units, such as an additional vehicle dynamics control unit (for vehicles with electrically adjustable dampers), an airbag control unit, etc., may be present in the vehicle. The networking of such control units, all of which belong to the powertrain category, is typically achieved using the CAN bus system (Controller Area Network) 104, which is standardized as an ISO standard, most commonly ISO 11898-1. Various sensors 171 to 173 in the vehicle, which are no longer intended to be connected solely to individual control units, are also designed to be connected to the bus system 104, with their sensor data being transmitted via the bus to the individual control units.Examples of sensors in motor vehicles include wheel speed sensors, steering angle sensors, acceleration sensors, yaw rate sensors, tire pressure sensors, distance sensors, knock sensors, air quality sensors, etc. Wheel speed sensors and steering angle sensors, in particular, are part of the vehicle's odometry system. The acceleration and yaw rate sensors can also be directly connected to the vehicle's measuring unit 170.

[0069] The Figures 4 and 5 This illustrates the basic functionality of an image processing system in its initial form, as used in vehicles. The image shown is taken by the front camera 150 of vehicle 10. The example chosen is traffic sign recognition. Fig. 4 The camera image shows the view from close range to traffic sign 15. Fig. 5The camera image shows traffic sign 15 from a greater distance. This is traffic sign 15, which indicates an absolute no-overtaking zone. The traffic sign recognition system is designed so that, under standard conditions, vehicle 10 should be able to detect traffic sign 15 at a standard detection distance of 80 m. Fig. 4 The camera image shows the situation in which the approach occurred to a distance of up to 20 m. Fig. 5The situation is depicted where vehicle 10 is still 40 m from traffic sign 15. Image analysis reveals that traffic sign recognition only occurred at a distance of 40 m. The image analysis system's verification function determines that traffic sign recognition occurred too late under the given conditions. The verification function then performs a test to pre-evaluate the captured images. As shown, this reveals that a truck 13 is visible in the camera image. Fig. 5 As can be seen, traffic sign 15 is directly to the right of truck 13. The verification function concludes from this that traffic sign 15 was obscured by the truck at a greater distance. For this reason, the images captured at a distance of 80 m are archived at a lower quality until the actual traffic sign recognition at a distance of 40 m.

[0070] The Figs. 4 and 5 This also serves to explain another misidentification scenario. Initially, traffic sign 15, indicating a restricted overtaking ban applicable to trucks, was recognized at a distance of 40 meters. Only at a distance of 20 meters from traffic sign 15 was it then recognized as traffic sign 15 indicating an absolute overtaking ban. In this situation, the verification function reacts as follows: The area 17 of the captured camera image, marked by a background, is archived in high quality among the images to be archived. The remaining parts of the images are recorded in lower quality. This applies to all images captured between a distance of 40 meters and a distance of 20 meters.

[0071] Fig. 6Figure 1 shows a flowchart for a program that implements the verification function. This version of the program is intended for the second variant described, in which the traffic sign was initially misidentified, and only upon closer approach was it correctly identified. The reference number 330 denotes the program start. In program step 331, the traffic sign recognition algorithm is executed. This is an object recognition algorithm that performs pattern recognition based on patterns stored in a table. All valid traffic signs are known, and their patterns can be stored in a table. Typically, the pattern recognition is improved by a convolution operation, in which the captured images are folded with the known patterns. Such algorithms are known to those skilled in the art and are readily available.If a traffic sign is detected in this way, the detection distance to the traffic sign is also determined and stored.

[0072] In program step 332, the system checks whether the traffic sign recognition complied with the standard. In the case under consideration, the traffic sign recognition was faulty for two reasons. First, the recognition did not occur at the standard recognition distance of 80 m from traffic sign 15. Furthermore, a restricted no-overtaking zone was initially detected, which was then changed to an absolute no-overtaking zone upon closer approach. If this program step detects compliant traffic sign recognition, no images need to be archived for verification, and the program ends at step 336. However, in this case, faulty traffic sign recognition was detected. The program then continues with program step 333. In program step 333, the number of problematic images is calculated. Because there is a deviation from the standard recognition distance, the calculation can be performed as follows.Initial detection occurred at a distance of 40 m. Correct detection only occurred at a distance of 20 m. Therefore, the images from the standard detection distance of 80 m down to a distance of 20 m are of interest.

[0073] In program step 333, an additional check is performed to determine whether the error detection can be explained. Image analysis reveals that a truck (number 13) is driving ahead and likely obscured traffic sign 15 until it was first detected. Therefore, it is concluded that images in the distance range of 80 m to 40 m are less important for subsequent verification. These images are therefore saved at a lower quality.

[0074] In program step 334, the problematic image sections are calculated. In the case of traffic sign recognition, the section in the Figs. 4 and 5The highlighted image area around traffic sign 15 has been selected as relevant. This image area is therefore selected with high image quality within a distance of 40 m to 20 m.

[0075] In program step 335, the problematic images and image sections are transferred from memory 60 to communication module 160, and communication module 160 then sends this image data via mobile network to the backend server 320, where it is archived. The archived images are later evaluated by experts in the data center or automatically using artificial intelligence. The purpose of this review is to identify potential problems with the image evaluation system. The results of the reviews can be used to improve the evaluation algorithms. Ideally, the improved evaluation algorithm can be transferred back to vehicle 10 via OTA (Over the Air) download and installed there. However, it is also possible that the review will reveal a systematic error resulting from a misalignment of camera 150.In that case, a message could be sent to vehicle 10 to inform the driver that he should visit the repair shop. The program ends at program step 336.

[0076] In Fig. 7The diagram depicts a situation in which a vehicle is approaching a priority road. Just before turning onto the priority road, an oncoming vehicle 12 with the right-of-way is detected through evaluation of camera images and / or data from radar and lidar sensors 154, 152. The autonomous vehicle 10 must therefore brake sharply. This particularly hard braking creates an unpleasant situation for the vehicle's occupants. Such braking maneuvers should be avoided, especially for autonomous vehicles, to ensure the occupants feel safe. At least during the test phase, the vehicle's electronics classify this incident as undesirable. The same can also occur during normal operation, i.e., after the test phase. There are several possibilities for this. One possibility is that this classification occurs when a predefined acceleration threshold is exceeded.Another possibility is to perform the classification when a negative statement from the occupants or driver is observed by the driver / occupant state detection system. A further possibility is to perform the classification when a defined minimum distance to object detection (in this case, the vehicle approaching from the right) has been breached.

[0077] To reduce the amount of image data to be stored, only the image sections from the ring buffer in which the object could have been located should be saved / transmitted.

[0078] The Fig. 8 shows a camera image for the same driving situation as in Fig. 7 at an earlier time. Vehicle 12, approaching from the right, is still further away from the intersection. Fig. 8The image even shows a camera image from a time when vehicle 12 has not yet been recognized as an approaching vehicle by the object recognition system. This can have various causes. The vehicle is still too far away, and the object recognition algorithm could not yet detect the vehicle with the available image information. Vehicle 12 is obscured by other objects (e.g., trees or bushes).

[0079] The reason why the approaching vehicle 12 was detected so late will be determined through subsequent analysis. For this purpose, the recorded camera images and, if applicable, the recorded images from other imaging sensors will be identified and archived for this analysis. The amount of data to be archived should be kept to a minimum.

[0080] How this happens will be explained below with the help of the Fig. 9 explained. Fig. 9Figure 340 shows a flowchart for a program that implements the verification function in the second embodiment of the invention. The program start is indicated by the reference number 340. The program is always started when a hazardous situation is detected in which a dynamic object was detected too late. In program step 342, the trajectory of the dynamically detected object is estimated. This is done as follows: At the time of object detection, the current speed and acceleration of the detected object are determined from several consecutive video frames using various sensor data from the observer vehicle 10. Using this information, and if necessary, the route geometry known from the map of the navigation system 130, a trajectory of the detected object is estimated.In conjunction with the vehicle's own motion estimation, a bounding box can be determined from the recorded image data for points in the past, within which the (at that time not yet detected) object must have been located. This bounding box is also referred to as the bounding box. It can be assumed that the bounding box will be larger at earlier points in time than at points in the more recent past. The size of the bounding box is influenced by the object parameters (e.g., size) at the time of detection, but also by uncertainties in these parameters, possible alternatives in the object trajectory, known sensor errors / inaccuracies, and empirically determined factors.

[0081] In program step 344, the estimated trajectory is used for a backward calculation to determine a number of problematic images. These images are used to verify why the approaching vehicle 12 could not be detected. The distance between the observer vehicle 10 and the approaching vehicle 12 is also taken into account. If the distance exceeds a predefined limit, no further past images need to be archived. The distances up to which the desired objects can still be detected are known for each of the various environmental monitoring sensors. The program then continues with program step 346, which calculates the number of problematic images.

[0082] In program step 348, the respective bounding boxes for the number of problematic images are calculated. This is also done taking the estimated trajectory into account.

[0083] In another variant, steps 344 and / or 346 also determine whether a failure to recognize an object could have been caused by the relevant object being obscured by other, possibly correctly recognized objects (other vehicles, houses / walls / noise barriers known from navigation data, trees, bushes, forests). In this case, the images may not be transmitted. The time period of problematic images may be limited because the presumed object trajectory suggests that the relevant object was not within the camera's field of view at an earlier time.

[0084] In program step 348, the problematic image sections 14 are transferred from the ring buffer of the storage device 60 to the communication module 160, and the communication module 160 sends this image data via mobile network to the backend server 320. There, they are archived. The archived images are later evaluated in the data center by experts or automatically using artificial intelligence. The purpose of the review is to identify potential problems with the image evaluation system. The results of the reviews can be used to improve the evaluation algorithms. Ideally, the improved evaluation algorithm can be transferred back to the vehicle 10 via OTA download (over the air) and installed there. However, it is also possible that the review will reveal a systematic error resulting from a misalignment of the camera 150.In that case, a message could be sent to vehicle 10 to inform the driver that he should visit the repair shop. The program ends at program step 350.

[0085] The number and quality of images to be archived can be influenced by various factors. In addition to the cases already presented, the following influencing factors are particularly noteworthy: Weather conditions

[0086] Visibility is severely limited in adverse weather conditions. This can be so severe that object detection becomes impossible. However, the other environmental sensors, such as radar and lidar, can deliver better results in such situations. In any case, it can be implemented to limit the number of images or image sections to be archived in adverse weather conditions, because the relevant objects cannot be detected from the standard detection distance. Positional accuracy

[0087] The accuracy of position determination based on GNSS and odometry signals can also be weather-dependent. It can also depend on other influencing factors. For example, the environment in which the vehicle is traveling can be a factor. In cities, dense development can restrict satellite signal reception. This can also be the case during cross-country journeys. Satellite signals can be weakened in forests. In mountainous regions, tectonics can also lead to poor reception. In such cases, it is therefore recommended that more images or image sections be recorded. This increases the likelihood that the relevant route segments are captured despite the positional inaccuracies. Road surface conditions

[0088] Road surface conditions can also have a similar influence. Strong vibrations occur on cobblestones, which can cause the recorded images to be blurry. On slippery surfaces caused by ice, snow, or rain, the traction control system calculates an estimate of the coefficient of friction. If the slip is sufficiently high, the odometry data becomes less reliable, and this influence should be taken into account, just as with the influence of positional inaccuracies. time

[0089] It is clear that image quality will vary greatly depending on the time of day. A distinction should at least be made between day and night. At night, data from radar or lidar sensors should be recorded rather than camera image data. Traffic conditions

[0090] Here, a distinction can be made between whether the vehicle is traveling in city traffic, on the motorway, on a country road, etc. In city traffic, particularly accurate detection of traffic signs is crucial. In this case, the number of images to be recorded can be increased. In traffic jams on the motorway, however, the number of images to be recorded can be reduced.

[0091] In vehicle 10, the images or image excerpts 14 to be archived are temporarily stored. Memory 60 can be used for this purpose. In practical implementation, a ring buffer is set up in memory 60. This is managed so that newly captured images are written sequentially to the free memory area of ​​the ring buffer. When the end of the free memory area is reached, the already written portion of the ring buffer is overwritten. By managing the allocated memory area as a ring buffer, the oldest part of the memory is always overwritten. The images or image excerpts 14 can be stored uncompressed or compressed. However, it is recommended to use a lossless compression method to prevent any loss of relevant image content. The FFmpeg codec is mentioned as an example of a lossless compression method.The image data is saved in a suitable file format. Various data container formats are suitable. Examples include the ADTF format, developed for the automotive sector, and the TIFF format. Other container formats for storing image and audio data include MPEG, Ogg, Audio Video Interleave, DIVX, QuickTime, Matroska, etc. If the images are transferred to the 320 backend server while driving, a ring buffer designed for a 20-second recording duration may be sufficient. Alternatively, separate storage can be provided in the vehicle for archiving the images. A removable USB hard drive, for example, is a good option for connecting to a computer used to analyze the archived images.

[0092] Fig. 10The diagram also shows how reducing the image quality can decrease the amount of data to be archived. From top to bottom, the amount of image data for the different image qualities is shown: Full HD with a resolution of 1920 x 1080 pixels, SD with a resolution of 640 x 480 pixels, and VCD with a resolution of 352 x 288 pixels.

[0093] The Fig. 11 The graph then shows the effect of varying the number of images or image segments (17) depending on positional accuracy, weather conditions, etc. If the recording duration is increased to a distance of 160 m due to positional inaccuracy, twice as much data is required for archiving. Reducing the distance to 40 m results in only half the amount.

[0094] All examples mentioned herein, as well as conditional formulations, are to be understood without limitation to such specifically cited examples. For instance, it is recognized by those skilled in the art that the block diagram shown here represents a conceptual view of an exemplary circuit arrangement. Similarly, it is understood that a flowchart, state transition diagram, pseudocode, and the like are different ways of representing processes that are essentially stored in computer-readable media and can thus be executed by a computer or processor. The object mentioned in the patent claims can expressly also be a person.

[0095] It should be understood that the proposed method and associated apparatus can be implemented in various forms of hardware, software, firmware, specialized processors, or a combination thereof. Specialized processors can include application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs), and / or field-programmable gate arrays (FPGAs). Preferably, the proposed method and apparatus are implemented as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. Typically, this is a machine based on a computer platform that includes hardware such as one or more central processing units (CPUs), random access memory (RAM), and one or more input / output (I / O) interfaces.A computer platform typically also has an operating system installed. The various processes and functions described here can be part of the application program or a component that is executed by the operating system.

[0096] The disclosure is not limited to the embodiments described here. There is scope for various adaptations and modifications that a person skilled in the art would consider based on their expertise and in relation to the disclosure itself.

[0097] The invention is explained in more detail in the exemplary embodiments using the example of its use in vehicles. The possible use in airplanes and helicopters, for example during landing maneuvers or search operations, etc., is also mentioned.

[0098] The invention can also be used in remotely controlled devices such as drones and robots, where image analysis is crucial. Other potential applications include smartphones, tablet computers, personal assistants, and smart glasses. Reference symbol list

[0099] 10 Observer vehicle 12 Approaching vehicle 13 Vehicle ahead 14 First relevant image section 15 Traffic sign 17 Second relevant image section 20 Head-up display 30 Touch-sensitive display unit 40 Processing unit 50 Input unit 60 Storage unit 70 Data line to display unit 80 Data line to storage unit 90 Data line to input unit 100 Data bus 110 Instrument cluster 120 Telephone 130 Navigation system 140 Radio 142 Gateway 144 On-board diagnostic interface 150 Camera 160 Communication module 170 Vehicle measuring unit 171 Sensor 1 172 Sensor 2 173 Sensor 3 180 Control unit for automatic driving function 181 Engine control unit 182 ESP control unit 183 Transmission control unit 200 Evolved Packet Core 210 Base station 300 Internet 310 Road Side Unit 320 Backend server 330-336 Various program steps of a first computer program 340-350 Various program steps of a second computer program Uu Air interface for communication UE and eNodeB PC5 Air interface forVehicle direct communication

Claims

1. Method for capturing image material for checking image analysis systems, comprising the following steps: - recording, in a memory (60), images captured by an image generating device of a vehicle; and - performing object recognition using an image analysis system; characterized by: - determining a distance between the vehicle and an object (12, 15) at which the object (12, 15) was first correctly detected; - establishing that the determined distance deviates from a standard recognition distance for object recognition beyond a tolerance limit; - ascertaining the images or image details (14, 17) that were captured between the determined distance and the standard recognition distance; and - archiving the ascertained images or image details (14, 17).

2. Method according to claim 1, wherein the step of ascertaining images or image details (14) for archiving includes calculating a trajectory of a moving object (12) and determining the images or image details (14) to be archived on the basis of the calculated trajectory.

3. Method according to claim 2, wherein a number of images or image details (14) in which the moving object (12) was presumably visible are determined using the trajectory.

4. Method according to any of the preceding claims, wherein the image analysis system is an image analysis system for a vehicle (10) and wherein the position data of the vehicle (10) and the position data of the recognized object (12, 15) are analyzed in order to determine the distance.

5. Method according to any of the preceding claims, wherein the images or image details (14, 17) to be archived are sent to an external archiving location (320) or are archived in a local memory unit (60).

6. Method according to any of the preceding claims, wherein the ascertained images or image details (14, 17) are archived with high quality and further images or other image details are archived with lower quality.

7. Apparatus for capturing image material for checking image analysis systems, comprising an image generating device, a computing device (40) and a memory device (60), the memory device (60) being designed to record images supplied by the image generating device and the computing device (40) being designed to perform object recognition on the images supplied by the image generating device, characterized in that the computing device (40) is furthermore designed: - to determine a distance at which an object (12, 15) was first correctly detected; - to establish that the determined distance deviates from a standard recognition distance for object recognition beyond a tolerance limit; - to ascertain the images or image details (14, 17) that were captured between the determined distance and the standard recognition distance; and - to issue a command to archive the ascertained images or image details (14, 17).

8. Apparatus according to claim 7, wherein the computing device (40) is designed to calculate a trajectory of a moving object (12) and, depending on the calculated trajectory, to determine the images or image details (14) to be archived.

9. Apparatus according to claim 7 or 8, wherein the apparatus comprises a communication module (160) and the communication module (160) is designed to send the ascertained images or image details (14, 17) to an external archiving location (320) after receiving the command to archive the ascertained images or image details (14, 17).

10. Apparatus according to any of claims 7 to 9, wherein the memory device (60) is designed as a circular buffer in which, in the event of a buffer overflow, the oldest previously stored images or image parts (14, 17) are overwritten by the new images or image parts (14, 17).

11. Apparatus according to any of claims 7 to 10, wherein the image generating device is a video camera (150) or a LIDAR or a RADAR sensor and wherein the communication module (160) is designed for wireless communication according to at least one of the communication systems WLAN in accordance with a standard of the IEEE 802.11 standard family or an LTE or 5G mobile communication system in accordance with a 3GPP standard.

12. Vehicle (10), characterized in that the vehicle (10) is equipped with an apparatus according to any of claims 7 to 11.

13. Computer program, characterized in that the computer program is designed to perform the steps of the method according to any of claims 1 to 6 when processed in a computing device (40).