A method for avoiding visual interference to operators targeting objects, an apparatus for performing the method, a vehicle, and a computer program.

By using image analysis and brightness adjustment technology, the problem of visual interference caused by augmented reality displays in vehicles has been solved, enabling rapid identification and elimination of visual interference, thereby improving driver safety and concentration.

CN111557019BActive Publication Date: 2026-11-10VOLKSWAGEN AG
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN201980006242.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-11-01
Filing Date
2019-10-09
Publication Date
2026-11-10
Estimated Expiration
2039-10-09

AI Technical Summary

Technical Problem

Existing augmented reality display technologies cause visual disturbances in vehicles, including obscuring real objects and diverting attention, especially due to flickering, jittering, or pulsation caused by the display of virtual information.

Method used

By using image analysis methods, the image is subdivided frame by frame and the fill degree is analyzed to identify visual interference. When interference is detected, the display brightness of virtual information is turned off or reduced. The image analysis unit and evaluation unit are used to achieve rapid identification and elimination of visual interference.

Benefits of technology

It can effectively identify and quickly eliminate visual interference, reduce potential dangers, prevent real objects from being obscured and attention from being diverted, and improve driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111557019B_ABST
    Figure CN111557019B_ABST
Patent Text Reader

Abstract

This invention relates to a method for avoiding visual interference for operators of an object. The visual interference may be caused by the display of virtual supplementary information via a display unit (20), particularly a head-up display (HUD) or data glasses of a vehicle (10). Here, the display of supplementary information assists the operator when operating the object (10). The method is characterized by performing image analysis to check whether visual interference is caused by the display of virtual supplementary information, and wherein, when visual interference is identified, measures to eliminate the visual interference are implemented.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to the technical field of driver information systems, also known as infotainment systems. These systems are primarily used in vehicles. However, the present invention can also be used for pedestrians, cyclists, etc., wearing digital glasses. The present invention also relates to a correspondingly designed apparatus for performing the method, as well as a vehicle and a computer program.

[0002] The automotive industry's vision for the future is to display virtual elements on the windshields of their vehicles to provide drivers with some benefits. This uses what's known as "Augmented Reality" (AR) technology. Less popular is the corresponding German term "erweiterten" (augmented reality). Here, the real environment is enriched through virtual elements. This has several advantages: because much relevant information is displayed on the windshield, there's no need to look down at other displays besides the windshield. Therefore, the driver's gaze doesn't need to deviate from the lane. Augmented reality displays are characterized by the ability to precisely position virtual elements within the real environment. Virtual elements are also displayed where the driver's gaze is directed in the real environment. From the user's perspective, these displays can "overlay" the real environment and add additional information, such as navigation routes. This results in lower cognitive load for the driver, as they don't need to comprehend abstract images but can achieve intuitive understanding within the scope of their normal perceptual habits.

[0003] Currently, head-up displays (HUDs) are used in vehicles. Their advantage is that the HUD image appears more realistically to the environment. These displays are essentially projection units that project images onto the windshield. However, from the driver's perspective, the image is positioned a few meters to 15 meters in front of the vehicle, depending on the module's construction. This reduces the strain on the driver's eyes from the visual accommodation process.

[0004] Here, the "image" is composed of elements that are less about virtual display and more accurately, a "keyhole" to a virtual world. Theoretically, the virtual environment is placed above the real world and includes virtual objects that support and inform the driver while driving. The limited display area of ​​the HUD results in only a portion of it being visible. People can see a portion of the virtual world through the HUD's display area. Because this virtual environment complements the real environment, it is also referred to as "mixed reality" in this context.

[0005] Currently, efforts are also underway to develop technologies that should enable autonomous driving in the future. One approach is to not completely relieve the driver of their duties, but rather to allow the driver to take over control of the vehicle at any time. Furthermore, the driver performs a monitoring function. Through updated technologies in the field of driver information systems, such as head-up displays (HUDs), drivers can be better informed about events in their vehicle's surroundings.

[0006] To achieve this, newer technologies are needed (vehicle-to-vehicle communication, databases, vehicle sensing technologies, etc.) to make rich information about objects (especially vehicles) in the immediate environment of the vehicle available. In particular, the following components enable environmental observation in the field of vehicle sensing technology: radar equipment (radio detection and ranging), lidar equipment (light detection and ranging), primarily used for distance detection / distance warning, and cameras with corresponding image processing systems for object recognition. This environmental data can serve as the basis for system-side driving advice, warnings, etc. Thus, for example, it is conceivable to display / warn about which direction other surrounding vehicles intend to turn (perhaps back into their own tracks).

[0007] During this period, vehicle-to-vehicle communication can also be achieved using mobile communications systems with technologies such as LTE (Long Term Evolution). The 3GPP organization has already adopted a technical specification called LTE V2X. Alternatively, systems based on WLAN technology can be used for direct communication between vehicles, especially systems according to WLAN p. Also mentioned is the latest generation of mobile wireless communication, 5G, which can also enable V2V or V2X communication.

[0008] Given the current trend towards higher levels of autonomous driving, while many vehicles remain driver-controlled as before, the starting point should be that the corresponding additional information can be used in manually controlled vehicles in the medium term, rather than only in highly automated systems in the long term. The solutions described below, applicable to both manually and automatically controlled vehicles, will be detailed in detail below.

[0009] A method for adjusting a head-up display (HUD) in a vehicle is known from patent document DE 10 2012 010 120 A1. This method can also be used to adjust the position of the eye-box. Here, the eye-box is the range within the vehicle within which the driver's eyes must be positioned in order to perceive a virtual image on the windshield. For this purpose, a test image is projected onto the windshield, and at least one edge region of the eye-box is captured by a camera.

[0010] A device for automatically calibrating a projection device for a head-up display used in vehicles is known from patent document DE 10 2005 037 797 A1, wherein calibration is particularly capable of being automated. Here, data for geometric correction and multi-projection calibration calculations are stored in a so-called warp-feldern field.

[0011] A major advantage of known augmented reality (AR) displays is that the displayed content is presented directly within or as part of the environment. Similar examples are mostly found in the field of navigation. Classical navigation displays (in traditional head-up displays) typically show illustrative views (e.g., a right-angled arrow pointing to the right as a sign that a right turn should be made at the next opportunity), while AR displays offer significantly more efficient possibilities. Because the displayed content can be presented as "part of the environment," it is much faster and more intuitive for the user to understand. However, known solutions also have various problems, for which there are currently no solutions. This is recognized within the scope of this invention. A number of problems arise from visual distractions caused by the virtual additional information appearing in the driver's field of vision. This visual distraction may involve obscuring objects in the real environment. Another distraction, in the case of vehicles, is the diversion of the operator's or driver's attention. This can occur when a flickering, jumping, or pulsating or pulsed view is produced due to the successive appearance of the display.

[0012] The displayed navigation routes and signs (speed limit signs, hazard signs, etc.) cover a large area and may obscure real objects. Therefore, further improvements are needed in displaying virtual supplementary information to the operator's field of vision. The technical problem this invention aims to solve is to find such a solution.

[0013] The aforementioned technical problem is solved by the method for avoiding visual interference to the operator of the target object according to the present invention, the apparatus for performing the method according to the present invention, the vehicle according to the present invention, and the computer program according to the present invention, namely:

[0014] A method for avoiding visual interference for an operator, wherein virtual supplementary information is displayed in the operator's field of vision via a display unit, characterized by performing image analysis to check whether visual interference is caused by the display of virtual supplementary information, and wherein, when visual interference is identified, measures for eliminating visual interference are implemented, wherein the image analysis for revealing the problem of obscuring areas in the operator's field of vision involves analyzing multiple sequential images, wherein the images are progressively divided into segments and said segments are analyzed individually, wherein the image analysis involves determining the fill level of a corresponding image or corresponding segment caused by the display of virtual supplementary information.

[0015] An apparatus for performing the method according to the present invention, the apparatus having a display unit capable of displaying virtual supplementary information into the field of vision of an operator of an object, the apparatus further having an interface for receiving an image, the image being used to display the virtual supplementary information into the field of vision of the operator of the object, characterized in that the apparatus further has an image analysis unit and an evaluation unit, the image analysis unit and the evaluation unit checking whether the display of virtual supplementary information causes visual interference, and the image analysis unit and the evaluation unit being designed to eliminate visual interference when visual interference is detected.

[0016] A means of transportation, characterized in that the means of transportation has a device according to the present invention.

[0017] A computer program, characterized in that the computer program is designed to execute the steps of the method for avoiding visual interference of an operator targeting an object, as described in the present invention, when running in a computing unit including an image analysis unit and an evaluation unit.

[0018] Advantageous extensions and improvements of the present invention lie in at least one of the following technical features:

[0019] Perform image analysis to check whether the displayed virtual additional information is causing an area to be obscured in the operator's field of vision.

[0020] Image analysis is performed on multiple sequentially displayed images with virtual additional information to check whether the display causes interfering flickering, jumping, or pulsation in the operator's field of vision.

[0021] The fill factor is calculated by measuring the ratio of the number of pixels filled by displaying virtual additional information to the total number of pixels that can be filled.

[0022] To check for occlusion of areas in the field of view, three consecutive images were analyzed. The fill degree of the entire image was determined for the first image, the fill degree of four image segments of the same size was determined for the second image, and the fill degree of nine image segments of the same size was determined for the third image.

[0023] The determined image fill and segment fill are graded and checked, and when at least one of the checked criteria is confirmed to be met during the check, a problem is identified in which an area is obscured from the operator's field of vision.

[0024] To check whether the display causes interfering flickering, jumping, or pulsation in the operator's field of vision, the determined fill level is analyzed through multiple sequential inspection processes. When a periodic change in the determined fill level is confirmed, the existence of an interfering flickering, jumping, or pulsation problem in the operator's field of vision is identified.

[0025] To eliminate visual interference, the display is turned off or its brightness is reduced.

[0026] The virtual supplementary information is displayed in a manner corresponding to augmented reality, which simulates contact with one or more identified objects and is brought into the operator's field of vision.

[0027] The image analysis unit and evaluation unit are also designed such that when visual interference is detected, the image analysis unit and evaluation unit will turn off the display or reduce the brightness of the display.

[0028] The proposed solution according to the present invention is a method for avoiding visual interference for operators, wherein virtual supplementary information is displayed in the operator's field of vision through a display unit. Image analysis is performed to check whether visual interference is caused by the display of virtual supplementary information, and when visual interference is identified, measures are taken to suppress or eliminate it. Using this solution, visual interference can be quickly identified and eliminated. The method is executed continuously, thus allowing for the inspection of all displays. This mitigates potential hazards associated with visual interference.

[0029] In a variant of the method, image analysis is performed to check whether the display is causing obstruction of an area in the operator's field of vision. Especially when moving objects are obscured, the obstruction can conceal significant potential hazards and lead to collisions because the operator cannot identify the object.

[0030] In another variation of the method, image analysis is performed on multiple sequentially displayed images with virtual additional information. This image analysis is used to check whether the displays cause interfering flickering, jittering, or pulsation in the operator's field of vision. The potential danger of such displays lies in the diversion of the operator's attention. Therefore, the advantage of this measure is that it also eliminates such visual interference.

[0031] In the specific design proposed in this invention, the image analysis used to reveal the problem of occlusion in the operator's field of vision involves analyzing multiple sequential images, wherein each image is progressively divided into increasingly finer segments and each segment is analyzed individually. This allows for the continuous definition of the local regions in the image where occlusion occurs.

[0032] In one variant design, the fill level of a given image or segment is determined during the image analysis step due to the display of virtual additional information. If, for example, the fill level in such a segment is 100%, then clearly objects that should be visible in that segment are obscured.

[0033] This fill factor can be advantageously achieved by calculating the ratio of the number of pixels, or pixels, that are filled due to the display of virtual additional information to the total number of pixels that can be filled. This calculation can be implemented without significant overhead using specialized chips, such as FPGA chips, which can also be used in products at low cost.

[0034] In one variant design, the following measures are taken to identify occlusion hazards. To check for occlusion of a region in the field of view, three consecutive images are analyzed. Specifically, the fill degree of the entire image is determined for the first image, the fill degree of four image segments of the same size is determined for the second image, and the fill degree of nine image segments of the same size is determined for the third image. Here, the total number of image segments covers the entire image. In other variant designs, additional images may be considered during the analysis step. This also depends on the performance of the dedicated chip that should perform the image analysis.

[0035] In another variant design, the following measures are taken to identify occlusion hazards. To check for occlusion of areas within the field of view, sequentially occurring images are analyzed, wherein the images are divided into different segments. First, the entire image is considered as one segment. Second, the image is divided into four image segments of equal size. Third, the image is divided into nine image segments of equal size. The fill factor of the corresponding image segments is then determined. The speed of analyzing the image segments depends on the performance of the dedicated chip that should perform the image analysis.

[0036] Advantageously, the determined image fill level and segment fill level are checked in stages, and a problem of occluding an area in the operator's field of vision is identified when at least one of the checked criteria is confirmed to be met during the inspection. If a larger segment is already largely filled, it also applies to the corresponding segment with a more finely divided section.

[0037] Furthermore, it is advantageous to analyze the fill level determined by multiple sequential inspection processes to check whether the display causes interfering flickering, jumping, or pulsation in the operator's field of vision. When a periodic change in the determined fill level is confirmed, the problem of interfering flickering, jumping, or pulsation in the operator's field of vision is identified. This algorithm is preferably implemented in software using a microcomputer.

[0038] To eliminate visual interference, it is advantageous to stop or turn off the display or reduce its brightness. This measure can be easily implemented using devices present in the head-up display. However, modifying the calculations used to display the image may require significantly more research and development.

[0039] Advantageously, the apparatus for performing the method includes a display unit that displays virtual supplementary information into the field of vision of an operator. The apparatus also has an interface for receiving images, which display the virtual supplementary information into the operator's field of vision. The apparatus includes an image analysis unit and an evaluation unit that check whether the display of virtual supplementary information causes visual interference. Furthermore, the image analysis unit and evaluation unit are designed to eliminate visual interference when it is identified. The apparatus can be implemented, in particular, as part of a head-up display (HUD) for vehicles. However, the apparatus can also be used in digital glasses or displays showing camera images, in which the supplementary information is displayed.

[0040] An advantageous variation of the device further includes an image analysis unit and an evaluation unit designed to, upon detecting visual interference, disable the display or reduce its brightness. This measure can be implemented particularly easily in head-up displays.

[0041] The device according to the invention can be used advantageously in vehicles. In vehicles, the invention is preferably implemented such that the device is fixedly mounted in the vehicle, for example, in the form of a head-up display.

[0042] As described above, the present invention can also be used advantageously when the display unit is digital glasses. The method according to the invention can thus be used for pedestrians, cyclists, motorcyclists, etc.

[0043] The corresponding advantages described with respect to the method according to the invention also apply to computer programs that run in the computing unit of the device to execute the method according to the invention.

[0044] Embodiments of the present invention are shown in the accompanying drawings and are further described below with reference to the drawings.

[0045] In the attached diagram:

[0046] Figure 1 This demonstrates the principle of displaying information in the driver's field of vision using a head-up display while the vehicle is in motion;

[0047] Figure 2 Showing a typical cockpit of a vehicle;

[0048] Figure 3 A block diagram showing the infotainment system of a vehicle;

[0049] Figure 4 This illustrates a typical example of displaying virtual supplemental information in the driver's field of vision;

[0050] Figure 5 A simplified block diagram of the head-up display unit is shown;

[0051] Figure 6 This shows a view of the analysis window when examining visual interference caused by AR display.

[0052] Figure 7 A flowchart is shown for the program used to perform image analysis;

[0053] Figure 8 A flowchart is shown illustrating the procedure for evaluating the results of image analysis in order to identify the first type of visual disturbance; and

[0054] Figure 9 A flowchart is shown for the procedure used to evaluate the results of image analysis in order to identify a second type of visual interference.

[0055] The description herein illustrates the principles of the disclosure of this invention. Therefore, it is self-evident that those skilled in the art can design different layouts that, although not explicitly described herein, also implement the principles of the disclosure of this invention and should be protected within their scope.

[0056] Figure 1This illustrates the basic working principle of a head-up display (HUD). The HUD, or display unit 20, is installed below / behind the dashboard area in the vehicle 10, or vehicle. Additional information is displayed in the driver's field of vision by projecting it onto the windshield. This additional information appears as if it were projected onto a projection surface 21 at a distance of 7-15 meters in front of the vehicle 10. However, the real world remains visible through this projection surface 21. Using the displayed additional information, a virtual environment is created to some extent. The virtual environment is theoretically set above the real world and contains virtual objects that assist and provide information to the driver while driving. However, since the projection only occurs on a portion of the windshield, the additional information cannot be arbitrarily positioned within the driver's field of vision.

[0057] Figure 2 The cockpit of vehicle 10 is shown. The vehicle shown is a passenger car (PKW). However, any other vehicle can be considered as vehicle 10. Other vehicles include, for example, buses, commercial vehicles, especially heavy-duty trucks (LKW), agricultural machinery, construction machinery, rail vehicles, etc. This invention can be widely applied to agricultural vehicles, rail vehicles, water vehicles, and air vehicles.

[0058] The infotainment system features three display units in the cockpit: a head-up display (HUD), a touch-sensitive screen 30 mounted on the center console, and a smaller display that is part of the instrument cluster 110. The center console is not in the driver's field of vision while driving. Therefore, AR content is displayed on the HUD during driving.

[0059] Here, the touch-sensitive screen 30 is specifically used for operating the functions of the vehicle 10. For example, it can be used to control the radio, navigation system, playback of stored music tracks and / or air conditioning, other electronic devices, or other comfort functions or applications of the vehicle 10. This is generally referred to as the "infotainment system." In vehicles, especially cars, the infotainment system refers to the sum of functions in the car radio, navigation system, hands-free calling device, driver assistance system, and central control unit. The term infotainment is a combination of the words Information and Entertainment. The infotainment system is primarily operated using the touch-sensitive screen 30 ("touchscreen"), which is particularly visible and operable by the driver of the vehicle 10, but also by the passenger. Furthermore, mechanical operating elements, such as buttons, adjustment knobs, or combinations thereof, such as press-to-adjust knobs, can be arranged in the input unit 50 below the touch-sensitive screen 30. Steering wheel operation can also typically be achieved using components of the infotainment system. This unit is not shown separately, but is considered as part of input unit 50.

[0060] Figure 3 A block diagram of an infotainment system 200 is schematically shown, along with exemplary examples of some subsystems or applications of the infotainment system. The operating device includes a touch-sensitive display unit, namely a touch-sensitive screen 30, a computing device 40, an input unit 50, and a memory unit 60. The display unit includes not only a display surface for displaying variable graphic information, but also an operating surface (touch-sensitive layer) arranged above the display surface for user input of commands.

[0061] The display unit is connected to the computing device 40 via data cable 70. The data cable can be designed according to the LVDS standard, i.e., Low Voltage Differential Signaling. The display unit receives control data from the computing device 40 via data cable 70 for controlling the display surface of the touchscreen. Control data for input commands is also transmitted from the touchscreen to the computing device 40 via data cable 70. Reference numeral 50 indicates an input unit. Input units include the previously mentioned operating elements, such as buttons, adjustment knobs, sliders, or press-to-adjust knobs, which operators can use to input data via menu guidance. The term input is generally understood as selecting a menu item, changing parameters, turning functions on or off, etc.

[0062] The memory unit 60 is connected to the computing device 40 via a data line 80. The memory unit 60 stores an icon catalog and / or a symbol catalog, which contains icons and / or symbols for possible display of additional information. Points / symbols used as the basis for computational grid display may also be stored here.

[0063] Other components of the infotainment system, including camera 150, radio 140, navigation device 130, telephone 120, and instrument cluster 110, are connected to the devices used to operate the infotainment system via data bus 100. The data bus 100 can be a high-speed variant of the CAN bus according to ISO standard 11898-2. Alternatively, an Ethernet-based bus system, such as BroadR-Reach, can also be considered. A bus system that transmits data via optical waveguides can also be used. Examples include the MOST bus (Media Oriented System Transport) or the D2B bus (Domestic Digital Bus). It should also be noted that camera 150 can be designed as a conventional video camera. In this case, the camera captures 25 full images per second, which is equivalent to 50.5 images per second in interlaced shooting mode. Alternatively, a dedicated camera can be used, capturing more images per second to improve object recognition accuracy when objects are moving rapidly. Multiple cameras can be used for environmental observation. In addition, as a supplement or alternative, the aforementioned radar or lidar systems can also be used to perform or extend environmental observation. The vehicle 10 is equipped with a communication module 160 for inward and outward wireless communication. This module is often referred to as an onboard unit. It can be designed for mobile wireless communication, such as mobile wireless communication according to the LTE standard, i.e., Long Term Evolution. It can also be designed for WLAN communication, i.e., Wireless LAN, for communication with devices of occupants in the vehicle, or for vehicle-to-vehicle communication, etc.

[0064] The following describes the analysis method according to the present invention based on an embodiment, which is used to avoid unacceptable visual interference in the HUD display unit due to information display.

[0065] First of all, Figure 4The diagram illustrates an example of information displayed via a head-up display. Here, a hazard symbol 28 is projected into the driver's field of vision. In the illustrated case, it is a warning triangle, which is also clearly outlined for emphasis. The display is made after an image evaluation of the image provided by camera 150 indicates that there is a deer 13 at the edge of the lane. The image evaluation is performed in computing unit 40. Known algorithms for object recognition can be used for this purpose. The display can also be positioned at the location of the deer 13 in the driver's field of vision in the form of "augmented reality". However, the danger of this is that the display of the hazard symbol 28 obscures the deer 13 and therefore the driver cannot accurately identify what the danger is.

[0066] Figure 5 A block diagram for a heads-up display (HUD) is shown. Figure 3 The head-up display (HUD) is connected to the computing unit 40. A connection for transmitting image data is present. Here, this refers to the data line 70 used to transmit the calculated LVDS format image data. The other bus connection 72 shown corresponds to a bus interface through which control data, configuration data, and status data can be exchanged. For example, an I2C type (Inter-Integrated Circuit Bus) bus interface can be considered. The HUD itself is essentially composed of three components. Of great importance is the projection unit 26, which, as an imaging unit, includes not only a light source but also optical devices for ray shaping. The projection unit 26 does not obtain image data directly from the computing unit 40, but rather from the control unit 22 inside the HUD. The control unit 22 converts the image data into a suitable format for controlling the projection unit 26. The control unit 22 also includes image analysis functions, which will be described more precisely below. Additionally, a microcontroller 24 is provided in the HUD. The microcontroller is used for brightness control of the HUD, as well as for controlling other settings. The microcontroller 24 also performs evaluation of the results of the image analysis function, which will be described further below.

[0067] First, let's further explain the image analysis function. Figure 6 The principle of image analysis is illustrated. Three consecutive images of the video data stream provided by camera 150 are shown. Typically, cameras provide 30 full images per second. This is consistent with the well-known TV standard NTSC. In the first image, the analysis window is adjusted to the full image size. In the second image, the image is divided into four segments of equal size. Image analysis is performed separately for each individual segment. In the third image shown, the image is analyzed separately in nine segments of equal size. Therefore, the image analysis becomes increasingly refined from left to right.

[0068] In the case of a 30Hz camera, the image analysis function is executed 10 times per second. Image analysis of a single segment or the entire image involves determining the fill factor. Here, the number of pixels required for the corresponding information display is counted, and this number is compared with the total number of pixels in the image that can be displayed by the HUD unit when the analysis window extends across the entire image, or in other words, its ratio is determined. When an image segment is analyzed, the ratio of the number of pixels presented in that segment to the total number of pixels in the corresponding segment is determined. The value thus determined is temporarily stored in the memory of the microcontroller 24 and is ready for algorithm evaluation.

[0069] According to Figure 7 The flowchart shown illustrates the data determination for the analysis function. The program runs in control unit 22. Reference numeral 310 indicates the start of the program. In program step 312, the fill degree for the first image to be analyzed is calculated. This is performed as described above. Next, in program step 314, the fill degree for the next image is calculated, which is divided into four segments of the same size. In program step 316, the fill degree for the third image is calculated, which is also divided into four segments of the same size. After the image analysis steps, in program step 318, the calculated data is transmitted to microcontroller 24. The received data is stored there in temporary memory and is ready for further evaluation. The program ends in program step 320.

[0070] according to Figure 8The flowchart illustrates the first part of the data evaluation. This program is run by microcontroller 24. After program start 330, in program step 332, it is determined whether the determined fill factor BG_1x1 of the analysis window not divided into segments is greater than 5%. If it is greater than 5%, there is a risk of obscuring the real object in the field of view. If it is less than 5%, in step 334, it is checked whether one of the determined fill factors BG_2x2 of the image divided into four segments of the same size exceeds a value of 10%. In this case, it is also inferred that object interference occlusion will occur. If it also does not exceed 10%, in program step 336, it is checked whether one of the determined fill factors BG_3x3 of the image divided into nine segments of the same size exceeds a value of 20%. If it also does not exceed 20%, it is inferred that there is no risk of occlusion, and the program ends in program step 340. In different cases, the program branches to program step 338, and in program step 338, measures to avoid the risk of occlusion are performed. Such measures include, for example, turning off the display and reducing the brightness of the display. While it's feasible to instruct the computing unit 40 to calculate the reduced display shape, this could present problems in practice. It would incur significantly higher research and development and testing costs. Furthermore, additional microcontrollers and interfaces might be necessary.

[0071] according to Figure 9 The flowchart illustrates the second part of the data evaluation. The procedure begins in step 350. In step 352, it is checked whether the consecutive values ​​BG_1x1i and BG_1x1i+3 are unequal, while BG_1x1i and BG1x1i+6 are equal. Here, the index i represents the image number. If the answer is yes, a further test is performed in step 354, namely, whether the consecutive values ​​BG_2x2i and BG_2x2i+3 are unequal, while BG_2x2i and BG_2x2i+6 are equal. If the answer is yes, a further test is performed in step 356, namely, whether BG_3x3i and BG_3x3i+3 are unequal, while BG_2x2i and BG_2x2i+6 are equal. If this condition is also met, it is inferred that a pulsating / flickering presentation has formed, which is undesirable because it distracts from events on the road. This particular form of the test can identify flickering at a frequency of approximately 4 Hz. The test can be adjusted accordingly to fit other frequencies.

[0072] However, this depends on the performance of the dedicated chip performing the image analysis, i.e., how many images it can analyze per second. In another variant implementation, the chip can analyze 60 image segments per second. Figure 6As shown, when the image is divided into 14 image segments, 14 image segments are analyzed in each loop. Due to the time required for this (approximately 250ms) and the continuously running video stream (30fps), involving 7 sequentially occurring images, this variant design operates as follows: In the first image, the entire image segment 1 and the image quadrant with number 2 are analyzed. In the second image in the video stream, the two image quadrants with numbers 3 and 4 are analyzed. In the third image in the video stream, the image quadrant with number 5 and the 3x3 image segment with number 6 are analyzed. In the fourth image in the video stream, the two 3x3 image segments with numbers 7 and 8 are analyzed. In the fifth image in the video stream, the two 3x3 image segments with numbers 9 and 10 are analyzed. In the sixth image in the video stream, the two 3x3 image segments with numbers 11 and 12 are analyzed. In the seventh image in the video stream, the two 3x3 image segments with numbers 13 and 14 are analyzed. For this variant design, adjustments need to be made accordingly in the appendix... Figure 7-9 The flowchart in the document.

[0073] In other implementations, the HUD display unit can be divided into different segments for testing. For example, the number of segments could be 4 x 4 = 16 segments.

[0074] The query to determine whether the value being compared is equal can also be designed such that a defined tolerance range for accepting deviations is still considered to be equivalent within that tolerance range when making comparisons.

[0075] All examples and conditional statements mentioned herein are not to be construed as limiting the specific examples described herein. Therefore, it will be appreciated, for example, by those skilled in the art, that the block diagrams shown herein present conceptual views of exemplary circuit arrangements. Similarly, it should be recognized that the flowcharts, state transition diagrams, pseudocode, and the like shown represent various variations for illustrating processes, which can be stored in a computer-readable medium and thus executed by a computer or processor. The objects referred to in this invention may also explicitly be human beings.

[0076] It should be understood that the proposed methods and corresponding apparatus can be implemented in various forms, including hardware, software, firmware, dedicated processors, or combinations thereof. Dedicated processors may include application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs), and / or field-programmable gate arrays (FPGAs). Preferably, the proposed methods and apparatus are implemented as a combination of hardware and software. The software is preferably installed as an application program on a program memory device. This is typically a computer-based machine with hardware such as one or more central processing units (CPUs), random access memory (RAM), and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described herein may be part of the application program or parts executed by the operating system.

[0077] The disclosure is not limited to the embodiments described herein. There is room for various adjustments and modifications, which those skilled in the art can consider, based on their expertise, to also be part of the disclosure.

[0078] The invention has been described in detail in the embodiments using its use in vehicles as an example. It should also be noted that (fixed-wing) aircraft and helicopters may be used, for example, in land-based mobility or in search and rescue operations.

[0079] However, it should be noted that the applications are not limited thereto. The invention can always be used when AR displays can enrich the field of vision of a driver, operator, or even simply someone wearing data glasses.

[0080] AR displays can also make operation easier in remote-controlled devices such as robots or drones (where remote control is performed via a monitor that displays camera images). There is also potential for its use in this area.

[0081] List of reference numerals

[0082] 10 means of transportation

[0083] 13 deer

[0084] 20 display units

[0085] 20' Instrument Cluster Display

[0086] 21 virtual projection surfaces

[0087] 22 Analysis Chips

[0088] 23LVDS bus

[0089] 24 microcontrollers

[0090] 25 bus

[0091] 26 projection units

[0092] 27 data cable

[0093] 28 Danger Symbols

[0094] 30 touch-sensitive screen

[0095] 40 computing units

[0096] 50 input units

[0097] 60 memory units

[0098] 70 Data cables leading to the display unit

[0099] 80 data lines to memory cells

[0100] 90 Data lines leading to the input unit

[0101] 100 data bus

[0102] 110 instrument cluster

[0103] 120 phone number

[0104] 130 navigation device

[0105] 140 radio

[0106] 150 cameras

[0107] 160 communication module

[0108] 200 infotainment system

[0109] 310-360 different program steps

Claims

1. A method for avoiding visual interference for operators of an object, wherein, Virtual supplementary information is displayed to the operator's field of vision through a display unit (20). The method is characterized by performing image analysis to check whether visual interference is caused by the display of virtual supplementary information, and wherein, when visual interference is identified, measures to eliminate visual interference are implemented. The image analysis used to reveal problems with areas obscuring the operator's field of vision involves analyzing multiple sequentially occurring images, wherein the images are progressively divided into segments and each segment is analyzed individually, wherein the total number of segments covers the entire image. The image analysis involves determining the fill factor of a corresponding image or segment resulting from the display of virtual supplementary information, wherein the fill factor is achieved by calculating the ratio of the number of pixels filled by the display of virtual supplementary information to the total number of pixels that can be filled.

2. The method according to claim 1, wherein, Perform image analysis to check whether the displayed virtual additional information is causing an area to be obscured in the operator's field of vision.

3. The method according to claim 1 or 2, wherein, Image analysis is performed on multiple sequentially displayed images with virtual additional information to check whether the display causes interfering flickering, jumping, or pulsation in the operator's field of vision.

4. The method according to claim 1 or 2, wherein, To check for occlusion of areas in the field of view, three consecutive images were analyzed. The fill degree of the entire image was determined for the first image, the fill degree of four image segments of the same size was determined for the second image, and the fill degree of nine image segments of the same size was determined for the third image.

5. The method according to claim 4, wherein, The determined image fill and segment fill are graded and checked, and when at least one of the checked criteria is confirmed to be met during the check, a problem is identified in which an area is obscured from the operator's field of vision.

6. The method according to claim 1 or 2, wherein, To check whether the display causes interfering flickering, jumping, or pulsation in the operator's field of vision, the determined fill level is analyzed through multiple sequential inspection processes. When a periodic change in the determined fill level is confirmed, the existence of an interfering flickering, jumping, or pulsation problem in the operator's field of vision is identified.

7. The method according to claim 1 or 2, wherein, To eliminate visual interference, the display is turned off or its brightness is reduced.

8. The method according to claim 1 or 2, wherein, The virtual supplementary information is displayed in a manner corresponding to augmented reality, which simulates contact with one or more identified objects and is brought into the operator's field of vision.

9. An apparatus for performing the method according to any one of the preceding claims, the apparatus having a display unit (20) by means of which virtual supplementary information can be displayed in the field of vision of an operator of an object, the apparatus further having an interface (70) for receiving an image by means of which virtual supplementary information can be displayed in the field of vision of an operator of an object, characterized in that, The device also includes an image analysis unit (22) and an evaluation unit (24) for checking whether visual interference is caused by the display of virtual additional information, and the image analysis unit (22) and the evaluation unit (24) are also designed to eliminate visual interference when it is detected.

10. The apparatus according to claim 9, wherein, The image analysis unit (22) and the evaluation unit (24) are also designed such that when visual interference is detected, the image analysis unit and the evaluation unit will cause the display to be turned off or the brightness of the display to be reduced.

11. A means of transportation, characterized in that, The vehicle (10) has the device according to claim 9 or 10.

12. A computer program, characterized in that, The computer program is designed to perform the steps of the method for avoiding visual interference to an operator of an object as described in any one of claims 1 to 8 when running in a computing unit including an image analysis unit (22) and an evaluation unit (24).

Citation Information

Patent Citations

  • Projection e.g. head up display, calibration system for e.g. motor vehicle, has control unit controlling unit for automatic adjustment of calibration, so that distortion free function, of projection is satisfied

    DE102005037797A1

  • Method for calibrating a head-up display

    DE102012010120A1

  • Method and device for augmented display of at least one additional piece of information in at least one image of an environment

    DE102014008152A1

  • Systems and Methods for Managing Augmented Reality Overlay Pollution

    US20160049013A1