Method for operating a display device in a vehicle

Abstract

The invention relates to a method for operating a display device (2) in a vehicle (1), comprising: operating (S1) the display device (2) in a first display mode (20) in which a bird's-eye view image (BEV image) of a first section is displayed; detecting (S2) an activation event (21) for which a display mode change to a second display mode (22) is specified; switching (S3) to the second display mode (22) in which a BEV image of a second section is displayed, which at least partially covers a different field of view compared to the first section; and operating (S4) the display device (2) in the second display mode (22). The method includes generating the BEV image for the first display mode (20) and / or the second display mode (22) by fusing a visual image (23) with point cloud information (24).

Description

[0001] The invention relates to a method for operating a display device in a vehicle. Furthermore, the invention relates to a control device for a vehicle, a vehicle itself, and a computer program for carrying out such a method.

[0002] A vehicle may be equipped with a display device configured to show a bird's-eye view (BEV) image of at least a portion of the vehicle's surroundings. Such a BEV image is particularly useful for the driver when maneuvering in confined spaces, such as parking lots and / or narrow streets, as it provides a comprehensive view of at least one object at a relatively close range to the vehicle. The object may be another vehicle or another obstacle. The field of view (FOV) covered by the BEV image is typically spatially limited. The FOV may be smaller than the detection range of a sensor device on the vehicle that provides the sensor information used to generate the BEV image.Depending on the current situation, a modified FOV may be desired, which, for example, covers a larger section of the environment than usual, at least locally.

[0003] US 2008 / 0007618 A1 discloses a device for generating a vehicle peripheral image and an image switching method which provides an additional rear view image from a vehicle peripheral image.

[0004] US 11,117,570 B1 discloses a device comprising a sensing device. The sensing device is configured to generate pixel data corresponding to an external view of the vehicle. A lighting device may be configured to generate light for the external view. A processor may be configured to generate video frames from the pixel data, perform computer vision operations on the video frames to detect objects in the video frames, and generate a control signal.

[0005] The object of the invention is to provide a particularly useful BEV image that is displayed in a vehicle.

[0006] The independent claims solve the problem.

[0007] A first aspect of the invention relates to a method for operating a display device in a vehicle. The display device can be a human-machine interface (HMI). The display device can be located in the front of the vehicle, for example, in a center console. Alternatively or additionally, the display device can be a head-up display. The display device can also include the vehicle's infotainment screen.

[0008] The display device is configured to show a bird's-eye view image (BEV image) of at least a portion of the vehicle's surroundings. The portion is a part of the surroundings; thus, it is a spatially limited area within the environment. The BEV image is a 360-degree view. Specifically, the BEV image is a top-down view that describes the vehicle and at least the portion of its surroundings. The BEV image can be a visual image captured by a camera system on the vehicle. This camera system can include multiple cameras mounted at the front, rear, and sides of the vehicle to capture a complete panoramic view of the vehicle's surroundings. The surroundings can alternatively be referred to as the vehicle's environment. The images captured by the cameras of the camera system are then stitched together and projected onto the display device.

[0009] The invention is based, at least, on the observation that the BEV image is displayed to show a comprehensive view of an object, in particular at least one other vehicle and / or another obstacle, in close proximity to the vehicle. Furthermore, the vehicle can be configured to provide a warning when an object is too close to the vehicle to the point of collision. The combination of the BEV image and the warning can enable the vehicle to be operated safely, as it can help the driver avoid a collision with the object. However, a typical BEV image is merely a visual representation of at least a portion of the vehicle's surroundings without any further information about the object in the surroundings. It is therefore advantageous to enhance the BEV image by adding details about at least one object in the surroundings described by the BEV image.This can be achieved through sensor fusion, i.e., fusing, for example, a visual image of the surroundings with point cloud information that describes at least one object in the environment. A radar device, a lidar device, and / or an ultrasonic sensor on the vehicle can acquire the point cloud information.

[0010] Furthermore, in specific situations, such as during a parking maneuver, the driver may be confronted with a limited field of view (FOV) of the BEV image. For example, the BEV image may not capture the entire surroundings relevant to the current situation, but only a portion of them. Additionally, the BEV image may not show a dynamic object, such as another vehicle approaching from far behind, because it is still outside the BEV's FOV. There may also be blind spots in the environment that are not visible in the BEV image. This can give the driver a false sense of control and safety, leading them to believe that the vehicle has detected all potential hazards, when in fact it may not be displaying them due to the limited field of view provided by the BEV image.The FOV of the BEV image should therefore be adaptable to a current situation.

[0011] FOV refers to the area of ​​the observable environment that can be captured by the camera or camera system at any given moment, or that is selected for display at that moment. The FOV can be expressed in degrees, representing an angular range visible to an observer and / or at a specific position of the camera and / or camera system. A wider FOV describes a larger portion of the environment, while a narrower FOV describes a smaller portion and thus a more focused view of a smaller area. In other words, the FOV can be used to describe an observer's perspective or the area that can be captured by a particular imaging system, such as a vehicle's camera system.

[0012] The method comprises operating the display device in a first display mode, in which a BEV image of a first section of the vehicle's surroundings is displayed. The display device shows the BEV image of the first section when it is operated in the first display mode. The first section can include a front area, a rear area, and a side area of ​​the vehicle, covering, for example, the entire area from an outer wall of the vehicle up to a distance of two meters from the outer wall.

[0013] The procedure involves detecting an activation event. A display mode change to a second display mode is predefined for the activation event. This means that the activation event is linked to the display mode change, so that whenever the activation event occurs, the display device switches to the second display mode. The activation event could, for example, be the activation of a specific function, particularly a specific driver assistance system, in the vehicle, provided that the display mode change is intended and thus predefined for this specific function or driver assistance system. The driver assistance system could, for example, be a parking assistant or another parking function that provides at least assisted, and in particular semi-automated or fully automated, parking into and / or exiting a parking space.

[0014] The activation event can be detected when the function and / or driver assistance system for which the display mode change is specified is manually activated. Manual activation can include operating a control element in the vehicle, such as a button, switch, key, and / or an element on a touchscreen of the display device, a voice command, and / or a gesture control command. Alternatively or additionally, the activation event can be detected by analyzing a steering angle, speed, acceleration, selected gear, data acquired by a rain sensor and / or a temperature sensor, a visual image of the surroundings, and / or a point cloud describing at least one object in the environment. Through such analysis, the vehicle can detect a parking maneuver and / or specific weather conditions for which the display mode change is specified.Alternatively or additionally, the activation event can be detected when the vehicle's current position is within a predetermined area where the display mode is to be changed. This means that a geofence can allow the detection of the activation event. A list of activation events and the display mode changes they trigger can be stored in a memory unit or storage device within the vehicle.

[0015] The procedure involves switching from the first display mode to the second display mode after detecting the activation event. The second display mode shows the BEV image of a second section of the vehicle's surroundings. This second section covers at least a partial different field of view (FOV) compared to the first. For example, the second section might cover a larger portion of the surroundings in the rear, front, and / or at least one of the sides of the vehicle compared to the first. In this example, the second display mode is an extended display mode compared to the first.

[0016] After switching to the second display mode, the procedure involves operating the display device in that mode. The display device then shows the BEV image of the second section when operating in the second display mode. In other words, the display device's mode can switch from the first to the second display mode for a specific situation identified by the detected activation event, thus changing the FOV of the displayed BEV image and adapting it to the current situation.

[0017] The method involves generating the BEV image displayed in the first and / or second display mode by fusing a visual image of the respective area with point cloud information describing at least one object in that area. When the fusing occurs in the first display mode, the visual image of the first area is fused with the point cloud information describing the at least one object in the first area. When the fusing occurs in the second display mode, the visual image of the second area is fused with the point cloud information describing at least one object in the second area. In a preferred example, the fusing occurs at least in the second display mode.

[0018] The advantage of the inventive method lies not only in adapting the displayed section of the environment to the current situation by taking the activation event into account, but also in adding the displayed BEV image by fusing the purely visual image with point cloud information. The point cloud information includes, for example, details about the shape, relative distance, and / or relative movement of at least one object in the environment. As a result, the driver has access to detailed information about the vehicle's surroundings and can thus better understand them. Therefore, the method provides a particularly useful BEV image displayed in the vehicle.

[0019] In a preferred example, a vehicle control device performs the inventive method. The control device receives, for example, the visual image and / or point cloud information from at least one sensor device of the vehicle. Typically, the visual image is captured by the vehicle's camera system and / or individual cameras, such as a front camera, a rear camera, and / or side cameras. The camera system and / or the respective camera can transmit the captured visual image to the control device, and the control device can then generate the BEV image based on the received visual images. Furthermore, at least one radar device, at least one lidar device, and / or at least one ultrasonic sensor of the vehicle can capture the point cloud information and transmit it to the control device, enabling the control device to generate the fused BEV image.The vehicle may include multiple radar devices, lidar devices and / or ultrasonic sensors located in the front, rear and / or side areas.

[0020] The method can be a computer-implemented method. In a preferred example, the control device is located in the vehicle. Alternatively or additionally, the control device, which performs at least some of the steps of the inventive method, can be located externally; it can be, for example, a backend, a server, and / or an external control device.

[0021] One embodiment includes the generation of the BEV image in the first display mode and / or the second display mode comprising object detection and / or object tracking applied to the visual image and / or the point cloud information. Object detection can be achieved by applying an object detection algorithm to the visual image and / or the point cloud information to detect at least one object located in the environment described by the visual image and / or the point cloud information. Object tracking can be achieved by applying an object tracking algorithm to the visual image and / or the point cloud information to track the detected object as it moves within the environment described by the visual image and / or the point cloud information. Object tracking can include predicting a trajectory for a moving object.Object detection and object tracking can use known techniques for detecting and tracking objects in images and / or point clouds.

[0022] The process involves generating a three-dimensional model of the detected and / or tracked object. This three-dimensional model can describe the detected and / or tracked object more realistically than a two-dimensional image of the object. For example, if the detected and / or tracked object is a pedestrian, the three-dimensional model can depict a three-dimensional model or representation of that pedestrian. The specific design of the three-dimensional model can depend on the visual image or object class of the detected object. The object class can be "pedestrian" for the detected pedestrian, or it can be a standard three-dimensional model of a pedestrian, independent of the actual pedestrian. The size or extent of the three-dimensional model can depend on the point cloud information.For example, the size or extent may depend on an area covered by points in the point cloud that describes the object.

[0023] Alternatively or additionally, the procedure can include determining at least one piece of object information. This object information describes a detected and / or predicted movement of the object. The object information is determined, for example, by applying an object information determination algorithm to the information obtained from object detection and / or object tracking. The object information determination algorithm can be applied to the visual image and / or the point cloud information. The object information determination algorithm can include at least one rule, the application of which leads to the determination of the object information.

[0024] The specific object information can describe the detected and / or predicted motion of the object by its velocity. Alternatively or additionally, it can describe the predicted motion by a predicted collision of the vehicle with the detected and / or tracked object. It is therefore possible to add information about the detected and / or predicted motion of the object, for example, by displaying the object information as a value and / or as text next to the three-dimensional model or the representation of the object in the BEV image.

[0025] The process involves inserting the generated three-dimensional model and / or specific object information into the BEV image. For example, if the detected and / or tracked object is another vehicle approaching the vehicle, the approaching vehicle can be displayed in the BEV image shown by the display device using a three-dimensional model of the vehicle, along with the speed of the other vehicle and / or the remaining time until collision. This makes it possible to display details about dynamic and / or static objects within the displayed area of ​​the environment to assist the driver.

[0026] This allows for a particularly useful BEV image, which uses the fusion of visual images and point cloud information to derive as much information as possible about objects and potential obstacles for the vehicle. It is thus possible to extend the field of view (FOV) of the BEV image of the environment using established three-dimensional modeling techniques. This is achieved by adjusting the position and orientation of the sensor devices that provide the visual image and / or the point cloud information; that is, by using more or less data from the sensor device in the second display mode compared to the first. The FOV of the BEV image can therefore be designed to capture both the existing environment and the extended area compared to the initial view.The field of view (FOV), particularly of the camera or camera system, should be wide enough to capture an entire area of ​​interest, which depends on the current situation and thus the activation event. The adjusted FOV of the BEV image is then rendered and displayed on the display device, allowing the driver to interact with the surroundings.

[0027] The invention can be based, at least in part, on the observation that integrating other sensor devices, such as radar and / or lidar devices with cameras, can significantly improve the accuracy and robustness of the vehicle, particularly in challenging environments. Combining depth information from multiple sensors, i.e., point cloud information, can improve the accuracy of a three-dimensional model of an object. Lidar devices can provide accurate depth measurements in almost all lighting conditions, although they may have a lower resolution. Radar devices can provide information about velocity vectors. By fusing the data from these sensors, it may be possible to create an accurate and robust three-dimensional model that includes object detection and tracking, especially in low-light conditions.

[0028] Another embodiment therefore involves object detection and / or object tracking only being applied when the ambient light level is below a predetermined threshold. It can be predetermined that switching from the first display mode to the second display mode can only occur when, for example, the ambient light intensity is already so low that the BEV view of the second display mode is required. For example, the generation of the three-dimensional model and / or the determination of object information are only performed at night, at dusk, and / or at dawn. This reduces the energy and computing power required for the process, as object detection and object tracking are only performed when necessary due to ambient light conditions.

[0029] Another embodiment involves the second view partially covering an expanded FOV and / or partially a reduced FOV compared to the first view. It is possible to define areas of interest in the vehicle's surroundings and adjust the FOV locally so that these areas of interest are included in the second view. This can result in larger FOVs locally compared to the first display mode, but it can also result in smaller FOVs locally if some parts of the first view are not areas of interest for the second display mode. For example, during a reversing parking maneuver, objects in front of the vehicle may be less important than objects behind the vehicle. Thus, the second view can be enlarged at the rear but reduced at the front compared to the first view. Other combinations of expanded and / or reduced FOVs are possible.This allows the displayed environment to be adjusted so that it only covers parts that are currently of interest and omits all parts that are not of interest.

[0030] A preferred embodiment involves multiple activation events. For each of these events, a display mode change to one of several secondary display modes is predefined. It is then possible to switch between different display modes. The first display mode can be understood as a default display mode, used whenever no activation event has been detected. However, in the case of a detected activation event, there can be several possible secondary display modes, depending on the specific activation event detected. For example, a parking maneuver can involve several possible activation events, depending on the vehicle's current direction of travel and / or maneuver path. During the parking maneuver, multiple secondary display modes can be activated as activation events are detected sequentially.

[0031] In a preferred example, after switching from the first display mode to the second display mode, a switch back to the first display mode is performed. Alternatively, the display device can switch from the second display mode to another second display mode without switching back to the first display mode in between. This means that the method can involve switching from the first display mode to the second display mode and from the second display mode to any other second display mode among several second display modes. The method is thus particularly versatile, as it is not limited to a single second display mode.

[0032] Another preferred embodiment involves the detected activation event being the act of parking in or out of a parking space. Alternatively, the detected activation event is at least a part of the parking or out-of-space maneuver. The parking or out-of-space maneuver is, in particular, a maneuver to park in or out of a parallel parking space located on a first side of the vehicle. Alternatively or additionally, the parking space can be a perpendicular parking space, such as a perpendicular parking space, a herringbone parking space, or any other angled parking space. Alternatively or additionally, the detected activation event is a specific weather condition, such as fog, rain, and / or snow.Whenever weather conditions are detected, a second display mode that fuses visual images with point cloud information can be useful, so a switch to such a second display mode can be performed.

[0033] Another embodiment involves the BEV image in the second display mode having an extended field of view (FOV), at least in the rear area of ​​the vehicle, compared to the first display mode, when the display device is activated in the second display mode after detecting reversing while parking. During parking, the FOV behind the vehicle can thus be enlarged to assist with reversing. Specifically, the extended FOV is selected at least for the first side. It is possible that the FOV is extended only in the rear area and on the first side of the vehicle, but not on an opposite second side facing away from the parallel parking space and in the front area of ​​the vehicle. On the second side and / or in the front area, the FOV can be reduced compared to the first display mode.In this example, the BEV image in the second display mode can describe the vehicle at an upper left or right corner, so that the extended FOV on the front side and in the rear area, but specifically only on the front side, can cover the rest of the BEV image. This means that only the areas of interest, which in this case are those near the parking space and in the rear area, can be in focus in the second display mode.

[0034] Another embodiment involves the BEV image in the second display mode having the same or a more extended field of view (FOV), at least in the rear area of ​​the vehicle, compared to the second display mode used for reversing while parking, when the display device is operated in the second display mode while exiting the parking space. During exiting, the FOV in the rear area can even be further enlarged compared to the reversing portion of the parking maneuver, allowing vehicles approaching from behind to be detected or seen earlier.

[0035] In particular, the same or a more extended field of view (FOV) is located at least on the second side of the vehicle, at least in the rear area. This second side is opposite the first. Therefore, the FOV on the side of the vehicle facing the parking space is not focused, but rather on the second side, which faces a lane adjacent to the parking space. It is assumed that the parking space is the one parallel to the parking space. The focus of the BEV image displayed in this second view switches to the lane, which is particularly useful for exiting the parking space.

[0036] It's possible that the FOV is extended only at the rear and on the second side of the vehicle, but not at the front or front. On the front and / or front, the FOV may be reduced compared to the first or second display mode for the portion of the reversing maneuver while parking. In this example, the BEV image in the second display mode might describe the vehicle from the upper left or right corner, allowing the extended FOV on the second side and rear (but specifically only on the second side) to cover the rest of the BEV image. This means that only the areas of interest—in this case, the road and the rear—can be in focus in the second display mode.

[0037] According to another embodiment, in the second display mode, the BEV image has an extended field of view (FOV) at least in a frontal area of ​​the vehicle compared to the first display mode, when the display device is operated in the second display mode after detecting forward movement during parking as the activation event. For example, if the vehicle changes direction from reversing to forward while parking, the extended FOV in the rear area is no longer needed, so the extension of the second view can be switched to the front of the vehicle compared to the first view or the second view described above. It is possible that the extended FOV is located at least on the front of the vehicle, at least on the first side, so that the focus is directed towards the parking space.This assumes that the parking space is the parallel parking space. Therefore, it is not only possible to extend the area covered by the BEV image at the rear, but also to extend this to the front area if necessary and useful.

[0038] It's possible that the FOV is only extended at the front and on the first side of the vehicle, but not at the rear. On the rear and / or in the second side, the FOV may be reduced compared to the first display mode, or the second display mode for reversing while parking or pulling out. In this example, the BEV image in the second display mode might describe the vehicle from a lower left or right corner, allowing the extended FOV on the front and front (but specifically only on the front) to cover the rest of the BEV image. This means that only the areas of interest, in this case the parking space and the front, can be in focus in the second display mode.

[0039] Furthermore, one embodiment includes the BEV image in the second display mode having both the same or a more extended FOV at the rear of the vehicle compared to the second display mode specified for reversing while parking, and the same or a more extended FOV at the front of the vehicle compared to the second display mode specified for driving forward while parking, when the display device is operated in the second display mode while exiting the parking space. It is thus possible not only to provide an extended FOV at the rear when exiting, but also an extended second view at the front of the vehicle. The same or a more extended FOV can also be applied to the second side. On the first side, the FOV can be reduced compared to the first display mode or the second display mode for parking.In this example, the BEV image in the second display mode can depict the vehicle at a central left or right corner, allowing the extended FOV to cover the rest of the BEV image on the second side, as well as in the front and rear areas, but specifically only the areas of interest. This means that only the areas of interest—in this case, the road, the front, and the rear—can be in focus in the second display mode. This allows for a particularly wide FOV when pulling out to detect other vehicles on the road as early as possible.

[0040] Another embodiment involves the BEV image in the second display mode having an extended field of view (FOV) on at least one side of the vehicle compared to the first display mode, when the display device is operated in the second display mode after detecting parking or exiting, and / or at least part of the parking or exiting process as the activation event. Thus, it is not only necessary to enlarge or extend the FOV in the front and / or rear areas, but also to enlarge it on one side of the vehicle. Specifically, the "at least one side" is the first side in the case of parking into the parking space, and the opposite second side in the case of exiting the parking space.The other side, i.e., the second side when parking or the first side when exiting a parking space, can be displayed with a reduced field of view (FOV) in the second display mode. This means that there is a focus on one side of interest, which can be the side facing away from the parallel parking space when parking, and the side facing the road when exiting.

[0041] However, the method can combine some or all of the described extended FOVs, i.e., an enlargement of the first view in the first display mode at the front, rear, first side, and second side, so that all boundaries of the BEV image are shifted outwards from the vehicle. For example, the second view can include an area at the front, rear, and sides of the vehicle, covering the entire area from an outer wall of the vehicle to a distance of three or four meters from that wall. In another example, additional extended areas at the front and rear are provided that extend further away from the outer wall than the extended areas on the sides. However, due to the larger FOV required when exiting a parking space compared to parking in, these can only be used for exiting, not for parking.

[0042] Another embodiment involves the display device reverting to the first display mode and operating in that mode when the activation event is no longer detectable. This means that detection of the activation event continues while the display device is in the second display mode. For example, if the activation event is parking, the system continuously verifies whether parking is still in progress. Once parking is complete, for example, when a parking assistant is deactivated, the display device reverts to the first display mode and operates in that mode until the activation event is detected again or another activation event is detected. Thus, the second display mode is used only temporarily and in specific situations.

[0043] Another aspect of the invention relates to a control unit for a vehicle. The control unit is configured to carry out the method described above. The control unit carries out the method described above.

[0044] The control unit can be understood as a computing unit or as a data processing device with processing circuitry. The control unit can therefore perform arithmetic operations for data processing and thus execute the procedure. These arithmetic operations can also include indexed access to a data structure, for example, a lookup table (LUT).

[0045] In particular, the control system may comprise at least one computer, at least one microcontroller, and / or at least one integrated circuit, for example, at least one application-specific integrated circuit (ASIC), at least one field-programmable gate array (FPGA), and / or at least one system-on-a-chip (SoC). The control system may comprise at least one processor, for example, at least one microprocessor, at least one central processing unit (CPU), at least one graphics processing unit (GPU), and / or at least one signal processor, in particular at least one digital signal processor (DSP). The control system may comprise a physical or virtual array of computers or other units of the aforementioned type.

[0046] The control system can have at least one hardware and / or software interface and / or at least one memory or storage unit. The memory or storage unit can be volatile data storage, for example, dynamic random access memory (DRAM) or static random access memory (SRAM), or non-volatile data storage, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, or ferroelectric random access memory.FRAM (ferroelectric random access memory), MRAM (magnetoresistive random access memory), or PCRAM (phase-change random access memory) can be designed as random access memory.

[0047] Another aspect of the invention relates to a vehicle. The vehicle is equipped to perform, or performs, the method described above. The vehicle may have a sensor device, such as a camera, radar, lidar, and / or ultrasonic sensor, that captures the visual image or point cloud information describing at least one object in the environment. The vehicle may have a control device. In a preferred example, the vehicle is a motor vehicle, for example, a passenger car, a truck, a bus, a motorcycle, and / or a moped.

[0048] One aspect of the invention relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions which, when the program is executed by a computer, such as the control device, cause the computer to perform the method described above.

[0049] The embodiments described in connection with the computer-implemented method apply both individually and in combination with one another when they are applicable to the control system, vehicle, and computer program product according to the invention. The invention comprises combinations of the described embodiments.

[0050] The characters show in: Fig. 1 a schematic representation of a vehicle starting a parking maneuver, Fig. 2 a schematic representation of a method for operating a display device in a vehicle, and Fig. 3 A schematic representation of bird's-eye view images.

[0051] The same components are labelled with the same reference symbols in the figures.

[0052] Fig. Figure 1 shows a vehicle 1, which has a display device 2. The display device 2 may be located in a control center of the vehicle 1 and / or may be a head-up display. The display device 2 can be understood as a human-machine interface (HMI) in the vehicle 1. The vehicle 1 may have a control device 3, which is configured to perform a procedure to operate the display device 2 in the vehicle 1.

[0053] Furthermore, the vehicle 1 can have several sensor devices. For example, it can have a front camera 4, a rear camera 5, and side cameras 6, which are located in the side mirrors of the vehicle 1. Additional sensor devices can be radar devices 7, which are located in the front and rear of the vehicle 1. Alternatively or in addition to the sensor devices shown, the vehicle 1 can have at least one lidar device and / or at least one ultrasonic sensor. Other positions and / or a different number of the sensor devices shown are possible. The arrangement of the sensor devices in Fig. 1 is just an example.

[0054] Vehicle 1 is currently performing a parking maneuver 9 into a parking space 8 located on the first side 15 of vehicle 1. The first side 15 is opposite a second side 16 of vehicle 1. The second side 16 faces a roadway on which the vehicle is located. The roadway is adjacent to parking space 8. Parking space 8 is, for example, a parallel parking space 8.

[0055] In Fig. Figure 1 shows several movement arrows. Vehicle 1 can first drive along an initial reverse path 10 into parking space 8, then adjust its position by driving forward 11 and then reversing 12 to a final parking position in parking space 8. This is just one example. Other movements of vehicle 1 during parking maneuver 9 are possible.

[0056] The driver of vehicle 1 should take into account another vehicle 13 approaching vehicle 1 from behind. This other vehicle 13 is performing a parking maneuver 9 to park in the other parking space 15 next to parking space 8. Furthermore, there may be moving objects on the first side 15 that should be considered during vehicle 1's parking maneuver. For example, these moving objects could be children 14 playing, who could be potential obstacles for vehicle 1 during the parking maneuver 9.

[0057] Fig. Figure 2 describes the steps of a method for operating the display device 2 in the vehicle 1. The described method can be performed by the control unit 3 of the vehicle 1. In a preferred example, it is a computer-implemented method. In step S1, the method includes operating the display device 2 in a first display mode 20. In the first display mode 20, a bird's-eye view image (BEV image) of a first section of the vehicle 1's environment is displayed. In step S2, the method includes detecting an activation event 21, for which a display mode change to a second display mode 22 is performed.

[0058] The activation event can be, for example, parking in or driving out of parking space 8. Alternatively or additionally, the activation event 21 can be at least part of the parking or driving out process. In particular, the parking or driving out process is carried out for a parallel parking space 8 located on the first side 15 of vehicle 1, as shown in Fig. 1 is shown. Alternatively or additionally, activation events can include specific weather conditions, such as fog, rain, or snow, and / or specific road conditions, where, for example, an extended field of view (FOV) of the BEV image may be required. In general, there can be more than one activation event 21 for which a switch to a second display mode is specified. In the case of multiple second display modes, for example, an activation event 21 is defined for each of the multiple second display modes. Therefore, there can be different second display modes 22 for the reverse travel path 10, 12 and the forward travel path 11.

[0059] The activation event can be a manual activation of a function in the vehicle 1, which may be a driver assistance system or any other vehicle function for which the activation event 21 is specified. Alternatively or additionally, an analysis of a steering angle, a selected gear, a speed, an acceleration, and / or a situation described by sensor information acquired by the sensor devices of the vehicle 1 can allow it to be determined whether a display mode change is specified for the current situation of the vehicle, so that an activation event 21 is detected.

[0060] Step S3 involves switching from the first display mode 20 to the second display mode 22 after detecting the activation event 21. The second display mode 22 is a mode in which the BEV image of the second section of the vehicle 1's environment is displayed. The second section covers at least partially different FOVs compared to the first section of the first display mode 20. For example, the second section may partially cover an extended FOV and / or partially a reduced FOV compared to the first section. Step S4 involves operating the display device in the second display mode 22. Step S4 is performed after step S3.

[0061] Step S5 can include verifying whether the activation event 21 is still detectable. For example, if the activation event 21 is no longer detectable, the display device 2 can switch back from the second display mode 22 to the first display mode 20 in step S6 and operate the display device 2 in the first display mode 20 in step S7. However, if the activation event is still detectable, the procedure can remain at repeating steps S4 and S5.

[0062] The method comprises generating the BEV image, which is displayed in the first display mode 20 and / or in the second display mode 22, by fusing a visual image 23 of the respective area with point cloud information 24 that describes at least one object in the respective area. The visual image 23 was acquired by the front camera 4, the rear camera 5, and / or the side cameras 6 of the vehicle 1. The point cloud information 24 was acquired by at least one of the radar devices 7 and / or by a lidar device and / or an ultrasonic sensor of the vehicle 1. In a preferred example, the fusing of the visual image 23 with the point cloud information 24 is performed only in the second display mode 22 and not in the first display mode 20. The fusing of the visual image 23 with the point cloud information 24 can also be performed in the first display mode 20.

[0063] Furthermore, the method can include object detection 25 and / or object tracking 26 applied to the visual image 23 and / or the point cloud information 24. This allows the method to generate a three-dimensional model 27 of the detected and / or tracked object and / or determine at least one piece of object information 28 that describes a detected and / or predicted movement of the object in the environment. The three-dimensional model 27 of the other vehicle 13 can be a three-dimensional representation of the vehicle 13. The object information 28 in this example can be details of the current speed of the other vehicle 13 and / or a potential point and / or time of collision between the other vehicle 13 and vehicle 1.The predicted collision can depend on predicted motion data derived from the visual image 23 and / or the point cloud information 24, as well as, for example, details of the current driving situation of the vehicle 1 itself, such as its speed, acceleration and / or parking trajectory.

[0064] The procedure can involve inserting the generated three-dimensional model 27 and / or the specified object information 28 into the BEV image of the second display mode 22. It can also be inserted into the BEV image of the first display mode 20. In general, object detection 25 and / or object tracking 26 can only be performed if the ambient light level is below a predetermined threshold. This threshold can define nighttime, dusk, and / or dawn as being below the threshold.

[0065] Fig. Figure 3 shows BEV images with different FOVs. Here, there is a model of vehicle 1 located in the center of the BEV image for the first display mode 20. However, there are other BEV images shown here. It is assumed that the activation event 21 describes at least part of the parking maneuver 9 while parking in or out of parking space 8. If, after detecting reversing during parking as the activation event 21, the display device 2 operates in the second display mode 22, which is the case, for example, for the reversing path 10, 12, the BEV image in the second display mode 22 can have an extended FOV 30, at least in a rear area of ​​vehicle 1, in particular at least on the first side 15, compared to the first display mode 20.An additional area at the rear, shown here as FOV 31 at least up to the dashed line 32 on the first page 15, can also be included in this BEV image. However, FOV 31 can only be selected for exiting and not for parking.

[0066] If the display device 2 is operated in the second display mode 22 while exiting parking space 8, particularly regardless of the direction of travel, the BEV image can show the same or even a more extended FOV, at least in the rear area of ​​the vehicle 1, and in particular at least on the second side 16, compared to the second display mode 22, which is specified for reversing during parking. This is shown here as FOV 36, in particular together with FOV 31, up to the dashed line 36 on the second side 16.

[0067] Alternatively or additionally, if the display device 2 is operated in the second display mode 22 after the detection of forward movement during parking as the activation event 23, the BEV image in the second display mode 22 can have an extended FOV 37 at least in a front area of ​​the vehicle 1, in particular at least on the first side 15, compared to the first display mode. This BEV image can even be extended to the FOV 34 on the first side 15 up to the dashed line 35.

[0068] For exiting, the extended FOV can be used in the rear area as well as in the front area, meaning that a combined BEV image of FOVs 38 and 36 can be displayed. Furthermore, this extended BEV image can also include FOV 34 on the second page 16 up to line 39 and FOV 31 on the second page 16 up to line 37.

[0069] As shown here, the BEV image for parking and exiting can have an extended FOV on at least one of the sides 15, 16 of vehicle 1. In particular, the at least one side 15, 16 that is extended is, in the case of parking into parking space 8, the first side 15 and the opposite second side 16 in the case of exiting parking space 8.

[0070] The BEV image can be a combination of all FOVs 30, 31, 33, 34, 36, 38. Fig. 3 indicates the front 40 and the rear 41 of vehicle 1.

[0071] Overall, the invention demonstrates an adaptive BEV image for parking support HMI on multiple data domains.

[0072] In the case of parking or exiting, the procedure may include: - Aligning Vehicle 1: The driver should align the rear of their vehicle 1 with the rear of vehicle 1 in front of the empty space (parking space 8). The standard BEV FOV is then displayed, which is the BEV image of the first display mode 20. - Reversing and turning the steering wheel: The driver should begin reversing slowly while turning the steering wheel fully towards the curb. The standard BEV FOV (first display mode 20) is then adjusted to the second display mode with an FOV of 30. - Straightening the wheels: When the front of vehicle 1 extends approximately halfway into parking space 8, the driver should straighten the wheels. The FOV will then be adjusted to FOV 33. - Resume Reversing: The driver should continue reversing until vehicle 1 is parallel to the curb and the vehicle is at the front and rear. Then the FOV is adjusted back to the standard BEV FOV range, which in this case is the BEV image of the first display mode 20. - During the reversing maneuver (driving out), the FOV is adjusted accordingly to FOV 38, 36, 34, 31.

[0073] Using information about dynamic and static objects from the three-dimensional model environment, all areas can announce / show all affected dynamic objects (vehicles 13, pedestrians) with additional information such as speed or probability of collision in order to avoid these collisions. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 2008 / 0007618 A1

[0003] US 11,117,570 B1

[0004]

Claims

[1] Method for operating a display device (2) in a vehicle (1), comprising: - Operating (S1) the display device (2) in a first display mode (20) in which a bird's-eye view image of a first section of the vehicle's (1) environment is displayed; - Detecting (S2) an activation event (21) for which a display mode change to a second display mode (22) is specified; - upon detection of the activation event (21), switching (S3) from the first display mode (20) to the second display mode (22), in which a bird's-eye view image of a second section of the vehicle's surroundings (1) is displayed, the second section covering at least partially a different field of view compared to the first section; and - Operating (S4) the display device (2) in the second display mode (22); wherein the method comprises generating the bird's-eye view image which is displayed in the first display mode (20) and / or the second display mode (22) by fusing a visual image (23) of the respective section of the environment with point cloud information (24) which describes at least one object in the respective section of the environment. [2] Method according to claim 1, comprising generating a bird's-eye view image in the first display mode (20) and / or the second display mode (22): - Object detection (25) and / or object tracking (26) applied to the visual image (23) and / or the point cloud information (24); - Generating a three-dimensional model (27) of the detected and / or tracked object and / or determining at least one object information (28) that describes a detected and / or predicted movement of the object; and - Inserting the generated three-dimensional model (27) and / or the specific object information (28) into the bird's-eye view image. [3] Method according to claim 2, wherein object detection (25) and / or object tracking (26) is only applied when the ambient light conditions are below a predetermined light condition threshold. [4] Method according to one of the preceding claims, wherein the second section partially covers an extended field of view and / or partially a reduced field of view compared with the first section. [5] Method according to one of the preceding claims, wherein there are several activation events (21) for which a display mode change to one of several second display modes (22) is specified. [6] Method according to one of the preceding claims, wherein the detected activation event (21) is parking into or exiting a parking space (8) and / or at least part of the parking or exiting, in particular for a parallel parking space (8) located on a first side (15) of the vehicle (1). [7] Method according to claim 6, wherein the bird's-eye view image in the second display mode (22) has an extended field of view at least in a rear area of ​​the vehicle (1), in particular at least on the first side (15), compared with the first display mode (20), when the display device (2) is operated in the second display mode (22) after detecting reversing during parking as the activation event (21). [8] Method according to claim 7, wherein the bird's-eye view image in the second display mode (22) has, at least in the rear area of ​​the vehicle (1), in particular at least on a second side (16) opposite the first side (15), the same or a more extended field of view compared to the second display mode (22) specified for reversing while parking, when the display device (2) is operated in the second display mode (22) while driving out of the parking space (8). [9] Method according to any one of claims 6 to 8, wherein the bird's-eye view image in the second display mode (22) has an extended field of view at least in a front area of ​​the vehicle (1), in particular at least on the first side (15), compared with the first display mode (20), when the display device (2) is operated in the second display mode (22) after detecting forward driving during parking as the activation event (21). [10] Method according to claims 8 and 9, wherein the bird's-eye view image in the second display mode (22) has both the same or a more extended field of view in the rear area of ​​the vehicle (1), in particular at least on the second side (16), compared with the second display mode (22) specified for reversing while parking, and also the same or a more extended field of view in the front area of ​​the vehicle (1), in particular at least on the second side (16), compared with the second display mode (22) specified for driving forward while parking, when the display device (2) is operated in the second display mode (22) while driving out of the parking space (8). [11] Method according to any one of claims 7 to 10, wherein the bird's-eye view image in the second display mode (22) has an extended field of view on at least one side (15, 16) of the vehicle (1) compared with the first display mode (20), when the display device (2) is operated in the second display mode (22) after detecting the parking or exiting and / or at least part of the parking or exiting as the activation event (21), wherein in particular the at least one side (15, 16) is the first side (15) in the case of parking in the parking space (8) and the opposite second side (16) in the case of exiting the parking space (8). [12] Method according to any one of claims 7 to 11, wherein, when the activation event (21) is no longer detectable, the display device (2) switches back to the first display mode (20) and is operated in the first display mode (20). [13] Control device (3) for a vehicle (1), wherein the control device (3) is configured to perform a method according to one of the preceding claims. [14] Vehicle (1) which is equipped to carry out a method according to any one of claims 1 to 12. [15] Computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform a method according to any one of claims 1 to 12.

Images