Method and system for displaying video call in vehicle

Abstract

In a method for displaying a video call in a vehicle (102), at least one first image having an image of a session participant (104) remote from the vehicle (102) is captured. First image data corresponding to at least the first image is generated. A background in the first image is determined based on the first image data. Second image data is generated based on the first image data and the determined background, the second image data corresponding to a second image having an image of the remote session participant (104), the background being transparently presented in the second image. In addition, a second image is displayed to a vehicle driver (106) of the vehicle (102) by means of a head-up display (118) of the vehicle (102) during travel.

Description

Technical Field

[0001] The present invention relates to a method for displaying a video call in a vehicle. The present invention also relates to a system for displaying a video call in a vehicle. Background Art

[0002] DE 10 2018 208 513 A1 discloses a camera monitor system for a motor vehicle having a telephone unit for a video call application. The camera monitor system is configured to display camera images of conversation participants on a monitor of the vehicle while the vehicle is in motion.

[0003] US 2006 / 0183514 A1 discloses a phone accessory having a generator for visual effects. The generator is configured to remove the background of an image, video or other visual information and replace it with a new background.

[0004] US 2018 / 189550 A1 discloses a system for performing video conferencing. The system is configured to remove backgrounds from images of participants in a conversation.

[0005] Driving a vehicle requires the driver to focus on the vehicle's surroundings, in particular the traffic situation around the vehicle. Therefore, in order not to endanger traffic safety, it is important to display the video call to the driver so that the driver is distracted from his or her driving task as little as possible. Summary of the Invention

[0006] The object of the present invention is therefore to provide a method and a system for displaying a video call in a vehicle, by means of which a video call can be displayed in a traffic-safe manner while driving without distracting the vehicle driver from his driving task.

[0007] This object is achieved by a method having the features of claim 1 and by a system having the features of the independent device claim. Advantageous further developments are specified in the dependent claims.

[0008] In the proposed method for displaying a video call in a vehicle, at least one first image is captured, including a remote conversation participant. First image data corresponding to at least the first image is generated. A background in the first image is determined based on the first image data. Second image data is generated based on the first image data and the determined background, the second image data corresponding to a second image including the remote conversation participant, wherein the background is transparently presented in the second image. Furthermore, the second image is displayed to the driver of the vehicle via a head-up display of the vehicle while driving.

[0009] In this method, a video call is conducted between a vehicle driver and a conversation participant who is remote from the vehicle. During the video call, a second image of the remote conversation participant is displayed to the vehicle driver. The second image is displayed on the vehicle's head-up display while driving. The head-up display is positioned in the vehicle driver's field of view, allowing the vehicle driver to view both the head-up display and the surrounding traffic situation almost simultaneously. Additionally, the background depicting the remote conversation participant is removed from the second image—that is, the remote conversation participant is cut out of the second image. This reduces the presentation of the remote conversation participant to the essential content. Thus, the second image is displayed to the vehicle driver via the head-up display, preventing the driver from being distracted from the traffic situation.

[0010] In one embodiment, the second image is presented semi-transparently via the vehicle's head-up display. In this embodiment, the remote conversation participants, which are cut out in the second image, are presented semi-transparently. This means that the remote conversation participants are displayed via the vehicle's head-up display so that objects behind the cut-out remote conversation participants are visible through the cut-out remote conversation participants from the vehicle driver's perspective. In other words, by displaying the cut-out remote conversation participants in the vehicle's head-up display, no area in the driver's field of view is completely obscured. In particular, the driver can thus perceive the traffic situation without interruption. Thus, the second image can still be displayed while maintaining traffic safety.

[0011] In another embodiment, multiple, temporally consecutive driver images are captured, each including at least a facial image of the vehicle driver. Driver image data corresponding to the driver images is generated. Based on the driver image data, it is determined whether the vehicle driver is distracted by the display of a second image. Furthermore, at least one parameter for displaying the second image is set based on whether the vehicle driver is distracted by the display of the second image. For example, based on the driver image data, pupil movement of the vehicle driver can be determined. From this pupil movement, it can be determined whether the vehicle driver is paying attention. A state of attention is characterized by reduced pupil movement. Furthermore, the driver image data can also be used to determine, in particular, the direction of the vehicle driver's gaze. If the vehicle driver's attention is high while viewing the second image, it can be assumed that the vehicle driver is distracted by the display of the second image. In this embodiment, if it is determined that the current presentation of the second image is excessively distracting to the vehicle driver, the display of the second image is adjusted to minimize the driver's distraction. Examples of the at least one parameter for displaying the second image, set to reduce distraction caused by the display of the second image, are described below in conjunction with other embodiments. The display of the second image may also be made more traffic-safe if the presentation of the second image is adjusted to reduce distraction of the vehicle driver caused by the display of the second image.

[0012] In another embodiment, a ratio is determined between a first time that the vehicle driver looks at the displayed second image and a second time that the vehicle driver observes the traffic situation based on the driver image data. In addition, it is determined based on the determined ratio whether the vehicle driver is distracted by the display of the second image. For example, it is possible to determine when the vehicle driver looks at the second image using a gaze direction determined by the vehicle driver's pupil movement. In particular, if the determined ratio is greater than a predetermined ratio, it is determined that the vehicle driver is distracted by the display of the second image. If the vehicle driver looks at the second image too frequently or for too long, it can be assumed that the vehicle driver is distracted from his driving task due to the display of the second image. Therefore, it is possible to very reliably determine whether the vehicle driver is distracted by the display of the second image based on the determined ratio.

[0013] In another embodiment, a driving situation of the vehicle is determined. Driving situation data corresponding to the driving situation is generated. Furthermore, at least one parameter for displaying a second image is set based on the determined driving situation data. The driving situation data specifically includes at least one of the following: lighting conditions, road conditions, vehicle speed, and traffic conditions in the vehicle's surroundings. If the vehicle includes a driver assistance system and / or is configured for autonomous driving, the driving situation data may also include the degree of driver assistance or autonomous driving, for example, according to BASt, VDA, and / or SAE Standard J3016. Certain driving situations (such as heavy traffic, high vehicle speeds, or poor road conditions) require a high level of driver concentration. Therefore, it is advantageous to adjust the display of the second image in such driving situations to reduce driver distraction caused by the display of the second image. Other driving situations, such as certain lighting conditions, may require, for example, increased contrast in the display of the second image to better enable the driver to perceive the second image. This also reduces driver distraction caused by the display of the second image, as the driver must focus less on the second image to perceive it.

[0014] In another embodiment, when the determined driving situation corresponds to a predetermined driving situation, the second image is presented translucently via the vehicle's head-up display. In this embodiment, the parameter for displaying the second image is a binary setting: whether the second image is displayed translucently or opaquely by the vehicle's head-up display. The translucent presentation of the second image has the advantages described above.

[0015] In another embodiment, the second image is presented translucently via the vehicle's head-up display. The at least one parameter for displaying the second image is the transparency with which the second image is presented. For a certain degree of transparency, the higher the transparency, the less distracting the display of the second image is. Thus, by adjusting the transparency, attention-demanding driving situations can be addressed. Conversely, if driving conditions, particularly lighting conditions, make the translucently presented image difficult to discern, the transparency can be reduced. This also reduces distractions caused by the display of the second image.

[0016] In another embodiment, the at least one parameter for displaying the second image is a display position within the field of view of the vehicle driver, at which display position the second image is displayed. Alternatively or additionally, the at least one parameter may also be a size, at which size the second image is displayed. By setting the display position and / or size of the second image in the field of view of the vehicle driver, it is possible to influence how much of the vehicle driver's attention is diverted from the second image to the vehicle driver. A large-sized presentation in the middle of the vehicle driver's field of view requires more attention than a small-sized presentation in the edge of the vehicle driver's field of view. Therefore, by setting the display position and / or size of the second image in the field of view of the vehicle driver, distraction of the vehicle driver caused by the display of the second image can also be reduced.

[0017] In another embodiment, facial recognition is performed based on the first image data to determine the background in the first image. Using facial recognition, the face of the remote session participant can be determined. Regions not corresponding to the face of the remote session participant are assigned to the background. This allows the remote session participant to be easily and reliably cut out from the background to generate the second image.

[0018] In another embodiment, a first image sequence of multiple temporally consecutive first images is acquired, each of which includes an image of a conversation participant remote from the vehicle. The first image data corresponds to the first image sequence. Based on the first image data, a background in the first image sequence is determined. Second image data is generated based on the first image data and the determined background, and the second image data corresponds to an image sequence of multiple temporally consecutive second images, each of which includes an image of a remote conversation participant, with the background being transparently displayed in the second image sequence. Furthermore, the second image sequence is displayed to the driver of the vehicle via a head-up display of the vehicle while driving. In this embodiment, the second image sequence, i.e., a video of the conversation participant remote from the vehicle, is displayed to the driver during a video call. By displaying the second image sequence on the vehicle's head-up display, the driver can view both the second image sequence and the surrounding traffic situation almost simultaneously. Additionally, the background of the second image sequence is removed, i.e., the remote conversation participant is cut out. Therefore, in this embodiment, the video call can be displayed in such a way that the driver is not distracted from his or her driving task.

[0019] In another embodiment, at least one movement in the first image sequence is determined based on the first image data. It is determined whether the movement can be presented without distraction. The second image data are generated such that if it is determined that the movement cannot be presented without distraction, the movement is not presented in the second image sequence. Particularly fast movements, in particular movements at the edge of the field of view, trigger instinctive reactions that can be very distracting. Therefore, in order to achieve the least distracting presentation of the second image sequence possible, certain movements can be filtered out from the second image sequence. As an alternative to determining individual movements, the dynamics of the first image sequence can also be evaluated, for example by determining characteristic values of the dynamics of the first image sequence. In such an embodiment, if the determined characteristic value of the dynamics of the first image sequence exceeds a predetermined threshold value, the dynamics of the second image sequence are reduced relative to the first image sequence based on the determined dynamics. The predetermined threshold value can be determined based on the current driving situation. The reduction in dynamics can be carried out, for example, with the aid of suitable machine learning methods.

[0020] In another embodiment, the second image data is generated by an image processing unit remote from the vehicle and transmitted to the vehicle. For example, the second image data is generated and transmitted to the vehicle at the location of the remote session participant. At the remote session participant's location, the first image is available at high image quality. For transmission, the image data is typically compressed to conserve bandwidth. Thus, at the remote session participant's location, the first image data, which contains more image information, can be processed compared to subsequent transmission to the vehicle. The high image quality allows for simple processing of the first image data into the second image data, resulting in particularly reliable background determination.

[0021] In another embodiment, the first image data is transmitted to an image processing unit of the vehicle. The second image data is generated by the image processing unit of the vehicle. In this embodiment, the second image data is generated by the image processing unit of the vehicle. This has the advantage that the second image data can be generated particularly easily, for example, to be adapted to the current driving situation.

[0022] The present invention also relates to a system for displaying a video call. The system includes an image acquisition unit, which is located remotely from the vehicle and is configured to acquire at least one first image having a remote conversation participant and generate at least first image data corresponding to the first image. Furthermore, the system includes a head-up display of the vehicle, which is located in the vehicle interior and is configured to display visual information to a driver of the vehicle. Furthermore, the system includes an image processing unit, which is configured to receive the first image data, determine a background in the first image based on the first image data, generate second image data based on the first image data and the determined background, the second image data corresponding to a second image having a remote conversation participant, wherein the background is transparently displayed in the second image, and operate the head-up display of the vehicle so that the head-up display displays the second image to the driver of the vehicle while the vehicle is in motion.

[0023] The system has the same advantages as the claimed method. In particular, the system can be improved by using the features of the dependent claims directed to the method. Furthermore, the above method can be improved by using the features described in conjunction with the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram illustrates a system for displaying a video call in a vehicle according to one embodiment; and

[0026] Figure 2 A flow chart illustrating a method for displaying a video call in a vehicle. DETAILED DESCRIPTION

[0027] Figure 1 A schematic diagram shows a system 100 for displaying a video call in a vehicle 102 according to one embodiment.

[0028] System 100 enables a video call between a conversation participant 104 remote from a vehicle 102 and a vehicle driver 106 of vehicle 102. In the illustrated embodiment, remote conversation participant 104 participates in the video call purely by way of example using a smart device 108. Alternatively, remote conversation participant 104 can also participate in the video call from another vehicle or from a stationary PC workstation, for example.

[0029] System 100 includes an image capture unit 110, which in the illustrated embodiment is implemented as a camera of smart device 108. Image capture unit 110 is configured to capture at least one first image of remote session participant 104 and generate image data corresponding to the first image. The first image includes an image of remote session participant 104 (particularly an image of the face of remote session participant 104) and a background. In particular, image capture unit 110 is configured to capture a first image sequence of temporally successive first images, i.e., a video sequence, image stream, or video stream of remote session participant 104.

[0030] exist Figure 1 In the illustrated embodiment, the image processing unit 112 of the system 100 is located at the location of the remote session participant 104. For example only, the image processing unit 112 is shown as part of the smart device 108. In an alternative embodiment, the image processing unit 112 of the system 100 is located in the vehicle 102, for example as part of the control unit 114 of the vehicle 102. The image processing unit 112 is configured to process first image data to generate second image data. In particular, the image processing unit 112 is configured to determine a background in the first image based on the first image data. To this end, the image processing unit 112 performs, for example, a facial recognition method to cut out at least the face of the remote session participant 104 from the background. The image processing unit 112 is further configured to generate second image data based on the determined background and the first image data. The second image data corresponds to a second image of the remote session participant 104, with the background being transparently displayed. When the first image sequence is acquired, the image processing unit 112 generates second image data corresponding to a second image sequence of second images, each of which includes an image of the remote session participant 104 with the background being transparently displayed.

[0031] When the first image sequence is acquired, the image processing unit 112 can also be configured to detect motion in the first image sequence. For example, the image processing unit 112 can compare temporally consecutive first images of the first image sequence with one another. Alternatively or additionally, the image processing unit 112 can also determine characteristic values of the dynamics of the first image sequence. Based on the determined motion in the first image sequence and / or the characteristic values of the dynamics of the first image sequence, the image processing unit 112 assesses whether the motion can be displayed without distraction. In particular, if the motion occurs too quickly or too suddenly, it cannot be displayed without distraction. If the image processing unit 112 has determined that the motion cannot be displayed without distraction, the image processing unit 112 generates second image data in such a way that the motion is filtered out in the second image sequence, i.e., either the motion is not displayed in the second image sequence or the motion is displayed in the second image sequence at a slower speed.

[0032] exist Figure 1 In the illustrated embodiment, the second image data—possibly along with other audio data, image data, and / or video data of the video call—is transmitted to vehicle 102 . The second image data is transmitted, in particular, via a remote data transmission network, such as the Internet. However, other computer networks, such as a LAN, MAN, or WAN network, or a mobile radio network, may also be used to transmit the second image data. If image processing unit 112 is located in vehicle 102 , the first image data is transmitted to vehicle 102 instead of the second image data.

[0033] In the illustrated embodiment, control unit 114 of vehicle 102 is configured to receive second image data or first image data via remote data transmission network 116. Furthermore, control unit 114 is configured to operate head-up display 118 of vehicle 102 so that head-up display 118 displays the second image or a second image sequence to vehicle driver 106 while vehicle 102 is in motion, i.e., while vehicle 102 is driving. Hereinafter, displaying the second image also refers to displaying the second image sequence as a possible alternative. To ensure traffic safety, control unit 114 may be configured to set at least one parameter for displaying the second image in order to adjust the display of the second image. For example, one of the parameters for displaying the second image may be whether the second image is to be displayed translucently or opaquely on head-up display 118. Furthermore, another parameter for displaying the second image may be the transparency with which the second image is displayed on head-up display 118. Furthermore, one of the parameters for displaying the second image may be the display position and / or size of the second image. For example, the second image may be displayed on head-up display 118 in a large and central manner, or in a small and peripheral position where it distracts vehicle driver 106 less from the current traffic situation.

[0034] The parameters for displaying the second image can be set, in particular, based on the driving situation. The driving situation is, in particular, the current level of driver assistance or automated driving of vehicle 102. Furthermore, the driving situation may also include traffic conditions or road conditions, which can each be detected using an external camera of vehicle 102. The driving situation may also include movement parameters of vehicle 102, such as the current speed of vehicle 102 or the distance to the vehicle ahead. In particular, the display of the second image is adjusted when a predetermined driving situation occurs. Predefined driving situations include, for example, situations in which the driver's attention is diverted, i.e., situations in which the driver must pay more or less attention than in previous driving situations. In situations in which the driver must pay more attention, such as when the distance to the vehicle ahead is very small or vehicle 102 is traveling on a highway, the display of the second image can be adjusted so that the display of the second image distracts driver 106 as little as possible.

[0035] The system 100 further includes an occupant detection unit 120, which Figure 1 The second image is exemplarily configured as an interior camera of vehicle 102. Occupant detection unit 120 is configured to capture a plurality of temporally sequential driver images including images of at least the face of vehicle driver 106 and to generate driver image data corresponding to the driver images. Control unit 114 is configured to receive the driver image data and, based on the driver image data, determine whether vehicle driver 106 is distracted by the display of the second image. Based on the determination of whether vehicle driver 106 is distracted by the display of the second image, one of the parameters for displaying the second image can be adjusted.

[0036] Figure 2 A flow chart illustrating a method for displaying a video call in vehicle 102 is shown.

[0037] The method begins in step S200. In step S202, image acquisition unit 110 acquires at least a first image having an image of conversation participant 104 remote from vehicle 102. In particular, image acquisition unit 110 acquires a plurality of temporally consecutive first images in this step, each of which is part of a first image sequence. Image acquisition unit 110 also generates first image data corresponding to the first image or first image sequence in this step. In optional step S204, image acquisition unit 110 transmits the first image data to vehicle 102. In step S206, image processing unit 112 determines the background in the first image or first image sequence based on the first image data. In step S208, image processing unit 112 generates second image data based on the first image data and the determined background. The second image data corresponds to a second image or a second image sequence having an image of remote conversation participant 104, wherein the background is transparent in the second image or second image sequence. In optional step S210, image processing unit 112 transmits the second image data to vehicle 102. During the execution of the method, either step S204 or step S210 is executed depending on the location of image processing unit 112. In step S212, head-up display 118 of vehicle 102 displays a second image or a second image sequence to vehicle driver 106 during driving. In step S214, the method ends.

[0038] With the help of Figure 1 and Figure 2 In the depicted embodiment, at least an image acquisition unit 110, an image processing unit 112, and a head-up display 118 form a system 100 for displaying a video call in a vehicle 102. Figure 1 and Figure 2 The elements and features shown in FIG and mentioned in the above description may be part of a system 100 for displaying a video call in a vehicle 102 . The method steps described with the aid of the system 100 may also be part of the claimed method.

[0039] Reference Signs List

[0040] 100 systems

[0041] 102 vehicles

[0042] 104 session participants

[0043] 106 vehicle drivers

[0044] 108 smart devices

[0045] 110 image acquisition unit

[0046] 112 image processing unit

[0047] 114 control unit

[0048] 116 Data Remote Transmission Network

[0049] 118 head-up display

[0050] 120 occupant detection unit

Claims

1. A method for displaying a video call in a vehicle (102), wherein: Acquiring at least one first image having an image of a conversation participant (104) remote from the vehicle (102), and generating at least first image data corresponding to the first image; determining a background in the first image based on the first image data; generating second image data based on the first image data and the determined background, the second image data corresponding to a second image having an image of the remote session participant (104), wherein the background is transparently presented in the second image; and The second image is displayed to a vehicle driver (106) of the vehicle (102) via a head-up display (118) of the vehicle (102) during driving.

2. The method according to claim 1, wherein The second image is presented semi-transparently via a heads-up display (118) of the vehicle (102).

3. The method according to claim 1 or 2, wherein: Acquiring a plurality of temporally successive driver images, the driver images comprising an image of at least the face of the vehicle driver (106), and generating driver image data corresponding to the driver images; determining, based on the driver image data, whether the vehicle driver (106) is distracted by the display of the second image; and At least one parameter for displaying the second image is set according to whether the vehicle driver (106) is distracted by the display of the second image.

4. The method according to claim 3, wherein: A ratio between a first time that the vehicle driver (106) is looking at the displayed second image and a second time that the vehicle driver (106) is observing a traffic situation is determined based on the driver image data; and whether the vehicle driver (106) is distracted by the display of the second image is determined based on the determined ratio.

5. The method according to any one of claims 1 to 4, wherein A driving situation of the vehicle (102) is determined, driving situation data corresponding to the driving situation is generated, and at least one parameter for displaying the second image is set based on the driving situation data.

6. The method according to claim 5, wherein: The second image is presented semi-transparently via a head-up display (118) of the vehicle (102) when the determined driving situation corresponds to a predetermined driving situation.

7. The method according to any one of claims 3 to 6, wherein The second image is presented semi-transparently via a head-up display (118) of the vehicle (102), and the at least one parameter for displaying the second image is a transparency with which the second image is displayed.

8. The method according to any one of claims 3 to 7, wherein The at least one parameter for displaying the second image is a display position within the field of view of the vehicle driver (106), at which display position the second image is displayed, and / or a size at which the second image is displayed.

9. A method according to any one of the preceding claims, wherein Facial recognition is performed based on the first image data to determine a background in the first image.

10. The method according to any one of the preceding claims, in, Acquiring a first image sequence of a plurality of temporally successive first images, each of which includes an image of a conversation participant (104) remote from the vehicle (102); wherein the first image data corresponds to the first image sequence; wherein determining a background in a first image sequence based on the first image data; wherein the second image data are generated based on the first image data and the determined background, the second image data corresponding to an image sequence of a plurality of temporally consecutive second images, each of which has an image of one of the remote session participants (104), the background being presented transparently in the second image sequence; and The second image sequence is displayed to the vehicle driver (106) of the vehicle (102) via a head-up display (118) of the vehicle (102) during driving.

11. The method according to claim 10, wherein: determining at least one motion in a first sequence of images based on the first image data; determining whether the motion can be presented without distraction; and The second image data is generated such that the motion is not presented in the second image sequence if it has been determined that the motion cannot be presented without being distracting.

12. A method according to any one of the preceding claims, wherein The second image data is generated by an image processing unit (112) remote from the vehicle (102) and transmitted to the vehicle (102).

13. The method according to any one of claims 1 to 11, wherein The first image data is transmitted to an image processing unit (112) of the vehicle (102), and the second image data is generated by the image processing unit (112) of the vehicle (102).

14. A system (100) for displaying a video call in a vehicle (102), the system comprising: an image acquisition unit (110) arranged at a location remote from the vehicle (102) and configured to acquire at least one first image having an image of a conversation participant (104) remote from the vehicle (102) and to generate at least first image data corresponding to the first image; A head-up display (118) of a vehicle (102), the head-up display being arranged in the vehicle interior and being designed to display visual information to a vehicle driver (106) of the vehicle (102); and An image processing unit (112) is configured to: receive first image data, determine a background in the first image based on the first image data, generate second image data based on the first image data and the determined background, the second image data corresponding to a second image having an image of the remote session participant (104), wherein the background is presented transparently in the second image, and operate the head-up display (118) of the vehicle (102) so that the head-up display (118) displays the second image to the vehicle driver (106) of the vehicle (102) during driving.

Images