Systems and methods for monitoring digital mirror displays of a vehicle

Abstract

An external image is captured by a first camera system of the vehicle. This external image shows a perspective from the vehicle's mirrors of an external environment outside the vehicle. An internal image is captured by a second camera system of the vehicle. This internal image shows an interior environment inside the vehicle. The internal image is a representation of the digital mirror display inside the vehicle. The digital mirror display is generated based on the external image. A determination is made as to whether the external image matches the image on the digital mirror display. Based on this determination, an image mismatch notification is generated for display on a vehicle display device.

Description

[0001] The technical field generally concerns vehicles and in particular systems and procedures for monitoring digital mirror displays of a vehicle.

[0002] Many vehicles have one or more digital mirror display devices. Examples of digital mirror display devices include a digital side-view mirror display device and a digital side-view mirror display device. External vehicle cameras capture images of the area outside the vehicle from the perspective of a left side-view mirror and a right side-view mirror. The images from the perspective of the left side-view mirror are displayed on the digital side-view mirror display device as digital side-view mirror displays, and the images from the perspective of the right side-view mirror are also displayed on the digital side-view mirror display device as digital side-view mirror displays.

[0003] DE 10 2010 023 891 A1 describes a method for detecting a faulty display of image data on a display unit. To enable timely and particularly reliable detection of a faulty display of image data on the display unit, the method proceeds as follows: the image data to be displayed is transmitted to the display unit; test data is acquired by electronically capturing at least a portion of the image displayed on the display unit; and a faulty display of the image data on the display unit is detected based on an electronic evaluation of at least a portion of the acquired test data.

[0004] DE 10 2017 220 571 A1 describes an arrangement comprising an image signal generation device suitable for generating an image-defining image signal, a display device for displaying the image, a connecting line which is connected to the image signal generation device at one end near the image signal generation device and to the display device at the other end, i.e., the end furthest from the image signal generation device, and which is suitable for transmitting the image signal from the image signal generation device to the display device, and an information recovery device for generating a test signal that enables verification of the function of the display device.It is intended that the information recovery device is connected to the end of the said connecting line furthest from the image signal generation device and transmits the test signal via this connecting line to the end of the line closest to the image signal generation device.

[0005] Accordingly, the object of the present invention is to provide systems and methods for monitoring digital mirror displays of a vehicle.

[0006] The problem is solved by the subject matter of the independent claims.

[0007] Other desirable features and characteristics will become apparent from the following detailed description and the attached claims in conjunction with the attached drawings and the preceding technical field and background.

[0008] A digital mirror display monitoring system for a vehicle according to the invention has at least one processor and at least one memory which is communicatively coupled to the at least one processor.The at least one memory contains instructions which, when executed by the at least one processor, cause the at least one processor to: receive an external image captured by a first camera system of the vehicle, wherein the external image is a perspective of a vehicle mirror view of an external environment outside the vehicle; receive an internal image captured by a second camera system of the vehicle, wherein the internal image is an image of an interior environment inside the vehicle, and wherein the internal image is an image of a digital mirror display inside the vehicle, and the digital mirror display is generated based on the external image; determine whether the external image matches the image of the digital mirror display; and generate an image mismatch notification for display on a vehicle display device of the vehicle based on the determination.

[0009] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor to: crop the internal image to create a region of interest (ROI) of the internal image that corresponds to the image of the digital mirror display; implement distortion correction of the ROI of the internal image to produce a corrected image of the digital mirror display; and determine whether the external image matches the image of the digital mirror display based on a comparison of the external image with the corrected image of the digital mirror display.

[0010] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor to use a trained deep neural network (DNN) model to: generate a first set of features based on the external image; generate a first vector based on the first set of features; generate a second set of features based on the corrected image of the digital mirror display; generate a second vector based on the second set of features; and determine whether the external image matches the corrected image of the digital mirror display based on a comparison of the first vector with the second vector.

[0011] In at least one embodiment, the DNN is a classifier DNN.

[0012] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor, upon determining that the external image does not match the image of the digital mirror display, to identify an image mismatch cause associated with the digital mirror display, wherein the image mismatch cause is one of a freezing of the digital mirror display, a color loss of the digital mirror display, and a black pixel detection in the digital mirror display.

[0013] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor, upon determining that the external image does not match the image of the digital mirror display, to transfer the external image to an infotainment display device of the vehicle for display on the infotainment display device.

[0014] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor to: restart a digital mirror display device configured to display the digital mirror display; determine whether restarting the digital mirror display device resulted in the external image matching the image of the digital mirror display; and issue a command to the infotainment display device to stop displaying the external image on the vehicle's infotainment display device based on the determination.

[0015] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor, upon determining that restarting the digital mirror display device did not result in the external image matching the image of the digital mirror display, to issue a command to switch off the digital mirror display device.

[0016] In at least one embodiment, the digital mirror display device is a digital side view mirror display device of the vehicle.

[0017] In at least one embodiment, the at least one memory further comprises instructions which, when executed by the at least one processor, cause the at least one processor to: modify at least one color in the external image according to a color-insufficient driver setting to produce a modified external image; issue a command to a digital mirror display device to display the digital mirror display based on the modified external image; determine whether the modified external image matches the image of the digital mirror display; and generate the mismatch notification of the digital mirror display device for display on the vehicle display device based on the determination.

[0018] A method according to the invention for monitoring a digital mirror display comprises the following: receiving, at a controller, an external image captured by a first camera system of the vehicle, wherein the external image is a perspective of a vehicle mirror view of an external environment outside the vehicle; receiving, at the controller, an internal image captured by a second camera system of the vehicle, wherein the internal image is an interior environment inside the vehicle, and wherein the internal image is an image of a digital mirror display inside the vehicle, and the digital mirror display is generated based on the external image; determining, by the controller, whether the external image matches the image of the digital mirror display; and generating, by the controller, an image mismatch notification for display on a vehicle display device of the vehicle based on the determination.

[0019] In at least one embodiment, the method further comprises: cropping, by control, the internal image to generate a region of interest (ROI) of the internal image that corresponds to the image of the digital mirror display; implementing, by control, a distortion correction of the ROI of the internal image to generate a corrected image of the digital mirror display; and determining, by control, whether the external image matches the image of the digital mirror display based on a comparison of the external image with the corrected image of the digital mirror display.

[0020] In at least one embodiment, the method further comprises: using, by controlling, a trained deep neural network (DNN) model to: generate a first set of features based on the external image; generate a first vector based on the first set of features; generate a second set of features based on the corrected image of the digital mirror display; generate a second vector based on the second set of features; and determine whether the external image matches the corrected image of the digital mirror display based on a comparison of the first vector with the second vector.

[0021] In at least one embodiment, the method further includes, in determining that the external image does not match the image of the digital mirror display, the identification, by the control, of an image mismatch cause that is associated with the digital mirror display, wherein the image mismatch cause is a freezing of the digital mirror display, a color loss of the digital mirror display and a black pixel detection in the digital mirror display.

[0022] In at least one embodiment, the method further includes, in the case of a determination that the external image does not match the image of the digital mirror display, the transmission, by means of the control, of the external image to an infotainment display device of the vehicle for display on the infotainment display device.

[0023] In at least one embodiment, the method further comprises: restarting, by the controller, a digital mirror display device configured to display the digital mirror image; determining, by the controller, whether restarting the digital mirror display device resulted in the external image matching the image of the digital mirror display; and issuing a command, by the controller, to the infotainment display device to stop displaying the external image on the vehicle's infotainment display device based on the determination.

[0024] In at least one embodiment, the method further includes, in the case of a finding that restarting the digital mirror display device did not result in the external image matching the image of the digital mirror display, the output, by the controller, of a command to switch off the digital mirror display device.

[0025] In at least one embodiment, the digital mirror display device is a digital side view mirror display device of the vehicle.

[0026] In at least one embodiment, the method further comprises: modifying, by the controller, at least one color in the external image according to a color-insufficient driver setting to produce a modified external image; issuing a command, by the controller, to a digital mirror display device to display the digital mirror display based on the modified external image; determining, by the controller, whether the modified external image matches the image of the digital mirror display; and generating, by the controller, the mismatch notification of the digital mirror display device for display on the vehicle display device based on the determination.

[0027] In one application example, a vehicle having a digital mirror display monitoring system according to the invention has at least one processor and at least one memory that is communicatively coupled to the at least one processor.The at least one memory contains instructions which, when executed by the at least one processor, cause the at least one processor to: receive an external image captured by a first camera system of the vehicle, wherein the external image is a perspective of a vehicle mirror view of an external environment outside the vehicle; receive an internal image captured by a second camera system of the vehicle, wherein the internal image is an image of an interior environment inside the vehicle, and wherein the internal image is an image of a digital mirror display inside the vehicle, and the digital mirror display is generated based on the external image; determine whether the external image matches the image of the digital mirror display; and generate an image mismatch notification for display on a vehicle display device of the vehicle based on the determination. Fig. 1 is a functional block diagram of a vehicle that has a digital mirror display monitoring system according to at least one embodiment; Fig. 2 a functional block diagram of a controller which has a digital mirror display monitoring system according to at least one embodiment; Fig. 3 a schematic representation of a top view of an exemplary vehicle configured to implement a digital mirror display monitoring system according to at least one embodiment; and Fig. 4 is a flowchart representation of an exemplary method for monitoring a digital mirror display according to at least one embodiment.

[0028] With reference to Fig. Figure 1 shows a functional block diagram of a vehicle 10 having a digital mirror display monitoring system 100 according to at least one embodiment. The vehicle 10 generally has a chassis 12, a body 14, front wheels 16, and rear wheels 18. While the vehicle 10 is shown as a passenger car in the illustrated embodiment, the vehicle 10 can be other types of vehicles, including trucks, sport utility vehicles (SUVs), and recreational vehicles (RVs).

[0029] In various embodiments, the body 14 is arranged on the chassis 12 and essentially encloses components of the vehicle 10. The body 14 and the chassis 12 can together form a frame. The wheels 16, 18 are rotatably coupled to the chassis 12 near their respective corners of the body 14.

[0030] In various embodiments, the vehicle 10 is an autonomous or semi-autonomous vehicle that is automatically controlled to transport passengers and / or cargo from one place to another. For example, in one exemplary embodiment, the vehicle 10 is a so-called Level Two, Level Three, Level Four, or Level Five automation system. Level Two automation means that the vehicle assists the driver with various driving tasks under driver supervision. Level Three automation means that, under certain circumstances, the vehicle can take over all driving functions. All major functions are automated, including braking, steering, and accelerating. At this level, the driver can disengage completely until the vehicle instructs the driver otherwise.A Level Four system indicates "high automation," which refers to the driving-mode-specific performance of an automated driving system in all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request for intervention. A Level Five system indicates "full automation," which refers to the full-time performance of an automated driving system in all aspects of the dynamic driving task under all road and environmental conditions that can be handled by a human driver.

[0031] As shown, the vehicle 10 generally comprises a drive system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. The controller 34 is configured to implement an automated driving system (ADS). The drive system 20 is configured to generate power to propel the vehicle. The drive system 20 comprises an internal combustion engine (ICE). In various embodiments, the drive system 20 may also include an electric machine, such as a traction motor, a fuel cell propulsion system, and / or any other type of drive configuration. The transmission system 22 is configured to transmit power from the drive system 20 to the vehicle wheels 16, 18 according to selectable speed ratios.According to various embodiments, the transmission system 22 can comprise a gear-ratio automatic transmission, a continuously variable transmission, or another suitable transmission. The braking system 26 is configured to provide a braking torque to the vehicle wheels 16 and 18. The braking system 26 can, in various embodiments, comprise friction brakes, brake-by-wire, a regenerative braking system such as an electric motor, and / or other suitable braking systems.

[0032] The steering system 24 is configured to influence the position of the vehicle wheels 16. Although, for illustrative purposes, it is shown in some embodiments considered within the scope of this description as having a steering wheel and steering column, the steering system 24 may not have a steering wheel and / or steering column. The steering system 24 has a steering column coupled, for example, by a rack and pinion or other mechanism (not shown), to an axle 50 associated with the front wheels 16. Alternatively, the steering system 24 may have a steer-by-wire system comprising actuators associated with each of the front wheels 16.

[0033] The sensor system 28 comprises one or more detection devices 40a-40n that detect observable conditions of the external environment and / or the internal environment of the vehicle 10. The detection devices 40a-40n may include, but are not limited to, radars, lidar, global positioning systems, optical cameras, thermal imaging cameras, ultrasonic sensors, a steering wheel sensor, and / or other sensors.

[0034] The vehicle dynamics sensors provide vehicle dynamics data, including longitudinal speed, yaw rate, lateral acceleration, longitudinal acceleration, etc. The vehicle dynamics sensors may include wheel sensors that measure information relating to one or more wheels of the vehicle 10. In one embodiment, the wheel sensors comprise wheel speed sensors coupled to each of the wheels 16, 18 of the vehicle 10. Furthermore, the vehicle dynamics sensors may include one or more accelerometers (provided as part of an inertial measurement unit, IMU) that measure information relating to the acceleration of the vehicle 10. In various embodiments, the accelerometers measure one or more acceleration values ​​for the vehicle 10, including lateral and longitudinal acceleration and yaw rate.In at least one embodiment, the vehicle dynamics sensors provide vehicle position and vehicle movement data.

[0035] The actuator system 30 comprises one or more actuator devices 42a-42n that control one or more vehicle features, such as, but are not limited to, one or more vehicle wheels 16, 18, the drive system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may further include, but are not limited to, interior and / or exterior vehicle features, such as doors, a trunk, and interior features such as air conditioning, music, lighting, etc. (not numbered).

[0036] The communication system 36 is configured to wirelessly communicate information to and from other entities 48, such as other vehicles (vehicle-to-vehicle communication, "V2V" communication), infrastructure (vehicle-to-infrastructure communication, "V2I" communication), remote systems, and / or personal devices, but is not limited to these. In one exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate over a wireless local area network (WLAN) using IEEE 802.11 standards or using cellular data communication. However, additional or alternative communication methods, such as a dedicated short-range communications (DSRC) channel, are also considered within the scope of this description.DSRC channels refer to one-way or two-way short-range to medium-range wireless communication channels specifically designed for automotive use, and to a corresponding set of protocols and standards.

[0037] The data storage device 32 stores data for use in the ADS of the vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps can be predefined by and received from a remote system. For example, the defined maps can be compiled by the remote system and communicated to the vehicle 10 (wirelessly and / or via a wired connection) and stored in the data storage device 32. It is understood that the data storage device 32 can be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.

[0038] The controller 34 comprises at least one processor 44 and a computer-readable memory device or computer-readable storage medium 46. The processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 34, a microprocessor-based semiconductor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer-readable memory device or computer-readable storage medium 46 can, for example, include volatile and non-volatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM).KAM is a persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is powered off. The computer-readable memory device or computer-readable storage medium 46 can be implemented using any number of known memory devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combined memory devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the vehicle 10. In at least one embodiment, the computer-readable memory device 46 is at least one memory configured to store the digital mirror display monitoring system 100.

[0039] The instructions can have one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by the processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, procedures, and / or algorithms to automatically control the components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control the components of the vehicle 10 based on the logic, calculations, procedures, and / or algorithms. Although only one controller 34 in Fig. As shown in Figure 1, embodiments of the vehicle 10 can have any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and that interact to process the sensor signals, execute logic, calculations, procedures and / or algorithms, and generate control signals to automatically control features of the vehicle 10. In various embodiments, the controller(s) 34 are configured to implement ADS.

[0040] With reference to Fig. Figure 2 shows a functional block diagram of a controller 34, which includes a digital mirror display monitoring system 100 according to at least one embodiment. The controller 34 includes at least one processor 44 and at least one memory 46. The at least one processor 44 is a programmable device that includes one or more instructions stored in or assigned to the at least one memory 46. The at least one memory 46 contains instructions that the at least one processor 44 is configured to execute. The at least one memory 46 includes an embodiment of the digital mirror display monitoring system 100.

[0041] The controller 34 is configured to communicate with one or more external cameras 200, one or more digital mirror displays 202, one or more internal cameras 204, a vehicle display 206, and an infotainment display 208. Each external camera 200 is configured to communicate with an associated digital mirror display 202. In at least one embodiment, the external cameras 200 are located outside the vehicle 10. In at least one embodiment, the external cameras 200 are located inside the vehicle 10.

[0042] Each digital mirror display device 202 is configured to display digital mirror displays representing an associated vehicle mirror view. Each external camera 200 is configured to capture external images of an environment outside the vehicle 10 from the perspective of a vehicle mirror view. Each external camera 200 is configured to transmit the captured external images to an associated digital mirror display device 202. Each digital mirror display device 202 is configured to display the received external images, representing the environment outside the vehicle 10, from the perspective of the associated vehicle mirror view as digital mirror displays.

[0043] The internal camera(s) 206 is / are configured to capture internal images of an interior environment within the vehicle 10. The internal images include images from the digital mirror displays shown on the digital mirror display devices 202.

[0044] The digital mirror display surveillance system 100 is configured to receive the external images captured by the external cameras 200 and the internal images captured by the internal camera(s) 206.

[0045] The digital mirror display monitoring system 100 is configured to determine whether the external images captured by an external camera 200 match the images of the digital mirror displays on a digital mirror display device 202 associated with the external camera 200. If the digital mirror display monitoring system 100 determines that the external images captured by the external camera 200 do not match the images of the digital mirror displays on the digital mirror display device 202 associated with the external camera 200, the digital mirror display monitoring system 100 is configured to generate an image mismatch notification for display on a vehicle display device 206 of the vehicle 10.A mismatch between an external image captured by an external camera 200 associated with a digital mirror display device 202 and an image of a digital mirror display shown on the digital mirror display device 202 in an internal image captured by an internal camera 204 indicates a display problem of the digital mirror display device 202. In at least one embodiment, the image mismatch notification identifies the specific digital mirror display device 202 associated with the image mismatch. In at least one embodiment, the digital mirror display monitoring system 100 is configured to transmit the external images associated with the digital mirror display device 202 experiencing the image mismatch to an infotainment display device 208 of the vehicle 10 for display on the infotainment display device 208.

[0046] The control unit 34 may include additional components that facilitate the operation of the digital mirror display monitoring system 100. The operation of the digital mirror display monitoring system 100 is described in more detail below.

[0047] With reference to Fig. Figure 3 shows a schematic top view of an exemplary vehicle 10 configured to implement a digital mirror display monitoring system 100 according to at least one embodiment. The vehicle 10 has a first external camera 200a, a second external camera 200b, a first digital mirror display device 202a, a second digital mirror display device 202b, an internal camera 204, a vehicle display device 206, and an infotainment display device 208. The first digital mirror display device 202a is a digital side-view mirror display device, and the second digital mirror display device 202b is also a digital side-view mirror display device. The first external camera 200a is associated with the first digital mirror display device 202a, and the second external camera 200b is associated with the second digital mirror display device 202b.

[0048] The first external camera 200a, associated with the first digital mirror display device 202a (the digital side-view mirror display device), is configured to capture external images of the environment outside the vehicle 10 from the perspective of a left side-view mirror of the vehicle 10. The external images of the environment outside the vehicle 10, from the perspective of the left side-view mirror of the vehicle 10, are displayed by the first digital mirror display device 202a (the digital side-view mirror display device) as real-time digital side-view mirror displays.The digital mirror display monitoring system 100 is configured to receive in real time the external images of the environment outside the vehicle 10 from the perspective of the left side view mirror of the vehicle 10 from the first external camera 200a, which is associated with the first digital mirror display device 202a (the digital side view mirror display device).

[0049] The second external camera 200b, which is associated with the second digital mirror display device 202b (the digital side-view mirror display device), is configured to capture external images of the environment outside the vehicle 10 from the perspective of a right side-view mirror of the vehicle 10. The external images of the environment outside the vehicle 10, from the perspective of the right side-view mirror of the vehicle 10, are displayed by the second digital mirror display device 202b (the digital side-view mirror display device) as real-time digital side-view mirror displays.The digital mirror display monitoring system 100 is configured to receive in real time the external images of the environment outside the vehicle 10 from the perspective of the right side view mirror of the vehicle 10 from the second external camera 200b, which is assigned to the second digital mirror display device 202b (the digital side view mirror display device).

[0050] The internal camera 204 is configured to capture internal images of an interior environment within the vehicle 10. The internal images include images from the digital mirror displays shown on the first digital mirror display device 202a (the digital side-view mirror display device) and the second digital mirror display device 202b (the digital side-view mirror display device). The digital mirror display monitoring system 100 is configured to receive the internal images captured by the internal camera 204 in real time. In at least one embodiment, the internal images captured by the internal camera 204 are field-of-view (FoV) images of the interior environment.

[0051] In at least one embodiment, the digital mirror display monitoring system 100 is configured to crop each internal image to generate a first area of ​​interest (ROI) and a second ROI. The first ROI is an image of the digital side-view mirror display shown on the first digital mirror display device 202a (the digital side-view mirror display device). The second ROI is an image of the digital side-view mirror display shown on the second digital mirror display device 202b (the digital side-view mirror display device).

[0052] In at least one embodiment, the digital mirror display monitoring system 100 is configured to implement distortion correction of the first ROI to produce a corrected image of the digital side-view mirror display that is shown on the first digital mirror display device 202a (the digital side-view mirror display device). The digital mirror display monitoring system 100 is configured to implement distortion correction of the second ROI to produce a corrected image of the digital side-view mirror display that is shown on the second digital mirror display device 202b (the digital side-view mirror display device).

[0053] The digital mirror display monitoring system 100 is configured to determine whether the external images received from the first external camera 200a match the corrected images of the digital side-view mirror displays shown in real time on the first digital mirror display device 202a (the digital side-view mirror display device). The digital mirror display monitoring system 100 is also configured to determine whether the external images received from the second external camera 200b match the corrected images of the digital side-view mirror displays shown in real time on the second digital mirror display device 202b (the digital side-view mirror display device).

[0054] In at least one embodiment, the digital mirror display monitoring system 100 is configured to use a trained deep neural network (DNN) model to compare the external images received from the first external camera 200a with the corrected images of the digital side-view mirror displays shown on the first digital mirror display device 202a (the digital side-view mirror display device), and the external images received from the second external camera 200b with the corrected images of the digital side-view mirror displays shown on the second digital mirror display device 202b (the digital side-view mirror display device) in real time. In at least one embodiment, the trained DNN model is a trained DNN classifier model.

[0055] If the digital mirror display monitoring system 100 determines that an external image captured by the first external camera 200a does not match a corrected image of a digital side-view mirror display shown on the first digital mirror display device 202a (the digital side-view mirror display device), the digital mirror display monitoring system 100 is configured to generate an image mismatch notification for display on the vehicle display device 206, indicating that there is a display problem associated with the first digital mirror display device 202a (the digital side-view mirror display device). The image mismatch notification identifies the first digital mirror display device 202a (the digital side-view mirror display device) as the one associated with the image mismatch.The digital mirror display monitoring system 100 is configured to transmit the external images associated with the first digital mirror display device 202a (the digital side-view mirror display device) to the infotainment display device 208 for display on the infotainment display device 208. In at least one embodiment, the digital mirror display monitoring system 100 is configured to identify a cause associated with the image mismatch associated with the first digital mirror display device 202a (the digital side-view mirror display device). Examples of causes include, but are not limited to, digital mirror display freezing, digital mirror display color fading, black pixel detection, and a time delay between the image captured by the external camera and the image captured by the internal camera in the digital mirror display.

[0056] The digital mirror display monitoring system 100 is configured to restart the first digital mirror display device 202a (the digital side-view mirror display device) and to determine whether restarting the first digital mirror display device 202a (the digital side-view mirror display device) resulted in an external image captured by the first external camera 200a matching a corrected image displayed on a digital side-view mirror display shown on the first digital mirror display device 202a (the digital side-view mirror display device).

[0057] When the digital mirror display monitoring system 100 determines that restarting the first digital mirror display device 202a (the digital side-view mirror display device) resulted in an external image captured by the first external camera 200a matching a corrected image displayed on a digital side-view mirror display on the first digital mirror display device 202a (the digital side-view mirror display device), the digital mirror display monitoring system 100 issues a command to the infotainment display device 208 to stop displaying the external images associated with the first digital mirror display device 202a (the digital side-view mirror display device) on the infotainment display device 208.In at least one embodiment, the infotainment display device 208 displays a notification about the problem resolution in the event of a successful restart.

[0058] If the digital mirror display monitoring system 100 determines that restarting the first digital mirror display device 202a (the digital side-view mirror display device) did not result in an external image captured by the first external camera 200a matching a corrected image displayed on the first digital mirror display device 202a (the digital side-view mirror display device), the digital mirror display monitoring system 100 issues a command to turn off the first digital mirror display device 202a (the digital side-view mirror display device). The first digital mirror display device 202a turns off in response to the command to turn off the first digital mirror display device 202a (the digital side-view mirror display device).In at least one embodiment, in the event of an unsuccessful restart, the infotainment display device 208 indicates a reversing monitoring system that requires a maintenance notification.

[0059] If the digital mirror display monitoring system 100 determines that an external image captured by the second external camera 200b does not match a corrected image of a digital side-view mirror display shown on the second digital mirror display device 202b (the digital side-view mirror display device), the digital mirror display monitoring system 100 is configured to generate an image mismatch notification for display on the vehicle display device 206, indicating that there is a display problem associated with the second digital mirror display device 202b (the digital side-view mirror display device). The image mismatch notification identifies the second digital mirror display device 202b (the digital side-view mirror display device) as the one associated with the image mismatch.The digital mirror display monitoring system 100 is configured to transmit the external images assigned to the second digital mirror display device 202b (the digital side view mirror display device) to the infotainment display device 208 for display on the infotainment display device 208.

[0060] The digital mirror display monitoring system 100 is configured to restart the second digital mirror display device 202b (the digital side-view mirror display device) and to determine whether restarting the second digital mirror display device 202b (the digital side-view mirror display device) resulted in an external image captured by the second external camera 200b (the digital side-view mirror display device) matching a corrected image displayed on a digital side-view mirror display shown on the second digital mirror display device 202b (the digital side-view mirror display device).

[0061] When the digital mirror display monitoring system 100 determines that restarting the second digital mirror display device 202b (the digital side-view mirror display device) resulted in an external image captured by the second external camera 200b matching a corrected image of a digital side-view mirror display shown on the second digital mirror display device 202b (the digital side-view mirror display device), the digital mirror display monitoring system 100 issues a command to the infotainment display device 208 to stop displaying the external images associated with the second digital mirror display device 202b (the digital side-view mirror display device) on the infotainment display device 208.In at least one embodiment, the infotainment display device 208 displays a notification about the problem resolution in the event of a successful restart.

[0062] If the digital mirror display monitoring system 100 determines that restarting the second digital mirror display device 202b (the digital side-view mirror display device) did not result in an external image captured by the second external camera 200b matching a corrected image displayed on a digital side-view mirror display on the second digital mirror display device 202b (the digital side-view mirror display device), the digital mirror display monitoring system 100 issues a command to turn off the second digital mirror display device 202b (the digital side-view mirror display device).The second digital mirror display device 202b (the digital side-view mirror display device) switches off in response to the command to switch off the second digital mirror display device 202b (the digital side-view mirror display device). In at least one embodiment, in the event of an unsuccessful restart, the infotainment display device 208 displays a reversing monitoring system that requires a maintenance notification.

[0063] While the operation of the digital mirror display monitoring system 100 has been described with reference to a digital side-view mirror display device and a digital side-view mirror display device, the digital mirror display monitoring system 100 can be used to identify image mismatches between other types of digital mirror display devices 202.

[0064] With reference to Fig. Figure 4 shows a flowchart representation of an exemplary method 400 for monitoring a digital mirror display in a vehicle 10 according to at least one embodiment. The method 400 is described with reference to an exemplary implementation of an embodiment of a digital mirror display monitoring system 100. As can be seen from the description, the order of operation within the method 400 is not limited to sequential execution, as shown in Figure 4. Fig. 4 illustrates, but can be performed in one or more different sequences as required and according to the present description.

[0065] In device 402, an external image of an environment outside a vehicle 10, captured by an external camera 200 and viewed from the perspective of the vehicle's rearview mirror, is received in real time by the digital mirror display monitoring system 100. The external camera 200 is associated with a digital mirror display device 202. The digital mirror display device 202 displays the external image captured by the external camera 200 as a digital mirror display.

[0066] At 404, an internal image of the interior environment within the vehicle 10, captured by an internal camera 204, is received in real time at the digital mirror display monitoring system 100. The internal image includes a picture of the digital mirror display shown on the digital mirror display device 202.

[0067] At 406, the internal image from the digital mirror display monitoring system 100 is cropped to generate a region of interest (ROI) of the internal image. The ROI of the internal image is the image of the digital mirror display shown on the digital mirror display device 202. At 408, a distortion correction of the ROI is implemented by the digital mirror display monitoring system 100 to generate a corrected image of the digital mirror display.

[0068] At 410, the digital mirror display monitoring system 100 determines whether the external image matches the corrected image of the digital mirror display. The digital mirror display monitoring system 100 uses a trained deep neural network (DNN) model to compare the external image with the corrected image of the digital mirror display. The trained DNN model generates a first set of features based on the external image and a second set of features based on the corrected image of the digital mirror display. The trained DNN model generates a first vector based on the first set of features based on the external image and a second vector based on the second set of features based on the corrected image of the digital mirror display.

[0069] The trained DNN model determines whether the external image matches the image of the corrected digital mirror display, based on a comparison of the first vector based on the external image with the second vector based on the image of the corrected digital mirror display.

[0070] If the digital mirror display monitoring system 100 determines that the external image matches the corrected image of the digital mirror display, the procedure returns to 402. If the digital mirror display monitoring system 100 determines that the external image does not match the corrected image of the digital mirror display, the digital mirror display monitoring system 100 identifies at 412 an image mismatch cause associated with the digital mirror display on the digital mirror display device 202. Examples of image mismatch causes include, but are not limited to, digital mirror display freezing, digital mirror display color fading, and digital mirror display black pixel detection.

[0071] At 414, the digital mirror display monitoring system 100 generates an image mismatch notification for display on a vehicle display device 206. At 416, the digital mirror display monitoring system 100 transmits the external images received from the external camera 200, which is associated with the digital mirror display device 202, to an infotainment display device 208 for display on the infotainment display device 208.

[0072] At 418, the digital mirror display monitoring system 100 restarts the digital mirror display device 202, which is associated with the image misalignment. At 420, the digital mirror display monitoring system 100 determines whether restarting the digital mirror display device 202 resulted in the external image matching the corrected image of the digital mirror display.

[0073] If the digital mirror display monitoring system 100 determines that restarting the digital mirror display device 202 resulted in the external image matching the corrected image of the digital mirror display, the digital mirror display monitoring system 100 issues a command at 422 to the infotainment display device 208 to stop displaying the image mismatch notification on the infotainment display device 208. In at least one embodiment, if the digital mirror display monitoring system 100 determines that restarting the digital mirror display device 202 resulted in the external image matching the corrected image of the digital mirror display, the digital mirror display monitoring system 100 issues a command to the infotainment display device 208 to stop displaying the external images on the infotainment display device 208.

[0074] If the digital mirror display monitoring system 100 determines that restarting the digital mirror display device 202 did not result in the external image matching the corrected image of the digital mirror display, the digital mirror display monitoring system 100 issues a command to the digital mirror display device 202 at 424 to turn off.

[0075] In at least one embodiment, external images received from an external camera 200 are modified according to a color-insufficient driver setting to generate modified external images. The digital mirror display device 202 displays the modified external images. The digital mirror display monitoring system 100 determines whether an external image received from the external camera 200, which is associated with a digital mirror display device 202, matches a digital mirror display shown on the digital mirror display device 202 by comparing the modified external image with the image of the digital mirror display. The digital mirror display monitoring system 100 generates the image mismatch notification associated with the digital mirror display device 202 for display on a vehicle display device 206 based on this determination.

Claims

[1] Digital mirror display monitoring system (100) for a vehicle (10), comprising: at least one processor (44); and at least one memory (46) that is communicatively coupled to the at least one processor (44), wherein the at least one memory (46) comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to do the following: Receiving an external image captured by a first camera system (200) of the vehicle (10), wherein the external image comprises a perspective of a vehicle mirror view of an external environment outside the vehicle (10); Receiving an internal image captured by a second camera system (204) of the vehicle (10), wherein the internal image comprises an interior environment within the vehicle (10), wherein the internal image includes an image of a digital mirror display within the vehicle (10) and the digital mirror display is generated based on the external image; Determine whether the external image matches the image on the digital mirror display; and Generating an image mismatch notification for display on a vehicle display device (206) of the vehicle (10) based on the determination. [2] System (100) according to claim 1, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to do the following: Cropping the internal image to create an area of ​​interest (ROI) of the internal image that is mapped to the image of the digital mirror display; Implementing a distortion correction of the internal image's ROI to produce a corrected image for the digital mirror display; and Determine whether the external image matches the image of the digital mirror display, based on a comparison of the external image with the corrected image of the digital mirror display. [3] System (100) according to claim 2, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to use a trained deep neural network (DNN) model to: Generating an initial set of features based on the external image; Generating a first vector based on the first set of features; Generating a second set of features based on the corrected image from the digital mirror display; Generating a second vector based on the second set of features; and Determine whether the external image matches the corrected image of the digital mirror display, based on a comparison of the first vector with the second vector. [4] System (100) according to claim 1, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to identify, upon determining that the external image does not match the image of the digital mirror display, an image mismatch cause associated with the digital mirror display, wherein the image mismatch cause is one of a freezing of the digital mirror display, a color loss of the digital mirror display and a black pixel detection in the digital mirror display. [5] System (100) according to claim 1, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to transfer the external image to an infotainment display device (208) of the vehicle (10) for display on the infotainment display device (208) when it is determined that the external image does not match the image of the digital mirror display. [6] System (100) according to claim 5, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to do the following: Restarting a digital mirror display device (202) that is configured to display the digital mirror display; Determine whether restarting the digital mirror display device (202) resulted in the external image matching the image of the digital mirror display; and Issuing a command to the infotainment display device (208) to stop displaying the external image on the vehicle infotainment display device (208) based on the determination. [7] System (100) according to claim 6, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to issue a command to switch off the digital mirror display device (202) when it is determined that restarting the digital mirror display device (202) did not result in the external image matching the image of the digital mirror display. [8] System (100) according to claim 1, wherein a digital mirror display device (202) is a digital side view mirror display device of the vehicle (10). [9] System (100) according to claim 1, wherein the at least one memory (46) further comprises instructions which, when executed by the at least one processor (44), cause the at least one processor (44) to do the following: Modifying at least one color in the external image according to a color-insufficient driver setting to produce a modified external image; Issuing a command to a digital mirror display device (202) to display the digital mirror display based on the modified external image; Determine whether the modified external image matches the image of the digital mirror display; and Generating the mismatch notification of the digital mirror display device (202) for display on the vehicle display device (206) based on the determination. [10] Method (400) for monitoring a digital mirror display, comprising: Receiving (402), at a controller, an external image captured by a first camera system (200) of a vehicle (10), wherein the external image comprises a perspective of a vehicle mirror view of an external environment outside the vehicle (10); Receiving (404) at the control, an internal image captured by a second camera system (204) of the vehicle (10), wherein the internal image comprises an interior environment within the vehicle (10), wherein the internal image comprises an image of a digital mirror display within the vehicle (10) and the digital mirror display is generated based on the external image; Determine (410) by controlling whether the external image matches the image of the digital mirror display; and Generating (414) by means of the control, an image mismatch notification for display on a vehicle display device (206) of the vehicle (10) based on the determination.

Images