Automobile Camera Monitoring System

Abstract

To provide a vision system comprising at least an exterior rear-view mirror that allows a field of view outside motor vehicles to be adjusted by moving a driver's head.SOLUTION: A camera monitoring system for motor vehicles 10 comprises: image capturing means, provided on a mounting assembly of the motor vehicle, for an exterior FOV of the motor vehicle extending sideward and rearward outside the motor vehicle; an ECU, connected to the image capturing means, obtaining a captured image 200; an electronic display device, connected to the ECU, and used by a driver 11 inside the motor vehicle; and a gesture detector configured to obtain a position of at least one part of the driver's body. The ECU is configured to adjust the FOV based on the obtained position, and the electronic display device is configured to display the adjusted FOV in a displayed image region 210, which can be moved within the captured image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] (Field of the Invention) The present invention can be applied to an automotive control system including an electronic (internal and / or external) rearview mirror assembly.

[0002] The present invention relates to a camera monitoring system (CMS) that manages the field of view from one or more rearview mirrors displayed to a vehicle driver by detecting the driver's gestures.

[0003] (Background of the Invention) A problem with the prior art in the field of automobiles is adjusting the field of view during driving.

[0004] US9073493B1 discloses a method and apparatus for adjusting a vehicle's side mirror by (i) obtaining a first angle of the side mirror, (ii) taking an image of the driver's head, (iii) determining a vertical distance of the side mirror, (iv) calculating a field of view angle based on the field of view distance and the vertical distance, and (v) adjusting the side mirror from the first angle to a second angle. US9073493B1 solves the technical problem of adjusting the field of view during driving by determining the driver's position. However, this adjustment rotates the side mirror using an actuator. Mechanical actuators imply cost, complexity, and the possibility of damage over time.

[0005] Also, a driver may want to move their head forward, especially when overtaking, to reduce the blind spot zone. At that time, the driver increases the relative angle of their head with respect to the image capture means (for example, a reflective rearview mirror in a conventional automotive mirror system or a camera in an automotive digital vision system) to expand the view of the blind spot zone. However, in a conventional reflective mirror, unfortunately, when the relative angle of the driver's head with respect to the image capture means increases, the opening angle (α) becomes smaller. The opening angle (α) can be defined as the angular range (from above) of a given scene captured by the image means (the captured image area). The opening angle (α) is converted into the horizontal length of the image area captured from the driver's viewpoint (on the mirror or on the display).

[0006] Therefore, it is highly desirable to provide a motor vehicle equipped with a control system for monitoring an intelligent rearview mirror system to change the field of view of the rearview mirror without mechanical or manual actuation and without touching the display. More specifically, an object of the present invention is to improve the visibility of the driver's out-of-vehicle scene by providing a vision system comprising at least an external rearview mirror that can adjust the out-of-vehicle view by moving the driver's head. To provide a better view for the driver and in particular to minimize the blind spot zone during overtaking, an improved view of the lateral adjacent area of the driver's vehicle is required. SUMMARY OF THE INVENTION

[0007] (Summary of the Invention) The present invention provides a camera monitoring system (CMS) for a vehicle, which solves the above problems and overcomes the state-of-the-art working constraints described above, and at least provides a display device configured to display an image captured from an electronic (external and / or side) rearview mirror of a vehicle on an electronic display. Specifically, the proposed CMS can adjust the field of view (FOV) of the rearview mirror based on the position of at least a part of the user's body (e.g., the driver's head) without touching the display. The movement of the head / body (or the change in the position of the head / body) may be detected by sensors based on different technologies (ultrasonic, image, ToF (Time of Flight), radar, etc.). The electronic display may refer to a screen (preferably, a touchless screen) or a display glass.

[0008] One aspect of the present invention is a camera monitoring system for a vehicle, having the following configuration. - Image capture means (e.g., a camera) is attached to a mounting assembly (e.g., a winglet or a shark fin) disposed outside the vehicle. The image capture means is configured to capture an image from the external field of view of the vehicle, and its field of view (FOV) extends at least rearward (preferably also laterally) outside the vehicle and includes a part of the external portion of the vehicle (preferably, the external lateral portion or side surface of the vehicle). - An electronic control unit (ECU) connected to the image capture means. The ECU is configured to select one or more (external FOV) image regions from the image captured by the image capture means. The image region selected by the ECU may be the entire captured image or smaller than the captured image (referred to as "crop") (in this case, preferably, the ECU can perform "digital panning", i.e., move the selected image region within the captured image). - At least one electronic display device installed inside the vehicle and used by the driver (therefore, the display device is installed in a place visible to the driver, such as inside the door). The display device is connected to an ECU and is configured to display an image area of an external FOV selected by the ECU. - Further, the camera monitoring system further includes a gesture detector that acquires / detects the position of at least a part of the driver's body (for example, the head or face) so that the ECU can adjust (optionally move if "pan" is configured) the image area of the external FOV displayed by the driver's electronic display device based on the position obtained by the gesture detector.

[0009] Another aspect of the present invention is an automobile equipped with two camera monitoring systems (CMS) as described above. One CMS is arranged on the left side of the vehicle (equipped with image capture means related to the left external rearview mirror), and the other CMS is arranged on the right side of the vehicle (equipped with image capture means related to the right external rearview mirror), and further includes an interior rearview mirror system (IRMS). The ECU of the vehicle controls all of the two camera monitoring systems and the interior rearview mirror system. Both the CMS and the IRMS operate with a symmetric image of an image from the external FOV of the vehicle captured by the image capture means, so the captured image refers to a symmetric image of the raw captured image.

[0010] The symmetric image can be obtained by (i) the image sensor of the camera, (ii) the ISP of the camera, or (iii) the ECU of the vehicle. Option (i) can obtain an image captured directly from the camera due to the hardware architecture. Options (ii) and (iii) include image processing for obtaining a captured image that can be moved within the image area by gestures.

[0011] For the camera that captures the (raw) image, in the case of CMS, the camera is on the side of the vehicle (e.g., within the "winglet" mounting assembly), and in the case of IRMS, the camera is on the roof of the vehicle (e.g., within the "shark fin" mounting assembly). Also, when the FOV monitor is used for parking, a parking camera on the rear bumper of the vehicle (this type of camera is also called a rear backup camera) is used.

[0012] The present invention can be applied to the control of an interior rear-view mirror (IRMS) or other electronic mirrors of a motor vehicle. The proposed CMS can use gesture detection technology to manage the FOV of the rear-view mirror without touching the display (i.e., via the touchless screen of the display device). The touchless screen (e.g., installed on the door of the motor vehicle) can be implemented in different ways, such as i) increasing the capacitive sensitivity of the screen (without using a camera), ii) using a camera, for example, based on an image classifier.

[0013] The present invention can be applied to the control of any external rear-view mirror on the (left or right) side of the vehicle without reducing the opening angle (α) of the FOV (image area) displayed to the driver even when the position of the driver's head changes. This solves the drawback of the conventional reflective mirror and the digital vision system imitating the conventional reflective mirror that when the opening angle (α) is reduced, the field of view of the blind spot area is undesirably reduced. The opening angle (α) is defined as the angular range of the displayed FOV, i.e., the displayed image area (crop). For this purpose, the present invention is configured at least as follows. (i) Fix the opening angle (α) of the image area so as not to depend on the detected relative angle of the driver's head with respect to the image capture means related to the external rear-view mirror (ERMS) installed on the driver's seat side of the vehicle, or (ii) As the relative angle of the driver's head with respect to the capture means increases, the angle (α) of the opening of the image area is increased.

[0014] According to the first embodiment of the present invention, the camera monitoring system (CMS) is configured to select one single image area ("crop") and is capable of moving the selected image area within the captured image based on changes in the head / face position or movement of the driver's body (preferably the torso, head and / or hands). Thereby, the field of view (FOV) of the electronic rearview mirror shown to the driver by the display is adjusted. According to this embodiment, the CMS is further configured to fix the opening angle (α) of the selected image area and is independent of the detected relative angle of the driver's head with respect to the CMS (i.e., the display device).

[0015] Preferably, the image region (“crop”) selected by the CMS ECU is off-center (not at the center) of the captured image but is located inside, at the right end, or at the left end of the captured image: when the crop is displayed on a display device located on the left side of the vehicle (the driver's display in the case of right-hand traffic), it is on the right side of the image, and when the crop is displayed on a display device located on the right side of the vehicle, it is on the left side. Thus, the driver is provided on the display with a path for moving the selected image region outward along the horizontal axis of the captured image, and the non-centered position of the crop is for maximizing the available path that can be moved on the display. More preferably, the length of the path is at least 20% of the length of the selected image region before starting the movement (i.e., displacement). Even more preferably, the length of the path is at least twice the length of the space (“crop”) defined on the opposite side of the selected region. Those skilled in the art recognize that the size of the raw image captured by the image capture means (i.e., the camera) may be limited. The opening angle (α) of the image region may depend on the lens of the camera and the image sensor (i.e., the imager). The opening angle (α) of the image region can be made wider by using a lens including a curved surface or by increasing the size of the image sensor. With a lens including a curved surface, the image may be distorted. Image processing may be required to correct the distorted image. Thus, the fact that the crop (image region) is not at the center of the captured image is particularly advantageous for maximizing the available path described above.

[0016] According to a second embodiment of the present invention, the ECU of the camera monitoring system is configured to select two image regions (crops), namely a first image region (i.e., a first crop) of the captured image and a second image region (i.e., a second crop) which is an additional extended view, wherein the additional extended view is adjacent to the first image region (i.e., the first crop), and the first crop and the second crop are combined to form an overall image region. The second crop is further configured to be enlarged and / or reduced based on changes in the position of the head / face or the movement of the driver's body (preferably the torso, head and / or hands). Thus, the displayed image region consists of a first image region (i.e., a first crop) and a second image region (i.e., a second, expandable, crop) which is an additional extended view. Thereby, the FOV (Field of View) of the electronic rearview mirror shown to the driver in the entire image region on the display is adjusted. According to this embodiment, the CMS is configured to increase the opening angle (α) of the second image region, and thus the opening angle (α) of the entire image region, as the relative angle of the driver's head with respect to the driver's camera of the CMS increases.

[0017] In the second embodiment, of the two crops, the first crop (non-expandable crop) is located not at the center of the captured image but inside the right or left end thereof: i) in the case of a left-hand drive vehicle sold in a right-hand traffic country such as Spain, it is on the right side of the image, and ii) in the case of a right-hand drive vehicle sold in a left-hand traffic country such as the UK, it is on the left side of the image. Thus, the driver is provided on the display with a path for expanding the second crop outward along the horizontal axis of the captured image. More preferably, the length of the path is at least 20% of the length of the selected image region before starting the movement (i.e., displacement). More preferably, the length of the path is at least twice the length of the space defined on the opposite side of the selected region ("crop").

[0018] According to the option of the second embodiment, the second image area (extensible crop) gradually expands its length at least horizontally outward within the captured image according to the movement of the driver's body in the determined driving direction, and the first image area ("crop") does not change in both size and position. According to another implementation option of the second embodiment, the display device is composed of two different parts: a first part where the first crop is displayed and a second part where an additional expanded view (second crop) is displayed. According to one example, the display device is a single screen, preferably a touch screen. According to another example, the display device consists of two different screens. The first part distinguished by the ECU may be implemented on the first screen that can be made touchless and inexpensive, and the second part may be implemented on the second screen that can be made touch-sensitive.

[0019] According to the second embodiment of the present invention, the ECU of the camera monitoring system is configured to select two image areas (the first crop and the second crop) and operate the display according to one of two states depending on the movement of the driver's head: i) When head movement is detected, the ECU sets the display to the first operating state and fully displays (i.e., switches on) the second image area on the display or preferably on the second part of the display device; ii) Conversely, when it is detected that the driver's head has returned to its original position, the ECU sets the display to the second state and does not display the second image area at all (i.e., switches off).

[0020] Furthermore, according to the fourth embodiment, the present invention enables the user to customize the system by configuring (i) the sensitivity for moving the displayed image in the captured image ("crop" if there is only one image area selected by the ECU, "first crop" if there is a total image area formed by two crops selected by the ECU), or (ii) the sensitivity for enlarging and / or reducing the additional enlarged display (second crop).

[0021] In the fifth embodiment, the electronic display device is mounted on a head-mounted device. This head-mounted device can also be applied to control the glasses of a driver equipped with AR (Augmented Reality). The AR glasses display an image captured from one or more rearview mirrors of an automobile; that is, the display is at least part of the glasses instead of a screen inside the automobile. Preferably, when the display is part of the AR glasses, the sensor is on the glasses themselves.

[0022] Movements of the head or body (or changes in the position of the head or body) are detected by sensors using various technologies (such as ultrasonic, image, ToF (Time of Flight), radar, etc.). When the display is a screen inside the vehicle (for example, installed on the driver's door or IRMS), the sensor is inside the vehicle. Also, when the display is part of the AR glasses, the sensor is on the AR glasses themselves.

[0023] The present invention has many advantages over the prior art, and they can be summarized as follows. - The present invention can adjust the FOV of a plurality of external cameras fixed to the vehicle body without requiring actuators or mechanical movements for these cameras and without touching the display. - The control actions for adjusting the FOV are triggered by the driver's gestures, rather than pressing digital buttons (overlays) on the displayed image or touching the display with a finger (which may dirty the screen), and the action area becomes wider. That is, the driver can perform related gestures in the air (using the head or finger) instead of on a small / limited-sized screen. Furthermore, this function is also effective for the calibration of the system. That is, when the ECU associates (i) the first position of the driver's head with the first position of the crop in the "captured image" and (ii) the second position of the head with the second position of the crop in the "captured image", the system can use the "gesture detection" to detect the movement of the head and determine the aforementioned positions for calibration. - In the present invention, the lateral visibility outside the vehicle can be improved to reduce the blind spot zone. That is, the driver can obtain a better view of the blind spot zone, especially when preparing to overtake, by not reducing the opening angle of the image area displayed on the display device. - In the present invention, the FOV can be adjusted only by the gestures performed by the driver (preferably, the movements of the driver's head and face) for driving operations such as overtaking and parking. That is, the displayed image area does not always move by the driver's gestures, but only when the gestures of the driver detected by the system exceed a threshold value, the displayed image area moves. - In the present invention, it is possible to adjust the FOV by detecting gestures related to the driver, and the detection is performed by comparing (by the ECU) a first position ("reference position") and a second position ("current position"). Both positions are based on at least one part of the driver's body, preferably the position of the face / head. The reference position is the position of the driver's head detected at the moment of the initial time, and the head is not a fixed object (such as the vehicle seat, headrest, belt, etc.), and its position in the vehicle may change from the moment of the initial time. Since the ECU determines the movement of the driver's head and takes the position of the head at the moment of the current time as the "current position", the "current position" depends only on the driver's head and is compared with the "reference position" that also depends only on the head (a "head-to-head" comparison rather than a "head-to-seat" comparison). That is, the comparison of the positions used for FOV adjustment depends only on the driver and does not depend on external elements of the driver. - The advantage of using the position of a part of the driver's body instead of another element such as the headrest of the vehicle seat is as follows. When the reference position is based on the seat, if the seat position is changed (for example, the driver usually adjusts the position of the seat or headrest using a red light), the reference position is lost or needs to be recalculated by complex processing or communication with the vehicle's CAN bus. Also, when the gesture detector is, for example, a radar, since it cannot detect the form of the seat, the reference position cannot be calculated. Furthermore, there are drivers who remove the headrest or wear a hat that covers the entire headrest while driving, which hinders the detection of that (reference) position. - The present invention enables the user to calibrate / customize the sensitivity of FOV adjustment according to the user's preference before starting driving.

[0024] These advantages and other advantages will become apparent in light of the detailed description of the present invention.

Brief Description of the Drawings

[0025] (Description of the Drawings) To assist in the understanding of the features of the present invention, in accordance with its preferred practical embodiments and to complement this description, the following figures, which are of an illustrative and non-limiting nature, are attached as an integral part thereof.

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

DETAILED DESCRIPTION OF THE INVENTION

[0027] (Detailed Description of the Invention) The matters defined in this detailed description are provided to assist in the comprehensive understanding of the present invention. Therefore, those skilled in the art will recognize that they can make modifications, changes, and amendments to the embodiments described in this specification without departing from the scope and spirit of the present invention. Also, descriptions of well-known functions and elements are omitted for clarity and brevity.

[0028] Of course, embodiments of the present invention can be implemented in various architecture platforms, operating and server systems, devices, systems, or applications. The specific architecture layout or implementation presented herein is provided for illustration and understanding only and is not intended to limit aspects of the present invention.

[0029] According to a preferred embodiment, the CMS comprises image processing means including a camera substantially facing rearwardly disposed on a side (right and / or left) of a vehicle (e.g., an automobile). The captured image includes at least a side / sidelong portion of the automobile and a zone behind the automobile. The ECU (Electronic Control Unit) is configured to perform image processing of receiving the captured image, selecting at least a part of the captured image, and transmitting it to a display installed in the vehicle interior. The capturing means can be fixedly attached to the outside of the vehicle.

[0030] According to a preferred embodiment, the CMS further comprises a gesture detector, and the driver can be commanded to adjust the FOV of the CMS by the movement of the driver's head or other parts of the body (e.g., eye movement or facial gesture). In an embodiment of the CMS using a gesture detector, the crop & pan function may be triggered by the movement of the driver's head. The movement of the driver's head may be tracked by a monitoring system. The monitoring system is preferably in the vehicle, more preferably fixed (i) in front of the driver, or (ii) an internal rearview mirror, or (iii) near the CMS display device. Preferably, the driver monitoring system may be composed of a camera. The monitoring system can grasp the position and distance of the driver's head. Therefore, when the gesture detector detects the movement of the driver's head, the displayed image area in the captured image is controlled by the forward movement of the driver's head. There are four implementation options for the gesture detector: i) In one embodiment, the gesture can be recognized from the captured image of the user by the same camera attached to the external assembly (winglet) used for the CMS to capture the external FOV, or by an additional camera inside the vehicle (in-cabin camera), for example, a camera attached near the display of the driver's seat or to the rearview mirror. ii) In another embodiment, without the need for an additional camera, the gesture can be executed without touching the display, and the gesture can be detected simply by increasing the capacitive sensitivity of the screen. iii) In a further embodiment, a 3D gesture controller using an electrical near-field (E-field) enables user gesture detection and motion tracking using a single-chip device that does not require host processing necessary for embedded applications. For this reason, the CMS can detect that the driver desires to open (enlarge) their FOV by simply turning their head or waving a hand, and as a result, the FFOV of the external camera can be enlarged. Thus, in this preferred embodiment, multiple locations where the driver can perform an action are provided. iv) A radar sensor system for occupancy detection can include an antenna system configured to generate a transmitted radar signal and a radar sensor configured to receive a sensor signal reflected as a result of the transmitted radar signal. Optionally, it may further include an accelerometer for obtaining acceleration data values. Additionally, it may include an algorithm for generating an improved signal by applying accelerometer data and offset removal to the sensor signal.

[0031] In another preferred embodiment, the display is within a head-mounted device (i.e., a near-eye device for the driver), preferably a pair of glasses with Augmented Reality (AR). AR glasses such as Google Glasses and Microsoft HoloLens already exist, for example. The head-mounted device (e.g., an interactive pair of AR glasses) includes: i) an optical assembly configured to display virtual content and be able to view at least a part of the surrounding environment; ii) an integrated processor for processing the displayed virtual content; and iii) an integrated image source for introducing the virtual content into the optical assembly. Further, the head-mounted device may include a communication function configured to connect the interactive head-mounted device to an external device such as an ECU of a camera monitoring system (CMS). The head-mounted device includes a gaze position detection unit for detecting the driver's gaze or fixation. The CMS using this head-mounted device can generate an instruction to move the crop (digital pan) in the captured image based on the position of the driver's head and the direction of the line of sight when the driver wears the AR glasses. The captured image may include an image symmetric to the raw captured image. Since the head movement electronically detected by the head-mounted device coincides with the direction of the driver's eyes / line of sight, the communication facility is configured to provide the ECU with a video feed that coincides with the head movement and the direction of the line of sight of the eyes. The video feed transmitted from the ECU to the head-mounted display via the communication facility is an enriched video stream including at least one of the data of (i) the displayed image area (crop) and (ii) the detection of blind spots. The displayed image area (crop) is displayed only when the detected driver's eyes are looking at a specific point on the optical assembly. For example, if the driver is not looking at the place where the rearview mirror should be, the display of the AR glasses is off; in this case, the ECU generates a signal to turn on the display of the AR glasses screen only when the "line-of-sight detector" catches that the driver is looking at the specific place.

[0032] In a preferred embodiment, the adjustment of the FOV includes the symmetry of the raw captured image (the original image captured by the image capturing means) in order to obtain a symmetric image. The symmetric image can be directly provided by the capturing means, or can also be provided by the ECU (especially the ISP) after performing image processing. In particular, when the capturing means directly provides a symmetric image, since the symmetric image can be obtained through the hardware of the image sensor, the computational amount of image processing (software) can be significantly reduced. Also, the adjustment of the FOV means (i) the movement of the crop within the symmetric image, or (ii) the enlargement of the additional extended view within the symmetric image.

[0033] FIG. 1 shows, according to a possible example, the relationship between the movement (111) of the driver's head (11) detected by the sensor (gesture detector) and the movement (211) or "pan" of the display image area (210) or "crop" within the captured image (200). In this example, the movement (111) of the driver's head (11) along the driving direction is converted into the displacement / movement (211) of the display image area (210) processed in the horizontal direction (by the ECU). Therefore, the change in the field of view (FOV) that the driver (11) sees is indirectly performed by changing the position of the crop, and as a result, a new FOV that the driver (11) sees is obtained.

[0034] In one embodiment, the ECU performs image processing as follows. - Obtain the captured image (200) as the symmetry of the (raw) image captured by the capturing means. This symmetry can be obtained by image processing by the ECU (usually, the symmetry is directly executed by the image sensor or imager of the imaging means). - Select the image area (210) of the captured image (200), that is, perform a "crop" that makes the image area (210) smaller than the captured image (200). - Execute the digital "pan" and move the crop within the image (symmetry) captured by the image capture means.

[0035] The symmetry of the raw image is performed with respect to its vertical axis. For example, other types of image processing, such as emulating a conventional rearview mirror, are not required. However, when the arm of the mounting means (i.e., the winglet or shark fin) on the outer side surface of the vehicle (10) is short or absent, it may be advantageous to perform an image processing operation to change the viewing point (i.e., perspective correction), for example, using a homography matrix. What is important here is that applying the homography matrix to i) the raw image (400), ii) the symmetric image (200), or iii) the image region (210) is definitely not the same or equivalent to emulating a conventional reflective rearview mirror. A conventional reflective rearview mirror is not the same or equivalent because as the relative angle between the driver's head and the display device increases, the opening angle (α) of the reflected image decreases, and the field of view of the blind spot region decreases in an undesirable manner.

[0036] Preferably, when the image capture means used is arranged on the outer side surface of the vehicle (10), the captured image (200) has a FOV in the range of 40° - 60°, and the image region (210) has a FOV between 60% and 80% of the length of the captured image (200). The ratio shape / geometry of the captured image (200) remains the same for the crop or the image region (210). When cropping using an 80% FOV, the FOV of the crop is 32° (calculated as 80% of 40°), and when cropping using a 60% FOV, the FOV of the crop is 36° (calculated as 60% of 60°). According to another non-limiting example, when the image capture means used is associated with an internal IRMS, the FOV of the captured image is 70°.

[0037] Figure 2 shows the relationship between another movement (112) of the driver's head (11) and the movement (212) of the displayed image area (210) according to another example. In this example, when the movement (112) of the driver's head (11) is upward, the movement (212) of the displayed image area (210) is downward in the vertical direction.

[0038] The cropped or displayed image area (210) is displaced according to a pre-established relationship. The pre-established relationship may be linear or non-linear (see Figure 3).

[0039] Table 1 below and Figure 3 show the horizontal movement of the crop according to the crop length when the driver's head moves forward along the driving direction. (Table 1) Forward head position variation (mm) Horizontal movement amount of crop (%) 0 0 100 10 200 40 300 100

[0040] As in the example of Table 1 above, when the horizontal movement amount according to the crop length when the head moves forward 300 mm is 100%, if the camera captures 60° and the crop is 60% of the camera, the FOV of the crop is 36°, that is, 24° of horizontal movement amount remains.

[0041] Figure 4 is the raw image (400) captured by the capture means arranged on the winglet on the outer side surface corresponding to the driver's seat side of the vehicle (10). Here, it is the raw image (400) captured by the capture means (for example, an external camera) arranged on the left side of the vehicle (10). This raw image (400) including the partial image of the left part of the vehicle (10) is what is seen by the capture means located on the side of the driver (11). The outer square corresponds to the image boundary of the entire FOV captured by the capture means, for example, a 60-degree FOV.

[0042] From the raw image (400) shown in FIG. 4, as symmetry with respect to the vertical axis of the raw image (400), the captured image (200) shown in FIG. 5 is obtained. Thus, the partial image of the vehicle (10) is seen on the right side of the captured image (200). According to this example, the image processing necessary to obtain the captured image (200) is only symmetry, and image processing such as emulation of a conventional rearview mirror is not performed. The inner square corresponds to the crop or image area (210) selected to be displayed, that is, the boundary line of the effective FOV displayed on the driver's (11) display, for example, an effective FOV of 36 degrees is displayed. In FIGS. 5 and 6, the effective FOV shows a highway including the horizon.

[0043] FIG. 5 shows the crop or image area (210) that is displayed when the ECU moves the crop horizontally by 40% (among the total movement amount of the entire crop) within the captured image (200). According to Table 1 and FIG. 3, this corresponds to the case where the driver moves his head 200 mm forward, that is, in the traveling direction of the vehicle (10).

[0044] Figure 6 shows the crop or image area (210) that is displayed when the ECU moves the crop 100% horizontally and reaches the limit of the captured image. According to Table 1 and Figure 3, this corresponds to the case where the driver moves their head 300 mm forward, i.e., in the driving direction of the vehicle (10). For example, when the image capture means is arranged in the mounting assembly of the left outer rearview mirror, the crop or image area (210) is preferably on the right side of the captured image when the driver's head is in the normal driving position, and when the head moves forward, as shown in Figure 6, the image area (210) moves to the left. That is, the image area (210) is not initially located at the center of the captured image, i.e., the image area (210) is off-center from the center of the captured image, but is usually located inside, on the right side, or on the left side: for example, in the case of a left-hand drive vehicle in a right-hand traffic country, it is located on the right side. Therefore, a path for moving the crop along the horizontal axis of the captured image is provided on the display for the driver, but this path is very long. For example, going to Figure 11, especially in the left figure, the displayed image area or crop (210, 210') is not centered in the captured image (200) so as to leave a space (i.e., a path) on its left side where it is not displayed. The length (L2) of said space is at least 20% of the length (L1) of the image area (210, 210') before starting the movement within the captured image (200). Further, the crop (210, 210') may leave a smaller space on its right side having a length (L3) such that L2≧L3, as illustrated in Figure 11. Those skilled in the art recognize that the size of the raw image (400) captured by the image capture means (i.e., the camera) may be limited. The opening angle (α) of the image area (210) may depend on the lens of the camera and the image sensor (i.e., the imager). A wider opening angle (α) of the image area (210) may be achieved by using a lens including a curved surface or by increasing the size of the image sensor. With a lens including a curved surface, the image may be distorted. Image processing may be required to correct the distorted image.Thus, it is particularly advantageous for the crop (image area) not to be at the center of the captured image (200) in order to maximize the available paths described above.

[0045] In addition, not only for the driver, but also for a co-driver (passenger) sitting in the passenger seat, a similar cropping operation can be performed, and for this purpose, another camera of the CMS is provided. In the case of a conventional rearview mirror, rearview mirrors with different positions and sizes were used. However, in the case of the CMS, the same FOV adjustment value can be used to maintain the symmetry of the FOV on both sides of the vehicle.

[0046] The relationship between the movement of the crop and the movement of the driver's head may be linear. However, for example, in the following driving scenarios, there may be other options. i) Since the driver is turning right on a highway and driving alone, the FOV of the CMS is in a dead zone where it is fixed or has very little variation. ii) Another zone where a larger movement amount of the crop is required due to merging onto a highway or changing lanes.

[0047] Therefore, the linear movement of the crop may be moved vertically, horizontally, and finally diagonally. Also, the crop may be arbitrarily zoomed in / out. Zooming out means increasing the size of the crop (display image), and zooming in means decreasing the size of the crop (display image). Furthermore, when the gesture detector is a camera, it is possible to detect the driver's line of sight, that is, which point the driver is looking at. Therefore, in another example, the ECU can perform a digital pan when there is head movement and the driver is looking at the display. In this example, if there is head movement but the driver is not looking at the display, the crop will not move. That is, the display image does not always move every time the driver moves their head, and the display image will only move when the detected gesture meets a threshold or a specific criterion.

[0048] To summarize the previous paragraph, the driver's head can be detected on three axes of x, y, and z. The driving axis indicated by x is essential, and the other two axes can be optional / additional. The ECU can correspond to both the position of the head and the relative angle of the head with respect to the driver's display (α angle when viewed from above). When the driver moves their head forward, the α angle increases, and the crop will move linearly to the left within the display along the horizontal axis x.

[0049] The ECU can perform operations and move the crop based on at least the change in the position of the head (defined by x, y, z coordinates), or at least the α (top), β (side), and γ angles of the head. Also, the ECU can perform digital panning (movement of the crop) based on the following data obtained by the gesture detector (data measured by sensors or data extracted from an image of the head captured by a camera): - Movement of the head (change in position) along the driving direction (longitudinal direction of the vehicle). - Relative α / β angles of the head considering only the driving direction. Changes in these angles depend only on the change in the position of the head because the display device of the driver's seat is fixed. The α angle is the relative angle between the head and the camera when viewed from above. - Movement of the head (change in position) along the driving direction and the horizontal axis of movement from the external (driver's) rearview mirror to the other (co-driver's) rearview mirror. - Relative α / β angles of the head considering the driving direction and the horizontal axis of left - right movement. - Movement of the head (change in position) along the driving direction, the horizontal axis of left - right movement (from one external rearview mirror to the other external rearview mirror), and the height of the head with respect to the ground (vehicle floor or road surface). - Relative angle of the head with respect to the display when viewed from the side of the vehicle, i.e., the angle formed by the height of the head with respect to the ground.

[0050] When the driver is in front of the rearview mirror (when there is no relative angle), the image reflected in the rearview mirror and the image displayed on the display match, that is, both images are exactly the same. However, as shown in FIG. 7, when there is a relative angle between the driver's head and the rearview mirror (1000) or there is head movement, these images are neither the same nor equivalent. When the head moves forward, the driver's eyes move from the first position (E51) to the second position (E52): the viewing angles (a1, a2) of the driver with respect to the reflective rearview mirror (1000) corresponding to the first position (E51) of the driver's eyes are different from the viewing angles (b1, b2) of the driver corresponding to the second position (E52). The case where the difference between the two images is large is when the head is moved forward at the same height as the mirror (1000). The captured image is simply a symmetric image of the "raw" image covering the external FOV, that is, captured by the camera associated with the rearview mirror, but the conventional rearview mirror is not emulated by the captured image. Therefore, computationally, advantageously, there is less cumbersome work. That is, the reflected image from the reflective mirror is not the same as, nor technically equivalent to, the display image obtained by the image processing (symmetry) + cropping + panning performed by the proposed CMS.

[0051] FIG. 7 shows a comparison between the light rays reflected by a conventional rearview mirror (1000) and the field of view captured by a camera disposed on the outer side surface of an automobile. As the driver's head moves forward from the first position (E51) toward the second position (E52), the reflected light rays (i.e., what the driver sees through the mirror) tend to become gradually parallel. Therefore, the second angles (b1, b2) corresponding to the second position (E52) are larger than the first angles (a1, a2) corresponding to the first position (E51). An increase in the angle means that the light rays tend to become parallel, and when the light rays become parallel, the image is distorted, so caution is required. When the driver's head is very far forward (almost at the same height as the mirror), the display image of the present invention (i.e., the image area (210)) is significantly different from what the driver can see with a conventional rearview mirror (1000). When the driver approaches and moves the head forward toward the mirror (1000), i.e., when the head is moved to the left in FIG. 7, the conventional reflective mirror (1000) is actually zoomed in because the head is in a position closer to the conventional reflective mirror (1000). Further, although the field of view displayed / reflected by the conventional reflective mirror (1000) changes as the driver moves the head, according to this particular embodiment in which the image area (210) is formed by a single crop, the field of view captured by the outer side surface camera displayed by the display device is always a constant FOV.

[0052] In FIG. 8, a “third” position of the driver's eye is further added to show the change in the light rays when the driver moves their head. The square (1001) drawn on the mirror (1000) represents the lens of the camera of the CMS, and the drawn triangle (1002) represents the FOV of the CMS camera associated with the rearview mirror (1000), that is, what corresponds to the captured image. FIG. 8 shows a comparison between the image captured by the external side camera associated with the rearview mirror (1000) and the light rays of a conventional reflective rearview mirror. In this example of FIG. 8, it can be clearly confirmed that the respective light rays corresponding to the first position (E61), the second position (E62), and the third position (E63) are gradually inclined. Therefore, as the driver's eye approaches the mirror (1000), due to the relative angle, the image observed by the reflective mirror (1000) does not match the true size (therefore, the difference between the CMS image (mere symmetry) and the reflective image is increasingly emphasized). It is very important to note that the two (the reflective image and the displayed image) are not the same, and the difference is emphasized as the position of the driver's head moves forward (that is, as the relative angle between the camera / mirror and the driver's head changes).

[0053] Figure 9 shows a triangle (1003) representing a crop within the captured image, and the captured images represented by other triangles are either the raw image or its symmetric image. Figure 9 shows the difference between the image perceived by the driver using the reflected image of a conventional mirror and the image displayed on the driver's display device. Even if the position of the driver's head changes, the opening angle (α) of the displayed FOV remains unchanged. In particular, even if the relative angle between the CMS and the driver's head changes, the value of the opening angle (α) is fixed. In contrast, when the position of the driver's head changes (the relative angle α changes), the image area (210) moves within the captured image, so the displayed FOV or crop (image area (210)) changes. When the driver's head moves forward, the opening angle (α) of the image area (210) remains fixed because it does not depend on the detected head / body position, but the image area (210) represented by the triangle (1003) in Figure 9 moves at least horizontally within the captured image. The captured image is merely a symmetric of the raw image captured by the image capture means of the CMS. If there is a wall behind the vehicle (10), the camera cannot capture the image behind the wall, so the triangle (1003) will be discarded. The opening, angle (α) of the triangle (1003) is the same for the first position (E61), the second position (E62), and the third position (E63).

[0054] That is, the displayed image area (210) is composed of an FOV with a fixed opening angle (α).

[0055] Figure 10 shows a comparison between the opening angles (a1', a2') of a conventional reflective rearview mirror (1000) "the upper quartered section A in Figure 10" and the opening angles (C1, C2) of a preferred embodiment of the present invention "the lower quartered section B in Figure 10". Also, Figure 10 shows the relative angles (X1, X2) between the position of a part of the driver's body obtained (for example, the driver's head) and the following. (i) When the image capture means is arranged on the left or right side of the vehicle (10) (for example, a camera attached to an external rearview mirror), the image capture means (for example, a camera). (ii) An electronic display device when an image capture means (for example, a camera associated with an interior rearview mirror) is disposed on the roof of a vehicle (10). FIG. 10 shows a first position (E11) of a driver's eye and a second position (E12) when the head moves forward. The angle (a1') of the driver's line of sight with respect to the reflective rearview mirror (1000) corresponding to this first position (E11) is greater than the angle (a2') of the driver's line of sight corresponding to the second position (E12) shown on the left side of FIG. 10, that is, as previously explained in FIG. 7, a1'>a2'. Also, FIG. 10 shows optical axes (E1, E2) as bisectors of triangles representing the FOVs of the first position (E11) and the second position (E12) respectively. In the case of the conventional system shown in the upper part (A) of FIG. 10, the FOV is directly shown to the driver in the reflective rearview mirror (1000), and according to a preferred embodiment of the present invention shown in the lower part (B) of FIG. 10, the FOV is displayed as a crop. When the driver's eye goes from the first position (E11) to the second position (E12), in the example of FIG. 10: - The relative angles (X1, X2) between the driver's eye and the external rearview mirror (1000) on the left side increase, that is, X2>X1; - The angles (EO1, EO2) of the optical axes (E1, E2) increase, that is, EO2>EO1; - The angles of the driver's field of view (a1', a2') with respect to the external rearview mirror (1000), these angles (a1', a2') define the opening of the FOV shown in the rearview mirror (1000) of the conventional system and decrease, that is, a1'>a2'; - The opening angles (C1, C2) of the crop or FOV displayed at the first position (E11) and the second position (E12) remain constant, that is, C1 = C2.

[0056] Therefore, with respect to the change in the angle from X1 to X2 due to the change in the position of the driver's eyes from E11 to E12, the opening of the crop will be kept constant at C1 = C2. Note that the relative angle between the driver's eyes and the conventional side exterior rearview mirror (1000) is not the relative angle between the driver's eyes and the camera arranged on the outer side surface and associated with the exterior rearview mirror. For example, in an automobile (usually not a truck), the reflective mirror has been replaced by an external camera, but this camera does not have to be in the exact position of the mirror (for example, it can be placed slightly lower and made invisible to the driver). In the case of a truck, it is highly likely that the rearview mirror has not been replaced by an external camera, and the mirror and the camera coexist and function complementarily.

[0057] Furthermore, there is another variable element, namely the distance between the driver's eyes and the rearview mirror. At the second position (E12), since the driver's head approaches the rearview mirror, "zooming in" occurs, and due to the "zooming in", the opening angle (C1, C2) of a single crop becomes smaller. Therefore, the angle (C1, C2) of the displayed FOV is constant with respect to the change in the angles X1, X2, but changes during "zooming in / out".

[0058] Furthermore, by making the vehicle camera that captures the driver's image function as a gesture detector, not only the position / movement of the driver's head but also the eyes can be detected. Then, the ECU can arbitrarily set the midpoint between the two eyes from the image provided by the vehicle camera (gesture detector), and the detection of the head can be set as the midpoint between the two eyes. It is also possible to calculate the contour of the head as an option. The camera can capture various parts of the face (ears, mouth / lips, nose, etc.).

[0059] FIG. 11 shows another second embodiment, and the displayed image area (210) is composed of two image areas or crops selected by the ECU (i.e., the first crop (210') and the second crop corresponding to the additional enlarged display (220)). The first crop (210') is selected by the electronic control unit (ECU) from the captured image (200) and includes a part of the exterior portion of the vehicle (10). The additional extended view (220) is a second crop selected from the captured image (200) by the ECU, and the additional extended view (220) is located in the vicinity of the first crop (210'). As shown in FIG. 11, it may be located particularly adjacent to the first crop (210'), i.e., adjacent to the first crop (210'), and preferably may have the same height as the first crop (210'). The gesture detector is configured to acquire at least one position of at least a part of the driver's body (e.g., the driver's head), and the ECU is configured to adjust the display image area (210) of the external FOV based on at least one acquired position by enlarging the length of the additional extended view (220) when the detected relative angles (X1, X2) between the driver's head and the camera monitoring system "i) the camera for ERMS or ii) the display for IRMS" increase. That is, when the gesture detector detects that the relative angles (X1, X2) have increased (i.e., when the driver sticks their head forward before overtaking), the ECU is configured to make the additional extended view (220) longer in the horizontal outward direction (i.e., expand). Further, when the gesture detector detects a decrease in the relative angles (X1, X2), the ECU is configured to return the additional extended view (220) to its original state and make it shorter in the horizontal inward direction (i.e., contract). It is preferable that the ECU is configured to gradually expand or contract as the relative angles (X1, X2) increase or decrease, respectively. In the illustrated example, it is important to note that as the relative angle of the driver's head with respect to the CMS increases, the opening angle (α) of the image area (210) increases.Therefore, the additional extended view (220) causes an improvement in the field of view of the lateral adjacent area outside the driver's vehicle in order to reduce the blind spot zone during overtaking. According to this example, it is important to note that even when the head moves forward, the driver can always see at least a part of the outer side surface of the vehicle. That is, the first crop (210') remains fixed (i.e., does not change) regardless of the movement of the driver's head.

[0060] Therefore, a first embodiment of the present invention in which only one crop is selected by the ECU is compared with a second embodiment of the present invention in which there are two crops selected by the ECU. - In the first embodiment, the opening angle (α) of the display image area (210) is fixed with respect to the change in the relative angle between the driver's head and the display device. - In the second embodiment, the opening angle (α) of the display image area (210) formed by the first crop (210') and the additional extended view (220) or the second crop increases as the relative angle between the driver's head and the display device increases.

[0061] Preferably, the length of the additional extended view (220) increases at least when the driver moves the head forward. The additional extended view (220) gradually increases / decreases its horizontal length as a result of the increase / decrease in the relative angle between the driver's head and the display device, while the first crop (210') does not change regardless of the movement of the driver's head, ensuring that the side surface portion outside the vehicle is permanently displayed. The first crop (210') once displayed is smaller than the display device. Therefore, the display device is large enough to display both crops ((210') and the additional extended view (220)).

[0062] According to another example, when the movement of the head is not detected (i.e., when the driver is not overtaking in the normal driving state), the additional extended view (220) may be turned off. When the driver's head moves forward, since the relative angle between the driver's head and the display device increases, when the ECU detects that the threshold value has been exceeded, the ECU is configured to generate the extended view (220) without modifying the first crop (210'). According to one embodiment, the first crop (210') is not displayed across the entire screen size. Due to the fact that the first crop (210') is smaller than the actual size of the display device, as depicted in FIG. 11, there is available space for display and then for enlarging the additional extended view (220). According to this example, the first crop (210') is always displayed during driving and is fixed both in terms of its position, size, and shape within the captured image (200) (i.e., it is constant over time). On the other hand, the length of the additional extended view (220) can be variable as it may grow. That is, based on the detected increase in the relative angles (X1, X2), the opening angle (α) of the entire displayed image region (210) increases.

[0063] Also, FIG. 11 shows the relationship between the movement of the driver's head detected by a sensor (gesture detector) according to an example and the enlargement of the additional extended view (220) displayed within the captured image (200). In this example, the movement of the driver's head along the driving direction is converted (by the ECU) into the extension of the horizontally processed additional extended view (220). Therefore, the change (i.e., adjustment) of the field of view (FOV) seen by the driver is indirectly effected by the length of the second crop (220), and as a result, a new FOV seen by the driver is obtained.

[0064] The ECU performs image processing as follows. - As the symmetry of the (raw) image captured by the capture means, obtain the captured image (200). This symmetry can be obtained by image processing by the ECU (usually directly performed by the image sensor or imager of the capture means). - Obtain a first crop (210) that is smaller than the captured image (200) and covers at least a part of the outer side surface of the vehicle (10). - Perform digital expansion of the displayed additional extended view (220) and increase its length.

[0065] The digital expansion of the displayed additional extended view (220) can follow a pre-established relationship. The pre-established relationship may be linear or non-linear (see Figure 3).

[0066] For example, the first crop (210') is permanently displayed while the vehicle is running. On the other hand, the ECU is configured to generate an instruction to expand the additional extended view (220) by 40% of the full length of the first crop (210'), at least in the horizontal direction. This corresponds to the case where the driver moves his head 200 mm forward, that is, in the traveling direction of the vehicle (10).

[0067] According to this illustrated embodiment, the ECU is configured to horizontally expand an additional extended view (220) that is displayed to reach the limit of the captured image (200). Preferably, when the driver's head is in the normal driving position, the first crop (210') is on the right side of the captured image, and when the head moves forward, as shown in FIG. 11, the additional extended view (220) expands to the left. That is, the first crop (210') is not normally centered but is usually located on the right side of the captured image. Thus, the driver is provided with a path on the display for making the length of the additional extended view (220) along the horizontal axis of the captured image much longer. In particular, the first crop (210') is not centered so as to leave available space on its left side within the captured image (200) that is not displayed. The length (L2) of said space is at least 20% of the length (L1) of the first crop (210'). Further, the length (L2) is at least twice the length of the minimum length (L3) from the right boundary of the captured image (200) to the left.

[0068] The linear movement of the additional extended view may be vertical, horizontal, and finally the second crop may move obliquely (not shown). Optionally, the sensor may be configured to detect at least a part of the driver's body in a three-dimensional space, and the ECU may be configured to select the additional extended view (220) located up to the first image area (210'), and the additional extended view (220) may gradually expand its height vertically downward within the captured image (200) in response to the determined vertical body movement, and the first image area (210') may remain fixed (i.e., unchanged) (not shown).

[0069] In addition, the first and second crops (210', 220) that form the display image area (210) may optionally be zoomed in / out. Zooming out means increasing the size of either of the two crops (210', 220), and zooming in means decreasing the size of either of the crops (210', 220). Further, when the gesture detector is a camera, it is possible to detect the driver's line of sight, i.e., which point the driver is looking at. Therefore, according to a further embodiment, the ECU can perform digital expansion of the additional extended view (220) when the movement of the head is detected and the driver is looking at the display device. Therefore, according to this example, the additional extended view (220) is not displayed when there is head movement but the driver is not looking at the display device. That is, the additional extended view (220) is not always displayed every time the driver moves their head, but is only displayed when the detected gesture meets a threshold or specific criteria.

[0070] To summarize the above, the driver's head can be detected on the three axes of x, y, and z, with the drive axis indicated by x being essential and the other two axes being optional / additional. The ECU can respond to both the position of the head and the relative angle of the head with respect to the driver's display. When the driver moves their head forward, the driver can increase the α angle and expand the additional extended view (220) linearly within the display to the left at least along the horizontal axis (x).

[0071] FIG. 12 is a top view of the increase in the mouth opening angles (α1, α2) when the driver's head moves forward to overtake another vehicle (10'). At the first position (E51) of the driver's eyes / head, the mouth opening angle is at the initial value (α1). When the driver of the vehicle (10) starts to overtake another vehicle (10'), the driver's head moves to the second position (E52), and the mouth opening angle takes another value (α2), where α2 > α1. As shown in FIG. 11, since the expandable second crop (220) is added to the first crop (210') and expands outward in the horizontal direction, this increase in the mouth opening angle is converted into an increase in the horizontal length of the display image area (210).

[0072] FIG. 13 shows another option of the expandable second crop (220) added to the first crop (210') to form the display image area (210). This crop can expand downward in the vertical direction when the driver's head moves a few millimeters upward, and is used, for example, when parking to look at the ground to check for curbs or other obstacles.

[0073] As shown in FIG. 14, the CMS includes an electronic display device (500) having two different portions: a first portion (5001) where a first crop (210') is displayed and a second portion (5002) where an additional extended view (220) is displayed. According to one example, the display device (500) is a single screen, preferably a touch screen. According to another example, the display device (500) consists of two different screens. By having two screens, the first portion (5001), which is distinguished by the ECU, may be implemented on a first screen that can be made touchless and inexpensive, and the second portion (5002) may be implemented on a second screen that can be made touch-sensitive. The second portion (5002) is preferably smaller than the first portion (5001). One of the technical advantages is the cost of the non-touch-sensitive screen. Further, the driver may perform a "touch & drag" operation in the second portion (5002) to move the display image area (210). The first calibration of the camera monitoring system (CMS) would advantageously be performed when the driver is about to start driving. The technical advantage of doing so is that the first portion (5001) does not get dirty. Further, the second portion (5002) may display parameters related to the brightness and / or contrast and / or color of the first portion (5001). As described, the second screen displays the additional extended view (220) only when the ECU determines that it must display the additional extended view (220), while the first screen permanently displays the first crop (210') while the engine of the vehicle (10) is on. For example, the driver can calibrate the CMS so that a movement of the driver's thumb indicating an OK symbol can turn on the additional extended view (220). Since the first crop (210') displayed on the first screen is fixed (i.e., does not change even if the driver changes the position of their head), even when the driver moves their head forward, it is ensured that at least a part of the outer side of the vehicle (10) is always displayed when the engine is running.

[0074] As shown in FIG. 14, in order to implement the above two parts (5001, 5002) in the display device (500), the CMS further includes a frame (300) that at least partially covers the display device (500). This frame (300) has the same dimensions as the display device (500) or has dimensions large enough to at least cover the entire display device (500). The frame (300) is preferably a cover made of plastic or glass, and since the touch screen may be relatively fragile, it protects the display device (500) from impact and damage. The frame (300) may be partially colored. Preferably, the color is black. The frame (300) does not allow all the light emitted by the display device (500) to pass through. Therefore, the driver cannot see the light emitted by the display where the frame is colored, i.e., in the silk screen area. The frame (300) is placed, for example, on the touch screen, and thus, since it is placed between the screen and the user, what the user sees is the "frame".

[0075] Alternatively, as shown in FIG. 15, the third embodiment of the present invention includes two operating states of the display device (500). The first state corresponds to, for example, overtaking when the gesture detector detects that the detected driver's head has exceeded a threshold, and then, as shown in the right diagram of FIG. 15, a second image area (220) as an additional extended view is fully displayed (i.e., switched on) in the second part (5002). The second state corresponds to the case where, as shown in the left diagram of FIG. 15, when it is detected that the driver's head has returned to its original position (i.e., normal driving), the second image area (220) is not displayed at all, i.e., the second part (5002) is switched off. The difference between the second embodiment and the third embodiment is that, while the enlarged view of the second embodiment continuously expands or contracts, the third embodiment is either fully displayed (i.e., the first state) or not displayed (i.e., the second state). Optionally, the ECU may generate a black image for the second state. Those skilled in the art will recognize that a dark color such as black consumes low power when displayed on an LLED screen such as an OLED display device.

[0076] As described, based on the movement of the driver's head, in practice, the FOV is changed (i.e., adjusted) by either (i) moving the image area (210) or the crop, or (ii) zooming in and out of the additional extended view (220). Thus, another further fourth embodiment relates to the sensitivity, i.e., the speed at which the FOV is changed (the speed at which the crop is moved or the size of the additional extended view is increased). For example, if the driver moves their head 10 cm, the crop moves 10 mm, but if they move it 20 cm, the crop only moves 10 mm. In another example, a first driver may want to move the displayed image in the captured image horizontally by 1 cm by moving their head 30 cm forward, while a second driver may want to move the displayed image in the captured image horizontally by 2 cm by moving their head 30 cm. That is, not all head movements produce the same displacement of the crop, and the system can be customized according to the driver's preference to control the speed of displacement / movement (panning). This sensitivity calibration also applies to the second embodiment where, as the detected relative angle increases, the additional extended view is expanded (extended / prolonged), increasing the aperture (α) of the image area (210).

[0077] The movement of a part of the driver's body detected by the sensor is at least in the driving direction of the vehicle, and accordingly, the panning is at least linear and horizontal. Preferably, it is also possible in the vertical plane (perpendicular to the ground plane), and the panning movement is accordingly linear and vertical, or a combination of horizontal and vertical, i.e., diagonal.

[0078] The movement of the crop based on the movement of the head is performed by comparing the reference position of the driver's head with its current position. Both positions can be calculated relative to the driver's head, or other parts of the driver's body or head, preferably the face. Thus, the reference position is calculated relative to a part of the driver's upper body, rather than relative to a (vehicle) fixed element. The system is calibrated before the driver starts driving in order to control / customize the sensitivity of the system when moving the crop. The system is configured to store the driver's settings in order to detect each user and load their stored profile. Therefore, during driving, the movement of the displayed image changes according to the customization (calibration) of the sensitivity. That is, one driver may slightly change the FOV by moving their head significantly (during driving), while another driver may desire a higher sensitivity such that the FOV changes significantly with only a slight movement of the head. The user can calibrate (customize the sensitivity) of the camera monitoring or vision system (CMS) of any electronic rearview mirror (internal and / or any side mirror).

[0079] According to one example, the customization of the sensitivity of the proposed system consists of the following steps. - Start the sensitivity calibration. When the driver purchases the vehicle or drives it for the first time, a specific "initial" position of the head is associated with the image to be displayed on the display (display image (crop)). This is done while the driver is moving their head, and ultimately, the image to be displayed when the driver places their head in the normal driving position is selected. The selection of the image on the display can be performed using the touch screen of the display device, for example, by clicking on virtual buttons or performing "touch and drag". Alternatively, and very advantageously, the selection of the image associated with the initial position of the driver's head can be performed using a gesture detector, which can detect the position of the driver's hand / finger, so the driver does not need to touch the screen of the display device. The sensor of the gesture detector can capture gestures and movements of any part of the driver's body (including the hand). The gesture detector transmits the movements of the finger or hand to the ECU until the driver selects a determined crop position on the display, and the ECU moves the crop. - When the image associated with the first (initial) position of the driver's head is selected, the driver moves their head to a second position. For example, it is the position of the head when the driver wants to overtake a vehicle. And, as before, the driver moves the crop on the display (using the touch screen, joystick, or preferably a gesture detector). - Finally, as an additional option for sensitivity calibration, the driver can select whether the change in the crop at the midpoint follows a linear (proportional) relationship or, preferably, a non-linear relationship (e.g., Table 1).

[0080] In one embodiment, the gesture detector is configured to perform "calibration". In the initial "calibration", the user / driver customizes the desired sensitivity. For this purpose, the gesture detection sensor detects a first position (e.g., of the head) and a second position (of the head). The driver selects / determines the FOV associated with the first position (determines the position of the crop) and selects / determines the second FOV of the second position (determines the adjustment position of the crop). The ECU determines the displacement of the displayed image / crop. During driving, the sensor detects the position of the driver's head relative to the initial reference position. By comparing the reference position (the first position) with the second position, the displayed image is moved. Since the driver can change the initial reference position of the head, both positions are dynamic rather than fixed values. The driver associates the first position with a predetermined FOV and the second position with another FOV.

[0081] Calibration is optional and is performed before driving. The user sits in the driver's seat and selects the first FOV (position of the crop) that he / she wants to see on the display in the first position, for example, in the normal driving posture. Then, a relationship is established between the first position and the associated first FOV (position of the crop) that the driver thinks he / she wants to see when in that position. Next, the driver changes the position of the body or head in the second position corresponding to the position when trying to change lanes to overtake the vehicle in front. Then, the driver selects the second FOV that he / she wants to be displayed on the display when in the second position. Therefore, the ECU has at least two positions of the driver's body or head and the respective first FOV and second FOV. At this time, the ECU establishes a (linear or non-linear) relationship based on these positions input by the driver. That is, when the driver is in an intermediate position during driving, the crop (displayed image) will move to an intermediate position between the position corresponding to the first FOV and the position corresponding to the second FOV. If this relationship is linear, the variation between the first position and the second position will be proportional, but it may not be proportional.

[0082] The reference position (first position) can be calculated as follows. i) A position preselected by the driver (i.e., where calibration is customized). Optionally, the driver is detected by face recognition and the driver's preferences / settings are loaded. ii) The head position measured during a pre-set time period (e.g., the first 20 seconds) (e.g., the average or mode value of the head position). iii) Discard head movements that are unlikely to be the reference position and permanently calculate a representative (average) value (or mode value) of the head displacement. That is, there is no time interval for collecting measurement values of the head position as in the previous points, and the average value or mode value (or other equivalent statistical parameters) is dynamically calculated at the current time. If the head movement at the current time corresponds to FOV adjustment for driving operation (e.g., the head moves forward), the head movement is discarded for the calculation of the reference position but considered as the "current position". Irregular or sudden head movements (head displacements deviating from the average / mode) are discarded. If an unstable head position is detected, the number of head detections per second can be increased. That is, if there are sudden movements or abnormal head positions (deviating from the average / mode), the number of detections can be increased to determine the reference position. Optionally, when there is such a sudden movement, it can be excluded from consideration for calculating the reference position.

[0083] To summarize, the ECU has several ways to determine (digital panning) the movement of the crop. (i) Based on the distance between the second position (current position) and the first position (reference position), the ECU refers to a look-up table that stores the detected positions. (ii) The ECU calculates the relative angular difference between the head and the display. The ECU is configured to move the display image / crop within the captured image based on the change in the angle of the head relative to the display. (iii) Comparison with a preset threshold: To move the crop, the detected head movement needs to exceed the threshold. This enables panning only when the driver actually moves their head intentionally, such as when overtaking another vehicle. Therefore, the displayed image (crop) will not move constantly (even slightly). (iv) Display based on the driver's line of sight (i.e., considering where the driver is looking). That is, the movement of the crop takes into account not only the movement of the driver's head but also whether the driver is looking at the display. By considering the driver's line of sight, it is possible to prevent the FOV from changing when the driver's head moves, for example, when scratching their back. Also, when the display is mounted on a head-mounted device (such as AR glasses), it is necessary to consider the driver's line of sight.

[0084] The sensor and gesture detector are, for example, a camera (it could also be radar or other technologies). Ideally, it is to always find representative points on the face and body. If it is a camera, since it detects facial expressions, especially the eyes, it can detect the midpoint between both eyes. If it is radar, it can find the contour of the face and detect the midpoint (head height ÷ 2, head width ÷ 2).

[0085] Furthermore, the movement of the crop, its speed (sensitivity as explained earlier), and / or the size of the crop can be changed according to the yaw angle, pitch angle, or roll angle, the turn signal, or the movement of the steering wheel.

[0086] The last example of the proposed camera monitoring system, CMS, is configured as follows. - The image capture means is arranged on an external attachment assembly of the vehicle (10) to capture a raw image (400) from an external FOV of the vehicle (10) that extends at least rearward of the vehicle (10) and includes a part of the external portion of the vehicle (10). Preferably, the image capture means comprises at least a camera associated with an external rearview mirror on the left or right side of the vehicle (10). - The electronic control unit (ECU) is connected to the image capture means, and the ECU (500) obtains a captured image (200) from the raw image (400). - The electronic display device is connected to the ECU to display an image including at least one image region (210) of the external FOV. At least one image region (210) is selected from the image (200) captured by the ECU, and the electronic display device is arranged inside the vehicle (10) and used by the driver (11) of the vehicle (10). The CMS further comprises a gesture detector configured to obtain at least one position of at least one part of the driver's body. The ECU is configured to adjust at least one image region (210) of the external FOV based on the obtained at least one position, and the display device is configured to display the adjusted image region (210) of the external FOV. The captured image (200) obtained from the raw image (400) comprises an image symmetric with respect to the vertical axis of the raw image (400). The adjusted image region (210) has an opening angle (α, α1, α2) defined as the angular range of the external FOV to be displayed. - The opening angle (α) is fixed regardless of the relative α angles (X1, X2). - The opening angles (α1, α2) increase with the increase of the relative α angles (X1, X2). Here, the relative angles (X1, X2) are defined as the angles between the position of a part of the driver's body and the following positions. (i) The image capture means arranged on the left or right side of the vehicle (10). (ii) The electronic display device.

[0087] The image area (210) to be adjusted is smaller than the captured image (200) and, optionally, is not centered within the captured image. Instead, the image area (210) displayed by the electronic display device is arranged inside, on the right side, or on the left side of the captured image, so that the driver is provided with a route on the display as follows. (i) Move the selected image area (210) within the captured image (200). (ii) Extend or retract the length of the additional extended view (220) outward along the horizontal axis of the captured image (200) according to the determined body movement in the driving direction. And in a specific example, the length (L2) of the route is at least 20% of the length (L1) of the image area (210, 210') before starting the adjustment within the captured image (200), and the length (L2) of the route is at least twice the length (L3) of the space defined on the other side of the image area (210, 210') before starting the adjustment within the captured image (200).

[0088] The image capturing means of the vehicle (10) for capturing the raw image (400) from the external view of the vehicle (10) may operate at least at 30 frames per second for all embodiments described in the present application, and optionally may operate at least at 60 frames per second.

[0089] In addition, in this specification, the term "comprising" and its derivatives (such as "including", "constituting", etc.) should not be understood in an exclusive sense, that is, these terms should not be construed as excluding the possibility that the things being described and defined may include further elements, steps, etc.

Claims

1. A camera monitoring system for an automobile (10), comprising: image capture means disposed in an external attachment assembly of the automobile (10) for capturing a raw image (400) from an external field of view (FOV) of the automobile (10), wherein the external field of view (FOV) extends at least to the outer rear of the automobile (10) and includes a part of an external portion of the automobile (10), and the image capture means includes an image sensor; the image capture means; an ECU (Electronic Control Unit) connected to the image capture means, the ECU (500) for obtaining a captured image (200) from the raw image (400); an electronic display device connected to the ECU and displaying an image including at least one image region (210) of the external field of view (FOV), wherein the at least one image region (210) is selected by the ECU from the captured image (200), and the electronic display device is disposed inside the automobile (10) and used by a driver (11) of the automobile (10); the electronic display device; a gesture detector configured to obtain at least one position of at least one part of the driver's body; characterized by the captured image (200) obtained by the ECU includes a symmetric image of the raw image (400) with respect to the vertical axis of the captured image (200), the symmetric image is generated by the image sensor of the image capture means, and the ECU is configured to move the at least one image region (210) in the captured image (200) according to a relative angle (X1, X2) defined as an angle between the obtained at least one position of the at least one part of the driver's body and the electronic display device; wherein, for a first relative angle (X1) defined with respect to a first obtained position and a second relative angle (X2) defined with respect to a second obtained position, when the second relative angle (X2) is greater than the first relative angle (X1); The ECU is configured to move the image area (210) at least to the left along the horizontal axis of the captured image (200) when the motor vehicle (10) is for right-hand traffic, and to move the image area (210) at least to the right along the horizontal axis of the captured image (200) when the motor vehicle (10) is for left-hand traffic. The external field of view (FOV) is adjusted based on the at least one position obtained, and the electronic display device is configured to display the adjusted external field of view (FOV) in the image area (210). Camera monitoring system.

2. The camera monitoring system according to claim 1, wherein the image area (210) includes an opening angle (α), the opening angle (α) is defined as an angular range of the displayed external field of view (FOV), and the opening angle (α) of the image area (210) is - fixed regardless of the relative angle (X1, X2) of the at least one part of the driver's body, or - increasing as the relative angle (X1, X2) of the at least one part of the driver's body increases, wherein the relative angle (X1, X2) is between the at least one position of the at least one part of the driver's body obtained and the following: (i) the image capturing means arranged on the left or right side of the motor vehicle (10); (ii) the electronic display device; defined therebetween. Camera monitoring system.

3. The camera monitoring system according to claim 1 or 2, wherein the image area (210) is smaller than the captured image (200) and is not at the center of the captured image (200). Camera monitoring system.

4. The camera monitoring system according to claim 3, wherein the image area (210) is located on the right or left inside the captured image (200). The image area (210) is located on the inside of the right side when the motor vehicle (10) is for right-hand traffic, and is located on the inside of the left side when the motor vehicle (10) is for left-hand traffic. Camera monitoring system.

5. The camera monitoring system according to any one of claims 1 to 4, wherein the gesture detector includes a sensor, - the sensor detects the first position of the at least one part of the driver's body. - The sensor detects the second position of the at least one part of the driver's body, wherein the first position is a reference position of the at least one detected part of the driver's body, the second position is the current position of the at least one detected part of the driver's body, the ECU is further configured to determine the movement of the at least one detected part of the driver's body based on a comparison of the second position with respect to the first position, at least along the driving direction, a camera monitoring system.

6. The camera monitoring system according to claim 5, wherein the ECU is further configured to move the image area (210) at least horizontally within the captured image (200) according to the determined movement of the driver's body in the driving direction. a camera monitoring system.

7. The camera monitoring system according to claim 5 or 6, wherein the ECU is further configured to move the image area (210) outward at least horizontally within the captured image (200) along the horizontal axis of the captured image (200) when the first relative angle (X1) is different from the second relative angle (X2). a camera monitoring system.

8. The camera monitoring system according to any one of claims 1 to 7, wherein the ECU is configured to divide the image area (210) into a first image area (210') and an additional extended view (220) for displaying the external field of view (FOV), the electronic display device is configured to display the additional extended view (220) adjacent to the first image area (210'). a camera monitoring system.

9. The camera monitoring system according to claim 8, wherein the ECU is configured to adjust the size of the image area (210) by gradually increasing or decreasing the length of the additional extended view (220) according to the at least one position of the at least one part of the driver's body obtained. The first image area (210') does not change in both size and arrangement. a camera monitoring system.

10. The camera monitoring system according to claim 8, wherein the ECU i) When a change in the at least one position of the at least one part of the driver's body is detected by the ECU, the additional extended view (220) is displayed in the first state. ii) When the ECU detects that the at least one position of the at least one part of the driver's body is equal to the initial position, the additional extended view (220) is not displayed. The electronic display device is configured to operate according to one of the two states. A camera monitoring system.

11. A camera monitoring system according to any one of claims 1 to 10, wherein the electronic display device is in a head-mounted device. A camera monitoring system.

12. A camera monitoring system according to any one of claims 1 to 11, wherein the ECU - selecting a first arrangement of the image region (210) in the captured image (200) corresponding to the first position of the at least one part of the driver's body; - selecting a second arrangement of the image region (210) in the captured image (200) corresponding to the second position of the at least one part of the driver's body; - defining the relationship between the selected first and second arrangements of the image region (210) as a linear or non-linear relationship; - selecting an intermediate position of the image region (210) according to the defined relationship to determine the sensitivity of the camera monitoring system, wherein the intermediate position corresponds to the position of the driver's body located between the first position and the second position; The camera monitoring system is configured to determine the sensitivity of the camera monitoring system by performing the above steps. A camera monitoring system.

13. A camera monitoring system according to any one of claims 1 to 12, wherein the image capturing means includes at least a camera associated with an external rearview mirror disposed on the left or right side of the automobile (10) and / or at least a camera associated with an internal rearview mirror, The field of view (FOV) associated with the external rearview mirror extends at least laterally outside the automobile (10) and includes a part of the external lateral surface portion of the automobile (10). A camera monitoring system.

14. A method for displaying an image of a camera monitoring system for a motor vehicle, comprising: - capturing a raw image (400) of an external field of view (FOV) of the motor vehicle (10) by image capture means arranged in an external attachment assembly of the motor vehicle (10), wherein the external field of view (FOV) extends at least to the outer rear of the motor vehicle (10) and includes a part of an external portion of the motor vehicle (10); - generating a symmetric image of the raw image (400); - providing the symmetric image to an ECU (Electronic Control Unit); - selecting, by the ECU, at least one image region (210) from a captured image (200), wherein the at least one image region (210) is smaller than the symmetric image and the captured image (200) obtained by the ECU includes the symmetric image of the raw image (400); - displaying the at least one image region (210) by an electronic display device; - obtaining, by a gesture detector, at least one position of at least one part of a driver's body; - moving, by the ECU, the at least one image region (210) within the captured image (200) according to a relative angle (X1, X2) defined as an angle between the obtained at least one position of the at least one part of the driver's body and the electronic display device, wherein, for a first relative angle (X1) defined with respect to an obtained first position and a second relative angle (X2) defined with respect to an obtained second position, when the second relative angle (X2) is greater than the first relative angle (X1), the ECU moves the image region (210) at least to the left along the horizontal axis of the captured image (200) if the motor vehicle (10) is for right-hand traffic and at least to the right along the horizontal axis of the captured image (200) if the motor vehicle (10) is for left-hand traffic; - adjusting the external field of view (FOV) based on the obtained at least one position. - The step of displaying, by the electronic display device, the adjusted external field of view (FOV) in the moved at least one image area (210); A method including the above.

15. The method according to claim 14, wherein the step of generating the symmetric image of the raw image (400) includes being executed by an image sensor of the image capturing means. A method.

16. The method according to claim 14 or 15, - The step of selecting, by the ECU, a first arrangement of the at least one image area (210) in the captured image (200) corresponding to the first position of the at least one part of the driver's body; - The step of selecting, by the ECU, a second arrangement of the at least one image area (210) in the captured image (200) corresponding to the second position of the at least one part of the driver's body; - The step of defining, by the ECU, the relationship between the selected first arrangement and the selected second arrangement of the at least one image area (210) as a linear relationship or a non-linear relationship; - The step of determining sensitivity by selecting an intermediate position of the at least one image area (210) according to the defined relationship by the ECU, wherein the intermediate position corresponds to the position of the driver's body that is intermediate between the first position and the second position, and the sensitivity corresponds to the speed at which the field of view (FOV) changes. A method further including the above.

Images