Indirect view system, method and interactive electronic rearview mirror system for vehicle

Abstract

The invention provides an indirect visual field system, an indirect visual field display method and an interactive electronic rearview mirror system applied to a vehicle. The indirect visual field system comprises a first camera device, a second camera device, a display and a processor. The first camera device is configured to capture a first image frame toward a first scene. The second camera device is configured to capture a second image frame toward a second scene. The display is coupled to the second camera device and configured to display a second image frame. The processor is configured to calculate a six-axis movement direction of a target point in a three-dimensional space through a current first image frame and another first image frame within a reference time. Calculating the displacement between the target point and the display in the six-axis movement direction; and changing the field of view of the second image frame displayed on the display according to the displacement amount. According to the invention, the driver can move at any position to watch the display or the electronic rearview mirror to obtain the required view range, and the effect that a picture is mutually connected with the driver is achieved.

Description

Technical Field

[0001] This invention relates to a field of view system and display method, and more particularly to an indirect field of view system and indirect field of view display method. Background Technology

[0002] Indirect vision devices are essential vehicle equipment to allow drivers to be aware of traffic conditions around their vehicles. They provide a means of observing traffic conditions, especially when the driver cannot directly observe the surrounding traffic with their eyes.

[0003] Indirect vision devices are devices that provide a clear view of the rear, sides, or front of a vehicle. These include devices that are integrated with traditional mirrors, cameras, or other devices that provide the driver with indirect vision information about the vehicle. These devices are collectively referred to as Exterior / Interior Monitoring Systems (CMS).

[0004] The definition of indirect visibility devices, such as those in the UN Regulations R46 / 04 series, categorizes vehicles into six types, from Category I to Category VII, each with a corresponding field of vision area. For example, Category I vehicles must provide indirect visibility devices that offer the following capabilities: Figure 1 The field of view shown.

[0005] Figure 1 In this vehicle 111, when the driver is seated, the driver's eye point 113 is located approximately 635 mm above the driver's seat and between the two eyes, which are approximately 65 mm apart. The line connecting the two eyes passes through the center of the driver's seat as specified by the manufacturer of the vehicle 111. The driver's field of vision in the vehicle 111 extends at least 60 meters behind the driver's eye point 113 and at least 20 meters laterally beyond this distance, forming a field of vision area 115.

[0006] In existing technology, when a driver wants to monitor the surrounding traffic conditions of vehicle 111, they often need to turn their head to check the left and right rearview mirrors, look up at the interior rearview mirror, or view the image provided by a camera device located at the rear of the vehicle. However, since the image reflected in the interior rearview mirror is reflected from inside the vehicle to outside, the driver's field of vision is still obstructed, and even turning the head cannot provide the required view. Furthermore, the rear camera device is fixed, and the image it provides is a fixed field of view, preventing the driver from freely viewing the desired angle. Therefore, providing the driver with a smooth field of vision is a problem encountered in this field. Summary of the Invention

[0007] One embodiment of the present invention is an indirect field-of-view system, including a first camera device, a second camera device, a display, and a processor. The first camera device is configured to capture a first image frame facing a first scene. The second camera device is configured to capture a second image frame facing a second scene. The display is coupled to the second camera device and configured to display the second image frame. The processor is coupled to the first camera device, the second camera device, and the display, and configured to perform the following operations: calculate the six-axis movement direction of a target point in three-dimensional space using the current first image frame and another first image frame within a reference time; calculate the displacement between the target point and the display in the six-axis movement direction; and change the field of view of the second image frame displayed on the display according to the displacement.

[0008] According to one embodiment of the present invention, wherein the processor calculates the displacement between the target point and the display in the six-axis movement direction, and is further configured to calculate the parallel plane displacement between the target point and the display in the six-axis movement direction.

[0009] According to one embodiment of the present invention, wherein the processor calculates the displacement between the target point and the display in the six-axis movement direction, and is further configured to calculate the current vertical plane displacement between the target point and the display in the six-axis movement direction.

[0010] According to one embodiment of the present invention, the processor changes the field of view of the second image frame displayed on the display according to the displacement amount, and is further configured to change the field of view of the second image frame displayed on the display based on a first average value of the parallel plane displacement amount.

[0011] According to an embodiment of the present invention, wherein the processor changes the field of view of the second image frame displayed on the display according to the displacement amount, and is further configured to store the current vertical displacement amount to the queue and update the standard deviation when the change in the current vertical displacement amount is less than the standard deviation of all the vertical displacement amounts in the queue or when the current first image frame is the Nth consecutive image frame in which the vertical displacement amount is greater than the standard deviation, where N is a positive integer; and to scale the field of view of the second image frame displayed on the display based on the second average value of all the vertical displacement amounts in the queue.

[0012] According to an embodiment of the present invention, the processor changes the field of view of the second image frame displayed on the display based on the first average value of the parallel plane displacement, and is further configured to control the display to change the field of view of the displayed second image frame toward a second direction when the target point moves in a first direction, wherein the first direction is opposite to the second direction and is either a vertical displacement direction or a horizontal displacement direction among the six-axis movement directions.

[0013] According to an embodiment of the present invention, after calculating the vertical plane displacement, the processor is further configured to directly store the current vertical plane displacement into the queue if it is determined that the number of vertical plane displacements stored in the queue is less than the capacity of the queue.

[0014] According to one embodiment of the present invention, after storing the current vertical plane displacement amount to the queue and updating the standard deviation, the processor is further configured to update the second average value of all the vertical plane displacement amounts in the queue, and update the scaling factor with the second average value, wherein the scaling factor is used to zoom in or out the field of view of the second image frame displayed on the display.

[0015] According to an embodiment of the present invention, the processor is configured to control the display to magnify the field of view of the second image frame displayed on the display when the target point is displaced forward perpendicular to the display; and to control the display to shrink the field of view of the second image frame displayed on the display when the target point is displaced backward perpendicular to the display.

[0016] According to one embodiment of the present invention, the processor is configured to control the display to display the second image frame with a preset field of view when the target point is not detected in the current first image frame.

[0017] Another embodiment of the present invention is an interactive electronic rearview mirror system for a vehicle, including a driver's camera device, a rearview camera device, and an electronic rearview mirror. The driver's camera device is disposed on the front side of the vehicle interior and captures a first image frame facing the driver's seat. The rearview camera device is disposed on the rear or outer side of the vehicle (at the position of the left and right outer rearview mirrors) and captures a second image frame facing the rear or outer scene of the vehicle. The electronic rearview mirror is disposed on the front side of the vehicle interior, coupled to the driver's camera device and the rearview camera device, and displays the second image frame facing the driver's seat. The driver camera device is configured to perform the following operations: calculate the six-axis movement direction of the driver's eye position in three-dimensional space using the current first image frame and another first image frame within a reference time; calculate the vertical and horizontal displacements of the driver's eye position in the six-axis movement direction; calculate the current vertical displacement between the driver's eye position and the electronic rearview mirror in the six-axis movement direction; when the change in the current vertical displacement is less than the standard deviation of all vertical displacements in the queue, or when the current first image frame is the Nth consecutive image frame with a vertical displacement greater than the standard deviation, store the current vertical displacement in the queue and update the standard deviation, where N is a positive integer; and adjust the field of view of the second image frame displayed on the electronic rearview mirror vertically or horizontally based on the first average value of the vertical or horizontal displacement, and zoom the field of view of the second image frame displayed on the electronic rearview mirror based on the second average value of all vertical displacements in the queue.

[0018] The indirect field of view system, indirect field of view display method, and interactive electronic rearview mirror system of the present invention improve the traditional rearview mirror by simulating the optical imaging principle, allowing drivers to move to any position to view the display or electronic rearview mirror without changing their existing driving habits, and also to see the required field of view, achieving the effect of mutual interaction between the image and the driver. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the indirect field of vision area applied to vehicle driving.

[0020] Figure 2 This is a schematic diagram illustrating the six-axis movement direction of the indirect vision system and the external scenery behind the vehicle according to an embodiment of the present invention.

[0021] Figure 3 This is a circuit block diagram of an indirect vision system according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the eye position and optical path of a driver before and after head movement, according to an embodiment of the present invention.

[0023] Figure 5 According to the present invention Figure 4A diagram illustrating the driver's field of vision before and after movement.

[0024] Figure 6 This is a schematic diagram illustrating optical reflection controlled by a vertical plane according to an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram illustrating the field of view variation controlled by the vertical plane according to another embodiment of the present invention.

[0026] Figure 8A and Figure 8B This is a schematic diagram illustrating a vertically controlled display screen according to an embodiment of the present invention.

[0027] Figure 9 This is a flowchart illustrating an indirect field-of-view display method according to an embodiment of the present invention.

[0028] Figure 10 This is a flowchart illustrating a method for further stabilizing the display in a parallel plane according to an embodiment of the present invention.

[0029] Figure 11 This is a flowchart illustrating a method for further stabilizing the display in the vertical plane in an indirect field of view display according to an embodiment of the present invention.

[0030] Figure 12 This is a schematic diagram illustrating queue data used for stable display in vertical plane control according to an embodiment of the present invention.

[0031] Figure 13 This is a circuit block diagram of an interactive electronic rearview mirror system according to an embodiment of the present invention.

[0032] The attached figures are labeled as follows:

[0033] 111: Vehicle 113: Driver's Eye Point

[0034] 115: Field of view area 211: Vehicles

[0035] 213: In-vehicle camera; 215: Electronic rearview mirror

[0036] 217: Driver; 219: Rearview camera

[0037] 231: Rearview camera field of view; 233, 235, 237: Scenery outside the vehicle.

[0038] 300: Indirect field of view system; 310: First camera device

[0039] 320: Second camera device; 330: Processor

[0040] 340: Monitor; 411, 431: Leftmost field of vision for the right eye

[0041] 413, 433: Rightmost visual field of the right eye; 417, 437: Rightmost visual field of the left eye

[0042] 419, 439: Binocular full field of vision 500: Image

[0043] 543, 545: Image frame

[0044] 611, 613, 621, 623, 711, 713, 721, 723: Optical paths

[0045] 800: Vehicle; 801, 803: Location

[0046] 810: First camera device; 820: Second camera device

[0047] 831, 833: Screen 1300: Interactive Electronic Rearview Mirror System

[0048] 1310: Driver's camera device; 1320: Rear-view camera device

[0049] 1340: Electronic rearview mirror OD, N-OD: Right eye position

[0050] OE, N-OE: Left eye position ID: Right eye virtual image

[0051] IE: Left eye virtual image I: Binocular virtual image

[0052] E: Monitor

[0053] S910~S940, S1010~S1020, S1110~S1150: Steps Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments, so that those skilled in the art can better understand the present invention and implement it accordingly. However, the embodiments are not intended to limit the present invention.

[0055] Figure 2 This is a schematic diagram illustrating the six-axis movement direction of the indirect vision system and the external scenery behind the vehicle according to an embodiment of the present invention.

[0056] like Figure 2 As shown, the six-axis direction is plotted with the electronic rearview mirror 215 as the coordinate system reference, with the X-axis, Y-axis and Z-axis drawn. The X-axis direction is the left-right movement direction (e.g., lateral movement) of the driver 217 relative to the electronic rearview mirror 215, the Y-axis direction is the forward-backward movement direction of the driver 217 relative to the electronic rearview mirror 215, and the Z-axis direction is the up-down extension direction of the driver 217 relative to the electronic rearview mirror 215.

[0057] In this paper, the plane formed by the X-axis and Z-axis is parallel to the electronic rearview mirror 215 and is called the parallel plane; the displacement of the driver in the X-axis or Z-axis direction is called the parallel plane displacement. The Y-axis direction is perpendicular to the electronic rearview mirror 215 and is called the vertical plane; the displacement of the driver in the Y-axis direction is called the vertical plane displacement.

[0058] The vehicle 211 is equipped with an in-vehicle camera 213, for example, mounted on the windshield, and the camera 213 is angled towards the driver's face. The vehicle 211 is also equipped with an electronic rearview mirror 215, for example, mounted on the windshield or the roof. The driver 217 can view the rear and surrounding environment of the vehicle 211 through the electronic rearview mirror 215. A rear-view camera 219 (not limited to the interior or exterior of the vehicle 211) is located at the rear of the vehicle 211. The rear-view camera 219 is used to capture images within the rearview camera's field of view 231, such as images of external objects 233, 235, and 237.

[0059] In one embodiment, the image from the rear-view lens 231 of the rear-view camera 219 can be displayed on the electronic rearview mirror 215. The driver 217 can view the display screen on the electronic rearview mirror 215 to understand the environmental conditions behind and around the vehicle 211.

[0060] The in-vehicle camera 213 can be a depth camera, a time-of-flight (ToF) camera, or any camera that can calculate and obtain the relative distance and position information of the driver 217.

[0061] The indirect vision system of the present invention will cause the electronic rearview mirror 215 to display image information that is useful to the driver 217 or that the driver expects to see, as the driver 217 moves (e.g., the head moves forward, backward, left, right or up and down).

[0062] Figure 3 This is a circuit block diagram of an indirect vision system according to an embodiment of the present invention.

[0063] The indirect viewing system 300 includes a first camera device 310, a second camera device 320, a processor 330, and a display 340.

[0064] The first camera device 310 is configured to capture a first image frame facing a first scene. The first camera device 310 may be installed inside the vehicle and angled towards the driver. In this embodiment, the first image frame is a scene (or first scene) that includes the driver's face, head, upper body, or hand gestures.

[0065] The second camera device 320 is configured to capture a second image frame facing the second scene. The second camera device 320 is located at the rear of the vehicle, and can be located inside or outside the vehicle. Its angle is such that it can capture the view of all lanes behind the vehicle or the field of view within the visible area behind the vehicle. In this embodiment, the second image frame is a view (or second scene) covering all lanes behind the vehicle.

[0066] In another embodiment, the second camera device 320 may be positioned on the left or right side of the vehicle and oriented at an angle that can capture the left or right side of the vehicle, so that the second scene covers the left or right side of the vehicle.

[0067] For the sake of simplicity, this article uses the example of the second camera device 320 being installed at the rear of the vehicle for illustration, but this invention is not limited to this.

[0068] Display 340 is coupled to second camera device 320 and configured to display second image frames. Display 340 can be installed inside a vehicle to provide a view of the second scene for the driver.

[0069] The processor 330 is coupled to the first camera device 310 and the second camera device 320. In another embodiment, the processor 330 may also be the processing chip of the first camera device 310 or a vehicle computer.

[0070] The processor 330 is configured to calculate the six-axis movement direction of the target point in three-dimensional space using the current first image frame and another first image frame within a reference time, calculate the displacement between the target point and the display 340 in the six-axis movement direction, and change the field of view of the second image frame displayed on the display 340 according to the displacement.

[0071] The reference time is, for example, 66 milliseconds. For instance, the first camera device 310 captures a first image frame every 66 milliseconds. The first camera device 310 captures the current first image frame at second 0 and another first image frame at second 65. It is worth noting that the reference time of this invention is not limited to 66 milliseconds.

[0072] The six-axis movement direction is, for example, Figure 2 The instructions specify the left-right, forward-backward, and up-down movement directions.

[0073] The target point is, for example, the midpoint between the driver's eyes (or the eye point position). In this embodiment, the processor 330 uses the current first image frame and another first image frame within a reference time to calculate the six-axis movement direction of the driver's eye point position in three-dimensional space.

[0074] In one embodiment, the processor 330 or the first camera device 310 may execute an artificial intelligence algorithm to identify and detect the relative position of a target point in the first image frame relative to the display 340.

[0075] The processor 330 calculates the parallel plane displacement between the target point and the display 340 in the six-axis movement direction using two first image frames. The parallel plane displacement is, for example, the displacement of the target point in the X-axis or Z-axis direction. Figure 2 The processor 330 then calculates the displacement of the object. Next, based on the calculated parallel plane displacement, the processor 330 changes the field of view of the second image frame displayed on the display 340. For example, the processor 330 calculates the parallel plane displacement of the target object from the current first image frame and another first image frame, and then calculates another parallel plane displacement of the target object from another first image frame and the next first image frame. This process continues, and the processor 330 calculates multiple parallel plane displacements. Next, the processor 330 calculates a first average of the multiple parallel plane displacements, and then changes the field of view of the second image frame displayed on the display 340 based on the first average.

[0076] In this embodiment, when the target point moves in a first direction, the processor 330 controls the display 340 to change the field of view of the displayed second image frame in a second direction. The first and second directions are opposite, and are either vertical or horizontal displacement directions within a six-axis movement direction. For example, when the target point moves upward relative to the display 340, the field of view of the second image frame displayed on the display 340 moves downward; when the target point moves downward relative to the display 340, the field of view of the second image frame displayed on the display 340 moves upward; when the target point moves to the left relative to the display 340, the field of view of the second image frame displayed on the display 340 moves to the right; and when the target point moves to the right relative to the display 340, the field of view of the second image frame displayed on the display 340 moves to the left.

[0077] On the other hand, the processor 330 calculates the vertical plane displacement between the target point and the display 340 in the six-axis movement direction using two first image frames. The vertical plane displacement is, for example, the displacement of the target point in the Y-axis direction (…). Figure 2 The processor 330 then adjusts the field of view of the second image frame displayed on the display 340 based on the calculated vertical plane displacement.

[0078] In detail, processor 330 calculates the current vertical plane displacement from the current first image frame and another first image frame. Processor 330 determines whether the change in the current vertical plane displacement is less than the standard deviation of all vertical plane displacements stored in a queue, or determines whether the current first image frame is the Nth consecutive image frame whose vertical plane displacement is greater than the standard deviation. The queue can be a memory (not shown) stored in processor 330 or the first camera device 310, and the queue stores multiple vertical plane displacements, for example, 10.

[0079] In one embodiment, when the processor 330 determines that the change in the current vertical plane displacement is less than the standard deviation of all vertical plane displacements in the queue, it stores the current vertical plane displacement in the queue to update the queue data, and recalculates the standard deviation based on the updated queue data as the updated standard deviation.

[0080] In another embodiment, when the processor 330 determines that the current first image frame is the Nth consecutive image frame in which the vertical plane displacement is calculated to be greater than the standard deviation, it stores the current vertical plane displacement to a queue to update the queue data, and updates the standard deviation based on the updated queue data, where N is a positive integer. Here, the value of N can be 5, meaning that the vertical plane displacement is calculated to be greater than the standard deviation from 5 consecutive image frames.

[0081] After updating the queue data, the processor 330 calculates a second average of all vertical plane displacements and scales the field of view of the second image frame displayed on the display based on the second average.

[0082] In one embodiment, when the target point (e.g., the driver's eye point) is displaced forward in a direction perpendicular to the display 340, the processor 330 controls the display 340 to magnify the field of view of the second image frame displayed on the display 340. Conversely, when the target point is displaced backward in a direction perpendicular to the display 340, the processor 330 controls the display 340 to narrow the field of view of the second image frame displayed on the display 340.

[0083] In detail, processor 330 calculates the average (or second average) of all vertical plane displacements in the queue. Then, processor 330 updates a scaling factor based on this second average, which is used to magnify or reduce the field of view of the second image frame displayed on display 340. For example, if the second average is less than the previous second average, it means that the driver's eye position is generally closer to display 340, and processor 330 magnifies the field of view of the second image frame displayed on display 340 based on the second average. In this case, the field of view of the second image frame on display 340 is magnified, meaning that people or objects in the displayed image are magnified and more detailed. Conversely, if the second average is greater than or equal to the previous second average, it means that the driver's eye position is generally farther from display 340, and processor 330 reduces the field of view of the second image frame displayed on display 340 based on the second average. In this case, the field of view of the second image frame on display 340 is reduced, meaning that people or objects in the displayed image are reduced and a larger area of ​​the environment is visible.

[0084] In one embodiment, after calculating the vertical displacement between the target point and the display 340, the processor 330 checks the number of vertical displacements stored in the queue. If it determines that the number of vertical displacements stored in the queue is less than the queue's capacity, the processor 330 directly stores the current vertical displacement into the queue (e.g., in a First-In-First-Out (FIFO) storage mode). Then, the processor 330 does not perform any further vertical displacement checks but returns to the operation of calculating the vertical displacement between the target object and the display 340 in the first image frame.

[0085] On the other hand, the processor 330 will detect whether the driver is sitting in the driver's seat. If no target point is detected in the current first image frame, it means that there is no one in the driver's seat. Then the processor 330 controls the display 340 to display the second image frame with a preset field of view. That is, the display 340 returns to the most preset display field of view and returns to the initial screen with no adjusted viewing angle.

[0086] In one embodiment, if the frame per second (FPS) of the first camera device 310 is less than a preset value (e.g., 20 FPS), the processor 330 disables the zoom function for the field of view of the second image frame.

[0087] Thus, the indirect vision system of the present invention can achieve the corresponding driver's eye point to the field of vision required by the moving driver, and through the average stabilization mechanism, allow the driver to see a relatively gentle field of vision movement, avoid the screen changes too drastically, and maintain driving safety.

[0088] To facilitate understanding of the operation of this invention, please refer to... Figure 4 . Figure 4 This is a schematic diagram illustrating the eye position and optical path of a driver before and after head movement, according to an embodiment of the present invention.

[0089] The driver sits in the driver's seat (not shown) and views display E (e.g., a rearview mirror). During time T, the driver's eyes view display E from positions OD (right eye) and OE (left eye). Under simulated optical paths, the right eye sees the right-eye virtual image ID, and the left eye sees the left-eye virtual image IE. The overlapping area visible to both eyes is the binocular virtual image I. On the other hand, based on optical imaging principles, the right eye at position OD can view the leftmost field of view 411 of the right eye through the leftmost edge of display E, and the rightmost field of view 413 of the right eye through the rightmost edge of display E; similarly, the left eye at position OE can view the leftmost field of view 415 of the left eye through the leftmost edge of display E, and the rightmost field of view 417 of the left eye through the rightmost edge of display E. Thus, the driver's eyes can see the full binocular field of view 419, encompassing the area between the leftmost field of view 411 of the right eye and the rightmost field of view 417 of the left eye.

[0090] On the other hand, at time T+1, the driver moves their head to the left. At this time, the driver's eyes move to the right eye position N-OD and the left eye position N-OE. After the movement, based on the simulated optical path of optical imaging principles, the right eye position N-OD can see the leftmost field of view 431 of the right eye through the leftmost end of the display E, and the rightmost field of view 433 of the right eye through the rightmost end of the display E; the left eye position N-OE can see the leftmost field of view 435 of the left eye through the leftmost end of the display E, and the rightmost field of view 437 of the left eye through the rightmost end of the display E. At this time, the driver's eyes can see the full field of view 439, that is, the field of view covering the area between the leftmost field of view 431 of the right eye and the rightmost field of view 437 of the left eye.

[0091] Please refer to Figure 5 . Figure 5 According to the present invention Figure 4 A diagram illustrating the driver's field of vision before and after movement.

[0092] exist Figure 5 The example image 500 can be generated by... Figure 3 The images captured by the second camera device 320, and by Figure 3 The display 340 displays all or part of the screen 500 in response to the movement of the driver's head position.

[0093] As mentioned above Figure 4In this embodiment, during time T, the image displayed on the display 340 is image frame 543 (solid line frame). During time T+1, because the driver's head moves to the left, the field of view of the display 340 shifts to the right, meaning the image displayed on the display 340 is image frame 545 (dashed line frame). In other words, the driver can see the expected surrounding environment on the display 340 as their head moves. Similarly, if the driver's head moves to the right, the field of view of the display 340 shifts to the left; if the driver's head moves upward, the field of view of the display 340 shifts downward; and if the driver's head moves downward, the field of view of the display 340 shifts upward. For the sake of brevity, similar images will not be repeated here.

[0094] Figure 6 This is a schematic diagram illustrating optical reflection controlled by a vertical plane according to an embodiment of the present invention.

[0095] Compared to Figure 4 and Figure 5 An example of a driver's head moving up, down, left, and right. Figure 6 This is an example of a driver's head moving back and forth. (Example:) Figure 6 As shown, at time T, the driver views display E (e.g., a rearview mirror). Under simulated optical paths, the optical paths visible to the left and right sides are optical paths 611 and 613, respectively. At time T+1, as the driver's body moves forward (e.g., away from the back of the driver's seat), the head also moves forward accordingly. At this time, the optical paths visible to the left and right sides are optical paths 621 and 623, respectively. From... Figure 6 As can be seen, because the driver's head moves forward, the driver's field of vision can see the environmental image to the right.

[0096] Figure 7 This is a schematic diagram illustrating the field of view variation controlled by the vertical plane according to another embodiment of the present invention.

[0097] Compared to Figure 6 The driver's position is in front of the driver on the side opposite to display E. Figure 7 The driver's position is shown in front of the center of display E. For example... Figure 7 As shown, at time T, the driver views display E (e.g., a rearview mirror). Under simulated optical paths, the optical paths visible to the left and right sides are optical paths 711 and 713, respectively. At time T+1, as the driver's body moves forward (e.g., away from the back of the driver's seat), the head also moves forward accordingly. At this time, the optical paths visible to the left and right sides are optical paths 721 and 723, respectively. From... Figure 6As can be seen, because the driver's head moves forward and is positioned in the middle of display E, the driver's field of vision can expand to the left and right simultaneously to view a wider range of environmental images.

[0098] To better understand Figure 6 and Figure 7 For instructions on how it works, please refer to [link / reference]. Figure 8A and Figure 8B . Figure 8A and Figure 8B This is a schematic diagram illustrating a vertically controlled display screen according to an embodiment of the present invention.

[0099] like Figure 8A and Figure 8B As shown, the driver is in vehicle 800, and the display E is positioned in a position visible to the driver (e.g., on the windshield). In this embodiment, a first camera device 810 is positioned above the display E and faces the driver's seat to capture images, and a second camera device 820 is positioned at the rear of vehicle 800 and faces the rear of vehicle 800 to capture images. The first camera device 810 is, for example, a... Figure 3 The first camera device 310 and the second camera device 820 described are, for example, Figure 3 The second camera device 320 is described.

[0100] At Figure 8A In the process, at time T, the first camera device 810 captures an image of the driver at position 801. The indirect field-of-view system calculates the displacement of the driver's head relative to the display E and controls the display E to show the image captured by the second camera device 820. At this time, the driver sees image 831 on the display E. Next, at time T+1, the indirect field-of-view system determines that the driver has moved from position 801 to position 803, that is, the driver is moving closer to the display E (i.e., vertical displacement). At this time, the driver sees image 833 on the display E. Because the driver is moving closer to the display E, compared to image 831, the driver can see a more rightward field of view in image 833.

[0101] Please refer to Figure 9 This is a flowchart illustrating an indirect field-of-view display method according to an embodiment of the present invention.

[0102] Figure 9 The steps of the indirect field-of-view display method can be derived from... Figure 3 The indirect vision system 300 is used to execute this.

[0103] In step S910, the first camera device 310 captures a first image frame toward the first scene and the second camera device 320 captures a second image frame toward the second scene.

[0104] In step S920, the processor 330 calculates the six-axis movement direction of a target point in three-dimensional space using the current first image frame and another first image frame within a reference time.

[0105] In step S930, the processor 330 calculates the displacement between the target point and the display in the six-axis movement direction.

[0106] In step S940, the processor 330 changes the field of view of the second image frame displayed on the display according to the displacement.

[0107] The indirect field of view system 300 and indirect field of view display method of the present invention can provide the driver with the surrounding environment image that the driver wants to see according to the driver's movement, thus solving the problem that the field of view of the traditional camera device is fixed, so that no matter how the driver moves, the angle relative to the display screen cannot change the field of view in the picture.

[0108] The indirect field of view system 300 and indirect field of view display method of the present invention provide the surrounding environment images that the driver wants to see according to the driver's movement. Furthermore, the present invention also proposes an indirect field of view system 300 and indirect field of view display method that can stabilize the image.

[0109] Please refer to Figure 10 This is a flowchart illustrating a method for further stabilizing the display in a parallel plane according to an embodiment of the present invention.

[0110] Figure 10 The steps of the indirect field-of-view display method can be derived from... Figure 3 The indirect vision system 300 is used to execute this.

[0111] Controlled in the parallel plane (e.g., up / down or left / right direction, or) Figure 2 In the process of controlling the viewpoint of the screen (in the X and Z axes), if the processor 330 directly controls the viewing angle based on the displacement of the driver, it may cause discomfort for the driver. The following presents a process for controlling a stable screen in a parallel plane.

[0112] In step S1010, the processor 330 calculates the amount of parallel plane displacement between the target point and the display 340 in the six-axis movement direction.

[0113] The driver's body uses the spine and hips as support points. When the body moves naturally, the head's movement in space will inevitably include displacement in six axes. When the driver's eye point position (the midpoint between the eyes) is taken as the target point, the processor 330 calculates the displacement between the target point and the display 340 in the X and Z axes.

[0114] Next, in step S1020, the processor 330 changes the field of view of the second image frame displayed on the display 340 based on a first average value of the parallel plane displacement.

[0115] In one embodiment, the processor 330 takes the two latest image frames to calculate the displacement in the X-axis and Z-axis directions, and calculates the average value of the displacement in the X-axis direction and / or the average value of the displacement in the Z-axis direction (here, the average value of the displacement in the X-axis and Z-axis directions is referred to as the first average value). The first average value is used to change the field of view of the second image frame displayed on the display 340, that is, the field of view of the environmental conditions behind the vehicle.

[0116] Please refer to Figure 11 This is a flowchart illustrating a method for further stabilizing the display in the vertical plane in an indirect field of view display according to an embodiment of the present invention.

[0117] Figure 11 The steps of the indirect field-of-view display method can be derived from... Figure 3 The indirect vision system 300 is used to execute this.

[0118] Control in the vertical plane (e.g., forward and backward direction, or) Figure 2 In the process of controlling the viewpoint (in the Y-axis direction), if the processor 330 directly controls the viewing angle of the screen based on the displacement of the driver's forward and backward movement, it may cause discomfort for the driver. The following presents a process for controlling a stable screen in the vertical plane according to the present invention.

[0119] In step S1110, the processor 330 calculates the current vertical plane displacement between the target point and the display in the six-axis movement direction.

[0120] In one embodiment, when the driver's eye position (the midpoint between the two eyes) is taken as the target point, the processor 330 calculates the displacement between the target point and the display 340 in the Y-axis direction, i.e., the vertical plane displacement.

[0121] In step S1120, the processor 330 determines whether the number of vertical plane displacement data stored in the queue is greater than or equal to the capacity of the queue.

[0122] The queue is used to store multiple vertical plane displacement values. The queue can be a memory stored in the processor 330 or the first camera device 310. The indirect field of view system 300 can set a preset capacity for the queue as the upper limit of the number of vertical plane displacement values ​​that the queue can store.

[0123] In step S1120, if the number of vertical plane displacement data stored in the queue is less than the queue's capacity, it means that the number of frames in the first image frame so far is slightly less, and stable averaging control is not yet required, then step S1150 is executed. Otherwise, step S1130 is executed.

[0124] In step S1130, the processor 330 determines whether the change in the current vertical plane displacement is less than the standard deviation, or whether the current first image frame is the Nth consecutive first image frame in which the vertical plane displacement is greater than the standard deviation.

[0125] After the first camera device 310 has captured images for a period of time, multiple first image frames have been generated. The processor 330 calculates multiple vertical plane displacements from the multiple first image frames and can calculate the standard deviation from all the vertical plane displacements stored in the queue.

[0126] In step S1130, the processor 330 compares the current vertical plane displacement with the standard deviation to assess the change in the current vertical plane displacement. If the current vertical plane displacement is less than the standard deviation, it means that the driver's vertical plane displacement is a routine action in focused driving behavior (i.e., excluding situations such as emergency braking or deliberately turning the head to talk), then step S1140 is executed.

[0127] On the other hand, in step S1130, the processor 330 determines that the vertical plane displacement calculated from the current first image frame is greater than the standard deviation, and further, if this current first image frame is the Nth consecutive first image frame in which the calculated vertical plane displacement is greater than the standard deviation, indicating that the driver made the change while concentrating on driving, then step S1140 is executed. In this embodiment, N is a positive integer, such as 5.

[0128] It is worth mentioning that step S1130 is optional; if the processor 330 evaluates that at least one condition is met, it will execute step S1140. Otherwise, the process returns to step S1110, continuously monitoring the driver's head movement.

[0129] On the other hand, the judgment condition used by processor 330: the Nth consecutive first image frame can be replaced by time. For example, the judgment condition can be the 5th consecutive first image frame; or, the judgment condition used by processor 330 can be 0.25 seconds, and then the sequence of first image frames is calculated by the number of frames per second of the camera device (which varies depending on the camera device).

[0130] In step S1140, the processor 330 stores the current vertical plane displacement to the queue to update the queue data, updates the second average and standard deviation of all vertical plane displacements, and scales the field of view of the second image frame displayed on the display with the updated second average of the vertical plane displacements.

[0131] The processor 330 updates the queue with the vertical plane displacement calculated from the current first image frame, for example, in a first-in-first-out (FIFO) order, and incorporates the calculation of the average and standard deviation of the vertical plane displacement to obtain updated average and standard deviation, which are provided as the judgment benchmark for the next round. Furthermore, the processor 330 uses the updated average of the vertical plane displacement to scale the field of view of the second image frame displayed on the monitor.

[0132] As described in step S1120 above, if the number of data in the queue is insufficient, step S1150 is executed, whereby the processor 330 stores the vertical displacement amount in the queue and returns to step S1110 to continuously monitor the driver's head movement.

[0133] This invention calculates a first average of multiple parallel plane displacements, resulting in a more even and stable change in the field of view of the second image frame displayed on the display 340, preventing the driver from experiencing severe screen shake when viewing the display 340. Furthermore, by calculating a second average of multiple vertical plane displacements, this invention correspondingly reduces or enlarges the field of view on the display 340, achieving an even and stable effect in changing the field of view.

[0134] Figure 12 This is a schematic diagram illustrating queue data used for stable display in vertical plane control according to an embodiment of the present invention.

[0135] Figure 12 Provides the standard deviation of the data in the queue and the four states of the current data.

[0136] In state (a), the current data falls within the standard deviation range of the data in the queue, which means that the driver's head has not moved significantly and can be included in the standard deviation update (such as the first judgment condition of step S1130 being met and proceeding to step S1140).

[0137] In state (b), if the current data falls outside the standard deviation range of the data in the queue, it means that the current data cannot be used to include the driver's focused driving behavior as a sample and should be excluded as noise (e.g., the first judgment condition of step S1130 is not met and the process returns to step S1110).

[0138] In state (c), the current data and several consecutive previous data points fall outside the standard deviation range of the data in the queue, indicating that the driver's focused driving behavior has changed and should not be considered noise (as the second judgment condition in step S1130 is met and the process proceeds to step S1140). At this point, the current data is used to calculate the second average and control the zoom of the field of view displayed on the monitor 340. After updating the standard deviation, as shown in state (d), the current data and the new data points are included within the standard deviation range.

[0139] In this way, the indirect field-of-view system 300 and the indirect field-of-view display method of the present invention can simultaneously ensure that the image providing the corresponding field of view after the driver moves, as well as the image providing different fields of view, remains stable during the process of changing fields of view. This avoids the problem of the driver being distracted by the unstable image and unable to immediately interpret the information in the image due to excessively drastic changes in the image. The present invention can provide the driver with a comfortable viewing experience, allowing them to focus more on traffic conditions and increasing driving safety.

[0140] Figure 13 This is a circuit block diagram of an interactive electronic rearview mirror system according to an embodiment of the present invention.

[0141] The interactive electronic rearview mirror system 1300 includes a driver camera device 1310, a rear camera device 1320, and an electronic rearview mirror 1340. The electronic rearview mirror 1340 is coupled to both the driver camera device 1310 and the rear camera device 1320.

[0142] The driver's camera device 1310 is located on the front side of the vehicle interior (not shown) and captures the first image frame facing the driver's seat.

[0143] The rear camera device 1320 is installed at the rear of the vehicle and captures a second image frame facing the scene behind the vehicle.

[0144] The electronic rearview mirror 1340 is located on the front side of the vehicle interior and displays a second image frame facing the driver's seat.

[0145] In one embodiment, the driver camera device 1310 is configured to perform the following operations:

[0146] Operation (a) calculates the six-axis movement direction of the driver's eye position in three-dimensional space using the current first image frame and another first image frame within a reference time. The six-axis movement direction is, for example,... Figure 2 The X, Y, and Z axes are the directions of movement.

[0147] Operation (b) calculates the vertical displacement of the driver's eye position in the six-axis movement directions (e.g., Figure 2 The Z-axis movement direction and the left and right displacement (e.g., the left and right displacement) Figure 2(X-axis movement direction).

[0148] Operation (c) calculates the current vertical displacement between the driver's eye position and the electronic rearview mirror 1340 in the six-axis movement direction (e.g., Figure 2 (Y-axis movement direction).

[0149] Operation (d) involves storing the current vertical plane displacement to the queue to update the queue data and the standard deviation when the change in the current vertical plane displacement is less than the standard deviation of all vertical plane displacements in the queue, or when the current first image frame is the Nth consecutive image frame whose vertical plane displacement is greater than the standard deviation. In one embodiment, N is a positive integer, such as the value 5.

[0150] Operation (e) involves changing the field of view of the second image frame displayed on the electronic rearview mirror 1340 vertically or horizontally based on a first average of the vertical or horizontal displacement. It also involves scaling the field of view of the second image frame displayed on the electronic rearview mirror 1340 based on a second average of all vertical plane displacements in the updated queue.

[0151] In terms of horizontal control operation, when the head moves to the right relative to the electronic rearview mirror 1340, the driver can see the display screen of the electronic rearview mirror 1340 move to the left accordingly; when the head moves to the left relative to the electronic rearview mirror 1340, the driver can see the display screen of the electronic rearview mirror 1340 move to the right accordingly; when the head moves upward relative to the electronic rearview mirror 1340, the driver can see the display screen of the electronic rearview mirror 1340 move downward accordingly; when the head moves downward relative to the electronic rearview mirror 1340, the driver can see the display screen of the electronic rearview mirror 1340 move upward accordingly.

[0152] In terms of vertical control operation, when the head moves forward relative to the electronic rearview mirror 1340, the driver can see that the display screen of the electronic rearview mirror 1340 has a wider field of view than the original display screen (i.e., it is wider on the left and right sides, and the field of view is further to the left and right), and the objects in the field of view are smaller (because more information is contained while the screen size remains the same); when the head moves backward relative to the electronic rearview mirror 1340, the driver can see that the display screen of the electronic rearview mirror 1340 has a narrower field of view than the original display screen, and the objects in the field of view are larger.

[0153] In summary, the indirect field of view system, indirect field of view display method, and interactive electronic rearview mirror system of the present invention improve upon the traditional rearview mirror by simulating the principle of optical imaging. This allows drivers to move to any position to view the display or electronic rearview mirror without changing their existing driving habits, and to see the required field of view, achieving a mutual effect between the image and the driver.

[0154] Furthermore, the second camera device and the rear camera device of the present invention are located at the rear of the vehicle, ensuring that the visible range is not obstructed by the vehicle's beams and pillars. By moving the head and changing the position of the target point or eye point through the indirect field of view display method of the present invention, the field of view can be adjusted in a timely manner, which is beneficial for identifying the driving blind spots around the vehicle.

[0155] Furthermore, the present invention proposes an average stabilization control mechanism in the time domain that can prevent viewing discomfort caused by the driver's head constantly shaking, which in turn causes the screen's field of vision to shake. Even with slight head movements, the image will not be over-adjusted, providing the driver with a more comfortable viewing experience.

[0156] In addition, the technology of this invention is applicable to the United Nations Vehicle Safety Regulation UN-R46 / 04 for various types of vehicles, and also to the Chinese standard GB-T / 15084 for the performance and installation requirements of indirect vision devices for motor vehicles.

[0157] The above description is merely a specific example of this case and does not limit the scope of the rights in this case. Therefore, all equivalent changes made using the content of this case are similarly included within the scope of this case and are hereby stated.

Claims

1. An indirect vision system characterized in that, comprising: a first camera configured to capture first image frames toward a first scene; a second camera configured to capture second image frames toward a second scene; a display coupled to the second camera and configured to display the second image frames; and a processor coupled to the first camera and the second camera and configured to: calculate a six-axis movement direction of a target point in a three-dimensional space by a current first image frame and another first image frame within a reference time; calculate a displacement amount between the target point and the display in the six-axis movement direction; and change a field of view of the second image frames displayed on the display according to the displacement amount. The processor calculates the displacement amount between the target point and the display in the six-axis movement direction and is configured to: calculate a parallel plane displacement amount between the target point and the display in the six-axis movement direction.

2. The indirect vision system of claim 1, wherein, The processor calculates the displacement amount between the target point and the display in the six-axis movement direction and is configured to: calculate a current perpendicular plane displacement amount between the target point and the display in the six-axis movement direction.

3. The indirect vision system of claim 1, wherein, The processor changes the field of view of the second image frames displayed on the display according to the displacement amount and is configured to: change the field of view of the second image frames displayed on the display based on a first average value of the parallel plane displacement amounts.

4. The indirect vision system of claim 2, wherein, The processor changes the field of view of the second image frames displayed on the display according to the displacement amount and is configured to: store the current perpendicular plane displacement amount to a queue and update a standard deviation of all the perpendicular plane displacement amounts in the queue when a variation of the current perpendicular plane displacement amount is less than the standard deviation or the current first image frame is a continuous Nth image frame with a perpendicular plane displacement amount greater than the standard deviation, wherein N is a positive integer; and 5. The indirect vision system of claim 3, wherein, scale the field of view of the second image frames displayed on the display based on a second average value of all the perpendicular plane displacement amounts in the queue. The processor changes the field of view of the second image frames displayed on the display based on the first average value of the parallel plane displacement amounts and is configured to: control the display to change the field of view of the second image frames displayed toward a second direction when the target point moves toward a first direction, wherein the first direction and the second direction are opposite directions and are up-down displacement directions or left-right displacement directions in the six-axis movement direction. The processor, after calculating the perpendicular plane displacement amount, is configured to:

6. The indirect vision system of claim 4, wherein, store the current perpendicular plane displacement amount to the queue if it is determined that a number of the perpendicular plane displacement amounts stored in the queue is less than a capacity of the queue. The processor, after storing the current perpendicular plane displacement amount to the queue and updating the standard deviation, is configured to:

7. The indirect vision system of claim 5, wherein, update the second average value of all the perpendicular plane displacement amounts in the queue and update a scaling rate with the second average value, wherein the scaling rate is used to enlarge or reduce the field of view of the second image frames displayed on the display. The processor is configured to:

8. The indirect vision system of claim 5, wherein, ​ ​ 9. The indirect vision system of claim 5, wherein, ​ when the target point is displaced forward perpendicularly to the display, controlling the display to zoom in a field of view of the second image frame displayed on the display; and when the target point is displaced backward perpendicularly to the display, controlling the display to zoom out the field of view of the second image frame displayed on the display.

10. The indirect vision system of claim 1, wherein, The processor is configured to: when the target point is not detected in the current first image frame, controlling the display to display the second image frame with a preset field of view.

11. An indirect vision display method to be applied to the indirect vision system according to any one of claims 1 to 10, characterized in that, comprising: capturing the first image frame by the first camera toward the first scene; capturing the second image frame by the second camera toward the second scene; performing the following steps by the processor coupled to the first camera and the second camera: calculating the six-axis movement direction of the target point in a three-dimensional space by the current first image frame and another first image frame within a reference time; calculating the displacement amount between the target point and the display in the six-axis movement direction; and changing the field of view of the second image frame displayed on the display according to the displacement amount.

12. An interactive electronic rearview mirror system for a vehicle, comprising: comprising: a driver camera disposed on a front side of the vehicle interior and capturing a first image frame toward a driver seat; a rear camera disposed on a rear side of the vehicle and capturing a second image frame toward a rear scene of the vehicle; an electronic rearview mirror disposed on the front side of the vehicle interior, coupled to the driver camera and the rear camera, and displaying the second image frame toward the driver seat; wherein the driver camera is configured to: calculate a six-axis movement direction of a driver eye point position in a three-dimensional space by the current first image frame and another first image frame within a reference time; calculate an up-down displacement amount and a left-right displacement amount of the driver eye point position in the six-axis movement direction; calculate a current perpendicular plane displacement amount between the driver eye point position and the electronic rearview mirror in the six-axis movement direction; when a variation of the current perpendicular plane displacement amount is less than a standard deviation of all the perpendicular plane displacement amounts in a queue or the current first image frame is a continuous Nth image frame with the perpendicular plane displacement amount greater than the standard deviation, store the current perpendicular plane displacement amount to the queue and update the standard deviation, wherein N is a positive integer; and change the field of view of the second image frame displayed on the electronic rearview mirror up or down based on a first average value of the up-down displacement amount or the left-right displacement amount, and zoom out the field of view of the second image frame displayed on the electronic rearview mirror based on a second average value of all the perpendicular plane displacement amounts in the queue.

Images