Vehicle periphery monitoring device, control method and storage medium

Abstract

A vehicle periphery monitoring device, control method and storage medium. The vehicle periphery monitoring device comprises: a side rear camera unit for photographing the side rear of the vehicle; a rear camera unit for photographing the rear of the vehicle; a detection unit for detecting object marks existing in a side rear overlap region overlapping with a rear image photographed by the rear camera unit in a side rear image photographed by the side rear camera unit and a rear overlap region overlapping with the side rear image in a rear image; a determination unit for determining whether the object marks detected by the detection unit include specific object marks visually recognized only in either the side rear overlap region in the side rear image or the rear overlap region in the rear image; and a display control unit for superimposing a specific object mark display image corresponding to the specific object mark in the side rear image or the rear image in which the specific object mark is not visually recognized, and displaying the specific object mark on at least one display unit provided in the vehicle cabin when the determination unit determines that the specific object mark is included.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2020-092677, filed on May 27, 2020, the disclosure of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a vehicle periphery monitoring device. Background Art

[0004] Japanese Patent Application Publication No. 2020-028027 discloses an electronic mirror device that combines a rear image captured by a rear camera that captures the rear of a vehicle and a side rear image captured by a side rear camera that captures the side rear of the vehicle to generate a first composite image, and displays the image on a display unit provided in the vehicle cabin.

[0005] The electronic mirror device determines whether there is an object in a blind spot caused by a following vehicle approaching from the rear of the vehicle, and generates a second synthetic image in which a side rear image including an image area corresponding to the object is superimposed on a rear image if there is an object, and is configured to switch an image displayed on a display unit to the second synthetic image.

[0006] In the technology described in Japanese Patent Application Laid-Open No. 2020-028027, the first synthetic image is visually recognized as a continuous image of the rear image and the side rear image, whereas in the second synthetic image, in the portion where the side rear image is displayed overlappingly in the rear image, the rear image and the side rear image have different viewing angles, so the images are displayed discontinuously. Therefore, even if it is the same object, it looks different when visually recognized in the first synthetic screen and when visually recognized in the second synthetic screen, so there is a problem that the occupant who visually recognizes the display unit feels irritated. Summary of the invention

[0007] The present disclosure has been made in consideration of the above-mentioned facts, and an object of the present disclosure is to provide a vehicle periphery monitoring device capable of improving the visibility of a display unit and enabling a vehicle occupant to effectively recognize the surrounding conditions.

[0008] The vehicle periphery monitoring device of the first scheme of the present disclosure comprises: a side rear camera unit for photographing the side rear of the vehicle; a rear camera unit for photographing the rear of the vehicle; a detection unit for detecting an object (object target) existing in a side rear overlapping area overlapping with a rear image photographed by the rear camera unit in a side rear image photographed by the rear camera unit and in a rear overlapping area overlapping with the side rear image in the rear image; a determination unit for determining whether the object targets detected by the detection unit include a specific object target that is visually recognized only in either the side rear overlapping area in the side rear image or the rear overlapping area in the rear image; and a display control unit for superimposing a specific object target display image corresponding to the specific object target on the side rear image or the rear image in which the specific object target is not visually recognized, and displaying the specific object target on at least one display unit provided in the vehicle cabin when the determination unit determines that the specific object target is included.

[0009] In the first scheme, the side rear camera unit photographs the side rear of the vehicle, and the rear camera unit photographs the rear of the vehicle. In addition, the display control unit causes at least one display unit arranged in the vehicle interior to display the side rear image photographed by the side rear camera unit and the rear image photographed by the rear camera unit. Here, regarding the side rear overlapping area in the side rear image that overlaps with the rear image, and the rear overlapping area in the rear image that overlaps with the side rear image, there is a situation where other vehicles approaching from the rear side of the vehicle become blind spots (dead angles) in the side rear image or in the rear image. In response to this, the detection unit detects object marks existing in the side rear overlapping area and the rear overlapping area. In addition, the determination unit determines whether the object marks detected by the detection unit include specific object marks that are only recognized in either the side rear overlapping area or the rear overlapping area. In addition, when the determination unit determines that the specific object marker is included, the display control unit superimposes a specific object marker display image corresponding to the specific object marker on the side rear image or the rear image where the specific object marker is not visually recognized, and displays it on the display unit. That is, among the object markers detected by the detection unit, the specific object marker recognized only in either the side rear overlap area or the rear overlap area is an object marker existing in an area that is blocked by other vehicles approaching the rear side of the vehicle and becomes a blind spot in the side rear image or the rear image. Therefore, by superimposing and displaying the specific object marker display image corresponding to the specific object marker on the side rear image or the rear image where the specific object marker is not visually recognized, the occupant can recognize that the specific object marker exists behind the other vehicles approaching the rear side of the vehicle. In addition, the display control unit superimposes and displays the specific object marker display image only on the side rear image or the rear image where the specific object marker is not visually recognized, so that an image with a constant viewing angle is displayed in the area other than the specific object marker display image in the image displayed on the display unit. Therefore, the specific object marker display image in the image can be easily recognized, and it is possible to suppress the specific object marker display image from being missed or misrecognized before or after the specific object marker display image is displayed.

[0010] In this way, the occupants can better recognize specific objects in the side rear image or in the blind spot in the rear image due to other vehicles approaching from the rear side of the vehicle. As a result, the visibility of the objects displayed on the display unit can be improved, so that the occupants of the vehicle can effectively recognize the surrounding conditions.

[0011] The vehicle periphery monitoring device of the second scheme of the present disclosure, based on the structure described in the first scheme, when the specific object marker exists in the first blind spot in the side rear image generated by the side rear approaching vehicle approaching the side rear of the vehicle, the display control unit superimposes the specific object marker display image corresponding to the specific object marker visually confirmed in the rear image on the side rear image in which the specific object marker is not visually confirmed due to the side rear approaching vehicle, and displays it on the display unit.

[0012] Thus, even if there is an object marker in the first blind spot in the side rear image generated by the side rear approaching vehicle approaching the side rear of the vehicle, the object marker can be determined as a specific object marker and displayed in the side rear image in the form of a specific object marker display image. Thus, the occupant can know that the specific object marker exists near the rear of the side rear approaching vehicle by visually confirming the specific object marker display image, and can clearly recognize the surrounding conditions on the rear side of the side rear approaching vehicle.

[0013] The vehicle periphery monitoring device of the third scheme of the present disclosure, based on the structure described in the first scheme or the second scheme, when the specific object marker exists in the second blind spot in the rear image generated by the rear approaching vehicle approaching the rear of the vehicle, the display control unit superimposes the specific object marker display image corresponding to the specific object marker visually confirmed in the side rear image on the rear image in which the specific object marker is not visually confirmed due to the rear approaching vehicle, and displays it on the display unit.

[0014] Thus, even if there is an object mark in the second blind spot in the rear image caused by the rear approaching vehicle approaching the rear of the vehicle, the object mark can be determined as a specific object mark and displayed in the rear image in the form of a specific object mark display image. Thus, the occupant can know that there is a specific object mark near the rear of the side rear approaching vehicle by visually confirming the specific object mark display image, and can clearly recognize the surrounding conditions on the rear side of the rear approaching vehicle.

[0015] The vehicle periphery monitoring device of the fourth scheme of the present disclosure is based on the structure recorded in any one of the first to third schemes, wherein the display control unit deletes the side rear overlapping area or the rear overlapping area from either the side rear image and the rear image on the virtual projection plane, synthesizes the side rear image and the rear image after deleting the side rear overlapping area or the rear overlapping area from either side into a single image, and displays it on one of the display units.

[0016] Thus, the side rear processed image and the rear processed image displayed on the display unit become continuous images, so the display unit can display an image close to the situation of observing the rear of the vehicle. As a result, the burden on the occupant when recognizing a specific object is reduced, and the visual confirmation of the display unit is improved.

[0017] The vehicle periphery monitoring device of the fifth scheme of the present disclosure is based on the structure recorded in any one of the first to third schemes, wherein the display control unit displays the side rear image including the side rear overlapping area and the rear image including the rear overlapping area on the virtual projection plane separately on multiple display units.

[0018] Thus, the passenger can observe the side rear image including the side rear overlap area and the rear image including the rear overlap area on the virtual projection plane, so the surrounding conditions behind the vehicle can be recognized in a wide range. In addition, since both the specific object marker and the specific object marker display image corresponding thereto are displayed on the multiple display units, the position of the specific object marker can be clearly recognized from the multiple display units. As a result, the visual confirmation of the display unit is improved.

[0019] The vehicle periphery monitoring device of the sixth scheme of the present disclosure is based on the structure recorded in any one of the first to fifth schemes, wherein the display control unit overlaps the display image of the specific object in the side rear image or the rear image and displays it on the display unit when the specific object is approaching the vehicle.

[0020] Thus, when the risk level is higher, that is, when the specific object is approaching the vehicle, the specific object display image is displayed on the display unit, thereby effectively attracting the attention of the occupant.

[0021] Furthermore, a vehicle periphery monitoring device according to a seventh aspect of the present disclosure is, in addition to the configuration described in any one of the first to sixth aspects, an electronic mirror device mounted on the vehicle.

[0022] Effects of the Invention

[0023] The vehicle periphery monitoring device according to the first aspect of the present disclosure has an excellent effect of being able to improve the visibility of the display unit and enable the vehicle occupant to effectively recognize the surrounding situation.

[0024] The vehicle periphery monitoring device according to the second aspect of the present disclosure has an excellent effect of enabling an occupant to clearly recognize the surrounding conditions on the rear side of a vehicle approaching from behind.

[0025] The vehicle periphery monitoring device according to the third aspect of the present disclosure has an excellent effect of enabling an occupant to clearly recognize the surrounding conditions on the rear side of a vehicle approaching from behind.

[0026] The vehicle periphery monitoring device according to the fourth aspect of the present disclosure has an excellent effect of being able to reduce the burden on the occupant when recognizing a specific object and improve the visibility of the display unit.

[0027] The vehicle periphery monitoring device according to the fifth aspect of the present disclosure has the excellent effect of improving the visibility of the display unit by displaying the periphery behind the vehicle in a wide range and displaying the specific object display image in a form that allows the position of the specific object to be clearly recognized.

[0028] The vehicle periphery monitoring device according to the sixth aspect of the present disclosure has an excellent effect of being able to effectively draw the attention of the occupant in relation to the specific object behind the vehicle.

[0029] The vehicle periphery monitoring device according to the seventh aspect of the present disclosure has an excellent effect of being able to improve the visibility of the display unit of the electronic mirror device and enable the vehicle occupant to effectively recognize the surrounding situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Hereinafter, exemplary embodiments of the present application will be described in detail based on the following drawings, wherein:

[0031] Figure 1 is a block diagram showing the hardware configuration of the electronic mirror device according to the first embodiment;

[0032] Figure 2 is a block diagram showing the functional configuration of the electronic mirror device according to the first embodiment;

[0033] Figure 3 1 is a diagram showing the arrangement positions of the side rear camera and the display unit of the electronic mirror device according to the first embodiment, and is a perspective view of the interior of a vehicle having the electronic mirror device as viewed from the rear side of the vehicle;

[0034] Figure 4 is a top view showing the shooting ranges of the side rear camera and the rear camera, and the side rear overlapping area of ​​the side rear image and the rear overlapping area of ​​the rear image;

[0035] Figure 5 is a schematic diagram showing an example of a state where a side rear processed image and a rear processed image are synthesized into a single image and displayed on a display unit;

[0036] Figure 6 is a flowchart showing an example of overlapping processing performed by the electronic mirror ECU;

[0037] Figure 7 is a top view showing an example of a first blind spot caused by a vehicle approaching from the rear and a situation in which a specific object exists in the first blind spot;

[0038] Figure 8 is a top view showing an example of a second blind spot caused by a vehicle approaching from behind and a situation where a specific object is present in the second blind spot;

[0039] Fig. 9 is shown with Figure 7 A schematic diagram of an example of a specific object marker display image corresponding to a specific object marker existing in a first blind zone;

[0040] Fig.10 is shown with Figure 8 A schematic diagram of an example of a specific object mark display image corresponding to a specific object mark existing in a second blind zone;

[0041] Fig.11 An example of a specific object display image is shown. Fig. 9 The corresponding schematic diagram;

[0042] Fig.12 is a block diagram showing the hardware configuration of the electronic mirror device according to the second embodiment;

[0043] Fig.13 is a block diagram showing the functional configuration of the electronic mirror device according to the second embodiment;

[0044] Fig.14 FIG. 1 is a diagram showing an example of a specific object marker display image corresponding to a specific object marker existing in the first blind spot. Fig. 9 the corresponding schematic diagram; and

[0045] Fig.15 An example of a specific object display image is shown. Fig.14 The corresponding schematic diagram. DETAILED DESCRIPTION

[0046] (Hardware Configuration)

[0047] Below, use Figures 1 to 11 , the first embodiment of the present invention is described. Figure 1 The vehicle-mounted system 40 shown includes a bus 42, to which a plurality of electronic control units and a plurality of sensor units that perform different controls are connected. Figure 1 Only a part of the vehicle-mounted system 40 related to the present invention is shown. Each electronic control unit is a control unit including a CPU, a memory, and a nonvolatile storage unit, and is hereinafter referred to as an ECU (Electronic Control Unit). The electronic mirror ECU 22 is included in the plurality of ECUs connected to the bus 42 .

[0048] The electronic mirror ECU 22 is connected to the left rear camera 12, the right rear camera 14, the rear camera 16, the electronic mirror display 18, and the camera storage ACT (actuator) 20. The electronic mirror ECU 22, the left rear camera 12, the right rear camera 14, the rear camera 16, the electronic mirror display 18, and the camera storage ACT 20 constitute the electronic mirror device 10, which is an example of a vehicle periphery monitoring device. In addition, the electronic mirror display 18 in the electronic mirror device 10 is an example of a display unit.

[0049] like Figure 3 As shown, the base of the camera support body 32L, which is roughly rectangular and has an arc-shaped top end, is mounted on the vehicle front end of the left door (front door, not shown) of the vehicle in the middle of the vehicle vertical direction in a manner that the top end of the camera support body 32L protrudes to the outside of the vehicle. The left rear camera 12 is mounted near the top end of the camera support body 32L, and the shooting optical axis (lens) of the left rear camera 12 faces the left rear of the vehicle to shoot the left rear of the vehicle. The camera support body 32L can rotate in the front-to-back direction of the vehicle, and rotates to the storage position of the camera support body 32L roughly along the outer side of the vehicle in the long direction, or the recovery position where the left rear camera 12 shoots the left rear of the vehicle by the driving force of the camera storage ACT 20.

[0050] Regarding the left rear camera 12, its lens is set to a fixed focus, and there is no mechanism for changing the direction of the shooting optical axis. When the camera support body 32L is in the restored position, Figure 4 In the present embodiment, the side rear image 62A of the left side of the vehicle projected onto the imaginary projection plane 66 behind the vehicle 50 is set as the side rear image captured by the left rear camera 12. In addition, the capturing range 62 of the left rear camera 12 partially overlaps with the capturing range 60 of the rear camera 16, and the side rear overlapping area of ​​the side rear image 62A that overlaps with the rear image 60A of the rear camera 16 projected onto the imaginary projection plane 66 is set as 62A1 and is shown in FIG. Figure 4 .

[0051] In addition, at the vehicle front end portion of the right side door (front side door, not shown) of the vehicle in the middle of the vehicle vertical direction, a base portion of a camera support body 32R having a shape symmetrical to the camera support body 32L is installed. A right rear camera 14 is installed near the top end portion of the camera support body 32R, and the shooting optical axis (lens) of the right rear camera 14 faces the right rear side of the vehicle to shoot the right rear side of the vehicle. The camera support body 32R can also rotate in the vehicle front-to-back direction, and rotate to a storage position roughly along the outer side surface of the vehicle in the long direction of the camera support body 32R, or a recovery position where the right rear camera 14 shoots the right rear side of the vehicle by the driving force of the camera storage ACT 20.

[0052] Regarding the right rear camera 14, its lens is set to a fixed focus, and there is no mechanism for changing the direction of the shooting optical axis. When the camera support body 32R is in the restored position, Figure 4 In the present embodiment, the side rear image 64A of the right side of the vehicle projected onto the imaginary projection plane 66 behind the vehicle 50 is set as the side rear image captured by the right rear camera 14. In addition, the capturing range 64 of the right rear camera 14 partially overlaps with the capturing range 60 of the rear camera 16, and the side rear overlapping area of ​​the side rear image 64A that overlaps with the rear image 60A of the rear camera 16 projected onto the imaginary projection plane 66 is set as 64A1 and is shown in FIG. Figure 4 .

[0053] The rear camera 16 is disposed on the vehicle 50 (see Figure 4 ) is positioned at the rear of the vehicle 50, with the shooting optical axis (lens) facing the rear of the vehicle, and shooting the rear of the vehicle 50. In addition, the rear camera 16 can be arranged at any position as long as it can shoot the rear of the vehicle 50. It can be arranged at the rear end of the vehicle 50 (for example, near the rear bumper) or can be arranged to shoot the rear of the vehicle 50 through the rear windshield. The lens of the rear camera 16 is set to a fixed focus, and there is no mechanism for changing the direction of the shooting optical axis. Figure 4 In the present embodiment, the rear image 60A at the center projected onto the virtual projection plane 66 at the rear of the vehicle 50 is set as the rear image captured by the rear camera 16. In addition, the area of ​​the rear image 60A that overlaps with the side rear image 62A on the left side of the vehicle and the side rear image 64A on the right side of the vehicle projected onto the virtual projection plane 66 is set as a rear overlap area 60A1 and is shown in FIG. Figure 4 .

[0054] In addition, a central monitor 34 is provided at the center of the dashboard of the vehicle, and an electronic mirror display 18 is provided at a position spaced apart from the central monitor 34 toward the upper part of the vehicle. The electronic mirror display 18 displays an image synthesized by performing a synthesis process described later on the left rear image (dynamic image) captured by the left rear camera 12, the right rear image (dynamic image) captured by the right rear camera 14, and the rear image (dynamic image) captured by the rear camera 16 through the electronic mirror ECU 22.

[0055] like Figure 1 As shown, the electronic mirror ECU 22 includes a CPU (Central Processing Unit) 24 , a ROM (Read Only Memory) 26 , a RAM (Random Access Memory) 28 , and a storage device 30 .

[0056] The CPU 24 is a central processing unit and executes various programs to control each unit. Specifically, the CPU 24 reads a program from the ROM 26 or the storage device 30 and executes the program using the RAM 28 as a work area.

[0057] (Functional composition)

[0058] Figure 2 1 is a block diagram showing the functional configuration of the electronic mirror ECU 22. As shown in the figure, the electronic mirror ECU 22 includes a side rear camera 220, a rear camera 230, a detection unit 240, a determination unit 250, and a display control unit 260. Each functional configuration is realized by reading an execution program stored in the ROM 26 or the storage device 30 and executing the execution program.

[0059] The side rear imaging unit 220 captures the left and right side rears of the vehicle 50 in the form of dynamic images using the left side rear camera 12 and the right side rear camera 14 , and outputs the captured side rear images to the detection unit 240 and the display control unit 260 .

[0060] The rear imaging unit 230 captures the rear of the vehicle 50 in a moving image through the rear camera 16 , and outputs the captured rear image to the detection unit 240 and the display control unit 260 .

[0061] The detection unit 240 analyzes the side rear images 62A, 64A and the rear image 60A, and detects object targets existing in the side rear overlap regions 62A1, 64A1 overlapping with the rear image 60A in the side rear images 62A, 64A, and the left and right rear overlap regions 60A1 overlapping with the side rear images 62A, 64A in the rear image 60A. Then, the detection unit 240 outputs information on the detected object targets to the determination unit 250. In addition, the detection unit 240 may be configured to detect object targets existing in regions corresponding to the side rear overlap regions 62A1, 64A1 and the rear overlap region 60A1 based on detection results of a radar or the like that has the rear of the vehicle 50 as a detection range.

[0062] The determination unit 250 determines whether the object marks present in the side rear overlap regions 62A1, 64A1 and the rear overlap region 60A1 include a specific object mark that is recognized only in one of the side rear overlap regions 62A1, 64A1 and the rear overlap region 60A1, based on the information on the object mark output from the detection unit 240. The determination unit 250 analyzes the side rear images 62A, 64A and the rear image 60A, and makes a determination by comparing the object marks detected in the side rear images 62A, 64A with the object marks detected in the rear image 60A.

[0063] The display control unit 260 has a function of generating an image by synthesizing the side rear images 62A, 64A and the rear image 60A output from the side rear imaging unit 220 and the rear imaging unit 230, and displaying the image on the electronic mirror display 18. The synthesis processing routine includes an overlay processing as a subroutine (see Figure 6 ), that is, when the vehicle 50 is detected approaching the rear side of the vehicle 52 (refer to Figure 7 ) or the rear approaching vehicle 54 approaching the rear of the vehicle 50 (refer to Figure 8 ), a specific object display image is displayed superimposed on the side rear image and the rear image.

[0064] (Function and Effect)

[0065] Next, use Figure 4 and Figure 5 The function of the first embodiment and the synthesis process of the electronic mirror ECU 22 causing the electronic mirror display 18 to display the side rear images captured by the left and right side rear cameras 12 and 14 and the rear image captured by the rear camera 16 are described. The synthesis process is described in detail below: The electronic mirror ECU 22 extracts the side rear images 62A and 64A on the virtual projection plane 66 captured by the left rear camera 12 and the right rear camera 14.

[0066] In the next step, the electronic mirror ECU 22 extracts an image corresponding to the extraction region 60A2 obtained by deleting the left and right rear overlapping regions 60A1 from the rear image 60A on the virtual projection plane 66 captured by the rear camera 16 .

[0067] In the next step, the electronic mirror ECU 22 combines the side rear image 62A with the left side of the extraction region 60A2 and combines the side rear image 64A with the right side of the extraction region 60A2 to generate a single image.

[0068] In the next step, the electronic mirror ECU 22 causes the electronic mirror display 18 to display the side rear images 62A, 64A and the extraction area 60A2 synthesized into a single image. An example of the image displayed on the electronic mirror display 18 through the above synthesis process is shown in FIG. Figure 5 .

[0069] like Figure 5 As shown, the image displayed on the electronic mirror display 18 is generated by deleting the left and right rear overlapping areas 60A1 from the rear image 60A on the virtual projection plane 66 to generate an image of the extraction area 60A2, and synthesizing the side rear image and the rear image into a single image. As a result, the side rear image and the rear image displayed on the electronic mirror display 18 become smooth and continuous images, and the electronic mirror display 18 can display an image close to the situation of observing the rear of the vehicle 50. As a result, the burden on the occupant when recognizing the specific object mark described later is reduced, and the visual confirmation of the electronic mirror display 18 is improved.

[0070] In the next step, the electronic mirror ECU 22 determines whether there is a rear approaching vehicle 52 (see FIG. 1 ) approaching the rear side of the vehicle 50. Figure 7 ) or the rear approaching vehicle 54 approaching the rear of the vehicle 50 (refer to Figure 8 ). In addition, the side rear approaching vehicle 52 and the rear approaching vehicle 54 can be detected, for example, by analyzing the side rear images 62A, 64A and the rear image 60A, and it is possible to determine whether the side rear approaching vehicle 52 or the rear approaching vehicle 54 exists at a relatively close distance based on whether the size of the image area corresponding to the side rear approaching vehicle 52 or the rear approaching vehicle 54 included in the side rear images 62A, 64A and the rear image 60A is greater than a predetermined value. In addition, the determination of whether there is a side rear approaching vehicle 52 or a rear approaching vehicle 54 can also be determined based on the detection results of a radar or the like that uses the rear of the vehicle 50 as a detection range. In this case, the distance between the vehicle 50 and the side rear approaching vehicle 52 or the rear approaching vehicle 54 can be determined more accurately.

[0071] If the above determination is negative, that is, if it is determined that the side rear approaching vehicle 52 and the rear approaching vehicle 54 are not present, the process returns to the beginning of the synthesis processing step and the above steps are repeated. During this period, the generation and display of a single image synthesized from the side rear image and the rear image are continued.

[0072] On the other hand, as an example, Figure 7 As shown, when there is a rear approaching vehicle 52 on the right side rear of the vehicle 50, the determination in the above step is yes, and the superimposition process included as a subroutine in the above-mentioned synthesis process is executed (refer to Figure 6 ).like Figure 7 As shown in FIG. 1 , when the side rear approaching vehicle 52 is traveling on the right side rear of the vehicle 50, a first blind spot 70 is generated in the side rear image 64A of the right rear camera 14 due to the side rear approaching vehicle 52. The first blind spot 70 includes the side rear overlap area 64A1 of the right rear camera 14. Therefore, when the two-wheeled vehicle 56 is present in the side rear overlap area 64A1 of the right rear camera 14, the two-wheeled vehicle 56 is blocked by the side rear approaching vehicle 52 and does not appear in the side rear image of the electronic mirror display 18 (see FIG. 1 ). Fig. 9 ). As a result, the presence of the two-wheeled vehicle 56 may not be recognized by the occupant who visually checks the electronic mirror display 18, so, for example, when the vehicle 50 attempts to change lanes to the right lane behind the vehicle 52 at the side rear, the risk of contact with the two-wheeled vehicle 56 increases. Therefore, in the subsequent overlapping processing, as described later, an image is generated and displayed in which the specific object marker display image M corresponding to the two-wheeled vehicle 56 is overlapped with the side rear image.

[0073] In addition, as an example, Figure 8 As shown, when there is a rear approaching vehicle 54 behind the vehicle 50, the determination in the above step is yes, and the overlap processing is also performed. Figure 8 As shown in FIG. 1 , when the rear approaching vehicle 54 is traveling behind the vehicle 50, a second blind spot 72 is generated in the rear image 60A of the rear camera 16 due to the rear approaching vehicle 54. The second blind spot 72 includes the rear overlap area 60A1 of the rear camera 16. Therefore, when the two-wheeled vehicle 56 is present in the rear overlap area 60A1 on the right side of the vehicle 50, the two-wheeled vehicle 56 is not visually recognized in the rear image of the electronic mirror display 18 due to the rear approaching vehicle 54, and is visually recognized in the side rear image in a state where it is partially blocked by the rear approaching vehicle 54 (see FIG. 1 ). Fig.10). As a result, the presence of the two-wheeled vehicle 56 may not be recognized by the occupant who visually checks the electronic mirror display 18, so when the vehicle 50 attempts to change lanes to the right lane, the risk of contact with the two-wheeled vehicle 56 increases. Therefore, an image in which the specific object display image M corresponding to the two-wheeled vehicle 56 is displayed superimposed on the rear image is generated and displayed through superimposition processing.

[0074] Specifically, if Figure 6 As shown, in step S100 , the electronic mirror ECU 22 detects object targets existing in the side rear overlapping regions 62A1 and 64A1 of the left and right side rear images 62A and 64A and the left and right rear overlapping regions 60A1 of the rear image 60A.

[0075] In the next step S101, the electronic mirror ECU 22 determines whether the object marks detected in the side rear overlap regions 62A1, 64A1 and the rear overlap region 60A1 include a specific object mark visually recognized only in either the side rear overlap region 62A1 and the rear overlap region 60A1 on the left side of the vehicle 50, and a specific object mark visually recognized only in either the side rear overlap region 64A1 and the rear overlap region 60A1 on the right side of the vehicle 50. If the determination in step S101 is yes, the process moves to step S102. If the determination in step S101 is no, the overlapping process is terminated and the process returns to the step of the synthesis process.

[0076] As an example, in the above Figure 7 In this way, when the side rear approaching vehicle 52 is traveling on the side rear of the right side of the vehicle 50, the two-wheeled vehicle 56 is present in the first blind spot 70 generated by the side rear approaching vehicle 52, and therefore is not visually recognized in the side rear image 64A captured by the right rear camera 14. Therefore, in step S100, through image recognition of the side rear image 64A and the rear image 60A, the existence of the object (two-wheeled vehicle 56) is not detected (visually recognized) from the side rear overlapping area 64A1 corresponding to the right rear camera 14 of the vehicle 50, while the two-wheeled vehicle 56 is detected (visually recognized) as an object from the right rear overlapping area 60A1 corresponding to the rear camera 16. Then, in the next step S101, the two-wheeled vehicle 56 detected in step S100 is set as a specific object that is visually recognized only in the rear overlapping area 60A1. Therefore, in step S101, the electronic mirror ECU 22 determines that the object target detected in step S100 (in Figure 7 In the example, only the two-wheeled vehicle 56) includes the specific object (the two-wheeled vehicle 56).

[0077] In addition, in Figure 8In that case, when the rear approaching vehicle 54 is traveling behind the vehicle 50, the two-wheeled vehicle 56 is present in the second blind spot 72 generated by the rear approaching vehicle 54, so it is not visually confirmed in the rear image 60A of the rear camera 16. Therefore, in step S100, through image recognition of the side rear image 64A corresponding to the right rear camera 14 and the rear image 60A, the existence of the object mark (the two-wheeled vehicle 56) is detected (visually confirmed) from the side rear overlapping area 64A1, and on the other hand, the existence of the object mark is not detected (visually confirmed) from the rear overlapping area 60A1. Then, in the next step S101, the two-wheeled vehicle 56 detected in step S100 is set as a specific object mark that is visually confirmed only in the side rear overlapping area 64A1. Therefore, in step S101, the electronic mirror ECU 22 determines that the object mark detected in step S100 (in Fig. 9 In the example, only the two-wheeled vehicle 56) includes the specific object (the two-wheeled vehicle 56).

[0078] In step S102, the electronic mirror ECU 22 determines whether the specific object is approaching the vehicle 50. The determination in step S102 can be made by determining whether there is a specific object (in the side rear image 62A, 64A or the rear image 60A) in which the specific object is recognized. Figure 7 , Figure 8 The area corresponding to the two-wheeled vehicle 56 is realized by increasing the size of the area as time passes. In addition, the determination may be made based on the detection result of a radar or the like that includes the side and rear area of ​​the vehicle 50 as the detection range.

[0079] If the determination in step S102 is yes, the process proceeds to step S103. If the determination in step S102 is no, the superimposition process is terminated and the process returns to the synthesis process. In step S103, the electronic mirror ECU 22 superimposes a specific object marker display image corresponding to the specific object marker on the side rear image or the rear image in which the specific object marker is not visually recognized, and displays it on the electronic mirror display 18.

[0080] As an example, in Figure 7 In the situation shown, the electronic mirror ECU 22 generates an image obtained by overlapping and displaying the specific object display image M corresponding to the two-wheeled vehicle 56 visually recognized in the right rear overlapping area 60A1 with the side rear image 64A corresponding to the right rear camera 14 in which the two-wheeled vehicle 56 as a specific object is not visually recognized. Fig. 9 As shown, the electronic mirror ECU 22 displays the generated rear side image on the electronic mirror display 18. The specific object display image M is displayed as a frame surrounding the specific object (two-wheeled vehicle 56), for example.

[0081] The specific object marker display image M is displayed superimposed on the rearward approaching vehicle 52 appearing in the rearward image 64A. Thus, the occupant who visually checks the electronic mirror display 18 can recognize that the two-wheeled vehicle 56 is behind the rearward approaching vehicle 52.

[0082] On the other hand, Figure 8 In the case of the situation shown in FIG. 1 , the electronic mirror ECU 22 generates an image obtained by overlapping and displaying the specific object display image M corresponding to the two-wheeled vehicle 56 visually recognized in the side rear overlapping area 64A1 of the right side rear camera 14, with respect to the rear image 60A corresponding to the rear camera 16 in which the two-wheeled vehicle 56 as the specific object is not visually recognized. Then, as shown in FIG. Fig.10 As shown, the electronic mirror ECU 22 displays the generated rearward image on the electronic mirror display 18 .

[0083] The specific object marker display image M is displayed superimposed on the rear approaching vehicle 54 appearing in the rear processed image. Thus, the occupant who visually checks the electronic mirror display 18 can recognize the presence of the two-wheeled vehicle 56 approaching to the rear side of the rear approaching vehicle 54 .

[0084] In addition, the specific object mark display image M may also be a predetermined icon. In addition, it may be configured so that the displayed specific object mark display image flashes, and the number of flashes increases as the distance between the specific object mark and the vehicle 50 approaches. Fig.11 As shown in the example, an image of a portion corresponding to the specific object (two-wheeled vehicle 56) recognized in the side rear images 62A, 64A or the rear image 60A is extracted as the specific object display image M. The image constituting the specific object display image M may be generated as a perspective image in a manner that the visual recognition is lower than the visual recognition of other images displayed on the electronic mirror display 18.

[0085] [Second embodiment]

[0086] Below, use Figure 12 to Figure 15 , the second embodiment of the present invention is described. In addition, the same number is marked on the same components as the first embodiment and the description thereof is omitted. The basic structure of the vehicle-mounted system 80 of the second embodiment is the same as that of the first embodiment, and is characterized in that "the side rear images and rear images captured by the left rear camera 12, the right rear camera 14, and the rear camera 16 are displayed separately on multiple display units" (refer to Fig.14 ).

[0087] (Hardware Configuration)

[0088] That is, the vehicle-mounted system 80 includes an electronic mirror device 90 composed of an electronic mirror ECU 92, a left rear camera 12, a right rear camera 14, a rear camera 16, an electronic mirror display 94, and a camera storage ACT 20. In addition, the electronic mirror display 94 as a display unit includes a left display 94A, a right display 94B, and a center display 94C. The left display 94A displays an image captured by the left rear camera 12. The right display 94B displays an image captured by the right rear camera 14. The center display 94C displays an image captured by the rear camera 16.

[0089] (Functional composition)

[0090] Fig.13 is a block diagram showing the functional configuration of the electronic mirror ECU 92. The display control unit 960 extracts Figure 4 The left and right side rear images 62A and 64A on the virtual projection plane 66 are displayed on the left display 94A and the right display 94B, respectively. That is, the display control unit 960 displays the side rear images 62A and 64A including the side rear overlap regions 62A1 and 64A1 on the electronic mirror display 94 as side rear images.

[0091] In addition, the display control unit 960 extracts Figure 4 The rear image 60A on the virtual projection plane 66 is displayed on the central display 94C. That is, the display control unit 960 displays the rear image 60A including the rear overlapping region 60A1 on the electronic mirror display 94 as the rear image.

[0092] In addition, the display control unit 960 executes the same as the display control unit 260 of the first embodiment. Figure 6 The side rear camera unit 920, the rear camera unit 930, the detection unit 940, and the determination unit 950 have the same functions as the side rear camera unit 220, the rear camera unit 230, the detection unit 240, and the determination unit 250 of the first embodiment, and thus their description is omitted.

[0093] (Function and Effect)

[0094] The digital mirror device 90 having the above-described structure is similar in structure to the digital mirror device 10 of the first embodiment except that the side rear processed image and the rear image are displayed separately, and therefore can obtain the same effects as those of the first embodiment.

[0095] In addition, if Fig.14As shown, the passengers in the vehicle can observe the side rear images 62A and 64A including the left and right side rear overlap regions 62A1 and 64A1 on the virtual projection plane 66 and the rear image 60A including the left and right rear overlap region 60A1 on the virtual projection plane 66. Therefore, the passengers observing the electronic mirror display 94 can recognize the surrounding conditions behind the vehicle in a wide range. In addition, as an example, Fig.14 The schematic diagram is in Figure 7 In the illustrated situation, the side rear image and the rear image are displayed on the electronic mirror display 94 .

[0096] In addition, in the present embodiment, the electronic mirror display 94 composed of the left display 94A, the right display 94B, and the center display 94C displays both the specific object (two-wheeled vehicle 56) and the corresponding specific object display image M. Therefore, the position of the specific object can be clearly recognized by visually confirming the image of the specific object and the specific object display image M on the plurality of displays. As a result, the visual confirmation of the electronic mirror display 94 is improved.

[0097] In addition, when both the specific object (two-wheeled vehicle 56) and the specific object display image M corresponding thereto are displayed on the electronic mirror display 94 as in the present embodiment, the following may be used. Fig.15 As shown in the example, a display image similar to the specific object display image M is displayed superimposed on the image of the specific object (two-wheeled vehicle 56). Thus, even when the specific object and the specific object display image are displayed on different displays, the occupant can quickly recognize that the information is related to the same object by observing the same display.

[0098] In the first embodiment, the side rear image 62A of the left rear camera 12 and the side rear image 64A of the right rear camera 14 on the virtual projection plane 66 are displayed on the electronic mirror display 18 as the side rear image of the present invention, and the extracted area 60A2 after deleting the left and right rear overlapping areas 60A1 from the rear image 60A of the rear camera 16 is displayed as the rear image of the present invention on the electronic mirror display 18. However, the present invention is not limited to this, and it is also possible to display the image extracted by deleting the side rear overlapping areas 62A1 and 64A1 from the side rear images 62A and 64A as the side rear image of the present invention on the electronic mirror display 18, and display the rear image 60A on the virtual projection plane 66 as the rear image of the present invention on the electronic mirror display 18.

Claims

1. A vehicle periphery monitoring device, comprising: A side and rear camera unit for shooting the side and rear of the vehicle; The rear camera unit is used to shoot the rear of the vehicle; a detection unit that detects an object mark existing in a side rear overlapping area in the side rear image captured by the side rear imaging unit and overlapping with the rear image captured by the rear imaging unit, and in a rear overlapping area in the rear image and overlapping with the side rear image; A display control unit deletes either a side rear overlapping area of ​​the side rear image or a rear overlapping area of ​​the rear image on the virtual projection plane, and synthesizes the side rear image from which the side rear overlapping area or the rear overlapping area is deleted and the rear image into a single image; a determination unit that determines whether the object marks detected by the detection unit include a specific object mark that is visually recognized only in one of the side rear overlapping region in the side rear image and the rear overlapping region in the rear image, The display control unit, when the determination unit determines that the specific object marker is included, superimposes a specific object marker display image corresponding to the specific object marker on the side rear image or the rear image in which the specific object marker is not visually recognized, and displays the image on at least one display unit provided in the vehicle interior. The display control unit, when the specific object is approaching the vehicle, superimposes the specific object display image on the side rear image or the rear image and displays the image on the display unit, The display control unit causes the specific object marker display image to flash when the display unit displays the specific object marker display image, and changes the flashing cycle according to the distance between the vehicle and the specific object marker, and increases the number of flashing times as the distance between the vehicle and the specific object marker decreases. When the specific object marker exists in a first blind spot in the side rear image caused by the side rear approaching vehicle approaching the side rear of the vehicle, The display control unit superimposes the specific object marker display image corresponding to the specific object marker visually recognized in the rear image on the side rear image in which the specific object marker is not visually recognized due to the side rear approaching vehicle, and displays the superimposed image on the display unit. When the specific object marker exists in a second blind spot in the rear image caused by a rear approaching vehicle approaching the rear of the vehicle, the display control unit superimposes the specific object marker display image corresponding to the specific object marker visually recognized in the side rear image on the rear image in which the specific object marker is not visually recognized due to the rear approaching vehicle, and displays the superimposed image on the display unit; The display control unit, when causing the display unit to display the specific object marker display image, causes the specific object marker display image to be displayed as a see-through image so that the visibility is lower than that of other images displayed on the display unit. The vehicle periphery monitoring device is an electronic mirror device mounted on the vehicle.

2. A control method, wherein: Take photos of the side and rear of the vehicle. Take a photo of the rear of the vehicle. detecting an object existing in a side rear overlapping region where a shooting range in the captured side rear image overlaps with a captured rear image, and in a rear overlapping region where a shooting range in the rear image overlaps with the side rear image, Deleting either the side rear overlapping area of ​​the side rear image or the rear overlapping area of ​​the rear image on the virtual projection plane, and synthesizing the side rear image and the rear image from which the side rear overlapping area or the rear overlapping area has been deleted into a single image, determining whether the detected object markers include a specific object marker that is visually recognized only in one of the side rear overlapping region in the side rear image and the rear overlapping region in the rear image, When it is determined that the specific object marker is included, a specific object marker display image corresponding to the specific object marker is superimposed on the side rear image or the rear image in which the specific object marker is not visually recognized, and displayed on at least one display unit provided in the vehicle interior. When the specific object marker is approaching the vehicle, the specific object marker display image is superimposed on the side rear image or the rear image and displayed on the display unit, When the display unit displays the specific object marker display image, the specific object marker display image is flashed, and a flashing cycle is changed according to the distance between the vehicle and the specific object marker, and the number of flashings increases as the distance between the vehicle and the specific object marker decreases. When the specific object marker exists in a first blind spot in the side rear image caused by the side rear approaching vehicle approaching the side rear of the vehicle, superimposing the specific object marker display image corresponding to the specific object marker visually recognized in the rear image on the side rear image in which the specific object marker is not visually recognized due to the side rear approaching vehicle, and displaying the specific object marker display image on the display unit, When the specific object marker exists in a second blind spot in the rear image caused by a rear approaching vehicle approaching the rear of the vehicle, superimposing the specific object marker display image corresponding to the specific object marker visually recognized in the side rear image on the rear image in which the specific object marker is not visually recognized due to the rear approaching vehicle, and displaying the superimposed specific object marker display image on the display unit, When the specific object marker display image is displayed on the display unit, the specific object marker display image is displayed as a see-through image so that visibility is lower than that of other images displayed on the display unit.

3. A storage medium that stores a program and can be read by a computer. When the program is executed by a processor, the following processing is performed: Take photos of the side and rear of the vehicle. Take a photo of the rear of the vehicle. detecting an object existing in a side rear overlapping region where a shooting range in the captured side rear image overlaps with a captured rear image, and in a rear overlapping region where a shooting range in the rear image overlaps with the side rear image, Deleting either the side rear overlapping area of ​​the side rear image or the rear overlapping area of ​​the rear image on the virtual projection plane, and synthesizing the side rear image and the rear image from which the side rear overlapping area or the rear overlapping area has been deleted into a single image, determining whether the detected object markers include a specific object marker that is visually recognized only in one of the side rear overlapping region in the side rear image and the rear overlapping region in the rear image, When it is determined that the specific object marker is included, a specific object marker display image corresponding to the specific object marker is superimposed on the side rear image or the rear image in which the specific object marker is not visually recognized, and displayed on at least one display unit provided in the vehicle interior. When the specific object marker is approaching the vehicle, the specific object marker display image is superimposed on the side rear image or the rear image and displayed on the display unit, When the display unit displays the specific object marker display image, the specific object marker display image is flashed, and a flashing cycle is changed according to the distance between the vehicle and the specific object marker, and the number of flashings increases as the distance between the vehicle and the specific object marker decreases. When the specific object marker exists in a first blind spot in the side rear image caused by the side rear approaching vehicle approaching the side rear of the vehicle, superimposing the specific object marker display image corresponding to the specific object marker visually recognized in the rear image on the side rear image in which the specific object marker is not visually recognized due to the side rear approaching vehicle, and displaying the specific object marker display image on the display unit, When the specific object marker exists in a second blind spot in the rear image caused by a rear approaching vehicle approaching the rear of the vehicle, superimposing the specific object marker display image corresponding to the specific object marker visually recognized in the side rear image on the rear image in which the specific object marker is not visually recognized due to the rear approaching vehicle, and displaying the superimposed specific object marker display image on the display unit, When the specific object marker display image is displayed on the display unit, the specific object marker display image is displayed as a see-through image so that visibility is lower than that of other images displayed on the display unit.

Images