Image processing device, system and program

Abstract

To reduce disappearance of an object from a synthesis image.SOLUTION: An image processor 10 includes: a synthesis unit 14 for synthesizing a rear image of the back of a vehicle and a side image of the left back side and the right back side of the vehicle; a detection unit 12 for detecting an object including a back vehicle from each image; and a determination unit 13 for determining whether the back vehicle detected from the rear image came closer to the vehicle. The synthesis unit 14 generates a synthesis image in a first mode when the determination unit has not determined that the back vehicle is coming closer to the vehicle, and generates a synthesis image in a specific synthesis mode when the determination unit has determined that the back vehicle is coming closer to the vehicle. In the first mode, the rear image is overlapped on a part of the side image and the overlapping region becomes larger as the back vehicle is coming closer. In the specific synthesis mode, the rear image is also overlapped with a part of the side image and the overlap of an object detected from the side image and the rear image is smaller than in the first mode or the rear image is not overlapped with the side image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a system, and a program. [Background technology]

[0002] Conventionally, driving assistance systems use cameras to capture images of the rear and left and right rear sides of a vehicle and display them on a display inside the vehicle. Because the camera positions and angles of view are different for the rear and rear sides, it can be difficult to judge the distance to the vehicle behind from a composite image that simply combines each captured image into a single image.

[0003] Therefore, in order to view the composite image in the same way as when viewed with the naked eye, it is common to apply image processing to each captured image during the composite process. For example, according to Patent Document 1, the captured image of the rear is reduced in size to make it easier to grasp the sense of distance to the vehicle behind. In addition, the captured images of the left and right rear sides are transformed so that the lanes and horizon are continuous between the captured images of the rear and rear sides. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4762698 Summary of the Invention [Problem to be solved by the invention]

[0005] In the above-mentioned Patent Document 1, image processing is performed to superimpose a rearward photographed image on a rearward photographed image so that the positions of the lanes in each photographed image match. If the rearward photographed image includes a rear vehicle traveling in an adjacent lane, the rearward photographed image may be superimposed on the rearward vehicle, causing the rearward vehicle to disappear from the composite image. As a result, the driver may be late in noticing the presence of the rearward vehicle.

[0006] The present invention aims to reduce the loss of objects from a synthetic image. [Means for solving the problem]

[0007] The present invention includes the following aspects. [1] An image processing device that generates a single composite image from multiple captured images of the rear of a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is less than in the first mode, or the rear image is not superimposed on the side image. Image processing device.

[0008] According to [1] above, synthesis is performed in the first mode until a vehicle behind approaches. In the first mode, the rear image is partially superimposed on a part of the side image, but because the vehicle behind is far away and the area where the rear image is superimposed is small, there is little loss of objects in the side image. On the other hand, when a vehicle behind approaches, synthesis is performed in a specific synthesis mode. In the specific synthesis mode, the area where the rear image is superimposed is smaller than in the first mode, or the rear image is not superimposed at all. Therefore, it is possible to reduce the loss of objects in the side image.

[0009] [2] The specific composite mode includes a second mode; In the second mode, the synthesis unit generates a reduced image of the rear image and overlays the reduced image at the center of the two connected side images or at a position where the reduced image does not overlap the object, or places the reduced image between the two side images and connects the images to generate the synthesized image. The image processing device according to [1] above.

[0010] According to [2] above, the rear image is reduced in size in the composite image generated by the second mode of composition, so the area of ​​the rear image superimposed on the side image is reduced. Also, the reduced image is positioned so that it is unlikely to overlap with objects in the side image, or so that it does not overlap. This reduces the loss of objects in the side image.

[0011] [3] The synthesis unit performs transparency processing on the reduced image. The image processing device according to [2] above.

[0012] According to [3] above, since the reduced image has high transparency, even if the reduced image is superimposed on an object in the side image, the object can be prevented from disappearing from the composite image.

[0013] [4] The synthesis unit enlarges the side image in the vertical direction, and increases the enlargement rate in the horizontal direction as the image approaches the vehicle side. The image processing device according to [2] or [3] above.

[0014] According to [4] above, the closer the side image is to the vehicle, the greater the vertical magnification. Objects in the side image are also magnified, making them less likely to disappear when the rear image is overlaid. This also creates a sense of depth in the side image, providing a natural composite image that is similar to what one would see visually.

[0015] [5] The specific composite mode includes a third mode; In the third mode, the synthesis unit generates reduced images of the side image and arranges the reduced images on the left and right of the rear vehicle in the rear image, thereby generating the synthesized image. The image processing device according to [1] above.

[0016] According to [5] above, the third mode synthesis reduces the size of the side image and generates a synthetic image in which the side image is overlaid on the left and right of the rear image. Because the rear image is not overlaid on the side image, the loss of objects in the side image due to the rear image can be reduced. In addition, because the rear vehicle in the rear image is usually located in the center of the rear image, the loss of objects in the rear image can also be reduced by overlaying the side image on the rear image.

[0017] [6] The specific mode includes a fourth mode; In the fourth mode, the synthesis unit performs transparency processing on both ends of the rear image and overlays the transparently processed rear image at the center of the two connected side images to generate the synthesis image, Of both ends of the rear image, a transparency processing range of the end on the side image side where the object is not detected is changed according to the position of the rear vehicle in the rear image, and a transparency processing range of the end on the side image side where the object is detected is fixed. The image processing device according to [1] above.

[0018] According to [6] above, by combining the fourth mode, a rear image with both edges transparently processed is placed in the center of the connected side images. The transparent processing range of the edges of the rear image is narrowed depending on the proximity level of the vehicle behind in the rear image, but the transparent processing range of the edges of the side image where the object is detected is fixed and not narrowed. Because the transparent processing range remains wide, it is possible to reduce the loss of objects in the side image.

[0019] [7] The determination unit further determines whether the object is detected from a specific region of the side image; The synthesis unit switches the first mode to the specific synthesis mode when it is determined that the rear vehicle in the rear image approaches the vehicle and the object is detected from the specific area. The image processing device according to any one of [1] to [6] above.

[0020] According to [7] above, when a rear image is superimposed on the vehicle-side end of a side image, it is possible to determine whether an object is located in a specific area and whether it is likely to be lost due to the superimposed rear image.

[0021] [8] The specific area is located at the end of the side image on the vehicle side, The determination unit determines a proximity level between the vehicle and the rear vehicle, and changes the size of the specific area in accordance with the proximity level. The image processing device according to [7] above.

[0022] According to the above [8], when a vehicle behind approaches and the display range of the rear image is expanded, it becomes easier to detect an object that disappears from the composite image.

[0023] [9] In the first mode, the synthesis unit performs transparency processing on both ends of the rear image, and generates the synthesis image by superimposing the transparently processed rear image on the center of the two connected side images, and changes the transparency processing range of both ends of the rear image depending on the position of the rear vehicle in the rear image. The image processing device according to any one of [1] to [8] above.

[0024] According to [9] above, in the first mode, the area in which the rear image is superimposed on the side image can be expanded to include the rear vehicle in the rear image, thereby reducing the possibility of the rear vehicle in the rear image disappearing from the composite image.

[0025]

[10] The determination unit determines that the rear vehicle is approaching the vehicle when the position of the edge of the rear vehicle detected from the rear image exceeds a reference line arranged in the horizontal or vertical direction of the rear image. The image processing device according to any one of [1] to [9] above.

[0026] According to the above

[10] , the approach of a vehicle behind can be easily determined by comparing with the reference line.

[0027]

[11] The detection unit detects the type of the object, The determination unit moves the position of the reference line depending on the type of the following vehicle. The image processing device according to

[10] above.

[0028] According to the above

[11] , by changing the position of the reference line depending on the type of the vehicle behind, it is possible to easily adjust the timing for determining whether the vehicle behind is approaching.

[0029]

[12] A system for providing a photographed image of a rear of a vehicle, an image processing device that generates a single composite image from a plurality of captured images of the area behind the vehicle; a display device that displays the composite image, The image processing device includes: a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is less than in the first mode, or the rear image is not superimposed on the side image. system.

[0030] According to

[12] above, synthesis is performed in the first mode until a vehicle behind approaches. In the first mode, the rear image is partially superimposed on a part of the side image, but because the vehicle behind is far away and the area where the rear image is superimposed is small, there is little loss of objects in the side image. On the other hand, when a vehicle behind approaches, synthesis is performed in a specific synthesis mode. In the specific synthesis mode, the area where the rear image is superimposed is smaller than in the first mode, or the rear image is not superimposed at all. Therefore, loss of objects in the side image can be reduced.

[0031]

[13] To the computer, a step of synthesizing a rear image taken behind the vehicle and two side images taken respectively of the left and right rear sides to generate one synthetic image; detecting an object including a rear vehicle from the rear image and the side image; a step of determining whether the rear vehicle detected from the rear image has approached the vehicle, the step of generating the composite image includes a step of generating the composite image in a first mode until it is determined that the rear vehicle is approaching, a step of switching the first mode to a specific composite mode when it is determined that the rear vehicle is approaching, and a step of generating the composite image in the specific composite mode, The step of generating the composite image in the first mode includes superimposing the rear image on a part of the side image, and widening the area where the rear image is superimposed as the rear vehicle approaches, In the step of generating the composite image by the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is less than in the first mode, or the rear image is not superimposed on the side image. program.

[0032] According to

[13] above, synthesis is performed in the first mode until a vehicle behind approaches. In the first mode, the rear image is partially superimposed on a part of the side image, but because the vehicle behind is far away and the area where the rear image is superimposed is small, there is little loss of objects in the side image. On the other hand, when a vehicle behind approaches, synthesis is performed in a specific synthesis mode. In the specific synthesis mode, the area where the rear image is superimposed is smaller than in the first mode, or the rear image is not superimposed at all. Therefore, it is possible to reduce the loss of objects in the side image. [Effects of the Invention]

[0033] According to the present invention, it is possible to reduce the loss of objects from the composite image. [Brief explanation of the drawings]

[0034] [Figure 1] 1 is a block diagram showing a configuration of a driving assistance system according to an embodiment; [Figure 2] FIG. 10 is a diagram showing an example of the arrangement of display devices in a vehicle. [Figure 3] 5 is a flowchart showing a synthesis process for generating a synthetic image in the first embodiment. [Figure 4A] 10A and 10B are diagrams illustrating a method for determining the approach of a rear vehicle. [Figure 4B] 10A and 10B are diagrams illustrating a method for determining the approach of a rear vehicle. [Figure 5A] FIG. 10 is a diagram illustrating a synthesis method of the first mode. [Figure 5B] FIG. 10 is a diagram illustrating a synthesis method of the first mode. [Figure 6A] FIG. 10 is a diagram illustrating the disappearance of a vehicle behind a vehicle from a composite image. [Figure 6B]FIG. 10 is a diagram showing an example of a composite image in a second mode. [Figure 7A] FIG. 10 is a diagram showing another example of a composite image in the second mode. [Figure 7B] FIG. 10 is a diagram showing another example of a composite image in the second mode. [Figure 7C] FIG. 10 is a diagram showing another example of a composite image in the second mode. [Figure 8] 10 is a flowchart showing a synthesis process in the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a specific region. [Figure 10] FIG. 10 is a diagram showing an example of a composite image in a third mode. [Figure 11] 11 is a flowchart showing a synthesis process in the third embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a composite image in a fourth mode. DETAILED DESCRIPTION OF THE INVENTION

[0035] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image processing apparatus, a system, and a program according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0036] First Embodiment FIG. 1 shows the configuration of a driving assistance system 100 according to the first embodiment. The driving assistance system 100 is mounted on a vehicle and assists the driver in driving by displaying captured images of the rear and rear sides of the vehicle.

[0037] 1, the driving assistance system 100 includes an image display system 25, three cameras 3C, 3L, and 3R, various sensors 4, and a measuring device 5. The image display system 25 includes an image processing device 10 and a display device 20.

[0038] Cameras 3C, 3L, and 3R continuously capture images of the rear, left rear side, and right rear side of the vehicle, respectively, and generate sequentially captured images in chronological order. Hereinafter, the captured image of the rear side will be referred to as a rear image, and the captured image of the rear side will be referred to as a side image. Camera 3C, which captures images of the rear side, is located, for example, near the license plate. Cameras 3L and 3R, which capture images of the left and right rear side, are located, for example, at the positions of the left and right side mirrors, respectively.

[0039] The measurement device 5 measures the position, direction, distance to the object, moving speed, etc. of an object around the vehicle, such as a vehicle or a person. Examples of the measurement device 5 include ultrasonic sonar and millimeter wave radar. The driving assistance system 100 can acquire measurement information of the object around the vehicle from these measurement devices 5.

[0040] The sensor 4 detects the state of the vehicle. Examples of the sensor 4 include various sensors mounted on the vehicle, such as a vehicle speed sensor, a steering angle sensor, and a turn signal sensor. The driving assistance system 100 can acquire detection signals from these sensors 4 as information indicating the state of the vehicle.

[0041] (display device) The display device 20 displays the composite image generated by the image processing device 10. The display device 20 may be an in-vehicle display having a general aspect ratio, or may be a rearview mirror type display having a long ratio of width to height.

[0042] FIG. 2 shows an example of the arrangement of the display device 20 in a vehicle. The display device 20 is installed on a dashboard 21 inside the vehicle or near the upper center of a windshield 22 in place of an inner mirror. In FIG. 2, the display device 20 is configured with a single horizontally long liquid crystal display or the like, and one composite image J is displayed on it.

[0043] The images (videos) taken by cameras 3C, 3L, and 3R are combined, and one composite image J is displayed on display device 20. In composite image J, image JL taken by camera 3L is displayed on the left side, image JC taken by camera 3C is displayed in the center, and image JR taken by camera 3R is displayed on the right side.

[0044] The display device 20 may be configured with a plurality of displays connected in a row. In this case, one composite image J is divided, and each divided image is displayed in the display area of ​​each display.

[0045] (Image processing device) The image processing device 10 combines the rear image captured by the camera 3C and the side images captured by the cameras 3L and 3R into one image.

[0046] The image processing device 10 includes a correction unit 11, a detection unit 12, a determination unit 13, a synthesis unit 14, and a display control unit 15. The functions of each unit of the image processing device 10 may be realized by software processing that is executed by reading a program by a computer such as an ECU (Electronic Control Unit) or a microcomputer mounted on a vehicle. Alternatively, each unit of the image processing device 10 may be configured by hardware such as an FPGA (Field-Programmable Gate Array).

[0047] The correction unit 11 performs correction processes such as distortion correction and color correction on the rear image and the side image.

[0048] The detection unit 12 detects objects included in each image after correction processing. Examples of objects to be detected include vehicles, people, animals, guardrails, buildings, etc. The detection method is not particularly limited, but detection can be performed using deep learning, for example.

[0049] In deep learning, the detection unit 12 uses multiple images of detection targets, such as vehicles of different models and perspectives, captured from various directions as training data, to learn the characteristics of the detection target objects. For example, the detection unit 12 can learn the characteristics of buses and passenger cars and detect image portions with similar characteristics as vehicles. By learning the characteristics of each model, the model of the vehicle can also be identified.

[0050] The detection unit 12 can set a detection frame in the image that surrounds the detected object. In this embodiment, the detection frame is rectangular, but the shape is not limited to a rectangle as long as the size of the detection frame approximately matches the size of the object. For example, the detection frame may have a shape that outlines the outline of the object, or may have an elliptical shape whose horizontal and vertical sizes are the same as those of the object.

[0051] The determination unit 13 determines whether a rear vehicle detected from the rear image by the detection unit 12 has approached the vehicle. The determination unit 13 can use information such as the state of the vehicle detected by the sensor 4 or the position and speed of an object measured by the measurement device 5 for the determination.

[0052] The synthesis unit 14 synthesizes the corrected rear image and side image to generate one synthesized image. The synthesis unit 14 has a plurality of synthesis modes and switches the synthesis mode depending on the determination result of the determination unit 13.

[0053] The display control unit 15 causes the display device 20 to display the composite image generated by the composition unit 14.

[0054] 3 shows the synthesis process in which the image processing device generates a synthesized image. This synthesis process is repeated every time one frame of captured image is input by each of the cameras 3C, 3L, and 3R. When the rear and side images are captured by the cameras 3C, 3L, and 3R, the correction unit 11 of the image processing device 10 corrects each image (step S11). The detection unit 12 detects objects and their types included in the corrected rear and side images (step S12).

[0055] If a vehicle is included in the objects detected from the rear image, that is, if a rear vehicle is detected from the rear image, the determination unit 13 determines whether the rear vehicle has approached the vehicle (step S13). The determination unit 13 can determine the approach of a rear vehicle based on measurement information acquired from the measurement device 5, or can determine the approach of a rear vehicle by analyzing the rear image.

[0056] 4A and 4B illustrate a method for determining the approach of a rear vehicle by analyzing a rear image. 4A, the determination unit 13 arranges, on the rear image JC, two reference lines L11 parallel to the vertical direction y and two reference lines L21 parallel to the horizontal direction x. The two reference lines L11 are arranged at positions on the left and right a certain distance away from the center line of the rear image JC in the horizontal direction x. The two reference lines L21 are arranged at positions each a predetermined distance away from the bottom edge of the rear image JC.

[0057] As the following vehicle 51 approaches the vehicle, the following vehicle 51 is positioned lower in the rear image JC and becomes larger in size in the image. Therefore, the determination unit 13 can determine that the following vehicle 51 is approaching the vehicle when the positions of the ends of the following vehicle 51 detected in the rear image JC, for example, the positions of the four sides of the detection frame set for the following vehicle 51, are outside and beyond all of the reference lines L11 and L21. In this way, the approach of the following vehicle 51 can be easily determined by comparing with the reference lines L11 and L21.

[0058] The positions of the reference lines L11 and L21 can be determined by measuring them in advance. For example, in a rear image JC when the rear vehicle 51 approaches to a position 15 m behind the vehicle, the positions of both ends of the rear vehicle 51 in the horizontal direction x and the vertical direction y can be measured and determined as the positions of the reference lines L11 and L21.

[0059] As in the present embodiment, it is preferable to perform the determination in both the horizontal direction x and the vertical direction y using the reference lines L11 and L21, as this provides high accuracy of determination. For a simpler determination, it is possible to perform the determination in only one of the horizontal direction x and the vertical direction y using either the reference lines L11 or L21.

[0060] The determination unit 13 can move the positions of the reference lines L11 and L21 depending on the type of vehicle 51 behind. For example, large vehicles such as trucks and buses have a wider vehicle width than standard vehicles and light vehicles. Therefore, as shown in FIG. 4B , by determining the approach using a reference line L12 that is closer to the reference line L11 or a reference line L22 that is located above the reference line L21, the timing for determining the approach can be advanced. The timing for determining the approach of the rear vehicle 51 can be easily adjusted by a simple operation of changing the positions of the reference lines L11 and L21.

[0061] If it is determined that the following vehicle 51 is not approaching (step S13: NO), the composition unit 14 switches the composition mode to the first mode (step S14). If it is determined that the following vehicle 51 is approaching (step S13: YES), the composition unit 14 switches the composition mode to the second mode (step S17). The composition unit 14 generates a composite image in the switched composition mode using the corrected rear image JC and side images JL and JR.

[0062] 5A and 5B illustrate the synthesis method according to the first mode. 5A, the shooting range 61C of the camera 3C partially overlaps with the shooting ranges 61L and 61R of the cameras 3L and 3R. The composition unit 14 adjusts the trimming ranges 62C, 62L, and 62R, pan, tilt, and roll of the rear image JC and the side images JL and JR so that the rear image JC and the side images JL and JR are seamlessly composed on a composition plane L3 n(m) (e.g., n=15) behind the vehicle 50.

[0063] The composition unit 14 connects the left and right side images JL and JR and overlays the rear image JC in the center to generate a layered composite image J (step S15). The composition unit 14 narrows the display range 63C of the rear image JC in the composite image J and widens the display ranges 63L and 63R of the side images JL and JR by performing transparency processing on both end portions Cs of the rear image JC (step S16).

[0064] 5A, areas that are not included in any of the trimming ranges 62C, 62L, and 62R are blind spots on the composite image J. If a rear vehicle 51 detected in the rear image JC moves into this blind spot due to a lane change or the like, the rear vehicle 51 will disappear from the composite image J. To prevent such disappearance, in the first mode, the composition unit 14 changes the position of the boundary line L1 that determines the display range 63C of the rear image JC in the composite image J in accordance with the position of the rear vehicle 51 in the rear image JC.

[0065] As shown in FIG. 5B , when the following vehicle 51 approaches the vehicle 50 and moves to a position on the right rear side, the composition unit 14 can move the position of the boundary line L1 toward the vehicle 50 so that the entire detection frame of the following vehicle 51 detected from the rear image JC is included in the rear image JC. As a result, the range of the right edge Cs to be subjected to transparency processing in the rear image JC can be reduced. This changes the trimming ranges 62C, 62L, and 62R, and the display range 63C of the rear image JC in the composite image J is expanded to the right, thereby reducing the display range 63R of the right side image JR by that amount. In this way, by changing the display range 63C of the rear image JC, the position of the blind spot in the composite image J is shifted, reducing the disappearance of the following vehicle 51.

[0066] In the first mode, the rear vehicles in the side images JL and JR may disappear from the composite image J due to a change in the display range 63C of the rear image JC. FIG. 6A shows an example of a composite image in the first mode. As shown in FIG. 6A, a rear vehicle 51 traveling in the same lane as the vehicle 50 is detected in the rear image JC, and a rear vehicle 52 traveling in the adjacent lane is detected in the left side image JL.

[0067] In the first mode, the closer the rear vehicle 51 is to the vehicle 50, the wider the display range 63C of the rear image JC becomes in order to prevent the rear vehicle 51 from disappearing. That is, the area of ​​the rear image JC that overlaps the side images JL and JR becomes wider. As a result, if the rear image JC also overlaps the rear vehicle 52 in the side image JL, the rear vehicle 52 will disappear from the composite image J. In order to prevent such disappearance, when it is determined that the rear vehicle 51 in the rear image JC has approached, the composite unit 14 switches to the second mode.

[0068] FIG. 6B shows an example of a composite image in the second mode. As shown in FIG. 6B, in the second mode, the synthesis unit 14 generates a reduced image JCa of the rear image JC (step S18).

[0069] The composition unit 14 connects the left and right side images JL and JR. At this time, the composition unit 14 can enlarge the left and right side images JL and JR. This also enlarges the size of the rear vehicle 52 in the side images JL and JR, making it easier to confirm the rear vehicle 52. The composition unit 14 overlays a reduced image JCa of the rear image JC in the center of the connected side images JL and JR to generate a composite image J (step S19). Because the area over which the rear image JC is overlaid is reduced by the reduction, the rear image JC does not overlap the rear vehicle 52, and the rear vehicle 52 does not disappear from the composite image J.

[0070] The composition unit 14 can display an icon image 50a of the vehicle 50 in the center of the bottom of the composite image J, and place a reduced image JCa of the rear image JC above it. Although the reduction in the rear image JC makes it difficult to grasp the sense of distance to the rear vehicle 51, the icon image 50a makes the positional relationship between the vehicle 50 and the rear vehicle 51 clear.

[0071] 7A to 7C show other examples of composite images in the second mode. 7A, the composition unit 14 can perform transparency processing on the reduced image JCa of the rear image JC. Because the reduced image JCa has high transparency, even when the reduced image JCa is superimposed on the rear vehicle 52 in the side image JL, the rear vehicle 52 can be prevented from disappearing from the composite image J.

[0072] When transparency processing is performed, the reduction ratio of the reduced image JCa may be smaller than when transparency processing is not performed. Because the size of the reduced image JCa is larger, it is easier for the driver to recognize the vehicle 51 behind. Even though the size is large, because transparency processing is performed, the vehicle 52 behind the vehicle 52 can also be seen.

[0073] 7B, the composition unit 14 can superimpose the reduced image JCa of the rear image JC at a position in the side image JL that does not overlap with the rear vehicle 52. For example, the composition unit 14 can detect the horizon or lane lines in the side image JL and place the reduced image JCa above these lines in an area that is not a road. Because the reduced image JCa does not overlap the rear vehicle 52, it is possible to prevent the rear vehicle 52 from disappearing from the composite image J.

[0074] 7C, the composition unit 14 may connect the images by placing a reduced image JCa of the rear image JC between the side images JL and JR. Since the rear image JC does not overlap with the side images JL and JR, it is possible to prevent the rear vehicle 52 from disappearing from the composite image J.

[0075] The synthesis unit 14 may modify the side images JL and JR to provide a natural-looking synthetic image J. Specifically, as shown in FIG. 7C , the synthesis unit 14 enlarges the side images JL and JR in the vertical direction y, increasing the magnification rate as they approach the vehicle 50. Furthermore, the synthesis unit 14 preferably reduces the side images JL and JR in the horizontal direction x. In the side images JL and JR modified in this way, the point at infinity becomes farther away, resulting in a natural image with depth. A synthetic image J can be provided that makes it easy to grasp the sense of distance to the vehicle 52 behind, and blind spots near the vehicle 50 on the synthetic image J are also reduced.

[0076] As described above, according to the first embodiment, the approach of the rear vehicle 51 is determined in the rear image JC, and the composite image J is generated in the first mode until the rear vehicle 51 approaches, and in the second mode when the rear vehicle 51 approaches. In the second mode, the rear image JC is reduced in size, so even if the rear image JC overlaps a portion of the side image JL or JR, the overlap is smaller than in the first mode. Alternatively, the rear image JC does not overlap the side image JL or JR. This reduces the possibility of the rear image JC overlapping with the rear vehicle 52 in the side images JL and JR and disappearing.

[0077] Second Embodiment In the second embodiment, the following synthesis process is performed in the image processing device 10. The configurations of the image processing device 10 and the driving assistance system 100 in the first and second embodiments are the same, except for the content of the synthesis process.

[0078] 8 shows the synthesis process in the second embodiment. This synthesis process is repeated every time a rear image JC and side images JL and JR are input from the cameras 3C, 3L and 3R. When the rear image JC and the side images JL and JR are generated by the cameras 3C, 3L, and 3R, the correction unit 11 of the image processing device 10 corrects each image (step S21). The detection unit 12 detects objects and their types included in the corrected rear image JC, side images JL, and JR (step S22).

[0079] When a following vehicle 51 is detected from the rear image JC, the determination unit 13 determines whether the following vehicle 51 has approached the vehicle 50 (step S23). The determination method is the same as in the first embodiment, and therefore a detailed description thereof will be omitted here.

[0080] If it is determined that the rear vehicle 51 in the rear image JC is approaching (step S23: YES), the determination unit 13 further determines whether or not an object has been detected in a specific area of ​​the side images JL and JR (step S24). The specific area is the area on which the rear image JC is superimposed in the first mode. That is, in step S24, it is determined whether or not there is an object in the side images JL and JR that may be lost from the composite image J due to the rear image JC. The object to be determined here is an object that should alert the driver of a vehicle, a person, or a motorcycle, etc.

[0081] FIG. 9 shows an example of a specific region. The specific area 66 is located at the end of the left side image JL and the right side image JR on the vehicle 50 side. The determination unit 13 can change the size of the specific area 66 depending on the approach level of the rear vehicle 51 in the rear image JC. When the rear vehicle 51 is located far from the vehicle 50, the display range 63C of the rear image JC is not expanded, so the size of the specific area 66 can be small. Conversely, when the rear vehicle 51 is close to the vehicle 50, the display range 63C is expanded, so by increasing the size of the specific area 66, it becomes easier to detect an object that disappears from the composite image J, such as the rear vehicle 52 in the side image JL.

[0082] The approach level of the rear vehicle 51 can be determined based on the distance of the rear vehicle 51 from vehicle 50, the moving speed, or the time it takes for the rear vehicle 51 to reach vehicle 50. The distance, moving speed, or arrival time can be measured by the measurement device 5, or the determination unit 13 can analyze the position, size, or the like of the rear vehicle 51 in the rear image JC. For example, large, medium, and small sized specific regions 66 are assigned to three approach levels: a distance of less than 15 m, 15 m to 30 m, and 30 m or more from vehicle 50. The determination unit 13 can determine the approach level from the distance measured by the measurement device 5, and use the specific region 66 of the size assigned to the approach level.

[0083] If the following vehicle 51 in the rear image JC is not approaching (step S23: NO), the composition unit 14 switches the composition mode to the first mode (step S25). Even if the following vehicle 51 in the rear image JC is approaching (step S23: YES), if no object is detected in the specific area 66 of the side image JL or JR (step S24: NO), the composition unit 14 switches to the first mode (step S25).

[0084] In the first mode, the composition unit 14 connects the side images JL and JR and overlays the rear image JC in the center to generate a composite image J (step S26). Furthermore, the composition unit 14 performs transparency processing on both end portions Cs of the rear image JC (step S27). The composition method in the first mode is the same as the method described in the first embodiment, so a detailed description will be omitted here.

[0085] On the other hand, if a rear vehicle 51 in the rear image JC approaches (step S23: YES) and an object is detected in the specific area 66 of the side images JL and JR (step S24: YES), the synthesis unit 14 switches the synthesis mode to the third mode (step S28).

[0086] FIG. 10 shows an example of a composite image in the third mode. In the third mode, the composition unit 14 enlarges the rear image JC to fill the entire display area of ​​the display device 20 (step S29). Next, the composition unit 14 generates reduced images JLa and JRa of the side images JL and JR and overlays them on the left and right of the rear image JC to generate a composite image J (step S30). The composition unit 14 can place the reduced images JLa and JRa on the left and right of the rear vehicle 51 in the rear image JC, for example, at the position of the door mirrors. This makes it easier to grasp the positional relationship and distance between the rear vehicle 51 and objects on the left and right sides, for example, rear side vehicles 52.

[0087] As described above, according to the second embodiment, the approach of the rear vehicle 51 in the rear image JC and objects (e.g., rear lateral vehicle 52, etc.) in the side images JL and JR are determined, and a composite image J is generated in the first mode until the rear vehicle 51 approaches and an object is detected, and in the third mode when the rear vehicle 51 approaches and an object is detected.

[0088] In the third mode, the rear image JC is not superimposed on the side images JL and JR, so the rear vehicle 52 does not disappear in the side images JL and JR. In addition, since the rear vehicle 51 in the rear image JC is usually positioned in the center of the rear image JC, the disappearance of the rear vehicle 51 due to the side images JL and JR can also be reduced.

[0089] <Third embodiment> In the third embodiment, the following synthesis process is performed in the image processing device 10. The configurations of the image processing device 10 and the driving assistance system 100 in the first and third embodiments are the same, except for the content of the synthesis process.

[0090] 11 shows the synthesis process in the third embodiment. This synthesis process is repeated every time a rear image JC and side images JL and JR are input from the cameras 3C, 3L and 3R. When the rear image JC and the side images JL and JR are generated by the cameras 3C, 3L, and 3R, the correction unit 11 of the image processing device 10 corrects each image (step S41). The detection unit 12 detects objects and their types included in the corrected rear image JC, side images JL, and JR (step S42).

[0091] When a following vehicle 51 is detected from the rear image JC, the determination unit 13 determines whether or not the following vehicle 51 has approached the vehicle 50 (step S43). The determination method is the same as in the first embodiment, and therefore a detailed description thereof will be omitted here.

[0092] If it is determined that the rear vehicle 51 in the rear image JC has approached (step S43: YES), the determination unit 13 further determines whether or not an object has been detected in the specific area 66 of the side images JL and JR (step S44). The determination method is the same as in the second embodiment, and therefore a detailed description thereof will be omitted here.

[0093] If the following vehicle 51 in the rear image JC is not approaching (step S43: NO), the composition unit 14 switches the composition mode to the first mode (step S45). Also, even if the following vehicle 51 in the rear image JC is approaching (step S43: YES), if no object is detected in the specific area 66 of the side image JL or JR (step S44: NO), the composition unit 14 switches to the first mode (step S45).

[0094] In the first mode, the composition unit 14 connects the side images JL and JR and overlays the rear image JC in the center to generate a composite image J (step S46). Furthermore, the composition unit 14 performs transparency processing on both end portions Cs of the rear image JC (step S47). The composition method in the first mode is the same as the method described in the first embodiment, so a detailed description will be omitted here.

[0095] On the other hand, if a rear vehicle 51 in the rear image JC approaches (step S43: YES) and an object is detected in the specific area 66 of the side images JL and JR (step S44: YES), the synthesis unit 14 switches the synthesis mode to the fourth mode (step S48).

[0096] FIG. 12 shows an example of a composite image in the fourth mode. In the fourth mode, the composition unit 14 superimposes the rear image JC on the side images JL and JR to compose them in the same manner as in the first mode (step S49). The composition unit 14 also performs transparency processing on the edge Cs of the rear image JC (step S50).

[0097] In the first mode, the range of the end Cs to be subjected to transparency processing varies depending on the position of the rear vehicle 51 in the rear image JC. In the fourth mode, the composition unit 14 also changes the range of transparency processing of the end Cs on the side image JL or JR side where the rear vehicle 52 is detected, depending on the position of the rear vehicle 51 in the rear image JC.

[0098] However, in the fourth mode, the composition unit 14 fixes the transparency processing range of the end Cs on the side image JL side where the rear vehicle 52 is detected, as shown in Fig. 12. This prevents the area where the rear image JC is superimposed on the side image JL from expanding even when the rear vehicle 51 approaches, thereby reducing the loss of the rear vehicle 52 due to the overlap of the rear image JC.

[0099] As described above, according to the third embodiment, the approach of the following vehicle 51 in the rear image JC and the object that may disappear in the side images JL and JR are determined. The composite image J is generated in the first mode until it is determined that the following vehicle 51 is approaching and an object is present, and in the fourth mode when it is determined that the following vehicle 51 is approaching and an object is present.

[0100] In the fourth mode, the display range 63C of the rear image JC is not expanded toward the side image JL or JR where an object is detected, and the display range 63L or 63R of the side image JL or JR is fixed. The rear image JC is superimposed on a portion of the side image JL or JR, but the overlap does not expand and the overlapping area is smaller than in the first mode. This reduces the possibility that the rear vehicle 52 in the side image JL or LR will disappear from the composite image J due to the rear image JC.

[0101] The above embodiment is a preferred example of the present invention, and the present invention is not limited to this and can be modified as appropriate. For example, in the second and third embodiments described above, the mode is switched to the third mode or the fourth mode when the following vehicle 51 approaches and there is an object in the specific area 66 of the side image JL or JR. This makes it possible to avoid unnecessary switching of the synthesis mode when there is no object in the specific area 66, since the object does not disappear. However, once it is determined that the following vehicle 51 is approaching, the mode may be switched to the third mode or the fourth mode regardless of whether there is an object in the specific area 66.

[0102] In the first embodiment, only the approach of the following vehicle 51 is determined, but as in the second and third embodiments, the presence or absence of an object in the specific area 66 of the side image JL or JR may also be determined. When it is determined that the following vehicle 51 is approaching and that an object exists in the specific area 66, the mode may be switched to the second mode. [Explanation of symbols]

[0103] 100 Driver Assistance Systems 25 Image display system 10 Image processing device 12 Detector 13 Judgment section 14 Synthesis section 15 Display control unit 20 Display device 3C, 3L, 3R cameras

Claims

1. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; the specific synthesis mode includes a second mode in which the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode; In the second mode, the synthesis unit generates a reduced image of the rear image and overlays the reduced image at a center of the two connected side images or at a position where the reduced image does not overlap the object, thereby generating the synthesized image. Image processing device.

2. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode, or the rear image is not superimposed on the side image; the specific blending mode includes a second mode, In the second mode, the synthesis unit generates a reduced image of the rear image, performs transparency processing on the reduced image, and generates the synthesized image by either superimposing the reduced image at the center of the two connected side images or at a position where the reduced image does not overlap the object, or by arranging the reduced image between the two side images and connecting the images. Image processing device.

3. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode, or the rear image is not superimposed on the side image; the specific blending mode includes a second mode, In the second mode, the synthesis unit generates a reduced image of the rear image, and overlays the reduced image at the center of the two connected side images or at a position where the reduced image does not overlap the object, or places the reduced image between the two side images and connects the images to generate the synthesized image; The synthesis unit vertically enlarges the side image, and increases the enlargement rate in the horizontal direction as the image approaches the vehicle side. Image processing device.

4. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode, or the rear image is not superimposed on the side image; the specific blending mode includes a third mode, In the third mode, the synthesis unit generates reduced images of the side image and arranges the reduced images on the left and right of the rear vehicle in the rear image, thereby generating the synthesized image. Image processing device.

5. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; the specific synthesis mode includes a fourth mode in which the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode; In the fourth mode, the synthesis unit performs transparency processing on both ends of the rear image and overlays the transparently processed rear image at the center of the two connected side images to generate the synthesis image, Of both ends of the rear image, a transparency processing range of the end on the side image side where the object is not detected is changed according to the position of the rear vehicle in the rear image, and a transparency processing range of the end on the side image side where the object is detected is fixed. Image processing device.

6. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode, or the rear image is not superimposed on the side image; the determination unit further determines whether the object is detected from a specific region of the side image; The synthesis unit switches the first mode to the specific synthesis mode when it is determined that the rear vehicle in the rear image approaches the vehicle and the object is detected from the specific area. Image processing device.

7. the specific area is located at an end of the side image on the vehicle side, The determination unit determines a proximity level between the vehicle and the rear vehicle, and changes the size of the specific area in accordance with the proximity level. The image processing device according to claim 6 .

8. An image processing device that generates a single composite image from multiple captured images of an area behind a vehicle, a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; In the specific synthesis mode, the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode, or the rear image is not superimposed on the side image; In the first mode, the synthesis unit performs transparency processing on both ends of the rear image, and generates the synthesis image by superimposing the transparently processed rear image on the center of the two connected side images, and changes the transparency processing range of both ends of the rear image depending on the position of the rear vehicle in the rear image. Image processing device.

9. The determination unit determines that the rear vehicle has approached the vehicle when a position of an edge of the rear vehicle detected from the rear image exceeds a reference line arranged in a horizontal or vertical direction of the rear image. The image processing device according to any one of claims 1 to 8.

10. the detection unit detects the type of the object; The determination unit moves the position of the reference line depending on the type of the following vehicle. The image processing device according to claim 9 .

11. A system for providing a captured image of the rear of a vehicle, an image processing device that generates a single composite image from a plurality of captured images of the area behind the vehicle; a display device that displays the composite image, The image processing device includes: a synthesis unit that synthesizes a rear image captured behind the vehicle with two side images captured respectively of the left and right rear sides of the vehicle to generate the synthesized image; a detection unit that detects objects including a rear vehicle from the rear image and the side image; a determination unit that determines whether the rear vehicle detected from the rear image has approached the vehicle, The synthesis unit generating the composite image in a first mode until it is determined that the rear vehicle has approached, and when it is determined that the rear vehicle has approached, switching the first mode to a specific composite mode to generate the composite image; In the first mode, the rear image is superimposed on a part of the side image, and the area where the rear image is superimposed is widened as the rear vehicle approaches; the specific synthesis mode includes a second mode in which the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode; In the second mode, the synthesis unit generates a reduced image of the rear image and overlays the reduced image at a center of the two connected side images or at a position where the reduced image does not overlap the object, thereby generating the synthesized image. system.

12. On the computer, a step of synthesizing a rear image taken behind the vehicle and two side images taken respectively of the left and right rear sides to generate one synthetic image; detecting an object including a rear vehicle from the rear image and the side image; a step of determining whether the rear vehicle detected from the rear image has approached the vehicle, the step of generating the composite image includes a step of generating the composite image in a first mode until it is determined that the rear vehicle is approaching, a step of switching the first mode to a specific composite mode when it is determined that the rear vehicle is approaching, and a step of generating the composite image in the specific composite mode, The step of generating the composite image in the first mode includes superimposing the rear image on a part of the side image, and widening the area where the rear image is superimposed as the rear vehicle approaches, the specific synthesis mode includes a second mode in which the rear image is superimposed on a part of the side image, but the overlap between the object detected from the side image and the rear image is smaller than in the first mode; The step of generating the composite image in the specific synthesis mode generates a reduced image of the rear image in the second mode, and overlays the reduced image at the center of the two connected side images or at a position that does not overlap the object, thereby generating the composite image. program.

Images