Image Processing Apparatus, System, and Program

Abstract

To provide a synthesis image that allows a user to easily grasp a back and an object in a back.SOLUTION: An image processor 10 includes: a synthesis unit 14 for synthesizing a rear image showing the back of a vehicle and two side images showing the left back side and the right back side of the vehicle; a detection unit 12 for detecting an object including a back vehicle from the rear image and the side images; and a determination unit 13 for determining whether there is a blind spot object which disappears from the synthesis image due to the back vehicle and is highly dangerous, among the objects detected from the side images, when the back vehicle is detected from the rear image. The synthesis unit 14 synthesizes alert information of the blind spot object on the synthesis image if the presence of the blind spot object has been determined.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a system, and a program.

Background Art

[0002] Conventionally, a driving support system has been used that captures the rear and left and right rear sides of a vehicle with a camera, synthesizes them into one image, and displays the image on a display inside the vehicle. Each captured image may be processed so that the synthesized image can be observed in the same way as when viewed. At this time, a rear vehicle, a pedestrian, etc. in the captured image may disappear from the synthesized image.

[0003] For example, image processing may be performed to superimpose the rear captured image on the rear side captured image so that the positions of the lanes in each captured image match. If a rear side vehicle traveling in an adjacent lane is included in the area where the rear captured image is superimposed, the rear side vehicle disappears from the synthesized image.

[0004] Thus, a technique has been proposed to warn the driver of the existence of an object that does not appear in the synthesized image. For example, according to Patent Document 1, a warning mark is superimposed on an object such as a motorcycle hidden behind a rear vehicle traveling in the same lane as the vehicle in the synthesized image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when warning marks are superimposed on the periphery of the rear vehicle, it becomes difficult for the driver to check the rear vehicle. When there are many objects hidden behind the rear vehicle, many warning marks are superimposed, making it difficult to grasp the positions where each object is hidden. Also, when there are many warning marks, it becomes impossible to distinguish objects that the driver should particularly pay attention to, such as vehicles traveling at high speed or motorcycles, among them.

[0007] An object of the present invention is to provide a composite image that makes it easy to grasp objects behind and at the rear sides of a vehicle.

Means for Solving the Problems

[0008] The present invention includes the following aspects. [1] An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, a composite unit that composites a rear image captured behind the vehicle and two side images captured respectively from the left rear side and the right rear side to generate the composite image; a detection unit that detects an object including a rear vehicle from the rear image and the side images; a determination unit that, when the rear vehicle is detected from the rear image, determines the presence or absence of a blind spot object that disappears from the composite image due to the rear image among the objects detected from the side images and has a high degree of danger, wherein when it is determined that there is the blind spot object, the composite unit superimposes warning information of the blind spot object on the composite image image processing apparatus.

[0009] According to the above [1], due to the warning information, the driver can early grasp the presence of the blind spot object in the side image that disappears from the composite image due to the rear vehicle in the rear image. The warning information warns not all blind spot objects but blind spot objects with a high degree of danger. Since the warning target is narrowed down, the driver can easily grasp the objects behind and at the rear sides that particularly require attention.

[0010] [2] The determination unit determines the level of approach of the blind spot object to the vehicle, and determines the degree of danger based on the level of approach. The image processing apparatus according to [1] above.

[0011] According to [2] above, it is possible to determine that a blind spot object with a high level of approach to the vehicle has a high degree of danger. As a result, the driver can be informed at an early stage of the existence of a blind spot object that may approach and collide with the vehicle.

[0012] [3] The composition unit changes the display form of the warning information according to the level of approach. The image processing apparatus according to [2] above.

[0013] According to [3] above, due to the difference in the display form of the warning information, the driver can easily grasp the level of approach of the blind spot object to the vehicle.

[0014] [4] The warning information includes an image indicating the type of the blind spot object. The image processing apparatus according to any one of [1] to [3] above.

[0015] According to [4] above, the driver can easily grasp the type of blind spot object such as a vehicle or a person.

[0016] [5] The composition unit arranges the warning information on the side image where the blind spot object is determined to be present. The image processing apparatus according to any one of [1] to [4] above.

[0017] According to [5] above, based on the position of the warning information, the driver can easily grasp on which side (left or right) of the rear side the blind spot object is located.

[0018] [6] The composition unit arranges the warning information in a predetermined area of the side image. The image processing apparatus according to [5] above.

[0019] According to [6] above, the warning information is always displayed at the same position and is easy to check.

[0020] [7] The determination unit determines whether or not a rear-side vehicle is detected from the side image while the rear vehicle is not detected from the rear image. When it is determined that the rear vehicle is not detected from the rear image and a rear-side vehicle is detected from the side image, the synthesizing unit overlays warning information of the rear-side vehicle on the side image. The image processing apparatus according to any one of [1] to [6] above.

[0021] According to [7] above, it is possible to warn a rear-side vehicle that does not disappear from the composite image due to a rear vehicle in the rear image.

[0022] [8] The determination unit acquires information on the blinker state of the vehicle, and changes a range for determining the presence or absence of the blind spot object in the side image according to the blinker state. The image processing apparatus according to any one of [1] to [7] above.

[0023] According to [8] above, it is possible to change the determination range in the side image in which the blinker is in the ON state to a wide area. Thereby, it is possible to widely target a left or right rear-side vehicle in which the blinker is in the ON state as a warning target.

[0024] [9] The determination unit When the rear vehicle is detected from the rear image and the rear vehicle approaches the vehicle, determines the presence or absence of the blind spot object. When the position of the end of the rear vehicle detected from the rear image exceeds a reference line arranged in the horizontal direction or the vertical direction of the rear image, it is determined that the rear vehicle has approached the vehicle. The image processing apparatus according to any one of [1] to [8] above.

[0025] According to [9] above, it is possible to easily determine the approach of the rear vehicle by comparison with the reference line.

[0026]

[10] The detection unit detects the type of the object. The determination unit moves the position of the reference line according to the type of the vehicle behind. The image processing apparatus according to [9] above.

[0027] According to

[10] above, by changing the position of the reference line according to the type of the vehicle behind, the timing for determining that the vehicle behind is approaching can be easily adjusted.

[0028]

[11] A system that provides a captured image of the rear of a vehicle, An image processing apparatus that generates one composite image from a plurality of captured images of the rear of the vehicle, A display device that displays the composite image, and is provided with, The image processing apparatus is, A composite unit that composites a rear image captured at the rear of the vehicle and two side images captured at the left rear side and the right rear side respectively, and generates the composite image, A detection unit that detects an object including a vehicle behind from the rear image and the side images, When the vehicle behind is detected from the rear image, a determination unit that determines the presence or absence of a blind spot object that disappears from the composite image due to the rear image among the objects detected from the side images and has a high risk, and is provided with, When it is determined that there is the blind spot object, the composite unit overlays warning information of the blind spot object on the composite image System.

[0029] According to

[11] above, the driver can early grasp the presence of the blind spot object in the side image that disappears from the composite image due to the vehicle behind in the rear image by the warning information. The warning information warns not all blind spot objects but blind spot objects with a high risk. Since the warning target is narrowed down, the driver can easily grasp the objects at the rear and the rear side that particularly require attention.

[0030]

[12] To a computer, A step of synthesizing a rear image captured at the rear of a vehicle and two side images captured at the left rear side and the right rear side respectively to generate one synthesized image; A step of detecting an object including a rear vehicle from the rear image and the side images; When the rear vehicle is detected from the rear image, a step of determining the presence or absence of a blind spot object that disappears from the synthesized image due to the rear image among the objects detected from the side images and has a high risk; When it is determined that there is the blind spot object, a step of superimposing warning information of the blind spot object on the synthesized image; A program for causing the above to be executed.

[0031] According to

[12] above, due to the warning information, the driver can early grasp the presence of the blind spot object of the side image that disappears from the synthesized image due to the rear vehicle in the rear image. The warning information warns not all blind spot objects but blind spot objects with a high risk. Since the warning target is narrowed down, the driver can easily grasp the rear and rear side objects that particularly require attention.

Effect of the Invention

[0032] According to the present invention, a synthesized image that makes it easy to grasp rear and rear side objects can be provided.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 10A

Figure 10B

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0034] Hereinafter, embodiments of the image processing apparatus, system, and program of the present invention will be described with reference to the drawings.

[0035] FIG. 1 shows the configuration of a driving support system 100 according to an embodiment. The driving support system 100 is mounted on a vehicle and supports the driving of a driver by displaying captured images of the rear and rear sides of the vehicle.

[0036] As shown in FIG. 1, the driving support system 100 includes an image display system 25, three cameras 3C, 3L, and 3R, various sensors 4, a measuring device 5, and the like. The image display system 25 includes an image processing apparatus 10 and a display device 20.

[0037] Cameras 3C, 3L, and 3R continuously capture images of the rear, left rear side, and right rear side of the vehicle respectively, and sequentially generate time-series captured images. Hereinafter, the captured image of the rear is referred to as a rear image, and the captured image of the rear side is referred to as a side image. The camera 3C that captures the rear is arranged, for example, near the license plate. The cameras 3L and 3R that capture the left and right rear sides are provided, for example, at the positions of the left and right side mirrors respectively.

[0038] The measuring device 5 measures the position, direction, distance to the object, moving speed of the object, etc. of objects around the vehicle, such as vehicles and people. Examples of the measuring device 5 include an ultrasonic sonar or a millimeter-wave radar. The driving support system 100 can acquire the measurement information of the objects around the vehicle from these measuring devices 5.

[0039] The sensor 4 detects the state of the vehicle. Examples of the sensor 4 include various sensors mounted on the vehicle such as a vehicle speed sensor, a steering angle sensor, and a wiper sensor. The driving support system 100 can acquire the detection signals from these sensors 4 as information indicating the state of the vehicle.

[0040] (Display device) The display device 20 displays the composite image generated by the image processing device 10. The display device 20 may be an in-vehicle display having a general aspect ratio, or may be a room mirror-type display having a long horizontal-to-vertical ratio.

[0041] FIG. 2 shows an example of the arrangement of the display device 20 in the vehicle. The display device 20 is installed on the dashboard 21 inside the vehicle or near the upper center of the front window 22 in place of the inner mirror. In FIG. 2, the display device 20 is composed of a horizontally long single liquid crystal display or the like, and one composite image J is displayed thereon.

[0042] The images (videos) captured by cameras 3C, 3L, and 3R are synthesized, and a single synthesized image J is displayed on the display device 20. In the synthesized image J, the captured image JL of camera 3L is displayed on the left side, the captured image JC of camera 3C is displayed in the center, and the captured image JR of camera 3R is displayed on the right side.

[0043] Note that the display device 20 may be composed of a plurality of displays connected in a row. In this case, a single synthesized image J is divided, and each divided image is displayed in the display area of each display.

[0044] (Image Processing Device) The image processing device 10 synthesizes the rear image captured by camera 3C and the side images captured by cameras 3L and 3R into one image.

[0045] The image processing device 10 includes a correction unit 11, a detection unit 12, a determination unit 13, a synthesis unit 14, and a display control unit 15. The functions of each part of the image processing device 10 may be realized by software processing in which a program is read and executed by a computer such as an ECU (Electronic Control Unit) or a microcomputer mounted on a vehicle. Also, each part of the image processing device 10 may be configured by hardware such as an FPGA (field-programmable gate array).

[0046] The correction unit 11 performs correction processing such as distortion correction and color correction on the rear image and the side images.

[0047] The detection unit 12 detects objects included in each image after the correction processing. Examples of the objects to be detected include vehicles, people, animals, guardrails, or buildings. The detection method is not particularly limited, but for example, detection using deep learning can be performed.

[0048] In deep learning, the detection unit 12 uses, as training data, a plurality of captured images obtained by photographing detection targets such as vehicles and people with different vehicle types and distances from various directions, and learns the features of the objects to be detected. For example, the detection unit 12 can learn the features of buses and passenger cars, and detect an image portion having features similar to these as a vehicle. By learning the features for each vehicle type, the vehicle type of the vehicle can also be determined.

[0049] The detection unit 12 can set a detection frame surrounding the detected object in the image. In the present embodiment, the detection frame is rectangular, but the shape is not limited to a rectangular shape as long as the size of the detection frame substantially matches the size of the object. For example, the detection frame may be a shape that outlines the contour of the object, or may be an elliptical shape or the like having the same horizontal and vertical sizes as the object.

[0050] The determination unit 13 determines whether the rear vehicle detected from the rear image by the detection unit 12 has approached the vehicle. The determination unit 13 can use information such as the state of the vehicle detected by the sensor 4 or the position and speed of the object measured by the measuring device 5 for the determination.

[0051] The composition unit 14 composes the corrected rear image and the side image to generate one composite image. The composition unit 14 has a plurality of composition modes, and switches the composition mode according to the determination result of the determination unit 13.

[0052] The display control unit 15 causes the display device 20 to display the composite image generated by the composition unit 14.

[0053] FIG. 3 shows a composite process in which the image processing apparatus generates a composite image. This composite process is repeated each time the captured images generated by the respective cameras 3C, 3L, and 3R are input.

[0054] When the rear image and the side images are captured by the respective cameras 3C, 3L, and 3R, in the image processing apparatus 10, the correction unit 11 corrects each image (step S1). The detection unit 12 detects an object and its type included in the corrected rear image and side images, and sets a detection frame around the object (step S2). On the other hand, the synthesis unit 14 generates a composite image using the corrected rear image and side images (step S3).

[0055] Figures 4A and 4B illustrate the synthesis method of the present embodiment. As shown in FIG. 4A, the imaging range 61C of the camera 3C partially overlaps with the imaging ranges 61L and 61R of the cameras 3L and 3R. The synthesis unit 14 adjusts the trimming ranges 62C, 62L, and 62R, pan, tilt, and roll of the rear image JC, the side images JL and JR so that the rear image JC, the side images JL and JR are seamlessly synthesized on the synthesis plane L3 at the rear n (m) (for example, n = 15) of the vehicle 50.

[0056] The synthesis unit 14 generates a composite image J having a layer structure by connecting the left and right side images JL and JR on the vehicle 50 side and overlapping the rear image JC in the center thereof. The synthesis unit 14 narrows the display range 63C of the rear image JC in the composite image J and widens the display ranges 63L and 63R of the side images JL and JR by performing a transparency process on the end portions Cs on both the left and right sides of the rear image JC.

[0057] In FIG. 4A, an area not included in any of the trimming ranges 62C, 62L, and 62R becomes a blind spot on the composite image J. When the rear vehicle 51 detected in the rear image JC moves to this blind spot due to a lane change or the like, the rear vehicle 51 disappears from the composite image J. To prevent such disappearance, the synthesis unit 14 changes the range of the end portion Cs to be subjected to the transparency process in the rear image JC according to the position of the rear vehicle 51 in the rear image JC.

[0058] As shown in FIG. 4B, when the following vehicle 51 approaches the vehicle 50 side and moves to the right rear side position, the synthesizing unit 14 reduces the range of the right end Cs to be subjected to the transparency process in the rear image JC so that the detection frame of the following vehicle 51 detected from the rear image JC is included in the rear image JC. As a result, the trimming ranges 62C, 62L, and 62R change, and the display range 63C of the rear image JC in the composite image J spreads to the right, and the display range 63R of the right side image JR decreases accordingly.

[0059] In this way, by changing the display range 63C of the rear image JC and shifting the position of the blind spot of the composite image J, the disappearance of the following vehicle 51 can be reduced. However, when the display ranges 63L or 63R of the side images JL or JR decrease, the objects in the side images JL and JR may disappear from the composite image J.

[0060] FIG. 5 illustrates the disappearance of an object from the composite image J. As shown in FIG. 5, a following vehicle 51 traveling in the same lane as the vehicle 50 is detected from the rear image JC. A rear side vehicle 52 traveling in the adjacent lane is detected from the left side image JL.

[0061] As the following vehicle 51 approaches the vehicle 50, in order to prevent the disappearance of the following vehicle 51, the display range 63C of the rear image JC expands during synthesis. That is, the area of the rear image JC overlapping the side images JL and JR increases. As a result, when the display range 63C of the rear image JC overlaps the rear side vehicle 52 in the side image JL, the rear side vehicle 52 disappears from the composite image J.

[0062] The determination unit 13 determines the presence or absence of an object disappearing from the composite image J by the rear image JC as described above. First, the determination unit 13 determines the presence or absence of the following vehicle 51 in the rear image JC based on the detection result of the object by the detection unit 12 (step S4). When it is determined that there is the following vehicle 51 (step S4: YES), the process proceeds to the first warning process (step S5). When it is determined that there is no following vehicle 51 (step S4: NO), the process proceeds to the second warning process (step S6).

[0063] As described above, as the following vehicle 51 approaches the vehicle 50, the display range 63C of the rear image JC expands, and the rear-side vehicle 52 is likely to disappear. Therefore, the determination unit 13 may shift to the first warning process when the following vehicle 51 is detected in the rear image JC and the following vehicle 51 is approaching the vehicle 50.

[0064] FIGS. 6A and 6B illustrate a method for determining the approach of the following vehicle 51 by analyzing the rear image JC. As shown in FIG. 6A, the determination unit 13 arranges two reference lines L11 parallel to the vertical direction y and two reference lines L21 parallel to the horizontal direction x on the rear image JC. The two reference lines L11 are respectively arranged at left and right positions at a certain distance from the center line of the rear image JC in the horizontal direction x. The two reference lines L21 are arranged at positions respectively at a predetermined distance from the lower end of the rear image JC.

[0065] As the following vehicle 51 approaches the vehicle 50, the following vehicle 51 is located on the lower side of the rear image JC, and the size in the image also becomes larger. Therefore, when the positions of the ends of the following vehicle 51 detected in the rear image JC, for example, the positions of the four sides of the detection frame set for the following vehicle 51, are all outside beyond all the reference lines L11 and L21, the determination unit 13 can determine that the following vehicle 51 has approached the vehicle 50. In this way, the approach of the following vehicle 51 can be easily determined by comparison with the reference lines L11 and L21.

[0066] The positions of the reference lines L11 and L21 can be measured and determined in advance. For example, in the rear image JC when the following vehicle 51 approaches to a position 15 m behind the vehicle 50, the positions of both ends in the horizontal direction x and both ends in the vertical direction y of the following vehicle 51 are measured, and these can be determined as the positions of the reference lines L11 and L21.

[0067] As in the present embodiment, it is preferable to perform determination in both the horizontal direction x and the vertical direction y using the reference lines L11 and L21 because the accuracy of the determination is high. When performing a simple determination, it is possible to perform the determination using only one of the horizontal direction x and the vertical direction y using either of the reference lines L11 and L21.

[0068] The determination unit 13 can move the positions of the reference lines L11 and L21 according to the vehicle type of the following vehicle 51. For example, large vehicles such as trucks and buses have a larger vehicle width than ordinary vehicles and light vehicles. Therefore, as shown in FIG. 6B, by determining the approach using the reference line L12 that is closer to the reference line L11 or the reference line L22 that is located above the reference line L21, it is possible to advance the timing of determining that an approach has occurred. By a simple operation of changing the positions of the reference lines L11 and L21, it is possible to easily adjust the timing of determining the approach of the following vehicle 51.

[0069] FIG. 7 shows the first warning process. In the first warning process, the determination unit 13 determines the presence or absence of an object that may disappear by synthesis (hereinafter referred to as a blind spot object) among the objects detected from the side images JL or JR (step S51). Specifically, as shown in FIG. 5, the determination unit 13 determines the presence or absence of a blind spot object such as the rear side vehicle 52 that is detected from the side image JL or JR but not detected from the rear image JC.

[0070] The determination unit 13 acquires information on the turn signal state from the sensor 4 and can change the range for determining the presence or absence of a blind spot object in the side image JL or JR on the rear side where the turn signal is ON. For example, when the left turn signal is ON, since the vehicle 50 is turning left or changing the driving lane to the left adjacent lane, the importance of checking for the rear side vehicle 52, people, motorcycles, etc. in the left side image JL increases. Therefore, by changing the determination range of the side image JL to a wider area, it is possible to widely inform the driver of the presence of an object located on the left rear side.

[0071] FIG. 8 shows an example of a range for determining an object in the side image JL or JR. When the right turn signal is in the OFF state, the determination unit 13 sets the right rear determination range 64R as the reference area. The reference area is set to a range from the lower end of the image to a position corresponding to, for example, 20 m behind the vehicle 50. On the other hand, when the left turn signal is in the ON state, the determination unit 13 changes the left rear determination range 64L to a wider area than the reference area. For example, the determination range 64L is a range from the lower end of the image to a position corresponding to 40 m behind the vehicle 50. As a result, the depth of the determination range 64L expands, and an object located far from the vehicle 50 can also be listed as a candidate for warning.

[0072] When it is determined that there is a blind spot object (step S51: YES), the determination unit 13 determines the risk level of the blind spot object according to the approach level of the blind spot object to the vehicle 50. The approach level can be determined by the distance from the vehicle 50 of the rear vehicle 51, the moving speed, or the time until reaching the vehicle 50. The distance, moving speed, or arrival time can be measured by the measuring device 5, or can also be analyzed by the determination unit 13 from the position, size, etc. of the rear vehicle 51 in the rear image JC.

[0073] The determination unit 13 compares the above distance, moving speed, or arrival time with a threshold value. When it is determined that the distance is short, the moving speed is fast, or the arrival time is short compared to the threshold value, the determination unit 13 determines that the risk level of the blind spot object is high, and otherwise determines that the risk level is low.

[0074] When there is a blind spot object in the side image JL or JR (step S51: YES) and it is determined that the risk level thereof is high (step S52: YES), the composition unit 14 generates warning information and overlays the warning information on the composite image J. The display control unit 15 causes the display device 20 to display the composite image J including the warning information (step S53).

[0075] FIGS. 9A to 9C show display examples of warning information. As shown in FIG. 9A, a rear-side vehicle 52 is detected from the left side image JL. Since the rear-side vehicle 52 has the rear image JC superimposed thereon and disappears from the composite image J, it is a blind spot object. When it is determined that the moving speed of the rear-side vehicle 52 is high and the risk level is high, warning information 55 is displayed on the composite image J. The warning information 55 is arranged on the side image JL where the rear-side vehicle 52 is detected.

[0076] The warning information 55 includes an icon image 551 representing the vehicle 50 and an icon image 552 representing the rear-side vehicle 52. The combining unit 14 arranges the icon image 551 on the vehicle 50 side within the frame of the warning information 55. Further, the combining unit 14 arranges the icon image 552 at a position corresponding to the approach level determined by the determination unit 13. For example, the closer the distance between the rear-side vehicle 52 and the vehicle 50, the closer the combining unit 14 arranges the icon image 552 to the icon image 551. In FIG. 9A, since the rear-side vehicle 52 is, for example, more than 30 m away from the vehicle 50, the icon image 552 is arranged at the position farthest from the icon image 551.

[0077] As shown in FIG. 9B, when the rear-side vehicle 52 approaches and the distance from the vehicle 50 cuts, for example, 20 m, the combining unit 14 arranges the icon image 552 at a position closer to the icon image 551. As shown in FIG. 9C, when the distance between the rear-side vehicle 52 and the vehicle 50 cuts, for example, 15 m, the combining unit 14 arranges the icon image 552 next to the icon image 551. The combining unit 14 not only changes the arrangement position of the icon image 552 but also changes the color of the icon image 552 in the order of blue, yellow, and red as the distance decreases, and blinks the icon image 552 when it is red.

[0078] In this way, the synthesizing unit 14 can vary the display form of the warning information 55 according to the approach level of the blind spot object to the vehicle 50. As a result, the driver can easily grasp the approach situation of the rear-side vehicle 52. Note that the display form to be changed may be not only the color or blinking state of the icon image 552, but also density, brightness, size, or the like. Further, the synthesizing unit 14 can also vary the display form according to the moving speed of the object or the arrival time of the object at the vehicle 50 instead of the distance.

[0079] The warning information 55 is not limited to the above. FIGS. 10A and 10B show other examples of the warning information 55. The synthesizing unit 14 can change the icon image 552 representing the blind spot object according to the type of the blind spot object. Specifically, when the blind spot object is a four-wheeled vehicle, the synthesizing unit 14 displays an icon image 552 imitating a four-wheeled vehicle as shown in FIG. 9A, and when the detected object is a two-wheeled vehicle, the synthesizing unit 14 displays an icon image 552 imitating a two-wheeled vehicle as shown in FIG. 10A. When the object is a person, an icon image imitating a person may be used. As a result, the driver can easily grasp the type of the blind spot object hidden behind the rear vehicle 51 and not visible from the composite image J.

[0080] As shown in FIG. 10A, the warning information 55 can include an icon image 553 of an arrow indicating the approach level. The synthesizing unit 14 lights up the three icon images 553 in order from left to right and increases the lighting speed as the blind spot object approaches.

[0081] As shown in FIG. 10B, the synthesizing unit 14 may display a side mirror icon image 554 instead of the icon images 551 and 552. The synthesizing unit 14 can prompt the driver to check the side mirror by increasing the number of the displayed icon images 554 as the blind spot object approaches.

[0082] The display forms of the above-described respective icon images 551 to 554 can be combined. For example, the combining unit 14 can increase the number of arrow icon images 553 as the blind spot object approaches, as in the icon image 554 of the side mirror. Further, the combining unit 14 may fix the number of icon images 554 of the side mirror and increase the lighting display speed, as in the arrow icon image 553, or may arrange the icon image 552 next to the icon image 554.

[0083] The warning information 55 is fixedly displayed in a predetermined area on the side image JL or JR where the blind spot object is detected. In the examples shown in FIGS. 9A to 9C, the warning information 55 has a rectangular shape extending in the horizontal direction x and is arranged at the lower end in the vertical direction y, but may be arranged at the upper end. Whether at the upper end or the lower end, it is difficult for the warning information 55 to overlap on the rear side vehicle 52, and even if it overlaps, it does not cover all of it, so the disappearance of the rear side vehicle 52 due to the warning information 55 can be reduced. For the same reason, the warning information 55 may have a rectangular shape extending in the vertical direction y and be arranged at the left end of the side image JL or the right end of the side image JR.

[0084] On the other hand, when there is no blind spot object corresponding to the side image JL or JR (step S51: NO), or when it is determined that the risk level is low even if there is a blind spot object (step S52: NO), the combining unit 14 does not generate the warning information as described above. The display control unit 15 causes the display device 20 to display the composite image J as it is (step S54).

[0085] Next, the second warning process will be described. FIG. 11 shows the second warning process. In the second warning process, the determination unit 13 determines the presence or absence of an object detected from the side image JL or JR (step S61). When there is an object detected from the side image JL or JR, the determination unit 13 determines whether the object has approached the vehicle 50 (step S62). The determination unit 13 can determine the approach of the object from the distance between the object and the vehicle 50, the moving speed of the object, or the arrival time at the vehicle 50, in the same manner as when determining the risk level in the first warning process.

[0086] If there is an object in the side image JL or JR (step S61: YES), and it is determined that the object is approaching (step S62: YES), the synthesizing unit 14 generates warning information and superimposes it on the composite image J. The display control unit 15 causes the display device 20 to display the composite image J including the warning information (step S63).

[0087] FIG. 12 shows an example of the display of warning information in the second warning process. As shown in FIG. 12, a rear-side vehicle 52 with a high moving speed is detected from the left side image JL. On the other hand, since nothing is detected from the rear image JC, the display range 63C of the rear image JC is not expanded. Since the rear image JC is not superimposed on the rear-side vehicle 52, the rear-side vehicle 52 does not disappear from the composite image J, but warning information 56 for alerting that the rear-side vehicle 52 is approaching is displayed on the composite image J.

[0088] The warning information 56 includes only an icon image 561 representing the rear-side vehicle 52. The icon image 561 may be an icon image representing the type of the object, similar to the icon image 552. The rear-side vehicle 52 is not a blind spot object that disappears from the composite image J, and since the driver can visually recognize the rear-side vehicle 52 traveling in the left lane from the composite image J, the icon image 561 alone can sufficiently alert the driver.

[0089] On the other hand, if there is no object in the side image JL or JR (step S61: NO), or if it is determined that the object is not approaching even though there is an object (step S62: NO), the synthesizing unit 14 does not generate the above-described warning information 56. Therefore, the display control unit 15 causes the display device 20 to display the composite image J as it is (step S64).

[0090] As described above, according to the present embodiment, when a rear vehicle 51 is detected from the rear image JC, the determination unit 13 determines the presence or absence of a blind spot object that disappears from the composite image J by the rear image JC and has a high risk among the objects detected from the side images JL or JR. When it is determined that there is a blind spot object, the composite unit 14 generates warning information 55 for the blind spot object and superimposes it on the composite image J.

[0091] With the warning information 55, the driver can quickly grasp the presence of the blind spot object in the side image JL or JR that disappears from the composite image J due to the rear vehicle 51 in the rear image JC. The warning information 55 warns not all blind spot objects but only those with a high risk. Since the warning target is narrowed down, the driver can easily grasp the rear vehicle 51 that particularly requires attention and the blind spot object at the rear side.

[0092] The above embodiment is a preferred example of the present invention and is not limited thereto, and can be changed as appropriate. For example, the present invention can also be applied to an image processing apparatus that performs a composition other than the above-described composition method as long as it is a composition method in which an object on the side image JL or JR may disappear by superimposing the side image JL or JR and the rear image JC.

Explanation of Signs

[0093] 100 Driving support system 25 Image display system 10 Image processing apparatus 12 Detection unit 13 Determination unit 14 Composite unit 15 Display control unit 20 Display device 3C, 3L, 3R Cameras

Claims

1. An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, comprising: a composite unit that composites a rear image captured behind the vehicle and two side images respectively captured from the left rear side and the right rear side to generate the composite image; a detection unit that detects an object including a vehicle behind from the rear image and the side images; a determination unit that, when the rear vehicle is detected from the rear image, determines the presence or absence of a blind spot object that disappears from the composite image due to the rear image and has a high risk among the objects detected from the side images; wherein when it is determined that there is the blind spot object, the composite unit overlays warning information of the blind spot object on the composite image; the determination unit acquires information on the state of a turn signal of the vehicle and widens a range for determining the presence or absence of the blind spot object in the side image on the side where the turn signal is in the ON state; Image processing apparatus.

2. The determination unit determines an approach level of the blind spot object to the vehicle and determines the risk level based on the approach level. The image processing apparatus according to claim 1.

3. The composite unit changes a display form of the warning information according to the approach level. The image processing apparatus according to claim 2.

4. The warning information includes an image indicating a type of the blind spot object. The image processing apparatus according to any one of claims 1 to 3.

5. The composite unit arranges the warning information on the side image on which it is determined that there is the blind spot object. The image processing apparatus according to any one of claims 1 to 4.

6. The composite unit arranges the warning information in a predetermined area of the side image. The image processing apparatus according to claim 5.

7. The determination unit determines whether or not a vehicle is detected from the side image when the rear vehicle is not detected from the rear image, and when it is determined that the rear vehicle is not detected from the rear image and a vehicle is detected from the side image, the composite unit overlays warning information of the vehicle on the side image. The image processing apparatus according to any one of claims 1 to 6.

8. The determination unit determines the presence or absence of the blind spot object when the rear vehicle is detected from the rear image and the rear vehicle approaches the vehicle. When the position of the end of the rear vehicle detected from the rear image exceeds a reference line arranged in the horizontal or vertical direction of the rear image, it is determined that the rear vehicle has approached the vehicle. The image processing apparatus according to any one of claims 1 to 7.

9. The detection unit detects the type of the object. When the position of the end of the rear vehicle detected from the rear image is outside all of two vertical reference lines parallel to each other and arranged at positions separated by a certain distance from the center line of the rear image in the horizontal direction and two horizontal reference lines parallel to each other and arranged at positions separated by a predetermined distance from the lower end of the rear image on the rear image, the determination unit determines that the rear vehicle has approached the vehicle. When the type of the rear vehicle is large, the determination unit narrows the interval between the two vertical reference lines parallel to each other or moves the two horizontal reference lines parallel to each other upward as compared with the case where the type is not large. The image processing apparatus according to claim 8.

10. A system that provides a captured image behind a vehicle, an image processing apparatus that generates one composite image from a plurality of captured images behind the vehicle, and a display device that displays the composite image. The image processing apparatus a composite unit that composites a rear image captured behind the vehicle and two side images captured on the left rear side and the right rear side respectively to generate the composite image, a detection unit that detects an object including a rear vehicle from the rear image and the side images, a determination unit that, when the rear vehicle is detected from the rear image, determines the presence or absence of a blind spot object that disappears from the composite image due to the rear image and has a high risk among the objects detected from the side images. When it is determined that there is the blind spot object, the composite unit overlays warning information of the blind spot object on the composite image. The determination unit acquires information on the state of the vehicle's blinker and widens the range for determining the presence or absence of the blind spot object in the side image on the side where the blinker is in the ON state. System.

11. In a computer, a step of generating one composite image by compositing a rear image captured behind the vehicle and two side images captured on the left rear side and the right rear side respectively, a step of detecting an object including a rear vehicle from the rear image and the side images, When the rear vehicle is detected from the rear image, a determination step of determining the presence or absence of a blind spot object that disappears from the composite image due to the rear image among the objects detected from the side image and has a high risk; When it is determined that there is the blind spot object, a step of superimposing warning information of the blind spot object on the composite image; to be executed, In the determination step, information on the state of the vehicle's blinker is acquired, and the range for determining the presence or absence of the blind spot object in the side image on the side where the blinker is in the ON state is widened Program.

Images