Image processing apparatus, system, and program

Abstract

To naturally switch display of a back object between taken images.SOLUTION: An image processor 10 includes: a synthesis unit 14 for generating a synthesis image by partially overlapping a rear image of the back of a vehicle with a part of the vehicle side of two side images of the left back side and the right back side, respectively; a detection unit 12 for detecting an object from the rear image and the side images; and a determination unit 13 for determining whether there is a target object detected from a target region of blending processing of the rear image and the side images on the basis of the result of detecting an object. The synthesis unit 14 performs blending processing including performing transmission-processing on the region of the target object in the rear image and the region of the target object in the side images, with different transmissivity, if the presence of the target object is determined.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a system, and a program.

Background Art

[0002] Conventionally, a driving support system has been used that captures images of the rear and the left and right rear sides of a vehicle and displays them on a display inside the vehicle. Since the imaging ranges of the respective cameras partially overlap, when synthesizing the rear imaging image and the rear side imaging image, image processing may be performed to partially overlap the respective images.

[0003] For example, in Patent Document 1, each imaging image of the rear and the rear side is arranged so as to represent the rear of the host vehicle from one virtual viewpoint, and the overlapping image portion is trimmed. At this time, when a rear vehicle is located on the boundary of each imaging image, the displayed rear vehicle may be distorted. In order to avoid such distortion, in Patent Document 1, by moving the boundary of each imaging image, the rear vehicle is positioned on either the rear imaging image side or the rear side imaging image side.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, due to the difference in the positions of the rear and rear side cameras, even for the same vehicle, the sizes of the rear imaging image and the rear side imaging image are different. Since the size of the rear vehicle suddenly changes when the boundary of each imaging image moves, the driver may not be able to understand the perspective of the rear vehicle. Also, if the boundary moves frequently, the rear vehicle will repeatedly grow and shrink, causing the screen to flicker and the driver may feel discomfort.

[0006] The present invention aims to naturally switch the display of a rear object between captured images.

Means for Solving the Problems

[0007] [1] An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, a composite unit that generates the composite image by overlapping a rear image captured behind the vehicle on a part of the vehicle side of two side images respectively captured from the left rear side and the right rear side, a detection unit that detects an object from the rear image and the side images, a determination unit that determines the presence or absence of a target object detected from a target region for blending processing of the rear image or the side image based on the detection result of the object, and includes: when it is determined that there is the target object, the composite unit executes a blending process of performing a transparency process on each image region of the target object in the rear image and the target object in the side image with different transmittance rates Image processing apparatus.

[0008] According to the above [1], the image regions of the target object in the rear image and the image regions of the target object in the side image are subjected to a transparency process with different transmittance rates. When the processed image regions are overlapped, the contour of the target object becomes blurred and it becomes difficult to recognize the difference in size between the target objects in the rear image and the side image. Since the target objects in both the rear image and the side image are displayed, even when the display of the target object is switched to only one of them, the size does not change suddenly, so the flickering is reduced and the discomfort is less. Therefore, the display of the rear object between the captured images can be naturally switched.

[0009] [2] The determination unit determines the degree of danger of the object detected from the target region, and determines the object detected from the target region and having a high degree of danger as the target object The image processing apparatus according to the above [1].

[0010] According to [2] above, it is possible to target not all target objects but rear objects with a high degree of danger for the blending process. The processing time until the composite image is displayed can be reduced.

[0011] [3] The determination unit determines whether the traveling direction of the object detected from the target area is the same as the traveling direction of the vehicle, and determines the object detected from the target area and having the same traveling direction as the target object. The image processing apparatus according to [1] or [2] above.

[0012] According to [3] above, it is possible to target rear objects that have the same traveling direction and may straddle between the rear image and the side image for the blending process.

[0013] [4] The determination unit determines whether the same target object exists in both the rear image and the side image, or whether the target object exists only in one of the rear image and the side image. When it is determined that the same target object exists in both the rear image and the side image, the composite unit, in the blending process, determines the image area of the target object in the rear image as the blending area, subjects the blending area to a transparency process, and reduces the transmittance from the vehicle side toward the side opposite to the vehicle, determines the image area of the target object in the side image as the blending area, subjects the blending area to a transparency process, and increases the transmittance from the vehicle side toward the side opposite to the vehicle. The image processing apparatus according to any one of [1] to [3] above.

[0014] According to [4] above, when there are target objects in both the rear image and the side image, the target object in the side image appears stronger closer to the vehicle, and the target object in the rear image appears stronger closer to the side opposite to the vehicle. Therefore, both target objects can be displayed in a transparent state.

[0015] [5] When it is determined that the target object exists only in one of the rear image and the side image, in the blending process, a region obtained by expanding the image region of the target object by n times on the side opposite to the vehicle is determined as a blending region, the blending region in the rear image is subjected to a transparency process, and its transmittance is increased from the vehicle side toward the side opposite to the vehicle, the blending region in the side image is subjected to a transparency process, and its transmittance is decreased from the vehicle side toward the side opposite to the vehicle The image processing apparatus according to the above [4].

[0016] According to the above [5], when the target object exists only in one of the rear image and the side image, the target object in the rear image appears stronger as it is closer to the vehicle, and the target object in the side image appears stronger as it is closer to the side opposite to the vehicle. Also, by expanding the blending region, it is possible to make the target object that was photographed but not detected also be a target for the transparency process. Therefore, not only the detected target object on one side but also the target object on the other side that was not detected but would have been photographed can be displayed in a transparent state.

[0017] [6] The synthesis unit determines the position of the boundary line between the rear image and the side image in the synthesized image to be the horizontal center of the blending region, sets the transmittance of the rear image and the side image at the boundary line to 50% respectively The image processing apparatus according to the above [4] or [5].

[0018] According to the above [6], the rear image and the side image can be overlapped by half at the center of the blending region, and the appearance of the target object in the blending region is not biased and becomes natural.

[0019] [7] A system that provides a photographed image behind a vehicle, an image processing apparatus that generates one synthesized image from a plurality of photographed images behind the vehicle, and a display device that displays the synthesized image. The image processing apparatus includes a synthesizing unit that generates the composite image by overlapping a rear image captured of the rear of the vehicle with a part on the vehicle side of two side images captured of the left rear side and the right rear side respectively; a detecting unit that detects an object from the rear image and the side images; and a determining unit that determines the presence or absence of a target object detected from a target area of blend processing of the rear image or the side image based on the detection result of the object. When it is determined that there is the target object, the synthesizing unit executes blend processing of performing transparency processing on each image area of the target object in the rear image and the target object in the side image at different transmittance rates. System.

[0020] According to [7] above, the image area of the target object in the rear image and the image area of the target object in the side image are subjected to transparency processing at different transmittance rates. When the processed image areas are overlapped, the contour of the target object becomes blurred and it becomes difficult to recognize the difference in size between the target objects in the rear image and the side image. Since the target objects in both the rear image and the side image are displayed, even when the display of the target object is switched to only one of them, the size does not change suddenly, so the flicker is reduced and the discomfort is less. Therefore, the display of the rear object between the captured images can be switched naturally.

[0021] [8] A step of causing a computer to generate one composite image by overlapping a rear image captured of the rear of the vehicle with a part on the vehicle side of two side images captured of the left rear side and the right rear side respectively; detect an object from the rear image and the side images; determine the presence or absence of a target object detected from a target area of blend processing of the rear image or the side image based on the detection result of the object; When it is determined that the target object exists, a step of executing a blending process of performing a transparency process on each image area of the target object in the rear image and the target object in the side image at different transmittances; A program for causing the execution.

[0022] According to [8] above, the image area of the target object in the rear image and the image area of the target object in the side image are subjected to a transparency process at different transmittances. When the respective image areas subjected to the transparency process are superimposed, the contour of the target object becomes blurred and it becomes difficult to recognize the difference in size between the target objects in the rear image and the side image. Since the target objects in both the rear image and the side image are displayed, even when the display of the target object is switched to only one of them, the size does not change suddenly, so the flickering is reduced and the discomfort is less. Therefore, the display of the rear object between the captured images can be switched naturally.

Effect of the Invention

[0023] According to the present invention, the display of the rear object between the captured images can be switched naturally.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8

Figure 9A

Figure 9B

Figure 10A

Figure 10B

Figure 10C

Embodiments for Carrying Out the Invention

[0025] Hereinafter, embodiments of the image processing apparatus, system, and program of the present invention will be described with reference to the drawings.

[0026] FIG. 1 shows the configuration of an operation support system 100 according to an embodiment. The operation support system 100 is mounted on a vehicle and supports the driving of the driver by displaying captured images of the rear and rear sides of the vehicle.

[0027] As shown in FIG. 1, the operation support system 100 includes an image display system 25, three cameras 3C, 3L, and 3R, various sensors 4, a measurement device 5, etc. The image display system 25 includes an image processing apparatus 10 and a display device 20.

[0028] Cameras 3C, 3L, and 3R continuously capture images of the rear, left rear side, and right rear side of the vehicle respectively, and sequentially generate time-series captured images. Hereinafter, the captured image of the rear is referred to as a rear image, and the captured image of the rear side is referred to as a side image. The camera 3C for capturing the rear is disposed, for example, near the license plate. The cameras 3L and 3R for capturing the left and right rear sides are respectively provided, for example, at the positions of the left and right side mirrors.

[0029] Sensor 4 detects the state of the vehicle. Examples of the sensor 4 include various sensors mounted on the vehicle such as a vehicle speed sensor, a steering angle sensor, and a turn signal sensor. The driving assistance system 100 can acquire detection signals from these sensors 4 as information indicating the state of the vehicle.

[0030] The measuring device 5 measures the position, direction, distance to the object, moving speed of the object, etc. of objects around the vehicle, such as vehicles and people. Examples of the measuring device 5 include an ultrasonic sonar or a millimeter wave radar. The driving assistance system 100 can acquire measurement information of objects around the vehicle from these measuring devices 5.

[0031] (Display device) The display device 20 displays the composite image generated by the image processing device 10. The display device 20 may be an in-vehicle display having a general aspect ratio, or may be a room mirror type display having a long horizontal-to-vertical ratio.

[0032] FIG. 2 shows an example of the arrangement of the display device 20 in the vehicle. The display device 20 is installed on the dashboard 21 inside the vehicle or near the upper center of the front window 22 instead of the inner mirror. In FIG. 2, the display device 20 is composed of a horizontally long single liquid crystal display or the like, and one composite image J is displayed thereon.

[0033] The images (videos) captured by cameras 3C, 3L, and 3R are synthesized, and a single synthesized image J is displayed on the display device 20. In the synthesized image J, the captured image JL of camera 3L is displayed on the left side, the captured image JC of camera 3C is displayed in the center, and the captured image JR of camera 3R is displayed on the right side.

[0034] Note that the display device 20 may be composed of a plurality of displays connected in a row. In this case, a single synthesized image J is divided, and each divided image is displayed in the display area of each display.

[0035] (Image Processing Device) The image processing device 10 synthesizes the rear image captured by camera 3C and the side images captured by cameras 3L and 3R into one image.

[0036] The image processing device 10 includes a correction unit 11, a detection unit 12, a determination unit 13, a synthesis unit 14, and a display control unit 15. The functions of each part of the image processing device 10 may be realized by software processing in which a program is read and executed by a computer such as an ECU (Electronic Control Unit) or a microcomputer mounted on a vehicle. Also, each part of the image processing device 10 may be constituted by hardware such as an FPGA (field-programmable gate array).

[0037] The correction unit 11 performs correction processing such as distortion correction and color correction on the rear image and the side images.

[0038] The detection unit 12 detects objects included in each image after the correction processing. Examples of the objects to be detected include vehicles, people, animals, guardrails, or buildings. The detection method is not particularly limited, but for example, detection using deep learning can be performed.

[0039] In deep learning, the detection unit 12 uses, as training data, a plurality of captured images obtained by photographing detection targets such as vehicles and people with different vehicle types and distances from various directions, and learns the features of the objects to be detected. For example, the detection unit 12 can learn the features of buses and passenger cars, and detect an image portion having features similar to these as a vehicle. By learning the features for each vehicle type, the vehicle type of the vehicle can also be discriminated. Thus, the detection unit 12 can also detect the type of the object.

[0040] The detection unit 12 can set a detection frame surrounding the detected object in the image. In the present embodiment, the detection frame is rectangular, but the shape is not limited to a rectangular shape as long as the size of the detection frame substantially matches the size of the object. For example, the detection frame may be a shape that outlines the contour of the object, or may be an elliptical shape or the like having the same horizontal and vertical sizes as the object.

[0041] Based on the detection result of the detection unit 12, the determination unit 13 determines the presence or absence of the target object detected from the target area of the blending process of the rear image or the side image. The determination unit 13 can determine the degree of danger and the traveling direction of the target object. The determination unit 13 can use information such as the state of the vehicle detected by the sensor 4 or the position and speed of the object measured by the measuring device 5 for the determination.

[0042] The combining unit 14 combines the corrected rear image and side image to generate one combined image.

[0043] The display control unit 15 causes the display device 20 to display the combined image generated by the combining unit 14.

[0044] FIG. 3 shows a combining process in which the image processing apparatus generates a combined image. This combining process is repeated each time the captured images generated by the respective cameras 3C, 3L, and 3R are input.

[0045] When the rear image and the side images are captured by the respective cameras 3C, 3L, and 3R, in the image processing apparatus 10, the correction unit 11 corrects each image (step S1). The detection unit 12 detects an object and its type included in the corrected rear image and side images, and sets a detection frame around the object (step S2).

[0046] Further, the composition unit 14 generates a composite image J by overlapping it on the central portions of the side images JL and JR obtained by concatenating the corrected rear image JC (step S3).

[0047] FIGS. 4A and 4B illustrate a normal composition method. As shown in FIG. 4A, the shooting range 61C of the camera 3C partially overlaps with the shooting ranges 61L and 61R of the cameras 3L and 3R. The composition unit 14 adjusts the trimming ranges 62C, 62L, and 62R, pan, tilt, and roll of the rear image JC, the side images JL and JR so that the rear image JC, the side images JL and JR are seamlessly composed on the composition plane L3 at the rear of the vehicle 50 by n (m) (for example, n = 15).

[0048] The composition unit 14 generates a composite image J having a layer structure by connecting the left and right side images JL and JR on the vehicle 50 side and overlapping the rear image JC in the center thereof. The composition unit 14 performs a transmission process on the left and right end portions Cs of the rear image JC at a transmittance of 100%, thereby narrowing the display range 63C of the rear image JC in the composite image J and widening the display ranges 63L and 63R of the side images JL and JR.

[0049] The composition unit 14 can determine the position of the boundary line L1 that determines the display range 63C of the rear image JC in the composite image J according to the position of the rear vehicle 51 in the rear image JC. For example, as shown in FIG. 4B, when the rear vehicle 51 moves to a position on the right rear side while approaching the vehicle 50 side, the composition unit 14 can move the position of the boundary line L1 to the vehicle 50 side so that the detection frame of the rear vehicle 51 detected from the rear image JC is included in the rear image JC. As a result, the range of the right end portion Cs to be subjected to the transmission process in the rear image JC can be reduced.

[0050] An area that is not within any of the trimming ranges 62C, 62L, and 62R becomes a blind spot on the composite image J. However, as described above, the display range 63C of the rear image JC can be changed by moving the boundary line L1. Thereby, the position of the blind spot of the composite image J can be shifted, and the disappearance of the rear vehicle 51 from the composite image J can be reduced.

[0051] In the above composite image J, when an object straddles the boundary line L1 between the rear image JC and the side image JL or JR, the display of the rear object may become unnatural. FIG. 5 shows an example of the composite image J when an object straddles the boundary line L1. As shown in FIG. 5, a rear-side vehicle 52 traveling in the adjacent lane of the vehicle 50 is included in both the rear image JC and the side image JR.

[0052] When the rear-side vehicle 52 straddles the boundary line L1 between the rear image JC and the side image JR, the left part of the rear-side vehicle 52 on the rear image JC and the right part of the rear-side vehicle 52 on the side image JR are connected on the boundary line L1. However, due to the difference in the camera positions, the sizes of the rear-side vehicle 52 in the rear image JC and the side image JR are different. Therefore, an unnatural composite image J is formed in which the sizes of the rear-side vehicle 52 are partially different.

[0053] Such a partial size difference temporarily occurs when the rear-side vehicle 52 moves between the rear image JC and the side image JL or JR beyond the boundary line L1. Each time the rear-side vehicle 52 crosses the boundary line L1, a temporary change in size occurs, so that the rear-side vehicle 52 appears to flicker and it becomes difficult to grasp the sense of perspective.

[0054] In order to reduce the above unnatural display, the combining unit 14 can execute a blending process during combination. The blending process is a process of performing a transparency process on each image area of the object in the rear image JC and the object in the side image JL or JR at different transmittance rates and then overlapping them. Thereby, the display of the object between each image can be switched naturally. The object to be blended (hereinafter referred to as the target object) can be determined by the determination unit 13.

[0055] The determination unit 13 determines the presence or absence of an object located in the target area for the blending process of the composite image J (step S4). FIG. 6A shows an example of the target area. As shown in FIG. 6A, the determination unit 13 determines whether an object is located in the target area 81 on the composite image J based on the object detection result by the detection unit 12.

[0056] The target area 81 is a triangular area that extends left and right starting from the center of the horizon in the composite image J. The position and size of the target area 81 on the composite image J may be determined in advance based on the width of a general lane, or may be determined each time by analyzing the composite image J. For example, the detection unit 12 detects the center of the horizon and the white line that separates the lane from the composite image J, and the area surrounded by two white lines extending from the center to the left or right side of the vehicle 50 can be set as the target area 81.

[0057] For example, when the center P1 of the lower side of the object detection frame 72 set by the detection unit 12 is located within the target area 81, the determination unit 13 determines that the object is located in the target area 81. In FIG. 6A, it is determined that the centers P1 to P3 of the three objects are located in the target area 81.

[0058] When an object is located in the target area 81 (step S4: YES), the determination unit 13 calculates the direction vector of the object. The calculated direction vector represents the traveling direction of the object. The determination unit 13 determines whether the traveling direction of the object is the same as that of the vehicle 50 based on the calculated direction vector (step S5).

[0059] The determination unit 13 obtains the position of the object, that is, the position coordinates (x, y) of the center P1 of the detection frame 72 in the horizontal direction x and the vertical direction y in the composite images J of the previous frame and the current frame. When the position coordinates of the center P1 of the previous frame are (x1, y1), the position coordinates of the current center P1 are (x2, y2), and the angle formed with respect to the horizontal direction x is θ, the direction vector β1 can be calculated as shown in the following equation [1]. β1=(x2 - x1) / cosθ [1]

[0060] FIG. 6B shows the direction vectors of each object in FIG. 6A. The traveling direction of the vehicle 50 is downward in the vertical direction y of the composite image J. The determination unit 13 determines that an object whose calculated direction vector is within a certain angle, for example, in the range of 0 to 70 degrees from the vertical direction y, is the same object as the traveling direction of the vehicle 50. In FIG. 6B, among the direction vectors β1 to β3 of each center P1 to P3, it is determined that the objects having the direction vectors β1 and β3 are the same objects as the vehicle 50 in the traveling direction.

[0061] When no object is detected from the target area 81 (step S4: NO), or even if an object is detected but the traveling direction of the object is different from that of the vehicle 50 (step S5: NO), the composite unit 14 performs normal transmission processing (step S6). That is, the end portion Cs on the side opposite to the vehicle 50 from the boundary line L1 of the rear image JC is subjected to transmission processing with a transmittance of 100%.

[0062] On the other hand, when an object is detected from the target area 81 (step S4: YES) and the traveling direction thereof is the same as that of the vehicle 50 (step S5: YES), the object is a candidate for the object to be subjected to blending processing. The determination unit 13 determines whether there is one or a plurality of candidates for the object to be subjected to blending processing (step S7). When there is one candidate (step S7: YES), the determination unit 13 determines the candidate as the object to be subjected to blending processing.

[0063] When there are not one but a plurality of candidate target objects (step S7: NO), the determination unit 13 determines the risk level of each candidate. The determination unit 13 determines the candidate with the highest risk level as the target object (step S8). The determination unit 13 can determine the risk level of the object according to the approach level or type of the object to the vehicle 50.

[0064] The approach level can be determined based on the distance from the object to the vehicle 50, the arrival time, or the moving speed. The distance, arrival time, or moving speed can be measured by the measuring device 5, or can also be analyzed by the determination unit 13 from the position, size, etc. of each object in the composite image J.

[0065] The determination unit 13 determines the degree of danger by comparing the above distance, arrival time, or moving speed with a plurality of thresholds. For example, the determination unit 13 determines that the higher the moving speed, the higher the degree of danger. Similarly, the shorter the distance or arrival time, the higher the degree of danger.

[0066] Also, the determination unit 13 can determine that the degree of danger is high in the order of, for example, person > two-wheeled vehicle > large vehicle > ordinary vehicle > light vehicle, from the type of the target object detected by the detection unit 12. This order is predetermined as the order in which the driver is required to be alerted, and usually a person is determined to have the highest degree of danger.

[0067] Next, the determination unit 13 determines whether there is the same target object in both the rear image JC and the side image JL or JR (step S9). For example, when it is determined that there is a target object in the side image JR, the determination unit 13 obtains the direction vector of the object located in the target region 81 of the rear image JC. The determination unit 13 can determine that an object having a direction vector whose difference from the direction vector of the target object in the side image JR is equal to or less than the threshold among the objects in the rear image JC is the same as the target object.

[0068] When it is determined that there is the same target object in both (step S9: YES), the compositing unit 14 performs a blending process on the composite image J. In the blending process, the compositing unit 14 determines the image region of the target object as the blending region, and performs a transparency process on the blending region with different transparency rates for each image (step S10).

[0069] Figures 7A and 7B illustrate the blending process of the target object. As shown in FIG. 7A, the synthesizing unit 14 determines the image area of the rear image JC corresponding to the detection frame 72 of the target object as the blending area 82C. The synthesizing unit 14 also determines the area of the detection frame 72 of the side image JR as the blending area 82R. The synthesizing unit 14 moves the boundary line L1 between the rear image JC and the side image JR to the center line in the horizontal direction x of the blending areas 82C and 82R.

[0070] As shown in FIG. 7B, the synthesizing unit 14 performs a transparency process on the area of the blending area 82C of the rear image JC that is the half area on the vehicle 50 side (left side) from the boundary line L1. At this time, the synthesizing unit 14 sets the transmittance FC at the left end on the vehicle 50 side to 100% and the transmittance FC at the boundary line L1 to 50%, and reduces the transmittance FC as it goes from the vehicle 50 side toward the boundary line L1 on the side opposite to the vehicle 50. Note that the synthesizing unit 14 performs a transparency process on the area of the end portion Cs on the side opposite to the vehicle 50 from the boundary line L1 outside the blending area 82C at a transmittance of 100%.

[0071] In addition, the synthesizing unit 14 performs a transparency process on the blending area 82R of the side image JR. The synthesizing unit 14 sets the transmittance FR at the left end on the vehicle 50 side to 0% and the transmittance FR at the right end on the side opposite to the vehicle 50 to 100%, and increases the transmittance FR as it goes from the vehicle 50 side toward the opposite side.

[0072] FIG. 8 shows an example of the synthesized image J when the blending process is executed. As shown in FIG. 8, within the detection frame 72 of the synthesized image J, the target object of the side image JR appears darker the closer it is to the vehicle 50 side, and the target object of the rear image JC gradually appears as it goes toward the boundary line L1. Since the contour of the target object blurs, it becomes difficult to visually recognize the difference in the sizes of the respective images. Since both the target objects of the rear image JC and the side image JR are displayed, the flicker due to the change in size is reduced even when switching to the display of only one of the images.

[0073] On the other hand, when it is determined that the target object is not in both the rear image JC and the side image JL or JR but only in one of them (step S9: NO), the composition unit 14 performs a blending process on the composite image J. In the blending process, the composition unit 14 determines a region obtained by expanding the image region of the target object by n times as the blending region, and performs a transparency process on the blending region at different transparency rates for each image (step S11).

[0074] FIGS. 9A and 9B illustrate the blending process of the target object. As shown in FIG. 9A, although the object is photographed in both the rear image JC and the side image JR, the object is not detected in the side image JR, and the object is detected only from the rear image JC and the detection frame 72 is set. In this case, the composition unit 14 determines, as the blending region 83C of the rear image JC, a region 83 obtained by expanding the region of the detection frame 72 by n times in the horizontal direction x on the side opposite to the vehicle 50. Further, the composition unit 14 moves the boundary line L1 between the rear image JC and the side image JR to the position of the center in the horizontal direction x of the blending region 83C. The composition unit 14 determines, as the blending region 83R, the image region of the side image JR that has the same position as the blending region 83C.

[0075] As shown in FIG. 9B, the composition unit 14 performs a transparency process on the blending region 83C of the rear image JC. At this time, the composition unit 14 sets the transparency rate FC at the left end on the vehicle 50 side to 0%, the transparency rate FC at the boundary line L1 to 50%, and the transparency rate FC at the right end on the side opposite to the vehicle 50 to 100%, and increases the transparency rate FC as it goes from the vehicle 50 side to the opposite side. Note that the composition unit 14 performs a transparency process on the region Cs of the end portion on the side opposite to the vehicle 50 from the boundary line L1 outside the blending region 83C at a transparency rate of 100%.

[0076] Further, the composition unit 14 performs a transparency process on the blending region 83R of the side image JR. The composition unit 14 sets the transparency rate FR at the left end on the vehicle 50 side to 100%, the transparency rate FR at the boundary line L1 to 50%, and the transparency rate FR at the right end on the side opposite to the vehicle 50 to 0%, and decreases the transparency rate FR as it goes from the vehicle 50 side to the opposite side.

[0077] In this way, even when the target object is not detected in either the rear image JC or one of the side images JL or JR, the flickering of the target object can be reduced by the blending process in the same manner as when it is detected from both. When there is a target object in only one of the rear image JC and the side image JL or JR, the target object in the rear image JC appears stronger as it gets closer to the vehicle 50, and the target object in the side image JL or JR appears stronger as it gets closer to the side opposite to the vehicle 50. Also, it is possible to make the target object that was not detected due to the expansion of the blending area also be a target for the transparency process. Therefore, not only the detected target object but also the other target object that was not detected but would have been photographed can be displayed in a transparent state.

[0078] Figures 10A to 10C show examples of the composite image J when the blending process is executed. As shown in Figure 10A, when the detection frame 72 of the rear vehicle 51 is included in the rear image JC, the blending process is not executed. As shown in Figure 10B, when the rear vehicle 51 crosses the boundary line L1 due to a lane change, the inside of the detection frame 72 is blended, and the rear vehicle 51 is displayed in a transparent state in the rear image JC and the side image JR. Further, as the rear vehicle 51 approaches and, as shown in Figure 10C, when all of the rear vehicle 51 moves beyond the boundary line L1 to the side image JR side, the blending process is cancelled.

[0079] Normally, when the rear vehicle 51 moves across the boundary line L1 from the rear image JC to the side image JR, the size of the rear vehicle 51 suddenly changes, resulting in flickering. However, according to the blending process, the contour of the rear vehicle 51 at the boundary line L1 becomes blurred, and the difference in size between the rear image JC and the side image JR becomes difficult to visually recognize, thus reducing such flickering.

[0080] As described above, according to the present embodiment, when there is a target object detected from the target area 81 of the rear image JC or the side images JL, JR, the image area of the target object in the rear image JC and the image area of the target object in the side image JL or JR are subjected to a transparency process with different transmittance rates by the blending process.

[0081] When the processed image regions are superimposed, the contour of the target object blurs, making it difficult to recognize the difference in the size of the target object between the rear image JC and the side image JL or JR. Since both the target objects in the rear image JC and the side image JL or JR are displayed, even when the display of the target object is switched to only one of them, the size does not change suddenly, so the flickering is reduced and the discomfort is less. Therefore, the display of the rear object between the captured images can be switched naturally.

[0082] The above embodiment is a preferred example of the present invention and is not limited thereto, and can be changed as appropriate. For example, as described above, if the position of the boundary line L1 is moved to the center of the horizontal direction x of the blend region and the transmittance at the boundary line L1 is set to 50%, the rear image JC and the side image JL or JR can be blended by half, and a more natural composite image can be obtained, which is preferable. However, if a natural composite image J can be obtained by the blending process, the position of the boundary line L1 is not limited to the center of the horizontal direction x, and it may be near the center or at a certain distance from the center. Also, instead of increasing or decreasing the transmittance at a constant gradient, the transmittance may be increased or decreased in a curved shape.

Explanation of Reference Numerals

[0083] 100 Driving support system 25 Image display system 10 Image processing apparatus 12 Detection unit 13 Determination unit 14 Composite unit 15 Display control unit 20 Display device 3C, 3L, 3R Cameras

Claims

1. An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, comprising: a composite unit that generates the composite image by overlapping a rear image captured behind the vehicle on a part on the vehicle side of two side images respectively captured on the left rear side and the right rear side; a detection unit that detects an object from the rear image and the side images; a determination unit that determines whether or not a target object detected from a target region for blending processing of the rear image or the side image is present based on the detection result of the object; and when it is determined that the target object is present, the composite unit performs a blending process of performing a transparency process on each image region of the target object in the rear image and the target object in the side image at different transmittance rates; the determination unit determines whether the same target object is present in both the rear image and the side image, or whether the target object is present only in one of the rear image and the side image; when it is determined that the same target object is present in both the rear image and the side image, the composite unit, in the blending process, determines an image region of the target object in the rear image as a blending region, performs a transparency process on the blending region, and reduces the transmittance rate from the vehicle side toward the side opposite to the vehicle; determines an image region of the target object in the side image as a blending region, performs a transparency process on the blending region, and increases the transmittance rate from the vehicle side toward the side opposite to the vehicle Image processing apparatus.

2. The determination unit determines the degree of danger of the object detected from the target region, and determines the object detected from the target region and having a high degree of danger as the target object The image processing apparatus according to claim 1.

3. The determination unit determines whether or not the traveling direction of the object detected from the target region is the same as the traveling direction of the vehicle, and determines the object detected from the target region and having the same traveling direction as the target object The image processing apparatus according to claim 1 or 2.

4. when it is determined that the target object is present only in one of the rear image and the side image, the composite unit, in the blending process, for the image in which it is determined that the target object is present, determines, as a blending region, a region obtained by expanding, by n times, an image region of the target object in the image toward the side opposite to the vehicle For an image determined to have no target object, an image area in the image that has the same position as the blend area of the image determined to have the target object is determined as the blend area. Perform a transparency process on the blend area corresponding to the rear image, and increase the transparency rate from the vehicle side toward the side opposite to the vehicle. Perform a transparency process on the blend area corresponding to the side image, and decrease the transparency rate from the vehicle side toward the side opposite to the vehicle. The image processing apparatus according to claim 1.

5. The synthesizing unit Determine the position of the boundary line between the rear image and the side image in the synthesized image at the horizontal center of the blend area. Set the transparency rates of the rear image and the side image at the boundary line to 50% respectively. The image processing apparatus according to claim 1 or 4.

6. A system that provides a captured image of the rear of a vehicle, comprising An image processing apparatus that generates one synthesized image from a plurality of captured images of the rear of the vehicle, and A display device that displays the synthesized image. The image processing apparatus A synthesizing unit that generates the synthesized image by overlapping a rear image captured at the rear of the vehicle with a part on the vehicle side of two side images captured at the left rear side and the right rear side respectively. A detection unit that detects an object from the rear image and the side images. A determination unit that determines the presence or absence of a target object detected from the target area of the blend process of the rear image or the side image based on the detection result of the object. When it is determined that there is the target object, the synthesizing unit executes a blend process of performing a transparency process on each image area of the target object in the rear image and the target object in the side image at different transparency rates. The determination unit determines whether there is the same target object in both the rear image and the side image, or whether the target object exists only in one of the rear image and the side image. When it is determined that there is the same target object in both the rear image and the side image, the synthesizing unit Determine the image area of the target object in the rear image as the blend area, perform a transparency process on the blend area, and decrease the transparency rate from the vehicle side toward the side opposite to the vehicle. Determine the image area of the target object in the side image as the blend area, perform a transparency process on the blend area, and increase the transparency rate from the vehicle side toward the side opposite to the vehicle. System

7. The computer is caused to generate a composite image by superimposing a rear image taken of the rear of the vehicle on a part on the vehicle side of two side images taken of the left rear side and the right rear side respectively; detect an object from the rear image and the side images; a determination step of determining whether or not a target object detected from a target region for blending processing of the rear image or the side image is present based on the detection result of the object; a composite step of, when it is determined that the target object is present, performing a blending process of performing a transparency process on each image region of the target object in the rear image and the target object in the side image at different transmittance rates; be executed, in the determination step, further execute a process of determining whether the same target object is present in both the rear image and the side image, or whether the target object is present only in one of the rear image and the side image; in the composite step, when it is determined that the same target object is present in both the rear image and the side image, in the blending process, determine the image region of the target object in the rear image as a blending region, perform a transparency process on the blending region, and reduce the transmittance rate from the vehicle side toward the side opposite to the vehicle; determine the image region of the target object in the side image as a blending region, perform a transparency process on the blending region, and execute a process of increasing the transmittance rate from the vehicle side toward the side opposite to the vehicle for a program.

Images