Image Processing Apparatus, System, and Program

Abstract

To provide a natural synthesis image.SOLUTION: An image processor 10 includes: a synthesis unit 14 for synthesizing a rear image of the back of a vehicle and two side images of the left back side and the right back side of the vehicle, respectively, and generating a synthesis image; and a detection unit 12 for detecting an object from the rear image and the side images. The synthesis unit 14 performs deformation processing. In the deformation processing, the synthesis unit 14 magnifies the side images in a vertical direction on the basis of the size of the object and the magnification rate becomes larger in a position closer to the vehicle in a horizontal direction.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a system, and a program.

Background Art

[0002] Conventionally, a driving support system has been used in which a camera captures the rear and the left and right rear sides of a vehicle, and combines them into a single image for display on a display inside the vehicle. Since the position and the angle of view of the camera are different between the rear and the rear sides, in a composite image obtained simply by connecting the captured images, the sense of distance may be difficult to understand.

[0003] Therefore, it is common to perform image processing on the captured images at the time of composition so as to provide a composite image as natural as when viewed. For example, in Patent Document 1, in order to give a sense of perspective to the composite image, the captured image is divided into a near region and a far region with respect to the vehicle, and the far region is compressed in the vertical direction.

[0004] Further, in Patent Document 2, the lower side of the captured image of the rear side with respect to the horizon is compressed in the horizontal direction, and the captured image of the rear is reduced and superimposed on the center thereof. As a result, the lanes of the captured images are connected, and a natural composite image is generated.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, small-sized subjects such as motorcycles and pedestrians may disappear when the captured image is reduced horizontally as described above. Also, the same vehicle may be partially included in the captured images of the rear and the rear sides, but the size of the same vehicle in the captured images is different due to the difference in the shooting positions. If they are directly synthesized, since the sizes of the vehicles are partially different, it is difficult to recognize them as one vehicle, and a sense of incongruity occurs. Also, the sense of distance of the vehicle is difficult to understand.

[0007] An object of the present invention is to provide a natural composite image.

Means for Solving the Problems

[0008] The present invention includes the following aspects. [1] An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, a composite unit that composites a rear image captured behind the vehicle and two side images captured from the left rear side and the right rear side respectively to generate the composite image; a detection unit that detects an object from the rear image and the side images, and the composite unit performs a deformation process on the rear image and the side images, and in the deformation process, the side image is enlarged vertically based on the size of the object, and the enlargement ratio is made larger as it is closer to the vehicle side in the horizontal direction Image processing apparatus.

[0009] According to the above [1], the side image can be enlarged vertically according to the difference in the size of the object in the rear image and the object in the side image. It is possible to match the vertical sizes of the same object partially included in the rear image and the side image, and it is possible to provide a natural composite image that is easily recognized as one object.

[0010] [2] In the deformation process, the composite unit moves the enlarged side image horizontally or vertically to match the positions of the respective objects detected from the side image and the rear image horizontally or vertically The image processing apparatus described in [1] above.

[0011] According to [2] above, since the horizontal or vertical position of an object straddling the rear image and the side image is also adjusted, a more natural composite image can be provided.

[0012] [3] The composite unit moves the side image so that the positions of the object in the rear image and the side image match. The image processing apparatus described in [2] above.

[0013] According to [3] above, when a part of an object in the rear image and a part of an object in the side image become one object by composition, since their positions match, the object in the composite image can be expressed naturally.

[0014] [4] It includes a determination unit that determines whether or not each object detected from the rear image and the side image satisfies the following conditions (1) to (3). (1) The object exists in the vicinity of the vehicle. (2) The object in the rear image and the side image partially disappears by composition. (3) The object in the rear image and the object in the side image are the same. The composite unit executes the deformation process when it is determined that the conditions (1) to (3) are satisfied. The image processing apparatus according to any one of [1] to [3] above.

[0015] According to [4] above, when there is an object that satisfies conditions (1) to (3), that is, an object that disappears from the composite image or an object that is the same but has a partially varying size, the deformation process can be performed.

[0016] [5] In the vertical direction, when the object in the rear image and the side image is located below the threshold value, the determination unit determines that the condition (1) is satisfied. The image processing apparatus according to [4] above.

[0017] According to [5] above, condition (1) can be easily determined from the positions of the objects in each image.

[0018] [6] When the object in the rear image is located within a certain range from both ends in the horizontal direction of the rear image and the object in the side image is located within a certain range from one end on the vehicle side of the side image, the determination unit determines that condition (2) is satisfied. The image processing apparatus according to [4] or [5] above.

[0019] According to [6] above, condition (2) can be easily determined from the positions of the objects in each image.

[0020] [7] When both of the following conditions (3-1) and (3-2) are satisfied, the determination unit determines that condition (3) is satisfied. (3-1) The centers of the object in the rear image and the object in the side image are located within the image regions of each other's objects. (3-2) The difference between the movement amount of the object in the rear image and the movement amount of the object in the side image is within a certain range. The image processing apparatus according to any one of [4] to [6] above.

[0021] According to [7] above, condition (3) can be easily determined from the positions of the center points of the objects in each image.

[0022] [8] A system that provides a captured image of the rear of a vehicle, An image processing apparatus that generates one composite image from a plurality of captured images of the rear of the vehicle, A display device that displays the composite image, and is provided with, The image processing apparatus is A composite unit that composites a rear image captured behind the vehicle and two side images captured on the left rear side and the right rear side respectively to generate the composite image, A detection unit that detects an object from the rear image and the side image, and is provided with. The synthesizing unit performs a deformation process on the rear image and the side image. In the deformation process, the side image is enlarged in the vertical direction based on the size of the object, and the enlargement ratio is made larger as it is closer to the vehicle side in the horizontal direction. System.

[0023] According to [8] above, the side image can be enlarged according to the difference in the size of the object in the rear image and the object in the side image. The vertical sizes of the same object partially included in the rear image and the side image can be made the same, and a natural composite image that is easily recognized as one object can be provided.

[0024] [9] A computer comprising the steps of synthesizing a rear image obtained by photographing the rear of a vehicle and two side images obtained by photographing the left rear side and the right rear side respectively to generate one composite image; detecting an object from the rear image and the side images; performing a deformation process on the rear image and the side images, and a program for causing the computer to execute the steps, wherein the step of performing the deformation process includes a step of performing a deformation process of enlarging the side image in the vertical direction based on the size of the object and making the enlargement ratio larger as it is closer to the vehicle side in the horizontal direction. Program.

[0025] According to [9] above, the side image can be enlarged according to the difference in the size of the object in the rear image and the object in the side image. The vertical sizes of the same object partially included in the rear image and the side image can be made the same, and a natural composite image that is easily recognized as one object can be provided.

Advantages of the Invention

[0026] According to the present invention, a natural composite image can be provided.

Brief Description of the Drawings

[0027]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 7A

Figure 7B

Figure 8A

Figure 8B

Figure 9A

Figure 9B

Figure 9C

Embodiments for Carrying Out the Invention

[0028] Hereinafter, embodiments of the image processing apparatus, system, and program of the present invention will be described with reference to the drawings.

[0029] FIG. 1 shows the configuration of an operation support system 100 according to an embodiment. The operation support system 100 is mounted on a vehicle and supports the driver's driving by displaying captured images of the rear and rear sides of the vehicle.

[0030] As shown in FIG. 1, the operation support system 100 includes an image display system 25, three cameras 3C, 3L, and 3R, various sensors 4, a measurement device 5, and the like. The image display system 25 includes an image processing apparatus 10 and a display device 20.

[0031] The cameras 3C, 3L, and 3R continuously capture the rear, left rear side, and right rear side of the vehicle, respectively, and sequentially generate time-series captured images. Hereinafter, the captured image of the rear is referred to as a rear image, and the captured image of the rear side is referred to as a side image. The camera 3C that captures the rear is disposed, for example, near the license plate. The cameras 3L and 3R that capture the left and right rear sides are provided, for example, at the positions of the left and right side mirrors, respectively.

[0032] The measurement device 5 measures the positions, directions, distances to objects, moving speeds of objects, etc. of objects around the vehicle, such as vehicles and people. Examples of the measurement device 5 include an ultrasonic sonar or a millimeter-wave radar. The operation support system 100 can acquire measurement information of objects around the vehicle from these measurement devices 5.

[0033] The sensor 4 detects the state of the vehicle. Examples of the sensor 4 include various sensors mounted on the vehicle, such as a vehicle speed sensor, a steering angle sensor, and a blinker sensor. The operation support system 100 can acquire detection signals from these sensors 4 as information indicating the state of the vehicle.

[0034] (Display device) The display device 20 displays the composite image generated by the image processing device 10. The display device 20 may be an in-vehicle display having a general aspect ratio, or may be a room mirror-type display having a long horizontal-to-vertical ratio.

[0035] FIG. 2 shows an example of the arrangement of the display device 20 in the vehicle. The display device 20 is installed on the dashboard 21 in the vehicle interior or near the upper center of the front window 22 instead of the inner mirror. In FIG. 2, the display device 20 is composed of a horizontally long single liquid crystal display or the like, and one composite image J is displayed thereon.

[0036] The images (videos) captured by the cameras 3C, 3L, and 3R are synthesized, and one composite image J is displayed on the display device 20. In the composite image J, the captured image JL of the camera 3L is displayed on the left side, the captured image JC of the camera 3C is displayed in the center, and the captured image JR of the camera 3R is displayed on the right side.

[0037] Note that the display device 20 may be composed of a plurality of displays connected in a row. In this case, one composite image J is divided, and each divided image is displayed in the display area of each display.

[0038] (Image Processing Device) The image processing device 10 synthesizes the rear image captured by the camera 3C and the side images captured by the cameras 3L and 3R into one image.

[0039] The image processing device 10 includes a correction unit 11, a detection unit 12, a determination unit 13, a synthesis unit 14, and a display control unit 15. The functions of each part of the image processing device 10 may be realized by software processing in which a program is read and executed by a computer such as an ECU (Electronic Control Unit) or a microcomputer mounted on the vehicle. Also, each part of the image processing device 10 may be composed of hardware such as an FPGA (field-programmable gate array).

[0040] The correction unit 11 performs correction processes such as distortion correction and color correction on the rear image and the side image.

[0041] The detection unit 12 detects an object included in each image after the correction process.

[0042] In deep learning, the detection unit 12 uses a plurality of captured images taken from various directions of detection targets such as vehicles and people with different vehicle types and perspectives as training data, and learns the characteristics of the objects to be detected. For example, the detection unit 12 can learn the characteristics of buses and passenger cars, and detect an image portion having similar characteristics thereto as a vehicle. By learning the characteristics for each vehicle type, the vehicle type of the vehicle can also be discriminated.

[0043] The detection unit 12 can set a detection frame surrounding the detected object in the image. In the present embodiment, the detection frame is rectangular, but the shape is not limited to a rectangular shape as long as the size of the detection frame substantially matches the size of the object. For example, the detection frame may be a shape that outlines the contour of the object, or may be an elliptical shape or the like having the same horizontal and vertical sizes as the object.

[0044] The determination unit 13 determines whether or not a rear vehicle detected from the rear image by the detection unit 12 has approached the vehicle. The determination unit 13 can use information such as the state of the vehicle detected by the sensor 4 or the position and speed of the object measured by the measuring device 5 for the determination.

[0045] The synthesis unit 14 synthesizes the rear image and the side image after the correction process to generate one synthesized image. The synthesis unit 14 has a plurality of synthesis modes, and switches the synthesis mode according to the determination result of the determination unit 13.

[0046] The display control unit 15 causes the display device 20 to display the synthesized image generated by the synthesis unit 14.

[0047] FIG. 3 shows the composite process by which the image processing apparatus 10 generates a composite image. This composite process is repeated each time the captured images generated by each of the cameras 3C, 3L, and 3R are input. When the rear image and the side images are captured by each of the cameras 3C, 3L, and 3R, in the image processing apparatus 10, the correction unit 11 corrects each image (step S1). The detection unit 12 detects the objects and their types included in the corrected rear image and side images, and sets detection frames around the detected objects (step S2).

[0048] Next, the determination unit 13 determines whether or not the condition for executing the deformation process is satisfied (step S3). In the composite image generated by the composite unit 14, the sizes of the same object straddling the rear image and the side images are different, and may look unnatural. Also, the object in the side image may disappear from the composite image due to the composition. The deformation process is a process for preventing the objects in the rear image and the side images from disappearing after the composition and making them look natural.

[0049] First, the composite method of the composite unit 14 will be described with reference to FIGS. 4A and 4B. As shown in FIG. 4A, the shooting range 61C of the camera 3C partially overlaps with the shooting ranges 61L and 61R of the cameras 3L and 3R. The composite unit 14 adjusts the trimming ranges 62C, 62L, and 62R, the pan, the tilt, and the roll of the rear image JC, the side images JL, and JR so that the rear image JC, the side images JL, and JR are seamlessly combined on the composite plane L3 at the rear n (m) (for example, n = 15) of the vehicle 50.

[0050] The composite unit 14 connects the left and right side images JL and JR, and generates a composite image J having a layer structure in which the rear image JC is superimposed in the center thereof. The composite unit 14 can narrow the display range 63C of the rear image JC in the composite image J and widen the display ranges 63L and 63R of the side images JL and JR by performing a transmission process on both end portions Cs of the rear image JC.

[0051] As shown in FIG. 4B, the rear vehicle 51 located in the vicinity from the composite surface L3 to the vehicle 50 may straddle the boundary line L1 between the trimming range 62C and the trimming range 62L or 62R. In this case, in either the rear image JC or the side image JL or JR, a part of the rear vehicle 51 is included, but not all of it. Since the size of the rear vehicle 51 in the image is different between the rear image JC and the side image JL or JR, an unnatural composite image J that is difficult to be recognized as the same vehicle is generated when they are directly combined.

[0052] The determination unit 13 determines such a case where an unnatural composite image J is generated. Specifically, the determination unit 13 determines whether the following three conditions are satisfied based on the detection result by the detection unit 12. (1) An object exists in the vicinity of the vehicle (2) The objects in the rear image and the side image are partially disappeared by the composition (3) The object in the rear image is the same as the object in the side image

[0053] When it is determined that all three conditions are satisfied (step S3: YES), the composition unit 14 executes the deformation processing in steps S4 to S6 before composing the rear image JC, the side images JL and JR. When it is determined that any of the three conditions is not satisfied (step S3: NO), the composition unit 14 composes the rear image JC, the side images JL and JR without executing the deformation processing (step S7).

[0054] First, the determination unit 13 determines whether the condition (1) is satisfied. The closer the distance to the vehicle 50 is, the lower the object is located in the vertical direction y in the rear image JC, the side images JL and JR. Since the detection frame is set to surround the object and its size is substantially the same as that of the object, the determination unit 13 can determine whether the object exists in the vicinity of the vehicle 50 based on the position of the detection frame by regarding the detection frame as the object.

[0055] FIG. 5A illustrates a method for determining an object in the vicinity of the vehicle 50. As shown in FIG. 5A, a rear vehicle 51 is detected from the rear image JC. At the time of detection, a detection frame 71 surrounding the rear vehicle 51 is set by the detection unit 12. When the rear vehicle 51, that is, its detection frame 71 is located below the threshold Th1 in the vertical direction y, the determination unit 13 determines that the rear vehicle 51 exists in the vicinity of the vehicle 50. In the side images JL and JR as well, the same determination can be made as in the rear image JC.

[0056] The threshold Th1 can be determined based on the relationship between the distance from the object to the vehicle 50 obtained in advance and the position of the detection frame 71 of the object on the image. Since the position of the camera 3C is shifted from those of the cameras 3L and 3R, the threshold Th1 may be different between the rear image JC and the side image JL or JR.

[0057] When condition (1) is satisfied, the determination unit 13 determines whether condition (2) is satisfied. FIG. 5B illustrates the determination method in the rear image JC. As shown in FIG. 5B, a rear vehicle 51 is detected from the rear image JC, and its detection frame 71 is set. When the right end 71R of the detection frame 71 is located to the right of the threshold Th22 set on the right side of the rear image JC in the horizontal direction x, or when the left end 71L of the detection frame 71 is located to the left of the threshold Th21 set on the left side of the rear image JC, the determination unit 13 determines that the rear vehicle 51 in the rear image JC will partially disappear by synthesis.

[0058] That is, when the left end 71L of the detection frame 71 is included within a certain range from the left end position x1 to the threshold Th21, or when the right end 71R of the detection frame 71 is included within a certain range from the threshold Th22 to the right end position x2 in the horizontal direction x, it is determined that the rear vehicle 51 in the rear image JC will partially disappear. In FIG. 5B, since the right end 71R of the detection frame 71 is located at the position x2, it is determined that the rear vehicle 51 will partially disappear.

[0059] The threshold values Th21 and Th22 may be at positions x1 and x2 at both ends in the horizontal direction x of the rear image JC, but as shown in FIG. 5B, it is preferable that they are located on the center side by a certain distance from both ends. Even for the same object, the size or position of the detection frame 71 may vary depending on the detection accuracy. However, by setting the threshold values Th21 and Th22 at positions slightly closer to the center from both ends, it is possible to accurately detect an object that partially disappears even when there is variation in the detection frame 71.

[0060] FIG. 5C illustrates a determination method in the side image JL. In the side image JL, among the positions x1 and x3 at both ends of the region where the rear image JC is superimposed in the horizontal direction x, the threshold value Th31 is set at a position on the vehicle 50 side by a certain distance from the position x1. When the right end 71R of the detection frame 71 in the side image JL is located on the right side (vehicle side) of the threshold value Th31, the determination unit 13 determines that the rear vehicle 51 in the side image JL partially disappears due to synthesis.

[0061] That is, in the side image JL, as shown in FIG. 5C, when the right end 71R of the detection frame 71 is included within a certain range from the threshold value Th31 to the position x3, it is determined that the rear vehicle 51 in the side image JL partially disappears. The threshold value Th31 may be the position x1 at the left end of the region where the rear image JC is superimposed, but as shown in FIG. 5C, it is preferably located on the vehicle side by a certain distance from the left end. Thereby, even when there is variation in the size of the detection frame for the same object, it is possible to accurately detect a partial object.

[0062] Although FIG. 5C shows an example of the left side image JL, in the case of the right side image JR, the determination can be made in the same manner except that the left - right relationship is reversed. When the determination unit 13 determines that an object is partially included in at least one of the left side image JL and the right side image JR together with the rear image JC, it determines that the condition (2) is satisfied.

[0063] When conditions (1) and (2) are satisfied, the determination unit 13 determines whether condition (3) is satisfied. In condition (3), the determination unit 13 determines that the objects are the same when both of the following two conditions (3-1) and (3-2) are satisfied. (3-1) The positions of the object in the rear image and the object in the side image are close to each other on the composite image (3-2) In both the rear image and the side image, the difference in the amount of movement of the object is within a certain range

[0064] Figure 6A illustrates the determination method for condition (3-1). The determination unit 13 calculates the position coordinates in the horizontal direction x and the vertical direction y for the center P1 of the detection frame 71 in the side image JR and the center P2 of the detection frame 72 in the rear image JC. As shown in Figure 6A, when the centers P1 and P2 of each detection frame 71 and 72 are within each other's detection frames, the determination unit 13 determines that condition (3-1) is satisfied. On the other hand, when they are not within each other's detection frames, the determination unit 13 determines that condition (3-1) is not satisfied.

[0065] Figure 6B illustrates the determination method for condition (3-2). The determination unit 13 calculates how much the detection frame 71 in the side image JR has moved compared to the side image JR one frame before. Specifically, the determination unit 13 calculates a vector β1 representing the amount of movement from the position P11(x1,y1) of the detection frame 71 one frame before to the current position P12(x2,y2). The positions P11 and P12 can be obtained, for example, as the center of the lower side of the detection frame 71. Let the angle formed by the vector with respect to the horizontal direction x be θ, then the vector β1 is calculated by the following equation [1]. β1=(x2-x1) / cosθ [1]

[0066] Similarly, the determination unit 13 calculates a vector β2 representing the amount of movement from the position P21(x3,y3) of the detection frame 72 in the rear image JC one frame before to the current position P22(x4,y4). Then, if the difference between the vectors β1 and β2 is within a certain range, the determination unit 13 determines that the objects in the rear image JC and the side image JR are the same.

[0067] Next, the deformation process will be described. In the deformation process, first, the synthesis unit 14 enlarges the side images JL and JR. FIGS. 7A and 7B illustrate the enlargement method. As shown in FIG. 7A, the rear image JC and the left side image JL each partially include a rear vehicle 51 that crosses the rear of the vehicle 50. The rear vehicle 51 in the rear image JC and the side image JL is the same, but the sizes in the images are different, and it is difficult to recognize them as the same vehicle if they are synthesized as they are.

[0068] As shown in FIG. 7B, the synthesis unit 14 enlarges the side images JL and JR in the vertical direction y, and increases the enlargement ratio as it gets closer to the vehicle 50 side in the horizontal direction x. The cameras 3L and 3R are farther from the objects behind than the camera 3C, and the objects in the side image JL or JR are likely to be smaller in size than the objects in the rear image JC. Therefore, the size of the object in the side image JL or JR can be adjusted to match the size of the object in the rear image JC by the above deformation.

[0069] The enlargement ratio is determined based on the ratio of the sizes of the detection frames of the rear image JC, the side images JL and JR. For example, when the detection frame 72 of the rear image JC is 1.3 times larger than the detection frame 73 in the left side image JL, the enlargement ratio at the left end of the side image JL can be set to 1.0, the enlargement ratio at the right end can be set to 1.3 times, and the enlargement ratio can be set to increase at a constant gradient from 1.0 to 1.3 in between.

[0070] Next, the synthesis unit 14 connects the left and right side images JL and JR in the horizontal direction x, and overlaps the rear image JC in the center. By itself, as shown in FIG. 7B, the positions of the detection frames 72 and 73 do not match. Therefore, the synthesis unit 14 aligns the positions of the rear vehicle 51 in the horizontal direction x or the vertical direction y.

[0071] FIGS. 8A and 8B illustrate the alignment. The synthesizing unit 14 calculates the difference d1 between the lower side position of the detection frame 73 in the left side image JL and the lower side position of the detection frame 72 in the rear image JC. The synthesizing unit 14 moves the position of the side image JL in the vertical direction y so that this difference d1 becomes 0, and makes the position of the detection frame 73 coincide with the detection frame 72 of the rear image JC.

[0072] After aligning the positions in the vertical direction y, as shown in FIG. 8B, the synthesizing unit 14 calculates the difference d2 between the left side (or right side) position of the detection frame 73 in the left side image JL and the detection frame 72 in the rear image JC. The synthesizing unit 14 moves the position of the side image JL in the horizontal direction x so that this difference d2 becomes 0, and makes the position of the detection frame 73 coincide with the detection frame 72 of the rear image JC. As a result, the same rear vehicle 51 that is partially photographed in the side image JL and the rear image JC is connected at the same size and the same position. A natural composite image J that is easily visible as one rear vehicle can be generated.

[0073] FIGS. 9A to 9C show other examples of the deformation process. As shown in FIG. 9A, the same rear vehicle 511 is included in the right side image JR and the rear image JC. The same rear vehicle 512 is also included in the left side image JL and the rear image JC. If each image is synthesized as usual without being subjected to the deformation process, as shown in FIG. 9B, a part of the rear vehicles 511 and 512 disappears on the composite image J. Also, the side image JR includes a person 513, but the rear image JC is overlaid and disappears from the composite image J.

[0074] On the other hand, in the case of the deformed process, as shown in FIG. 9C, the rear vehicles 511 and 512 that straddle the two images have substantially the same size and position in the composite image J and can be visually recognized as one vehicle. An in-depth feeling is also generated by the enlargement of the side images JL and JR, and a natural composite image J similar to when viewed visually is obtained. Also, since the person 513 on the side image JR is not disappeared from the composite image and is displayed, the driver can recognize the presence of the person 513.

[0075] As described above, according to the present embodiment, based on the sizes of the detection frame 72 in the rear image JC and the detection frame 73 in the side image JL or JR, the side image JL or JR is enlarged in the vertical direction y, and a deformation process is executed in which the magnification rate is increased as it approaches the vehicle 50 side in the horizontal direction x.

[0076] By the enlargement, it is possible to match the sizes in the vertical direction y of the same object partially included in the rear image JC and the side image JL or JR, and it is possible to provide a natural composite image J that is easily recognized as one object.

[0077] The above deformation process is executed when it is determined that the conditions (1) to (3) are satisfied. Therefore, when there is the same object partially included in the rear image JC and the side image JL or JR and an unnatural composite image J is generated, the composite image J can be corrected by the deformation process.

[0078] The above embodiment is a preferred example of the present invention and is not limited thereto, and can be changed as appropriate. For example, the present invention can also be applied to an image processing apparatus that performs synthesis other than the above-described synthesis method as long as it is a synthesis method in which there is a possibility that an object on the side image disappears when the side image and the rear image are overlapped.

Explanation of Reference Numerals

[0079] 100 Driving support system 25 Image display system 10 Image processing apparatus 12 Detection unit 13 Determination unit 14 Synthesis unit 15 Display control unit 20 Display device 3C, 3L, 3R Cameras

Claims

1. An image processing apparatus that generates one composite image from a plurality of captured images behind a vehicle, a composite unit that composites a rear image captured behind the vehicle and two side images respectively captured from the left rear side and the right rear side to generate the composite image; a detection unit that detects the same object from the rear image and the side images, and the composite unit performs a deformation process on the rear image and the side images, and in the deformation process, the side image is enlarged vertically based on the ratio of the size of the object in the rear image to the size of the object in the side image, and the magnification rate is made larger as it is closer to the vehicle side in the horizontal direction Image processing apparatus.

2. In the deformation process, the composite unit moves the enlarged side image in the horizontal direction or the vertical direction to align the positions of the respective objects detected from the side image and the rear image in the horizontal direction or the vertical direction The image processing apparatus according to claim 1.

3. The composite unit moves the side image so that the positions of the objects in the rear image and the side image coincide. The image processing apparatus according to claim 2.

4. The apparatus further includes a determination unit that determines whether each object detected from the rear image and the side images satisfies the following conditions (1) to (3): (1) The object exists in the vicinity of the vehicle (2) The objects in the rear image and the side images are partially disappeared by the composition (3) The object in the rear image and the object in the side image are the same The composite unit executes the deformation process when it is determined that the conditions (1) to (3) are satisfied. The image processing apparatus according to any one of claims 1 to 3.

5. When, in the vertical direction, the object in the rear image and the side image is located below a threshold value, the determination unit determines that the condition (1) is satisfied. The image processing apparatus according to claim 4.

6. When the object in the rear image is located within a certain range from both ends in the horizontal direction of the rear image and the object in the side image is located within a certain range from one end on the vehicle side of the side image, the determination unit determines that the condition (2) is satisfied. The image processing apparatus according to claim 4 or 5.

7. When both of the following conditions (3-1) and (3-2) are satisfied, the determination unit determines that the condition (3) is satisfied. (3-1) The centers of the object in the rear image and the object in the side image are located within the image regions of each other's objects. (3-2) The difference between the amount of movement of the object in the rear image and the amount of movement of the object in the side image is within a certain range. The image processing apparatus according to any one of claims 4 to 6.

8. A system that provides a captured image of the rear of a vehicle, An image processing apparatus that generates one composite image from a plurality of captured images of the rear of the vehicle, A display device that displays the composite image, and The image processing apparatus, A composite unit that composites a rear image captured behind the vehicle and two side images captured on the left rear side and the right rear side respectively to generate the composite image, A detection unit that detects the same object from the rear image and the side image, and The composite unit performs a deformation process on the rear image and the side image. In the deformation process, the side image is enlarged vertically based on the ratio of the sizes of the object in the rear image and the object in the side image, and the enlargement ratio is made larger as it is closer to the vehicle side in the horizontal direction. System.

9. A program for causing a computer to synthesize a rear image taken of the rear of a vehicle and two side images taken of the left rear side and the right rear side respectively to generate one synthesized image; detect the same object from the rear image and the side images; perform a deformation process on the rear image and the side images, the program comprising: the step of performing the deformation process includes performing a deformation process of vertically enlarging the side image based on the ratio of the size of the object in the rear image to the size of the object in the side image, and making the enlargement rate larger as it is closer to the vehicle side in the horizontal direction Program.

Images