Image Control System

The image control system optimizes target display in panoramic images by adjusting azimuth angles to prioritize critical targets, addressing processing load and safety concerns in conventional systems.

JP7582158B2Active Publication Date: 2024-11-13TOYOTA JIDOSHA KK
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2021177843
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-13
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

Conventional image control systems struggle to appropriately display multiple detected targets in a panoramic composite image without increasing processing load, leading to potential reductions in vehicle driving safety.

Method used

An image control system that adjusts the magnitudes of azimuth angles independently for rear, left rear, and right rear imaging devices to prioritize the display of targets closest to the vehicle, minimizing overlapping and hidden areas while suppressing processing load.

Benefits of technology

Effectively displays critical targets in the composite image, reducing the likelihood of safety hazards by minimizing overlapping and hidden targets, thus enhancing vehicle safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582158000001
    Figure 0007582158000001
  • Figure 0007582158000002
    Figure 0007582158000002
  • Figure 0007582158000003
    Figure 0007582158000003
Patent Text Reader

Abstract

To properly display a target having a relatively high possibility of affecting travel of a vehicle in a composite image and suppress increase in a processing load related to generation of the composite image.SOLUTION: An image control system comprises: a rear camera 11Re; a left rear (right rear) camera 11L (11R); a display device 12; and an image control device 10 which generates a composite image on the basis of image data of the cameras to display the image on a display device. When an azimuth angle of a left (right) boundary line defining a field angle in the horizontal direction of a rear effective range used for generation of the composite image in an imaging range of the rear camera is defined as a rear left side (right side) azimuth angle, the image control device determines a rear target and a left rear (right rear) target closest to a vehicle in an own traffic lane and a left traffic lane (right traffic lane), and changes magnitude of the rear left side (right side) azimuth angle on the basis of a combination of presence / absence of the rear target and a degree of distance to the target and presence / absence of the left rear (right rear) target and a degree of distance to the target.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image control system that generates a composite image in a panoramic format from image data acquired by a plurality of image capture devices and displays the image on a display device. [Background technology]

[0002] Conventionally, a technique is known for generating a panoramic composite image from image data acquired by a plurality of imaging devices (typically, vehicle-mounted cameras) and displaying the image on a display device. For example, an image display control device described in Patent Document 1 (hereinafter also referred to as a "conventional device") acquires a first rear image captured from the rear of a vehicle body, a second rear image captured from the right rear of the vehicle body, and a third rear image captured from the left rear of the vehicle body. When the conventional device determines that a target has been detected based on these rear images, it converts each of the first to third rear images using a homography calculated based on the distance from the rear of the vehicle body to the target so that the angle of view of the first rear image becomes wider as the distance becomes shorter. Then, the converted rear images are combined to generate a panoramic composite image with the rear end of the vehicle body as the viewpoint, and the image is displayed on a display device. In Patent Document 1, the target is described as an "obstacle." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-189576 A Summary of the Invention

[0004] Patent Document 1 describes that the conventional device can smooth the joints of the composite image and display the entire target in the composite image. However, in the configuration of the conventional device, when multiple targets are detected, these targets may not be properly displayed in the composite image. That is, the conventional device is configured to convert the first to third rear images for one detected target so that the angle of view of the first rear image becomes wider as the distance from the rear of the vehicle body to the target becomes shorter. For this reason, it is considered that the entire target can be displayed by performing the above-mentioned conversion process for the target. However, Patent Document 1 does not consider how to convert each rear image when other targets are detected, so these other targets may not be properly displayed in the composite image, which may result in a decrease in the driving safety of the vehicle.

[0005] However, if an attempt is made to properly display all detected targets in a composite image, the conversion process becomes complicated, and this may increase the processing load of the conventional device.

[0006] The present invention has been made to address the above-mentioned problems. That is, one of the objects of the present invention is to provide an image control system that can appropriately display, in a composite image, a target that is relatively likely to affect the driving of a vehicle, while suppressing an increase in the processing load required for generating the composite image.

[0007] The image control system according to the present invention (hereinafter also referred to as the "system of the present invention") comprises: A rear imaging device (11Re) that captures images of targets and lane markings present in a rear imaging range (Rre) extending behind the vehicle (V); A left rear imaging device (11L) that captures an image of a target and a marking line present in a left rear imaging range (Rl) that extends to the left rear of the vehicle and partially overlaps with the rear imaging range; A right rear imaging device (11R) that captures an image of a target and a marking line present in a right rear imaging range (Rr) that extends to the right rear of the vehicle and partially overlaps with the rear imaging range; a display device (12) having a display screen (12a); Obtaining rear image data obtained by imaging using the rear imaging device, left rear image data obtained by imaging using the left rear imaging device, and right rear image data obtained by imaging using the right rear imaging device, A composite image in a panoramic format is generated based on the image data. displaying the composite image on the display screen of the display device; An image control device (10) configured as described above; Equipped with. Among the rear imaging range, the left rear imaging range, and the right rear imaging range, imaging ranges used for generating the composite image are defined as a rear effective range (Rrea), a left rear effective range (Rla), and a right rear effective range (Rra), respectively; Among the boundaries (30la, 30ra) that define the horizontal angle of view of the rear effective range when the orientation of the optical axis (Are) of the rear imaging device is taken as a rear reference orientation, the angles of a rear left boundary line (30la) that is the boundary line on the left side of the vehicle and a rear right boundary line (30ra) that is the boundary line on the right side of the vehicle from the rear reference orientation are defined as a rear left azimuth angle (θl) and a rear right azimuth angle (θr), respectively; Among the boundaries (40ia, 40oa) that define the horizontal angle of view of the left rear effective range when the orientation of the optical axis (Al) of the left rear image capture device is defined as a left rear reference orientation, an angle of a left rear inner boundary line (40ia), which is a boundary line on the inside in the vehicle width direction of the vehicle, from the left rear reference orientation is defined as a left rear inner azimuth angle (θli); When the orientation of the optical axis (Ar) of the right rear image capture device is defined as a right rear reference orientation, the angle of a right rear inner boundary line (50ia), which is a boundary line on the inside in the vehicle width direction of the vehicle, from the right rear reference orientation is defined as a right rear inner azimuth angle (θri), The image control device (10) The rear effective range (Rrea) can be changed by changing the magnitudes of the rear left azimuth angle (θl) and the rear right azimuth angle (θr) independently of each other within the range of the horizontal angle of view of the rear imaging device, The left rear effective range (Rla) can be changed by changing the magnitude of the left rear inner azimuth angle (θli) within the range of the horizontal angle of view of the left rear imaging device, The right rear effective range (Rra) can be changed by changing the magnitude of the right rear inner azimuth angle (θri) within the range of the horizontal angle of view of the right rear imaging device, Based on the dividing lines (d1, d2, d3, d4) detected from the image data, the positions and shapes of the lane (L) in which the vehicle is located, the left lane (Ll) adjacent to the lane on the left side thereof, and the right lane (Lr) adjacent to the lane on the right side thereof are calculated; When generating the composite image, Among the targets located rearward of the rear end portion of the vehicle detected from the image data, a rear target, a left rear target, and a right rear target are identified as targets that are closest to the vehicle in the own lane, the left lane, and the right lane, respectively; changing the magnitude of the rear left azimuth angle (θl) based on a combination of the presence or absence of the rear object target and the degree of distance to the rear object target, and the presence or absence of the left rear object target and the degree of distance to the left rear object target; changing the magnitude of the rear right azimuth angle (θr) based on a combination of the presence or absence of the rear object target and the distance to the rear object target, and the presence or absence of the right rear object target and the distance to the right rear object target; When the magnitude of the rear left azimuth angle (θl) or the rear right azimuth angle (θr) is changed to a relatively small value, the magnitude of the left rear inner azimuth angle (θli) or the right rear inner azimuth angle (θri) is changed to a relatively large value, thereby generating the composite image; generating the composite image by changing the magnitude of the left rear inner azimuth angle or the right rear inner azimuth angle to a relatively small value when the magnitude of the left rear azimuth angle or the right rear inner azimuth angle is changed to a relatively large value, It is structured as follows.

[0008] In the system of the present invention, when generating a synthetic image, the magnitude of the rear left azimuth (the angle of the boundary line on the left side of the vehicle (rear left boundary line) among the boundaries that define the horizontal angle of view of the rear effective range from the rear reference azimuth) is changed based on a combination of the presence or absence of a rear object target (the object closest to the vehicle in the own lane) and the degree of distance to the rear object target (for example, either "close", "middle", or "far") and the presence or absence of a left rear object target (the object closest to the vehicle in the left lane) and the degree of distance to the left rear object target. Then, when the magnitude of the rear left azimuth is changed, the magnitude of the left rear inner azimuth (the angle of the boundary line on the inside in the vehicle width direction (left rear inner boundary line) among the boundaries that define the horizontal angle of view of the left rear effective range from the left rear reference azimuth) is changed accordingly. By appropriately changing the magnitude of the rear left side azimuth angle and the magnitude of the left rear inner azimuth angle based on the above combination, the rear target object and the left rear target object (or the detected target object, if only one of them is detected) can be appropriately displayed in the composite image.

[0009] Similarly, in the system of the present invention, when generating a composite image, the magnitude of the rear right azimuth (the angle of the boundary line on the right side of the vehicle (rear right boundary line) among the boundaries that define the horizontal angle of view of the rear effective range from the rear reference azimuth) is changed based on a combination of the presence or absence of a rear object target and the distance to the rear object target, and the presence or absence of a right rear object target (the target closest to the vehicle in the right lane) and the distance to the right rear object target. When the magnitude of the rear right azimuth is changed, the magnitude of the right rear inner azimuth (the angle of the boundary line on the inside in the vehicle width direction (right rear inner boundary line) among the boundaries that define the horizontal angle of view of the right rear effective range from the right rear reference azimuth) is changed accordingly. By appropriately changing the magnitudes of the rear right azimuth and the right rear inner azimuth based on the above combination, the rear object target and the right rear object target (if only one of them is detected, the detected object) can be appropriately displayed in the composite image.

[0010] Among the objects located behind the vehicle, the objects that are relatively likely to affect the traveling of the vehicle are the objects closest to the vehicle in the own lane, the left lane, and the right lane (i.e., the rear object, the left rear object, and the right rear object). Therefore, according to the configuration of the system of the present invention, the objects that are relatively likely to affect the traveling of the vehicle can be appropriately displayed in the composite image. In addition, the rear left azimuth and the left rear inner azimuth, and the rear right azimuth and the right rear inner azimuth are changed based only on information about the rear target, the left rear target, and the right rear target (i.e., the presence or absence of these targets and the distance to these targets), and information about other targets is not taken into consideration. Therefore, it is possible to suppress an increase in the processing load required for generating a composite image.

[0011] In one aspect of the invention, The image control device (10) The rear left azimuth angle (θl) is changeable between a first rear left angle (θn) and a second rear left angle (θw) that is greater than the first rear left angle, The rear right azimuth angle (θr) is changeable between a first rear right angle (-θn) and a second rear right angle (-θw) that is greater than the first rear right angle, The left rear inner azimuth angle (θli) is changeable between a first left rear inner angle (−θ1) and a second left rear inner angle (−θ2) that is greater than the first left rear inner angle, The right rear inner azimuth angle (θri) is changeable between a first right rear inner angle (θ1) and a second right rear inner angle (θ2) that is greater than the first right rear inner angle. It is structured as follows.

[0012] According to this configuration, the rear target, the left rear target, and the right rear target can be appropriately displayed in the composite image with a relatively simple configuration.

[0013] In one aspect of the invention, The image control device (10) A virtual projection plane (Sp) perpendicular to the ground contact surface of the vehicle is set at a position rearwardly spaced a predetermined projection distance (D3) determined as a fixed value from the rear end of the vehicle (V), generating, as a composite image, an image obtained by projecting the targets and the lane markings present in the rear effective range (Rrea), the left rear effective range (Rla), and the right rear effective range (Rra) onto the projection surface based on a viewpoint (Pv) that is virtually set on or near the vehicle; When generating the composite image, changing the rear left azimuth angle (θl), the left rear inner azimuth angle (θli), the rear right azimuth angle (θr), and the right rear inner azimuth angle (θri) so that a left intersection point (Pl) which is an intersection point between the rear left boundary line (30la) and the left rear inner boundary line (40ia) and a right intersection point (Pr) which is an intersection point between the rear right boundary line (30ra) and the right rear inner boundary line (50ia) are located on the projection plane in a plan view of the vehicle; It is structured as follows.

[0014] In one aspect of the present invention, the left intersection (intersection of the rear left boundary line and the left rear inner boundary line) is located on the projection surface in a plan view of the vehicle. Therefore, the overlapping portion (overlapping area) between the rear effective range and the left rear effective range can be minimized, and the area in which targets and / or lane lines are displayed in double in the composite image can be minimized. Similarly, in the system of the present invention, the right intersection (intersection of the rear right boundary line and the right rear inner boundary line) is located on the projection surface in a plan view of the vehicle. Therefore, the overlapping portion (overlapping area) between the rear effective range and the right rear effective range can be minimized, and the area in which targets and / or lane lines are displayed in double in the composite image can be minimized. Hereinafter, the double display of targets and / or lane lines in the composite image is also referred to as the "doubling phenomenon."

[0015] The left intersection is located at a juncture between the rear image data and the left rear image data in the composite image (strictly speaking, image data corresponding to the rear effective range in the rear image data and image data corresponding to the left rear effective range in the left rear image data), while the right intersection is located at a juncture between the rear image data and the right rear image data in the composite image (strictly speaking, image data corresponding to the rear effective range in the rear image data and image data corresponding to the right rear effective range in the right rear image data).

[0016] in this case, The area between a first straight line (21) that passes through the rear end of the vehicle (V) and extends in the vehicle width direction of the vehicle, and a second straight line (22) that is spaced a predetermined first distance (D1) rearward from the first straight line and parallel to the first straight line is defined as a first zone (Z1), the area between the second straight line and a third straight line (23) that is spaced a predetermined second distance (D2) rearward from the first straight line and parallel to the first straight line is defined as a second zone (Z2), and the area behind the third straight line (23) is defined as a third zone (Z3), the projection distance (D3) is greater than the first distance and less than the second distance; If the intersections of a left imaginary line and a right imaginary line extending rearward from the left rear corner (Cl) and the right rear corner (Cr) of the vehicle in parallel with the front-rear axis (A) of the vehicle and the projection plane (Sp) are defined as a first intersection (P11) and a second intersection (P12), respectively, When changing the magnitude of the rear left azimuth angle (θl) or the rear right azimuth angle (θr) to the relatively small value, in a plan view of the vehicle, the left intersection (Pl) or the right intersection (Pr) is located on or near the first intersection, or on or near the second intersection, When the magnitude of the rear left azimuth angle or the rear right azimuth angle is changed to the relatively large value, the left intersection point or the right intersection point is located at a position that is a predetermined distance away from the first intersection point or the second intersection point, respectively, to the outside in the vehicle width direction of the vehicle, the distance being equal to or greater than a typical lane width.

[0017] According to this configuration, regardless of the combination of the rear object and the left rear object (or the right rear object), the rear object, the left rear object and the right rear object can be appropriately displayed in the composite image.

[0018] in this case, The projection distance (D3) is smaller than half the sum of the first distance (D1) and the second distance (D2).

[0019] According to this configuration, regardless of the combination of the rear object and the left rear object (or the right rear object), the rear object, the left rear object and the right rear object can be more appropriately displayed in the composite image.

[0020] In one aspect of the invention, The image control device (10) When the rear target is located in the first zone (Z1) and the left rear target is located behind the first zone, the magnitude of the rear left azimuth angle (θl) is changed to the relatively large value; When the rear target is located in the first zone and the right rear target is located behind the first zone, the magnitude of the rear right azimuth angle (θr) is changed to the relatively large value. It is structured as follows.

[0021] In this configuration, depending on the positional relationship between the rear object and the left rear object (or the right rear object), the left rear object (or the right rear object) may be hidden behind the rear object, and the left rear object (or the right rear object) may not be displayed in the composite image (hereinafter, this phenomenon is also referred to as the "hiding phenomenon"). On the other hand, if the magnitude of the rear left azimuth angle (or the rear right azimuth angle) is changed to a relatively small value, a part of the rear object (typically, a front corner part) may be cut off from the composite image (hereinafter, this phenomenon is also referred to as the "cut-off phenomenon"). That is, the turn signal lamp of the rear object is not displayed in the composite image, and as a result, the driver of the vehicle may not be able to determine from the composite image whether the rear object has an intention to change lanes. If the vehicle changes lanes without noticing the intention of the rear object, even though the rear object has an intention to change lanes, the rear object may come too close to or come into contact with the vehicle, and the driving safety of the vehicle may be reduced.

[0022] Therefore, in one aspect of the present invention, when the rear object is located in the first zone and the left rear object (or the right rear object) is located behind the first zone, the magnitude of the rear left azimuth (or the rear right azimuth) is changed to a relatively large value. That is, improving the visibility of the rear object is prioritized over improving the hiding phenomenon of the left rear object (or the right rear object). With this configuration, at least the rear object can be appropriately displayed in the composite image, thereby suppressing a decrease in the driving safety of the vehicle.

[0023] In one aspect of the invention, The image control device (10) When the rear target is located in the first zone (Z1) and the left rear target does not exist, the magnitude of the rear left azimuth angle (θl) is changed to the relatively large value; When the rear target is located in the first zone and the right rear target does not exist, the magnitude of the rear right azimuth angle (θr) is changed to the relatively large value. It is structured as follows.

[0024] If the magnitude of the rear left azimuth (or rear right azimuth) is changed to a relatively small value, a phenomenon occurs in which the rear object is cut off, and the vehicle may change lanes without the driver of the vehicle not noticing that the rear object has an intention to change lanes. In this case, the rear object may approach or contact the vehicle excessively, decreasing the driving safety of the vehicle. Therefore, in one aspect of the present invention, when the rear object is located in the first zone and the left rear object (or the right rear object) does not exist, the magnitude of the rear left azimuth (or rear right azimuth) is changed to a relatively large value. With this configuration, the rear object can be appropriately displayed in the composite image, so that the decrease in the driving safety of the vehicle can be suppressed.

[0025] In one aspect of the invention, The image control device (10) When the rear target and the left rear target are both located in the first zone (Z1), the magnitude of the rear left azimuth angle (θl) is changed to the relatively small value; When the rear target and the right rear target are both located in the first zone, the magnitude of the rear right azimuth angle (θr) is changed to the relatively small value. It is structured as follows.

[0026] When the magnitude of the rear left azimuth angle (or rear right azimuth angle) is changed to a relatively small value, the left rear object (or right rear object) may be properly displayed in the composite image, but the rear object may be cut off, and the vehicle may change lanes without the driver of the vehicle not noticing that the rear object has an intention to change lanes. In this case, the rear object may approach or contact the vehicle excessively, decreasing the driving safety of the vehicle. On the other hand, when the magnitude of the rear left azimuth angle (or rear right azimuth angle) is changed to a relatively large value, the cut-off phenomenon of the rear object is resolved, but the left rear object (or right rear object) may not be displayed in the composite image (disappear in the composite image) due to being located in a blind spot area between the rear effective range and the left rear effective range (or the right rear effective range) (hereinafter, this phenomenon is also referred to as a "disappearance phenomenon"). If the vehicle changes lanes without the driver noticing the presence of the left rear target (or right rear target) due to the disappearance phenomenon, the left rear target (or right rear target) may come into contact with the vehicle while traveling straight, reducing the vehicle's driving safety.

[0027] Here, even if the rear object is cut off, a part of the rear object is displayed in the composite image, so the driver can recognize the presence of the rear object itself. On the other hand, if the left rear object (or the right rear object) is lost, the target is not displayed at all in the composite image, so the driver cannot recognize the presence of the target. Therefore, in one aspect of the present invention, when the rear object and the left rear object (or the right rear object) are both located in the first zone, the magnitude of the rear left azimuth (or the rear right azimuth) is changed to a relatively small value. That is, the improvement of the loss of the left rear object (or the right rear object) is given priority over the improvement of the cut-off phenomenon of the rear object. According to this configuration, the left rear object (or the right rear object) can be appropriately displayed in the composite image, so that the deterioration of the driving safety of the vehicle can be suppressed.

[0028] In one aspect of the invention, The image control device (10) When the rear target and the left rear target are both located in the third zone (Z3), the magnitude of the rear left azimuth angle (θl) is changed to the relatively large value; When the rear target and the right rear target are both located in the third zone, the magnitude of the rear right azimuth angle (θr) is changed to the relatively large value; When the rear target does not exist and the left rear target is located in the third zone, the magnitude of the rear left azimuth angle (θl) is changed to the relatively large value; When the rear target does not exist and the right rear target is located in the third zone, the magnitude of the rear right azimuth angle (θr) is changed to the relatively large value. It is structured as follows.

[0029] According to this configuration, the doubling phenomenon does not occur in the left rear object (or the right rear object), and the appearance of the composite image is improved.

[0030] In one aspect of the invention, The image control device (10) In the case of generating the composite image, When changing the rear left azimuth angle (θl) and the left rear inside azimuth angle (θli), these azimuth angles are changed continuously, When changing the rear right azimuth angle (θr) and the right rear inner azimuth angle (θri), these azimuth angles are changed continuously. It is structured as follows.

[0031] According to this configuration, the movement of the joints of the composite images becomes smooth, so that the composite images can be switched smoothly.

[0032] In the above description, in order to facilitate understanding of the invention, the symbols used in the embodiments are added in parentheses to the constituent elements of the invention corresponding to the embodiments, but each constituent element of the invention is not limited to the embodiment defined by the said symbols. [Brief description of the drawings]

[0033] [Figure 1] 1 is a schematic configuration diagram of an image control system according to an embodiment of the present invention. [Diagram 2] 2 is a plan view showing the installation positions and imaging ranges of the cameras in the image control system. FIG. [Diagram 3] 4 is a flowchart showing a routine executed by a CPU of an image control ECU 10 provided in the image control system. [Figure 4] FIG. 2 is a plan view showing the relative positions of other vehicles with respect to the host vehicle. [Figure 5A] 11 is a plan view showing the effective ranges Rrea, Rla, and Rra and the projection surface when the azimuth angles θl and θr are changed to angles θn and -θn, respectively. FIG. [Figure 5B] 11 is a plan view showing the effective ranges Rrea, Rla, and Rra and the projection surface when the azimuth angles θl and θr are changed to angles θw and -θw, respectively. FIG. [Figure 6A] This is the matrix M1 used to determine the azimuth angle θl. [Figure 6B] This is the matrix M2 used to determine the azimuth angle θr. [Figure 7A] FIG. 11 is a plan view showing a state in which there is no target in the own lane and an target in the left lane is located in each zone when θl=θn. [Figure 7B] FIG. 11 is a plan view showing a state in which there is no target in the own lane and an target in the left lane is located in each zone when θl=θw. [Figure 8A] FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z3 and objects in the left lane are located in each zone when θl=θn. [Figure 8B] FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z3 and objects in the left lane are located in each zone when θl=θw. [Figure 9A]FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z2 and objects in the left lane are located in each zone when θl=θn. [Figure 9B] FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z2 and objects in the left lane are located in each zone when θl=θw. [Figure 10A] FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z1 and objects in the left lane are located in each zone when θl=θn. [Figure 10B] FIG. 11 is a plan view showing a state in which an object in the own lane is located in zone Z1 and objects in the left lane are located in each zone when θl=θw. [Figure 11] FIG. 2 is a plan view showing an imaging range of an image control system as a first comparative example, and is used to explain the phenomenon in which a target is cut off in a composite image. [Figure 12] FIG. 2 is a plan view showing an imaging range of an image control system as a first comparative example, and is used to explain a doubling phenomenon of a target in a composite image. [Figure 13] FIG. 11 is a plan view showing an imaging range of an image control system as a second comparative example, and is used to explain the disappearance and hiding phenomena of a target in a composite image. [Figure 14A] 13 is a modified matrix M3 used to determine the azimuth angle θl. [Figure 14B] 4 is a modified matrix M4 used to determine the azimuth angle θr. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] (composition) FIG. 1 shows a schematic configuration diagram of an image control system 1 according to an embodiment of the present invention. As shown in FIG. 1, the image control system 1 includes an image control ECU 10 as an image control device, a camera 11, and a display device 12. The camera 11 and the display device 12 are connected to the image control ECU 10. The image control ECU 10 includes a microcomputer as a main part. ECU is an abbreviation of Electronic Control Unit. The microcomputer includes a CPU, a ROM, a RAM, an interface (I / F), etc., and the CPU realizes various functions by executing instructions (programs, routines) stored in the ROM. Hereinafter, a vehicle equipped with the image control system 1 is referred to as a "host vehicle". The image control ECU 10 is also simply referred to as an "ECU 10".

[0035] The camera 11 includes a rear camera 11Re as a rear imaging device, a left side camera 11L as a left rear imaging device, and a right side camera 11R as a right rear imaging device. As shown in Fig. 2, the rear camera 11Re is installed in the center of the rear end of the vehicle V so that its optical axis Are is approximately parallel to the ground surface of the vehicle V. In other words, the optical axis Are is parallel to the front-rear axis A of the vehicle V. However, the extension direction of the optical axis Are is not limited to this, and for example, the rear camera 11Re may be installed so that the optical axis Are is slightly inclined downward.

[0036] The left side camera 11L and the right side camera 11R are respectively installed under the left and right side mirrors of the vehicle V such that their optical axes Al and Ar are approximately parallel to the ground surface of the vehicle V. However, the extending direction of the optical axes Al and Ar is not limited to this, and for example, the left side camera 11L and the right side camera 11R may be installed such that the optical axes Al and Ar are slightly inclined downward. Hereinafter, the rear camera 11Re, the left side camera 11L, and the right side camera 11R will also be referred to as "camera 11Re, camera 11L, and camera 11R," respectively. Note that the installation positions of the cameras 11L and 11R are not limited to this, and for example, they may be built into the left and right side mirrors, respectively.

[0037] The camera 11Re captures the scenery behind the vehicle V (strictly speaking, directly behind and diagonally behind the vehicle V). The range Rre is a plan view of the imaging range of the camera 11Re (the rear imaging range extending behind the vehicle V). The imaging range Rre has a sector shape centered on the position of the camera 11Re and is line-symmetrical with respect to the longitudinal axis A. The horizontal and vertical angles of view of the camera 11Re are preset to values ​​such that targets (typically, other vehicles and motorcycles) and lane lines located behind the vehicle V can be included in the imaging range Rre. In this embodiment, the horizontal angle of view of the camera 11Re is set to 44 degrees, but is not particularly limited. The boundary line 30l and the boundary line 30r are the left and right boundary lines of the vehicle V, respectively, among the boundary lines that define the horizontal angle of view of the camera 11Re. Hereinafter, the boundary lines 30l and 30r are also simply referred to as "boundaries 30l and 30r of the imaging range Rre". The camera 11Re captures an image of a subject (typically, a target and a lane marking) present in an imaging range Rre, and outputs the obtained image data to the ECU 10 as rear image data.

[0038] The camera 11L captures the scenery to the left rear of the vehicle V (more precisely, to the left rear side). The range Rl is a plan view of the imaging range of the camera 11L (left rear imaging range expanding to the left rear of the vehicle V). The imaging range Rl has a sector shape centered on the position of the camera 11L. The horizontal angle of view and the vertical angle of view of the camera 11L are preset to values ​​such that the landmarks and lane markings located to the left rear of the vehicle V can be included in the imaging range Rl. In this embodiment, the horizontal angle of view of the camera 11L is set to 45 degrees, but is not particularly limited. The boundary line 40i and the boundary line 40o are the inner boundary line and the outer boundary line in the vehicle width direction of the vehicle V, respectively, among the boundaries that define the horizontal angle of view of the camera 11L. Hereinafter, the boundary lines 40i and 40o are also simply referred to as "boundaries 40i and 40o of the imaging range Rl". The boundary line 40i is parallel to the front-rear axis A, but the extension direction of the boundary line 40i is not limited to this. The imaging range Rl partially overlaps with the imaging range Rre (see the hatched portion). The camera 11L captures an image of a subject located in the imaging range Rl and outputs the obtained image data to the ECU 10 as left rear image data.

[0039] The camera 11R captures the scenery to the right rear (strictly speaking, to the right rear side) of the vehicle V. The range Rr is a plan view of the imaging range of the camera 11R (the right rear imaging range expanding to the right rear of the vehicle V). The imaging range Rr has a sector shape centered on the position of the camera 11R. The horizontal angle of view and the vertical angle of view of the camera 11R are preset to values ​​such that the target and the lane markings located to the right rear of the vehicle V can be included in the imaging range Rr. In this embodiment, the horizontal angle of view of the camera 11R is set to 45 degrees, but is not particularly limited. The boundary line 50i and the boundary line 50o are the inner boundary line and the outer boundary line in the vehicle width direction of the vehicle V, respectively, among the boundaries that define the horizontal angle of view of the camera 11R. Hereinafter, the boundary lines 50i and 50o are also simply referred to as "boundaries 50i and 50o of the imaging range Rr". The boundary line 50i is parallel to the front-rear axis A, but the extension direction of the boundary line 50i is not limited to this. The imaging range Rr partially overlaps with the imaging range Rre (see the hatched portion). The camera 11R captures an image of a subject located in the imaging range Rr and outputs the obtained image data to the ECU 10 as right rear image data. In FIG. 2, in order to make the drawing easier to see, the scale of the host vehicle V and the scale of the image capture ranges Rre, Rl, and Rr are illustrated to be different from each other.

[0040] Returning to FIG. 1, the explanation will be continued. The display device 12 has a display 12a (display screen) provided at a position visible to the driver. The display device 12 may be, for example, an inner mirror (room mirror) or a display device provided in a navigation system. In the former case, the mirror surface of the inner mirror is used as the display 12a, and in the latter case, a touch panel is used as the display 12a.

[0041] The ECU 10 acquires the rear image data, the left rear image data, and the right rear image data output from the cameras 11Re, 11L, and 11R at every predetermined time T. Then, based on the acquired image data, the ECU 10 generates a panoramic composite image (described in detail later) and displays it on the display 12a of the display device 12.

[0042] (Details of operation) Next, a detailed description will be given of the operation of the ECU 10. The CPU of the ECU 10 is configured to repeatedly execute a routine shown in the flowchart of Fig. 3 every time a predetermined time T has elapsed while the ignition switch is on.

[0043] At a predetermined timing, the CPU starts the process from step 300 in FIG.

[0044] In step 310, the CPU acquires the rear image data, the left rear image data, and the right rear image data output from the cameras 11Re, 11L, and 11R, respectively.

[0045] In step 320, the CPU analyzes each image data acquired in step 310 to detect a target, and calculates the relative relationship between the host vehicle V and the target. Here, the "relative relationship between the host vehicle V and the target" includes the distance from the host vehicle V to the target, the direction and relative speed of the target with respect to the host vehicle V, etc.

[0046] In addition, strictly speaking, the "distance from the vehicle V to the target" means the distance (shortest distance) from an arbitrary reference position (typically, the center of the rear end) of the vehicle V to the target. That is, the CPU calculates the distance to the target detected from each image data (in other words, the distance to the target when the positions of the cameras 11Re, 11L, and 11R are used as references), converts the distance to a distance from the reference position of the vehicle V, and calculates the converted distance as the "distance from the vehicle V to the target". In addition, strictly speaking, the "orientation of the target relative to the vehicle V" means the orientation of the target relative to the reference position of the vehicle V. That is, the CPU calculates the orientation of the target detected from each image data (in other words, the orientation of the target when the positions of the cameras 11Re, 11L, and 11R are used as references), converts the orientation to an orientation relative to the reference position of the vehicle V, and calculates the converted orientation as the "orientation of the target relative to the vehicle V". This allows the position (distance and direction) of the target relative to the host vehicle V (its reference position) to be calculated with high accuracy.

[0047] In addition, the CPU analyzes each image data to detect a dividing line. The dividing line is a line marked on a road to divide the traffic of vehicles into different directions. The CPU calculates the position and shape of the lane based on the detected dividing line. A lane is defined as an area between two adjacent dividing lines extending on a roadway. The camera 11Re is capable of detecting dividing lines constituting at least the own lane L, the left lane Ll, and the right lane Lr. Here, the own lane L is the lane in which the own vehicle V is located, the left lane Ll is the lane adjacent to the own lane L on its left side, and the right lane Lr is the lane adjacent to the own lane L on its right side. The camera 11L is capable of detecting dividing lines constituting at least the left lane Ll. The camera 11R is capable of detecting dividing lines constituting at least the right lane Lr.

[0048] In the example of Fig. 4, the imaging range Rre includes other vehicles Vre, Vl, and Vr2 as targets, and also includes four lane markings d1 to d4. Therefore, the CPU analyzes the rear image data to detect the other vehicles Vre, Vl, and Vr2, and calculates the relative relationships between the host vehicle V and the other vehicles Vre, Vl, and Vr2. In addition, the CPU analyzes the rear image data to detect the lane markings d1 to d4, and calculates the positions and shapes of the host lane L, the left lane Ll, and the right lane Lr. The imaging range Rl includes the other vehicle Vl as a target and two lane markings d1 and d2. Therefore, the CPU analyzes the left rear image data to detect the other vehicle Vl and calculates the relative relationship between the host vehicle V and the other vehicle Vl. In addition, the CPU analyzes the left rear image data to detect the lane markings d1 and d2 and calculates the position and shape of the left lane Ll. Note that the target (in this example, the other vehicle Vl) and lane markings (in this example, the lane markings d1 and d2) located in the overlapping portion between the imaging range Rre and the imaging range Rl are detected separately from the rear image data and the left rear image data. The imaging range Rr includes other vehicles Vr1 and Vr2 as targets, as well as two lane markings d3 and d4. Therefore, the CPU analyzes the right rear image data to detect the other vehicles Vr1 and Vr2, and calculates the relative relationships between the host vehicle V and the other vehicles Vr1 and Vr2. In addition, the CPU analyzes the right rear image data to detect the lane markings d3 and d4, and calculates the position and shape of the right lane Lr. Note that the target (in this example, the other vehicle Vr2) and lane markings (in this example, the lane markings d3 and d4) located in the overlapping portion between the imaging range Rre and the imaging range Rr are detected separately from the rear image data and the right rear image data.

[0049] In step 330, the CPU determines in which lane each of the detected targets is located, based on the calculation result in step 320. Then, the CPU identifies the target in each lane (the own lane L, the left lane Ll, and the right lane Lr). Here, "target in a lane" means the target located behind the rear end of the own vehicle V in the lane and having the shortest distance from the own vehicle V. Note that "target located behind the rear end of the own vehicle V" includes "target at least a part of which is located behind the rear end of the own vehicle V." Hereinafter, "shortest distance from the own vehicle V" is also simply referred to as "closest to the own vehicle V." On the other hand, if the CPU does not detect a target object located behind the rear end of the vehicle V on a certain lane, it determines that there is no target object on that lane.

[0050] In the example of Fig. 4, the CPU determines, based on the calculation results obtained by analyzing the rear image data, that the other vehicle Vre is located in the own lane L, the other vehicle Vl is located in the left lane Ll, and the other vehicle Vr2 is located in the right lane Lr. The CPU also determines, based on the calculation results obtained by analyzing the left rear image data, that the other vehicle Vl is located in the left lane Ll. Furthermore, based on the calculation results obtained by analyzing the right rear image data, the CPU determines that both the other vehicles Vr1 and Vr2 are located in the right lane Lr.

[0051] The targets on the own vehicle lane L are included only in the rear image data. Therefore, the CPU identifies the target closest to the own vehicle V among the targets determined to be located on the own vehicle lane L (the other vehicle Vre in the example of FIG. 4) as the target on the own vehicle lane L (rear target).

[0052] The object on the left lane Ll may be included in both the rear image data and the left rear image data. Therefore, the CPU specifies the object closest to the host vehicle V among the object Ol(re) (in the example of FIG. 4, another vehicle Vl) determined to be located on the left lane Ll based on the rear image data and the object Ol(l) (in the example of FIG. 4, another vehicle Vl) determined to be located on the left lane Ll based on the left rear image data, as the object in the left lane Ll. When the distances from the host vehicle V to the object Ol(re) and the object Ol(l) are equal to each other, the CPU specifies one of the objects (in the example of FIG. 4, for example, the other vehicle Vl detected from the rear image data) as the object in the left lane Ll (left rear object). This is also true when specifying the object in the right lane Lr.

[0053] The object on the right lane Lr may be included in both the rear image data and the right rear image data. Therefore, the CPU specifies the object closest to the host vehicle V (the other vehicle Vr1 in the example of FIG. 4) among the "object Or(re) determined to be located on the right lane Lr based on the rear image data (the other vehicle Vr2 in the example of FIG. 4)" and the "object Or(r) determined to be located on the right lane Lr based on the right rear image data (the other vehicles Vr1 and Vr2 in the example of FIG. 4)" as the object in the right lane Lr (right rear object).

[0054] In step 340, the CPU determines the zone in which each target object is located based on the position of the target object in each lane identified in step 330. Specifically, as shown in FIG. 4, the CPU sets zones Z1, Z2, and Z3 behind the host vehicle V. Zone Z1 is an area between a straight line 21 (first straight line) that passes through the rear end of the host vehicle V and extends in the vehicle width direction (direction perpendicular to the front-rear axis A) and a straight line 22 (second straight line) that is separated rearward from the straight line 21 by a predetermined distance D1 and is parallel to the straight line 21. Zone Z2 is an area between the straight line 22 and a straight line 23 (third straight line) that is separated rearward from the straight line 21 by a predetermined distance D2 and is parallel to the straight line 21. Zone Z3 is an area behind the straight line 23. Zone Z1 is an area that is relatively close to the host vehicle V, zone Z3 is an area that is relatively far from the host vehicle V, and zone Z2 is an area between zone Z1 and zone Z3.

[0055] The distance D1 can be set in advance by experiment or simulation so that the number of targets located in the zone Z1 is relatively small when the host vehicle V is traveling at a relatively medium or high speed, and the number of targets located in the zone Z1 is relatively large when the host vehicle V is stopped or traveling at a relatively low speed. In this embodiment, D1 = 15 [m]. The distance D2 can be set in advance by experiment or simulation so that the number of targets located in the zone Z2 is relatively small when the host vehicle V is traveling at a relatively high speed, and the number of targets located in the zone Z2 is relatively large when the host vehicle V is traveling at a relatively medium speed. In this embodiment, D2 = 30 [m]. The zone Z3 is an area where the number of targets is likely to be relatively large when the host vehicle V is traveling at a relatively high speed.

[0056] In the example of Figure 4, the CPU determines that the other vehicle Vre, which is an object in the own lane L, is located in zone Z3, determines that the other vehicle Vl, which is an object in the left lane Ll, is located in zone Z2, and determines that the other vehicle Vr1, which is an object in the right lane Lr, is located in zone Z1.

[0057] In step 350, the CPU determines the azimuth angles θl and θr by referring to matrices M1 and M2, which will be described later. In the following, the azimuth angles θl and θr will be described first, and then the process of step 350 will be described in detail.

[0058] The data ranges in the horizontal direction of the rear image data, the left rear image data, and the right rear image data correspond to the horizontal angles of view of the cameras 11Re, 11L, and 11R (in other words, the imaging ranges Rre, Rl, and Rr), respectively. As shown in FIG. 2, the imaging ranges Rre and Rl partially overlap (in the horizontal direction), and the imaging ranges Rre and Rr partially overlap (in the horizontal direction). For this reason, if these image data are used as is to generate a composite image, the range in which targets and / or lane lines are displayed double in the composite image will become large, making it difficult to achieve smooth seams. Therefore, the ECU 10 trims the horizontal ends of these image data as necessary, so that the seams in the composite image will be smooth. Hereinafter, the imaging range Rre corresponding to the trimmed rear image data will be referred to as the “effective range Rrea (rear effective range)”, the imaging range Rl corresponding to the trimmed left rear image data will be referred to as the “effective range Rla (left rear effective range)”, and the imaging range Rr corresponding to the trimmed right rear image data will be referred to as the “effective range Rra (right rear effective range)”. The effective ranges Rrea, Rla, and Rra are the imaging ranges among the imaging ranges Rre, Rl, and Rr that are used to generate a composite image.

[0059] 5A and 5B are diagrams showing examples of effective ranges Rrea, Rla, and Rra, respectively. The boundary line 30la and the boundary line 30ra are the boundary line on the left side (rear left boundary line) and the boundary line on the right side (rear right boundary line) of the vehicle V, respectively, among the boundaries that define the horizontal angle of view of the effective range Rrea. Hereinafter, the boundary lines 30la and 30ra are also simply referred to as "boundaries 30la and 30ra of the effective range Rrea". When the orientation of the optical axis Are of the camera 11Re is the rear reference orientation (0°), the angle of the boundary line 30la from the rear reference orientation is defined as "azimuth angle θl (rear left azimuth angle)", and the angle of the boundary line 30ra from the rear reference orientation is defined as "azimuth angle θr (rear right azimuth angle)". Note that the value of the azimuth angle on the left side (right side of the paper) of the vehicle V with respect to the rear reference orientation is positive, and the value of the azimuth angle on the right side (left side of the paper) of the vehicle V with respect to the rear reference orientation is negative. The ECU 10 is configured to be able to change the effective range Rrea by changing the magnitudes of the azimuth angles θl and θr independently of each other within the range of the horizontal angle of view of the camera 11Re.

[0060] Specifically, the ECU 10 can change the azimuth angle θl between an angle θn (>0) (first rear left angle) and an angle θw (>0) (second rear left angle), and can change the azimuth angle θr between an angle -θn (<0) (first rear right angle) and an angle -θw (<0) (second rear right angle). The angle θw is greater than the angle θn, and the magnitude of the angle -θw is greater than the magnitude of the angle -θn. FIG. 5A is an example when the azimuth angles θl and θr are changed to angles θn and -θn, respectively, and FIG. 5B is an example when the azimuth angles θl and θr are changed to angles θw and -θw, respectively. As shown in FIG. 5B, when the azimuth angle θl is an angle θw, the boundary line 30la of the effective range Rrea coincides with the boundary line 30l of the imaging range Rre, and when the azimuth angle θr is an angle -θw, the boundary line 30ra of the effective range Rrea coincides with the boundary line 30r of the imaging range Rre. That is, the angle θw is also the azimuth angle (angle from the rear reference azimuth) of the boundary line 30l of the imaging range Rre, and the angle -θw is also the azimuth angle (angle from the rear reference azimuth) of the boundary line 30r of the imaging range Rre. This means that "when the azimuth angle θl is changed to the angle θw, the left end of the rear image data is not trimmed, and when the azimuth angle θr is changed to the angle -θw, the right end of the rear image data is not trimmed." As is clear from the above explanation, the angles θn, -θn, θw, and -θw are all angles within the range of the horizontal angle of view of the camera 11Re.

[0061] The magnitudes of the angles θw and -θw are not limited to being configured to match the magnitudes of the azimuth angles of the boundary lines 30l and 30r of the imaging range Rre, respectively, and may be configured to be smaller than the magnitudes of the azimuth angles of the boundary lines 30l and 30r. That is, when θl=θw or θr=-θw, the boundary lines 30la and 30ra of the effective range Rrea do not have to match the boundary lines 30l and 30r of the imaging range Rre, respectively.

[0062] The boundary line 40ia and the boundary line 40oa are, respectively, the boundary line on the inside in the vehicle width direction (left rear inner boundary line) and the boundary line on the outside in the vehicle width direction of the vehicle V among the boundaries that define the horizontal angle of view of the effective range Rla. Hereinafter, the boundary lines 40ia and 40oa are also simply referred to as "boundaries 40ia and 40oa of the effective range Rla". When the orientation of the optical axis Al of the camera 11L is the left rear reference orientation (0°), the angle of the boundary line 40ia from the left rear reference orientation is defined as "azimuth angle θli (left rear inner azimuth angle)". Note that the value of the azimuth angle on the outside in the vehicle width direction (right side of the paper) with respect to the left rear reference orientation is positive, and the value of the azimuth angle on the inside in the vehicle width direction (left side of the paper) is negative. The ECU 10 is configured to be able to change the effective range Rla by changing the magnitude of the azimuth angle θli within the range of the horizontal angle of view of the camera 11L. The boundary line 40oa of the effective range Rla coincides with the boundary line 40o of the imaging range Rl (see Figs. 5A and 5B). This means that "the left end of the left rear image data is not trimmed." In other words, the ECU 10 is configured to trim only the right end as necessary when generating a composite image.

[0063] Specifically, the ECU 10 can change the azimuth angle θli between an angle -θ1 (<0) (first left rear inner angle) and an angle -θ2 (<0) (second left rear inner angle). The magnitude of the angle -θ2 is greater than the magnitude of the angle -θ1. FIG. 5A shows an example when the azimuth angle θli is changed to the angle -θ2, and FIG. 5B shows an example when the azimuth angle θli is changed to the angle -θ1. As shown in FIG. 5A, when the azimuth angle θli is the angle -θ2, the boundary line 40ia of the effective range Rla coincides with the boundary line 40i of the imaging range Rl. That is, the angle -θ2 is also the azimuth angle (angle from the left rear reference azimuth) of the boundary line 40i of the imaging range Rl. This means that "when the azimuth angle θli is changed to the angle -θ2, the right end of the left rear image data is not trimmed (that is, the effective range Rla coincides with the imaging range Rl)." As is clear from the above description, the angles -θ1 and -θ2 are both within the range of the horizontal angle of view of the camera 11L.

[0064] The magnitude of the angle -θ2 is not limited to the configuration that matches the magnitude of the azimuth angle of the boundary line 40i of the imaging range Rl, and may be configured to be smaller than the magnitude of the azimuth angle of the boundary line 40i. In other words, when θli=-θ2, the boundary line 40ia of the effective range Rla does not have to match the boundary line 40i of the imaging range Rl.

[0065] The boundary line 50ia and the boundary line 50oa are the inner boundary line (right rear inner boundary line) and the outer boundary line in the vehicle width direction of the vehicle V, respectively, among the boundaries that define the horizontal angle of view of the effective range Rra. Hereinafter, the boundary lines 50ia and 50oa are also simply referred to as "boundaries 50ia and 50oa of the effective range Rra". When the orientation of the optical axis Ar of the camera 11R is the right rear reference orientation (0°), the angle of the boundary line 50ia from the right rear reference orientation is defined as "azimuth angle θri (right rear inner azimuth angle)". Note that the value of the azimuth angle on the inner side in the vehicle width direction (right side of the paper) with respect to the right rear reference orientation is positive, and the value of the azimuth angle on the outer side in the vehicle width direction (left side of the paper) is negative. The ECU 10 is configured to be able to change the effective range Rra by changing the magnitude of the azimuth angle θri within the range of the horizontal angle of view of the camera 11R. The boundary line 50oa of the effective range Rra coincides with the boundary line 50o of the imaging range Rr (see FIGS. 5A and 5B). This means that "the right edge of the right rear image data is not trimmed." In other words, the ECU 10 is configured to trim only the left edge as necessary when generating a composite image.

[0066] Specifically, the ECU 10 can change the azimuth angle θri between an angle θ1 (>0) (first right rear inner angle) and an angle θ2 (>0) (second right rear inner angle). The angle θ2 is greater than the angle θ1. FIG. 5A shows an example when the azimuth angle θri is changed to the angle θ2, and FIG. 5B shows an example when the azimuth angle θri is changed to the angle θ1. As shown in FIG. 5A, when the azimuth angle θri is the angle θ2, the boundary line 50ia of the effective range Rra coincides with the boundary line 50i of the imaging range Rr. That is, the angle θ2 is also the azimuth angle (angle from the right rear reference azimuth) of the boundary line 50i of the imaging range Rr. This means that "when the azimuth angle θri is changed to the angle θ2, the left end of the right rear image data is not trimmed." As is clear from the above description, the angles θ1 and θ2 are both angles within the range of the horizontal angle of view of the camera 11R.

[0067] The angle θ2 is not limited to being configured to coincide with the azimuth angle of the boundary line 50i of the imaging range Rr, and may be configured to be smaller than the azimuth angle of the boundary line 50i. In other words, when θri=θ2, the boundary line 50ia of the effective range Rra does not have to coincide with the boundary line 50i of the imaging range Rr.

[0068] The ECU 10 sets a projection plane Sp behind the host vehicle V and sets a viewpoint Pv on the host vehicle V, and generates a composite image by projecting the subjects (targets and lane lines) existing in the effective ranges Rrea, Rla, and Rra onto the projection plane Sp with the viewpoint Pv as a reference. Here, the projection plane Sp is a virtual plane set at a position separated rearward by a predetermined distance D3 (projection distance) from the straight line 21 (in other words, from the rear end of the host vehicle V) so as to be orthogonal to the road surface (the grounding surface of the host vehicle V). The projection plane Sp is desirably set within the zone Z2. That is, it is desirable that D1 < D3 < D2 holds. More specifically, the projection plane Sp is desirably set in a region in front of the central position in the longitudinal axis A direction within the zone Z2. That is, it is desirable that D1 < D3 < (D1 + D2) / 2 holds. In the present embodiment, D3 = 20 [m] (that is, a fixed value), and D1 < D3 < (D1 + D2) / 2 holds. However, depending on the values of D1 and D2, the projection plane Sp may be set in another zone (zone Z1 or zone Z3).

[0069] The viewpoint Pv is set at the intersection of the extension line of the boundary line 40oa and the extension line of the boundary line 50oa. The position of the viewpoint Pv does not change due to the trimming of the left rear image data and / or the right rear image data. Note that the position of the viewpoint Pv is not limited to being on the host vehicle V. Depending on the extending directions of the boundary line 40oa and the boundary line 50oa, the viewpoint Pv may be located in front of the host vehicle V instead of on the host vehicle V.

[0070] Thus, since the distance D3 is a fixed value and the extending directions of the boundary lines 40oa and 50oa are invariant regardless of the presence or absence of trimming, the relative positions of the point P21 (the intersection of the boundary line 40oa and the projection plane Sp in the plan view of the host vehicle V) and the point P22 (the intersection of the boundary line 50oa and the projection plane Sp in the plan view of the host vehicle V) with respect to the host vehicle V are invariant respectively. Hereinafter, the "plan view of the host vehicle V" is also simply referred to as the "plan view".

[0071] As shown in FIG. 5A, when changing the azimuth angle θl to the angle θn, the ECU 10 is configured to change the azimuth angle θli to the angle -θ2. The values ​​of the angles θn and -θ2 are set in advance so that the point Pl (left intersection) of the boundary line 30la of the effective range Rrea and the boundary line 40ia of the effective range Rla is located on the projection surface Sp in a planar view. In this embodiment, the angle -θ2 is also the azimuth angle of the boundary line 40i of the imaging range Rl. Therefore, the position of the intersection of the boundary line 40ia (40i) and the projection surface Sp in a planar view can be uniquely determined based on the mounting position and performance (typically, the horizontal angle of view) of the camera 11L. The angle θn is set in advance to a value such that the boundary line 30la passes through the position of the intersection of the boundary line 40ia and the projection surface Sp determined in this manner. According to this configuration, when the azimuth angle θl=θn, the point Pl is located on the projection surface Sp in a planar view. Hereinafter, the point Pl when the azimuth angle θl=θn is specifically referred to as "point Pln."

[0072] Similarly, the ECU 10 is configured to change the azimuth angle θri to the angle θ2 when changing the azimuth angle θr to the angle -θn. The values ​​of the angles -θn and θ2 are set in advance so that the point Pr (right intersection) of the boundary line 30ra of the effective range Rrea and the boundary line 50ia of the effective range Rra is located on the projection surface Sp in a planar view. In this embodiment, the angle θ2 is also the azimuth angle of the boundary line 50i of the imaging range Rr. Therefore, the position of the intersection of the boundary line 50ia (50i) and the projection surface Sp in a planar view can be uniquely determined based on the mounting position and performance (typically, the horizontal angle of view) of the camera 11R. The angle -θn is set in advance to a value such that the boundary line 30ra passes through the position of the intersection of the boundary line 50ia and the projection surface Sp determined in this manner. According to this configuration, when the azimuth angle θr=-θn, the point Pr is located on the projection surface Sp in a planar view. Hereinafter, the point Pr when the azimuth angle θr=−θn is also referred to as the “point Prn”.

[0073] As shown in FIG. 5B, when changing the azimuth angle θl to the angle θw, the ECU 10 is configured to change the azimuth angle θli to the angle -θ1. The values ​​of the angles θw and -θ1 are set in advance so that the point Pl is located on the projection surface Sp in a planar view. In this embodiment, the angle θw is also the azimuth angle of the boundary line 30l of the imaging range Rre. Therefore, the position of the intersection between the boundary line 30la (30l) and the projection surface Sp in a planar view can be uniquely determined based on the mounting position and performance (typically, the horizontal angle of view) of the camera 11Re. The angle -θ1 is set in advance to a value such that the boundary line 40ia passes through the position of the intersection between the boundary line 30la and the projection surface Sp determined in this manner. According to this configuration, when the azimuth angle θl=θw, the point Pl is located on the projection surface Sp in a planar view. Hereinafter, the point Pl when the azimuth angle θl=θw is also particularly referred to as the "point Plw".

[0074] Similarly, the ECU 10 is configured to change the azimuth angle θri to the angle θ1 when changing the azimuth angle θr to the angle -θw. The values ​​of the angles -θw and θ1 are set in advance so that the point Pr is located on the projection surface Sp in a planar view. In this embodiment, the angle -θw is also the azimuth angle of the boundary line 30r of the imaging range Rre. Therefore, the position of the intersection between the boundary line 30ra (30r) and the projection surface Sp in a planar view can be uniquely determined based on the mounting position and performance (typically, the horizontal angle of view) of the camera 11Re. The angle θ1 is set in advance to a value such that the boundary line 50ia passes through the position of the intersection between the boundary line 30ra and the projection surface Sp determined in this manner. According to this configuration, when the azimuth angle θr=-θw, the point Pr is located on the projection surface Sp in a planar view. Hereinafter, the point Pr in the case of the azimuth angle θr=-θw is also particularly referred to as the "point Prw".

[0075] The point Pl is located at a seam (left seam) between the rear image data and the left rear image data after trimming (in other words, image data corresponding to the effective range Rrea in the rear image data and image data corresponding to the effective range Rla in the left rear image data) in the composite image. The point Pr is located at a seam (right seam) between the rear image data and the right rear image data after trimming (in other words, image data corresponding to the effective range Rrea in the rear image data and image data corresponding to the effective range Rra in the right rear image data) in the composite image. The ECU 10 is configured to be able to change the position of the left seam (point Pl) in the composite image between points Pln and Plw by changing the azimuth angle θl between angles θn and θw. The ECU 10 is configured to be able to change the position of the right seam (point Pr) in the composite image between points Prn and Prw by changing the azimuth angle θr between angles -θn and -θw.

[0076] In addition, an object existing in the effective range Rrea is projected onto the area between points Pl and Pr on the projection surface Sp. An object existing in the effective range Rla is projected onto the area between points Pl and P21 on the projection surface Sp. An object existing in the effective range Rra is projected onto the area between points Pr and P22 on the projection surface Sp.

[0077] The positions of points Pln and Prn will be described in more detail. As shown in FIG. 5A, points P11 (first intersection) and P12 (second intersection) are intersections of a left virtual line and a right virtual line extending backward from a left rear corner Cl and a right rear corner Cr of the vehicle V in parallel with the longitudinal axis A (see FIG. 2), respectively, and a projection plane Sp. In this embodiment, points Pln and Prn are located on points P11 and P12, respectively, in a plan view. Note that point Pln is not limited to being located on point P11, and may be configured to be located near point P11 based on the values ​​of angle θn and angle -θ2. Similarly, point Prn is not limited to being located on point P12, and may be configured to be located near point P12 based on the values ​​of angle -θn and angle θ2.

[0078] The positions of points Plw and Prw will be described in more detail. As shown in FIG. 5B, point Plw is located at a position spaced outward in the vehicle width direction from point P11, and the spacing distance is sufficiently larger than the lane width of the left lane Ll. Also, point Prw is located at a position spaced outward in the vehicle width direction from point P12, and the spacing distance is sufficiently larger than the lane width of the right lane Lr. Note that the spacing distance of point Plw from point P11 may be the same as the lane width of the left lane Ll, and the spacing distance of point Prw from point P12 may be the same as the lane width of the right lane Lr. That is, points Plw and Prw can be configured to be located at positions spaced outward in the vehicle width direction from points P11 and P12 by a predetermined distance equal to or greater than the general lane width, respectively.

[0079] Here, it is assumed that when each of the lanes L, Ll, and Lr is straight and the host vehicle V is traveling along the extending direction of the host lane L (the front-rear axis A is parallel to the extending direction). In this case, point Pln is located slightly inward in the vehicle width direction from the dividing line d2 (the left dividing line constituting the host lane L), and point Prn is located slightly inward in the vehicle width direction from the dividing line d3 (the right dividing line constituting the host lane L) (see FIG. 5A). Also, point Plw is located outward in the vehicle width direction from the dividing line d1 (the left dividing line constituting the left lane Ll), and point Prw is located outward in the vehicle width direction from the dividing line d4 (the right dividing line constituting the right lane Lr) (see FIG. 5B). Further, the projection plane Sp on which points Pl (Pln, Plw) and point Pr (Prn, Prw) are located is located in zone Z2, and D1 < D3 < (D1 + D2) / 2 holds.

[0080] At this time, the position where the blind spot area Rb and / or overlap area Rov (both described later) of each zone Z1 to Z3 occurs has characteristics according to the value of θl (or |θr|) (i.e., θn or θw). A specific explanation will be given below. Note that the blind spot area Rb is the area between the effective ranges Rrea and Rla, and the area between the effective ranges Rrea and Rra. An object present in the blind spot area Rb is not displayed in the composite image. The overlap area Rov is the area where the effective ranges Rrea and Rla partially overlap, and the area where the effective ranges Rrea and Rra partially overlap. An object present in the overlap area Rov is displayed doubly in the composite image. In this embodiment, points Pl and Pr are located on the projection plane Sp. Therefore, the blind spot region Rb occurs in front of the projection surface Sp (in other words, in front of points Pl and Pr), and the overlap region Rov occurs behind the projection surface Sp (in other words, behind points Pl and Pr) (see Figures 5A and 5B).

[0081] First, the case where θl=θn or θr=-θn will be described with reference to Fig. 5A. Note that the characters in parentheses indicate the characteristics when θr=-θn. In zone Z1, the following characteristics are obtained. (Feature 1n) The blind spot area Rb occurs in the vehicle's lane L except for the center and the left end (right end). (Feature 2n) The blind spot area Rb does not occur in the left lane Ll (right lane Lr). (Feature 3n) The overlapping region Rov does not occur in the own lane L or the left lane Ll (right lane Lr).

[0082] In zone Z2, the following characteristics are obtained: (Feature 4n) The blind spot area Rb occurs slightly in the portion of the own vehicle lane L in front of the point Pln (point Prn) excluding the center and the left end (right end). (Feature 5n) The blind spot area Rb does not occur in the left lane Ll (right lane Lr). (Feature 6n) The overlapping region Rov occurs slightly at the left end (right end) of the own lane L behind point Pln (point Prn). (Feature 7n) The overlapping region Rov rarely occurs in the left lane Ll (right lane Lr).

[0083] Zone Z3 has the following characteristics: (Feature 8n) The blind spot area Rb does not occur in the vehicle's own lane L or the left lane Ll (right lane Lr). (Feature 9n) The overlapping region Rov occurs slightly at the left edge (right edge) of the own vehicle lane L. (Feature 10n) The overlapping region Rov occurs at the right end (left end) of the left lane Ll (right lane Lr), and its range becomes larger toward the rear.

[0084] Next, the case where θl=θw or θr=-θw will be described with reference to Fig. 5B. In the zone Z1, the following characteristics are obtained. (Feature 1w) The blind spot area Rb occurs slightly on the left (right) side of the front part of the host vehicle's lane L. (Feature 2w) The blind spot area Rb occurs diagonally rearward and to the left across the left lane Ll (right lane Lr). (Feature 3w) The overlapping region Rov does not occur in the own lane L and the left lane Ll (right lane Lr).

[0085] In zone Z2, the following characteristics are obtained: (Feature 4w) The blind spot area Rb does not occur in the vehicle's lane L. (Feature 5w) The blind spot area Rb does not occur in the left lane Ll (right lane Lr). (Feature 6w) The overlapping region Rov does not occur in the own lane L. (Feature 7w) The overlapping region Rov does not occur in the left lane Ll (right lane Lr).

[0086] Zone Z3 has the following characteristics: (Feature 8w) The blind spot area Rb does not occur in the vehicle's own lane L or the left lane Ll (right lane Lr). (Feature 9w) The overlapping region Rov does not occur in the own lane L. (Feature 10w) The overlapping region Rov does not occur in the left lane Ll (right lane Lr).

[0087] Next, the process of step 350 will be specifically described. Using the determination result of step 340, the CPU determines (changes or maintains) the azimuth angle θl based on a combination of "the presence or absence of an object target in the own lane L and the zone in which the object target is located" and "the presence or absence of an object target in the left lane Ll and the zone in which the object target is located". FIG. 6A shows a matrix M1 representing the combination. With respect to the object target in the own lane L, "Z1 (close)", "Z2 (middle)", "Z3 (far)", and "none" in the matrix M1 respectively represent that the object target is "located in zone Z1", "located in zone Z2", "located in zone Z3", and "no object target exists". This is the same for the object target in the left lane Ll. The same is also true for the object targets in the own lane L and the right lane Lr in the matrix M2 described later. In addition, "narrow" and "wide" in the matrix M1 respectively mean "changing the azimuth angle θl to angle θn" and "changing to angle θw." "No change" means not changing the azimuth angle θl and keeping it at the value of the previous cycle. Note that the CPU changes the azimuth angle θli to the above-mentioned angle (-θ2 or -θ1) in response to the change in the azimuth angle θl.

[0088] In the example of Fig. 4, the object (other vehicle Vre) in the own lane L is located in zone Z3, and the object (other vehicle Vl) in the left lane Ll is located in zone Z2. In this case, according to the matrix M1, the azimuth angle θl is "unchanged," so the CPU does not change the azimuth angle θl. That is, for example, if θl=θw in the immediately preceding cycle, the CPU maintains the azimuth angle θl at angle θw in the current cycle (see Fig. 5B).

[0089] Similarly, the CPU uses the judgment result of step 340 to determine (change or maintain) the azimuth angle θr based on a combination of "the presence or absence of an object target in the own lane L and the zone in which the object target is located" and "the presence or absence of an object target in the right lane Lr and the zone in which the object target is located." FIG. 6B shows a matrix M2 representing the combination. "Narrow" and "Wide" in the matrix M2 mean "changing the azimuth angle θr to angle -θn" and "changing to angle -θw," respectively. "No change" means not changing the azimuth angle θr and maintaining it at the value of the previous cycle. The CPU changes the azimuth angle θri to the above-mentioned angle (θ2 or θ1) according to the change in the azimuth angle θr.

[0090] In the example of Fig. 4, the object in the own lane L (another vehicle Vre) is located in zone Z3, and the object in the right lane Lr (another vehicle Vr1) is located in zone Z1. In this case, according to the matrix M2, the azimuth angle θr is "narrow," so the CPU changes the azimuth angle θr to an angle -θn (see Fig. 5A).

[0091] In step 360, the CPU crops the image data acquired in step 310 based on the azimuth angles θl and θr determined in step 350, and generates a panoramic composite image by a known method using the cropped image data. As a result, the position of the left seam in the composite image is point Pln when θl=θn (see FIG. 5A), and point Plw when θl=θw (see FIG. 5B). Also, the position of the right seam in the composite image is point Prn when θr=-θn (see FIG. 5A), and point Prw when θr=-θw (see FIG. 5B).

[0092] In step 370, the CPU displays the composite image generated in step 360 on the display 12a of the display device 12. After that, the CPU proceeds to step 395 to temporarily end this routine.

[0093] Next, problems with an image control system as a comparative example will be described with reference to Fig. 11 to Fig. 13. In the image control system of Fig. 11 to Fig. 13, the position of the seam of the composite image cannot be changed. Note that the same components as those in the image control system 1 are given the same reference numerals and detailed description will be omitted.

[0094] In the examples of Figures 11 and 12, an image control system as a first comparative example is mounted on a vehicle V1. This image control system includes cameras 11Ren, 11L, and 11R. The horizontal angle of view of the camera 11Ren is relatively narrow, and the horizontal angles of view of the cameras 11L and 11R are relatively wide. For this reason, the imaging range Rren has a long and narrow sector shape, and the imaging ranges Rl and Rr have wide sector shapes. Points P2 and P3 respectively represent the positions of the seams in the composite image. The positions of points P2 and P3 cannot be changed. The vehicle V1 is traveling in its own lane L.

[0095] As shown in FIG. 11, a blind spot area Rb occurs between the imaging ranges Rren and Rl in front of the point P2. Similarly, a blind spot area Rb occurs between the imaging ranges Rren and Rr in front of the point P3. Behind the vehicle V1, a motorcycle Mre and another vehicle Vre are traveling in the vehicle's own lane L. Most of the motorcycle Mre, ​​including its center, is located in the blind spot area Rb, and only the left and right ends are included in the imaging ranges Rl and Rre, respectively. For this reason, the center of the motorcycle Mre is cut off from the composite image, and only the left and right ends are displayed. On the other hand, the center of the other vehicle Vre is located in the imaging range Rren, and the left and right ends are located in the blind spot area Rb. For this reason, the left and right ends of the other vehicle Vre are cut off from the composite image, and only the center is displayed.

[0096] 12, the imaging ranges Rren and Rl partially overlap behind the point P2, creating an overlapping area Rov. Similarly, the imaging ranges Rren and Rr partially overlap behind the point P3, creating an overlapping area Rov. Behind the vehicle V1, another vehicle Vl is traveling in the left lane Ll. A part of the other vehicle Vl is located in the overlapping area Rov. Therefore, a part of the other vehicle Vl is displayed doubly in the composite image.

[0097] In this way, when the horizontal angle of view of the rear camera is relatively narrow, there is a problem that a part of the target is cut off from the composite image, or at least a part of the target is displayed doubly in the composite image. Hereinafter, the former is also referred to as the "cut-off phenomenon" and the latter is also referred to as the "doubling phenomenon."

[0098] In the example of FIG. 13, an image control system as a second comparative example is mounted on a vehicle V2. This image control system includes cameras 11Re, 11Ln, and 11Rn. The horizontal angle of view of the camera 11Re is relatively wide, and the horizontal angles of view of the cameras 11Ln and 11Rn are relatively narrow. Therefore, the imaging range Rre has a wide sector shape, and the imaging ranges Rln and Rrn have slightly elongated sector shapes. Points P5 and P6 respectively represent the positions of the seams in the composite image. The positions of points P5 and P6 cannot be changed. The vehicle V2 is traveling in the own lane L.

[0099] As shown in FIG. 13, between the imaging ranges Rre and Rln, and between the imaging ranges Rre and Rrn, there are blind spots Rb in front of points P5 and P6. Behind the vehicle V2, another vehicle Vre is traveling in the own lane L, and the motorcycle Mre and the other vehicle Vl are traveling in the left lane Ll. The motorcycle Mre is entirely located in the blind spot Rb. Therefore, the motorcycle Mre is not displayed in the composite image (disappears in the composite image). In addition, although the other vehicle Vre and the other vehicle Vl are both present in the imaging range Rre, the light from the other vehicle Vl is blocked by the other vehicle Vre and does not reach the camera 11Re, so the camera 11Re cannot capture the other vehicle Vl. As a result, the other vehicle Vl is not displayed in the composite image (is hidden behind other targets in the composite image).

[0100] In this way, when the horizontal angle of view of the rear camera is relatively wide, problems occur in which targets disappear in the composite image, or targets on one lane are hidden behind targets on the other lane. Hereinafter, the former is also referred to as the "disappearance phenomenon" and the latter is also referred to as the "hiding phenomenon."

[0101] The vanishing phenomenon may occur in the examples of Figures 11 and 12. The cut-off phenomenon may occur in the example of Figure 13 (although the doubling phenomenon may occur, it is not considered in this specification because it does not occur at least in the own lane L, the left lane Ll, and the right lane Lr). However, the obscuration phenomenon tends to occur when the horizontal angle of view of the rear camera is relatively wide and a target in the left lane Ll or the right lane Lr is located diagonally behind a target in the own lane L. For this reason, the obscuration phenomenon cannot occur when the horizontal angle of view of the rear camera is relatively narrow. As described above, when the horizontal angle of view of the rear camera is relatively narrow, problems such as cut-off, disappearance and doubling of the target in the composite image become problematic. On the other hand, when the horizontal angle of view of the rear camera is relatively wide, problems such as cut-off, disappearance and hiding of the target in the composite image become problematic.

[0102] In response to this, the image control system 1 is configured to be able to change the position of the seam in the composite image by changing the magnitude of the azimuth angles θl and θr between the angles θn and θw. Therefore, among the above-mentioned phenomena (cut-off, disappearance, doubling, and hiding phenomena), it is possible to suppress the occurrence of at least the phenomenon that is most likely to affect the traveling of the vehicle V. Hereinafter, a specific description will be given with reference to the matrix M1 using Figs. 7A to 10B (since the matrix M2 has the same configuration as the matrix M1, detailed description thereof will be omitted). In Figs. 7A to 10B, there is no target object in the right lane Lr. In this case, according to the matrix M2, when there is no target object in the lane L (see Figs. 7A and 7B), when the target object is located in zone Z3 (see Figs. 8A and 8B), and when the target object is located in zone Z2 (see Figs. 9A and 9B), the CPU maintains the azimuth angle θr at the value of the previous cycle ("no change"). 7A to 9B, the azimuth angle θr is illustrated assuming that the value of the azimuth angle θr when the value of the previous cycle is maintained is equal to the value of the azimuth angle θl in each figure. On the other hand, according to the matrix M2, when the target object in the own lane L is located in zone 1 (see FIGS. 10A and 10B), the CPU changes the azimuth angle θr to angle -θw ("wide"). For this reason, in FIGS. 10A and 10B, the azimuth angle θr is illustrated so that the azimuth angle θr is angle -θw.

[0103] 1. When there is no target object in the own lane L (FIGS. 7A and 7B) 1-1. When the target in the left lane Ll is another vehicle Vl1 The other vehicle Vl1 is located in zone Z1. Therefore, the CPU changes the azimuth angle θl to angle θn ("narrow") with reference to the matrix M1. As shown in FIG. 7A, in this case, since features 2n and 3n hold, the other vehicle Vl1 can be appropriately displayed in the composite image. In contrast, when the azimuth angle θl is changed to angle θw, features 2w and 3w hold (see FIG. 7B), and the other vehicle Vl1 cannot be appropriately displayed in the composite image due to the cut-off phenomenon. Therefore, in the case of 1-1, by changing the azimuth angle θl to angle θn, the object in the left lane Ll can be appropriately displayed in the composite image.

[0104] 1-2. When the target in the left lane Ll is another vehicle Vl2 The other vehicle Vl2 is located in zone Z2. Therefore, the CPU refers to the matrix M1 and maintains the azimuth angle θl at the value of the previous period ("no change"). If the azimuth angle θl in the previous period was angle θn, the CPU maintains θl=θn in the current period. In this case, as shown in FIG. 7A, the features 5n and 7n are established, so the other vehicle Vl2 can be appropriately displayed in the composite image. On the other hand, if the azimuth angle θl in the previous period was angle θw, the CPU maintains θl=θw in the current period. In this case, as shown in FIG. 7B, the features 5w and 7w are established, so the other vehicle Vl2 can be appropriately displayed in the composite image. That is, in the case of 1-2, no matter which angle (θn or θw) the azimuth angle θl is changed to, the object in the left lane Ll can be appropriately displayed in the composite image.

[0105] 1-3. When the target in the left lane Ll is another vehicle Vl3 The other vehicle Vl3 is located in zone Z3. Therefore, the CPU changes the azimuth angle θl to angle θw ("wide") with reference to the matrix M1. As shown in FIG. 7B, in this case, the features 8w and 10w hold, so the other vehicle Vl3 can be properly displayed in the composite image. In contrast, when the azimuth angle θl is changed to angle θn, the features 8n and 10n hold (see FIG. 7A), so the other vehicle Vl3 cannot be properly displayed in the composite image due to the doubling phenomenon. Therefore, in the case of 1-3, by changing the azimuth angle θl to angle θw, the object in the left lane Ll can be properly displayed in the composite image.

[0106] 1-4. When there is no target in the left lane Ll In this case, since there is no target object in either the own lane L or the left lane Ll, the above-mentioned phenomena (cut-off, disappearance, doubling, and hiding phenomena) cannot occur no matter what angle the azimuth angle θl is changed to.

[0107] 2. When the object (another vehicle Vre) in the own lane L is located in zone Z3 (FIGS. 8A and 8B) 2-1. When the target in the left lane Ll is another vehicle Vl1 The other vehicle Vl1 is located in zone Z1. Therefore, the CPU changes the azimuth angle θl to angle θn ("narrow") by referring to the matrix M1. As shown in FIG. 8A, in this case, since the features 8n and 9n are satisfied for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image. Also, since the features 2n and 3n are satisfied for the other vehicle Vl1, the other vehicle Vl1 can be appropriately displayed in the composite image. In contrast, when the azimuth angle θl is changed to the angle θw (see FIG. 8B), the features 8w and 9w are satisfied for the other vehicle Vre, so the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, the features 2w and 3w are satisfied for the other vehicle Vl1, so the other vehicle Vl1 cannot be appropriately displayed in the composite image due to the cut-off phenomenon. Therefore, in the case of 2-1, by changing the azimuth angle θl to the angle θn, the objects in the own lane L and the left lane Ll can be appropriately displayed in the composite image.

[0108] 2-2. When the target in the left lane Ll is another vehicle Vl2 The other vehicle Vl2 is located in zone Z2. Therefore, the CPU refers to the matrix M1 and maintains the azimuth angle θl at the value of the previous cycle ("no change"). If the azimuth angle θl in the previous cycle was angle θn, the CPU maintains θl = θn in the current cycle. In this case, as shown in FIG. 8A, since features 8n and 9n hold for the other vehicle Re, the other vehicle Vre can be appropriately displayed in the composite image. Furthermore, since features 5n and 7n hold for the other vehicle Vl2, the other vehicle Vl2 can be appropriately displayed in the composite image. On the other hand, if the azimuth angle θl in the previous cycle was angle θw, the CPU maintains θl=θw in the current cycle. In this case, as shown in FIG. 8B, the features 8w and 9w are satisfied for the other vehicle Vre, so the other vehicle Vre can be appropriately displayed in the composite image. Also, the features 5w and 7w are satisfied for the other vehicle Vl2, so the other vehicle Vl2 can be appropriately displayed in the composite image. Therefore, in the case of 2-2, no matter which angle (θn or θw) the azimuth angle θl is changed to, the objects in the own lane L and the left lane Ll can be appropriately displayed in the composite image.

[0109] 2-3. When the target in the left lane Ll is another vehicle Vl3 The other vehicle Vl3 is located in zone Z3. Therefore, the CPU changes the azimuth angle θl to angle θw ("wide") by referring to the matrix M1. As shown in FIG. 8B, in this case, the features 8w and 9w hold for the other vehicle Vre, so the other vehicle Vre can be appropriately displayed in the composite image. Also, the features 8w and 10w hold for the other vehicle Vl3, so the other vehicle Vl3 can be appropriately displayed in the composite image. In contrast, when the azimuth angle θl is changed to the angle θn (see FIG. 8A), the features 8n and 9n are satisfied for the other vehicle Vre, so that the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, the features 8n and 10n are satisfied for the other vehicle Vl3, so that the other vehicle Vl3 cannot be appropriately displayed in the composite image due to the doubling phenomenon. Therefore, in the case of 2-3, by changing the azimuth angle θl to the angle θw, the objects in the own lane L and the left lane Ll can be appropriately displayed in the composite image.

[0110] 2-4. When there is no target in the left lane Ll In this case, the CPU refers to the matrix M1 and maintains the azimuth angle θl at the value of the previous period ("no change"). If the azimuth angle θl in the previous period was angle θn, the CPU maintains θl=θn in the current period. In this case, as shown in FIG. 8A, the features 8n and 9n are established for the other vehicle Re, so that the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, if the azimuth angle θl in the previous period is angle θw, the CPU maintains θl=θw in the current period. In this case, as shown in FIG. 8B, the feature 8w and the feature 9w are established for the other vehicle Vre, so that the other vehicle Vre can be appropriately displayed in the composite image. Therefore, in the case of 2-4, even if the azimuth angle θl is changed to any angle (θn or θw), the object in the own lane L can be appropriately displayed in the composite image.

[0111] 3. When the object (another vehicle Vre) in the own lane L is located in zone Z2 (FIGS. 9A and 9B) 3-1. When the target in the left lane Ll is another vehicle Vl1 The other vehicle Vl1 is located in zone Z1. Therefore, the CPU changes the azimuth angle θl to angle θn ("narrow") by referring to the matrix M1. As shown in FIG. 9A, in this case, since the features 4n and 6n are satisfied for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image. Also, since the features 2n and 3n are satisfied for the other vehicle Vl1, the other vehicle Vl1 can be appropriately displayed in the composite image. In contrast, when the azimuth angle θl is changed to the angle θw (see FIG. 9B), the features 4w and 6w are satisfied for the other vehicle Vre, so the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, the features 2w and 3w are satisfied for the other vehicle Vl1, so the other vehicle Vl1 cannot be appropriately displayed in the composite image due to the cut-off phenomenon. Therefore, in the case of 3-1, by changing the azimuth angle θl to the angle θn, the objects in the own lane L and the left lane Ll can be properly displayed in the composite image.

[0112] 3-2. When the target in the left lane Ll is another vehicle Vl2 The other vehicle Vl2 is located in zone Z2. Therefore, the CPU refers to the matrix M1 and maintains the azimuth angle θl at the value of the previous cycle ("no change"). If the azimuth angle θl in the previous cycle was angle θn, the CPU maintains θl = θn in the current cycle. In this case, as shown in FIG. 9A, since features 4n and 6n hold for the other vehicle Re, the other vehicle Vre can be appropriately displayed in the composite image. Furthermore, since features 5n and 7n hold for the other vehicle Vl2, the other vehicle Vl2 can be appropriately displayed in the composite image. On the other hand, if the azimuth angle θl in the previous cycle was angle θw, the CPU maintains θl=θw in the current cycle. In this case, as shown in FIG. 9B, since the features 4w and 6w are satisfied for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image. Also, since the features 5w and 7w are satisfied for the other vehicle Vl2, the other vehicle Vl2 can be appropriately displayed in the composite image. Therefore, in the case of 3-2, no matter which angle (θn or θw) the azimuth angle θl is changed to, the objects in the own lane L and the left lane Ll can be appropriately displayed in the composite image.

[0113] 3-3. When the target in the left lane Ll is another vehicle Vl3 The other vehicle Vl3 is located in zone Z3. Therefore, the CPU changes the azimuth angle θl to angle θn ("narrow") by referring to the matrix M1. As shown in FIG. 9A, in this case, since the features 4n and 6n are satisfied for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, since the features 8n and 10n are satisfied for the other vehicle Vl3, the other vehicle Vl3 cannot be appropriately displayed in the composite image due to the doubling phenomenon. In contrast, when the azimuth angle θl is changed to the angle θw (see FIG. 9B), the features 4w and 6w are satisfied for the other vehicle Vre, and therefore the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, although the features 8w and 10w are satisfied for the other vehicle Vl3, since the other vehicle Vl3 is located diagonally to the left rear of the other vehicle Vre, depending on the positional relationship between the two, a concealment phenomenon may occur. Generally, the doubling phenomenon is a problem that reduces the quality of the composite image, whereas the hiding phenomenon is a problem that may cause a decrease in the driving safety of the vehicle. For this reason, in 3-3, the improvement of the hiding phenomenon is prioritized over the improvement of the doubling phenomenon. According to this configuration, in the case of 3-3, by changing the azimuth angle θl to the angle θn, the target objects in the own lane L and the left lane Ll can be properly displayed in the composite image.

[0114] 3-4. When there is no target in the left lane Ll In this case, the CPU refers to the matrix M1 and maintains the azimuth angle θl at the value of the previous period ("no change"). If the azimuth angle θl in the previous period was angle θn, the CPU maintains θl=θn in the current period. In this case, as shown in FIG. 9A, the features 4n and 6n are established for the other vehicle Re, so that the other vehicle Vre can be appropriately displayed in the composite image. On the other hand, if the azimuth angle θl in the previous period was angle θw, the CPU maintains θl=θw in the current period. In this case, as shown in FIG 9B, the feature 4w and the feature 6w are established for the other vehicle Vre, so that the other vehicle Vre can be appropriately displayed in the composite image. Therefore, in the case of 3-4, even if the azimuth angle θl is changed to any angle (θn or θw), the object in the own lane L can be appropriately displayed in the composite image.

[0115] Among the 16 components of matrix M1, the "no change" components may be set in advance to either "narrow" or "wide". The same applies to matrix M2. With this configuration, the frequency of switching the composite image increases, but the control content can be simplified.

[0116] 4. When the object (another vehicle Vre) in the own lane L is located in zone Z1 (FIGS. 10A and 10B) 4-1. When the target in the left lane Ll is another vehicle Vl1 The other vehicle Vl1 is located in zone Z1. Therefore, the CPU changes the azimuth angle θl to angle θn ("narrow") by referring to the matrix M1. As shown in FIG. 10A, in this case, since the features 1n and 3n are satisfied for the other vehicle Vre, the other vehicle Vre cannot be properly displayed in the composite image due to the cut-off phenomenon. On the other hand, since the features 2n and 3n are satisfied for the other vehicle Vl1, the other vehicle Vl1 can be properly displayed in the composite image. In contrast, when the azimuth angle θl is changed to the angle θw (see FIG. 10B), the features 1w and 3w are satisfied for the other vehicle Vre, so that the other vehicle Vre can be appropriately displayed in the composite image unless the other vehicle Vre is excessively close to the host vehicle V. On the other hand, the features 2w and 3w are satisfied for the other vehicle Vl1, so that the other vehicle Vl1 cannot be appropriately displayed in the composite image due to the cut-off phenomenon. Depending on the size of the blind spot area Rb, the disappearance phenomenon may occur. In addition, when the object target in the left lane Ll is a motorcycle, the disappearance phenomenon is more likely to occur.

[0117] According to feature 1n, when the cut-off phenomenon occurs in the other vehicle Vre (i.e., the object in the own lane L), there is a high possibility that the left front corner of the other vehicle Vre (i.e., the part where the left turn signal lamp is installed) is located in the blind spot area Rb. For this reason, the left turn signal lamp of the other vehicle Vre is not displayed in the composite image, and the driver of the own vehicle V cannot determine from the composite image whether the other vehicle Vre is about to change lanes. Therefore, if the other vehicle Vre changes lanes to the left lane Ll at the same time that the own vehicle V changes lanes to the left lane Ll, the other vehicle Vre may come too close to or come into contact with the own vehicle V depending on the trajectory of both vehicles, which may reduce the driving safety of the own vehicle V.

[0118] On the other hand, if the other vehicle Vl1 is cut off or disappears, the driver may not be able to properly recognize the presence of the other vehicle Vl1 from the composite image. If the host vehicle V changes lanes to the left lane Ll without noticing the presence of the other vehicle Vl1, the host vehicle V may come into contact with the other vehicle Vl1 traveling straight in the left lane Ll, which may reduce the driving safety of the host vehicle V.

[0119] Here, according to the feature 1n, the blind spot area Rb is not so large, so the possibility of the disappearance phenomenon occurring in the other vehicle Vre is extremely low. That is, although the cut-off phenomenon occurs in the other vehicle Vre, the driver can recognize the presence of the other vehicle Vre itself from the synthetic image. In contrast, when the disappearance phenomenon occurs in the other vehicle Vl1, the driver cannot recognize the presence of the other vehicle Vl1 from the synthetic image. For this reason, the cut-off or disappearance phenomenon of the object target (other vehicle Vl1) in the left lane Ll is more likely to affect the running of the host vehicle V than the cut-off phenomenon of the object target (other vehicle Vre) in the host lane L. Therefore, in 4-1, the improvement of the cut-off or disappearance phenomenon of the object target in the left lane Ll is prioritized over the improvement of the cut-off phenomenon of the object target in the host lane L. According to this configuration, in the case of 4-1, the object targets in the host lane L and the left lane Ll can be appropriately displayed in the synthetic image by changing the azimuth angle θl to the angle θn.

[0120] 4-2. When the target in the left lane Ll is another vehicle Vl2 The other vehicle Vl2 is located in zone Z2. Therefore, the CPU changes the azimuth angle θl to angle θw ("wide") by referring to the matrix M1. As shown in FIG. 10B, in this case, since the features 1w and 3w are established for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image as long as the other vehicle Vre is not excessively close to the host vehicle V. On the other hand, although the features 5w and 7w are established for the other vehicle Vl2, since the other vehicle Vl2 is located diagonally to the left rear of the other vehicle Vre, depending on the positional relationship between the two, a hiding phenomenon may occur. In contrast, when the azimuth angle θl is changed to the angle θn (see FIG. 10A), the features 1n and 3n are satisfied for the other vehicle Vre, and therefore the other vehicle Vre cannot be appropriately displayed in the composite image due to the cut-off phenomenon. On the other hand, the features 5n and 7n are satisfied for the other vehicle Vl2, and therefore the other vehicle Vl2 can be appropriately displayed in the composite image.

[0121] As in the case of 4-1 above, when the cut-off phenomenon occurs in the other vehicle Vre, the driver cannot determine from the composite image whether the other vehicle Vre is about to change lanes. This may result in a decrease in the driving safety of the host vehicle V. In general, the cut-off phenomenon is more likely to affect the traveling of the vehicle V than the obscuration phenomenon. For this reason, in 4-2, improvement of the cut-off phenomenon is prioritized over improvement of the obscuration phenomenon. According to this configuration, in the case of 4-2, by changing the azimuth angle θl to the angle θw, the target objects in the vehicle lane L and the left lane Ll can be appropriately displayed in the composite image.

[0122] 4-3. When the target in the left lane Ll is another vehicle Vl3 The other vehicle Vl3 is located in zone Z3. Therefore, the CPU changes the azimuth angle θl to angle θw ("wide") by referring to the matrix M1. As shown in FIG. 10B, in this case, since the features 1w and 3w are established for the other vehicle Vre, the other vehicle Vre can be appropriately displayed in the composite image as long as the other vehicle Vre is not excessively close to the host vehicle V. On the other hand, although the features 8w and 10w are established for the other vehicle Vl3, since the other vehicle Vl3 is located diagonally to the left rear of the other vehicle Vre, depending on the positional relationship between the two, a hiding phenomenon may occur. In contrast, when the azimuth angle θl is changed to the angle θn (see FIG. 10A), the features 1n and 3n are satisfied for the other vehicle Vre, so the other vehicle Vre cannot be appropriately displayed in the composite image due to the cut-off phenomenon. Also, the features 8n and 10n are satisfied for the other vehicle Vl3, so the other vehicle Vl3 cannot be appropriately displayed in the composite image due to the doubling phenomenon.

[0123] As in the case of 4-1 above, when the cut-off phenomenon occurs in the other vehicle Vre, the driver cannot determine from the composite image whether the other vehicle Vre is about to change lanes. This may result in a decrease in the driving safety of the host vehicle V. As described above, the cut-off phenomenon is more likely to affect the traveling of the vehicle V than the obscuration phenomenon. For this reason, in 4-3, improvement of the cut-off phenomenon is prioritized over improvement of the obscuration phenomenon. According to this configuration, in the case of 4-3, by changing the azimuth angle θl to the angle θw, the target objects in the vehicle lane L and the left lane Ll can be appropriately displayed in the composite image.

[0124] 4-4. When there is no target in the left lane Ll In this case, the CPU changes the azimuth angle θl to the angle θw ("wide") by referring to the matrix M1. As shown in Fig. 10B, in this case, the features 1w and 3w are established for the other vehicle Vre, so that the other vehicle Vre can be appropriately displayed in the composite image as long as the other vehicle Vre is not excessively close to the host vehicle V. In contrast, when the azimuth angle θl is changed to the angle θn (see FIG. 10A), the features 1n and 3n are established for the other vehicle Vre, and therefore the other vehicle Vre cannot be properly displayed in the composite image due to the cut-off phenomenon. Therefore, in the case of 4-4, by changing the azimuth angle θl to the angle θw, the object in the own lane L can be properly displayed in the composite image.

[0125] As described above, in the image control system 1, when generating a composite image, the magnitude of the azimuth angle θl (or the azimuth angle θr) is changed based on a combination of "the presence or absence of an object target in the own lane L and the distance to the object target" and "the presence or absence of an object target in the left lane Ll (or the right lane Lr) and the distance to the object target". Then, when the magnitude of the azimuth angle θl (or the azimuth angle θr) is changed, the magnitude of the azimuth angle θli (or the azimuth angle θri) is changed accordingly. By appropriately changing the magnitude of the azimuth angle θl (or the azimuth angle θr) and the magnitude of the azimuth angle θli (or the azimuth angle θri) independently based on this combination, it is possible to appropriately display objects that are relatively likely to affect the traveling of the vehicle (i.e., objects in the own lane L and objects in the left lane Ll (or the right lane Lr)) in the composite image.

[0126] Furthermore, the azimuth angle θl (or the azimuth angle θr) and the azimuth angle θli (or the azimuth angle θri) are changed based only on information about the object in the own lane L and the object in the left lane Ll (or the object in the right lane Lr), and information about other objects is not taken into consideration. Therefore, it is possible to suppress an increase in the processing load required for generating a composite image.

[0127] In particular, in this embodiment, the azimuth angle θl (or the azimuth angle θr) can be changed between two angles, that is, the angle θn (or -θn) and the angle θw (or -θw). Therefore, with a relatively simple configuration, the objects in the lanes L, Ll, and Lr can be appropriately displayed in the composite image.

[0128] Although the image control system according to the present embodiment has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the object of the present invention.

[0129] For example, as shown in Fig. 14A, the azimuth angle θl may be determined by referring to a matrix M3 as a modified example instead of the matrix M1. Similarly, as shown in Fig. 14B, the azimuth angle θr may be determined by referring to a matrix M4 as a modified example instead of the matrix M2. The matrix M3 (or M4) differs from the matrix M1 (or M2) in the following points. Among the elements of matrix M3 (or M4), the elements that correspond to the "unchanged" elements of matrix M1 (or M2) are set to "narrow". Among the components of matrix M3 (or M4), the component when there is no target in the own lane L and the target in the left lane Ll (or right lane Lr) is located in zone Z3 (hereinafter referred to as "component C1") is set to "narrow." Among the components of matrix M3 (or M4), the component when both the object in the own lane L and the object in the left lane Ll (or right lane Lr) are located in zone Z3 (hereinafter referred to as "component C2") is set to "narrow."

[0130] According to this configuration, since 13 of the 16 components of the matrix M3 (or M4) are set to "narrow", the frequency of changing the azimuth angle θl (or θr) (i.e., the frequency of the movement of the joint position of the composite image) is significantly reduced. This reduces the frequency of switching the composite image, so that the possibility that the driver feels annoyed due to frequent switching can be reduced. In addition, the frequency of the driver noticing the switching and turning his / her eyes to the display 12a can be reduced. On the other hand, when the components C1 and C2 are set to "narrow", a double phenomenon occurs in the object target in the left lane Ll (see FIG. 7A and FIG. 8A). However, as described above, the double phenomenon only deteriorates the appearance of the composite image, so that the traveling of the host vehicle V is not affected much. That is, in this configuration, "reducing the frequency of switching the composite image" is prioritized over "the occurrence of the double phenomenon" (the occurrence of the double phenomenon is allowed).

[0131] According to this configuration, even if the ECU 10 can only calculate the distance to a relatively nearby target (typically, a target located in zone Z1) (in other words, the calculation limit distance of the ECU 10 includes only zone Z1), the azimuth angle θl (or θr) can be determined according to the matrix M3 (or M4). That is, the ECU 10 is configured to change the azimuth angle θl (or θr) to an angle θw (-θw) ("wide") only when the target in the own lane L is located within the calculation limit distance and the target in the left lane Ll (or right lane Lr) is not located within the calculation limit distance (including the case where the target does not exist), and to change the azimuth angle θl (or θr) to an angle θn (-θn) ("narrow") in other cases. This allows the present invention to be applied to an image control system in which the calculation limit distance of the ECU 10 does not cover zone Z3.

[0132] Furthermore, the ECU 10 may be configured to change the azimuth angle θl (or θr) and the azimuth angle θli (or θri) continuously. With this configuration, the movement of the joints of the composite images becomes smooth, so that the composite images can be switched smoothly (naturally), and the possibility that the driver will feel annoyed by the way the composite images are switched can be reduced.

[0133] Furthermore, when the target object is traveling near the boundary between zones Z1 and Z2, or near the boundary between zones Z2 and Z3, if the position of the target object frequently changes between zones Z1 and Z2, or between zones Z2 and Z3, the azimuth angle θl (or θr) may frequently change between angles θn and θw (or between -θn and -θw). In this case, the synthetic image may be frequently switched (chattering occurs), and the driver may find the switching of the synthetic image annoying. Therefore, the value of the distance D1 when the target object enters zone Z1 from zone Z2 may be set to D1in (for example, 15 [m]), and the value of the distance D1 when the target object enters zone Z2 from zone Z1 may be set to D1out (for example, 17 [m]) larger than D1in. Similarly, the value of the distance D2 when the target enters zone Z2 from zone Z3 may be set to D2in (e.g., 30 [m]), and the value of the distance D2 when the target enters zone Z2 from zone Z3 may be set to D2out (e.g., 34 [m]) larger than D2in. That is, hysteresis may be introduced into the distances D1 and D2. This can suppress the occurrence of chattering. Alternatively, when the composite image is switched with a change in the azimuth angle θl (or θr), the ECU 10 may be configured not to change the azimuth angle θl (or θr) (i.e., not to switch the composite image) until a predetermined period (>T) has elapsed, even if a condition for changing the azimuth angle θl (or θr) is satisfied during the period. This configuration can also suppress the occurrence of chattering.

[0134] Furthermore, the image control system 1 may include a distance measuring sensor such as a radar, an ultrasonic sensor, or a laser radar, which improves the accuracy of calculating the distance to the target.

[0135] Furthermore, the ECU 10 may be configured to display an overhead view (shown in FIG. 5A or FIG. 5B) showing the current effective ranges Rrea, Rla, and Rra on a part of the display 12a. This may be performed all the time while the ignition switch is on, or may be performed only when switching the composite image.

[0136] Furthermore, the ECU 10 may be configured not to display the composite image on the display 12a when the host vehicle V is stopped or traveling at a low speed, but to display the composite image on the display 12a only when the host vehicle V is traveling at a relatively medium or high speed. When the host vehicle V is stopped or traveling at a low speed, this typically occurs when the host vehicle V is waiting at a traffic light or caught in traffic, and in such cases, the possibility that the objects in the lanes L, Ll, and Lr will affect the traveling of the host vehicle V is relatively low (i.e., there is not much need to display the composite image). In this case, ECU 10 may determine the azimuth angle θl (or θr) by referring to a matrix (not shown) in which the leftmost column of matrix M1 (or M2) (i.e., the column indicating the component when the object in the own lane L is located in zone Z1) has been deleted. According to this configuration, the configuration of the image control system can be simplified, and costs can be reduced. [Explanation of symbols]

[0137] 10: Image control ECU, 11Re: Rear camera, 11L: Left rear camera, 11R: Right rear camera, 12: Display device, 12a: Display sensor, 21, 22, 23: Straight line, 30la: Boundary line (rear left boundary line), 30ra: Boundary line (rear right boundary line), 40ia: Boundary line (left rear inner boundary line), 40oa: Boundary line, 50ia: Boundary line (right rear inner boundary line), 50oa: Boundary line

Claims

1. A rear imaging device that captures images of targets and lane markings present in a rear imaging range extending behind the vehicle; a left rear imaging device that captures an image of a target and a lane marking that are present in a left rear imaging range that extends to the left rear of the vehicle and partially overlaps with the rear imaging range; a right rear imaging device that captures an image of a target and a lane marking that are present in a right rear imaging range that extends to the right rear of the vehicle and partially overlaps with the rear imaging range; a display device having a display screen; Obtaining rear image data obtained by imaging using the rear imaging device, left rear image data obtained by imaging using the left rear imaging device, and right rear image data obtained by imaging using the right rear imaging device, A composite image in a panoramic format is generated based on the image data. displaying the composite image on the display screen of the display device; An image control device configured as above; In an image control system comprising: Among the rear imaging range, the left rear imaging range, and the right rear imaging range, imaging ranges used for generating the composite image are defined as a rear effective range, a left rear effective range, and a right rear effective range, respectively; among boundaries defining a horizontal angle of view of the rear effective range when an orientation of an optical axis of the rear image capture device is defined as a rear reference orientation, angles of a rear left boundary line, which is a boundary line on the left side of the vehicle, and a rear right boundary line, which is a boundary line on the right side of the vehicle, from the rear reference orientation are defined as a rear left azimuth angle and a rear right azimuth angle, respectively; a left rear inner azimuth angle is defined as an angle of a left rear inner boundary line, which is a boundary line on the inside in a vehicle width direction of the vehicle, from the left rear reference azimuth when the azimuth of the optical axis of the left rear image capture device is defined as a left rear reference azimuth angle; When the orientation of the optical axis of the right rear image capture device is defined as a right rear reference orientation, the angle of a right rear inner boundary line, which is a boundary line on the inner side in the vehicle width direction of the vehicle, from the right rear reference orientation is defined as a right rear inner azimuth angle. The image control device includes: the rear effective range can be changed by changing the magnitudes of the rear left azimuth angle and the rear right azimuth angle independently of each other within a range of a horizontal angle of view of the rear imaging device, the left rear effective range can be changed by changing the magnitude of the left rear inner azimuth angle within a range of a horizontal angle of view of the left rear imaging device, the right rear effective range can be changed by changing the magnitude of the right rear inner azimuth angle within a range of a horizontal angle of view of the right rear imaging device, Calculating the positions and shapes of a lane in which the vehicle is located, a left lane adjacent to the lane on the left side thereof, and a right lane adjacent to the lane on the right side thereof, based on the lane markings detected from the image data; When generating the composite image, Among the targets located rearward of the rear end portion of the vehicle detected from the image data, a rear target, a left rear target, and a right rear target are identified as targets that are closest to the vehicle in the own lane, the left lane, and the right lane, respectively; changing the magnitude of the rear left azimuth angle based on a combination of the presence or absence of the rear object target and the degree of distance to the rear object target, and the presence or absence of the left rear object target and the degree of distance to the left rear object target; changing the magnitude of the rear right azimuth angle based on a combination of the presence or absence of the rear object target and the degree of distance to the rear object target, and the presence or absence of the right rear object target and the degree of distance to the right rear object target; generating the composite image by changing the magnitude of the left rear inner azimuth angle or the right rear inner azimuth angle to a relatively large value when the magnitude of the left rear azimuth angle or the right rear inner azimuth angle is changed to a relatively small value, generating the composite image by changing the magnitude of the left rear inner azimuth angle or the right rear inner azimuth angle to a relatively small value when the magnitude of the left rear azimuth angle or the right rear inner azimuth angle is changed to a relatively large value, It was configured as follows: Image control system.

2. 2. The image control system according to claim 1, The image control device includes: the rear left azimuth angle is variable between a first rear left angle and a second rear left angle greater in magnitude than the first rear left angle; the rear right azimuth angle is variable between a first rear right angle and a second rear right angle that is greater in magnitude than the first rear right angle; the left rear inner azimuth angle is variable between a first left rear inner angle and a second left rear inner angle greater in magnitude than the first left rear inner angle; the right rear inner azimuth angle is variable between a first right rear inner angle and a second right rear inner angle having a magnitude greater than the first right rear inner angle; It was configured as follows: Image control system.

3. 3. The image control system according to claim 1, The image control device includes: a virtual projection plane perpendicular to the ground contact surface of the vehicle is set at a position spaced rearward from the rear end of the vehicle by a predetermined projection distance determined as a fixed value; generating, as the composite image, images obtained by projecting, onto a projection surface, the targets and the lane markings that are present in the rear effective range, the left rear effective range, and the right rear effective range, based on a viewpoint that is virtually set on or near the vehicle; When generating the composite image, changing the rear left azimuth, the left rear inner azimuth, the rear right azimuth, and the right rear inner azimuth so that a left intersection point, which is an intersection point between the rear left boundary line and the left rear inner boundary line, and a right intersection point, which is an intersection point between the rear right boundary line and the right rear inner boundary line, are located on the projection plane in a plan view of the vehicle; It was configured as follows: Image control system.

4. 4. The image control system according to claim 3, When a region between a first straight line passing through the rear end of the vehicle and extending in the vehicle width direction of the vehicle, and a second straight line spaced a predetermined first distance rearward from the first straight line and parallel to the first straight line is defined as a first zone, a region between the second straight line and a third straight line spaced a predetermined second distance rearward from the first straight line and parallel to the first straight line is defined as a second zone, and a region rear of the third straight line is defined as a third zone, the projection distance is greater than the first distance and less than the second distance; If the intersections of a left imaginary line and a right imaginary line extending rearward from the left rear corner and the right rear corner of the vehicle in parallel with the front-rear axis of the vehicle and the projection plane are defined as a first intersection and a second intersection, respectively, when changing the magnitude of the rear left azimuth angle or the rear right azimuth angle to the relatively small value, in a plan view of the vehicle, the left intersection point or the right intersection point is located on or in the vicinity of the first intersection point, or on or in the vicinity of the second intersection point, When changing the magnitude of the rear left azimuth angle or the rear right azimuth angle to the relatively large value, the left intersection point or the right intersection point is located at a position that is apart from the first intersection point or the second intersection point by a predetermined distance that is equal to or greater than a typical lane width, on the outer side in a vehicle width direction of the vehicle. Image control system.

5. 5. The image control system according to claim 4, the projection distance is less than half the sum of the first distance and the second distance; Image control system.

6. 6. The image control system according to claim 4, The image control device includes: When the rear object is located in the first zone and the left rear object is located behind the first zone, the magnitude of the rear left azimuth angle is changed to the relatively large value; when the rear object is located in the first zone and the right rear object is located behind the first zone, the magnitude of the rear right azimuth angle is changed to the relatively large value; It was configured as follows: Image control system.

7. 7. The image control system according to claim 4, The image control device includes: When the rear target is located in the first zone and the left rear target does not exist, the magnitude of the rear left azimuth angle is changed to the relatively large value; when the rear target is located in the first zone and the right rear target does not exist, changing the magnitude of the rear right azimuth angle to the relatively large value; It was configured as follows: Image control system.

8. The image control system according to any one of claims 4 to 7, The image control device includes: When the rear target and the left rear target are both located in the first zone, the magnitude of the rear left azimuth angle is changed to the relatively small value; when the rear target and the right rear target are both located in the first zone, the magnitude of the rear right azimuth angle is changed to the relatively small value; It was configured as follows: Image control system.

9. The image control system according to any one of claims 4 to 8, The image control device includes: When the rear target and the left rear target are both located in the third zone, the magnitude of the rear left azimuth angle is changed to the relatively large value; When the rear target and the right rear target are both located in the third zone, the magnitude of the rear right azimuth angle is changed to the relatively large value; When the rear target does not exist and the left rear target is located in the third zone, the magnitude of the rear left azimuth angle is changed to the relatively large value; when the rear target does not exist and the right rear target is located in the third zone, changing the magnitude of the rear right azimuth angle to the relatively large value; It was configured as follows: Image control system.

10. The image control system according to any one of claims 1 to 9, The image control device includes: In the case of generating the composite image, When changing the rear left azimuth angle and the left rear inner azimuth angle, these azimuth angles are changed continuously, When changing the rear right azimuth angle and the right rear inner azimuth angle, these azimuth angles are changed continuously. It was configured as follows: Image control system.

Citation Information

Patent Citations

  • Vehicle surrounding condition display device

    JP1998257482A

  • Image display control device

    JP2016189576A

  • Visual recognition device for vehicle and visual recognition image display method for vehicle

    JP2018074503A

  • Image display device

    JP2019118016A

  • Display system and method thereof

    US20140277940A1