Travel assistance device

Abstract

The present application provides a travel support device. A processor of the driving support device divides a region corresponding to an object detectable region in a peripheral image G into a plurality of divisions, assigns a standard value to transparency of an object detection image that is superimposed on a first division of the plurality of divisions in a case where the object detection image is not superimposed on all divisions adjacent to the first division, and assigns a value higher than the standard value to transparency of the object detection image in a case where the object detection image is superimposed on the first division and at least one division adjacent to the first division.

Description

Technical Field

[0001] The present invention relates to a driving support device, which includes a camera for capturing images of the surrounding area of ​​the vehicle and obtaining surrounding images, and an object detection sensor for obtaining information related to the position of a three-dimensional object existing in the surrounding area, and has the function of overlaying the information related to the position of the three-dimensional object obtained by the object detection sensor onto the surrounding images. Background Technology

[0002] A driving support device is proposed, which includes a camera for capturing images of the surrounding area of ​​the vehicle and obtaining a surrounding image (G), and an object detection sensor for obtaining the position of a three-dimensional object existing in the surrounding area. The device also has the function of overlaying information related to the position of the three-dimensional object obtained by the object detection sensor onto the surrounding image (see, for example, Patent Document 1 below). In this driving support device (hereinafter referred to as the "conventional device"), when no three-dimensional object is detected by the object detection sensor, it has the function of overlaying an image representing the detectable range of the object detection sensor onto the surrounding image.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2008-186132 Summary of the Invention

[0004] Furthermore, in order to draw the driver's attention to three-dimensional objects located near the vehicle, driving support devices are sometimes configured by overlaying an image representing the detection results of an object detection sensor (an object detection image) onto a predetermined area of ​​a surrounding image (the area corresponding to the position (or orientation) of the three-dimensional object obtained by the object detection sensor). For example, a predetermined area in the surrounding image indicating the direction in which the three-dimensional object exists is colored "yellow" or "red." In this case, a portion of the surrounding image may be obscured by the object detection image, so there is a possibility that the driver may have difficulty visually identifying that portion of the image in the surrounding image (a photograph taken by a camera (e.g., a photograph of the three-dimensional object itself and white lines drawn on the surrounding road surface)).

[0005] One of the objectives of this invention is to provide a driving support device that has the function of overlaying information obtained by an object detection sensor onto a surrounding image representing the vicinity of the vehicle, and has the function of displaying the information in an appropriate manner corresponding to the situation.

[0006] To achieve the above objectives, the driving support device (1) of the present invention comprises:

[0007] The vehicle-mounted sensor (20) includes a camera (22) for acquiring a peripheral image (G) of the surrounding area of ​​the vehicle and an object detection sensor (21) for acquiring information relating to the position of a three-dimensional object in a predetermined object-detectable area (D) within the surrounding area.

[0008] Display device (30) displays images; and

[0009] The processor (10) is configured to control the display device in such a way that it displays an object detection image (S) representing the detection result of the three-dimensional object, which overlaps with the area corresponding to the position of the three-dimensional object detected by the object detection sensor in the entire area of ​​the peripheral image.

[0010] The processor is configured as follows:

[0011] The region (R) in the surrounding image corresponding to the detectable area of ​​the object is divided into multiple regions (Bk).

[0012] In the case where the object detection image is displayed overlapping in the first zone of the plurality of zones, and the object detection image is not displayed overlapping in all zones adjacent to the first zone, a standard value (T0) is assigned to the transparency (T) of the object detection image displayed overlapping in the first zone.

[0013] When the object detection image is displayed in the first zone and at least one zone adjacent to the first zone, the transparency of these object detection images is assigned a value (T1) that is higher than the standard value.

[0014] When a three-dimensional object is detected by an object detection sensor, and the object detection image, as a result of the detection, is overlaid on a surrounding image, there is a possibility that the driver may have difficulty visually identifying that part of the surrounding image (a portion of the surrounding image) because it is partially covered by the object detection image. Furthermore, if the transparency of the object detection images is uniformly increased to address this issue, there is a possibility that the driver may have difficulty recognizing the displayed object detection images. Therefore, the processor of the driving support device according to the present invention adjusts the transparency of each object detection image according to the situation when overlaying object detection images on a surrounding image. That is, the processor adjusts the transparency of the first region (a portion of the surrounding image) of the object detection image. Figure 2 In the example, zone B2) overlays the object detection image, and all zones adjacent to this first zone ( Figure 2 In the example where regions B1 and B3 in the image do not overlap, in the first case where the object detection image is displayed, a standard value (a value with relatively low transparency) is assigned to the transparency of the object detection image of the first region. On the other hand, the processor in the surrounding image of the first region ( Figure 3In the example, zone B2) overlays the object detection image, and at least one zone adjacent to the first zone ( Figure 3 In the example of zone B3) overlapping object detection images in the second condition, a higher value than the standard value is assigned to the transparency of these object detection images. In the first condition, because the transparency of the object detection images is relatively low, the object detection images are easily noticeable. That is, they can attract the driver's attention to three-dimensional objects in the direction of the object detection images. In this first condition, it is slightly difficult for the driver to visually identify the image of the inner part of the first zone in the surrounding image, but because its range is relatively narrow, the driver can predict the inner part of the first zone from the image around the first zone. In contrast, in the second condition, because the transparency of the object detection images is relatively high, the driver can easily visually identify the image of the part covered by these object detection images in the surrounding image. In this second condition, compared with the first condition, the object detection images are less noticeable, but because multiple object detection images are arranged adjacently (the area occupied by the object detection images in the surrounding image is relatively large), even if the transparency of these object detection images is relatively high, the driver can still recognize that the object detection images are displayed. As described above, according to the present invention, a driving support device is provided that has the function of overlaying information acquired by an object detection sensor onto a surrounding image representing the periphery of the vehicle and the function of displaying the information in an appropriate manner corresponding to the situation.

[0015] In one aspect of the driving support device according to the present invention,

[0016] The surrounding image is equivalent to a top view of the vehicle and its surrounding parts from directly above.

[0017] The multiple divisions are equivalent to dividing the annular area surrounding the vehicle in the peripheral image into multiple regions in the circumferential direction.

[0018] Therefore, the driver can easily grasp the positional relationship between the vehicle and the three-dimensional object (the direction from which the three-dimensional object is viewed from the vehicle).

[0019] In other aspects of the driving support device according to the present invention,

[0020] The processor is configured as follows:

[0021] When the object detection image is displayed overlapping in the first zone, overlapping in the second zone adjacent to one side of the first zone, and not overlapping in the third zone adjacent to the other side of the first zone, the transparency of these object detection images is assigned a first value higher than the standard value.

[0022] When the object detection image is displayed in the first overlapping region, the object detection image is displayed in the second overlapping region, and the object detection image is displayed in the third overlapping region, a second value higher than the first value is assigned to the transparency of these object detection images.

[0023] In the case where there are three adjacent object detection images (Situation A), compared to the case where there are two adjacent object detection images (Situation B), there is a possibility of reduced visual recognition of the surrounding images. According to the driving support device of this method, the transparency of the object detection images in Situation A is higher than that in Situation B. This suppresses the reduction in visual recognition of the surrounding images. Furthermore, the display area of ​​the object detection images in Situation A is larger than that in Situation B. Therefore, even if the transparency of the object detection images in Situation A is higher than that in Situation B, the driver can still recognize the displayed object detection images.

[0024] In other embodiments of the driving support device of the present invention,

[0025] The object detection sensor includes a positioning sensor that emits light, radio waves, or ultrasonic waves outward from the periphery of the vehicle and detects three-dimensional objects based on their reflected waves.

[0026] Therefore, it is possible to use known sensors to relatively easily obtain the position of a three-dimensional object relative to the vehicle.

[0027] In other embodiments of the driving support device of the present invention,

[0028] The object detection image is a graphic of the same shape and size as the region containing the distribution of reflection points reflecting the emitted radio waves, light, or ultrasonic waves.

[0029] In this driving support device, the areas corresponding to the position (or orientation) of the detected three-dimensional object are filled in among multiple zones set in the peripheral image. This allows the driver to grasp the approximate position (or orientation) of the three-dimensional object relative to the vehicle. Attached Figure Description

[0030] Figure 1 This is a block diagram of a driving support device according to one embodiment of the present invention.

[0031] Figure 2 This is an example of an image displayed on an image display device when there is a pedestrian diagonally in front of the right side of the vehicle.

[0032] Figure 3This is an example of an image displayed on an image display device when there are two pedestrians diagonally in front of the right side of the vehicle.

[0033] Figure 4 This is a flowchart of the program executed by the CPU to achieve the attention-awakening function. Detailed Implementation

[0034] (summary)

[0035] like Figure 1 As shown, a driving support device 1 according to one embodiment of the present invention is applied to a vehicle V0 (hereinafter referred to as "the vehicle"). The vehicle has an automatic driving function. The driving support device 1 has the following function (attention awakening function): when the automatic driving function of the vehicle is disabled and the driver performs manual driving operation, the information obtained by the object detection sensor mounted on the vehicle is superimposed on the image obtained by the camera mounted on the vehicle, thereby arousing the driver's attention to the detected object.

[0036] (Specific structure)

[0037] Next, the structure of the driving support device 1 will be described in detail. For example... Figure 1 As shown, the driving support device 1 includes an ECU 10, an on-board sensor 20, and an image display device 30.

[0038] ECU10 includes a microcomputer with CPU10a, ROM10b, RAM10c, timer10d, etc. ECU10 is connected to other ECUs via CAN (communication network).

[0039] The vehicle-mounted sensor 20 includes a sonar 21, which serves as an object detection sensor (position sensor) for acquiring information about three-dimensional objects OBx (x=a, b, ...) located around the vehicle. Additionally, the vehicle-mounted sensor 20 includes a camera 22 for acquiring images of the area surrounding the vehicle.

[0040] Sonar 21 has multiple transmitting and receiving units. These units are mounted, for example, on the front, side, and rear faces of the vehicle. Each transmitting and receiving unit emits ultrasonic waves to the surrounding area (front, right, left, and rear) of the vehicle and receives ultrasonic waves (reflected waves) reflected by three-dimensional objects (e.g., other vehicles parked at parking spots adjacent to where the vehicle is parked, pedestrians near where the vehicle is parked) within these emission ranges. Sonar 21 also has a signal processing unit. Based on the time from the emission of ultrasonic waves by the transmitting and receiving units to the reception of reflected waves, the signal processing unit provides information representing the distribution of ultrasonic wave reflection points (the areas occupied by each three-dimensional object OBx around the vehicle) as three-dimensional object information to ECU 10.

[0041] Here, the object detection area of ​​sonar 21 appears as a band (oval) surrounding the vehicle when viewed from above.

[0042] Camera 22 includes an imaging device. The imaging device may incorporate an imaging element such as a CCD (charge-coupled device) or a CIS (CMOS image sensor). The imaging device may be located, for example, at the front, right, left, and rear ends of the vehicle. Each imaging device captures images (image data) of the front, right, left, and rear areas of the vehicle at a predetermined frame rate. Each imaging device provides the acquired images to ECU 10.

[0043] The image display device 30 obtains the image display command from the ECU 10 and displays the image according to the command.

[0044] (Note the wake-up function)

[0045] ECU10 acquires stereoscopic information sequentially from sonar 21. In addition, ECU10 acquires images of the front, rear, left and right areas sequentially from each camera device of camera 22, and synthesizes (deforms) these images to generate (acquire) a top view (hereinafter referred to as "peripheral image G") as an image of the surrounding area of ​​the vehicle viewed from directly above.

[0046] To draw the driver's attention to the three-dimensional object OBx (x=a, b, ...) detected by sonar 21, ECU 10 controls the image display device 30 to overlay a predetermined graphic S (object detection image) representing the detection result onto the surrounding image G. Specifically, as Figure 2 As shown, in the peripheral image G, a region R corresponding to the object detectable area of ​​sonar 21 is defined. As mentioned above, the object detectable area appears as an oblong ring (strip) when viewed from above. Therefore, region R also appears as an oblong ring. This region R is equally divided into multiple divisions (angular ranges) in its circumference. For example, as shown in the figure, region R is divided into 16 divisions (divisions B1 to B16). ECU10 controls the image display device 30 to overlay a graphic S of the same shape and size as division Bk (in a way that the interior of division Bk is filled) with the division Bk (k=1, 2, ..., 16) corresponding to the detection result of sonar 21 (the area occupied by the stereoscopic object OBx).

[0047] Here, when graphic S is superimposed on the area Bk (k=1, 2, ..., 16) in the surrounding image G, graphic S is displayed transparently (semi-transparently) in such a way that the driver can visually identify the inner part of the area Bk (a portion of the photograph captured by camera 22). Here, in order to make graphic S stand out, a relatively low value is assigned to the transparency T of graphic S, which increases the effect of attracting the driver's attention, but reduces the visual recognizability of the part of the surrounding image G covered by graphic S (the inner part of the area Bk).

[0048] Therefore, as explained below, ECU10 adjusts the transparency T of graphic S according to the situation (transparency adjustment processing).

[0049] When a three-dimensional object exists in a relatively narrow area, or when a small three-dimensional object exists alone, the driver may have difficulty noticing it. Therefore, in this situation, the ECU 10 assigns a standard value T0 to the transparency T of the graphic S. On the other hand, when a three-dimensional object exists over a relatively wide area, or when a large three-dimensional object exists, the driver is more likely to notice it. Therefore, in this situation, the ECU 10 assigns a value T1, which is higher than the standard value T0, to the transparency T of the graphic S.

[0050] More specifically, ECU10, such as Figure 2 As shown, in region R, graphic S is displayed overlapping with zone Bk. When graphic S is not displayed overlapping with zones Bkcw (clockwise adjacent to zone Bk) and Bkccw (counterclockwise adjacent to zone Bk), a standard value T0 is assigned to the transparency T of graphic S overlapping with zone Bk. That is, when there are three-dimensional objects in the area corresponding to zone Bk but not in the areas corresponding to zones Bkcw and Bkccw on either side of it, the transparency T of graphic S overlapping with zone Bk is adjusted so that zone Bk is more prominent in the surrounding image G (making graphic S easily recognizable to the driver).

[0051] On the other hand, ECU10, such as Figure 3 As shown, when graphic S is displayed overlapping with zone Bk, and when graphic S is displayed overlapping with at least one of zones Bkcw and Bkccw, the transparency T of the graphic S overlapping these zones is assigned a value T1 higher than the standard value T0. That is, when there is a three-dimensional object in the area corresponding to multiple zones Bk that are continuous in the circumferential direction of region R, the transparency T of graphic S is adjusted in a way that makes it easier for the driver to visually identify the image of the inner part of these zones Bk.

[0052] Next, refer to Figure 4This section describes the program PR1 executed by CPU 10a (hereinafter referred to as "CPU") to achieve the aforementioned function of the driving support device 1 (the function of setting the transparency T of the graphic S according to the situation). The CPU executes program PR1 in a predetermined scenario. Specifically, based on information obtained from camera 22, the CPU starts executing program PR1 at predetermined intervals when the frame line of the parking point is detected (a state where the probability of the vehicle being parked at the parking point is high).

[0053] (Program PR1)

[0054] The CPU starts executing program PR1 from step 100 and proceeds to step 101.

[0055] In step 101, the CPU acquires three-dimensional object information from sonar 21. Then, the CPU proceeds to step 102.

[0056] In step 102, the CPU acquires images obtained from the cameras of camera 22 capturing images of the areas in front, behind, to the left, and to the right. These images are then combined (distorted) to generate (acquire) a surrounding image G. Next, the CPU proceeds to step 103.

[0057] In step 103, the CPU obtains the number of zones Bk, kmax. Figure 2 as well as Figure 3 In the example shown, the number kmax is "16". Next, the CPU proceeds to step 104.

[0058] In step 104, the CPU initializes the index k of the region used to select (specify) the processing object (the object for which steps 107 to 110 described below are performed) to "0". Then, the CPU proceeds to step 105.

[0059] In step 105, the CPU increments the index k. Then, the CPU proceeds to step 106.

[0060] In step 106, the CPU selects zone Bk as the processing target. Then, the CPU proceeds to step 107.

[0061] In step 107, the CPU determines whether graphic S is overlaid on region Bk. That is, the CPU determines whether a three-dimensional object exists in the area corresponding to region Bk based on the object information. If the CPU determines that graphic S is overlaid on region Bk (107: "Yes"), it proceeds to step 108. On the other hand, if the CPU does not determine that graphic S is overlaid on region Bk (107: "No"), it proceeds to step 111.

[0062] In step 108, the CPU determines whether the graphic S overlaps with at least one of the adjacent regions Bkcw and Bkccw. That is, based on the three-dimensional object information, the CPU determines whether a three-dimensional object exists in at least one of the regions corresponding to region Bkcw and region Bkccw. If the CPU determines that the graphic S overlaps with at least one of the adjacent regions Bkcw and Bkccw (108: "Yes"), it proceeds to step 109. Conversely, if the CPU does not determine that the graphic S overlaps with at least one of the adjacent regions Bkcw and Bkccw (108: "No"), it proceeds to step 110.

[0063] In step 109, the CPU assigns a value T1, which is higher than the standard value T0, to the transparency T of the graphic S that is overlaid on the area Bk. Then, the CPU proceeds to step 111.

[0064] In step 110, the CPU assigns a standard value T0 to the transparency T of the graphic S overlaid on the area Bk. Then, the CPU proceeds to step 111.

[0065] In step 111, the CPU determines whether the index k is consistent with the number of partitions Bk, kmax. If the CPU determines that the index k is consistent with the number kmax (111: "Yes"), it proceeds to step 112. On the other hand, if the CPU does not determine that the index k is consistent with the number kmax (111: "No"), it returns to step 105.

[0066] In step 112, the CPU terminates the execution of program PR1.

[0067] (Effect)

[0068] When a three-dimensional object is detected by sonar 21, and the resulting graphic S is overlaid on the surrounding image G, a portion of the surrounding image G may be obscured by the graphic S, making it difficult for the driver to visually identify that portion of the image (part of the surrounding image G). To address this issue, if the transparency T of the graphic S is consistently increased, it may also make it difficult for the driver to recognize the displayed graphic S. Therefore, the ECU 10 of the driving support device 1 according to this embodiment adjusts the transparency T of each graphic S according to the situation when overlaying the graphic S on the surrounding image G. That is, the ECU 10 adjusts the transparency T of each graphic S in the first division of the surrounding image G (… Figure 2 In the example, zone B2) overlaps with graphic S, and all zones adjacent to this first zone ( Figure 2In the example where regions B1 and B3 do not overlap, in the first case where graphic S is displayed, a standard value T0 (a value with relatively low transparency) is assigned to the transparency T of graphic S in the first region. On the other hand, the first region (B1 and B3) in the peripheral image G of ECU10 Figure 3 In the example, zone B2) overlaps with graphic S, and at least one zone adjacent to the first zone ( Figure 3 In the example of the overlapping display of graphic S in section B3, a value T1 higher than the standard value T0 is assigned to the transparency T of these graphic S. In the first condition, the transparency T of graphic S is relatively low, so graphic S is easily noticeable. That is, it can attract the driver's attention to the three-dimensional object in the direction of graphic S. In this first condition, it is slightly difficult for the driver to visually identify the image of the inner part of the first section in the peripheral image G, but its range is relatively narrow, so the driver can predict the inner part of the first section from the image around the first section. In contrast, in the second condition, the transparency T of graphic S is relatively high, so the driver can easily visually identify the image of the part covered by these graphic S in the peripheral image G. In this second condition, graphic S is less noticeable than in the first condition, but because multiple graphic S are arranged adjacently (graphic S occupies a relatively large area in the peripheral image G), the driver can recognize that graphic S is displayed even though the transparency T of these graphic S is relatively high. As described above, according to this embodiment, a driving support device is provided that has the function of displaying information acquired by sonar 21 as an overlay on a peripheral image G representing the periphery of the vehicle, and has the function of displaying graphics S in an appropriate manner (transparency) according to the situation.

[0069] (Variation Example 1)

[0070] In the above embodiment, when multiple consecutive sections Bk in the circumferential direction of region R overlap to display graphic S, the transparency T of these graphic S is assigned a value T1 that is higher than the standard value T0. That is, when two or more consecutive sections Bk in the circumferential direction of region R overlap to display graphic S, the transparency T of graphic S is the same (T1>T0), regardless of the number of consecutively arranged graphic S in the circumferential direction of region R. Alternatively, when two or more consecutive sections Bk in the circumferential direction of region R overlap to display graphic S, the more consecutively arranged graphic S there are, the higher the transparency T of these graphic S is assigned. That is, when ECU10 overlaps graphic S in sections B2 and B3 but does not overlap graphic S in section B1, the transparency T of these graphic S is assigned a first value T1 (e.g., "30%) that is higher than the standard value T0 (e.g., "20%)". In addition, when the graphic S is displayed in overlapping areas B1, B2 and B3, the ECU10 assigns a second value T2 (e.g., "40%) to the transparency T of these graphics S, which is higher than the first value T1.

[0071] (Variation Example 2)

[0072] The color of the graphic S can also be set based on the distance Δd between the three-dimensional object detected by sonar 21 and the vehicle. For example, the color of the graphic S can be set to "yellow" when the distance Δd exceeds the threshold Δdth, and the color of the graphic S can be set to "red" when the distance Δd is below the threshold Δdth.

[0073] (Variation Example 3)

[0074] Alternatively, a LIDAR can be used to replace (or add) sonar 21, emitting lasers towards the area surrounding the vehicle and detecting three-dimensional objects around the vehicle based on the reflected light. Alternatively, a millimeter-wave radar can be used to replace (or add) sonar 21, emitting millimeter-wave radio waves towards the area surrounding the vehicle and detecting three-dimensional objects around the vehicle based on the reflected waves.

[0075] (Variation Example 4)

[0076] In the above embodiment, the detectable area of ​​sonar 21 (object detection sensor) covers the entire circumference of the vehicle. Alternatively, the detectable area of ​​sonar 21 may not cover the entire vehicle but may be limited to a predetermined area. For example, sonar 21 may also be unable to detect objects located on the sides of the vehicle (e.g., Figure 2 as well as Figure 3 The three-dimensional objects in the divisions B4, B5, B12, and B13.

[0077] Explanation of reference numerals in the attached figures

[0078] 1: Driving support device; 10: ECU; 20: On-board sensors; 30: Image display device

Claims

1. A driving support device, comprising: The vehicle-mounted sensor includes a camera for acquiring a peripheral image of the surrounding area of ​​the vehicle, and an object detection sensor for acquiring information relating to the position of a three-dimensional object in a predetermined object-detectable area present in the surrounding area. Display device, displaying images; as well as The processor is configured to control the display device in such a way that it displays an object detection image representing the detection result of the stereoscopic object, overlapping the area corresponding to the position of the stereoscopic object detected by the object detection sensor within the entire area of ​​the peripheral image. The processor is configured as follows: The region in the surrounding image corresponding to the detectable area of ​​the object is divided into multiple zones. In a plurality of zones, the object detection image is displayed in an overlapping manner in the first zone, and in a situation where the object detection image is not displayed in an overlapping manner in all zones adjacent to the first zone, a standard value is assigned to the transparency of the object detection image displayed in the overlapping first zone. When the object detection image is displayed in the first zone and at least one zone adjacent to the first zone, the transparency of these object detection images is assigned a value higher than the standard value.

2. The driving support device according to claim 1, wherein, The surrounding image is equivalent to a top view of the vehicle and its surrounding parts from directly above. The multiple divisions are equivalent to dividing the annular area surrounding the vehicle in the peripheral image into multiple regions in the circumferential direction.

3. The driving support device according to claim 2, wherein, The processor is configured as follows: When the object detection image is displayed overlapping in the first zone, overlapping in the second zone adjacent to one side of the first zone, and not overlapping in the third zone adjacent to the other side of the first zone, the transparency of these object detection images is assigned a first value higher than the standard value. When the object detection image is displayed in the first overlapping region, the object detection image is displayed in the second overlapping region, and the object detection image is displayed in the third overlapping region, a second value higher than the first value is assigned to the transparency of these object detection images.

4. The driving support device according to any one of claims 1 to 3, wherein, The object detection sensor includes a positioning sensor that emits light, radio waves, or ultrasonic waves outward from the periphery of the vehicle and detects three-dimensional objects based on their reflected waves.

5. The driving support device according to claim 4, wherein, The object detection image is a graphic of the same shape and size as the region containing the distribution of reflection points reflecting the emitted radio waves, light, or ultrasonic waves.

Images