Display control device

Abstract

The display control device disclosed herein can reduce user discomfort when switching the display of an object according to its type identification status. [Solution] The display control device includes: an object detection unit that detects the presence of an object using other sensors provided by the vehicle other than the camera and identifies the type of object from an image taken by the camera provided by the vehicle; and a display control unit that displays a transition state indicating that the object display is being switched when the display unit switches between a first object display in a state where the object type cannot be identified because it is outside the identification range of the object type by the camera but within the detection range of the other sensors, and a second object display in a state where the object type can be identified because it is within the identification range of the camera.

Description

Technical Field

[0001] This disclosure relates to a display control device.

Background Art

[0002] Patent Document 1 discloses an object detection device that can more accurately determine the type of an object using a captured image. In Patent Document 1, a pedestrian or a two-wheeled vehicle as an object existing around the host vehicle is detected by performing image processing on a captured image obtained by capturing the traveling direction of the host vehicle. The type of the object is tentatively determined by analyzing the captured image, and the type of the tentatively determined object is finally determined using the moving speed of the object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In addition, in the prior art, for example, there is a known technique of displaying a target as object presence information detected by a sensor or the like on a display device using an icon. However, when the target is outside the detection range of the camera and within the detection range of another sensor, even if the presence of the target can be identified, the type cannot be specified, and there is a problem that it cannot be displayed unless it enters the identification range of the camera. Therefore, when it enters the identification range of the camera, it will be switched to an appropriate type of icon, but if the icon is switched to a completely different type, it may cause a sense of discomfort to the user.

[0005] An object of this disclosure is to provide a display control device that can reduce the user's sense of discomfort when switching the display of a target according to the identification state of the type.

Means for Solving the Problems

[0006] The display control device according to claim 1 includes: an object detection unit that detects the presence of an object using other sensors provided by the vehicle other than a camera and identifies the type of object from an image captured by the camera provided by the vehicle; and a display control unit that displays a transition state indicating that the display of an object is being switched when the display unit switches between a first object display in a state where the type of object cannot be identified because it is outside the identification range of the object type by the camera but within the detection range of the other sensors, and a second object display in a state where the type of object can be identified because it is within the identification range of the camera.

[0007] The display control device described in claim 1 displays the transition state during target switching. This allows the user to easily understand that the type of target has not been identified, and to identify the target without any difficulty when the target is identified and switched.

[0008] The display control device according to claim 2 is the display control device according to claim 1, wherein the display control unit displays the transition state display with reduced brightness or saturation. According to the display control device according to claim 2, the user's discomfort can be reduced by changing the display manner of the display during switching.

[0009] The display control device according to claim 3 is the display control device according to claim 1, wherein the display control unit sets the second target display as an icon corresponding to the type of identified target, and the first target display as a general-purpose target set to a lower height than the icon, and draws the switching of the display from the general-purpose target to the icon in the transition state display as a continuous animation. According to the display control device according to claim 3, the user's discomfort can be reduced by continuously changing the display of the targets.

[0010] The display control device according to claim 4 is the display control device according to claim 1, wherein when the target of the second target display moves out of the identification range of the camera, the second target display is displayed so as to gradually disappear. According to the display control device according to claim 4, the user's discomfort can be reduced by continuously changing the display of the target.

[0011] The display control device according to claim 5 is the display control device according to claim 1, and when the target of the first target display moves out of the identification range of the other sensor, the display of the first target display is made to gradually disappear. According to the display control device according to claim 5, the user's discomfort can be reduced by continuously changing the display of the target.

[0012] The display control device according to claim 6 is the display control device according to claim 1, which causes the first target display to appear gradually when a target that was not detected within the identification range of the other sensor enters. According to the display control device according to claim 6, the user's discomfort can be reduced by continuously changing the display of the target. [Effects of the Invention]

[0013] According to the technology disclosed herein, when switching the display of an object according to its type identification status, it is possible to reduce user discomfort. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 shows the configuration of the in-vehicle system according to this embodiment. [Figure 2A] Figure 2A is a schematic diagram illustrating an example of target detection. [Figure 2B] Figure 2B is a schematic diagram illustrating an example of target detection. [Figure 3] Figure 3 is a diagram illustrating general-purpose targets categorized by size and type. [Figure 4] Figure 4 is a diagram illustrating the timing of displaying transition states. [Figure 5A] FIG. 5A is a diagram showing an example of the transition of the transition state display. [Figure 5B] FIG. 5B is a diagram showing an example of the transition of the transition state display. [Figure 6A] FIG. 6A is a diagram showing an example of estimating a likely type from the frequency information of the types of targets identified in the driving lane. [Figure 6B] FIG. 6B is a diagram showing an example of estimating a likely type from the frequency information of the types of targets identified in the driving lane. [Figure 7] FIG. 7 is a flowchart for explaining the flow of display control processing as a display control method executed by the display control ECU of the present embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0015] Hereinafter, an example of an embodiment of the present disclosure will be described in detail with reference to the drawings. As shown in FIG. 1, an in-vehicle system 10 according to the present embodiment includes a communication bus 12. A peripheral situation acquisition device group 14, a vehicle traveling state detection sensor group 26, an ADAS (Advanced Driver-Assistance Systems)-ECU 34, and a display control ECU 42 are connected to the communication bus 12, respectively. Note that FIG. 1 shows only a part of the in-vehicle system 10. Also, hereinafter, a vehicle equipped with the in-vehicle system 10 is referred to as the host vehicle.

[0016] The peripheral situation acquisition device group 14 includes a GNSS (Global Navigation Satellite System) device 16, an in-vehicle communicator 18, a navigation system 20, a camera device 24, a radar device 22, etc. as devices for acquiring peripheral information representing the situation of the peripheral environment of the host vehicle.

[0017] The GNSS device 16 receives GNSS signals from a plurality of GNSS satellites to measure the position of the host vehicle. The in-vehicle communicator 18 performs at least one of vehicle-to-vehicle communication with other vehicles and vehicle-to-roadside unit communication with the roadside unit. The navigation system 20 includes a map information storage unit 20A that stores map information, and based on the position information obtained from the GNSS device 16 and the map information stored in the map information storage unit 20A, it performs processes such as displaying the position of the host vehicle on the map or determining and guiding the route to the destination.

[0018] The radar device 22 is a group of radars that detects objects such as other vehicles and pedestrians existing around the host vehicle as point cloud information, and detects objects in front of, on the side of, and behind the host vehicle. Also, the radar device 22 acquires the relative position and relative speed of the detected object with respect to the host vehicle. The radar device 22 is an example of another sensor of the present disclosure. In the display control ECU 42 described later, the presence of a target is detected from the detection result of the radar device 22.

[0019] Further, the radar device 22 excludes roadside objects such as noise and guardrails from the monitoring targets based on changes in the relative position and relative speed with respect to individual objects, and outputs information such as the relative position and relative speed with respect to the targets that are monitoring target objects such as other vehicles and pedestrians.

[0020] The camera device 24 is a plurality of cameras that photograph the periphery of the host vehicle and outputs the photographed images. In the present embodiment, the case where the camera device 24 photographs the front of the host vehicle will be described as an example, but a side camera that photographs the rear and a side camera that photographs the side closest to the host vehicle from the side mirror are also conceivable. In the display control ECU 42 described later, the type of the target is identified from the information of the objects existing in the photographed image. The camera device 24 is an example of the camera of the present disclosure.

[0021] As described above, it is detected from the detection result of the radar device 22 that a target exists, but the type of the target is not identified, and the type is identified by analyzing the image of the camera device 24.

[0022] The vehicle driving state detection sensor group 26 includes a steering angle sensor 28 for detecting the steering angle of the vehicle, a vehicle speed sensor 30 for detecting the driving speed of the vehicle, and an acceleration sensor 32 for detecting the acceleration applied to the vehicle, as multiple sensors for acquiring the driving state of the vehicle.

[0023] The ADAS-ECU34 is connected to the throttle ACT36, which changes the throttle opening of the vehicle; the brake ACT38, which changes the braking force generated by the vehicle's braking system; and the steering ACT40, which changes the amount of steering by the vehicle's steering system.

[0024] The ADAS-ECU34, although not shown in the diagram, includes a CPU, memory such as ROM and RAM, storage such as an HDD or SSD, and a communication interface (I / F). The storage contains a predetermined program that enables the ADAS-ECU34's CPU to function as a support unit 35. When the vehicle is in driving assistance mode, the support unit 35 controls the throttle ACT36, brake ACT38, and steering ACT40 based on information obtained from the surrounding situation acquisition device group 14 and the vehicle driving state detection sensor group 26, as well as the target detection results from the display control ECU42, to perform support processing that assists at least one of the vehicle's acceleration / deceleration and steering. Examples of driving assistance functions realized by this support processing include ACC (Adaptive Cruise Control), which drives the vehicle to follow a preceding vehicle in its lane, and LKA (Lane Keeping Assist), which assists steering to keep the vehicle in the center of its lane.

[0025] The display control ECU 42 includes a CPU 44, memory 46 such as ROM or RAM, storage 48 such as HDD or SSD, and a communication I / F 50. The CPU 44, memory 46, storage 48, and communication I / F 50 are connected to each other via an internal bus 52 so that they can communicate with one another. The display control program 54 is stored in the storage 48. The display control ECU 42 reads the display control program 54 from the storage 48 and expands it into the memory 46, and the display control program 54 expanded into the memory 46 is executed by the CPU 44. The CPU 44 functions as a target detection unit 62 and a display control unit 64, and performs the display control processing described later. Note that the function of the target detection unit 62 may be an external function of the display control ECU 42, and a separate target detection ECU may be provided.

[0026] The display control ECU42 is for AR (Augmented Reality) - Head-Up Display (and Below, an AR-HUD 56 (referred to as an AR-HUD) and a meter display 58 are connected. The AR-HUD 56 according to this embodiment is a small HUD that displays an image (forming an image in the lower foreground) on a portion of the forward field of view of the vehicle's occupants by reflection from the windshield glass, etc. The meter display 58 is a display provided on the instrument panel of the vehicle. The display control ECU 42 controls the display of information on the AR-HUD 56 and the meter display 58. Note that the display control ECU 42 is an example of a display control device according to this disclosure, and the AR-HUD 56 and meter display 58 are examples of display units in this disclosure.

[0027] The target detection unit 62 detects the presence of targets around the vehicle based on the detection results from the radar device 22. The target detection unit 62 also identifies the type of target from the image captured by the camera device 24. In this embodiment, the target detection unit 62 identifies at least passenger cars, large commercial vehicles such as trucks, motorcycles, and pedestrians as types of targets.

[0028] Figure 2 schematically illustrates an example of target detection. In the examples in Figures 2A and 2B, the vehicle image is 78, the targets around the vehicle are 80, the detection range of the radar device 22 is 90, and the identification range of the camera device 24 is 92. Note that the detection range 90 shown in Figures 2A and 2B is shown for illustrative purposes only as the detection range to the side of the vehicle that does not overlap with the identification range 92 of the camera device 24, and the front and rear areas that overlap with the identification range 92 of the camera device 24 may also be included in the detection range.

[0029] In the example in Figure 2A, the target 80 is located within the detection range 90 of the radar device 22 and within the identification range 92 of the camera device 24, and the type of target 80 can be identified. In the example in Figure 2B, the target 80 is located within the detection range 90 of the radar device 22, but outside the identification range 92 of the camera device 24, and the type of target 80 cannot be identified. As shown in Figure 2A, when the target 80 can be identified, an icon image 80A indicating the type of target 80, which is prepared according to the type, is displayed. As shown in Figure 2B, when the target 80 cannot be identified, a general-purpose target image 80B is displayed. The general-purpose target image 80B is a roughly rectangular parallelepiped with a lower height than the icon displayed according to the type. By displaying the general-purpose target in this way, it is possible to indicate that there are targets whose type has not yet been identified. The height of the general-purpose target is lower than the icon of the target type. By displaying general-purpose targets with different sizes and appearances from the type icons, it is possible to prevent users from mistaking them for type icons and making it easier to understand that the type is unidentified. Furthermore, by reducing the height of the general-purpose targets and displaying them at the minimum size necessary for identification, the estimated type can be more easily conveyed through size. The target shape can also be limited to a roughly rectangular prism, including trapezoids and truncated cones. Additionally, by temporarily displaying general-purpose targets that do not specify passenger cars, commercial vehicles, motorcycles, or pedestrians, it is possible to reduce user discomfort when the type is identified and the corresponding icon image is displayed.

[0030] Figure 3 is a diagram illustrating general-purpose targets of different sizes corresponding to different target types. Figures 3(1) to (4) show the respective icon images and their corresponding general-purpose target images. The general-purpose target images are set to approximately the same width and depth as the corresponding icon images of each type. (1) is the icon image (80A-1) representing a passenger car and the general-purpose target (80B-1) corresponding to the passenger car type. (2) is the icon image (80A-2) representing a commercial vehicle and the general-purpose target (80B-2) corresponding to the commercial vehicle type. The general-purpose target (80B-2) for commercial vehicles is set to be slightly larger in both width and depth than that for passenger cars. (3) is the icon image (80A-3) representing a motorcycle and the general-purpose target (80B-3) corresponding to the motorcycle type. The size of the general-purpose marker for motorcycles (80B-3) is set to be about half the width of a passenger car, with a depth about the same as the depth of a passenger car, resulting in a vertically elongated, roughly rectangular shape. (4) is an icon image (80A-4) representing a pedestrian and a general-purpose marker (80B-4) corresponding to the type of commercial vehicle. The size of the general-purpose marker for pedestrians (80B-4) is set to be about the same width as a motorcycle, with a depth about half the depth of a motorcycle. In this embodiment, the display control unit 64 controls which size of general-purpose marker to display.

[0031] The display control unit 64 displays an icon indicating the type of target if the camera device 24 can identify the type of target detected by the radar device 22. If the display control unit 64 cannot identify the type of target detected by the radar device 22, it displays a general-purpose target. The display control unit 64 can also estimate the type of general-purpose target most likely to be present around the vehicle and display a general-purpose target of the size corresponding to the estimated type. The estimation method will be described later.

[0032] In this embodiment, the display control unit 64 switches between a general-purpose target (first target display) set to a lower height than the icon, and an icon corresponding to the type of identified target (second target display), according to the transition of the target. When switching, the display control unit 64 displays a transition state indicating that the target display is being switched, according to the identification state of the camera device 24. In this embodiment, the transition state display in the display switching will be explained using as an example the case when switching from a general-purpose target (first target display) to an icon corresponding to the type (second target display), and the case when the icon disappears as the type of the identified target (second target display) moves away from the identifiable area of ​​the vehicle.

[0033] Figure 4 is a diagram illustrating the timing of the transition state display. The example in Figure 4 shows the case where the transition state display is performed at the timing of switching from a general-purpose target to an icon indicating the type of target. In (1-1), the target 80 was located within the detection range 90 of the radar device 22 but outside the identification range 92 of the camera device 24. In (1-2), the target transitions into the identification range 92 of the camera device 24. At the timing of (1-2), the general-purpose target is displayed in the transition state (CH). (1-3) is the timing when the transition state display (CH) ends and the display switches to the icon. Here, in the (1-2) transition state display (CH), the display switches to the icon indicating the type of target by drawing an animation such as (CH1) where the front of the general-purpose target is gradually made to disappear, while (CH2) where the front of the passenger car icon appears. Note that the display is just an example, and the general-purpose target may be made to disappear first, and then the icon may be displayed from the front with a gradient. The gradient of the transition state display is displayed with reduced brightness or saturation. Furthermore, the display may be drawn to gradually disappear during any transition, not limited to a fade-out, such as a mosaic display, a dissolve display, or a slide-out aligned with the direction of movement. Similarly, when the display reappears, as described in (2-2), it may be drawn during any transition, not limited to a fade-in, such as a mosaic display, a dissolve display, or a slide-in. By switching the display to gradually disappear or reappear while changing the display of a continuous animation in this way, the user's discomfort caused by the display change can be reduced.

[0034] Figures 5A and 5B show examples of transitions in the display of transition states. Figure 5A shows an example of the display of transition states when switching from a general-purpose target to an icon. (2-1) is the state where no target is detected and no target is displayed. (2-2) is the state where an object has been detected and the general-purpose target (80B-1) is displayed. When displaying (2-2), the display control unit 64 may also control the display to fade in the general-purpose target when a target that was not detected within the identification range of the radar device 22 enters. In this case, it is displayed by fading in from the front. (2-3) is the state in which the camera device 24 has started to detect the type of target within the identification range, and the display is switching to the transition state display (CH1) by fading out the front. Next, the display of transition states moves to the state shown in Figure 5B.

[0035] Figure 5B shows an example of a transition state display from the time a detected target transitions until it disappears. (2-4) is a transition state display (CH2) in which only the front of the passenger car is identified within the identification range, and the display is transitioning from a general-purpose target to an icon image of the target type via animation. (2-5) is a state in which the display has switched to the icon image of the target type (80A-1). State (2-6) is a state in which the passenger car has moved further ahead of the identification range of the vehicle, and only the rear of the passenger car is within the identification range. Display (2-6) is a transition state display (CH3) in which the front of the passenger car fades out as the icon image of the target moves out of the identification range of the camera device 24, indicating that it is transitioning to the disappearance state.

[0036] The estimation method for general-purpose targets is described below. The methods for estimating the most likely types of targets to be present around the display control unit 64 are illustrated below as follows: (1) In the first identification method, the display control unit 64 estimates the most likely type from the type of lane the vehicle is in and displays a general-purpose target of the size corresponding to the estimated type. For example, if the lane in which the vehicle is located, as measured by the GNSS device 16, is a general road, the most likely type to be present is a passenger car or a commercial vehicle. If the lane in which the vehicle is located, as measured, is near an intersection, the most likely type to be present is a pedestrian. (2) In the second identification method, the display control unit 64 estimates the most likely type from the frequency information of the types of targets identified in the lane and displays a general-purpose target of the size corresponding to the estimated type. (3) In the third identification method, the display control unit 64 estimates the most likely type from the date and time information and displays a general-purpose target of the size corresponding to the estimated type. The display control unit 64 acquires the position of the vehicle measured by the GNSS device 16 and map information from the map information storage unit 20A, and identifies the lane the vehicle is traveling in by comparing the vehicle's position with the lanes on the map. Data for estimating the most likely type is stored in pre-prepared setting data for a specific date and time and lane set. The setting data may be a table data that stores the most likely types for a specific date and time and lane set, or it may be model data that has learned the most likely types according to the characteristics of the lane and the date and time. The display control unit 64 uses the setting data to estimate the most likely type corresponding to the lane at the current date and time, and displays a general-purpose marker of the corresponding size. For example, the setting data may be set or learned by distinguishing between "weekdays" and "holidays / weekends" as specific dates and times. If a lane tends to have many commercial vehicles on "weekdays" or at "night," commercial vehicles may be set or learned as the most likely type. If a lane tends to have many passenger cars on "holidays / weekends" or during "daytime," passenger cars may be set or learned as the most likely type. In this embodiment, one of the methods (1) to (3) is used. By using the estimation methods described above, a general-purpose target corresponding to a highly accurate type can be displayed.

[0037] Figures 6A and 6B illustrate examples of estimating the most likely type of object based on frequency information of object types identified in the driving lane. In both Figures 6A and 6B, the types of three objects in front have been identified, while the type of one object to the side has not been identified. In Figure 6A, a passenger car (80A-1), a vehicle in the center (80A-1), and a commercial vehicle (80A-2) are identified to the front left, and an unidentified object (80n) is detected to the side. Since two of the three identifiable objects are passenger cars (80A-1), the most likely type of the unidentified object (80n) is estimated to be a passenger car (80A-1). In Figure 6B, a commercial vehicle (80A-2), a vehicle in the center (80A-1), and a commercial vehicle (80A-2) are identified to the front left, and an unidentified object (80n) is detected to the side. Since two out of the three identifiable targets are commercial vehicles (80A-2), the most likely type of the unidentified target (80n) is estimated to be a commercial vehicle (80A-2).

[0038] (Control flow) Figure 7 is a flowchart illustrating the flow of the display control process as a display control method executed by the display control ECU 42 in this embodiment. The display control process should be executed at predetermined intervals or when detection occurs by the radar device 22 and the camera device 24.

[0039] In step S100, the target detection unit 62 detects the presence of targets in the vicinity of the vehicle based on the detection results of the radar device 22.

[0040] In step S102, the target detection unit 62 identifies the type of target detected from the image captured by the camera device 24 and displays an icon image corresponding to the identified type of target on the display unit.

[0041] In step S104, the target detection unit 62 determines whether or not there are any targets among the detected targets whose type cannot be identified. Targets whose type cannot be identified are those detected outside the target type identification range of the camera device 24 but within the detection range of the radar device 22. If there are targets whose type cannot be identified, the process proceeds to step S106; if there are no targets whose type cannot be identified, the process proceeds to step S110.

[0042] In step S106, the display control unit 64 estimates the types of objects that are most likely to be present in the vicinity of the vehicle.

[0043] In step S108, the display control unit 64 displays a general-purpose object of the size corresponding to the estimated type of object.

[0044] In step S110, the display control unit 64 determines whether a display switching transition has occurred for the general-purpose target and the icon corresponding to the type of identified target. If a display switching target transition has occurred, the process proceeds to step S112; otherwise, it returns to step S100 and repeats the display control process.

[0045] In step S112, the display control unit 64 displays a transition state indicating that the display of the target is being switched according to the identification status of the camera device 24. The transition state display causes targets that have changed from unidentifiable to identifiable to display an icon image corresponding to the target type, rather than a general-purpose target. Targets that have changed from identifiable to unidentifiable disappear from the display. After this step, the process returns to step S100 and is repeated to update the target display as needed. The display control process is repeated until the vehicle stops or no more targets are detected in the vicinity.

[0046] As described above, the display control ECU42 of this embodiment can reduce user discomfort when switching the display of an object according to the identification status of its type.

[0047] Furthermore, in the above embodiment, a configuration was described in which a display control program 54 for causing the display control ECU 42 to function as a target detection unit 62 and a display control unit 64 is pre-stored (installed) in the storage 48. However, the display control program 54 can also be provided in a form recorded on a non-temporary recording medium such as an HDD, SSD, or DVD.

[0048] The processing flow described in the above embodiment is just one example, and unnecessary steps may be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main point. [Explanation of Symbols]

[0049] 10 In-vehicle systems 22. Radar equipment (other sensors) 24 Camera equipment (camera) 42 Display Control ECU (Display Control Unit) 62 Target detection unit 64 Display Control Unit

Claims

1. A target detection unit that detects the presence of a target using other sensors on the vehicle besides the camera, and identifies the type of the target from the image captured by the camera on the vehicle, A display control unit that displays a transition state indicating that the display of an object is being switched, when the display unit switches between a first object display in a state where the object type cannot be identified because it is outside the identification range of the camera but within the detection range of the other sensor, and a second object display in a state where the object type can be identified because it is within the identification range of the camera, in accordance with the transition of the object, A display control device that includes a display control device.

2. The display control device according to claim 1, wherein the display control unit displays the transition state display with reduced brightness or saturation.

3. The display control device according to claim 1, wherein the display control unit sets the second target display as an icon corresponding to the type of identified target, and the first target display as a general-purpose target set to a lower height than the icon, and the switching of the display from the general-purpose target to the icon in the transition state display is drawn as a continuous animation.

4. The display control device according to claim 1, wherein when the target of the second target display moves out of the identification range of the camera, the display of the second target display is made to gradually disappear.

5. The display control device according to claim 1, which causes the first target display to gradually disappear when the target of the first target display moves out of the identification range of the other sensor.

6. The display control device according to claim 1, which causes the first target display to gradually appear when a target that has not been detected within the identification range of the other sensor enters the area.

Images