Display control device and display control program

Abstract

To provide a technology that enables superimposed display with fewer displacements or senses of discomfort in a display control device and a display control program for controlling display operation in a display device that, by being mounted to a host vehicle, superimposedly displays a display image in the foreground of the host vehicle.SOLUTION: A recognition result acquisition unit (121) acquires recognition result information that corresponds to the image recognition result of a target based on the foreground image captured by an image capturing device (13). A reliability calculation unit (125) calculates the reliability of recognition result information on the basis of feature quantity information that represents intermediate information in a process in which recognition result information is acquired or the recognition result information. A display state setup unit (126) sets, on the basis of the reliability calculated by the reliability calculation unit, a display state that represents the status of a display image being displayed or not, or the visibility of the display image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display control device and a display control program that control a display operation of a display device that is mounted in a vehicle and displays a display image in a superimposed manner on a foreground of the vehicle. [Background technology]

[0002] The driving assist system described in Patent Document 1 displays predetermined image information superimposed on the windshield of the vehicle, and is characterized in that the length of the arrow indicating the guidance direction in the image information is changed according to the reliability of the vehicle position information. When the reliability of the vehicle position information is low, this driving assist system displays guidance using an arrow with a short arrowhead, thereby preventing unnecessary confusion for the driver. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-203053 A Summary of the Invention [Problem to be solved by the invention]

[0004] As described in Patent Document 1, a technology for displaying various types of information such as route guidance using a so-called head-up display has been known in the past. Furthermore, in recent years, there has been a demand for this type of display technology to use image processing technology used in ADAS to recognize targets such as white lines, and to utilize information on the vehicle's attitude and the position of the target to present information that helps the driver to recognize and make decisions in the scene he or she is viewing without positional deviation. ADAS is an abbreviation for Advanced Driver Assistance System.

[0005] Here, for example, in image recognition technology, under adverse conditions such as when white lines are blurred and difficult to distinguish, the accuracy of position detection by recognition deteriorates. In addition, with the position accuracy of in-vehicle navigation systems currently in widespread use, the forward / backward / left / right position guided may deviate by about 10 m. Specifically, such position deviation occurs in situations where positioning signals from satellites such as GPS cannot be received or are difficult to receive due to the influence of high-rise buildings, tunnels, multi-story parking lots, etc. In addition, such position deviation occurs, for example, when the actual road shape has changed from the map information, causing deviations when matching the map with the vehicle's position estimation.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, etc. That is, the present invention provides a technique that enables superimposed display with less misalignment and discomfort, for example. [Means for solving the problem]

[0007] The display control device (12) described in claim 1 is configured to be mounted on a vehicle (1) and to control a display operation of a display device (11) that superimposes a display image (IM) on the foreground of the vehicle. This display control device includes: a recognition result acquisition unit (121) that acquires recognition result information corresponding to an image recognition result of a target based on an image of the foreground captured by an image capturing device (13); a reliability calculation unit (125) that calculates the reliability of the recognition result information based on intermediate information in the process of acquiring the recognition result information or feature amount information that is the recognition result information; a display state setting unit (126) that sets a display state of the display image based on the reliability calculated by the reliability calculation unit; a display information generation unit (124) that generates display information for displaying the display image on the display device based on the recognition result information; Equipped with picture, the reliability calculation unit calculates the reliability of an estimation result of a road shape, which is a road gradient or a road curvature of a road ahead of the host vehicle; the display information generation unit generates the display information corresponding to a route display image, which is the display image for displaying a travel route of the host vehicle, taking into account the estimation result of the road shape; The display state setting unit sets the display state in the route display image based on the reliability of the estimation result of the road shape. Claim to 8The display control program described is a program executed by the display control device, The process executed by the display control device includes: a recognition result acquisition process for acquiring recognition result information corresponding to an image recognition result of a target based on an image of the foreground captured by an image capturing device (13); a reliability calculation process for calculating reliability of the recognition result information based on intermediate information in a process of acquiring the recognition result information or feature amount information which is the recognition result information; a display state setting process for setting a display state of the display image based on the reliability calculated by the reliability calculation process; a display information generating process for generating display information for displaying the display image on the display device based on the recognition result information; Including fruit, the reliability calculation process calculates the reliability of an estimation result of a road shape, which is a road gradient or a road curvature of a road ahead of the host vehicle; the display information generation process generates the display information corresponding to a route display image, which is the display image for displaying a travel route of the host vehicle, taking into account the estimation result of the road shape; The display state setting process sets the display state in the route display image based on the reliability of the estimation result of the road shape.

[0008] In addition, in the application documents, each element may be given a reference number in parentheses. However, even in this case, such a reference number merely indicates an example of the correspondence between each element and the specific means described in the embodiment described later. Therefore, the present invention is not limited in any way by the description of the above reference numbers. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a schematic configuration of an in-vehicle system including a display control device according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing a schematic functional configuration of the in-vehicle system shown in FIG. 1. [Diagram 3] 3 is a block diagram showing an overview of reliability calculation according to the first embodiment of the display control device shown in FIG. 2. FIG. [Figure 4A] FIG. 11 is a schematic diagram showing a display example according to a comparative example. [Figure 4B] FIG. 2 is a schematic diagram showing a display example according to the first embodiment. [Diagram 5]4 is a block diagram showing an overview of reliability calculation according to a second embodiment of the display control device shown in FIG. 2. FIG. [Figure 6A] FIG. 11 is a schematic diagram showing a display example according to a comparative example. [Figure 6B] FIG. 11 is a schematic diagram showing a display example according to the second embodiment. [Figure 7] 10 is a diagram showing an overview of reliability calculation according to a third embodiment of the display control device shown in FIG. 2. FIG. [Figure 8] 13 is a diagram showing an overview of reliability calculation according to a fourth embodiment of the display control device shown in FIG. 2. FIG. [Figure 9] 13 is a diagram showing an overview of reliability calculation according to a fifth embodiment of the display control device shown in FIG. 2. FIG. [Figure 10] 13 is a diagram showing an overview of reliability calculation according to a sixth embodiment of the display control device shown in FIG. 2. FIG. [Figure 11] 13 is a diagram showing an overview of reliability calculation according to a seventh embodiment of the display control device shown in FIG. 2. FIG. [Figure 12] 13 is a block diagram showing an overview of reliability calculation according to an eighth embodiment of the display control device shown in FIG. 2. FIG. [Figure 13] 13 is a time chart showing an example of operation of the display control device shown in FIG. 2 according to the eighth embodiment. [Figure 14] 13 is a diagram showing an overview of reliability calculation according to an eighth embodiment of the display control device shown in FIG. 2. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Embodiment) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that various modified examples applicable to one embodiment may be hindered from being understood if they are inserted in the middle of a series of explanations related to the embodiment. For this reason, the modified examples will be explained together after the series of explanations related to the embodiment, rather than in the middle of the series of explanations related to the embodiment.

[0011] (In-vehicle system) Referring to FIG. 1, the vehicle 1 is a so-called four-wheeled vehicle, and includes a vehicle body 2 that is substantially rectangular in plan view when the vehicle 1 is viewed from vertically above. For convenience of the following description, the concepts of "front", "rear", "left", "right", "up" and "down" in the vehicle 1 are defined as indicated by arrows in FIG. 1. These directional concepts are based on the driver D, who is an occupant sitting in the driver's seat of the vehicle 1 while the vehicle 1 is traveling forward (i.e. while traveling in a shift range other than reverse). The front-rear direction is synonymous with the vehicle overall length direction. The left-right direction is synonymous with the vehicle width direction. The up-down direction is synonymous with the vehicle height direction. The vehicle overall length direction is a direction perpendicular to the vehicle width direction and perpendicular to the vehicle height direction. The vehicle height direction is a direction that defines the vehicle height of the vehicle 1, and is a direction parallel to the direction of gravity when the vehicle 1 is stably placed on a horizontal plane in a travelable state.

[0012] The front of the vehicle cabin 3, which is the interior space of the vehicle body 2 where passengers sit, is covered by a front windshield 4. The front windshield 4 is formed into a plate shape from translucent glass or synthetic resin. The front windshield 4 is inclined so that the upper end is located further rearward than the lower end when viewed from the side in a direction parallel to the vehicle width direction.

[0013] A dashboard 5 is disposed below the front windshield 4 within the vehicle interior 3. A meter panel 6 is provided on the dashboard 5. A steering column 7 extends from the dashboard 5 toward the driver D. A steering wheel 8 is attached to the steering column 7.

[0014] A vehicle 1 is equipped with an in-vehicle system 10. The vehicle 1 equipped with the in-vehicle system 10 is hereinafter referred to as the "own vehicle". The in-vehicle system 10 is configured to be capable of executing various controls when the own vehicle is driven and various display operations associated therewith. The in-vehicle system 10 is provided to configure an in-vehicle network that complies with a predetermined communication standard such as CAN (internationally registered trademark: international registration number 1048262A). CAN (internationally registered trademark) is an abbreviation for Controller Area Network.

[0015] The in-vehicle system 10 includes a head-up display device 11 that forms a display image IM using AR technology, and a display control device 12 that is configured to control the display operation of the head-up display device 11. AR is an abbreviation for Augmented Reality. The head-up display device 11 is a display device that is mounted on the vehicle and superimposes a display image IM, which is a virtual image, on a view in front of the vehicle (e.g., a road surface, etc.) visible through the front windshield 4, and is housed in the dashboard 5 below the front windshield 4.

[0016] Fig. 2 shows a schematic functional block configuration related to the operation control of the head-up display device 11 in the in-vehicle system 10. Below, a schematic head-up display control configuration in the in-vehicle system 10 will be described with reference to Fig. 1 and Fig. 2. In addition to the head-up display device 11 and the display control device 12, the in-vehicle system 10 includes an image capturing device 13, a recognition unit 14, a vehicle state sensor 15, and a navigation device 16.

[0017] The image pickup device 13 is a so-called front camera, and is provided so as to be able to pick up images of the front and front sides of the vehicle. Specifically, the image pickup device 13 is a digital camera device, and is provided with an image sensor such as a CCD or a CMOS. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor. Hereinafter, the image pickup device 13 may be simply referred to as a "camera". The recognition unit 14 is configured to perform image recognition of targets (e.g., white lines, etc.) included in the foreground based on an image of the foreground captured by the image pickup device 13. The "targets" include three-dimensional objects such as traffic lights TL in addition to road markings such as an intersection IS, a road dividing line LD, a stop line LL, and a pedestrian crossing ZC shown in FIG. 4A and the like. In this embodiment, the image pickup device 13 and the recognition unit 14 are configured so as to be able to detect the three-dimensional position of the target relative to the vehicle using a monocular stereo or compound eye stereo technique. The recognition unit 14 is provided so as to be able to provide the image recognition results of the target to each unit including the display control device 12.

[0018] The vehicle state sensor 15 is provided to detect the driving state and driving environment of the vehicle. The "driving state" includes various quantities related to the driving operation state by the driver D or the driving automation system, such as the accelerator opening, braking amount, shift position, steering angle, etc. The "driving state" also includes physical quantities related to the behavior of the vehicle, such as the vehicle speed (i.e., the running speed of the vehicle), angular velocity, forward / rearward acceleration, and left / right acceleration. The "driving environment" includes the natural environment, such as the outside temperature, rainfall, and illuminance. The "driving environment" also includes the presence state of moving objects and stationary objects in a predetermined detection range around the vehicle. The "moving objects" include pedestrians, cyclists, animals, and other vehicles that are being driven. The "stationary objects" include fallen objects on the road, guardrails, curbs, other parked vehicles, road signs, and structures such as walls and buildings. Moving objects and stationary objects are included in the "targets". In other words, the vehicle condition sensor 15 is a general term for various sensors other than the image capturing device 13, such as an accelerator opening sensor, a steering angle sensor, a wheel speed sensor, an angular velocity sensor, an acceleration sensor, an outside air temperature sensor, a raindrop sensor, an illuminance sensor, a radar sensor, an ultrasonic sensor, etc., for the sake of simplicity of illustration and explanation.

[0019] The navigation device 16 is configured to calculate the vehicle's position and the travel route of the vehicle based on map information and positioning information from positioning satellites. The "vehicle position" is the current position of the vehicle on the map information. The "travel route" is the planned travel route of the vehicle from the vehicle's position to the destination. The navigation device 16 also has links connecting nodes indicating intersections as branch information, and is configured to be able to calculate the angle between the travel direction of the vehicle and the link.

[0020] (Display Control Device) The display control device 12 has a configuration as an HCU. HCU is an abbreviation for HMI Control Unit. HMI is an abbreviation for Human Machine Interface. The display control device 12 is configured as a so-called in-vehicle computer having a processor and memory programmed to execute one or more functions embodied by a computer program. The display control device 12 has a recognition result acquisition unit 121, a vehicle position acquisition unit 122, a route information acquisition unit 123, a display information generation unit 124, a reliability calculation unit 125, and a display state setting unit 126 as functional configurations or functional units realized on a microcomputer.

[0021] The recognition result acquisition unit 121 is adapted to acquire an image recognition result of a target based on an image of the foreground captured by the image capturing device 13 (i.e., recognition result information which is information corresponding to the image recognition result) from the recognition unit 14. The vehicle position acquisition unit 122 is adapted to acquire the vehicle position from the navigation device 16. The route information acquisition unit 123 is adapted to acquire a travel route from the navigation device 16.

[0022] The display information generating unit 124 generates display information for displaying the display image IM on the head-up display device 11 based on the recognition result information acquired by the recognition result acquiring unit 121, the vehicle position acquired by the vehicle position acquiring unit 122, and the travel route acquired by the route information acquiring unit 123. The reliability calculating unit 125 calculates the reliability of the recognition result information based on the feature amount information. Here, the "feature amount information" includes the recognition result information and / or intermediate information in the process of acquiring the recognition result information. The display state setting unit 126 sets the display state of the display image IM based on the reliability calculated by the reliability calculating unit 125. The "display state" here includes the presence or absence of display and the display form. The "display form" includes various elements (e.g., brightness, saturation, hue, line diameter, display size, etc.) that affect the visibility of the display image IM.

[0023] That is, the processes executed by the display control device 12 include a recognition result acquisition process, a reliability calculation process, and a display state setting process. The recognition result acquisition process is a process for acquiring recognition result information corresponding to the image recognition result of the target based on the image of the foreground captured by the image capturing device 13. The reliability calculation process is a process for calculating the reliability of the recognition result information acquired in the recognition result acquisition process based on intermediate information in the process of acquiring the recognition result information or feature amount information which is the recognition result information. The display state setting process is a process for setting the display state in the display image IM based on the reliability calculated by the reliability calculation process.

[0024] (Operation overview) The following describes the operation of the in-vehicle system 10 and the display control device 12 according to this embodiment, and an overview of the display control method and the display control program executed thereby, along with the effects achieved by this embodiment. In the following description, the configuration of the in-vehicle system 10 and the display control device 12 according to this embodiment, and the series of processes of the display control method and the display control program executed thereby, may be collectively referred to as "this embodiment."

[0025] The recognition unit 14 performs image recognition of targets included in the foreground visible to the driver D through the front windshield 4 in front and to the sides of the vehicle based on the captured image of the foreground by the image capturing device 13. The image recognition result of the targets by the recognition unit 14 is used for the autonomous driving operation or ADAS operation of the vehicle, as well as for the display operation by the head-up display device 11.

[0026] The display control device 12 generates a display image IM, which is a virtual image, based on signals or information acquired from the recognition unit 14, the vehicle state sensor 15, and the navigation device 16, and uses the display image IM to superimpose various information on the foreground. Specifically, the recognition result acquisition unit 121 acquires, from the recognition unit 14, an image recognition result of a target based on an image of the foreground captured by the image capture device 13. The vehicle position acquisition unit 122 acquires the vehicle position from the navigation device 16. The route information acquisition unit 123 acquires a travel route from the navigation device 16. The display information generation unit 124 generates display information for superimposing the display image IM on the foreground, based on the image recognition result of the target acquired by the recognition result acquisition unit 121, the vehicle position acquired by the vehicle position acquisition unit 122, and the travel route acquired by the route information acquisition unit 123.

[0027] Here, in the image recognition technology, the recognition accuracy is highly dependent on the environment in which the image is captured and the state of the target object to be captured. For this reason, image recognition cannot always be performed with the same accuracy. Specifically, under adverse conditions such as a situation in which the white lines are blurred and difficult to distinguish, the accuracy of position detection by recognition deteriorates. Therefore, if the image recognition result is simply used, even if the recognition accuracy is reduced, the display control is performed using the image recognition result as it is, which may cause display deviation or discomfort in the display. In addition, with the position accuracy of the popular in-vehicle navigation, the forward / backward / left / right position to be guided may be shifted by about 10 m. Specifically, such position deviation occurs in a situation in which, for example, a high-rise building, a tunnel, a multi-storey parking lot, etc. make it difficult or impossible to receive a positioning signal from a satellite such as a GPS. In addition, such position deviation occurs, for example, when the actual road shape has changed from the map information, and the vehicle position is estimated with deviation when matching the map with the map. A typical example is that image recognition accuracy tends to decrease at long distances (for example, about 100 m ahead), and correspondingly, superimposed displays of distant targets are prone to display misalignment or an unnatural appearance.

[0028] In this regard, the inventor has found that it is possible to suppress display misalignment and discomfort as much as possible by expressing the image recognition accuracy in the form of reliability and controlling the display of the display image IM according to the reliability. Specifically, for example, the display state of the display image IM can be set according to the reliability by correcting (for example, multiplying) the calculated value or estimated value of the target information corresponding to the image recognition result by the reliability, or by performing a display control threshold determination according to the reliability. This reliability can be calculated using the image recognition result or feature amount information, which is intermediate information generated when deriving the image recognition result. In addition, for example, by using an integrated reliability obtained by integrating the reliability obtained by different feature amount information using any integration means (for example, Bayesian estimation, etc.), it is possible to further suppress display misalignment and discomfort. In this way, according to this embodiment, superimposed display with less display misalignment and discomfort is possible.

[0029] (First embodiment: road gradient) FIG. 3, FIG. 4A, and FIG. 4B show an example of a technique for suppressing deviation of road gradient in route display. That is, FIG. 3 shows an example of an operation in the reliability calculation unit 125 shown in FIG. 2. FIG. 4A shows a display example in a comparative example in which a display image IM for displaying route information taking into account road gradient is simply displayed based on a recognition result without using reliability. FIG. 4B shows a display example according to this embodiment. In FIG. 4A and FIG. 4B, the vehicle is assumed to be approaching an intersection IS ahead while traveling straight on a traveling lane LR, which is the right lane of a traveling road RR having two lanes in each direction. Hereinafter, a display example according to this embodiment will be described with reference to FIG. 1 to FIG. 3 and FIG. 4A and FIG. 4B.

[0030] In this embodiment, the recognition unit 14 recognizes the road shape of the road ahead of the vehicle. That is, the recognition unit 14 calculates, that is, estimates, the road gradient of the road ahead of the vehicle. The recognition result acquisition unit 121 acquires the estimation result of the road gradient of the road ahead of the vehicle from the recognition unit 14. The display information generation unit 124 generates display information corresponding to the route display image, which is a display image IM for displaying the travel route of the vehicle, taking into account the estimation result of the road gradient. The reliability calculation unit 125 calculates the reliability of the estimation result of the road gradient. Then, the display state setting unit 126 sets the display state of the route display image based on the reliability of the estimation result of the road gradient. Specifically, for example, the display state setting unit 126 sets the display state so that the visibility of the route display image increases as the reliability increases. Alternatively, for example, the display state setting unit 126 sets the display area and the non-display area depending on whether the reliability exceeds a reliability threshold.

[0031] Furthermore, in this embodiment, as shown in FIG. 3, the reliability calculation unit 125 calculates an integrated reliability P by integrating a plurality of reliability (i.e., a first reliability P1, a second reliability P2, ...). The first reliability P1 is obtained using a distribution distance, which is one type of first feature information among a plurality of types of feature information, and a first reliability map prepared corresponding to the distribution distance. The first reliability map as the first correspondence information is a map that defines the correspondence between the distribution distance, which is the first feature information, and the first reliability P1, and is represented by P1=f1 (distribution distance). The second reliability P2 is obtained using a distance score, which is the other type of second feature information among a plurality of types of feature information, and a second reliability map prepared corresponding to the distance score. The second reliability map as the second correspondence information is a map that defines the correspondence between the distance score, which is the second feature information, and the second reliability P2, and is represented by P2=f2 (distance score). The number of distance points, which is one of the intermediate information, is the number of distance points used in the quadratic curve approximation of the road gradient, that is, the number of feature points whose distance, that is, the relative three-dimensional position from the vehicle, is calculated by image recognition. The distribution distance, which is one of the intermediate information, is the 95th percentile distance of the number of distance points used in the gradient estimation. For example, the distribution distance indicates the spread in the depth direction of the distance points used in the estimation, and as shown in FIG. 3, the longer the distribution distance, the higher the estimation accuracy tends to be. As for the number of distance points, if there is a certain number or more, the estimation is possible. Also, the more the distance points, the higher the estimation accuracy tends to be. In addition to the above-mentioned distribution distance and number of distance points, the feature amount information that can be used when calculating the reliability of the road gradient estimation result includes the estimation error, vehicle speed, day and night, etc. The above map is created by experiments, computer simulations, etc., and is stored in advance in memory.

[0032] The integration of the multiple reliabilities is performed, for example, using the following formula 1. Specifically, the first reliability P1 and the second reliability P2 are integrated into an integrated reliability P 12 In formula 1, (a, b)=(1, 2), i.e., Pa=P1 and Pb=P2. In addition, the integrated reliability P 123In equation 1, Pa = P 12 , Pb=P3. The integrated reliability P 1234 In equation 1, Pa = P 123 , Pb=P4. Integration of five or more confidence levels is performed in the same manner. That is, integration of three or more confidence levels can be performed by repeating the integration of two confidence levels according to Equation 1. Formula 1:P ab (Pa,Pb) =Pa·Pb / {(Pa·Pb)+(1-Pa·(1-Pb)}

[0033] In this way, the likelihood of the gradient estimation being correct (i.e., reliability) according to the number of distance points and the distribution distance is mapped, and these reliability values ​​are integrated and reflected in the gradient display mode in the route display, thereby enabling the gradient display to have less deviation and discomfort. Specifically, the road gradient estimation result is affected by the recognition result of the white line, i.e., the road dividing line LD. That is, for example, if the road dividing line LD is scratched or the road dividing line LD is cut off by an intersection IS, the accuracy of the road gradient estimation result based on the calculation result of the curvature of the road dividing line LD decreases. In addition, the road gradient estimation result based on the curvature of the road dividing line LD that is accurately recognized relatively close to the vehicle does not necessarily match the actual road gradient in the distance. For this reason, if the road dividing line LD is not recognized well in the distance, a large error occurs in the estimation result of the road gradient in the distance. Then, as shown in FIG. 4A, a large deviation occurs in the route display in the distance area. Therefore, by hiding the display or reducing the visibility in the distant area where the calculated value of the integrated reliability P is low, it is possible to perform route display with less deviation and discomfort. In the display example shown in FIG. 4B, the distant area where the reliability, i.e., the integrated reliability P, is less than the threshold value Pth is hidden, but the present invention is not limited to this aspect. That is, for example, the display image IM may be displayed so that the visibility, such as brightness, changes stepwise according to the reliability value.

[0034] (Second embodiment: road curvature) Fig. 5, Fig. 6A, and Fig. 6B show an example of a technique for suppressing deviation of road curvature (i.e., curve curvature) in route display. That is, Fig. 5 shows an example of an operation of the reliability calculation unit 125 shown in Fig. 2. Fig. 6A shows, as a comparative example, a display example in which a display image IM for displaying route information taking into account road curvature is simply displayed based on a recognition result without using reliability. Fig. 6B shows a display example according to this embodiment.

[0035] In this embodiment, the recognition unit 14 recognizes the road shape of the road ahead of the vehicle. That is, the recognition unit 14 calculates, i.e., estimates, the road curvature of the road ahead of the vehicle. The recognition result acquisition unit 121 acquires the estimation result of the road curvature of the road ahead of the vehicle from the recognition unit 14. The display information generation unit 124 generates display information corresponding to the route display image, which is a display image IM for displaying the travel route of the vehicle, taking into account the estimation result of the road curvature. The reliability calculation unit 125 calculates the reliability of the estimation result of the road curvature. Then, the display state setting unit 126 sets the display state of the route display image based on the reliability of the estimation result of the road curvature. Specifically, for example, the display state setting unit 126 sets the display state so that the visibility of the route display image increases as the reliability increases. More specifically, for example, when the reliability is high, the display is made darker, and when the reliability is low, the display is made lighter or off.

[0036] Relying only on the recognition result cannot reduce the display misalignment. On the other hand, if the navigation device 16 is a widespread model, the map information and vehicle position information used therein are not accurate enough to display the route without misalignment. Even if the navigation device 16 uses high-precision digital map information and high-precision positioning information that conforms to the ADASIS standard, reception problems with positioning signals may occur while driving, as described above. ADASIS stands for Advanced Driver Assistance Systems Interface Specification.

[0037] In this regard, the inventors have noticed that by focusing on intermediate information of the image recognition result, it is possible to predict a state in which display deviation or discomfort is likely to occur. Therefore, in this embodiment, the reliability of lateral deviation in the depth direction is calculated based on the road shape (i.e., for example, Bspline model) estimated by KF and DNN edge points, and a display is performed according to the reliability. KF is an abbreviation for Kalman filter. DNN is an abbreviation for Deep Neural Network. The edge points are feature points on the boundary between the road division line LD and other parts. The DNN edge points are edge points recognized using DNN technology. Specifically, for example, the longer the recognition distance, which is the depth direction position of the recognition, i.e., the calculation, of the road curvature, the lower the estimation accuracy. For this reason, the reliability can be obtained using a map created so that the longer the recognition distance, the lower the estimation accuracy.

[0038] Further, as shown in FIG. 5, in this embodiment, the reliability calculation unit 125 calculates the integrated reliability P by integrating multiple reliability, as in the first embodiment. The integration of multiple reliability is performed, for example, using the above formula 1. Examples of feature information that can be used when calculating the reliability of the estimation result of the road curvature include the recognition distance, the edge point lateral position error, the preceding vehicle distance, the lane width, and the road edge lateral position error. The edge point lateral position error is the lateral position difference between the road shape estimated by the KF and the DNN edge point, and is shown as "lateral position difference" in FIG. 5 for the sake of simplicity. The preceding vehicle distance is the distance between the host vehicle and the preceding vehicle. That is, when the calculation value of the lateral position based on the image recognition result of the preceding vehicle is used to estimate the road curvature, if the preceding vehicle is far away from the host vehicle, the accuracy of the estimation result decreases. Therefore, by reflecting the preceding vehicle distance in the reliability, it is possible to calculate the reliability with higher accuracy.

[0039] As shown in FIG. 6A, if the image recognition result is used as it is to display the display image IM for route display reflecting the road curvature, deviation and discomfort will be large. In contrast, as shown in FIG. 6B, in the distant area where the calculated value of the integrated reliability P is low, the display is hidden or the visibility is reduced, making it possible to perform route display with less deviation and discomfort. In the display example shown in FIG. 6B, the distant area where the reliability, i.e., the integrated reliability P, is less than the threshold value Pth is hidden, but the present invention is not limited to such an embodiment. That is, for example, the display image IM may be displayed so that the visibility, such as brightness, changes stepwise according to the reliability value.

[0040] (Third embodiment: POI) This embodiment shows an example of displaying a display image IM for emphasizing and notifying information about a POI, which is a target that should be noticed by a driver D. POI is an abbreviation for Point Of Interest, and examples of POI include pedestrians, preceding vehicles, traffic signs, road markings, etc.

[0041] FIG. 7 shows a reliability map corresponding to each of the recognition distance, DNN recognition score, tracking state, and cumulative tracking count as feature information. As shown in the map (A) in FIG. 7, the reliability is increased as the recognized distance is closer. Also, as shown in the map (B) in FIG. 7, the reliability is increased as the score value obtained as the identification result by the DNN is higher. Also, as shown in the map (C) in FIG. 7, the reliability is decreased when a ghost state, that is, a temporary unrecognized state due to an obstruction, etc. occurs. Also, as shown in the map (D) in FIG. 7, the reliability is increased according to the cumulative number of detections. The reliability in this embodiment can be obtained using at least one of these reliability maps. The process of calculating the integrated reliability P using multiple reliability maps can be performed in the same manner as each of the above embodiments.

[0042] Furthermore, when the superimposed display target of the display image IM is a pedestrian and any of the following situations apply, the brightness may be increased more than usual or the display image may be controlled to blink. When you are near a pedestrian crossing When you are at a traffic light, a stop line, or an intersection (between a stop line and a traffic light that is a certain distance away). Furthermore, when the display image IM is to be superimposed on a traffic sign, the brightness may be increased or the image may be controlled to blink when the display image IM is a traffic sign or other regulatory indication that requires special attention, such as a stop sign, no entry sign, or speed sign.

[0043] (Fourth embodiment: vehicle position within lane) This embodiment shows an example of calculating the reliability of the estimation result of the lateral position information corresponding to the vehicle position in the lane. The vehicle position in the lane is the position of the vehicle in the road width direction in the lane LR in which the vehicle is traveling. The route display is displayed based on the center of the lane LR in which the vehicle is traveling, not the center of the vehicle. Therefore, by performing the display according to the recognition reliability of the vehicle position in the lane, it is possible to display the route with less discomfort. Specifically, for example, a display control may be performed to cancel the display by setting a threshold value for the reliability.

[0044] FIG. 8 shows a reliability map corresponding to the total length of the white lines and the recognition state of the white lines as feature information. "White line" is an abbreviation of road dividing line LD. The lateral position of the vehicle in the lane depends on the recognition state of the left and right white lines. For this reason, as shown in the map in FIG. 8(A), the reliability is increased as the total length of the white lines is longer. Also, as shown in the map in FIG. 8(B), the reliability is set as follows according to the recognition state of the left and right white lines of the vehicle: · Recognize both left and right: High reliability (e.g. 90%) - Only detects left or right side: Lane width cannot be detected and relies on history, so reliability is medium (e.g. 25%) · If both the left and right are not recognized...Position calculation is impossible, so the reliability is lowest (for example, 1%)

[0045] (Fifth embodiment: lane number) This embodiment shows an example of calculating the reliability of the estimation result of the lateral position information corresponding to the lane number. The lane number is an ID number indicating the position of the lane LR in which the vehicle is traveling in the road width direction on the road RR. For example, the lane numbers are assigned in ascending order from 1 from right to left. The lane number is necessary to prompt the vehicle to move to an appropriate lane position in advance and to display guidance information related to landmarks such as stores at an appropriate lateral position from the vehicle in order to smoothly guide the route without difficulty.

[0046] Identification of lane numbers, which can be realized only by camera recognition, can be performed by considering the lane width and the width of the road shoulder based on (1) the line type pattern of the white lines recognized by the camera (e.g., a solid line on the left and a dashed line on the right) and (2) the lateral offset distance from the road edge recognized by the camera (e.g., the bottom edge of the road shoulder, etc.) to the vehicle. In reality, (1) the line type results output by white line recognition are discrete and unstable information, while (2) the road edge recognition may or may not be possible because it may be blocked by surrounding vehicles. Therefore, as shown in the map (A) in FIG. 9, the standard deviation is taken for the value for which the lane was identified within a certain period of time, and the temporal fluctuation of the lane number is regarded as the reliability. If there are times when the lane could not be identified within a certain period of time, a predetermined value of standard deviation is added according to the number of times, and the state is changed to one with large fluctuations, that is, the lane number is not reliable. In addition, since road edge recognition is often hindered by the number of surrounding vehicles within a predetermined distance (e.g., 30 m) ahead of the vehicle in the left and right lanes adjacent to the vehicle, the number of adjacent vehicles on the left and right within a predetermined distance is also included in the reliability map, as shown in (B) and (C) of FIG. 9. The reliability in this embodiment can also be obtained using at least one of these reliability maps. The process of calculating the integrated reliability P using multiple reliability maps can be performed in the same manner as in each of the above embodiments.

[0047] (Sixth embodiment: intersection position) This embodiment shows an example of calculating the reliability of an intersection position estimation result. The intersection position is the relative position of the center of the intersection IS with respect to the vehicle. The intersection position to be estimated is typically the forward / rearward position, i.e., the planned travel distance from the current position of the vehicle in the current traveling direction of the vehicle.

[0048] In the position accuracy of conventional in-vehicle navigation, the front and rear positions of intersections IS that are targets for guidance on the route or travel direction may be shifted. This is caused by, for example, a decrease in the accuracy of the estimation of the vehicle's position when a positioning signal such as a GPS cannot be received or is difficult to receive in a high-rise building, a tunnel, or a multi-story parking lot. Alternatively, this is caused by, for example, an error in the estimation of the vehicle's position due to the actual road shape being changed from the map when map matching is performed with the map. In this way, when an error occurs in the estimation or calculation of the vehicle's position, an error also occurs in the relative positional relationship from the vehicle's position of the intersection IS that is a target for guidance on the route or travel direction. Then, a display element for guiding the route or travel direction that should be displayed corresponding to the intersection IS is displayed at a position shifted from the intersection IS. Therefore, by estimating or calculating the intersection position using the camera recognition result, it is possible to display a route without any shift or discomfort.

[0049] When analyzing an intersection IS with traffic lights, the things that can be recognized by a camera are a traffic light TL and a stop line LL, with the traffic light TL permanently installed on the back side of the intersection IS and the stop line LL permanently installed on the front side of the intersection IS. Therefore, the logic is set up so that the midpoint of the traffic light TL on the back side of the intersection IS and the stop line LL on the front side of the intersection IS is assumed to be the center of the intersection IS, and the front-rear position is calculated. In addition, the lateral position of the center of the intersection IS is calculated based on the lane ID, which is the lane number of the lane LR in which the vehicle is traveling. Note that if a traffic light TL is also installed on the front side of the intersection IS, the estimated position based on the image recognition results may be reflected in the estimation of the intersection position.

[0050] Here, if the intersection IS cannot be detected after it has started to be detected once, it is called a ghost, and if it cannot be detected a predetermined number of times in succession (for example, four times), the intersection position is reset. If the intersection IS is too close to be recognized, the vehicle speed and yaw rate are displayed as a complement to the display of the vehicle.

[0051] As described above, the front and rear positions of the intersection IS are characterized by being dependent on the state of recognition of the traffic light TL and the stop line LL. For this reason, the reliability is estimated by a reliability map using the following feature amount information. FIG. 10 shows an example of these reliability maps. It is to be noted that not only the reliability of the camera recognition but also the reliability of the vehicle position by the navigation device 16 may be reflected. The reliability in this embodiment may also be obtained by using at least one of these reliability maps. The process of calculating the integrated reliability P using multiple reliability maps may be performed in the same manner as in each of the above embodiments. -DNN recognition score value of stop line LL, confidence according to tracking state - Traffic light TL DNN recognition score value, confidence according to tracking state -Reliability based on recognition distance: the closer, the more reliable Accumulative detection count: The more, the more trustworthy

[0052] (Seventh embodiment: reliability setting based on scene) This embodiment shows an example in which the driving scene, i.e., the driving situation of the vehicle, is reflected in the reliability calculation. When the route display by the head-up display device 11 is controlled by camera recognition, the display may be displaced in a scene where the camera recognition is poor. Therefore, in such a scene, the reliability is lowered in advance before the display displacement occurs, so that an erroneous display is not made. That is, in this embodiment, when the image recognition result includes a driving situation of the vehicle that reduces the reliability, the reliability calculation unit 125 corrects the reliability in a direction to lower it.

[0053] Typical scenes where errors are likely to occur in the recognition of road shapes (i.e., road gradient and road curvature) by a camera include, for example, approaching an intersection IS, backlighting, night, etc. Regarding road curvature, errors are likely to occur in the recognition of road shapes even in scenes where a vehicle is approaching a fork. Specifically, at an intersection IS, there are many road markings such as arrows, and the white lines are often faded. Furthermore, since there are no white lines inside the intersection IS, errors or failure to recognize by the camera are likely to occur. Regarding a fork, if the camera recognizes the white line on the fork side, it will be displayed incorrectly. Regarding backlighting, the tar repair marks and the road surface are shiny, which is likely to cause errors or failure to recognize by the camera. At night, the camera's exposure is high, and the pattern of the black parts on the road surface becomes difficult to see due to blurring and noise, which makes it easy to make mistakes in estimating the road gradient and road curvature.

[0054] Therefore, as shown in the map (A) in FIG. 11, the closer the distance to the intersection IS based on the image recognition result of the intersection position, the lower the reliability. Also, as shown in the map (B) in FIG. 11, the closer the distance to the intersection IS or the branch on the map information, the lower the reliability. Also, as shown in the map (C) in FIG. 11, the greater the amount of whiteout area in the upper half of the camera image, the lower the reliability. Also, as shown in the map (D) in FIG. 11, the greater the exposure of the camera, the lower the reliability. The reliability in this embodiment can also be obtained using at least one of these reliability maps. The process of calculating the integrated reliability P using multiple reliability maps can be performed in the same manner as in each of the above embodiments.

[0055] (Eighth embodiment: reliability setting based on recognition instability determination) In this embodiment, an example is shown in which display output is suppressed when the recognized road shape is unstable, such as a snowy road, an unprepared road, or a rainy road surface reflection scene. That is, in this embodiment, the amount of fluctuation in the recognition result, such as the recognized horizontal position of the white line and the estimated curve radius, is used as a feature, and the "unstable state of the road shape," which means whether the road shape is unstable, is probabilistically estimated, and display is suppressed based on this. Figures 12 to 14 show an overview of the reliability calculation and display state control in this embodiment. In FIG. 12, "likelihood" indicates the degree of instability of the road shape recognition, and is also called "instability degree." This "instability degree" is the opposite concept of reliability in each of the above embodiments, and corresponds to a value obtained by subtracting the reliability degree from 100%. The vertical axis P' of each map in FIG. 14 indicates instability degree.

[0056] Specifically, as shown in Fig. 12, the reliability calculation unit 125 calculates the amount of fluctuation in instantaneous likelihood, which is the instantaneous value of likelihood, for each feature amount, and calculates an integrated likelihood by integrating multiple instantaneous likelihoods. The method of integrating multiple likelihoods is the same as the method of integrating reliability in each of the above-mentioned embodiments. In addition, the reliability calculation unit 125 improves the calculation accuracy of the integrated likelihood by linear prediction using a so-called "weighted regression" method or the like. That is, the display information generation unit 124 generates display information corresponding to the route display image based on the road shape estimation result by weighted regression using the reliability of the road shape estimation result.

[0057] The display state setting unit 126 sets the display state of the display image IM according to the integrated likelihood calculated by the reliability calculation unit 125, i.e., the fluctuation state of the image recognition result of the target. Specifically, as shown in FIG. 13, the display state setting unit 126 turns on the display suppression (e.g., non-display) when the instability, i.e., the integrated likelihood, becomes equal to or greater than the upper threshold THH (e.g., 90%), and then turns off the display suppression and redisplays when the instability becomes less than the lower threshold THL (e.g., 50%). After time t1 when the integrated likelihood becomes equal to or greater than the upper threshold THH and the display suppression is turned on, the display suppression is maintained in the on state until time t2 when the integrated likelihood becomes less than the lower threshold THL, even if the integrated likelihood becomes less than the upper threshold THH. In this way, the display state setting unit 126 performs hysteresis control of the display state using the upper threshold THH and the lower threshold THL.

[0058] The method of determining the degree of instability is as follows. For white line recognition, the feature information includes the white line detection flag variation, the white line estimation R variation, and the white line lateral position variation. For the white line detection flag variation, as shown in the map in FIG. 14A, the recognition is deemed unstable if the detection / non-detection is repeated. For the white line estimation R variation, as shown in the map in FIG. 14B, the recognition is deemed unstable if the variation in the direction of the recognized white line is large. For the white line lateral position variation, as shown in the map in FIG. 14C, the recognition is deemed unstable if the variation in the recognized white line position is large. For the road gradient, the feature information includes the road gradient estimation success flag variation and the variation in the estimated value of the road gradient. For the road gradient estimation success flag variation, as shown in the map in FIG. 14D, the estimation is deemed unstable if the estimation is repeated successful / unsuccessful. Regarding the amount of change in road gradient, as shown in maps (E) and (F) in Fig. 14, if the amount of change in the estimated gradient is large, the estimation is deemed unstable, and separate maps are used for longitudinal and transverse travel. Alternatively, for example, if it is possible to determine whether or not recognition is possible based on the camera output without using the recognition result information (for example, the entire screen remains completely white, the camera temperature is abnormally high, the rain is too heavy and visibility is poor, the vehicle speed is too fast and operation is not guaranteed, etc.), this may be used to obtain the degree of instability.

[0059] (Modification) The present invention is not limited to the above embodiment. Therefore, the above embodiment can be modified as appropriate. Representative modified examples will be described below. In the following description of the modified examples, differences from the above embodiment will be mainly described. In addition, the same reference numerals are used for parts that are the same or equivalent in the above embodiment and the modified examples. Therefore, in the following description of the modified examples, the description of the above embodiment can be appropriately used for components having the same reference numerals as the above embodiment, unless there is a technical contradiction or special additional explanation.

[0060] The present invention is not limited to the specific device configuration shown in the above embodiment. That is, for example, the vehicle 1 equipped with the in-vehicle system 10 is not limited to a normal car. Specifically, the vehicle 1 may be a large car such as a cargo truck. There is no particular limitation on the number of wheels, and the vehicle may be a three-wheeled car, or a six- or eight-wheeled car such as a cargo truck. The type of the vehicle 1 may be a conventional car equipped with only an internal combustion engine, an electric car or a fuel cell car without an internal combustion engine, or a so-called hybrid car. The shape and structure of the body 2 of the vehicle 1 are also not limited to a box shape, that is, a substantially rectangular shape in a plan view. There are no particular limitations on the use of the vehicle 1, the position of the steering wheel 8, the number of occupants, etc. That is, the driver D may be an occupant of the vehicle who is in charge of or performs a dynamic driving task. In other words, as long as the driver D can perform driving operations, there is no particular limitation on the seating position of the driver D. In addition, instead of or together with the steering wheel 8, any operation device such as a joystick may be used.

[0061] The communication standard for the in-vehicle system 10 may be other than CAN (international registered trademark), such as FlexRay (international registered trademark). The communication standard for the in-vehicle system 10 is not limited to one type. For example, the in-vehicle system 10 may have a sub-network line that complies with a communication standard such as LIN. LIN is an abbreviation for Local Interconnect Network.

[0062] The display control device 12 is not limited to a configuration as a so-called in-vehicle computer having a processor and a memory. Specifically, for example, all or a part of the display control device 12 may be configured to include a digital circuit configured to enable the above-mentioned operation, such as an ASIC or an FPGA. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0063] The program according to the present invention, which enables the execution of various operations, procedures, or processes described in the above embodiment, can be stored in a computer-readable non-transient tangible storage medium as instructions executed by a computer. That is, each of the above functional configurations and methods can be expressed as a computer program including procedures for realizing the same, or a non-transient tangible storage medium storing the program. Such a program can be downloaded or upgraded via V2X communication. V2X is an abbreviation for Vehicle to X. Alternatively, such a program can be downloaded or upgraded via a terminal device provided in a manufacturing plant, a maintenance plant, a dealer, or the like of the vehicle 1. The storage destination of such a program may be a memory card, an optical disk, a magnetic disk, or the like.

[0064] The program according to the present invention, which enables the execution of various operations, procedures, or processes described in the above embodiments, can be downloaded or upgraded via V2X communication. V2X is an abbreviation for Vehicle to X. Alternatively, such a program can be downloaded or upgraded via a terminal device provided in a manufacturing plant, a maintenance plant, a dealer, or the like of the vehicle. The storage destination of such a program may be a memory card, an optical disk, a magnetic disk, or the like.

[0065] The head-up display device 11 is not limited to a type that projects onto the front windshield 4. That is, for example, the head-up display device 11 may be of a so-called combiner type. The combiner type head-up display device 11 may be of a so-called retrofit type.

[0066] The image capturing device 13 as a camera is not limited to a type fixedly mounted on the vehicle body 2 or the vehicle interior 3. That is, for example, a camera mounted on a so-called aftermarket drive recorder may be used as the image capturing device 13. Alternatively, for example, a camera mounted on a mobile terminal brought into the vehicle interior 3 by an occupant (for example, an out-camera mounted on a so-called smartphone) may be used as the image capturing device 13. In this case, instead of a driver status monitor (not shown) for detecting the viewpoint of the driver D or a camera for capturing images inside the vehicle interior, another camera mounted on the mobile terminal (for example, an in-camera mounted on a so-called smartphone) may be used. Alternatively, instead of a camera, a LiDAR may be used as the image capturing device 13.

[0067] The recognition unit 14 and the recognition result acquisition unit 121 may be integrated. That is, the recognition result acquisition unit 121 may be configured to perform image recognition of a target included in the foreground based on an image of the foreground captured by the image capturing device 13. In other words, the present invention may be suitably applied to a vehicle 1 that does not include an automatic driving system or an ADAS system that utilizes an image recognition result of the foreground captured by the image capturing device 13 for driving control or the like.

[0068] The navigation device 16 is not limited to a type fixedly attached to the dashboard 5. That is, for example, a mobile terminal brought into the vehicle compartment 3 by a passenger may be used as the navigation device 16. In this case, as described above, a camera equipped in the mobile terminal may be used as the image capturing device 13 and / or a sensor for detecting the viewpoint of the driver D.

[0069] The present invention is not limited to the specific operation modes or operation examples shown in the above embodiment. For example, the reliability or the integrated reliability may be subjected to low-pass filter processing. Moreover, instead of or in addition to adjusting the display form, the shape of the display image IM itself may be corrected. Specifically, for example, the current calculated value of the road gradient may be calculated as road gradient = estimated value by image recognition x reliability, and low-pass filter processing may be performed as road gradient = α x current calculated value + (1 - α) x past value. The same applies to the road curvature, etc. Moreover, even if the route display by the display image IM in the head-up display device 11 is suppressed, the route display in the navigation device 16 may not be suppressed.

[0070] The present invention is not limited to the specific processing modes shown in the above embodiment. That is, for example, "above threshold" and "exceed threshold" can be appropriately replaced within a range that is not technically inconsistent. Similarly, "below threshold" and "below threshold" can be appropriately replaced within a range that is not technically inconsistent.

[0071] So-called "weighted regression" can be applied to the road shape estimation in each of the above-mentioned embodiments. By performing road shape estimation using more reliable recognition points, the road shape estimation accuracy is improved, and superimposed display with less display deviation and discomfort is possible. Note that the details of the "weighted regression" itself are already publicly known or well-known at the time of filing this application, so a description thereof will be omitted in this specification.

[0072] Mutually similar expressions such as "obtain," "estimate," "detect," "sensing," "calculate," "extract," and "generate" may be substituted as appropriate within the scope not causing any technical contradiction.

[0073] Needless to say, the elements constituting the above-mentioned embodiment are not necessarily essential, except when expressly stated as essential or when considered to be obviously essential in principle. Furthermore, when the numbers, amounts, ranges, etc. of the components are mentioned, the present invention is not limited to the specific values, except when expressly stated as essential or when clearly limited to a specific value in principle. Similarly, when the shapes, directions, positional relationships, etc. of the components are mentioned, the present invention is not limited to the shapes, directions, positional relationships, etc., except when expressly stated as essential or when clearly limited to a specific shape, direction, positional relationship, etc. in principle.

[0074] The modified examples are not limited to the above examples. A plurality of modified examples may be combined with each other. The above-mentioned embodiments may be applied in a redundant manner. All or a portion of the above-mentioned embodiments and all or a portion of the modified examples may be combined with each other. [Explanation of symbols]

[0075] 1 Vehicle 11 Head-up display device (display device) 12 Display control device 13. Image capture device 121 Recognition result acquisition unit 122 Vehicle position acquisition unit 123 Route information acquisition unit 124 Display information generation section 125 Center position estimation section TD direction display

Claims

1. A display control device (12) configured to control a display operation of a display device (11) that is mounted on a vehicle (1) and superimposes a display image (IM) on a foreground of the vehicle, a recognition result acquisition unit (121) for acquiring recognition result information corresponding to an image recognition result of a target based on an image of the foreground captured by an image capturing device (13); a reliability calculation unit (125) that calculates the reliability of the recognition result information based on intermediate information in the process of acquiring the recognition result information or feature amount information that is the recognition result information; a display state setting unit (126) that sets a display state of the display image based on the reliability calculated by the reliability calculation unit; a display information generating unit (124) that generates display information for displaying the display image on the display device based on the recognition result information; Equipped with the reliability calculation unit calculates the reliability of an estimation result of a road shape, which is a road gradient or a road curvature of a road ahead of the host vehicle; the display information generation unit generates the display information corresponding to a route display image, which is the display image for displaying a travel route of the host vehicle, taking into account the estimation result of the road shape; the display state setting unit sets the display state in the route display image based on the reliability of the estimation result of the road shape. Display control device.

2. The reliability calculation unit is acquiring the first reliability by using first correspondence information that is prepared corresponding to a first feature amount information that is one type of the plurality of types of feature amount information and that defines a correspondence relationship between the first feature amount information and a first reliability; acquiring the second reliability by using second correspondence information that is prepared corresponding to another type of second feature information among the plurality of types of feature information and that defines a correspondence relationship between the second feature information and the second reliability; calculating the reliability by integrating the acquired first reliability and the acquired second reliability; The display control device according to claim 1 .

3. the display state setting unit sets the display state such that the visibility of the display image increases as the reliability increases. The display control device according to claim 1 .

4. The reliability calculation unit corrects the reliability in a direction lowering the reliability when the image recognition result includes a traveling state of the host vehicle that lowers the reliability. The display control device according to any one of claims 1 to 3.

5. The display state setting unit sets the display state according to a change state of the image recognition result of the target. The display control device according to any one of claims 1 to 4.

6. the display information generation unit generates the display information corresponding to the route display image based on the estimation result of the road shape by weighted regression using the reliability of the estimation result of the road shape. The display control device according to any one of claims 1 to 5.

7. the reliability calculation unit calculates the reliability of an estimation result of lateral position information corresponding to a position in a road width direction of the road on which the host vehicle is traveling, the display state setting unit sets the display state in the display image for displaying guidance information related to features around the vehicle based on the reliability of the estimation result of the lateral position information. The display control device according to any one of claims 1 to 6.

8. A display control program executed by a display control device (12) configured to control a display operation of a display device (11) that is mounted on a vehicle (1) and superimposes a display image (IM) on a foreground of the vehicle, the display control program comprising: The process executed by the display control device includes: A recognition result acquisition process for acquiring recognition result information corresponding to an image recognition result of a target based on an image of the foreground captured by an image capturing device (13); a reliability calculation process for calculating reliability of the recognition result information based on intermediate information in a process of acquiring the recognition result information or feature amount information which is the recognition result information; a display state setting process for setting a display state of the display image based on the reliability calculated by the reliability calculation process; a display information generating process for generating display information for displaying the display image on the display device based on the recognition result information; Including, the reliability calculation process calculates the reliability of an estimation result of a road shape, which is a road gradient or a road curvature of a road ahead of the host vehicle; the display information generation process generates the display information corresponding to a route display image, which is the display image for displaying a travel route of the host vehicle, taking into account the estimation result of the road shape; the display state setting process sets the display state in the route display image based on the reliability of the estimation result of the road shape. Display control program.

9. The reliability calculation process includes: acquiring the first reliability by using first correspondence information that is prepared corresponding to a first feature amount information that is one type of the plurality of types of feature amount information and that defines a correspondence relationship between the first feature amount information and a first reliability; acquiring the second reliability by using second correspondence information that is prepared corresponding to another type of second feature information among the plurality of types of feature information and that defines a correspondence relationship between the second feature information and the second reliability; calculating the reliability by integrating the acquired first reliability and the acquired second reliability; The display control program according to claim 8.

10. The display state setting process sets the display state such that the visibility of the display image increases as the reliability increases.

10. The display control program according to claim 8.

11. When the image recognition result includes a driving condition of the vehicle that reduces the reliability, the reliability calculation process corrects the reliability to a lower level. The display control program according to any one of claims 8 to 10.

12. The display state setting process sets the display state according to a change state of the image recognition result of the target. The display control program according to any one of claims 8 to 11.

13. The display information generation process generates the display information corresponding to the route display image based on the estimation result of the road shape by weighted regression using the reliability of the estimation result of the road shape. The display control program according to any one of claims 8 to 12.

14. The reliability calculation process calculates the reliability of an estimation result of lateral position information corresponding to a position in a road width direction of the road while the vehicle is traveling, the display state setting process sets the display state in the display image for displaying guidance information related to features around the vehicle, based on the reliability of the estimation result of the lateral position information. The display control program according to any one of claims 8 to 13.

Images