Vehicle control device, vehicle control method, and vehicle control program

Abstract

A vehicle control device (1) for controlling a vehicle equipped with an optical sensor (40) for acquiring a reflected light image (Ior) by sensing reflected light irradiated with light, and a sensing camera (41) for acquiring a camera image (Ic) corresponding to the intensity of external light in a sensing area overlapping with the optical sensor (40), comprises: an extraction unit (100) for extracting a mismatched pixel group (Po) by comparing the reflected light image (Ior) with the camera image (Ic); and a control unit (120) for issuing a control instruction corresponding to a water-related substance (Mw) estimated to correspond to the mismatched pixel group (Po) to the vehicle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Japanese Patent Application No. 2020-153238 filed in Japan on September 11, 2020, and Japanese Patent Application No. 2021-124655 filed in Japan on July 29, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to control techniques utilized by sensor systems in vehicles. Background Art

[0004] It is known that the presence of water-related substances such as rain, fog, or water vapor in the sensing area of ​​the vehicle's sensor system leads to a decrease in sensing accuracy. Therefore, the presence of water-related substances affects the control of the vehicle. In particular, in recent years, the presence of water-related substances in the vehicle's automatic driving mode has become an important control factor because it affects the continuity of automatic driving.

[0005] Patent Document 1 discloses a technique for determining dirt such as water droplets attached to an incident surface where light enters a sensing region in a sensor system. In this disclosed technique, dirt on the incident surface is determined based on changes in the intensity of reflected light in response to light irradiation.

[0006] Patent Document 1: Japanese Patent Application Publication No. 2005-10094

[0007] However, in the disclosed technology of Patent Document 1, the presence of the water-related substance cannot be determined unless the water-related substance adheres to the incident surface. Therefore, there is a concern that it is difficult to suppress the deterioration of the appropriateness of vehicle control due to the reduction in the sensing accuracy of the sensor system after the water-related substance adheres to the incident surface. Summary of the invention

[0008] The present disclosure provides a vehicle control device that improves the appropriateness of vehicle control. Another object of the present disclosure is to provide a vehicle control method that improves the appropriateness of vehicle control. Another object of the present disclosure is to provide a vehicle control program that improves the appropriateness of vehicle control.

[0009] Hereinafter, the technical means of the present disclosure for solving the problems will be described.

[0010] A first aspect of the present disclosure is a vehicle control device for controlling a vehicle equipped with an optical sensor for acquiring a reflected light image by sensing reflected light of light irradiation, and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor, comprising:

[0011] an extraction unit, extracting mismatched pixel groups by comparing the reflected light image and the camera image; and

[0012] The control unit issues a control instruction to the vehicle corresponding to the water-related substance estimated to correspond to the mismatched pixel group.

[0013] A second aspect of the present disclosure is a vehicle control method for controlling a vehicle equipped with an optical sensor for acquiring a reflected light image by sensing reflected light of light irradiation, and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor, comprising:

[0014] an extraction step, extracting mismatched pixel groups by comparing the reflected light image and the camera image; and

[0015] The control step issues a control instruction to the vehicle corresponding to the water-related substance estimated to correspond to the mismatched pixel group.

[0016] A third aspect of the present disclosure is a vehicle control program including instructions for causing a processor to execute in order to control a vehicle equipped with an optical sensor for acquiring a reflected light image by sensing reflected light irradiated with light, and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor.

[0017] The command contains:

[0018] an extraction step, causing the processor to extract mismatched pixel groups by comparing the reflected light image with the camera image; and

[0019] The control step causes the processor to issue a control instruction corresponding to the water-related substance estimated to correspond to the mismatched pixel group to the vehicle.

[0020] According to the first to third aspects, a reflected light image acquired by an optical sensor by sensing reflected light irradiated with light and a camera image acquired by a sensing camera based on the intensity of external light in a sensing area overlapping with the sensor are compared. It is possible to correctly estimate that the mismatched pixel group extracted by such comparison corresponds to a water-related substance that is easy to sense reflected light, that is, a water-related substance that is easy to be reflected in the reflected light image. Therefore, by issuing a control instruction corresponding to the water-related substance to the vehicle as a result of the estimation, the appropriateness of the vehicle control can be improved.

[0021] A fourth aspect of the present disclosure is a vehicle control device for controlling a vehicle equipped with an optical sensor that acquires a reflected light image by sensing reflected light from light irradiation and acquires an external light image corresponding to external light intensity when light irradiation is stopped, comprising:

[0022] an extraction unit, extracting mismatched pixel groups by comparing the reflected light image with the external light image; and

[0023] The control unit issues a control instruction to the vehicle corresponding to the water-related substance estimated to correspond to the mismatched pixel group.

[0024] A fifth aspect of the present disclosure is a vehicle control method for controlling a vehicle equipped with an optical sensor that acquires a reflected light image by sensing reflected light from light irradiation and acquires an external light image corresponding to external light intensity when light irradiation is stopped, comprising:

[0025] An extraction step, extracting mismatched pixel groups by comparing the reflected light image with the external light image; and

[0026] The control step issues a control instruction to the vehicle corresponding to the water-related substance estimated to correspond to the mismatched pixel group.

[0027] A sixth aspect of the present disclosure is a vehicle control program including instructions for causing a processor to execute in order to control a vehicle equipped with an optical sensor that acquires a reflected light image by sensing reflected light from light irradiation and acquires an external light image corresponding to the external light intensity when the light irradiation is stopped.

[0028] The command contains:

[0029] an extraction step, causing the processor to extract mismatched pixel groups by comparing the reflected light image with the external light image; and

[0030] The control step causes the processor to issue a control instruction corresponding to the water-related substance estimated to correspond to the mismatched pixel group to the vehicle.

[0031] According to the fourth to sixth aspects, a reflected light image acquired by an optical sensor by sensing reflected light for light irradiation and an external light image acquired by the sensor according to the external light intensity when the light irradiation is stopped are compared. It can be correctly estimated that the mismatched pixel group extracted by such comparison corresponds to a water-related substance that is easy to sense reflected light, that is, a water-related substance that is easy to be reflected in the reflected light image. Therefore, by issuing a control instruction corresponding to the water-related substance to the vehicle as a result of the estimation, the appropriateness of the vehicle control can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a side view showing a state where the automatic driving unit according to the first embodiment is mounted on a vehicle.

[0033] Figure 2 It is a cross-sectional schematic diagram showing the overall structure of the automatic driving unit according to the first embodiment.

[0034] Figure 3 This is a block diagram showing the functional configuration of the vehicle control device according to the first embodiment.

[0035] Figure 4 1 is a schematic cross-sectional view showing the optical sensor and the sensing range of the sensing camera according to the first embodiment.

[0036] Figure 5 It is a schematic diagram for explaining an image acquired by the optical sensor according to the first embodiment.

[0037] Figure 6 1 is a schematic diagram for explaining an image captured by the sensing camera according to the first embodiment.

[0038] Figure 7 1 is a schematic diagram for explaining an interpolated image of a sensing camera according to the first embodiment.

[0039] Figure 8 This is a flowchart showing the vehicle control method according to the first embodiment.

[0040] Fig. 9 It is a block diagram showing the functional configuration of a vehicle control device according to the second embodiment.

[0041] Fig.10 This is a flowchart showing a vehicle control method according to the second embodiment.

[0042] Fig.11 It is a block diagram showing the functional configuration of a vehicle control device according to the third embodiment.

[0043] Fig.12 It is a schematic diagram for explaining an image acquired by the optical sensor according to the third embodiment.

[0044] Fig.13 This is a flowchart showing a vehicle control method according to the third embodiment.

[0045] Fig.14 It is a block diagram showing the functional configuration of a vehicle control device according to a fourth embodiment.

[0046] Fig.15 This is a flowchart showing a vehicle control method according to a fourth embodiment.

[0047] Fig.16 It is a schematic diagram showing the detailed structure of a sensing camera according to the fifth embodiment.

[0048] Fig.17 It is a block diagram showing the functional configuration of a vehicle control device according to a fifth embodiment.

[0049] Fig.18 This is a flowchart showing a vehicle control method according to the fifth embodiment.

[0050] Fig.19 : is a flowchart showing a vehicle control method according to a modified example.

[0051] Fig. 20 : is a flowchart showing a vehicle control method according to a modified example.

[0052] Fig.21 : is a flowchart showing a vehicle control method according to a modified example.

[0053] Fig. 22 : is a flowchart showing a vehicle control method according to a modified example. DETAILED DESCRIPTION

[0054] Hereinafter, a plurality of embodiments are described based on the accompanying drawings. In addition, sometimes, by attaching the same reference numerals to the corresponding constituent elements in each embodiment, repeated descriptions are omitted. In addition, when only a part of the composition is described in each embodiment, the other parts of the composition can apply the composition of other embodiments described in advance. Moreover, not only the combination of the composition explicitly stated in the description of each embodiment, but also the composition of multiple embodiments can be partially combined with each other even if it is not explicitly stated, as long as the combination does not particularly cause an obstacle.

[0055] (First Embodiment)

[0056] like Figure 1 As shown, an automatic driving unit ADU having a vehicle control device 1 of the first embodiment is mounted on a vehicle 2. The vehicle 2 can always or temporarily drive automatically in an automatic driving mode based on autonomous driving control or advanced driving assistance control. The automatic driving unit ADU is configured to include a vehicle control device 1, a housing 3, a sensor system 4, and a cleaning system 5. In the following description, front, rear, left, right, top, and bottom are defined based on the vehicle 2 on a horizontal plane.

[0057] like Figure 1 , 2 The housing 3 is formed into a hollow flat box shape by metal or the like, for example. The housing 3 is disposed on the roof 20 of the vehicle 2. A plurality of sensor windows 31 are formed in the wall 30 of the housing 3. Each sensor window 31 is covered by a plate-shaped light-transmitting cover 32. The outer surface of each light-transmitting cover 32 constitutes an incident surface 33 for light to be incident from the outside of the vehicle 2.

[0058] like Figures 2 to 4As shown, the sensor system 4 includes an optical sensor 40. The optical sensor 40 is a so-called LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) that acquires optical information that can be used in the autonomous driving mode of the vehicle 2. The optical sensor 40 includes a light emitting element 400, an imaging element 401, and an imaging circuit 402.

[0059] The light emitting element 400 is, for example, a semiconductor element that generates directional laser light, such as a laser diode. The light emitting element 400 makes the laser irradiation toward the outside of the vehicle 2 be in the form of an intermittent pulse beam. The imaging element 401 is, for example, a semiconductor element that is highly sensitive to light, such as a SPAD (Single Photon Avalanche Diode). An incident surface 33o dedicated to the optical sensor 40 is arranged on the external side of the imaging element 401 (the front side in this embodiment). The element 401 is exposed by light incident on the incident surface 33o from the outside world in a sensing area Ao determined according to the viewing angle of the imaging element 401. The imaging circuit 402 is an integrated circuit that controls the exposure and scanning of multiple pixels in the imaging element 401, and processes and digitizes the signal from the element 401.

[0060] In the reflected light mode in which the imaging circuit 402 exposes the imaging element 401 by irradiating light from the light emitting element 400, the object point in the sensing area Ao becomes a reflection point of the laser. As a result, the laser light reflected at the reflection point (hereinafter referred to as reflected light) enters the imaging element 401 through the incident surface 33o. At this time, the imaging circuit 402 senses the reflected light by scanning a plurality of pixels of the imaging element 401. In particular, the imaging circuit 402 senses the reflected light by scanning a plurality of pixels of the imaging element 401 according to the following example. Figure 4 The reflection point distance dr of the reflected light sensed as shown in the figure is converted into three-dimensional data according to the distance value obtained for each of the plurality of pixels as each pixel value, and the obtained Figure 5 The distance image Iord as the reflected light image Ior is shown.

[0061] like Figures 2 to 4 The sensor system 4 shown in the figure also includes a sensing camera 41 in addition to the optical sensor 40. The sensing camera 41 is a so-called external camera that acquires optical information that can be used in the automatic driving mode in the vehicle 2. The sensing camera 41 includes an imaging element 411 and an imaging circuit 402.

[0062] The imaging element 411 is, for example, a semiconductor element such as a CMOS. An incident surface 33c dedicated to the sensing camera 41 is arranged on the external side (the front side in this embodiment) of the imaging element 411. The element 411 is exposed by light incident on the incident surface 33c from the external world in a sensing area Ac determined by the viewing angle of the imaging element 411. Figure 4 The sensing area Ac of the sensing camera 41 partially overlaps with the sensing area Ao of the optical sensor 40. The overlap rate of the sensing areas Ac and Ao, that is, the ratio of the overlapping area Aoc in these areas Ac and Ao, is, for example, 50% or more, preferably 70% or more, and more preferably 90% or more. The imaging circuit 412 is an integrated circuit that controls the exposure and scanning of multiple pixels in the imaging element 411 and processes the signals from the element 411 for data conversion.

[0063] In the exposure mode in which the imaging circuit 412 exposes the imaging element 411, the object point in the sensing area Ac becomes a reflection point of the external light. As a result, the external light reflected at the reflection point passes through the incident surface 33c and enters the imaging element 411. At this time, the imaging circuit 412 senses the reflected external light by scanning the multiple pixels of the imaging element 411. In particular, the imaging circuit 412 converts the brightness value obtained for each of the multiple pixels according to the intensity of the sensed external light into two-dimensional data as each pixel value, thereby obtaining a value such as Figure 6 Camera image Ic as shown.

[0064] Figures 1 to 3 The cleaning system 5 shown cleans a plurality of incident surfaces 33 including the incident surfaces 33o and 33c on which light is incident from the sensing areas Ao and Ac in the optical sensor 40 and the sensing camera 41. The cleaning system 5 includes a cleaning module 50 for each incident surface 33. Each cleaning module 50 may have a cleaning nozzle for spraying a cleaning gas such as air as a cleaning fluid for cleaning the incident surface 33 toward the incident surface 33. Each cleaning module 50 may also have a cleaning nozzle for spraying a cleaning liquid as a cleaning fluid for cleaning the incident surface 33 toward the incident surface 33. Each cleaning module 50 may also have a cleaning blade for cleaning the incident surface 33 by wiping.

[0065] Figure 2 , 3The vehicle control device 1 shown is connected to the electrical components 4 and 5 of the automatic driving unit ADU via at least one of a LAN (Local Area Network), a wiring harness, and an internal bus, for example. The vehicle control device 1 is configured to include at least one dedicated computer. The dedicated computer constituting the vehicle control device 1 may be a driving control ECU that cooperates with an ECU (Electronic Control Unit) in the vehicle 2 to control the automatic driving mode. The dedicated computer constituting the vehicle control device 1 may also be an actuator ECU that independently controls the driving actuator of the vehicle 2. The dedicated computer constituting the vehicle control device 1 may also be a locator ECU that estimates the state quantity of the vehicle 2 including its own position. The dedicated computer constituting the vehicle control device 1 may also be a navigation ECU that navigates the driving route of the vehicle 2. The dedicated computer constituting the vehicle control device 1 may also be an HCU (HMI (Human Machine Interface) Control Unit) that controls the information presentation of the information presentation system of the vehicle 2.

[0066] The vehicle control device 1 is configured to include such a dedicated computer, and at least one memory 10 and a processor 12 are provided. The memory 10 is at least one non-transitorytangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that non-temporarily stores programs and data that can be read by the computer. The processor 12 includes at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU as a core.

[0067] The processor 12 executes a plurality of commands included in the vehicle control program stored in the memory 10. Thus, the vehicle control device 1 constructs a plurality of functional units (i.e., functional modules) for controlling the vehicle 2. In this way, in the vehicle control device 1, the vehicle control program stored in the memory 10 for controlling the vehicle 2 causes the processor 12 to execute a plurality of commands, thereby constructing a plurality of functional units. Figure 3 The plurality of functional units constructed by the vehicle control device 1 shown include an extraction unit 100 and a control unit 120 .

[0068] In each control cycle, the extraction unit 100 receives a distance image Iord as a reflected light image Ior from the optical sensor 40, and receives a camera image Ic from the sensing camera 41. The extraction unit 100 extracts the mismatched pixel group Po by comparing the distance image Iord and the camera image Ic. To this end, the extraction unit 100 includes a pixel interpolation unit 102, an edge detection unit 104, and a matching determination unit 106 as sub-functional units.

[0069] The pixel interpolation unit 102 performs pixel interpolation on the distance image Iord, which is the reflected light image Ior, and the camera image Ic so as to correspond to the high-resolution side and the low-resolution side. In this embodiment, the number of pixel groups capturing the overlapping area Aoc in the sensing areas Ao and Ac in the camera image Ic is set to be greater than that in the distance image Iord. That is, the camera image Ic becomes the high-resolution side, while the distance image Iord becomes the low-resolution side. Therefore, the pixel interpolation unit 102 interpolates the pixel value of each pixel coordinate in the camera image Ic based on each pixel coordinate in the distance image Iord. As a result of the interpolation, Figure 7 The pixel interpolation unit 102 generates a camera image Ic having substantially the same resolution as the distance image Iord with respect to the overlap area Aoc.

[0070] The edge detection unit 104 detects edges from the camera image Ic by performing at least one edge filtering process on the pixel-interpolated camera image Ic. The edge detection unit 104 also detects edges from the image Iord by performing at least one edge filtering process on the reflected light image Ior of the overlap area Aoc having substantially the same resolution as the camera image Ic through pixel interpolation, that is, the distance image Iord.

[0071] The matching determination unit 106 determines the matching status of these images Iord and Ic by comparing the edges detected in the distance image Ior and the camera image Ic, which are the reflected light image Ior. At this time, the matching determination unit 106 makes a mismatch determination, for example, when the pixel values ​​constituting the edge are standardized with each other and the difference is calculated, and it is confirmed that the difference is in a pixel group outside the matching range. Here, in particular, in the first embodiment, for example, a mismatch determination based on water-related substances Mw (described in detail later) present in the distance image Iord and not in the camera image Ic is made by at least one of comparison between the past and present of the images Iord and Ic and object recognition. This is to avoid making a mismatch determination due to mirror-reflecting objects such as black car bodies other than water-related substances Mw that are objects present in the camera image Ic and not in the distance image Iord. By making such a determination, the matching determination unit 106 Figure 5In this way, the pixel group Po corresponding to the edge that does not match the camera image Ic in the distance image Iord is extracted. In addition, a pixel group refers to a group of multiple pixels. In particular, the pixel group Po determined to be mismatched can be a group of multiple adjacent and continuous pixels. In addition, for the convenience of explanation, Figure 5 The thick solid circle is given a reference symbol Po to show the mismatched pixel group Po.

[0072] like Figure 3 As shown in FIG. 1 , the control unit 120 receives the pixel group Po extracted by the mismatch determination of the matching determination unit 106 from the distance image Iord as the reflected light image Ior. Figure 5 As shown, it is estimated that such a mismatched pixel group Po corresponds to the presence of water-related substances Mw that are easily sensed in the reflected light of the strong laser. Here, the water-related substance Mw is, for example, at least one of rain (including water droplets), fog, water vapor (including components in exhaust gas), and sludge containing water. In addition, Figure 5 For convenience, the example schematically shows the existence of mist as the water-related substance Mw for easy understanding.

[0073] The control unit 120 also inputs the value of the reflection point distance dr of the pixel value on the mismatched pixel group Po of the distance image Iord from the matching judgment unit 106. Therefore, the control unit 120 judges the length of the reflection point distance dr on the mismatched pixel group Po by comparing it with the specified setting range. Here, the setting range is defined as a range less than the threshold value related to the reflection point distance dr, or below the threshold value. In addition, the threshold value of the setting range is set to 1 to 3m when stopped, and to 3 to 10m when driving. Of course, the threshold value of the setting range can also be set to an appropriate value of such an example value. Under such regulations and settings, in the control unit 120, when the reflection point distance dr is within the setting range, it is predicted that the water-related substance Mw corresponding to the mismatched pixel group Po is likely to be attached to the incident surface 33o of the optical sensor 40. Among the water-related substances Mw predicted to be attached to the incident surface 33o in this way, rain, fog, water vapor, and sludge are assumed as the above-mentioned example substances.

[0074] Thus, when it is determined that the water-related substance Mw estimated to correspond to the mismatched pixel group Po is predicted to adhere to the incident surface 33o of the optical sensor 40, the control unit 120 issues a cleaning control instruction to the cleaning system 5 of the vehicle 2 in the automatic driving mode to clean the incident surface 33o of the optical sensor 40. The cleaning system 5 receives the instruction and controls the incident surface 33o of the optical sensor 40. Figure 2In this way, the cleaning module 50o corresponding to the incident surface 33o is driven and controlled to perform the cleaning process of the incident surface 33o. At this time, the cleaning control instruction related to the incident surface 33c of the sensing camera 41 is not necessary, so the cleaning module 50c is not an essential element. However, when the attachment prediction of the water-related substance Mw to the incident surface 33o of the optical sensor 40 is established, it is considered that the attachment prediction on the incident surface 33c of the sensing camera 41 arranged as close as possible to the sensor 40 is also established, and the cleaning control instruction related to the surface 33c is issued.

[0075] The following is based on Figure 8 The flow of a vehicle control method in which the vehicle control device 1 controls the vehicle 2 through the cooperation of the functional units 100 and 120 will be described. In addition, each "S" in this flow represents a plurality of steps executed by a plurality of commands included in the vehicle control program.

[0076] First, in S101, the pixel interpolation unit 102 of the extraction unit 100 performs pixel interpolation on the high-resolution camera image Ic based on the distance image Iord as the reflected light image Ior on the low-resolution side. Next, in S102, the edge detection unit 104 of the extraction unit 100 detects edges from the distance image Iord and the camera image Ic, respectively. Next, in S103, the matching determination unit 106 of the extraction unit 100 determines the matching state of these images Iord and Ic by comparing the edges detected in the distance image Iord and the camera image Ic with each other.

[0077] If the result of S103 is that a pixel group Po determined to be mismatched with the camera image Ic is extracted from the distance image Iord as the reflected light image Ior, the process moves to S104. In S104, the control unit 120 determines whether it can be predicted that the water-related substance Mw estimated to correspond to the mismatched pixel group Po is attached to the incident surface 33o of the optical sensor 40 based on the reflection point distance dr of the mismatched pixel group Po.

[0078] If the result of S104 is that the reflection point distance dr of the mismatched pixel group Po is within the set range and the attachment of the water-related substance Mw to the incident surface 33o can be predicted, the process moves to S105. In S105, the control unit 120 issues an instruction to the cleaning system 5 to perform cleaning control on the incident surface 33o of the optical sensor 40 as control of the vehicle 2 in the automatic driving mode.

[0079] In addition, if the result of S103 is that a complete match of the pixel group Po determined as mismatched is not extracted, and if the result of S104 is that the attachment of the water-related substance Mw to the incident surface 33o is not predicted, the flow moves to S106. In S106, the control unit 120 issues a command to stop the cleaning control to the cleaning system 5. As described above, in the first embodiment, S101, S102, and S103 correspond to the extraction process, and S104, S105, and S106 correspond to the control process.

[0080] (Effect)

[0081] Next, the effects of the first embodiment described above will be described.

[0082] According to the first embodiment, the reflected light image Ior acquired by the optical sensor 40 through sensing of reflected light irradiated with light and the camera image Ic acquired by the sensing camera 41 based on the external light intensity in the sensing area Ac overlapping with the sensor 40 are compared. It can be accurately estimated that the mismatched pixel group Po extracted through such comparison corresponds to the water-related substance Mw that is easy to sense reflected light, that is, the water-related substance Mw that is easy to be reflected in the reflected light image Ior. Therefore, by issuing a control instruction corresponding to the water-related substance Mw to the vehicle 2 as a result of the estimation, the appropriateness of the vehicle control can be improved.

[0083] According to the first embodiment, in order to clean the incident surface 33o where light enters from the sensing area Ao in the optical sensor 40, the washing system 5 is mounted on the vehicle 2. According to this, it is possible to accurately estimate that the mismatched pixel group Po extracted by comparing the reflected light image Ior and the camera image Ic acquired by the optical sensor 40 and the sensing camera 41, respectively, corresponds to the water-related substance Mw. Therefore, by issuing a washing control instruction to the washing system 5 of the vehicle 2 in the automatic driving mode for the incident surface 33o predicted as a result of the estimation to have a high possibility of the water-related substance Mw being attached, the appropriateness of the washing control can be improved.

[0084] According to the first embodiment, as a reflected light image Ior corresponding to the reflection point distance dr of the reflected light for light irradiation, the distance image Iord is compared with the camera image Ic. It can be accurately estimated that the mismatched pixel group Po extracted by such comparison corresponds to the water-related substance Mw that is easy to sense the reflection point distance dr, that is, the water-related substance Mw that is easy to be reflected in the distance image Iord. Therefore, as a result of the estimation, an instruction for cleaning control corresponding to the water-related substance Mw is issued, which can improve the appropriateness of the cleaning control.

[0085] According to the first embodiment, among the mismatched pixel groups Po, in particular, the pixel groups Po whose reflection point distance dr in the distance image Iord as the reflected light image Ior is within the set range can be accurately estimated to correspond to the water-related substance Mw predicted to be within the set range and easily attached to the incident surface 33o of the optical sensor 40. Therefore, it is possible to appropriately issue an instruction to control the cleaning of the incident surface 33o under a condition where the possibility of the attachment of the water-related substance Mw is high.

[0086] By comparing the edges of the reflected light image Ior and the camera image Ic as in the first embodiment, it is easy to accurately capture and extract the mismatched pixel group Po from these edges. Therefore, it is possible to appropriately issue instructions and execute control corresponding to the water-related substance Mw estimated to correspond to the mismatched pixel group Po.

[0087] According to the first embodiment, pixel interpolation is performed on the high-resolution side based on the low-resolution side of the reflected light image Ior and the camera image Ic, and then these images Ior and Ic are compared. According to such a comparison, it is possible to suppress the situation where the pixel group that should have matched is erroneously extracted due to the difference in resolution. Therefore, it is possible to appropriately issue an instruction and execute control corresponding to the water-related substance Mw estimated to correspond to the pixel group Po that does not match.

[0088] (Second Embodiment)

[0089] The second embodiment is a modification of the first embodiment.

[0090] like Fig. 9 The vehicle 2 of the second embodiment is equipped with a driving control device 6. The driving control device 6 cooperates with the automatic driving unit ADU to perform driving control of the vehicle 2 including the automatic mode based on various sensing information of the sensor system 4. For this purpose, the driving control device 6 is configured as at least one dedicated computer such as the ECU included in the vehicle 2 described in the first embodiment. In addition, the cleaning system 5 of the second embodiment is not essential. Fig. 9 In the figure, the illustration of the system 5 is omitted.

[0091] In the control unit 2120 of the second embodiment, the reflection point distance dr in the unmatched pixel group Po is not input, and the length determination of the distance dr is omitted. Therefore, the control unit 2120 issues a driving control instruction in the automatic driving mode to the driving control device 6 of the vehicle 2, so that the unmatched pixel group Po estimated to correspond to the water-related substance Mw is excluded from the recognition as an obstacle.

[0092] In the process of the vehicle control method of the second embodiment, as shown in FIG. Fig.10As shown, S101, S102, and S103 are executed in accordance with the first embodiment, and S205 and S206 are executed instead of S104, S105, and S106. Specifically, in S205, the control unit 2120 issues a command for driving control in the automatic driving mode to the driving control device 6 of the vehicle 2 so that the water-related substance Mw is excluded from the identification as an obstacle. In contrast, in S206, the control unit 2120 issues a command to stop the driving control to exclude the water-related substance Mw from the obstacle identification to the driving control device 6. According to the above, in the second embodiment, S101, S102, and S103 are equivalent to the extraction process, and S205 and S206 are equivalent to the control process.

[0093] (Effect)

[0094] Next, the effects of the second embodiment described above will be described.

[0095] According to the second embodiment, a driving control instruction in the automatic driving mode is issued to the vehicle 2 so that the water-related substance Mw estimated to correspond to the mismatched pixel group Po extracted by comparing the reflected light image Ior and the camera image Ic is excluded from the recognition as an obstacle. Accordingly, the continuity of the automatic driving mode can be ensured by appropriate driving control that suppresses the situation where the water-related substance Mw is mistakenly recognized as an obstacle.

[0096] (Third Embodiment)

[0097] The third embodiment is a modification of the first embodiment.

[0098] like Fig.11 The optical sensor 3040 of the third embodiment shown has a function of acquiring a reflected light image Ior during light irradiation based on the first embodiment, and has an image acquisition function during the stop of the light irradiation. Specifically, in an external light mode in which the photographing circuit 3402 exposes the photographing element 401 during the stop of the intermittent light irradiation from the light emitting element 400, the object point in the sensing area Ao becomes a reflection point of the external light. As a result, the external light reflected at the reflection point enters the photographing element 401 through the incident surface 33o. At this time, the photographing circuit 3402 senses the reflected external light by scanning multiple pixels of the photographing element 401.

[0099] In particular, the imaging circuit 3402 converts the brightness value obtained for each of the plurality of pixels according to the intensity of the sensed external light into two-dimensional data as each pixel value, thereby obtaining Fig.12At this time, the sensing of the external light in the acquisition function of the external light image Ioo and the sensing of the reflected light in the acquisition function of the reflected light image Ior are realized by the same imaging element 401. In addition, the sensing camera 41 of the third embodiment is not essential. Fig.11 In the figure, the camera 41 is omitted.

[0100] In the extraction unit 3100 of the third embodiment, the distance image Iord as the reflected light image Ior and the external light image Ioo are sequentially input from the optical sensor 3040 in each control cycle. In the extraction unit 3100, the pixel interpolation unit 102 as a sub-functional unit is omitted by sensing by the same element 401 described above. In addition, the extraction unit 3100 implements the functions of the edge detection unit 104 and the matching determination unit 106 according to the first embodiment between the external light image Ioo replacing the camera image Ic and the reflected light image Ior.

[0101] In the process of the vehicle control method of the third embodiment, as shown in FIG. Fig.13 As shown, first, in S302, the edge detection unit 104 of the extraction unit 3100 detects edges from the distance image Iord, which is the reflected light image Ior, and the external light image Ioo. Next, in S303, the matching determination unit 106 of the extraction unit 3100 determines the matching status of these images Iord and Ioo by comparing the edges detected in the distance image Iord and the external light image Ioo.

[0102] In addition, S104, S105, and S106 after S303 are performed according to the first embodiment. According to the above, in the third embodiment, S302 and S303 correspond to the extraction process, and S104, S105, and S106 correspond to the control process.

[0103] (Effect)

[0104] Next, the effects of the third embodiment described above will be described.

[0105] According to the third embodiment, the reflected light image Ior acquired by the optical sensor 3040 by sensing the reflected light of the light irradiation and the external light image Ioo acquired by the sensor 3040 according to the external light intensity during the stop of the light irradiation are compared. It can be accurately estimated that the mismatched pixel group Po extracted by such comparison corresponds to the water-related substance Mw that is easy to sense the reflected light, that is, the water-related substance Mw that is easy to be reflected in the reflected light image Ior. Therefore, by issuing a control instruction corresponding to the water-related substance Mw to the vehicle 2 as a result of the estimation, the appropriateness of the vehicle control can be improved.

[0106] According to the third embodiment, in order to clean the incident surface 33o where light enters from the sensing area Ao in the optical sensor 3040, the washing system 5 is mounted on the vehicle 2. According to this, it is possible to accurately estimate that the mismatched pixel group Po extracted by comparing the reflected light image Ior and the external light image Ioo acquired by the optical sensor 3040 corresponds to the water-related substance Mw. Therefore, based on the incident surface 33o to which the possibility of the attachment of the water-related substance Mw is predicted as a result of the estimation, an instruction for washing control is issued to the washing system 5 of the vehicle 2 in the automatic driving mode, and the appropriateness of the washing control can be improved.

[0107] According to the third embodiment, as a reflected light image Ior corresponding to the reflection point distance dr of the reflected light for light irradiation, the distance image Iord is compared with the external light image Ioo. It can be accurately estimated that the mismatched pixel group Po extracted by such comparison corresponds to the water-related substance Mw that is easy to sense the reflection point distance dr, that is, the water-related substance Mw that is easy to be reflected in the distance image Iord. Therefore, as a result of the estimation, an instruction for cleaning control corresponding to the water-related substance Mw is issued, which can improve the appropriateness of the cleaning control.

[0108] According to the third embodiment, among the mismatched pixel groups Po, in particular, the pixel groups Po whose reflection point distance dr in the distance image Iord as the reflected light image Ior is within the set range can be accurately estimated to correspond to the water-related substance Mw predicted to be within the set range and easily attached to the incident surface 33o of the optical sensor 3040. Therefore, it is possible to appropriately issue an instruction to control the cleaning of the incident surface 33c under a condition where the possibility of the attachment of the water-related substance Mw is high.

[0109] According to the third embodiment, it is possible to suppress the occurrence of errors in the judgment of inside and outside the set range due to the axis offset between the reflection point distance dr sensed by the same element 401 as the external light intensity in the optical sensor 3040. Therefore, it is possible to appropriately issue instructions and control the cleaning of the incident surface 33o in which the attachment of the water-related substance Mw is expected, corresponding to the pixel group Po whose reflection point distance dr is within the set range.

[0110] (Fourth Embodiment)

[0111] The fourth embodiment is a modification of the third embodiment.

[0112] like Fig.14 As shown in the fourth embodiment, the vehicle 2 is equipped with the driving control device 6 based on the second embodiment. At the same time, the control unit 4120 of the fourth embodiment has the command function of the driving control based on the second embodiment. In addition, the cleaning system 5 of the fourth embodiment is not necessary. Fig.14In the figure, the illustration of the system 5 is omitted.

[0113] In the process of the vehicle control method of the fourth embodiment, as shown in FIG. Fig.15 As shown, according to the third embodiment, S302 and S303 are executed, and on the other hand, S405 and S406 according to S205 and S206 of the second embodiment are executed instead of S104, S105, and S106. That is, in S405, the control unit 4120 issues a driving control instruction in the automatic driving mode to the driving control device 6 of the vehicle 2 so that the water-related substance Mw is excluded from the identification as an obstacle. In contrast, in S406, the control unit 4120 issues a driving control stop instruction to the driving control device 6 to exclude the water-related substance Mw from the obstacle identification. According to the above, in the fourth embodiment, S302 and S303 are equivalent to the extraction process, and S405 and S406 are equivalent to the control process.

[0114] (Effect)

[0115] Next, the effects of the fourth embodiment described above will be described.

[0116] According to the fourth embodiment, a driving control instruction in the automatic driving mode is issued to the vehicle 2 so that the water-related substance Mw estimated to correspond to the mismatched pixel group Po extracted by comparing the reflected light image Ior and the external light image Ioo is excluded from the recognition as an obstacle. Accordingly, by appropriately controlling the driving to suppress the situation where the water-related substance Mw is mistakenly recognized as an obstacle, the continuity of the automatic driving mode can be ensured.

[0117] (Fifth Embodiment)

[0118] The fifth embodiment is a modification of the first embodiment. In the following description of the fifth embodiment, the incident surface 33c of the sensing camera 41 is referred to as a camera incident surface 33c, and the incident surface 33o of the optical sensor 40 is referred to as an optical incident surface 33o.

[0119] like Fig.16 As shown, in the sensing camera 41 of the first embodiment applied to the fifth embodiment, the camera incident surface 33c where light is incident from the sensing area Ac overlapping with the optical sensor 40 is arranged on the inner side of the optical axis L compared to the focal position pf on the outer side of the imaging lens system 413 that images the incident light onto the imaging element 411. That is, the camera incident surface 33c is located within the focal length df on the sensing area Ac side of the imaging lens system 413. Here, within the focal length df, in the camera image Ic of the sensing camera 41, the possibility that the edge of the water-related substance Mw is not detected even by the edge filter processing becomes high.

[0120] like Fig.17 The extraction unit 5100 of the fifth embodiment shown in the figure has a search unit 5108 as a sub-functional unit in addition to the pixel interpolation unit 102, the edge detection unit 104 and the matching determination unit 106. The search unit 5108 searches for an object point whose reflection point distance dr is within the set range Δd in the distance image Iord of the optical sensor 40 (hereinafter also referred to as an object point within the set range). This means searching for an object point near the optical incident surface 33o whose reflection point distance dr is within the set range Δd. Here, the set range Δd in the fifth embodiment is set to a range in which the edge of the water-related substance Mw can be detected by edge filtering processing in the distance image Iord of the optical sensor 40 according to the focal length df on the sensing area Ac side in the imaging lens system 413.

[0121] In such a fifth embodiment, after edge detection by the edge detection unit 104 is performed on the distance image Iord including the object within the set range searched by the search unit 5108 and the camera image Ic subjected to pixel interpolation by the pixel interpolation unit 102, the edges are compared with each other by the matching determination unit 106. The matching determination unit 106 that determines the matching state can extract the pixel group Po representing the edge of the object within the set range in the distance image Iord as the pixel group Po that does not match the camera image Ic.

[0122] Therefore, the control unit 5120 of the fifth embodiment, when extracting the mismatched pixel group Po representing the edge of the target object within the set range, determines that it is predicted that the water-related substance Mw estimated to correspond to the mismatched pixel group Po is attached to the optical incident surface 33o. That is, when the reflection point distance dr in the mismatched pixel group Po is within the set range Δd corresponding to the focal length df, the control unit 5120 predicts that the water-related substance Mw is attached to the optical incident surface 33o.

[0123] In this way, since the reflection point distance dr in the mismatched pixel group Po is within the set range Δd, the control unit 5120 determines that the water-related substance Mw is expected to adhere to the optical incident surface 33o, and issues a command to the cleaning system 5 to clean the optical incident surface 33o using the cleaning gas. After the cleaning control command, the control unit 5120 further issues a command to clean the optical incident surface 33o using the cleaning liquid to the cleaning system 5, when the extraction unit 5100 extracts the pixel group Po with the reflection point distance dr within the set range Δd from the distance image Iord. The extraction of the pixel group Po at this time is achieved by performing edge detection on the mismatched pixel group Po in the distance image Iord before the command by the edge detection unit 104 in the latest distance image Iord after the cleaning control command using the cleaning gas. In addition, after the cleaning control command using the cleaning gas, when there is no pixel group Po with the reflection point distance dr within the set range Δd in the distance image Iord, the control unit 5120 issues a command to stop the cleaning control to the cleaning system 5.

[0124] In the process of the vehicle control method of the fifth embodiment, as shown in FIG. Fig.18 As shown, in S500, the search unit 5108 of the extraction unit 5100 searches for an object whose reflection point distance dr in the distance image Iord is within the set range Δd, that is, an object within the set range. As a result, if an object within the set range is found, the process moves to S501.

[0125] In S501, the pixel interpolation unit 102 of the extraction unit 5100 performs pixel interpolation on the high-resolution camera image Ic based on the distance image Iord as the reflected light image Ior on the low-resolution side. Next, in S502, the edge detection unit 104 of the extraction unit 5100 detects edges from the distance image Iord and the camera image Ic.

[0126] Next, in S503, the matching determination unit 106 of the extraction unit 5100 determines the matching status of the images Iord and Ic by comparing the edges detected in the distance image Iord and the camera image Ic. As a result, when the pixel group Po representing the edge of the object within the set range in the distance image Iord is extracted as the pixel group Po that does not match the camera image Ic, the process moves to S504.

[0127] In S504, the control unit 5120 estimates that the mismatched pixel group Po representing the edge of the object within the set range corresponds to the water-related substance Mw predicted to be attached to the optical incident surface 33o. In other words, it is predicted that the water-related substance Mw estimated to correspond to the mismatched pixel group Po is attached to the optical incident surface 33o. Therefore, in the next S505, the control unit 5120 issues an instruction to the cleaning system 5 to execute the cleaning control of the optical incident surface 33o using the cleaning gas as the control of the vehicle 2 in the automatic driving mode.

[0128] In S506 after the cleaning control instruction for the optical incident surface 33o using the cleaning gas is issued, it is determined whether the edge detection unit 104 of the extraction unit 5100 has extracted the pixel group Po of the reflection point distance dr within the set range Δd from the distance image Iord. As a result, when the edge detection unit 104 has extracted the pixel group Po of the reflection point distance dr within the set range Δd, the present flow moves to S507. In S507, the control unit 5120 issues an instruction to the cleaning system 5 to execute the cleaning control for the optical incident surface 33o using the cleaning liquid as the control of the vehicle 2 in the automatic driving mode.

[0129] On the other hand, if the result of S506 is that the edge detection unit 104 has not extracted the pixel group Po that is the reflection point distance dr within the set range Δd, the flow moves to S508, and the control unit 5120 issues a command to stop the cleaning control to the cleaning system 5. In addition, if the result of S500 is that the object within the set range is not searched, and if the result of S503 is that the pixel group Po representing the edge of the object within the set range is not completely matched, the flow also moves to S508, and the control unit 5120 issues a command to stop the cleaning control to the cleaning system 5. According to the above, in the fifth embodiment, S500, S501, S502, S503, and S506 correspond to the extraction process, and S504, S505, S507, and S508 correspond to the control process.

[0130] (Effect)

[0131] Next, the effects of the fifth embodiment described above will be described.

[0132] According to the fifth embodiment, the camera incident surface 33c of the sensing camera 41, which senses light incident from the sensing area Ac overlapping with the optical sensor 40, is located within the focal length df of the imaging lens system 413 that images the incident light on the sensing area Ac side of the imaging element 411. Accordingly, when the reflection point distance dr in the mismatched pixel group Po is within the set range Δd corresponding to the focal length df, it is predicted that the judgment of the attachment of the water-related substance Mw is correct. Therefore, it is possible to appropriately issue an instruction and control the cleaning of the incident surface 33o in a situation where the possibility of the attachment of the water-related substance Mw is high.

[0133] According to the fifth embodiment, when the reflection point distance dr is within the set range Δd, a command for cleaning control using cleaning gas is issued to the cleaning system 5, and then when the extraction unit extracts the pixel group Po of the reflection point distance dr within the set range Δd from the distance image Iord, a command for cleaning control using cleaning liquid is issued to the cleaning system 5. Accordingly, by using the cleaning gas and the cleaning liquid separately according to the type or adhesion state of the water-related substance Mw, it is possible to save the cleaning liquid. For example, if the water-related substance Mw is rain, fog, or water vapor, it is easy to complete the cleaning even with the cleaning gas, so the cleaning liquid can be effectively used for cleaning sludge, etc., which is the water-related substance Mw.

[0134] (Other Embodiments)

[0135] Although a plurality of embodiments have been described above, the present disclosure is not limited to these embodiments and can be applied to various embodiments and combinations within a scope not departing from the gist of the present disclosure.

[0136] In a modified example, the dedicated computer constituting the vehicle control device 1 may also be at least one external central computer capable of communicating with the vehicle 2. In a modified example, the dedicated computer constituting the vehicle control device 1 may also include at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit refers to at least one of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a SOC (System on a Chip), a PGA (Programmable Gate Array), and a CPLD (Complex Programmable Logic Device). In addition, such a digital circuit may also have a memory storing a program.

[0137] In the modified example, the light-transmitting cover 32 forming the incident surface 33o may be provided in the optical sensor 40 or 3040. In the modified example, the incident surface 33o may be formed by an optical component such as a lens in the optical sensor 40 or 3040.

[0138] In the modified example, in the optical sensor 40, 3040, an intensity image in which the brightness value obtained for each of the plurality of pixels according to the reflected light intensity is two-dimensionally digitized as each pixel value may be obtained as the reflected light image Ior. In this case, the functions of the sub-function units constituting the extraction unit 100, 3100 are implemented between the camera image Ic or the external light image Ioo and the intensity image as the reflected light image Ior. However, in this case, the determination function of the reflection point distance dr based on the control unit 120 in the first and third embodiments may be implemented based on the distance image Iord as the reflected light image Ior, or may be omitted.

[0139] In the modified example, Fig.19 , 20 In other words, in the vehicle control method, S104 of the first and third embodiments may be omitted. That is, in the process of the vehicle control method, S105 may be executed only by predicting the possibility of the water-related substance Mw adhering to the incident surface 33o based on the mismatch determination in S103 or S303.

[0140] In a modified example, Fig.21 The combination shown realizes the first embodiment and the second embodiment. That is, if the adhesion of the water-related substance Mw to the incident surface 33o is not predicted as a result of S104, the process of the vehicle control method moves to S205, and if it is completely matched as a result of S103, the process moves to S106 and S206 in sequence.

[0141] In a modified example, Fig. 22 The combination shown realizes the third embodiment and the fourth embodiment. That is, if the adhesion of the water-related substance Mw to the incident surface 33o is not predicted as a result of S104, the process of the vehicle control method moves to S405, and if it is completely matched as a result of S303, the process moves to S106 and S406 in sequence.

[0142] In the modified example, the cleaning control of the fifth embodiment may be replaced or the driving control based on the second embodiment may be performed on the basis of the cleaning control. In the modified example, instead of executing S505 to S507 of the fifth embodiment, the step of executing the control unit 5120 to issue a cleaning control instruction to the cleaning system 5 using at least one of the cleaning gas and the cleaning liquid may be replaced. In the modified example, in S506 of the fifth embodiment, the edge detection unit 104 of the extraction unit 5100 may realize the extraction and determination of the pixel group Po by executing the processing based on S500 to S503. In the modified example, in S105 of the first and third embodiments, the steps S506 to S508 based on the fifth embodiment may be performed (however, the setting range is the range defined in the first embodiment).

Claims

1. A vehicle control device for controlling a vehicle, wherein an optical sensor for acquiring a reflected light image by sensing reflected light of light irradiation and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor are mounted on the vehicle, the vehicle control device comprising: an extracting unit that extracts a mismatched pixel group by comparing a distance image as the reflected light image corresponding to a distance of a reflection point of the reflected light with respect to the light irradiation with the camera image; and The control unit issues a control instruction corresponding to the water-related substance estimated to correspond to the mismatched pixel group to the vehicle.

2. The vehicle control device according to claim 1, wherein: The vehicle is equipped with a cleaning system to clean the incident surface of the optical sensor through which light enters the sensing area. The control unit issues a command for washing control of the incident surface to the washing system of the vehicle in the automatic driving mode when predicting that the water-related substance corresponding to the mismatched pixel group is attached to the incident surface.

3. The vehicle control device according to claim 2, wherein: The control unit determines that adhesion of the water-related substance is predicted when the distance between the reflection points in the mismatched pixel group is within a set range.

4. The vehicle control device according to claim 3, wherein: In the above-mentioned sensing camera, the incident surface from which light is incident from the sensing area overlapping the above-mentioned optical sensor is located within the focal length of the imaging lens system for imaging the incident light onto the sensing area side of the imaging element, The control unit determines that adhesion of the water-related substance is predicted when the reflection point distance in the mismatched pixel group is within the set range corresponding to the focal length.

5. The vehicle control device according to claim 3, wherein: The control unit sends a command for cleaning control using a cleaning gas to the cleaning system when the reflection point distance is within the set range, and then sends a command for cleaning control using a cleaning liquid to the cleaning system when a pixel group with the reflection point distance within the set range is extracted from the distance image by the extraction unit.

6. The vehicle control device according to any one of claims 1 to 5, wherein: The control unit issues a command for driving control of the vehicle in the automatic driving mode so as to exclude the water-related substance estimated to correspond to the mismatched pixel group from recognition as an obstacle.

7. A vehicle control method, comprising: a vehicle including an optical sensor for acquiring a reflected light image by sensing reflected light of light irradiation, and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor, the vehicle control method comprising: an extraction step of extracting a mismatched pixel group by comparing a distance image as the reflected light image corresponding to a distance of a reflection point of the reflected light to the light irradiation with the camera image; and The control step is to issue a control instruction corresponding to the water-related substance estimated to correspond to the pixel group that does not match the vehicle.

8. The vehicle control method according to claim 7, wherein: The vehicle is equipped with a cleaning system to clean the incident surface of the optical sensor through which light enters the sensing area. The control step issues a command for washing control of the incident surface to the washing system of the vehicle in the automatic driving mode when it is predicted that the water-related substance corresponding to the mismatched pixel group is attached to the incident surface.

9. The vehicle control method according to claim 8, wherein: The control step determines that adhesion of the water-related substance is predicted when the distance between the reflection points in the mismatched pixel group is within a set range.

10. The vehicle control method according to claim 9, wherein: In the above-mentioned sensing camera, the incident surface from which light is incident from the sensing area overlapping the above-mentioned optical sensor is located within the focal length of the imaging lens system for imaging the incident light onto the sensing area side of the imaging element, The control step determines that adhesion of the water-related substance is predicted when the distance of the reflection points in the mismatched pixel group is within the set range corresponding to the focal length.

11. The vehicle control method according to claim 9, wherein: The control process sends a cleaning control instruction using a cleaning gas to the cleaning system when the reflection point distance is within the set range, and then sends a cleaning control instruction using a cleaning liquid to the cleaning system when a pixel group with the reflection point distance within the set range is extracted from the distance image through the extraction process.

12. The vehicle control method according to any one of claims 7 to 11, wherein: The control step issues a command for driving control of the vehicle in the automatic driving mode so as to exclude the water-related substance corresponding to the non-matching pixel group from recognition as an obstacle.

13. A storage medium storing a vehicle control program, the vehicle control program including instructions for causing a processor to execute in order to control a vehicle, wherein the vehicle is equipped with an optical sensor for acquiring a reflected light image by sensing reflected light of light irradiation, and a sensing camera for acquiring a camera image corresponding to the intensity of external light in a sensing area overlapping with the optical sensor, The above command contains: an extraction step of causing the processor to extract a mismatched pixel group by comparing a distance image corresponding to a reflection point distance of the reflected light to the light irradiation with the camera image as the reflected light image; and The control step causes the processor to issue a control instruction corresponding to the water-related substance estimated to correspond to the mismatched pixel group to the vehicle.

14. The storage medium according to claim 13, wherein: The vehicle is equipped with a cleaning system to clean the incident surface of the optical sensor through which light enters the sensing area. The control step causes the processor to issue a command for cleaning control of the incident surface to the cleaning system of the vehicle in the automatic driving mode when it is predicted that the water-related substance corresponding to the mismatched pixel group is attached to the incident surface.

15. The storage medium according to claim 14, wherein: The control step causes the processor to determine that adhesion of the water-related substance is predicted when the distance between the reflection points in the non-matching pixel group is within a set range.

16. The storage medium according to claim 15, wherein: In the above-mentioned sensing camera, the incident surface from which light is incident from the sensing area overlapping the above-mentioned optical sensor is located within the focal length of the imaging lens system for imaging the incident light onto the sensing area side of the imaging element, The control step causes the processor to determine that adhesion of the water-related substance is predicted when the distance of the reflection points in the mismatched pixel group is within the set range corresponding to the focal length.

17. The storage medium according to claim 15, wherein: The control process causes the processor to issue a cleaning control instruction using a cleaning gas to the cleaning system when the reflection point distance is within the set range, and then to issue a cleaning control instruction using a cleaning liquid to the cleaning system when a pixel group with the reflection point distance within the set range is extracted from the distance image through the extraction process.

18. The storage medium according to any one of claims 13 to 17, wherein: The control step issues a command for driving control of the vehicle in the automatic driving mode so as to exclude the water-related substance corresponding to the non-matching pixel group from recognition as an obstacle.

Images