Sensor system, method for sensing, and program

Abstract

To provide a sensor system, a method for sensing, and a program applicable to changes in the surrounding environment of a vehicle.SOLUTION: A sensor system (1) is mounted on a vehicle (7) and includes at least a radar (10) and a camera (20). The sensor system (1) includes a detection unit (42, 171) configured to detect an object existing around the vehicle (7) on the basis of at least one of radar information detected by the radar (10) and optical sensor information detected by the camera (20); and an operation changing unit (19, 44) configured to change an operation mode of the vehicle (7) on the basis of external environment information around the vehicle (7).SELECTED DRAWING: Figure 3C

Description

[Technical Field]

[0001] The present invention relates to a sensor system, a sensing method, and a program. [Background technology]

[0002] In recent years, the importance of safe driving assistance and automated driving has been increasing for mobility vehicles that travel on roads and off-road work sites. For example, various mobility vehicles are becoming more sophisticated, such as those used on public roads, intersections, highways, special vehicles for transporting goods, work vehicles for transporting goods within construction sites and factories, and agricultural machinery used on farms. These mobility vehicles are equipped with sensor systems, and decisions are made and the mobility is controlled based on the information recognized by these systems. This must be possible in all external situations, and various forms and methods for achieving this are being considered.

[0003] For example, Patent Document 1 describes an object detection method that combines a camera and radar as an example of sensor fusion. Patent Document 1 shows examples of applying the detection results of multiple sensors to make the vehicle safer, such as calculating the reliability of each sensor, weighting the detection results, and integrating the target detection, or using information detected on the closer side of the sensor output information depending on the situation.

[0004] Patent Document 2 describes a technology for detecting weather changes and bad weather through image recognition. In Patent Document 2, the technology determines the decline in the camera's ability to recognize landmarks, the decline in detection distance, and other factors, as well as the surrounding conditions such as rainfall, and is also able to determine the type of adverse environment using the camera, such as rainfall, fog, snow, sandstorms, etc.

[0005] Patent Document 3 describes a radar threshold processing method using CFAR detection and a method for setting a predetermined threshold based on a noise signal. CFAR alone cannot detect weak targets around strong targets, and targets with length go undetected. Furthermore, because there are several issues, such as noise being detected, an example is given in which a fixed threshold setting is introduced.

[0006] Patent Document 4 describes an example of how a vehicle responds to weather changes by changing the search and navigation modes of an autonomous vehicle based on rain information detected by a rainfall sensor, etc. In this example, when rain is detected, covered parking lots are given priority for navigation. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-6123 [Patent Document 2] Japanese Patent Application Publication No. 2022-14729 [Patent Document 3] Japanese Patent Publication No. 2022-174938 [Patent Document 4] Japanese Patent Application Publication No. 2018-105814 Summary of the Invention [Problem to be solved by the invention]

[0008] However, while Patent Document 1 implements sensor fusion using multiple sensors to make the vehicle safer, such as in preventing collisions, it does not describe how to respond to dynamic changes in the surrounding environment, such as weather. Patent Document 2 also describes how cameras and image recognition can be used to detect unsuitable conditions for sensors, such as weather changes and bad weather, but does not describe specific measures to be taken on the vehicle or on the sensor in such conditions. Patent Document 3 also discloses a threshold processing method as a method for improving radar detection of surrounding objects, but does not describe a method suitable for bad weather or a method that responds to dynamic changes in the surrounding environment. Patent Document 4 discloses a method for changing the information provision mode for a specific application in response to dynamic changes in the surrounding environment, such as weather, but requires communication with an external vehicle to implement this method. It does not describe autonomous methods, such as independent vehicle decisions using sensors mounted on the vehicle or changing sensor operation modes.

[0009] An object of the present invention is to provide a sensor system that can be adapted to changes in the surrounding environment of a vehicle. [Means for solving the problem]

[0010] (1) The sensor system is mounted on a vehicle and includes at least a radar and an optical sensor, and includes a detection unit that detects objects present around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor, and an operation change unit that changes the operation mode of the vehicle based on external environmental information around the vehicle.

[0011] (2) In the sensor system described in (1), the external information is detected by at least one of light of a visible wavelength and light of an infrared wavelength from the optical sensor.

[0012] (3) In the sensor system described in (1) or (2), the operation change unit changes the operation mode of the radar together with the operation mode of the vehicle based on the external environment information.

[0013] (4) In the sensor system described in (3), the change in the operation mode of the radar is a change in a threshold value for distinguishing between an object and noise in a signal included in the radar information.

[0014] (5) In the sensor system described in (4), the operation change unit changes the operation mode of the radar by setting the threshold relatively higher than the normal operation mode of the radar through CFAR processing in the distance direction.

[0015] (6) In the sensor system described in (4), when the threshold is set by CFAR processing in the direction of distance during the radar's normal operation mode, the operation change unit changes the radar's operation mode by setting the threshold in the CFAR processing in the direction of distance relatively higher than that during the radar's normal operation mode.

[0016] (7) In the sensor system described in any one of (4) to (6), the operation change unit changes the operation mode of the radar by setting the threshold value having a different gradient or change in the distance direction so that the threshold value is relatively higher in the vicinity of the vehicle than in the normal operation mode.

[0017] (8) In the sensor system described in any one of (4) to (7), the operation change unit sets the threshold value in the distance direction to be relatively higher than that in the normal operation mode based on historical information of past environmental noise when it is determined that external information is unsuitable for the radar or the optical sensor.

[0018] (9) In the sensor system described in any one of (4) to (8), the operation change unit changes the operation mode of the radar by setting a threshold that is relatively higher near the vehicle in the distance direction through CFAR processing in the speed direction.

[0019] (10) In the sensor system described in any one of (3) to (9), the radar has a transmitting antenna and a receiving antenna each having multiple channels, and is capable of detecting angles in both the horizontal angle direction and the elevation and depression angle direction, and the sensor system further includes a processing unit that uses the radar information to simultaneously create a map of the surrounding environment and estimate its own position.

[0020] In the sensor system described in either (11) or (10), the operating mode of the radar is changed by lowering the reliability of the point cloud compared to the normal operating mode when creating a map of the surrounding environment and estimating the radar's own position based on the point cloud output from the radar.

[0021] (12) In the sensor system according to any one of (1) to (11), the change in the operation mode of the vehicle is to slow down the moving speed of the vehicle compared to the normal operation mode.

[0022] (13) In the sensor system according to any one of (1) to (12), the change in the operation mode of the vehicle is to move the vehicle to or return to a predetermined position.

[0023] The sensor system described in either (14) or (13) further includes a GNSS device capable of receiving GNSS signals, and the predetermined position is set in a global reference geodetic system using the GNSS signals together with a three-dimensional map that the vehicle has in advance, and the sensor system identifies its own position and the predetermined position in the global reference geodetic system based on a map of the surrounding environment created using radar information and the three-dimensional map.

[0024] (15) In the sensor system described in (14), the predetermined location includes a starting point where the vehicle starts traveling, a garage, or a location with a roof.

[0025] (16) In the sensor system described in (3), the operation change unit determines whether the external environment information is appropriate for the radar and the optical sensor based on the level of environmental noise around the vehicle, and if it determines that the external environment information is inappropriate, it changes the operation mode of the vehicle and the operation mode of the radar, and if it determines that the external environment information has recovered from an inappropriate state to an appropriate state, it switches the operation mode of the vehicle and the operation mode of the radar after a certain time has passed since the determination.

[0026] (17) The sensing method is a sensing method using a sensor system mounted on a vehicle and equipped with at least a radar and an optical sensor, and includes a detection step of detecting an object present around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor, and an operation change step of changing the operation mode of the vehicle based on external environment information around the vehicle.

[0027] (18) The program is a program to be executed by a computer of a sensor system mounted on a vehicle and equipped with at least a radar and an optical sensor, and causes the computer to execute a detection process of detecting objects present around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor, and an operation change process of changing the vehicle's operation mode based on external environmental information around the vehicle. [Effects of the Invention]

[0028] According to the present invention, a sensor system that can be adapted to changes in the surrounding environment of a vehicle can be provided. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a schematic diagram illustrating a sensor system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of the configuration of a radar according to an embodiment of the present invention; [Figure 3A] 1 is a block diagram showing a hardware configuration of a radar according to an embodiment of the present invention; [Figure 3B] 1 is a block diagram showing a hardware configuration of an ECU according to an embodiment of the present invention; [Figure 3C] 2 is a block diagram illustrating a configuration of functional blocks of a computer of an ECU according to an embodiment of the present invention. FIG. [Figure 3D] 4 is a flowchart showing an example of processing executed by a computer of an ECU according to an embodiment of the present invention. [Figure 4A] 10 is a graph showing the relationship between the quality of the environment around the vehicle and the performance of the radar, camera, and vehicle body operation. [Figure 4B] 10 is a graph showing the results when the quality of the environment around the vehicle, the radar performance, and the vehicle body motion performance are simultaneously degraded. [Figure 5A] 10A and 10B are diagrams illustrating the spectrum of a signal detected under normal circumstances, a threshold, noise, and a peak of an object. [Figure 5B] 10A and 10B are diagrams illustrating the spectrum of a detected signal, a threshold, environmental noise, and a peak of an object in bad weather. [Figure 6A] FIG. 10 is a diagram showing the spectrum of environmental noise detected in bad weather and having high levels near the vehicle. [Figure 6B] FIG. 10 is a diagram showing the spectrum of low environmental noise detected in bad weather on the near side of the vehicle. [Figure 6C] FIG. 10 is a diagram showing environmental noise detected in bad weather and a threshold value set higher near the vehicle. [Figure 6D] FIG. 10 is a schematic diagram showing a situation when moving from an area with environmental noise to an area without environmental noise. [Figure 6E] FIG. 10 is a diagram illustrating thresholds set in OS-CFAR processing and detected signals in the presence of an object and thermal noise. [Figure 6F] FIG. 6C shows the thresholds set and the detected signals in the OS-CFAR process performed in the middle order (the eighth smallest value relative to the reference cell 16) when environmental noise is added from the state of FIG. 6E. [Figure 6G] FIG. 6C is a diagram showing thresholds set and detected signals in OS-CFAR processing performed by changing the magnification of the threshold from the setting shown in FIG. 6F. [Figure 6H] FIG. 6B is a diagram showing thresholds set and signals detected in OS-CFAR processing performed at an order greater than the intermediate order when environmental noise is added from the state of FIG. 6E. [Figure 6I] FIG. 10 is a diagram showing an example of a threshold value obtained by changing the magnification of the speed CFAR in the distance direction in bad weather. [Figure 7A]FIG. 10 is a diagram illustrating an example of a flowchart of a process for changing an operation mode in accordance with external information from a sensor system mounted on an agricultural machine. [Figure 7B] FIG. 10 is a diagram showing an example of a flowchart of a process for changing an operation mode in response to external information from a sensor system mounted on a general automobile. [Figure 8] FIG. 1 is a schematic diagram showing a sensor system that controls vehicle operation by fusing radar information and GNSS signals. [Figure 9] FIG. 1 is a diagram illustrating an example of a predetermined position in a three-dimensional map defined in a global reference geodetic system. DETAILED DESCRIPTION OF THE INVENTION

[0030] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0031] <Sensor system configuration> The sensor system 1 according to this embodiment is a system that is mounted on a vehicle 7, detects objects such as obstacles present around the vehicle 7, and controls the operation of the vehicle 7 and various sensors according to the detection results. First, an outline of the configuration of the sensor system 1 will be described with reference to Figs. 1 to 3D.

[0032] 1, the sensor system 1 of this embodiment includes a radar 10, a camera 20, a GNSS (Global Navigation Satellite System) device 30, and an ECU 40. The vehicle 7 on which the sensor system 1 is mounted is not particularly limited, and may be, for example, a general vehicle, or a work vehicle such as a construction machine, civil engineering machine, transport vehicle, or agricultural machine. In addition, notifying the operator of the vehicle 7 is also included in controlling the state of the vehicle.

[0033] The ECU 40 is also connected to control the vehicle body drive units 70, such as the brakes and steering of the vehicle 7, and controls the movement of the vehicle 7. The radar 10, the camera 20, and the GNSS device 30 are each mounted on the vehicle 7 and connected to the ECU 40. The ECU 40 controls the operation of the vehicle 7 based on sensor information from the radar 10, the camera 20, and the GNSS device 30. In this specification, the radar information, the camera information, and the GNSS signal are collectively referred to as sensor information, and the radar 10 and the camera 20 are referred to as sensors 2.

[0034] The GNSS device 30 includes an antenna and receives GNSS signals and the like. The GNSS signals are transmitted from navigation satellites that constitute GNSS, such as the Global Positioning System (GPS) or the Quasi-Zenith Satellite System. The GNSS device 30 transmits the received GNSS signals to the ECU 40. The system is not limited to the above, and may be RTK-GNSS.

[0035] The radar 10 acquires objects around the vehicle 7 as a point cloud by transmitting and receiving electromagnetic waves. As shown in Fig. 2, the radar 10 includes a local oscillator 11, a modulator 12, an antenna 13, a demodulator 14, an AD converter 15, and a signal processor (detector) 16. Some of these may be integrated into an IC (Integrated Circuit). The radar 10 transmits radio waves from a transmitting antenna 131, receives radio waves reflected by the object with a receiving antenna 132, and performs signal processing to detect the object.

[0036] The local oscillator 11 generates a signal of a predetermined frequency and supplies it to the modulator 12 .

[0037] The modulator 12 modulates the local oscillation signal of a predetermined frequency supplied from the local oscillator 11 and supplies the modulated signal to the antenna 13. Note that various modulation methods other than frequency modulation, such as pulse modulation, phase modulation, and multi-level modulation, may also be used.

[0038] The antenna unit 13 is composed of a transmitting antenna 131 that transmits electromagnetic waves and multiple receiving antennas 132 that receive the electromagnetic waves reflected by objects from the transmitting antenna 131. The antenna unit 13 transmits the frequency-modulated signal supplied from the modulator 12 to the surrounding area as electromagnetic waves, and also captures and converts the reflected signals reflected by surrounding objects into electrical signals. The transmitting antenna 131 and the receiving antenna 132 each have multiple channels.

[0039] The electric signal from the antenna unit 13 is amplified by an amplifier (not shown) and supplied to a demodulator 14. The demodulator 14 demodulates the electric signal supplied from the amplifier unit using a local oscillation signal supplied from the local oscillator 11, and supplies the demodulated signal to an AD converter 15.

[0040] The AD converter 15 samples the electrical signal supplied from the demodulation unit 14 at a predetermined period, converts it into a digital signal, and supplies it to the signal processing unit 16 .

[0041] The signal processing unit 16 acquires a point cloud and detects targets by performing signal processing on the received digital signals supplied from the AD converter 15. The signal processing unit 16 executes distance processing, velocity processing, and angle processing as signal processing, and acquires a point cloud with specified position information.

[0042] The distance processing executed by the signal processing unit 16 is a process of obtaining distance data corresponding to the distance to an object from the frequency difference between the transmitted wave and the reflected wave for each of a plurality of measurement periods. The velocity processing is a process of obtaining information corresponding to the velocity of an object by processing a plurality of distance data repeatedly obtained for each of a plurality of measurement periods. The angle processing is a process of identifying the angle of the point cloud.

[0043] Here, the angle resolution improves as the number of channels provided by each of the transmitting antenna 131 and the receiving antenna 132 increases. In the radar 10 of this embodiment, the transmitting antenna 131 and the receiving antenna 132 each having a plurality of channels are arranged so as to enable angle detection in the horizontal angle direction and angle detection in the elevation and depression angle directions. With this configuration, the radar 10 can acquire a three-dimensional point cloud, which is a point cloud having dimensions in both the horizontal direction and the elevation and depression angle directions in addition to distance.

[0044] Acquiring a three-dimensional point cloud improves the accuracy of obstacle detection by the vehicle 7 and also enables the generation of a three-dimensional map using SLAM (Simultaneous Localization and Mapping). In particular, it is possible to detect and acquire height information for objects with a vertical shape in environments with three-dimensional structures, such as curbs and guardrails around the roadway on roads, structures around work sites for work vehicles, and trees and vegetation on farms. Furthermore, monitoring these surrounding conditions during autonomous driving allows for determination of whether or not the vehicle can travel and for driving control. When considering vehicle movement in locations requiring various vertical structures, it is preferable to use a radar 10 capable of detecting angles in both directions. Furthermore, especially when considering that the detection performance of other sensors 2 may be reduced due to environmental changes such as rain, fog, snow, and dust, it is necessary to achieve autonomous driving using the radar 10 alone. By increasing the number of channels and improving angular resolution, it becomes possible to more accurately determine the drivable area around the vehicle and detect obstacles.

[0045] The camera 20 is an optical sensor that detects objects present around the vehicle 7. The camera 20 transmits camera information to the ECU 40 and the radar 10. In addition to the camera 20, other optical sensors include, for example, a LiDAR and a raindrop sensor.

[0046] In the sensor system 1 using multiple sensors 2, in terms of electromagnetic wave wavelength, the wavelength of millimeter waves or the like acquired by the radar 10 is longer than the optical wavelength acquired by an optical sensor such as the camera 20. Therefore, when changes in the external environment such as rain, fog, snow, or dust occur around the vehicle 7, these external environments will each have some kind of fine particles distributed therein, and although the radio wave wavelengths of the radar 10 or the like are not significantly affected by these, the optical wavelengths acquired by the camera 20 or the like will be significantly affected by these.

[0047] The optical wavelengths of the camera 20 and the like can recognize the surroundings of the vehicle 7 with high resolution, but are significantly affected by changes in the external environment. When rain, fog, snow, dust, etc. are considered as environmental noise, the environmental noise is easy to detect, but the environmental noise makes it difficult to detect the intended detection target. This change is more noticeable than with the radar 10 and the like. In this specification, unwanted signals caused by the environment, such as rain, fog, snow, and dust, are referred to as environmental noise, and information about the environmental noise around the vehicle 7 is referred to as external information. Examples of external information include the presence or absence of environmental noise and the amount of environmental noise.

[0048] The sensor system 1 may include at least one of a LiDAR and a raindrop sensor as an optical sensor together with the camera 20. That is, the identification of environmental changes such as rain, fog, snow, and dust may be achieved not only by the camera 20 but also by the LiDAR or the raindrop sensor. In terms of detecting external information, the use of the camera 20 is preferable because it can identify environmental changes in detail as image information, in addition to point cloud information from the LiDAR.

[0049] Furthermore, for example, the sensor system 1 may be configured to include a communication device that communicates with the outside of the vehicle 7, and to acquire weather information and the like from the outside of the vehicle 7 through the communication device.

[0050] Although it is possible to acquire weather information and the like from outside the vehicle 7 via communication, it is preferable to acquire information from sensors mounted on the vehicle 7 in order to detect local environmental changes around the vehicle 7 in real time. Cameras, LiDAR, and raindrop sensors all detect environmental changes around the vehicle 7 using visible light or infrared wavelengths. While raindrop sensors are specialized for detecting raindrops and can accurately detect precipitation, other sensors 2 are preferable for responding to other events. Furthermore, a configuration in which the LiDAR, camera 20, etc. are connected to the radar 10, etc. via the ECU 40, etc., is efficient because it allows for operation as sensor fusion in normal conditions and for sharing external environment identification information with the radar 10 as described above.

[0051] The sensor system 1 of this embodiment monitors the periphery of the vehicle 7 and obtains information necessary for safe driving assistance and autonomous driving, specifically, detecting obstacles for collision avoidance and determining the vehicle's own position for autonomous driving. To ensure accuracy, the sensor system 1 fuses radar information, camera information (optical sensor information), GNSS signals, and other sensor information. Furthermore, the sensor system 1 changes the operation mode of the sensors 2, such as the radar 10, and the operation mode of the vehicle 7 based on external information acquired by the camera 20, etc., so that optimal processing can be performed in response to changes in the surrounding environment of the vehicle 7.

[0052] Next, an example of the hardware configuration of the radar 10 will be described with reference to Fig. 3A. Fig. 3A is a block diagram showing the hardware configuration of the radar 10.

[0053] The radar 10 includes a computer 110, a storage unit 111, and an I / F unit 112. A bus 113 and the like connect these units together.

[0054] The computer 110 includes a processor 114 and a read-only memory (ROM) 115 and a random-access memory (RAM) 116 as main storage devices. The processor 114 may be a central processing unit (CPU), a microprocessing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 114 may be a combination of these. The processor 114 may also be a combination of these with a hardware accelerator or the like. The processor 114 controls each unit to implement various functions of the radar 10 based on programs such as firmware, system software, and application software stored in the ROM 115, the RAM 116, or an auxiliary storage device that is part of the storage unit 111. Note that some or all of the programs may be incorporated into the circuitry of the processor 114.

[0055] The storage unit 111 is a storage area for storing various programs and various data for causing the hardware group to function as the radar 10, and can be configured with a ROM, RAM, flash memory, a solid-state drive (SSD), a hard disk drive (HDD), etc. Specifically, the storage unit 111 stores programs for causing the computer 110 to execute each function of this embodiment.

[0056] The I / F unit 112 is a wired communication interface for the radar 10 to communicate with the ECU 40.

[0057] Next, an example of the hardware configuration of the ECU 40 will be described with reference to Fig. 3B. Fig. 3B is a block diagram showing the hardware configuration of the ECU 40.

[0058] The ECU 40 includes a computer 410, a storage unit 411, and an I / F unit 412. A bus 413 and the like connect these units together.

[0059] The computer 410 includes a processor 414 and a read-only memory (ROM) 415 and a random-access memory (RAM) 416 as main storage devices. The processor 414 may be a central processing unit (CPU), a microprocessing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 414 may be a combination of these. The processor 414 may also be a combination of these with a hardware accelerator or the like. The processor 414 controls each unit to realize various functions of the ECU 40 based on programs such as firmware, system software, and application software stored in the ROM 415, the RAM 416, or an auxiliary storage device that is part of the storage unit 411. Note that some or all of the programs may be incorporated into the circuitry of the processor 414.

[0060] The storage unit 411 is a storage area for storing various programs and various data for causing the hardware group to function as the ECU 40, and can be configured with a ROM, a RAM, a flash memory, a solid-state drive (SSD), a hard disk drive (HDD), etc. Specifically, the storage unit 411 stores programs for causing the computer 410 to execute each function of this embodiment.

[0061] The I / F unit 412 is a wired communication interface for the ECU 40 to communicate with the radar 10, the camera 20, the GNSS 30, and the vehicle body drive unit 70.

[0062] Next, functional blocks for executing the process of changing the operation mode of the sensor 2 and the process of changing the operation mode of the vehicle 7 and an overview of the process will be described with reference to FIGS. 3C and 3D.

[0063] For example, as shown in FIG. 3C, the radar 10 includes a signal processing unit 16, an external noise information acquisition unit 17, a detection unit 171, a point cloud processing unit 18, and an operation modification unit 19 as functional units operating on the processor 114.

[0064] The signal processing unit 16 acquires, as radar information, a point cloud obtained by signal processing the received signal received by the receiving antenna 132 as described above. The signal processing unit 16 executes so-called threshold processing, which distinguishes signals due to an object from noise components and acquires them from all information in each dimension of the signal processing, that is, the radar cube, in the distance processing, velocity processing, and angle processing. In the threshold processing, the threshold for distinguishing noise components is changed by the operation modification unit 19, which will be described later, based on external information.

[0065] The external noise information acquisition unit 17 acquires external information acquired by the camera 20 or the like via the ECU 40 .

[0066] The detection unit 171 detects objects present around the vehicle 7 by referring to the radar information acquired by the signal processing unit 16 and the information acquired from a camera or the like by the external noise information acquisition unit 17.

[0067] The point cloud processing unit 18 has a configuration in which the transmitting antenna 131 and the receiving antenna 132 each have multiple channels and are capable of angle detection in both the horizontal direction and the elevation / depression angle direction, thereby enabling the radar 10 to use the detected data to perform three-dimensional obstacle detection and coordinate transformation before estimating the self-position, etc. Furthermore, an application that uses self-position information acquired via the ECU 40 or a sensor further therefrom, or an application that estimates the self-position information may be used, although this is a minor process compared to that performed by the ECU 40.

[0068] The operation change unit 19 executes a process of changing the operation mode of the radar 10 based on the external environment information. The operation change unit 19 determines whether the external environment information is appropriate for the radar 10 and an optical sensor such as the camera 20, for example, based on the level of environmental noise around the vehicle 7. If the operation change unit 19 determines that the external environment information is not appropriate, it changes the operation mode of the radar 10. The specific contents of the process of changing the operation mode of the radar 10 will be described later.

[0069] The ECU 40 includes, as functional units operating on a processor 414, a sensor information acquisition unit 41, a detection unit 42, a processing unit 43, and an operation change unit 44, as shown in FIG. 3C, for example.

[0070] The sensor information acquisition unit 41 executes a process of acquiring camera information (optical sensor information) from the camera 20, radar information from the radar 10, and a GNSS signal from the GNSS device 30.

[0071] The detection unit 42 executes the detection step shown in step S1 in Fig. 3D. That is, the detection unit 42 executes a process of detecting objects present around the vehicle 7 based on at least one of radar information and camera information. The detection unit 42 monitors the periphery of the vehicle 7 and may fuse radar information, camera information, and sensor information such as GNSS signals to detect obstacles for safe driving assistance or collision avoidance in autonomous driving.

[0072] The processing unit 43 executes a process of estimating the self-position information of the vehicle 7. The self-position information may be, for example, information such as the latitude, longitude, and altitude of the location where the vehicle 7 is located. The processing unit 43 may also estimate the self-position information based on a GNSS signal. For example, the processing unit 43 may perform SLAM by fusing radar information, camera information, and the GNSS signal. Note that SLAM refers to simultaneously performing map creation and self-position estimation.

[0073] The operation change unit 44 executes an operation change step shown in step S2 of Fig. 3D. That is, the operation change unit 44 executes a process of changing the operation mode of the vehicle 7 based on external environment information. The operation change unit 44 determines whether the external environment information is appropriate for the radar 10 and optical sensors such as the camera 20, for example, based on the level of environmental noise around the vehicle 7. Then, when the operation change unit 44 determines that the external environment information is not appropriate, it changes the operation mode of the vehicle 7. The specific contents of the process of changing the operation mode of the vehicle 7 will be described later.

[0074] Here, optical sensors such as the camera 20 are subject to a large degree of performance degradation due to environmental changes such as rain, fog, snow, and dust, but are easy to identify environmental changes. Furthermore, it is preferable for the sensor 2 used for sensor fusion to be able to autonomously identify environmental changes such as rain, fog, snow, and dust in real time, and the camera 20 is suitable for this purpose. On the other hand, the radar 10 is subject to a small degree of performance degradation due to environmental changes, but is difficult to identify environmental changes. If the detection performance of the camera 20 is reduced to a certain level due to environmental noise, the detection performance as a sensor fusion system will be reduced, making it difficult to provide the vehicle 7 with sufficient information about the detection target that should be obtained. For these reasons, when the camera 20 identifies environmental changes around the vehicle or determines performance degradation, it is possible to reduce the vehicle 7's movement performance from normal, as shown in FIG. 4A, and switch to a safer operating mode.

[0075] At the same time, by adopting a configuration in which the sensors 2 are connected to each other, it is possible to share the identification results of the camera 20 with other sensors 2 such as the radar 10. The radar 10 may experience performance degradation or false detections due to failure to recognize the presence of environmental noise such as rain, fog, snow, and dust. For example, it may become difficult to detect distant objects, and environmental noise may be mistakenly detected as a target in the vicinity. By recognizing the presence of environmental noise using the camera 20, it is possible for the radar 10 to deal with these situations. Specifically, as shown in FIG. 1 , the identified information can be sent to the radar 10 via the ECU 40, and the operating mode of the radar 10 can be affected.

[0076] For example, the detection results of sensor 2 are affected by environmental noise, and even if detection is possible at a distance, the accuracy will be lower than usual. When the situation at a distance becomes uncertain, obstacles can only be detected and judged nearby, and assuming the vehicle is traveling, there will be no time to take control measures such as braking to a stop or steering to avoid an obstacle. To address these issues, by slowing the travel speed of vehicle 7 in the operating mode compared to the normal operating mode, the time it takes for the control to be reflected in the movement of vehicle 7 can be shortened, thereby ensuring safety.

[0077] As described above, it is preferable to simultaneously change the vehicle 7's operation mode to correspond to a performance degradation of the sensor fusion of the camera 20, radar 10, etc., and address the performance degradation of the sensor 2. As described above, when the vehicle travels at a slow speed, the required detection distance for the sensor 2 is shorter than normal. It is not necessary to retain uncertain distant detection results as usual; accuracy can be improved by eliminating uncertain information. For example, in SLAM using the radar 10, previously detected point cloud information is retained during movement to map the surrounding environment. However, uncertain acquired point cloud information is unnecessary; only accurate information is required. Requiring the sensor to maintain a detection distance similar to normal even when the vehicle's speed is set to a slow speed is an excessive performance requirement, resulting in a degradation of accuracy. Conversely, maintaining the vehicle 7's travel speed at, for example, normal speed despite setting the detection distance of the sensor 2 to a short value clearly reduces safety. For these reasons, it is preferable to simultaneously change the vehicle 7's operation mode and the operation mode of the sensor 2, such as the radar 10, as shown in FIG. 4B.

[0078] Up to this point, we have considered environmental noise to be changes in the environment around the vehicle, such as rain, fog, snow, and dust. However, this does not only include bad weather, but also environmental noise caused by the operation of the vehicle itself. For example, dust generated by the vehicle's movement, or spray from agricultural machinery while in operation, can also be environmental noise around the vehicle. Furthermore, as mentioned above, environmental noise is not limited to certain particulate matter, but can also adhere to the sensor surface.

[0079] <About changing the operation mode of Radar 10> The process of changing the operation mode of the radar 10 by the operation change unit 19 will be described.

[0080] The operation modification unit 19 changes the threshold value used in threshold processing based on external environment information. For example, the operation modification unit 19 changes the setting of the threshold value that distinguishes between an object and noise in the spectrum of a received signal included in radar information.

[0081] In a typical radar 10, CFAR (Constant False Alert Rate) processing may be applied to extract signals from noise components. CFAR processing basically compares data of interest with surrounding data, and determines a signal if the strength is greater than a certain magnification. There are various methods for acquiring the surrounding data, and the surrounding data for the data of interest varies depending on the axis, range, and representative value method. In addition to CFAR processing, it is also possible to set a threshold based on some other information, and determine a signal if the strength is greater than that threshold.

[0082] The operation modification unit 19 changes the operation mode of the radar 10 by, for example, setting a threshold value relatively higher than that of the normal operation mode of the radar 10 through CFAR processing in the distance direction. This processing will be described with reference to FIGS. 5A and 5B, which illustrate a normal state and a state in which environmental noise, such as rain, fog, snow, or dust, is present. The normal state refers to, for example, sunny or cloudy weather, a state in which dust is not flying, etc. FIG. 5A is a graph showing the signal and threshold value detected by the signal processing unit 16 and the distance from the vehicle 7 under normal conditions. FIG. 5B is a graph showing the signal and threshold value detected by the signal processing unit 16 and the distance from the vehicle 7 under bad weather conditions. In FIGS. 5A and 5B, the vertical axis represents the signal amplitude, and the horizontal axis represents the distance from the vehicle 7. In FIG. 5A, the dashed line represents the threshold value for distinguishing a signal due to an object from noise, and the solid line represents thermal noise. In Figure 5B, the thin dashed line represents the same threshold as that shown in Figure 5A, the dashed line represents environmental noise, the thick dashed line represents the threshold set by performing CFAR processing under normal conditions, and the thick solid line represents the threshold set by performing CFAR processing with a magnification increased from the threshold of the thick dashed line.

[0083] For example, consider a signal on the range axis of a radar cube, as shown in Figure 5A. When there is no environmental noise and few unnecessary structures around the target, a simple threshold setting that separates the signal component from thermal noise can be applied to target detection. For example, CFAR processing may not be performed in the range direction, or, if it is performed, CA (Cell Averaging)-CFAR or OS-CFAR (Ordered Statistics) may be used with a certain magnification factor before and after the range direction. On the other hand, when environmental noise such as rain, fog, snow, or dust is present, these noise components will be superimposed.

[0084] Environmental noise such as rain, fog, snow, and dust spreads spatially and continuously along the distance axis. As shown in Figure 5B, the environmental noise exceeds the thermal noise level, so if the threshold is set assuming the target is thermal noise, the environmental noise will be erroneously detected as a signal.

[0085] Although it is difficult for the radar 10 alone to determine whether or not environmental noise is present, as described above, it is possible to deal with these situations by acquiring information on the presence of environmental noise from an external device, such as the camera 20, outside the radar 10. As described above, for environmental noise that continuously spreads along the distance axis, performing CFAR on the distance axis is an effective countermeasure against false detection of environmental noise. Furthermore, if CFAR is already being performed on the distance axis under normal conditions, it is possible to further reduce false detection of environmental noise by changing the magnification setting or method in CFAR when determining the presence of environmental noise.

[0086] Furthermore, we consider various variations in outdoor environments. While environmental noise is widespread and unrestricted in a completely outdoor environment, there are cases where environmental noise is present in only a limited area, or where indoor and outdoor environments are mixed. For example, there are situations where environmental noise exists only on one side, such as the situation shown in Figure 6A, in which environmental noise occurs from the sensor surface to the air and the level of environmental noise decreases at a distance due to distance attenuation, or the situation shown in Figure 6B, in which a vehicle enters an outdoor environment from a foggy or snow-free area or a covered garage. In Figures 6A and 6B, the vertical axis represents the signal amplitude, the horizontal axis represents the distance from the vehicle 7, the dashed line represents environmental noise, and the solid line represents thermal noise.

[0087] In such cases, when using a conventional CFAR to cyclically acquire noise values ​​for nearby and distant locations, or when using CA-CFAR, OS-CFAR, or SOCA (Smallest Of Cell Averaging)-CFAR, environmental noise may be recognized as a detection target in situations where noise is high only on the nearby side, as shown in Figure 6C, or where noise increases rapidly across a boundary, such as an outdoor exit, as shown in Figure 6D. In Figure 6D, the thick solid line represents rain, and the thin solid line extending in the distance direction represents the building roof. In situations where rain, fog, snow, dust, etc. are recognized outside the room, it is also effective to take measures to prevent false detections due to more localized environmental noise, such as changing the radar operating mode to OS-CFAR or GOCA (Greatest Of Cell Averaging)-CFAR, which have a CFAR setting greater than the middle order.

[0088] As an example, Figure 6B shows the simulation results for threshold setting and detection in a situation where environmental noise occurs starting from a certain distance. Note that if the starting distance is set to 0, the situation will be similar to Figure 6A. Figure 6E shows a situation where a target object and thermal noise are present. While the units are arbitrary, the horizontal axis represents distance and the vertical axis represents amplitude. The circled line represents the signal, the solid line represents the threshold set by CFAR or other methods, and the thick dashed line indicates whether the signal exceeds the threshold. The thick dashed line shows a situation where targets are detected at positions greater than 0, approximately 10, 11, and 53. The same notation is used in Figures 6F, 6G, and 6H. Figure 6E shows that OS-CFAR was implemented, and the target object was successfully detected from the signal. Here, OS-CFAR is configured by sorting the amplitude values ​​of a specific number of reference cells from smallest to largest, and then multiplying the Kth largest value by a multiplier. Therefore, the number of reference cells, the Kth reference cell, the multiplier, etc. can be setting parameters. In this example, at least the Kth reference cell is set to approximately the middle order, for example, 8th for a reference cell count of 16.

[0089] Figure 6F assumes a situation in which environmental noise occurs starting from a certain distance indicated by the two-dot chain line. Environmental noise is added to the signal, causing the level of the circled line to rise beyond the two-dot chain line. Even though the signal level rises sharply at the two-dot chain line, if the CFAR threshold is determined based on the signal level on the non-rising side, false positives may occur at this boundary (dotted circle, thick dashed line). If the presence of environmental noise can be detected by other sensors, such as a camera, the CFAR settings can be changed. For example, in Figure 6G, the threshold magnification is changed, which may avoid false positives. However, in this case, false positives cannot be avoided, and the target object remains undetected. On the other hand, in Figure 6H, the Kth setting in the OS-CFAR is changed to a value larger than the middle order, e.g., the 10th smallest value for a reference cell count of 16. This setting change avoids false positives at the boundary and correctly detects the target object. When environmental noise is added beyond a certain boundary or locally, it is preferable to determine the CFAR threshold based on the signal level increased by the environmental noise. As in this example, not only OS-CFAR, which is set to an order larger than the middle, but also GOCA-CFAR is suitable for application to environmental noise.

[0090] Furthermore, environmental noise such as rain, fog, snow, and dust becomes more pronounced near the radar 10. When these noises are present over a wide range, reflection, scattering, and attenuation phenomena occur over the entire distance, making it more likely that false detections will occur near the radar 10 where the reflection is less attenuated. Furthermore, adhesion of these noises to the radome of the radar 10, the cover that houses the radar 10, and the bumper can also cause false detections. In either case, a higher threshold value than usual, as shown in Figure 6C, is required in the vicinity or very vicinity where the influence of environmental noise is significant.

[0091] For example, the operation modification unit 19 may change the operation mode of the radar 10 by setting a threshold value with a different gradient or change in the distance direction so that the threshold value is relatively higher near the vehicle 7 than in the normal operation mode. For example, as a method other than the above-described CFAR for setting the threshold value, a fixed value may be used, as shown in FIG. 6C , or a threshold value calculated from a noise value detected by the radar 10 may be multiplied by the calculated value. In either case, it is effective to set a higher threshold value nearer the distance axis than the normal setting. For example, as a countermeasure against false detection due to the above-described adhesion to the radar 10, a fixed threshold may be set in the very vicinity of the stationary object component, or the signal itself may be removed so that it is not detected. Furthermore, it is also possible to set a threshold value with a gradient near the distance based on environmental noise information acquired by some means, and to set a multiplier for calculating the threshold value.

[0092] Furthermore, there are various cases where environmental noise, such as rain, fog, snow, and dust, occurs locally. For example, as shown in FIG. 6D , when the vehicle 7 moves from outdoors to an indoor area, such as a garage, where there is no environmental noise, the presence of environmental noise may be very localized depending on the position of the vehicle or the sensor 2. Under these localized conditions, it is expected that instantaneous CFAR or instantaneous threshold processing will result in false detection. In such cases, data acquired when the vehicle is in a different location or past thresholds are used or retained while it is determined that environmental noise is occurring within the detection range. That is, the operation modification unit 19 may set a threshold value in the distance direction that is relatively higher than that in the normal operation mode based on historical information about environmental noise when past external information was determined to be unsuitable for radar or optical sensors. This makes it possible to avoid false detection of local environmental noise.

[0093] Furthermore, CFAR is generally effective as a threshold process not only on the distance axis but also on the speed axis. When environmental noise such as rain, fog, snow, or dust is present or the vehicle 7 is traveling through such noise, the environmental noise spreads widely on the distance axis but spreads partially on the speed axis. When speed CFAR is performed under such circumstances and determines that environmental noise is occurring in a manner different from normal conditions, applying a different threshold magnification, as shown in FIG. 6I, can avoid false detections due to environmental noise. The solid line in FIG. 6I indicates the threshold value under normal conditions, and the dashed line indicates the threshold value under bad weather. For example, the operation modification unit 19 may change the operation mode of the radar 10 by performing CFAR processing in the speed direction and setting a threshold value that is relatively higher near the vehicle 7 in the distance direction.

[0094] This has an effect similar to changing the magnification of the distance CFAR mentioned above. Also, as opposed to the method of using a gradient in the distance direction as a threshold, by setting the magnification of the velocity CFAR for each distance, it is possible to have different thresholds on the distance axis, which has an effect similar to changing the setting to one with a gradient on the distance axis mentioned above. For example, it is possible to set quantitatively according to distance, such as avoiding false detections in the vicinity or removing noisy signals in the distance. In this case, it is possible to set a relative threshold compared to the method of calculating from a fixed value or noise value on the distance axis mentioned above.

[0095] When the radar 10 detects an object in a location where an optical sensor such as the camera 20 identifies the presence of clear environmental noise, the operation modification unit 19 may determine that the detection result by the radar 10 is a false detection. This is not a detailed change in the settings of the radar 10 as described above, but is a logical process for the sensor system 1 having multiple sensors 2, and is a simple response.

[0096] The radar 10 of this embodiment also features SLAM, but when environmental noise is present, the noise floor becomes high, reducing the accuracy of the output point cloud. For example, when the presence of environmental noise is determined by a sensor 2 other than the radar 10, such as a camera 20, which can more easily determine environmental noise, reliability information is added to the output point cloud acquired by the radar 10. For example, when performing SLAM based on the point cloud output from the radar 10, the operation change unit 19 may change the operation mode by performing SLAM with a lower reliability of the point cloud than in the normal operation mode. The acquired reliability information determines the processing method for applying the point cloud to SLAM. In the most extreme case, when it is determined that environmental noise is present, new map information is not reflected, and only existing map information is used to perform self-location estimation. It is also possible to lower the contribution to map creation compared to normal situations, or to tighten the conditions for map generation. For example, in cases where a grid map is to be created, it is possible to tighten the conditions for generating the grid map. In adverse conditions such as rain, fog, snow, and dust, a new map update is not necessarily required, but it is necessary to avoid losing track of the vehicle's own position.

[0097] <About changing the operation mode of vehicle 7> The process of changing the operation mode of the vehicle 7 by the operation change unit 44 when it is determined that adverse conditions such as rain, fog, snow, or dust are present will be described.

[0098] After identifying an environmental change using an optical sensor such as the camera 20, i.e., after determining whether external information is appropriate based on the level of environmental noise around the vehicle 7, the operation change unit 44 changes the operation mode of the vehicle 7 to one that corresponds to a decrease in performance as sensor fusion of the radar 10, the camera 20, etc. For example, in order to avoid events such as a collision, the operation change unit 44 has the vehicle 7 travel at a slower speed than normal to ensure safety, but there are also more suitable travel methods depending on the application situation of the vehicle 7.

[0099] For example, if the vehicle 7 is a work machine such as an agricultural machine, it may be difficult to continue working in rainy weather. In pest control, for example, pesticides may not be sprayed on vegetation and may be washed away in rainy weather, which may necessitate halting work in addition to driving safely. These work machines typically depart from a predetermined location, such as a garage or warehouse, when work begins and return to the same location when work is completed. If the surrounding conditions deteriorate during work and the work must be stopped, it is preferable for the work machine to be able to return to the starting point of the work. An example of a decision flow, including low-speed driving, is shown in Figure 7A.

[0100] 7A, in step S11, the operation modification unit 44 determines whether the environment around the vehicle 7 is normal based on the external environment information acquired by the sensor information acquisition unit 41, and if it is determined that the environment is normal, for example, sunny or cloudy (step S11; YES), it moves the vehicle 7 in the normal operation mode and ends the processing. On the other hand, if it is determined that the environment around the vehicle 7 has deteriorated due to rain or the like (step S11; NO), it operates the vehicle 7 at a low speed and proceeds to step S13.

[0101] In step S13, the operation modification unit 44 determines whether or not to stop the work due to worsening surrounding conditions, etc. If it is determined that there is no need to stop the work (step S13; NO), the operation modification unit 44 changes the operation of the vehicle 7 to an operation for low-speed work. On the other hand, if it is determined that there is a need to stop the work (step S13; YES), the operation modification unit 44 returns the vehicle 7 to the starting point.

[0102] Furthermore, for ordinary automobiles, there may be events such as dense fog or heavy snow that make it difficult to continue driving. In such cases, stopping the vehicle 7 is the easiest safety measure, but stopping a large number of vehicles 7 can cause chaos in the traffic environment, and how autonomous vehicles respond to traffic environments in bad weather is a major challenge. In such situations, driving at low speeds and avoiding the traffic toward the starting point of the vehicle, moving to a designated evacuation location, a place recognized as a relatively large space, a parking lot, etc., is effective in avoiding confusion. An example of a decision flow including low-speed driving is shown in Figure 7B.

[0103] 7B, in step S21, the operation modification unit 44 determines whether the environment around the vehicle 7 is normal based on the external environment information acquired by the sensor information acquisition unit 41. If the operation modification unit 44 determines that the environment around the vehicle 7 is normal, such as sunny or cloudy (step S21; YES), the operation modification unit 44 moves the vehicle 7 in a normal operation mode and ends the processing. On the other hand, if the operation modification unit 44 determines that the environment around the vehicle 7 has deteriorated due to dense fog or heavy snow (step S21; NO), the operation modification unit 44 operates the vehicle 7 at a low speed (step S22) and proceeds to step S23.

[0104] In step S23, the operation modification unit 44 determines whether or not to continue traveling. If it is determined that traveling should be continued (step S23; YES), the operation modification unit 44 causes the vehicle 7 to travel at a low speed. On the other hand, if it is determined that traveling needs to be stopped (step S23; NO), the operation modification unit 44 causes the vehicle 7 to move to a predetermined evacuation location.

[0105] In either case, it is necessary to move the vehicle to a predetermined location, and it is preferable to perform these by monitoring the surroundings and estimating the vehicle's own position using radar 10, which has little performance degradation even in adverse conditions such as rain, fog, dust, snow, etc. Furthermore, it is preferable that the driver, a remote operator, etc., intervene in the execution of these avoidance movements.

[0106] Furthermore, after determining that the surrounding environment is in adverse conditions, the vehicle 7's operating mode may be changed to, for example, low-speed driving. If the surrounding environment improves and an optical sensor such as the camera 20 detects this improvement, the vehicle's operating mode may be returned to normal mode. In this case, if the sensor 2 detects an unfavorable environmental change, the operating mode is immediately changed to ensure safety. However, when it is determined that the environment has recovered, temporary recovery may occur in changeable conditions such as sudden rain or dust. Therefore, it is preferable to return to normal mode after a certain period of recovery has been determined. Instantaneously changing the operating mode of the vehicle 7 or the radar 10 due to temporary recovery may result in the need to redo the operating mode change or chattering. It is preferable to minimize the number of operating mode changes and ensure stability.

[0107] Furthermore, autonomous driving based on radar 10, which is robust against adverse conditions such as rain, fog, snow, and dust, requires radar 10 to recognize the surroundings and estimate its own position using SLAM. GNSS is typically effective for self-position estimation, and no major issues arise when the GNSS radio wave environment is favorable. However, GNSS may malfunction depending on the surrounding environment. To accurately estimate self-position in all cases, it is useful to use a sensor 2 other than GNSS, such as radar 10, and these self-position estimates must be fused as shown in Figure 8. With a sensor 2 such as radar 10 mounted on the vehicle, the surrounding detection results can be defined relative to the vehicle's coordinate system. However, in fusion, the two coordinate systems must ultimately match, and the self-position estimation results by radar 10 are preferably defined based on the global reference geodetic system used by the GNSS device 30. That is, the vehicle-centered SLAM map position obtained by radar 10 is compared with information based on GNSS signals and converted to an absolute position. Furthermore, when performing three-dimensional perimeter monitoring or SLAM using the radar 10, a three-dimensional global reference geodetic system is preferable. Although the notation itself may use offset values ​​via a local coordinate system, definitions are allowed in the global reference geodetic system. These coordinate system definitions can be performed in advance via common landmarks or the like at predetermined locations.

[0108] Furthermore, when autonomous driving is performed using the radar 10 alone in adverse conditions as described above, a predetermined location that will be the driving destination must be set in the three-dimensional map recognized by the radar 10, and this predetermined location is set in a global reference geodetic system or the like. For example, as shown in FIG. 9, the departure point 3 of the host vehicle and the corresponding return point 4 of the host vehicle must be defined in the three-dimensional map recognized by the radar 10. For this reason, it is preferable to acquire the predetermined locations in the global reference geodetic system in advance. Note that the departure point 3 and return point 4 of the host vehicle are often located in a garage or a roof for a work machine, etc., and direct coordinate acquisition using GNSS may be difficult. However, it is possible to coordinate these locations in advance using some indirect method. If at least this predetermined location can be set in a radar three-dimensional map defined in a global reference geodetic system, travel to this location is possible, and return, etc., is possible using radar SLAM even in a garage or a roof where the GNSS radio wave environment is poor.

[0109] In both of the above cases, a 3D map created beforehand based on the point cloud of the radar 10 is required before the vehicle travels. The 3D map acquired during preliminary travel with the radar in operation or travel in a good environment determined to have little environmental noise must be stored in memory. Since the amount of data required for storing this data is large, it can also be stored in an ECU or other device outside the radar module.

[0110] According to the embodiment described above, the following effects are achieved.

[0111] The sensor system 1 of this embodiment is mounted on a vehicle 7 and includes at least a radar 10 and a camera 20, and includes a detection unit 42, 171 that detects objects present around the vehicle 7 based on at least one of radar information detected by the radar 10 and optical sensor information detected by the camera 20, and an operation change unit 19, 44 that changes the operation mode of the vehicle 7 based on external environment information around the vehicle 7.

[0112] As a result, by installing at least optical sensors such as the radar 10 and the camera 20, it becomes possible to monitor the surroundings at different wavelengths. In sensor fusion, by using the radar 10 that transmits and receives electromagnetic waves with a relatively long wavelength, it becomes possible to detect surrounding objects even in external environments such as rain, fog, snow, and dust. Furthermore, by using external information acquired by various means, such as within the sensor fusion or through communications outside the vehicle, it becomes possible to determine whether the external environment is suitable or unsuitable for the vehicle-mounted sensors, and to change the operating mode of the vehicle 7 accordingly, thereby ensuring safety.

[0113] In the sensor system 1 according to this embodiment, external information is detected by at least one of light of visible wavelength and light of infrared wavelength from the optical sensor.

[0114] This simplifies the process by allowing information to be acquired by the sensor system 1 mounted on the vehicle, eliminating the need for communication with the outside of the vehicle 7. Furthermore, in sensor fusion, by using sensors with relatively short wavelengths compared to radar such as cameras, it becomes possible to more clearly detect rain, fog, snow, dust, etc., and acquire information about the outside world with high precision.

[0115] In the sensor system 1 according to this embodiment, the operation change units 19 and 44 change the operation mode of the radar 10 together with the operation mode of the vehicle 7 based on the external environment information.

[0116] As a result, when the external environment, such as the surroundings of the vehicle 7, is unsuitable for the sensor 2 mounted on the vehicle 7, the operating mode of the vehicle 7 is simultaneously changed to correspond to the deterioration of the detection capability of the sensor 2 by changing the operating mode of the radar 10 in response to the external environmental noise, and the performance of the sensor 2 and the vehicle 7 can be adjusted in a consistent manner in the unsuitable external environment.

[0117] In the sensor system 1 according to this embodiment, the change in the operation mode of the radar 10 is the change in the threshold value for distinguishing between an object and noise in the signal included in the radar information.

[0118] This allows the radar 10 to change the threshold setting to a different value from that used in normal operation, thereby avoiding the false detection of environmental noise such as rain, fog, snow, and dust that do not normally exist as targets.

[0119] Furthermore, in the sensor system 1 according to this embodiment, the operation change units 19 and 44 change the operation mode of the radar 10 by setting the threshold value relatively higher than that of the normal operation mode of the radar 10 through CFAR processing in the distance direction.

[0120] As a result, environmental noises such as rain, fog, snow, and dust that do not normally exist are distributed spatially around the vehicle and sensor to a certain extent, unlike the objects to be detected. Therefore, by using a CFAR mode in the distance direction that is different from the normal state, or by setting the threshold value in the CFAR in the distance direction to make them more difficult to detect, it is possible to avoid falsely detecting these as objects.

[0121] In addition, in the sensor system 1 of this embodiment, when the threshold is set by CFAR processing in the distance direction when the radar 10 is in the normal operation mode, the operation change unit 19, 44 may change the operation mode of the radar 10 by setting the threshold in the CFAR processing in the distance direction relatively higher than that in the normal operation mode of the radar 10.

[0122] As a result, environmental noises such as rain, fog, snow, and dust that do not normally exist are distributed spatially around the vehicle and sensor to a certain extent, unlike the objects to be detected. Therefore, by using a CFAR mode in the distance direction that is different from the normal state, or by setting the threshold value in the CFAR in the distance direction to make them more difficult to detect, it is possible to avoid falsely detecting these as objects.

[0123] In addition, in the sensor system 1 according to this embodiment, the operation change unit 19 changes the operation mode of the radar by setting a threshold value having a different gradient or change in the distance direction that is relatively higher in the vicinity of the vehicle 7 than in the normal operation mode.

[0124] This makes it possible to avoid false detections due to environmental noise such as rain, fog, snow, and dust, which are difficult to handle using distance CFAR, etc. Specifically, by raising the threshold value especially on the near side, false detections in the vicinity can be avoided due to the tendency for environmental noise to appear more pronounced at closer distances, and the adhesion of unwanted matter to the cover on which the sensor 2 is installed, etc.

[0125] In addition, in the sensor system 1 according to this embodiment, the operation modification unit 19 sets a threshold value in the distance direction that is relatively higher than that in the normal operation mode, based on historical information of past environmental noise when the external environment information was determined to be inappropriate for the radar 10 or the camera 20.

[0126] This makes it possible to avoid false detections by setting a threshold generated based on relatively spatially spread environmental noise acquired in the past, in contrast to local environmental noise that is difficult to deal with using instantaneous distance CFAR or threshold settings.

[0127] In addition, in the sensor system 1 according to this embodiment, the operation change unit 19 changes the operation mode of the radar by setting a threshold that is relatively high near the vehicle in the distance direction through CFAR processing in the speed direction.

[0128] This makes it possible to simultaneously detect noise present as a velocity component and deal with environmental noise that appears more prominent at closer distances.

[0129] In addition, in the sensor system 1 according to this embodiment, the radar 10 has a transmitting antenna 131 and a receiving antenna 132, each having multiple channels, and is capable of detecting angles in both the horizontal angle direction and the elevation / depression angle direction. The sensor system 1 further includes a processing unit 43 that uses radar information to simultaneously create a map of the surrounding environment and estimate its own position.

[0130] This allows for multiple channels and angle detection in both horizontal and elevation / depression directions, enabling three-dimensional obstacle detection and mapping, and more accurate self-location estimation. Therefore, three-dimensional detection and self-location estimation enable radar 10-based control of the vehicle 7's operation and autonomous driving.

[0131] In addition, in the sensor system 1 according to this embodiment, when creating a map of the surrounding environment and estimating the radar's own position based on the point cloud output from the radar 10, the operating mode of the radar 10 is changed by lowering the reliability of the point cloud compared to the normal operating mode.

[0132] This reduces the impact of point clouds that are less accurate than normal, ensuring the accuracy of the surrounding map generated by SLAM.

[0133] In the sensor system 1 according to this embodiment, the change in the operation mode of the vehicle 7 is to make the moving speed of the vehicle 7 slower than in the normal operation mode.

[0134] As a result, in order to cope with the decline in the detection capability of the sensor 2, particularly the decline in the detection distance and the delay in detection, the vehicle operation is slowed down and the deceleration time and stopping time of the vehicle 7 are shortened, thereby making it possible to avoid contact with an obstacle.

[0135] In the sensor system 1 according to this embodiment, the change of the operation mode of the vehicle 7 is to move the vehicle 7 to or return to a predetermined position.

[0136] As a result, if it is determined that continuing to drive or work is unsuitable in bad weather, the vehicle will retreat to a relatively safe location in a sensor operation mode or vehicle operation mode suited to the unsuitable conditions.

[0137] In the sensor system 1 according to this embodiment, the predetermined position includes the starting point where the vehicle 7 starts traveling, a garage, or a place with a roof.

[0138] This allows a predetermined position to be clearly defined, and allows movement to a location where safety is ensured, or where bad weather such as rain or snow can be avoided.

[0139] In addition, the sensor system 1 of this embodiment further includes a GNSS device 30 capable of receiving GNSS signals, and the predetermined position is set in the global reference geodetic system using the GNSS signals together with a three-dimensional map that the vehicle has in advance, and the sensor system identifies its own position and the predetermined position in the global reference geodetic system based on a map of the surrounding environment created using radar information and the three-dimensional map.

[0140] As a result, by storing a three-dimensional map and predetermined locations in advance based on the signal of the radar 10, which has relatively little performance degradation due to environmental noise such as rain, fog, snow, and dust, it becomes possible to move to a predetermined location when an unsuitable situation actually occurs with other sensors as long as the radar 10 can identify its own position. Furthermore, since the coordinate definition of this radar 10 is stored in the global reference geodetic system, it becomes possible to fuse it with information such as the radar's own position and predetermined position based on GNSS signals.

[0141] Furthermore, in the sensor system 1 according to this embodiment, the operation change unit 19 determines whether the external environment information is appropriate for the radar 10 and the camera 20 based on the level of environmental noise around the vehicle 7, and if it determines that the external environment information is inappropriate, it changes the operation mode of the vehicle 7 and the operation mode of the radar 10, and if it determines that the external environment information has recovered from an inappropriate state to an appropriate state, it switches the operation mode of the vehicle 7 and the operation mode of the radar 10 a certain time after the determination.

[0142] This allows the operating mode to be changed immediately to ensure safety when an environmental change occurs that is unsuitable for sensor 2. However, since the environment may be judged to have recovered only temporarily, recovery after a certain period of time can be achieved to ensure safety and stabilize the operating mode.

[0143] The sensing method of this embodiment is a sensing method using a sensor system 1 mounted on a vehicle 7 and equipped with at least a radar 10 and a camera 20, and includes a detection process of detecting objects present around the vehicle 7 based on at least one of radar information detected by the radar 10 and optical sensor information detected by the camera 20, and an operation change process of changing the operation mode of the vehicle 7 based on external environment information around the vehicle 7.

[0144] The program of this embodiment is a program to be executed by a computer 110, 410 of a sensor system 1 that is mounted on a vehicle 7 and has at least a radar 10 and a camera 20, and causes the computer 110, 410 to execute a detection process of detecting objects present around the vehicle 7 based on at least one of radar information detected by the radar 10 and optical sensor information detected by the camera 20, and an operation change process of changing the operation mode of the vehicle 7 based on external environment information around the vehicle 7. [Explanation of symbols]

[0145] 1 Sensor System 7 vehicles 10. Radar 19, 44 Operation change section 20 Camera (optical sensor) 42, 171 Detector 110, 410 Computer

Claims

1. A sensor system mounted on a vehicle and including at least a radar and an optical sensor, a detection unit that detects an object present around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor; and an operation change unit that changes the operation mode of the vehicle based on external environmental information about the surroundings of the vehicle.

2. The sensor system according to claim 1 , wherein the external information is detected by at least one of light of a visible wavelength and light of an infrared wavelength from the optical sensor.

3. The sensor system according to claim 1 , wherein the operation change unit changes the operation mode of the radar together with the operation mode of the vehicle based on the external environment information.

4. The sensor system according to claim 3 , wherein the change in the operation mode of the radar is a change in a threshold value for distinguishing between an object and noise in a signal included in the radar information.

5. The sensor system according to claim 4 , wherein the operation change unit changes the operation mode of the radar by setting the threshold relatively higher than that of a normal operation mode of the radar through CFAR processing in the distance direction.

6. 5. The sensor system according to claim 4, wherein, when the threshold is set by CFAR processing in the distance direction during the radar's normal operation mode, the operation change unit changes the radar's operation mode by setting the threshold in the CFAR processing in the distance direction to be relatively higher than that during the radar's normal operation mode.

7. The sensor system according to claim 4 , wherein the operation change unit changes the operation mode of the radar by setting the threshold value having a different gradient or change in the distance direction so that the threshold value is relatively higher in the vicinity of the vehicle than in a normal operation mode.

8. The sensor system according to claim 4, wherein the operation change unit sets the threshold value in the distance direction to be relatively higher than that in a normal operation mode based on historical information of past environmental noise when it is determined that external information is unsuitable for the radar or the optical sensor.

9. The sensor system according to claim 4 , wherein the operation change unit changes the operation mode of the radar by setting a threshold that is relatively high on the side closer to the vehicle in the distance direction through CFAR processing in the direction of velocity.

10. The radar has a transmitting antenna and a receiving antenna each having a plurality of channels, and is capable of detecting angles in both horizontal and elevation / depression angles; The sensor system according to claim 3 , further comprising a processing unit that uses the radar information to simultaneously create a map of the surrounding environment and estimate a self-position.

11. The sensor system according to claim 10, wherein the change in the operating mode of the radar is performed by lowering the reliability of the point cloud compared to the normal operating mode when creating a map of the surrounding environment and estimating the radar's own position based on the point cloud output from the radar.

12. The sensor system according to claim 10 , wherein the change in the vehicle operation mode is to slow down the vehicle's moving speed compared to a normal operation mode.

13. The sensor system according to claim 10 , wherein the change in the vehicle operation mode is to move the vehicle to or return to a predetermined location.

14. Further comprising a GNSS device capable of receiving GNSS signals; the predetermined position is set in a global reference geodetic system using the GNSS signals together with a three-dimensional map previously stored in the vehicle; The sensor system according to claim 13 , wherein the sensor system identifies its own position and the predetermined position in a global reference geodetic system based on a map of the surrounding environment created using the radar information and the three-dimensional map.

15. The sensor system of claim 14 , wherein the predetermined location includes a starting point where the vehicle starts traveling, a garage, or a location where there is a roof.

16. the operation change unit determines whether the external environment information is appropriate for the radar and the optical sensor based on a level of environmental noise around the vehicle; When it is determined that the external environment information is inappropriate, the operation mode of the vehicle and the operation mode of the radar are changed, and A sensor system as described in any one of claims 3 to 15, wherein when it is determined that the external information has recovered from an inappropriate state to an appropriate state, the change between the vehicle's operating mode and the radar's operating mode is switched after a certain period of time after the determination.

17. A sensing method using a sensor system mounted on a vehicle and including at least a radar and an optical sensor, a detection step of detecting an object existing around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor; and an operation change step of changing the operation mode of the vehicle based on external environmental information about the surroundings of the vehicle.

18. A program to be executed by a computer of a sensor system mounted on a vehicle and including at least a radar and an optical sensor, a detection step of detecting an object existing around the vehicle based on at least one of radar information detected by the radar and optical sensor information detected by the optical sensor; an operation change step of changing an operation mode of the vehicle based on external environment information about the vehicle.

Images