Monitoring device

Abstract

The objective is to obtain a monitoring device that is less susceptible to changes in illuminance due to weather conditions such as changes in ambient solar radiation direction and intensity, and that does not require a large information processing load such as image contour extraction. [Solution] The monitoring device comprises a detector that detects surrounding objects as a point cloud, a reference information storage unit that stores the position of a reference object and point cloud information of the reference object when no fog is present as reference position and reference point cloud information, a reference position point cloud information acquisition unit that acquires point cloud information of the reference position detected by the detector, and a fog level estimation unit that estimates the fog level around the detector by comparing the point cloud information acquired by the reference position point cloud information acquisition unit with the reference point cloud information stored in the reference information storage unit.

Description

Technical Field

[0006] , , , , , , ,

[0005] ,

[0001] This application relates to a monitoring device.

Background Art

[0002] In recent years, the introduction of autonomous driving technology into vehicles has been desired. Technologies for realizing autonomous driving by performing driving support for vehicles using information obtained by detectors of roadside monitoring devices provided along roads have been disclosed. The roadside monitoring device detects surrounding objects by a detector, identifies vehicles, pedestrians, buildings, and other objects, and provides information on the position, speed, and type of the detected objects.

[0003] As the detector of the roadside monitoring device, an optical detector is often used to identify the type of an object. The roadside monitoring device is often installed outdoors, and it is conceivable that fog may occur in the installation environment. When fog occurs, it is known that the recognition performance of the detector of the roadside monitoring device deteriorates. In an optical detector, light is absorbed and scattered by fog, making it difficult to detect objects, and there may be problems such as erroneously detecting an object that does not actually exist.

[0004] ​​​​​​​​​​​​​​​​​​​​​​​In the monitoring device described in Patent Document 1, buildings, signs, etc., in the vicinity of the monitoring device are used as markers. If a marker that is farther away than the detected object cannot be identified, the detected object is determined to be a false detection. Marker identification is determined by identifying the contour of the captured image using edge extraction, etc., and comparing it with the contour extracted from an image under good conditions to determine whether the type of marker was correctly recognized.

[0007] The images captured by the markers are affected by changes in solar radiation direction and intensity, as well as weather conditions such as rain and fog. Therefore, there is a problem in that the system is not robust to changes in illuminance on the markers. Furthermore, image contour extraction and other processes place a significant information processing load. This necessitates expensive processing equipment and a certain amount of processing time.

[0008] This disclosure aims to provide a monitoring device that can solve these problems. Specifically, it aims to provide a monitoring device that is less susceptible to changes in illuminance due to weather conditions such as changes in ambient solar radiation direction and intensity, and that does not require a large information processing load such as image contour extraction. [Means for solving the problem]

[0009] The monitoring device related to this disclosure is A detector that detects surrounding objects as a point cloud, which is a collection of three-dimensional positions. The reference information storage unit stores the position of a reference object, which is used as a reference for estimating the fog level, and the point cloud information of the reference object in a fog-free state, as detected by the detector, as reference position and reference point cloud information. A reference position point cloud information acquisition unit that acquires point cloud information of a reference position detected by a detector, and The system includes a fog level estimation unit that estimates the fog level around the detector by comparing the point cloud information acquired by the reference position point cloud information acquisition unit with the reference point cloud information stored in the reference information storage unit. [Effects of the Invention]

[0010] According to the monitoring device described herein, a detector that detects surrounding objects as a point cloud, which is a collection of three-dimensional positions, can be used to estimate the fog level by comparing the point cloud information of a reference object when no fog is present with the point cloud information detected by the detector. This makes it possible to obtain a monitoring device that is less affected by weather conditions such as changes in solar radiation direction and intensity, rain, and fog, and does not require a large information processing load such as image contour extraction. As a result, the stability of the monitoring device's performance can be improved, and the cost of the monitoring device can be reduced. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the configuration of the monitoring device according to Embodiment 1. [Figure 2] This is a hardware configuration diagram of the monitoring device according to Embodiment 1. [Figure 3] This figure shows the monitoring device and the object to be detected according to Embodiment 1. [Figure 4] This figure shows the monitoring device according to Embodiment 1 and the point cloud of the detected object. [Figure 5] This is the first figure showing the monitoring device and the region of the reference object according to Embodiment 1. [Figure 6] This is the first figure showing the reference position and reference point cloud information of the reference object according to Embodiment 1. [Figure 7] This is the first figure showing an example of point cloud information of a reference object according to Embodiment 1. [Figure 8] This diagram shows the relationship between visibility and fog level according to Embodiment 1. [Figure 9] This is the first figure showing the relationship between the number of reference point cloud data, the number of measured point cloud data, and the fog level according to Embodiment 1. [Figure 10] This is the first figure showing an example of the number of measured point cloud data and fog level according to Embodiment 1. [Figure 11] This is the second figure showing the relationship between the number of reference point cloud data, the number of measured point cloud data, and the fog level according to Embodiment 1. [Figure 12]It is a second diagram showing an example of the number of measurement point group data and the fog level according to Embodiment 1. [Figure 13] It is a second diagram showing the regions of the monitoring device and the reference object according to Embodiment 1. [Figure 14] It is a second diagram showing the reference position and reference point cloud information of the reference object according to Embodiment 1. [Figure 15] It is a second diagram showing an example of the point cloud information of the reference object according to Embodiment 1. [Figure 16] It is a third diagram showing an example of the number of measurement point group data and the fog level according to Embodiment 1. [Figure 17] It is a fourth diagram showing an example of the number of measurement point group data and the fog level according to Embodiment 1. [Figure 18] It is a flowchart showing the processing of the monitoring device according to Embodiment 1. [Figure 19] It is a configuration diagram of the monitoring device according to Embodiment 2. [Figure 20] It is a flowchart showing the processing of the monitoring device according to Embodiment 2. [Figure 21] It is a configuration diagram of the monitoring device according to Embodiment 3. [Figure 22] It is a first flowchart showing the processing of the monitoring device according to Embodiment 3. [Figure 23] It is a second flowchart showing the processing of the monitoring device according to Embodiment 3.

Modes for Carrying Out the Invention

[0012] The embodiments will be described in detail below with reference to the drawings. Note that the drawings are schematic representations, and for the sake of clarity, some components may be omitted or simplified as appropriate. Furthermore, the relative sizes and positions of components shown in different drawings are not necessarily precisely represented and may be modified as appropriate. In the following description, similar components will be denoted by the same reference numerals, and their names and functions will also be the same. Therefore, detailed explanations of these components may be omitted to avoid redundancy.

[0013] 1. Embodiment 1 <Configuration of the monitoring device> Figure 1 is a configuration diagram of the monitoring device 200 according to Embodiment 1. The monitoring device 200 receives information about surrounding objects detected by the detector 20. The monitoring device 200 processes the information received from the detector 20 and outputs it to the information transmission device 40.

[0014] The monitoring device 200 may be, for example, a roadside monitoring device. However, the monitoring device 20 is not limited to this, and may be a security monitoring device for monitoring intruders. Furthermore, the monitoring device 200 may be an external monitoring device mounted on a mobile vehicle. Here, we will explain using a roadside monitoring device as an example.

[0015] The detector 20 can detect surrounding objects as a point cloud, which is a collection of three-dimensional positions. The detector 20 is a sensor that emits light or radio waves and obtains information about objects from the reflected light or waves, such as LiDAR (Light Detection and Ranging), laser radar, or millimeter-wave radar (MMWR). The following explanation will use LiDAR, which is commonly installed in roadside monitoring devices, as an example.

[0016] The information transmission device 40 may be a transmitter that receives the output of the monitoring device 200 and transmits it to another device. The information transmission device 40 is not limited to this and may be a display device that issues a warning about the surrounding fog level. The detector 20 and the information transmission device 40 may be built into the monitoring device 200.

[0017] The monitoring device 200 includes a reference information storage unit 201, a reference position point cloud information acquisition unit 202, a fog level estimation unit 203, and an object extraction unit 209. The reference information storage unit 201 stores the position of a reference object, which is an object used as a reference for estimating the fog level, as the reference position. Furthermore, the point cloud information of the reference object detected by the detector 20 when no fog is present is stored as reference point cloud information.

[0018] The reference position point cloud information acquisition unit 202 acquires point cloud information of the reference position detected by the detector 20. The fog level estimation unit 203 compares the point cloud information acquired by the reference position point cloud information acquisition unit 202 with the reference point cloud information stored in the reference information storage unit 201 to estimate the fog level around the detector. The object extraction unit 209 extracts objects from the point cloud information received from the detector 20 and outputs information about the objects to the information transmission device 40.

[0019] <Hardware configuration of the monitoring device> Figure 2 is a hardware configuration diagram of the monitoring device 200. In this embodiment, the monitoring device 200 is an electronic control device installed on the roadside to detect surrounding objects in order to assist the driving of a vehicle and realize autonomous driving. Each function of the monitoring device 200 is realized by the processing circuit provided in the monitoring device 200. Specifically, the monitoring device 200 includes, as a processing circuit, a arithmetic processing unit 90 (computer) such as a CPU (Central Processing Unit), a storage device 91 that exchanges data with the arithmetic processing unit 90, an input circuit 92 that inputs external signals to the arithmetic processing unit 90, and an output circuit 93 that outputs signals from the arithmetic processing unit 90 to the outside. Each piece of hardware, such as the arithmetic processing unit 90, the storage device 91, the input circuit 92, and the output circuit 93, is connected to each other by a wired network such as a bus or a wireless network.

[0020] The arithmetic processing unit 90 may include an ASIC (Application Specific Integrated Circuit), an IC (Integrated Circuit), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), various logic circuits, and various signal processing circuits. Furthermore, multiple arithmetic processing units 90 of the same or different types may be provided, with each processing unit performing a portion of the tasks. The storage device 91 may include a RAM (Random Access Memory) configured to read and write data from the arithmetic processing unit 90, or a ROM (Read Only Memory) configured to read data from the arithmetic processing unit 90. The storage device 91 may use non-volatile or volatile semiconductor memory such as flash memory, an SSD (Solid State Drive), an EPROM, or an EEPROM. The input circuit 92 is connected to various sensors such as a detector 20 and an image detector 30, switches, and communication lines, and includes an A / D converter, communication circuits, etc., which input the output signals and communication information of these sensors and switches to the arithmetic processing unit 90. The output circuit 93 includes a drive circuit, a communication circuit, etc., that outputs control signals from the arithmetic processing unit 90, and is connected to the information transmission device 40, etc. The interfaces of the input circuit 92 and the output circuit 93 may be based on specifications such as CAN (Control Area Network) (registered trademark), Ethernet (registered trademark), USB (Universal Serial Bus) (registered trademark), DVI (Digital Visual Interface) (registered trademark), HDMI (High-Definition Multimedia Interface) (registered trademark).

[0021] Each function of the monitoring device 200 is realized by the arithmetic processing unit 90 executing software (programs) stored in a storage device 91 such as ROM, and cooperating with other hardware of the monitoring device 200, such as the storage device 91, input circuit 92, and output circuit 93. Setting data such as thresholds and judgment values ​​used by the monitoring device 200 are stored in the storage device 91 such as ROM as part of the software (program). Each function of the monitoring device 200 may be composed of software modules, or it may be composed of a combination of software and hardware.

[0022] <Monitoring device and detected object> Figure 3 shows the monitoring device 200 and the object to be detected according to Embodiment 1. In Figure 3, the monitoring device 200 is shown as an integral part of the structure supporting the detector 20. In practice, the monitoring device 200 may be mounted on top of a structure several meters above the ground, together with the detector 20 which is equipped with external monitoring equipment such as a lidar, millimeter-wave radar, visible light camera, or infrared camera.

[0023] Figure 3 shows an example where objects 300, 400, 500, and 600 are present around the detector 20. The dashed line indicates the range in which the detector 20, acting as a lidar, emits laser light and detects the reflected light as a point cloud. For example, it shows the case where object 300 is a building, object 400 is a sign, object 500 is a vehicle, and object 600 is a pedestrian. In this case, objects 300 and 400 are stationary objects, while objects 500 and 600 are moving objects.

[0024] Figure 4 shows the monitoring device 200 and the point cloud of the detected object according to Embodiment 1. The lidar rapidly performs the procedure of briefly irradiating with laser light, checking the reflected light, changing the irradiation angle, and briefly irradiating with laser light again, thereby detecting surrounding objects as a point cloud. The point cloud obtained by the lidar is usually acquired at intervals of a few Hz to 10 Hz, and the number of point cloud data per frame is usually several thousand to tens of thousands.

[0025] A point cloud is obtained when laser light emitted from a lidar is reflected by the surfaces of objects within the irradiation range. This point cloud contains information such as three-dimensional position and reflection intensity. By performing processes such as clustering on the point cloud, objects such as vehicles, people, and structures within the target range can be detected. Note that when performing object detection from a point cloud, it is common practice to remove points that are presumed to be the ground before processing; therefore, points reflected from the ground are not depicted in Figure 4.

[0026] <First embodiment (when the reference object is singular)> The first embodiment describes the case where there is a single reference object. Figure 5 is the first diagram showing the monitoring device 200 and region A of the reference object 300 according to Embodiment 1. Lidars using laser light surpass the identification accuracy of other sensors in distance measurement. However, since laser light is scattered, dispersed, or absorbed by fog and rain, it can be difficult to distinguish objects under adverse weather conditions, and data noise may increase. To address these issues, it is important to estimate the fog level around the detector 20.

[0027] To estimate the fog level around the detector 20, a fixed object such as a nearby building is designated as a reference object. In Figure 5, a building is designated as the reference object 300. Region A is defined as the area for detecting point clouds relative to the reference object 300. In Figure 5, region A, defined by the dashed line, covers only a portion of the reference object 300. This approach simplifies processing by limiting the number of data points used as a reference. Region A may also cover the entire reference object 300.

[0028] Region A, which covers part or all of the reference object 300, can also be said to indicate the reference position, which is the location of the reference object 300. Region A and the point cloud information of region A when no fog is present are stored in the reference information storage unit 201 in advance as the reference position and reference point cloud information of the reference object 300.

[0029] Let's consider the case where point cloud information is newly acquired for region A, which is the reference location. By comparing this newly acquired point cloud information with the reference point cloud information stored in the reference information storage unit 201, the current fog level can be estimated. This is because when fog occurs, the number of point cloud data obtained decreases because the laser light from the lidar is scattered, dispersed, or absorbed.

[0030] <Reference position and reference point cloud information of the reference object> Figure 6 is the first figure showing the reference position and reference point cloud information of the reference object 300 according to Embodiment 1. Figure 6 shows information for representing the position and range of region A. In the figure, it is described as a rectangular region (center position coordinates and width, length, height, and rotation in the azimuth direction), but this definition method is not limited to this. Any information that can represent the position and range of a specific region is acceptable.

[0031] Figure 6 shows the reference position (region A) of the reference object 300, along with reference point cloud information, which represents the expected characteristics of the point cloud in region A when no fog is present. The figure shows the number of point cloud data points and the average value of the point cloud's reflectance. In addition, information representing the characteristics of the point cloud in region A when no fog is present may be included, such as information on the reflectance for each point cloud location in region A, the average, variance, maximum, and minimum values ​​of the reflectance.

[0032] The reference position (region A) and reference point cloud information stored in the reference information storage unit 201 are pre-set, but can also be configured to be rewritable externally via the input circuit 92. By making it rewritable, it becomes possible to reset to an appropriate region even if the building structure or terrain changes due to construction work, for example. Furthermore, as the reference object, it is desirable to use the side of a building or structure that minimizes factors other than fog that cause the number of point cloud data points to change (such as puddles on the ground or occlusion caused by occlusion between the reference object and the lidar).

[0033] Furthermore, the reference point cloud information stored in the reference information storage unit 201 may be updated at all times with the data obtained from the detector 20 when the number of point cloud data points acquired by the reference position point cloud information acquisition unit 202 reaches its maximum value for the week, month, etc. Alternatively, the system may determine from separately obtained weather information whether the point cloud information obtained by the lidar is not obstructed by weather factors such as fog, and update the data with the point cloud information at that time. In this way, it becomes possible to flexibly respond to deterioration of the laser light source, changes in reflectivity due to aging of the reference object, etc.

[0034] <Detected point cloud information> Figure 7 is the first diagram showing an example of point cloud information of a reference object 300 according to Embodiment 1. The reference position point cloud information acquisition unit 202 acquires point cloud information of the reference position from the detector 20 based on the reference position (region A) acquired from the reference information storage unit 201. Figure 7 shows an example of the acquired point cloud information. For each point where reflected light is obtained, the coordinates and the intensity of the reflected light are indicated.

[0035] The reference position (region A) and the actually obtained point cloud information are linked, and the point cloud information is transmitted to the fog level estimation unit 203. Here, it is not necessary to use the point cloud information obtained by the reference position point cloud information acquisition unit 202 as is; any operations such as downsampling to reduce processing load may be performed. Another method to estimate the fog level from the change in the intensity of reflected light at each point in the point cloud is also conceivable.

[0036] Figure 8 shows the relationship between visibility and fog level according to Embodiment 1. Here, visibility can be defined as the distance at which a given shape can be seen with the naked eye. In Figure 8, the fog level is divided into four stages, but the level division is not limited to four stages.

[0037] <Estimation of fog level based on the number of measurement point cloud data points> Figure 9 is the first figure showing the relationship between the number of reference point cloud data points Nst, the number of measured point cloud data points Ndt, and the fog level Lfg according to Embodiment 1. The reference position point cloud information acquisition unit 202 can estimate the fog level by focusing on the number of point cloud data points from the point cloud information of the reference position obtained from the detector 20 and comparing it with the number of point cloud data points of the reference point cloud information stored in the reference information storage unit 201.

[0038] If the number of measured point cloud data Ndt, which is the number of point cloud data acquired by the reference position point cloud information acquisition unit 202, is less than the value obtained by subtracting a predetermined point cloud data threshold Nth from the number of reference point cloud data Nst, which is the number of point cloud data in the reference point cloud information stored in the reference information storage unit 201, then it can be estimated that the fog level around the detector is high. That is, if the number of measured point cloud data Ndt is greater than or equal to the value obtained by subtracting the first point cloud data threshold Nth1 from the number of reference point cloud data Nst, then the fog level is set to 0. If the number of measured point cloud data Ndt is less than the value obtained by subtracting the first point cloud data threshold Nth1 from the number of reference point cloud data Nst, then the fog level is set to 1. If the number of measured point cloud data Ndt is less than the value obtained by subtracting the second point cloud data threshold Nth2 from the number of reference point cloud data Nst, then the fog level is set to 2. If the number of measured point cloud data points Ndt is less than the value obtained by subtracting the third point cloud data point threshold Nth3 from the number of reference point cloud data points Nst, the fog level will be set to 3.

[0039] By setting multiple point cloud data thresholds Nth in this way, the fog level increases as the number of measured point cloud data points Ndt decreases, allowing for gradual changes in the fog level. Therefore, the fog level can be estimated more precisely, improving the accuracy of fog level estimation.

[0040] As described above, since it only involves a simple comparison of the number of point cloud data points, the fog level can be estimated quickly and easily. Furthermore, since there is no need to perform image contour extraction, it eliminates the need for high-speed, large-scale information processing, contributing to a reduction in the cost of monitoring equipment.

[0041] Figure 10 is the first figure showing an example of estimating the number of measured point cloud data points Ndt and the fog level Lfg according to Embodiment 1. Since the number of measured point cloud data points Ndt is 130, the fog level Lfg is 1 according to the conditions in Figure 9.

[0042] Figure 11 is a second figure showing the relationship between the number of reference point cloud data Nst, the number of measured point cloud data Ndt, and the fog level Lfg according to Embodiment 1. When the point cloud data ratio RF (=Ndt / Nst), obtained by dividing the number of measured point cloud data Ndt acquired by the reference position point cloud information acquisition unit 202 by the number of reference point cloud data Nst of the reference point cloud information stored in the reference information storage unit 201, is smaller than a predetermined point cloud data ratio threshold (RFth), it can be estimated that the fog level (Lfg) around the detector is at a high level.

[0043] Specifically, if the point cloud data ratio RF is greater than or equal to the first point cloud data ratio threshold RFth1, the fog level is set to 0. If the point cloud data ratio RF is less than the first point cloud data ratio threshold RFth1, the fog level is set to 1. If the point cloud data ratio RF is less than the second point cloud data ratio threshold RFth2, the fog level is set to 2. If the point cloud data ratio RF is less than the third point cloud data ratio threshold RFth3, the fog level is set to 3.

[0044] By setting multiple point cloud data ratio thresholds (RFth) in this way, the fog level increases as the point cloud data ratio RF decreases, allowing for gradual changes in the fog level. Therefore, the fog level can be estimated more precisely, improving the accuracy of fog level estimation.

[0045] As described above, since it only involves a simple comparison of the point cloud data ratio RF, which is the ratio of the number of measured point cloud data points Ndt to the number of reference point cloud data points Nst, the fog level can be estimated quickly and easily. Furthermore, since there is no need to perform image contour extraction, it eliminates the need for high-speed, large-scale information processing, contributing to a reduction in the cost of monitoring equipment.

[0046] Figure 12 is a second figure showing an example of estimating the number of measured point cloud data points Ndt and the fog level Lfg according to Embodiment 1. Since the number of measured point cloud data points Ndt is 130, the fog level Lfg is 1 according to the conditions in Figure 11.

[0047] <Second example (when there are multiple reference objects)> The second embodiment describes the case where there are multiple reference objects. Figure 13 is the second figure showing the region of the reference object according to Embodiment 1. Figure 13 shows region A for reference object 300 and region B for reference object 400. As a countermeasure against fog, it is important to estimate the fog level around the detector 20. Therefore, by acquiring point cloud information for multiple reference objects and comparing it with the reference point cloud information for each, the fog level can be estimated more appropriately. Three or more reference objects may be provided, but here we will explain using the case where there are two reference objects as an example.

[0048] To estimate the fog level around detector 20, fixed objects such as surrounding buildings and signs are defined as reference objects. In Figure 13, reference object 300, which is a building, and reference object 400, which is a sign, are defined. Region A is defined as the area for point cloud detection relative to reference object 300. Region B is defined as the area for point cloud detection relative to reference object 400. In Figure 13, regions A and B are indicated by dashed lines.

[0049] Region A covers a portion of the reference object 300, but it may also cover the entire reference object 300. Region B covers the entire reference object 400, but it may also cover a portion of the reference object 400.

[0050] Regions A and B, which cover part or all of reference object 300 and reference object 400, can be rephrased as indicating reference positions, which are the positions of the reference objects, respectively. Point cloud information of region A and region A when no fog is present is stored in the reference information storage unit 201 in advance as the reference position and reference point cloud information of reference object 300. Point cloud information of region B and region B when no fog is present is stored in the reference information storage unit 201 in advance as the reference position and reference point cloud information of reference object 400.

[0051] Let's consider the case where point cloud information for reference location region A and point cloud information for reference location region B are newly acquired. By comparing this newly acquired point cloud information with the reference point cloud information stored in the reference information storage unit 201, the current fog level can be estimated. This is because when fog occurs, the number of point cloud data obtained decreases because the laser light from the lidar is scattered, dispersed, or absorbed.

[0052] <Reference position and reference point cloud information of the reference object> Figure 14 is a second figure showing the reference position and reference point cloud information of the reference object according to Embodiment 1. Figure 14 shows information for representing the position and range of region A and region B. In the figure, it is described as a rectangular parallelepiped region (center position coordinates and width, length, height, and rotation in the azimuth direction), but this definition method is not limited to this. Any information that can represent the position and range of a specific region is acceptable.

[0053] Figure 14 shows the reference position of reference object 300 (region A) and the reference position of reference object 400 (region B). Along with this, it includes reference point cloud information that represents the expected characteristics of the point clouds in regions A and B when fog is not present. In the figure, the number of point cloud data points and the average value of the point cloud's reflectance are shown for regions A and B. In addition, information that represents the characteristics of the point clouds in regions A and B when fog is not present may be included, such as information on the reflectance at each position in the point clouds of regions A and B, the average, variance, maximum, and minimum values ​​of the reflectance.

[0054] The reference positions (regions A and B) and reference point cloud information stored in the reference information storage unit 201 are pre-set, but the system can also be configured to allow external rewriting via the input circuit 92. Furthermore, the reference point cloud information stored in the reference information storage unit 201 may be constantly updated with data from the point cloud information obtained from the detector 20, specifically the data obtained by the reference position point cloud information acquisition unit 202 when the number of point cloud data points for the week, month, etc., is at its maximum. This would allow for flexible responses to degradation of the laser light source, changes in reflectivity due to aging of the reference object, and other factors.

[0055] <Detected point cloud information> Figure 15 is the first figure showing an example of point cloud information for reference object 300 and reference object 400 according to Embodiment 1. The reference position point cloud information acquisition unit 202 acquires point cloud information of the reference position from the detector 20 based on the reference position (region A and region B) acquired from the reference information storage unit 201. Figure 15 shows an example of the acquired point cloud information. For each point where reflected light is obtained, the coordinates and intensity of the reflected light are indicated.

[0056] The reference positions (regions A and B) are linked to the actually obtained point cloud information, and the point cloud information is transmitted to the fog level estimation unit 203. Here, it is not necessary to use the point cloud information obtained by the reference position point cloud information acquisition unit 202 as is; any operations such as downsampling to reduce processing load may be performed. Another method for estimating the fog level is to consider the change in the intensity of reflected light at each point in the point cloud.

[0057] <Estimation of fog level using the average value of the point cloud data ratio RF> Figure 16 is a third figure showing an example of the number of measured point cloud data Ndt and fog level Lfg according to Embodiment 1. The reference position point cloud information acquisition unit 202 can estimate the fog level Lfg by focusing on the number of point cloud data from the point cloud information of the reference position (region A, region B) obtained from the detector 20 and comparing it with the number of reference point cloud data Nst of the reference point cloud information stored in the reference information storage unit 201.

[0058] Here, we will estimate the fog level Lfg by applying the relationship shown in Figure 11, based on the value of the point cloud data ratio RF, which is obtained by dividing the number of measured point cloud data Ndt by the number of reference point cloud data Nst. For region A, the number of measured point cloud data Ndt is 90, so the point cloud data ratio RF = 90 / 200 = 0.45, and the fog level Lfg = 1. For region B, the number of measured point cloud data Ndt is 65, so the point cloud data ratio RF = 60 / 100 = 0.65, and the fog level Lfg = 2. Here, we will calculate the point cloud data ratio RF for each of the two reference objects by dividing the number of measured point cloud data Ndt by the number of reference point cloud data Nst, and estimate the fog level Lfg according to the average ratio obtained by averaging the point cloud data ratio RF calculated for each reference object.

[0059] In Figure 16, the average value of the point cloud data ratio RF is 0.55, and the fog level Lfg is estimated to be 1. Here, the fog level Lfg is estimated based on the average value of multiple point cloud data ratio RFs, but it may also be estimated based on the minimum value of multiple point cloud data ratio RFs.

[0060] By setting a point cloud data ratio threshold RFth, as shown in Figure 11, to compare with the average value of multiple point cloud data ratios RF, the fog level Lfg can be continuously estimated, improving the accuracy of the fog level Lfg estimation. Furthermore, by using the number of measured point cloud data Ndt for multiple reference objects, the reliability of the fog level Lfg estimation can be improved.

[0061] <Estimation of fog level using weighted mean ratio of point cloud data points (RF)> Figure 17 is the fourth figure showing an example of the number of measured point cloud data points Ndt and fog level Lfg according to Embodiment 1. Figure 17 differs from Figure 16 in that, in estimating the fog level using the average value of the point cloud data point ratio RF, the average value is calculated based on the value of the point cloud data point ratio RF obtained by dividing the number of measured point cloud data points Ndt of each reference object by the number of reference point cloud data points Nst, multiplied by a weighting coefficient (W).

[0062] In Figure 17, the point cloud data ratio RF of the reference object 300 related to region A is multiplied by a weighting coefficient W1 = 1.1. Then, the point cloud data ratio RF of the reference object 400 related to region B is multiplied by a weighting coefficient W2 = 0.8. The average value of the point cloud data ratio RF multiplied by the weighting coefficients is then calculated, and the fog level Lfg is determined based on the point cloud data ratio threshold RFth shown in Figure 11. The estimation result in Figure 17 is fog level Lfg = 1. In this case, however, the fog level Lfg may also be estimated according to the minimum value obtained by multiplying the weighting coefficient W of multiple point cloud data ratios RF.

[0063] The way in which the number of measured point cloud data points Ndt changes in response to fog generation is expected to vary depending on the distance from the lidar to the reference object, the material of the reference object, and other factors. Therefore, by setting a weighting coefficient W for each reference object, the fog level Lfg can be appropriately estimated.

[0064] <Processing by monitoring device> Figure 18 is a flowchart showing the processing of the monitoring device 200 according to Embodiment 1. The processing shown in Figure 18 is executed by the arithmetic processing unit of the monitoring device 200. This processing may be executed at predetermined intervals (for example, every minute). Alternatively, it may be executed in response to events, such as whenever the monitoring device 200 receives point cloud information from the detector 20, rather than at predetermined intervals.

[0065] The process shown in Figure 18 is initiated, and in step S101, the monitoring device 200 receives point cloud information from the detector 20. The received point cloud information may be output to the information transmission device 40 via the object extraction unit 209 as needed.

[0066] In step S102, the reference position point cloud information acquisition unit 202 acquires the point cloud information from the detector 20 that corresponds to the reference position (region of the reference object). In step S105, the reference point cloud information is read from the reference information storage unit 201.

[0067] In step S106, the fog level Lfg is estimated from the number of measured point cloud data Ndt extracted from the point cloud corresponding to the reference position and the number of reference point cloud data Nst extracted from the reference point cloud information.

[0068] In step S107, the estimated fog level Lfg is output to the information transmission device 40. Then, the process ends.

[0069] Through the processing described above, the monitoring device 200 can estimate the fog level by comparing point cloud information for a reference object when no fog is present with point cloud information detected by the detector. This makes it possible to obtain a monitoring device that is less affected by weather conditions such as changes in solar radiation direction and intensity, rain, and fog, and does not require a large information processing load such as image contour extraction. As a result, the stability of the monitoring device's performance can be improved, and the cost of the monitoring device can be reduced.

[0070] 2. Embodiment 2 <Configuration of the monitoring device> Figure 19 is a configuration diagram of the monitoring device 200 according to Embodiment 2. The configuration diagram in Figure 19 differs from the configuration diagram in Figure 1 in that a point cloud information usage prohibition unit 204 has been added. The other configurations are the same as those in Figure 1 according to Embodiment 1. The hardware configuration diagram of the monitoring device 200 in Figure 2 can also be applied to the monitoring device 200 according to Embodiment 2.

[0071] The point cloud information usage prohibition unit 204 decides whether to use the point cloud information obtained from the detector 20 to perform fog level estimation by the fog level estimation unit 203, or to prohibit fog level estimation. For example, if other objects such as a vehicle 500 or pedestrian 600 pass through the visibility range between the detector 20 and the reference object, the number of point cloud data within the region of the reference object is expected to change rapidly due to the effect of occlusion. In that case, the accuracy of fog level estimation may deteriorate.

[0072] <Prohibited use of point cloud information> To avoid this, the point cloud information usage prohibition unit 204 prohibits the fog level estimation unit 203 from using the point cloud information acquired by the reference position point cloud information acquisition unit 202 if the absolute value of the rate of change per unit time ΔNdt of the number of measured point cloud data Ndt acquired by the reference position point cloud information acquisition unit 202 is greater than a predetermined rate of change threshold ΔNdtth. This is because if the number of measured point cloud data Ndt changes significantly, the accuracy of fog level estimation may decrease.

[0073] It is assumed that the change in the number of measured point cloud data points Ndt due to fog does not occur abruptly. "If there is a significant change (e.g., a decrease of 50% or more) between the number of measured point cloud data points Ndt at the time of the previous measurement and the number of measured point cloud data points Ndt at the time of the current measurement, it will be judged that some kind of anomaly has occurred in that area and it will not be used in the calculation of the fog level." This will allow areas affected by factors other than fog to be excluded from the calculation of the fog level, and is expected to improve the accuracy of fog level estimation.

[0074] <Processing by monitoring device> Figure 20 is a flowchart showing the processing of the monitoring device 200 according to Embodiment 2. The processing shown in Figure 20 is executed by the arithmetic processing unit of the monitoring device 200. This processing may be executed at predetermined intervals (for example, every minute). Alternatively, it may be executed in response to events, such as whenever the monitoring device 200 receives point cloud information from the detector 20, rather than at predetermined intervals.

[0075] The difference between the process in Figure 20 and that in Figure 18 is that steps S103 and S104 are inserted between steps S102 and S105. Only the differences will be explained here.

[0076] After step S102, in step S103, the rate of change per unit time ΔNdt for the number of measured point cloud data points Ndt is calculated. Then, in step S104, it is determined whether the absolute value of the rate of change per unit time ΔNdt is greater than the rate of change threshold ΔNdtth. If the absolute value of the rate of change per unit time ΔNdt is greater than the rate of change threshold ΔNdtth (determination is YES), the process is terminated without performing fog level estimation. If the absolute value of the rate of change per unit time ΔNdt is not greater than the rate of change threshold ΔNdtth (determination is NO), the process proceeds to step S105.

[0077] By performing this process, if the number of measured point cloud data points Ndt changes significantly, the fog level estimation unit 203 can avoid using the point cloud information. This improves the accuracy of fog level estimation.

[0078] 3. Embodiment 3 <Configuration of the monitoring device> Figure 21 is a configuration diagram of the monitoring device 200 according to Embodiment 3. The configuration diagram in Figure 21 differs from the configuration diagram in Figure 19 in that it receives a signal from an external image detector 30 and an internal reference position image information acquisition unit 205 has been added. The other configurations are the same as those in Figure 19 according to Embodiment 2. The hardware configuration diagram of the monitoring device 200 in Figure 2 can also be applied to the monitoring device 200 according to Embodiment 3.

[0079] The reference position image information acquisition unit 205 acquires image information from the image detector 30. The image detector 30 is a visible light camera, an infrared light camera, etc. Similar to the detector 20, the image detector 30 is installed at a position and field of view in which the reference object is visible, and images are taken at intervals of typically a few Hz to 10 Hz, and the image information is transmitted via any communication method such as a USB / LAN cable or wireless communication. The coordinates of the camera and the real world are assumed to be calibrated in advance.

[0080] In the monitoring device 200 according to Embodiment 3, the reference information storage unit 201 stores as a template image an image of the region of a reference object when there are no weather changes such as rain or fog, and no occlusion caused by an object passing in front of it. The point cloud information usage prohibition unit 204 compares the image acquired from the reference position image information acquisition unit 205 with the template image acquired from the reference information storage unit 201 and determines whether the image of the reference position matches the template image.

[0081] Existing algorithms such as template matching are used to determine the match. If a match is not determined, the point cloud information usage prohibition unit 204 determines that the point cloud information for that region will not be used for estimating the fog level of the reference object. This identifies conditions that affect fog level calculation, such as puddles or snow cover, and prohibits fog level estimation. Therefore, in the configuration of Embodiment 2, it is possible to determine whether to prohibit the use of point cloud information even in situations where determination is difficult. This is expected to improve the accuracy of fog level estimation.

[0082] <Processing by monitoring device> Figure 22 is a first flowchart showing the processing of the monitoring device 200 according to Embodiment 2. Figure 23 is a first flowchart. Figure 23 shows a continuation of the flowchart in Figure 22. The processing shown in Figures 22 and 23 is executed by the arithmetic processing unit of the monitoring device 200. This processing may be executed at predetermined intervals (for example, every minute). Rather than at predetermined intervals, it may be executed by events such as each time the monitoring device 200 receives point cloud information from the detector 20.

[0083] The difference between the processes in Figures 22 and 23 and Figure 20 is that the process in Figure 22 is inserted before step S101 in Figure 20. Here, only the difference will be explained.

[0084] The process shown in Figure 22 is initiated, and in step S201, the monitoring device 200 receives image information from the image detector 30. The received image information is sent to the reference position image information acquisition unit 205. At this point, the image information may also be transmitted directly to the information transmission device 40 and sent externally.

[0085] In step S102, the reference position image information acquisition unit 205 acquires image information of the reference position (region) where the reference object exists. Then, in step S203, the template image of the reference object is read from the reference information storage unit 201.

[0086] In step S204, the image of the reference position (region) where the reference object is located, received from the image detector 30, is compared with the template image to check if they match. In step S205, it is determined whether they match. If they match (judgment is YES), the process proceeds to step S101. The subsequent process is the same as in Figure 20. In step S205, if they do not match (judgment is NO), the process is terminated.

[0087] By performing this process, the fog level estimation can be stopped if the image does not match the reference object, if an object obstructing the view is passing between the image detector 30 and the reference object, or if the reference object cannot be identified due to snowfall, heavy rain, etc. This improves the accuracy of the fog level estimation.

[0088] While this application describes various exemplary embodiments and examples, the various features, aspects, and functions described in one or more embodiments are not limited to the application of a particular embodiment, but are applicable individually or in various combinations to the embodiments. Accordingly, countless variations not illustrated are envisioned within the scope of the art disclosed herein. These include, for example, modifying, adding or omitting at least one component, or even extracting at least one component and combining it with a component from another embodiment.

[0089] The various aspects of this disclosure are summarized below as an appendix.

[0090] (Note 1) A detector that detects surrounding objects as a point cloud, which is a collection of three-dimensional positions. The reference information storage unit stores the position of a reference object, which is used as a reference for estimating the fog level, and the point cloud information of the reference object in a fog-free state, as detected by the detector, as the reference position and reference point cloud information. A reference position point cloud information acquisition unit acquires point cloud information of the reference position detected by the detector, and A monitoring device comprising a fog level estimation unit that estimates the fog level around the detector by comparing the point cloud information acquired by the reference position point cloud information acquisition unit with the reference point cloud information stored in the reference information storage unit. (Note 2) The monitoring device according to Appendix 1, wherein the fog level estimation unit estimates the fog level around the detector by comparing the number of point cloud data acquired by the reference position point cloud information acquisition unit with the number of point cloud data of the reference point cloud information stored in the reference information storage unit. (Note 3) The monitoring device according to Appendix 2, wherein the fog level estimation unit estimates that the fog level around the detector is high when the number of point cloud data acquired by the reference position point cloud information acquisition unit is less than the value obtained by subtracting a predetermined point cloud data threshold from the number of point cloud data of the reference point cloud information stored in the reference information storage unit. (Note 4) The monitoring device described in Appendix 2, wherein the fog level estimation unit estimates that the fog level around the detector is high when the ratio obtained by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit is smaller than a predetermined point cloud data ratio threshold. (Note 5) The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to Appendix 4, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector is high when the average ratio obtained by averaging the ratios calculated for each reference object is smaller than the point cloud data number ratio threshold. (Note 6) The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to Appendix 4, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector is high when the weighted average ratio obtained by multiplying the ratio calculated for each reference object by a weight coefficient predetermined for each reference object is smaller than the point cloud data number ratio threshold. (Note 7) The monitoring device according to Appendix 2, wherein the fog level estimation unit estimates that the fog level around the detector increases as the ratio obtained by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit decreases. (Note 8) The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to Appendix 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the average ratio obtained by averaging the ratios calculated for each reference object decreases. (Note 9) The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to Appendix 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the weighted average ratio, obtained by multiplying the ratio calculated for each reference object by a weight coefficient predetermined for each reference object and then averaging it, decreases. (Note 10) The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to Appendix 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the minimum value of the weighting ratio obtained by multiplying the ratio calculated for each reference object by a weighting coefficient predetermined for each reference object decreases. (Note 11) The detector is a monitoring device as described in any one of the appendices 1 to 10, which is a laser-based measuring instrument. (Note 12) The monitoring device is a roadside monitoring device installed around a road, as described in any one of the items 1 to 11 of the appendices. (Note 13) A monitoring device according to any one of the appendices 1 to 12, further comprising a point cloud information usage prohibition unit that prohibits the use of point cloud information acquired by the reference position point cloud information acquisition unit by the fog level estimation unit. (Note 14) The point cloud information usage prohibition unit is a monitoring device as described in Appendix 13, which prohibits the fog level estimation unit from using the point cloud information acquired by the reference position point cloud information acquisition unit when the absolute value of the amount of change per unit time of the number of point cloud data points acquired by the reference position point cloud information acquisition unit is greater than a predetermined change threshold. (Note 15) An image detector that detects images of surrounding objects, and The system includes a reference position image information acquisition unit that acquires an image of the reference position detected by the image detector, The reference information storage unit stores the reference position of the reference object, the reference point cloud information, and the reference image. The monitoring device according to Appendix 13 or 14, wherein the point cloud information use prohibition unit determines that the image acquired by the reference position image information acquisition unit deviates from the reference image stored in the reference information storage unit, and the fog level estimation unit prohibits the use of the point cloud information acquired by the reference position point cloud information acquisition unit. [Explanation of symbols]

[0091] 20 Detector, 30 Image detector, 200 Monitoring device, 201 Reference information storage unit, 202 Reference position point cloud information acquisition unit, 203 Fog level estimation unit, 204 Point cloud information usage prohibition unit, 205 Reference position image information acquisition unit, 300, 400 Reference object

Claims

1. A detector that detects surrounding objects as a point cloud, which is a collection of three-dimensional positions. The reference information storage unit stores the position of a reference object, which is used as a reference for estimating the fog level, and the point cloud information of the reference object in a fog-free state, as detected by the detector, as the reference position and reference point cloud information. A reference position point cloud information acquisition unit acquires point cloud information of the reference position detected by the detector, and A monitoring device comprising a fog level estimation unit that estimates the fog level around the detector by comparing the point cloud information acquired by the reference position point cloud information acquisition unit with the reference point cloud information stored in the reference information storage unit.

2. The monitoring device according to claim 1, wherein the fog level estimation unit estimates the fog level around the detector by comparing the number of point cloud data acquired by the reference position point cloud information acquisition unit with the number of point cloud data of the reference point cloud information stored in the reference information storage unit.

3. The monitoring device according to claim 2, wherein the fog level estimation unit estimates that the fog level around the detector is high when the number of point cloud data acquired by the reference position point cloud information acquisition unit is smaller than the value obtained by subtracting a predetermined point cloud data threshold from the number of point cloud data of the reference point cloud information stored in the reference information storage unit.

4. The monitoring device according to claim 2, wherein the fog level estimation unit estimates that the fog level around the detector is high when the ratio obtained by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit is smaller than a predetermined point cloud data ratio threshold.

5. The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to claim 4, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector is high when the average ratio obtained by averaging the ratios calculated for each reference object is smaller than the point cloud data number ratio threshold.

6. The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to claim 4, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector is high when the weighted average ratio obtained by multiplying the ratio calculated for each reference object by a weight coefficient predetermined for each reference object is smaller than the point cloud data number ratio threshold.

7. The monitoring device according to claim 2, wherein the fog level estimation unit estimates that the fog level around the detector increases as the ratio obtained by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit decreases.

8. The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to claim 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the average ratio obtained by averaging the ratios calculated for each reference object decreases.

9. The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to claim 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the weighted average ratio, obtained by multiplying the ratio calculated for each reference object by a weight coefficient predetermined for each reference object and then averaging it, decreases.

10. The aforementioned reference information storage unit stores reference positions and reference point cloud information of multiple reference objects. The monitoring device according to claim 7, wherein the fog level estimation unit calculates a ratio for each reference object by dividing the number of point cloud data acquired by the reference position point cloud information acquisition unit by the number of point cloud data of the reference point cloud information stored in the reference information storage unit, and estimates that the fog level around the detector increases as the minimum value of the weighting ratio obtained by multiplying the ratio calculated for each reference object by a weighting coefficient predetermined for each reference object decreases.

11. The monitoring device according to any one of claims 1 to 10, wherein the detector is a laser-based measuring instrument.

12. The monitoring device is a roadside monitoring device installed around a road, according to any one of claims 1 to 10.

13. The monitoring device according to any one of claims 1 to 10, further comprising a point cloud information usage prohibition unit that prohibits the use of point cloud information acquired by the reference position point cloud information acquisition unit by the fog level estimation unit.

14. The monitoring device according to claim 13, wherein the point cloud information usage prohibition unit prohibits the fog level estimation unit from using the point cloud information acquired by the reference position point cloud information acquisition unit if the absolute value of the amount of change per unit time of the number of point cloud data points of the point cloud information acquired by the reference position point cloud information acquisition unit is greater than a predetermined change threshold.

15. An image detector that detects images of surrounding objects, and The system includes a reference position image information acquisition unit that acquires an image of the reference position detected by the image detector, The reference information storage unit stores the reference position of the reference object, the reference point cloud information, and the reference image. The monitoring device according to claim 13, wherein the point cloud information use prohibition unit determines that the image acquired by the reference position image information acquisition unit deviates from the reference image stored in the reference information storage unit, and the fog level estimation unit prohibits the use of the point cloud information acquired by the reference position point cloud information acquisition unit.

Images