Monitoring and warning device, monitoring and warning method, and program

Abstract

To ensure safety and safety during simple inspection in which main line regulation of a highway is not performed, and enable an inspector to inspect without carefulness to rearward (an anti-travel direction of a vehicle).SOLUTION: A monitoring alarm device monitors a rear side of a road from a road shoulder 91 of the road, and alerts a user of a vehicle coming into the road shoulder 91. The monitoring alarm device includes a camera 30 for imaging the rear side of the road, image processing means 51 for detecting and identifying and tracking a vehicle and a class from each image frame of a moving image, borderline setting means S21 for setting a borderline 9 so as to overlap a white line band 90 on a road surface partitioning a vehicle traveling lane 92 and the road shoulder 91, and means S41 for comparing an area ratio of two regions B1 and B2 obtained by dividing the vehicle region BB occupied by the vehicle being tracked by the borderline 9 with a warning threshold of the class of the vehicle, and for warning when the threshold is exceeded.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring and warning device that, when a vehicle enters a predetermined monitoring area, checks the danger of the vehicle and outputs a warning if the danger exists. For example, this relates to a device that detects vehicles approaching from behind, determines the risk, and issues a warning in the event of danger, in order to ensure the safety of inspection workers who temporarily park an inspection vehicle on the shoulder or leftmost lane of a highway and get out of the vehicle to visually inspect the surrounding conditions. The present invention also relates to a method executed by the above device and a program for causing a computer to function as the above device. [Background technology]

[0002] Japanese Patent Application Publication No. 2017-163374 (Patent Document 1) discloses a technology for accurately estimating the behavior of an object moving on a road within a specified area on the road based on an image input from an imaging device that captures the area. Japanese Patent Laid-Open Publication No. 2018-022234 (Patent Document 2) discloses a technology for accurately detecting objects such as other vehicles present around a vehicle from a captured image.

[0003] Object detection using deep learning is called object detection, and has been published in overseas papers (Non-Patent Documents 1, 2) since around 2016. It has also been implemented in open source software, and by spring 2018, real-time processing (30 fps or more) was possible using existing computers. For example, in object detection and identification (classification) using deep learning, relatively well-known implementations such as SSD and YOLO use transfer learning, which uses a pre-trained model using a huge number of training images, making it possible to build a model using original data with a minimal amount of computation.

[0004] Deep-SORT, an extension of SORT (Simple Online and Realtime Tracking), is an example of an object tracking technique. Deep-SORT is described in, for example, Non-Patent Document 3. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2017-163374 [Patent Document 2] Japanese Patent Application Publication No. 2018-022234 [Non-Patent Document 1] YOLOv3: An Incremental Improvement / Joseph Redmon, Ali Farhadi / University of Washington [Non-patent document 2] SSD: Single Shot MultiBox Detector / Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg [Non-patent document 3] https:nanonets.com / blog / object-tracking-deepsort / (as of June 16, 2020) Summary of the Invention [Problem to be solved by the invention]

[0006] For example, in the case of minor inspections of expressways that do not involve main lane restrictions, an inspection vehicle is temporarily parked on the shoulder of the road and a visual inspection is carried out by walking ahead of the vehicle (in the direction of travel). However, on rare occasions, vehicles may travel on the shoulder of the road, so monitoring is required to ensure safety. In other words, there is a demand for inspectors to ensure safety and security without being distracted by what is behind them (the direction opposite to the vehicle's travel direction). There is also a demand for an easy way to ensure safety and security. The present invention aims to meet such demands. [Means for solving the problem]

[0007] Examples of the present invention are described below in [1] to [8]. Note that in this section ([Means for Solving the Problem]) and the next section ([Effects of the Invention]), the symbols are used to facilitate understanding, and are not intended to limit the present invention to the symbol configuration. [1] Invention 1 A monitoring and warning device that monitors the rear side of a road from a shoulder 91 or the leftmost lane of a road and warns a user of a vehicle entering the shoulder 91 or the leftmost lane, a camera 30 that captures an image of the rear side of the road from a predetermined position in the road shoulder 91 or the leftmost lane; an image processing means 51 for detecting, identifying and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera 30; a boundary line setting means S21 for setting a virtual boundary line 9 in each of the image frames so as to overlap a white line 90 on the road surface that separates a vehicle travel lane 92 of the road from the shoulder 91 or the leftmost lane; a threshold value storage means 61 that stores, for each vehicle class, a warning threshold value for the area ratio of two areas B1, B2 that are formed by dividing a vehicle area BB occupied by a vehicle being tracked by the image processing means 51 with the boundary line 9; area ratio processing means S21, S23 for calculating the area ratio for each vehicle being tracked by the image processing means 51 and comparing the area ratio with a warning threshold value for the class to which the vehicle belongs; a warning means S41 for outputting a warning when the area ratio processing means S23, S31 determine that the area ratio of any vehicle being tracked by the image processing means 51 exceeds the warning threshold value for the class of the vehicle; A monitoring and warning device comprising:

[0008] <Terminology details> The image processing means (detector, classifier, tracker) 51 can be configured by implementing, for example, Deep-SORT and YOLOv3. Depending on the required performance, object detection and identification (classification) can be performed using, for example, SSD instead of YOLOv3. Appropriate open source (library / platform / AI application) can be selected taking into consideration the behavior of the object to be detected in the detection area and the accuracy and performance required of the object information processing device. Object detection and identification (classification) can be performed simultaneously, or in a two-stage application. Furthermore, it is preferable to use a large number of images of various vehicles (trucks, passenger cars, light trucks, buses, motorcycles, etc.) taken from the front as training data. When the area ratio is defined as B1 / B2 or B1 / (B1+B2), "exceeding the warning threshold" means "the calculated value of the area ratio is equal to or greater than the warning threshold." Conversely, if the area ratio is defined as B2 / B1 or B2 / (B1+B2), "exceeding the warning threshold" means "warning threshold ≧ (or >) the calculated value of the area ratio." Here, B1 refers to the area of ​​the area on the shoulder side of the vehicle area, and B2 refers to the area of ​​the area on the driving lane side of the vehicle area (see FIGS. 7(b) and 7(c)). The warning threshold stored in the threshold storage means is assumed to be a threshold that conforms to the above definition. For example, if B1 / B2 is used, a threshold that is determined to be dangerous for the road type in question based on the results of measurement by B1 / B2 is assumed to be prepared for each class (truck, passenger car, light truck, bus, motorcycle, etc.). For example, in the case of a highway, since approaching vehicles are traveling at high speed, the threshold is set to a value that indicates a relatively small amount of intrusion into the shoulder area so that the inspection worker has sufficient time to take shelter. The warning means S41 is, for example, a means for controlling a speaker to output a warning sound, or a means for sending warning data to a wireless receiver carried by an inspector.

[0009] [2] Invention 2 In Invention 1, The vehicle area occupied by the vehicle being tracked by the image processing means 51 is the area of ​​the bounding box BB of the vehicle. A monitoring and warning device characterized by: [3] Invention 3 In Invention 1 or Invention 2, The boundary line setting means S21 sets the boundary line 9 in response to an input from the operation input device 20. A monitoring and warning device characterized by: [4] Invention 4 In Invention 1 or Invention 2, When the white line 90 on the road surface is curved, the boundary line setting means S21 sets the boundary line 90 on the road shoulder 91 or the leftmost lane Predetermined monitoring area into a plurality of polygons 9a to 9e, and the sides of the polygons that overlap the white line strip 90 are set approximately as parts of the boundary line 9. A monitoring and warning device characterized by: [5] Invention 5 In invention 4, The polygons 9a to 9e are Predetermined monitoring area A trapezoid is formed by a plurality of parallel lines set in order at intervals from the back to the front, with the upper and lower sides being in turn A monitoring and warning device characterized by: The terms "far side" and "near side" refer to the rear side of the road (opposite the direction of travel of the vehicle). [6] Invention 6 In any one of Inventions 1 to 5, The image processing means 51 is adjusted by deep learning using a large number of image data created based on photographed images of the front side of each vehicle type, such as trucks, buses, passenger cars, light trucks, and motorcycles, as training data for vehicle detection and identification. A monitoring and warning device characterized by: [7] Invention 7 A program for causing a computer to function as any one of the monitoring and warning devices of Inventions 1 to 6.

[0010] [8] A monitoring and warning method for monitoring the rear side of a road from a shoulder 91 or the leftmost lane of a road and warning a user of a vehicle entering the shoulder 91 or the leftmost lane, Detecting, identifying and tracking a vehicle and the class to which the vehicle belongs from each image frame of a video input from a camera 30 that captures an image of the rear side of the road from the road shoulder 91 or a predetermined position in the leftmost lane, A virtual boundary line 9 is set in each image frame so as to overlap a white line 90 on the road surface that separates a vehicle driving lane 92 of the road from the shoulder 91 or the leftmost lane (S21). For each vehicle being tracked, the area ratio of two areas B1, B2 formed by dividing the vehicle area BB occupied by the vehicle by the boundary line 9 is calculated (S21), and each is compared with the warning threshold of the class to which the vehicle belongs (S23). If the calculated area ratio of any vehicle being tracked exceeds a warning threshold value for the class of the vehicle (S31), a warning is output (S41); A monitoring and warning method comprising: [Effects of the Invention]

[0011] Invention 1 is a monitoring and warning device that monitors the rear side of a road from a shoulder 91 or the leftmost lane of a road and warns a user of a vehicle entering the shoulder 91 or the leftmost lane, the monitoring and warning device comprising: a camera 30 that captures an image of the rear side of the road from a predetermined position in the shoulder 91 or the leftmost lane; image processing means 51 that detects, identifies, and tracks a vehicle and the class to which the vehicle belongs from each image frame of a video input from the camera 30; boundary line setting means S21 that sets a virtual boundary line 9 in each image frame so as to be superimposed on a white line 90 on the road surface that separates a vehicle travel lane 92 of the road from the shoulder 91 or the leftmost lane; and threshold values ​​that are stored for each vehicle class as warning thresholds for the area ratio of two areas B1, B2 formed by dividing a vehicle area BB occupied by a vehicle being tracked by the boundary line 9. Since the monitoring and warning device is characterized by having a memory means 61, area ratio processing means S21, S23 in which the image processing means 51 calculates the area ratio for each vehicle being tracked and compares it with the warning threshold for the class to which the vehicle belongs, and a warning means S41 that outputs a warning when the area ratio processing means S23, S31 determines that the area ratio of any vehicle being tracked by the image processing means 51 has exceeded the warning threshold for the class of that vehicle, for example, when patrolling a highway without main line restrictions, an inspection vehicle can be temporarily parked on the shoulder of the road, and when walking ahead of it (in the direction of vehicle travel) to perform a visual inspection, the inspection worker (user) can be warned at an appropriate time (a time when sufficient time can be secured to evacuate) that a vehicle is entering the shoulder and approaching. For inventions 2 to 6, specific configuration examples that can achieve the above can be given. Invention 7 can realize the above by a computer. Invention 8 is the same as Invention 1. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating the function and configuration of a monitoring / warning device according to an embodiment; [Figure 2] 4 is a flowchart illustrating the image processing procedure of the monitoring / alarm device of FIG. 1. [Figure 3]FIG. 2 is a block diagram illustrating a specific configuration example of the monitoring / alarming device and peripheral devices of FIG. 1. [Figure 4] 4 is a flowchart illustrating an example of a processing procedure in the monitoring / alarm device of FIG. [Figure 5] An explanatory diagram of the relationship between roads, shoulders, boundary lines, etc. [Figure 6] (a) is an explanatory diagram of sufficient time to ensure safety (warning timing), (b) is an explanatory diagram of the truck vehicle area BB, and (c) is an explanatory diagram of the passenger car vehicle area. [Figure 7] (a) is an explanatory diagram of the road, shoulder, boundary line, and vehicle area BB, (b) is an explanatory diagram of the boundary line that divides the vehicle area, and (c) is an explanatory diagram of two areas formed by dividing the vehicle area. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, the monitoring / warning device of this embodiment includes a camera 30, an image processing device 5 that receives and processes images from the camera 30, an operation input device 20 that may be configured as a keyboard or a game pad, a storage device 60, a wireless transmitter 58, and a control device (computer) 50 that controls each of these components. Each of these devices may be an independent device or an integrated device. The image processing device 5 and the control device 50 are generally implemented as an image processing board or a control board. The storage device 60 may be configured as a memory, a hard disk, an SSD, or a combination thereof.

[0014] The camera 30 is a device that is temporarily installed on the road shoulder 91 (see FIG. 7; in a location where there is no road shoulder, the leftmost lane, etc.) and is capable of capturing images of the area to be captured on the rear side of the road from a fixed position, and outputs, for example, 30 fps video images to the image processing device 5. An example of the fixed position is a position that provides a panoramic view of a predetermined range on the rear side of the road (a range in which vehicle detection and tracking can be started with sufficient timing to ensure safety; including at least the range of the monitoring area in FIG. 6(a)), as shown in each diagram in FIG. 5. The camera 30 is temporarily fixed in this position to capture the desired video image. The camera 30 may be configured to be operable while still mounted on the inspection vehicle. Because it is temporarily fixed, it can be easily installed and removed.

[0015] The image processing device 5 has an image processing unit 51 that detects, identifies, and tracks vehicles (trucks, passenger cars, light trucks, motorcycles, etc.) from each image frame of the moving image input from the camera 30. Although not shown, a display device may be provided downstream of the image processing device 5 that can display the moving image input from the camera 30 as an image in which a bounding box BB is superimposed on the detected vehicle in real time / still state / repeatedly in response to an instruction from the control device 50.

[0016] The image processing unit 51 can be a known AI application capable of performing image processing (detection, identification, and tracking of each vehicle) with the performance and accuracy required depending on the characteristics of the imaged object (such as the size and speed of the vehicle to be detected). In this embodiment, a configuration based on YOLO (You Only Look Once) is adopted, but other known AI applications may be adopted as appropriate. For example, instead of YOLO (You Only Look Once), it may be considered to use SSD (Single Shot Multibox Detector) or Faster-R-CNN as long as the required performance is met. Alternatively, it may be considered to adopt Deep-SORT-YOLO.v3 or the like.

[0017] The control device 50 controls each device in response to a user's operation input from the operation input device (game pad, keyboard, etc.) 20. For example, when an instruction to "start capturing" is given from the operation input device 20, the control device 50 instructs the camera 30 to start capturing, causing it to start capturing. The camera 30 outputs moving images being captured, i.e., image frames in chronological order, to the image processing unit 51 of the image processing device 5. In this embodiment, the moving images output by the camera 30 are 30 fps.

[0018] The image processing unit 51 detects and identifies (classifies, divides into classes) vehicles (passenger cars, trucks, motorcycles, light trucks, etc.) from each image frame of the video input from the camera 30, tracks them, and stores data of each vehicle (position and size of bounding box BB, vehicle type (class), ID, image frame number, etc.) in an appropriate memory area (not shown) and outputs the data to the control device 50.

[0019] When the control device 50 receives data of the current image frame from the image processing unit 51, it executes the processing shown in Fig. 2. That is, it calculates the area ratio of each of two areas obtained by dividing the vehicle area (area of ​​bounding box BB) of each vehicle in the current image frame by the boundary line 9, compares the calculated area ratio of each vehicle with the warning threshold of the class to which the vehicle belongs (passenger car / truck / motorcycle / light truck, etc.), and executes processing to output a warning to the user if the area ratio exceeds the warning threshold.

[0020] The explanation will be given with reference to FIG. The control device 50 acquires various data detected in the current image frame from the image processing device 5 (S11). For example, the control device 50 acquires the information on each vehicle detected in the current image frame, its class, and the size and position of the vehicle area (bounding box) BB of that vehicle, as well as each vehicle detected in the previous image frame and being tracked in the current image frame, its class, and the size and position of the vehicle area (bounding box) BB of that vehicle. It is assumed that data on the boundary line 9 indicating the boundary between the travel lane 92 and the road shoulder 91 has already been acquired by initial setting when the monitoring and warning device is installed, for example.

[0021] Next, using the acquired data, the area ratio of each of the two regions formed by dividing the vehicle region of each vehicle by the boundary line 9 is calculated (S21). In this embodiment, the area ratio is calculated as B1 / (B1+B2). Here, B1 is the area of ​​the area on the shoulder 91 side when the vehicle area (bounding box) BB is divided by the boundary line 9, and B2 is the area of ​​the area on the driving lane 92 side (FIG. 7). That is, in this embodiment, the area ratio is the ratio of the area of ​​the area on the road shoulder 91 side when the vehicle area (bounding box) BB is divided by the boundary line 9 to the area of ​​the vehicle area.

[0022] This area ratio is compared with the warning threshold for the class to which the vehicle belongs (passenger car / truck / motorcycle / light truck, etc.) (S23). This warning threshold is a value found by trial and error from the perspective of when the above area ratio reaches a certain level, in other words, at what distance and to what extent the vehicle being detected and tracked must enter the shoulder of the road before a warning is issued to the user, ensuring sufficient time for the user to take refuge. Naturally, as for the warning threshold value, values ​​calculated according to the principle corresponding to the above definition "B1 / (B1+B2)" are prepared for each class. In this embodiment, the distance from the monitoring / warning device to the vehicle is obtained using input data from the distance sensor 70.

[0023] If the comparison result in step S23 indicates that the area ratio of any vehicle exceeds the threshold value of the class to which the vehicle belongs (YES in S31), a warning is output (S41). In this embodiment, "data indicating that a warning should be sent" is output to the wireless transmitter 58. As a result, the wireless transmitter 58 wirelessly transmits the data to each wireless receiver 80 carried by each inspection worker. Each wireless receiver 80 outputs a signal that will reliably attract the attention of the inspection worker. For example, a loud warning sound is output.

[0024] 1, the GNSS receiver 10 is a device that outputs the current time and current position based on information received from GNSS satellites. This allows the monitoring / warning device of this embodiment to obtain accurate position information and time information for performing the above-mentioned monitoring / warning.

[0025] In the above embodiment, "B1 / (B1+B2)" is used as the area ratio. However, instead of this, (1) "B1 / B2" (2) "B2 / (B1+B2" (3) "B2 / B1" In that case, the threshold for each class should naturally be a threshold that conforms to the definition. In the above cases (2) and (3), the numerical value of the area ratio decreases as the vehicle advances further toward the shoulder 91. Therefore, "exceeding the threshold" means "going beyond the threshold and the area ratio decreasing."

[0026] Although the above embodiment describes a case where the monitoring and warning device of the present invention is installed on the shoulder of a highway, the present invention is not limited to such an embodiment. For example, when the device is installed at the entrance or exit of a service area, similar processing is possible by setting an appropriate area as the danger area to be monitored instead of the shoulder. Alternatively, the present invention can be applied to a train station platform or the like where the approaching tracks are curved and therefore trains cannot be seen early, by setting a danger area to be monitored in the same way instead of the shoulder. [Explanation of symbols]

[0027] 10 GNSS receivers 20 Operation input device 30 Camera 40 Power supply 5. Image processing device 51 Image processing section 50 Control device 58 Radio transmitter 60 Storage device 61 Threshold memory unit 70 Distance Sensor 80 Radio receiver 9. Boundary 9a,9b,9c,9d,9e Trapezoid 90 White Line 91 Shoulder or leftmost lane 92 Vehicle Lane BB Vehicle Area B1 Area on the shoulder side of the vehicle area B2 Area on the driving lane side within the vehicle area

Claims

1. A monitoring and warning device that monitors the rear side of a road from a shoulder or the leftmost lane of a road and warns a user of a vehicle entering the shoulder or the leftmost lane, a camera that captures an image of the rear side of the road from a predetermined position in the road shoulder or the leftmost lane; an image processing means for detecting, identifying, and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera; a boundary setting means for setting a virtual boundary line within each of the image frames so as to overlap a white line on a road surface that separates a vehicle driving lane of the road from the shoulder or the leftmost lane; a threshold storage means for storing, for each vehicle class, a warning threshold value for the area ratio of two areas obtained by dividing a vehicle area occupied by a vehicle being tracked by the image processing means by the boundary line; an area ratio processing means for calculating the area ratio for each vehicle being tracked by the image processing means and comparing the area ratio with a warning threshold value for the class to which the vehicle belongs; a warning means for outputting a warning when the area ratio processing means determines that the area ratio of any vehicle being tracked by the image processing means exceeds a warning threshold value for the class of the vehicle; A monitoring and warning device comprising:

2. In claim 1, The vehicle area occupied by the vehicle being tracked by the image processing means is the area of ​​the bounding box of the vehicle. A monitoring and warning device characterized by:

3. In claim 1 or claim 2, the boundary line setting means sets the boundary line in response to an input from an operation input device. A monitoring and warning device characterized by:

4. In claim 1 or claim 2, When the white line on the road surface is curved, the boundary line setting means divides the predetermined monitoring area of ​​the road shoulder or the leftmost lane into a plurality of polygons, and sets the sides of the polygons that overlap the white line as approximately parts of the boundary lines. A monitoring and warning device characterized by:

5. In claim 4, The polygon is a trapezoid formed by using a plurality of parallel lines, which are set in order at intervals from the back side to the front side within the predetermined monitoring area, as upper and lower sides in turn. A monitoring and warning device characterized by:

6. A monitoring and warning device that monitors from within a predetermined monitoring area adjacent to a vehicle's driving area and warns a user of a vehicle entering the predetermined monitoring area, a camera that captures images from a predetermined position within the predetermined monitoring area; an image processing means for detecting, identifying, and tracking a vehicle and the class to which the vehicle belongs from each image frame of the video input from the camera; a boundary setting means for setting a virtual boundary line within each of the image frames so as to overlap a linear band that separates the predetermined monitoring area from a vehicle travel area adjacent to the monitoring area; a threshold storage means for storing, for each vehicle class, a warning threshold value for the area ratio of two areas obtained by dividing a vehicle area occupied by a vehicle being tracked by the image processing means by the boundary line; an area ratio processing means for calculating the area ratio for a vehicle being tracked by the image processing means and comparing the area ratio with a warning threshold for the class to which the vehicle belongs; a warning means for outputting a warning when the area ratio processing means determines that the area ratio of the vehicle being tracked by the image processing means exceeds a warning threshold value for the class of the vehicle; A monitoring and warning device comprising:

7. In claim 6, When the linear band is curved, the boundary line setting means divides the predetermined monitoring area into a plurality of polygons and sets sides of the polygons that overlap the linear band as approximately parts of the boundary lines. A monitoring and warning device characterized by:

8. In any one of claims 1 to 7, The image processing means is adjusted by deep learning using a large number of image data created based on photographed images of the front side of each vehicle type, such as trucks, buses, passenger cars, light trucks, and motorcycles, as training data for vehicle detection and identification. A monitoring and warning device characterized by:

9. A program for causing a computer to function as the monitoring / warning device according to any one of claims 1 to 8.

10. A monitoring and warning method for monitoring a rear side of a road from a shoulder or a leftmost lane of a road and warning a user of a vehicle entering the shoulder or the leftmost lane, comprising: Detecting, identifying and tracking a vehicle and the class to which the vehicle belongs from each image frame of a video input from a camera that captures an image of the rear side of the road from a predetermined position in the road shoulder or the leftmost lane; a virtual boundary line is set in each of the image frames so as to overlap a white line on the road surface that separates the vehicle driving lane of the road from the shoulder or the leftmost lane; For each vehicle being tracked, calculate the area ratio of two regions obtained by dividing the vehicle region occupied by the vehicle by the boundary line, and compare the area ratio with the warning threshold of the class to which the vehicle belongs; outputting a warning when the calculated area ratio of any of the vehicles being tracked exceeds a warning threshold for the class of the vehicle; A monitoring and warning method comprising:

Images