Falling object detection system and falling object detection method

Abstract

There was a possibility that the system might not be able to correctly detect fallen objects even if road conditions changed in real time. [Solution] A falling object detection system 1 for detecting falling objects on a road using multiple different falling object detection algorithms, comprising a computing device (CPU 31) that performs at least the following: a process of identifying the current road conditions from information on road conditions obtained by photographing the road; a process of selecting and switching a falling object detection algorithm based on the current road conditions; and a process of executing the selected falling object detection algorithm to detect falling objects on the road. The computing device refers to a switching table 60 that stores the scores of each falling object detection algorithm for each road condition label, calculates an evaluation value from the scores of the falling object detection algorithms corresponding to each label of the current road conditions, and selects and switches falling object detection algorithms based on the evaluation value.

Description

Technical Field

[0001] The present invention relates to a falling object detection system and a falling object detection method for detecting falling objects on a road.

Background Art

[0002] Conventionally, there is a road control system that monitors roads and vehicles using information such as images from CCTV (Closed-Circuit Television) cameras installed on roads. The road control system collects information such as the number of vehicles and vehicle speeds from the images of CCTV cameras, and also determines abnormal events such as falling objects,故障 vehicles, and逆行.

[0003] Patent Document 1 discloses a road monitoring system that monitors a road using an image of the road. This road monitoring system uses either both or one of two falling object detection algorithms using camera images as a first criterion, and detects a falling object when a second criterion based on a threshold of the number of times or frequency of satisfying the first criterion is satisfied. In this road monitoring system, a falling object detection algorithm based on object identification and a falling object detection algorithm based on lane change are used.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Incidentally, the road monitoring system described in Patent Document 1 has the following main characteristics. Firstly, since road conditions such as time of day, weather, and traffic volume change in real time, the accuracy of detecting fallen objects is unstable due to changes in road conditions. Secondly, each fallen object detection algorithm has road conditions under which it is good at detecting fallen objects and road conditions under which it is not good. Thirdly, it is necessary to decide in advance whether to use both or one of the fallen object detection algorithms. Due to these characteristics, there is a possibility that fallen objects cannot be correctly detected when road conditions change in real time.

[0006] Given the above situation, there was a need for a method to accurately detect fallen objects even when road conditions change in real time. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the present invention provides a falling object detection system that includes a computing device for executing a program, a storage device for storing the program, and a plurality of different falling object detection algorithms, and is a falling object detection system for detecting falling objects on a road. The computing unit performs at least the following processes: identifying the current road conditions from information about road conditions obtained by photographing the road; selecting and switching a falling object detection algorithm based on the current road conditions; and executing the selected falling object detection algorithm to detect falling objects on the road. Furthermore, in the process of selecting and switching falling object detection algorithms, the computing unit refers to switching information that stores the scores of each falling object detection algorithm for each road condition label, calculates an evaluation value from the scores of the falling object detection algorithms corresponding to each label of the current road condition, and selects a falling object detection algorithm based on the evaluation value. [Effects of the Invention]

[0008] According to at least one aspect of the present invention, road conditions can be identified in real time from information about road conditions obtained by photographing the road, and the system can automatically switch to an appropriate falling object detection algorithm according to those road conditions. This makes it possible to correctly detect falling objects even when road conditions change in real time. Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of a road control system according to one embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention. [Figure 3] This figure shows an example of a dataset in which ground truth information and labels for each road condition are attached to each video of a falling object according to one embodiment of the present invention. [Figure 4] This figure shows an example of the evaluation results of each falling object detection algorithm for a dataset related to one embodiment of the present invention. [Figure 5] This figure shows an example of a switching table created from the score calculation results related to one embodiment of the present invention. [Figure 6] This figure shows an example of a table illustrating the risk level of falling objects in relation to road conditions, according to one embodiment of the present invention. [Figure 7] This figure shows an example of the current road conditions according to one embodiment of the present invention. [Figure 8] This figure shows an example of the output result of the falling object detection algorithm execution unit according to one embodiment of the present invention. [Figure 9] This flowchart shows an example of the processing procedure of the switching table creation unit according to one embodiment of the present invention. [Figure 10] This flowchart shows an example of the overall processing procedure for a road control system according to one embodiment of the present invention. [Figure 11]It is a flowchart showing an example of the procedure of the processing of the falling object detection algorithm switching unit in step S12. [Figure 12] It is a flowchart showing an example of the procedure of the processing of the falling object detection algorithm execution unit in step S13.

Embodiments for Carrying out the Invention

[0010] Hereinafter, an example of an embodiment for carrying out the present invention (hereinafter referred to as "embodiment") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, the same reference numerals are given to common components or similar components, and duplicate explanations are omitted. Also, when there are a plurality of identical or similar components, they may be described with different subscripts attached to the same reference numeral. The number of each component may be singular or plural unless otherwise specified.

[0011] In the following examples of the embodiments, various information will be described in a table format, but the various information may be in a data format other than the table format. Also, for example, various designations such as "XX information", "XX table", "XX list", "XX list", "dataset", etc. are interchangeable. Also, when explaining identification information, expressions such as "identification information", "name", "ID", etc. may be used, and these are interchangeable.

[0012] [Configuration of Road Control System] First, the configuration of the road control system according to an embodiment of the present invention will be described with reference to FIG. 1. FIG. 1 is a diagram showing a configuration example of the road control system according to an embodiment of the present invention. The road control system 1 shown in FIG. 1 is a system that monitors roads and vehicles from images captured by cameras such as CCTV cameras. The road control system 1 has a function of detecting falling objects and is an example of a falling object detection system.

[0013] In FIG. 1, the road control system 1 receives various information through communication networks such as LAN (Local Area Network) and WAN (Wide Area Network). For example, the road control system 1 acquires data of images captured by the camera 11 (hereinafter referred to as "camera images") through the LAN. The camera 11 captures an image of a road (hereinafter also referred to as "falling object image") that is the target of falling object detection, and transmits the image data to the road control system 1. The camera 11 is generally installed for each section of a road (route). Information for identifying the camera 11 is linked to information on the installation location of the camera 11 (for example, route name). The camera image is an example of information regarding the road condition.

[0014] In addition, the road control system 1 acquires various information from the weather forecast system 21, the road traffic information center 22, the connected car system 23, the road emergency dial 24, the external recognition sensor 25, etc. through the WAN. The various information acquired using these systems, etc. is also an example of information regarding the road condition.

[0015] The weather forecast system 21 is a system that predicts and notifies the meteorological conditions (weather, atmospheric pressure, wind direction, wind speed, temperature, humidity, etc.) from the current day to several days later for each region.

[0016] The road traffic information center 22 provides services to provide road traffic information, etc. to drivers through the communication function of vehicles. The road traffic information includes, in addition to road traffic information and traffic jam information, construction information, etc. Note that construction information may be provided to the road control system 1 by other systems or services.

[0017] The connected car system 23 is a system consisting of a connected car and an information center that receives data transmitted from the connected car. The connected car is an automobile that functions as an ICT (Information and Communication Technology) terminal. The information center receives vehicle data from the connected car system 23. The road control system 1 is an example of an information center. The vehicle data includes data related to driving conditions and data related to vehicle status. For example, data related to driving conditions includes vehicle speed, brake operation status (e.g., braking frequency), airbag deployment, and location information.

[0018] The Road Emergency Dial 24 is a hotline for reporting road damage such as potholes and shoulder collapses, as well as road abnormalities such as fallen objects and road surface contamination. It allows road users to directly report any abnormalities they discover on main roads to the road administrator (management organization), ensuring safety by enabling the road administrator to respond quickly to the problem. After a road user dials the designated number, they identify the road administrator based on their location and the selection of numbers provided by the automated voice guidance system, and are then transferred to the appropriate department. In recent years, reports using SNS (Social Networking Service) applications have also been accepted.

[0019] The external environment recognition sensor 25 is a sensor mounted on the vehicle. The external environment recognition sensor 25 acquires information based on the external environment conditions of the vehicle and converts it into an electrical signal. Examples of external environment recognition sensors 25 include monocular cameras, compound cameras, LiDAR (Light Detection and Ranging), millimeter-wave radar, and ultrasonic sensors. The output data of the external environment recognition sensor 25 may also include location information.

[0020] The road control system 1 comprises a switching table creation unit 2, a hazard level table registration unit 3, a switching table management unit 4, a hazard level table management unit 5, a road condition identification unit 6, a falling object detection algorithm switching unit 7, and a falling object detection algorithm execution unit 8.

[0021] (Switching table creation section) The switching table creation unit 2 acquires or creates a switching table (see Figure 5, described later) used for switching the falling object detection algorithm. The switching table is an example of switching information and may be in other data formats. For example, the switching table creation unit 2 acquires a switching table stored in any memory unit of the road control system 1. Alternatively, the switching table creation unit 2 acquires a switching table from an external server or the like via a network such as a WAN.

[0022] Furthermore, the switching table creation unit 2 receives road conditions and scores manually entered by the administrator and creates a switching table. Alternatively, the switching table creation unit 2 may be configured to automatically create a switching table based on the entered road conditions. The switching table creation unit 2 then outputs the switching table to the switching table management unit 4.

[0023] Road conditions include road information and environmental information. Road information mainly concerns road traffic, such as traffic volume, vehicle speed, and construction. Environmental information mainly concerns the road environment, such as time of day and weather.

[0024] The switching table creation unit 2 creates the switching table by following the steps of (1) creating the dataset, (2) calculating the average precision, and (3) calculating the score. (1) Creating a dataset The administrator manually creates a dataset for each sample video of a fallen object, labeling it with correct information and the corresponding road conditions.

[0025] Figure 3 shows an example of a dataset in which ground truth information and labels for each road condition are attached to each video of a falling object. In Figure 3, dataset 40 consists of N data points (Data 1 to Data N). Each data point includes a video of a falling object, the correct answer, time of day, weather, traffic volume, vehicle speed, and construction work.

[0026] The "Falling Object Video" section stores file information for the falling object video. This file information includes, for example, the file name, data format, and a link to the video data.

[0027] The "correct answer" file contains file information that summarizes the coordinates of the bounding box representing the falling object for each frame of the falling object video. For example, the coordinates of the bounding box representing the falling object are, for instance, the coordinates of the four corners of the rectangle that circumscribes the falling object object.

[0028] The "Time of Day" field stores information about the time period in which the video of the falling object was captured. For example, the labels (sub-items) for the time of day could be morning, noon, and night.

[0029] The "Weather" field stores weather information for the time period in which the video of the fallen object was taken. For example, weather labels may include sunny, rainy, and snowy. Weather information may be obtained from the weather forecasting system 21, or the road condition identification unit 6 may determine the weather from the video of the fallen object.

[0030] The "traffic volume" section stores information about the traffic volume on the roads visible in the footage of the fallen objects. For example, the traffic volume may be labeled as "high" or "low."

[0031] The "Vehicle Group Speed" column stores information about the speed of the group of vehicles traveling on the road as seen in the video of the fallen object. For example, the labels for vehicle group speed could be "high" or "low."

[0032] The "Construction" section stores information about construction work being carried out on roads visible in the footage of falling objects. For example, the label for construction may be either "Present" or "Absent".

[0033] Figure 3 shows an example where, for Data 1, the video of the fallen object is “video1.mp4”, the correct answer is “true1.json”, the time of day is “daytime”, the weather is “sunny”, the traffic volume is “low”, the vehicle group speed is “low”, and there is “no construction”. Note that the major categories and labels (sub-categories) of road conditions are not limited to the types shown in Figure 3, but may be of other types as well.

[0034] (2) Calculation of average accuracy The switching table creation unit 2 executes each falling object detection algorithm on each data point in the dataset and calculates the average accuracy. More specifically, for each falling object detection algorithm, the switching table creation unit 2 calculates the detection accuracy for each frame image that makes up each data point (falling object video) in the dataset, and then calculates the average value (average accuracy) obtained by averaging the detection accuracy for each frame image over the number of frames. In the following explanation, the average accuracy for each data point (falling object video) of a given falling object detection algorithm will simply be referred to as "accuracy". Note that the average accuracy may also be expressed as an F1 score, etc. The F1 score is the harmonic mean of precision (accuracy) and recall (sensitivity).

[0035] Figure 4 shows an example of the evaluation results for each falling object detection algorithm on the dataset. In Figure 4, the evaluation results 50 of the falling object detection algorithms show the detection accuracy for each algorithm for data 1 to data N. Here, the detection accuracy is represented by a value between 0 and 1, with a value closer to 1 indicating higher detection accuracy.

[0036] For example, the object recognition algorithm for detecting falling objects has a detection accuracy of "0.7" for data 1, "0.8" for data 2, "0.8" for data 3, and "0.3" for data N. Furthermore, the algorithm for detecting objects falling due to lane changes has a detection accuracy of "0.6" for data 1, "0.7" for data 2, "0.7" for data 3, and "0.7" for data N.

[0037] Figure 4 shows five types of falling object detection algorithms (detection methods). The following describes these five types of falling object detection algorithms. However, the types of falling object detection algorithms are not limited to these examples.

[0038] • Detection by object recognition The object recognition detection method uses an object detection / identification model to detect objects other than vehicles as fallen objects.

[0039] • Detection due to lane changes The detection method based on lane changes determines that there is a fallen object if a predetermined number of vehicles or more are making a temporary lane change within a common area.

[0040] • Detection using background subtraction The background subtraction detection method calculates the difference between the pixel values ​​of a normal image (background image) that does not show the fallen object or vehicle and the current camera image. Areas with a large difference are considered candidates for fallen objects, and if that area is maintained for a certain period of time, it is detected as a fallen object. Because this detection method takes time, its accuracy is low immediately after switching camera images (zooming in, changing the shooting angle). Therefore, when switching camera images, background subtraction detection may be withheld or its priority reduced for a certain period immediately after the change in shooting conditions. For example, a configuration could be considered in which the score for each label of the background subtraction fallen object detection algorithm in the switching table (Figure 5) is reduced for a certain period of time, or the algorithm is excluded from selection. Note that shooting conditions such as camera zoom and shooting angle are related to the judgment of road information, so they may be included in the road information.

[0041] Detection by likelihood The likelihood-based detection method uses a generative model (such as a Variational Autoencoder: VAE) trained on normal images to detect falling objects based on the low probability of occurrence in the current camera image. The probability of occurrence is the probability of the current image occurring relative to normal images.

[0042] • Detection due to recovery error The detection method using reconstruction error detects falling objects based on the magnitude of the reconstruction error, using a generative model (such as a VAE) trained on normal images including vehicles. The error between the current camera image and the reconstructed normal image is calculated by reconstructing a normal image with a vehicle from the current camera image (input image). This detection method can detect falling objects immediately without waiting a certain amount of time, even if the camera image is switched. Therefore, when the camera image is switched, the priority of the reconstruction error-based falling object detection algorithm may be increased over the background subtraction-based falling object detection algorithm for a certain period of time. Alternatively, the score for each label in the reconstruction error-based falling object detection algorithm in the switching table (Figure 5) may be increased over the score for each label in the background subtraction-based falling object detection algorithm for a certain period of time. However, this detection method requires a diverse and large number of normal images to achieve high detection accuracy, and its processing time is longer compared to other detection methods.

[0043] (3) Calculation of the score The switching table creation unit 2 calculates the average accuracy of data (fallen object videos) with common road condition labels for each fallen object detection algorithm, and uses this as the score for each fallen object detection algorithm for each label. For example, the average accuracy of the daytime label for a given time period is the average detection accuracy of data labeled "daytime" among data 1 to data N in the dataset for a given fallen object detection algorithm.

[0044] While each falling object detection algorithm processes essentially the same dataset, there are various methods for calculating accuracy (finding the correct answer). For example, the method differs depending on whether the camera footage includes a frame around the falling object or not (without specifying its location). In the case of no frame around the falling object, it is sufficient to detect objects on the road, allowing for, for example, knowing when objects were on the road. After checking the relevant time period, the administrator can then search for the fallen object in the camera footage. In this way, by performing the processes in (2) and (3) using the dataset created in (1), a switching table can be automatically created.

[0045] Figure 5 shows an example of a switching table created from the score calculation results. In Figure 5, the switching table 60 shows the scores for each road condition label for each falling object detection algorithm. For example, the falling object detection algorithm using object recognition has scores of "0.7", "0.8", and "0.4" for the time of day ("morning", "noon", and "night"), and "0.9", "0.3", and "0.2" for the weather conditions ("sunny", "rainy", and "snowy"). It also has scores of "0.7" and "0.8" for the traffic volume ("high" and "low"), "0.7" and "0.5" for the vehicle group speed ("high" and "low"), and "0.7" and "0.8" for the presence of construction ("present" and "absent").

[0046] Furthermore, the algorithm for detecting objects falling due to lane changes has scores of "0.6", "0.7", and "0.6" for the time of day ("morning", "noon", and "night"), and "0.7", "0.7", and "0.7" for the weather conditions ("sunny", "rainy", and "snowy"). Additionally, the scores for "high" and "low" traffic volume are "0.9" and "0.1" respectively, the scores for "high" and "low" vehicle group speed are "0.5" and "0.8" respectively, and the scores for "present" and "absent" construction are "0.1" and "0.7" respectively.

[0047] (Risk Level Table Registration Section) Returning to the explanation of Figure 1, the Hazard Table Registration Unit 3 registers a Hazard Table that represents the degree of danger to road conditions from falling objects. The Hazard Table Registration Unit 3 receives input from the administrator, sets the Hazard Level for each road condition label, and creates the Hazard Table. Alternatively, the Hazard Table Registration Unit 3 may be configured to set the Hazard Level for each road condition label and create the Hazard Table based on pre-set standard values ​​for each road condition. The Hazard Table Registration Unit 3 then outputs the Hazard Table to the Hazard Table Management Unit 5. Here, the Hazard Level is represented on a scale from 1 to 5, with higher values ​​indicating a higher degree of danger. In this way, the Road Control System 1 has a Hazard Table in which the Hazard Level to falling objects is set for each road condition label.

[0048] Figure 6 shows an example of a table illustrating the risk of falling objects based on road conditions. In Figure 6, the falling object risk level table 70 has risk levels assigned to each road condition label. Here, the risk level is represented by an integer from 1 to 5, with higher numbers indicating a higher risk. Basically, it is assumed that the administrator will manually assign a risk level to each label and register the risk level table in the road control system 1. However, the risk level table registration unit 3 may be configured to automatically set the risk level based on past accident cases for each label.

[0049] The following rules can be considered for determining the degree of risk posed by falling objects in relation to road conditions. • Falling objects are difficult to see at night → High possibility of collision → High risk • Braking distance increases in the rain → Higher likelihood of collision → Higher risk • Heavy traffic makes it difficult to avoid fallen objects → Higher likelihood of collision → Higher risk • High vehicle speed → High probability of collision (high impact of collision) → High level of danger • Construction work increases the likelihood of falling materials and tools → Higher risk of collision → Higher danger level

[0050] In Figure 6, the risk level table 70 shows that the risk levels for the time of day "morning," "noon," and "night" are "2," "1," and "4," respectively, and the risk levels for the weather conditions "sunny," "rainy," and "snowy" are "1," "3," and "5," respectively. In addition, the risk levels for "high" and "low" traffic volume are "4" and "1," respectively, the risk levels for "high" and "low" vehicle group speed are "5" and "1," respectively, and the risk levels for "present" and "absent" construction are "3" and "1," respectively.

[0051] (Switching Table Management Department) Returning to the explanation of Figure 1, the switching table management unit 4 stores and manages the switching tables input from the switching table creation unit 2. The switching table management unit 4 also outputs the switching tables to the falling object detection algorithm switching unit 7.

[0052] (Risk Level Table Management Department) The risk level table management unit 5 stores and manages the switching table input from the risk level table registration unit 3. The risk level table management unit 5 also outputs the risk level table to the falling object detection algorithm switching unit 7. Furthermore, the risk level table management unit 5 outputs the risk level table to the falling object detection algorithm execution unit 8. The risk level table is used in the falling object detection algorithm execution unit 8 when adjusting the threshold for falling object detection (hereinafter referred to as the "detection threshold") of the falling object detection algorithm.

[0053] Note that the aforementioned hazard level table registration unit 3 and hazard level table management unit 5 are not mandatory. However, by providing the hazard level table registration unit 3 and hazard level table management unit 5 and reflecting the information in the hazard level table in the falling object detection, it is possible to perform falling object detection while taking into account the hazard level according to the road conditions.

[0054] (Road Condition Identification Department) The road condition identification unit 6 acquires video footage (hereinafter referred to as "camera footage") that is subject to falling object detection, captured by cameras 11 installed along each road (route), via the LAN, and identifies the road conditions from the camera footage. The road condition identification unit 6 then outputs information about the identified road conditions to the falling object detection algorithm switching unit 7. The road condition identification unit 6 can analyze the camera footage using, for example, a trained model, to identify road information (traffic volume, vehicle speed, construction, etc.) and environmental information (weather, etc.) as road conditions.

[0055] Figure 1 shows an example in which camera images 10a of route A and camera images 10b of route B are taken up by the road condition identification unit 6. A fallen object 12 is visible in the camera image 10a of route A. In this embodiment, it is assumed that a CCTV camera is used as camera 11. However, camera 11 can be any device that can photograph an object. For example, camera 11 may be a LiDAR that uses laser light to measure the distance to an object and the shape of the object, or a depth camera that can measure the distance to an object.

[0056] Furthermore, the road condition identification unit 6 may identify road conditions based on information other than camera footage, in addition to camera footage. For example, the road condition identification unit 6 may acquire information on weather, traffic volume, vehicle group speed, and construction, etc., from the weather forecasting system 21, the road traffic information center 22 (weather information, construction information), and the connected car system 23 via the WAN.

[0057] Figure 7 shows an example of the current road conditions identified by the road condition identification unit 6. In Figure 7, the current road conditions 80 are flagged (logical value "1") for each of the following: time of day "daytime", weather "sunny", traffic volume "low", vehicle group speed "low", and construction "none". The time of day may be determined from the time information included in the camera image data of camera 11 (see Figure 1), or from the time information of the terminal operating the road control system 1.

[0058] (Falling object detection algorithm switching unit) The falling object detection algorithm switching unit 7 calculates an evaluation value from the score corresponding to the current road conditions, based on a score representing the strengths and weaknesses of each falling object detection algorithm for the given road conditions, and switches the falling object detection algorithm according to the evaluation value. The method for calculating the evaluation value and the criteria for deciding whether to switch will be described in detail later.

[0059] (Falling object detection algorithm execution unit) The falling object detection algorithm execution unit 8 executes the falling object detection algorithm selected by the falling object detection algorithm switching unit 7 and detects falling objects in the camera image captured by the camera 11. The falling object detection algorithm execution unit 8 then outputs the execution result, i.e., the falling object detection result, to a terminal used by the air traffic controller. The falling object detection algorithm execution unit 8 may output the execution result only when a falling object is detected in the camera image.

[0060] The falling object detection algorithm execution unit 8 may also detect falling objects in the camera image based on the camera image and other information. For example, the falling object detection algorithm execution unit 8 may acquire the above-mentioned other information from the external environment recognition sensor 25, the connected car system 23, and the road emergency dial 24, etc.

[0061] The arrow pointing from the falling object detection algorithm execution unit 8 to the road condition identification unit 6 indicates that the processes of the road condition identification unit 6, the falling object detection algorithm switching unit 7, and the falling object detection algorithm execution unit 8 are repeatedly executed. The falling object detection algorithm executes its process 30 times per second if the road camera image is, for example, 30 fps. The road condition identification unit 6 and the falling object detection algorithm switching unit 7 periodically (for example, every minute) at a time longer than the frame cycle, to identify the road conditions and switch the falling object detection algorithm. Alternatively, the road condition identification unit 6 and the falling object detection algorithm switching unit 7 may be executed only when the road conditions change.

[0062] Figure 8 shows an example of the output result of the falling object detection algorithm execution unit 8. The output result 90 of the falling object detection algorithm execution unit 8 shown in Figure 8 is what is displayed on the display device 34 of each terminal. The output result 90 includes, as an example, the items of falling object detection result, time, route, falling object image, and accuracy.

[0063] The "Detection Results" column stores information about the detection results regarding the presence or absence of falling objects. Note that in configurations where the results are output only when a falling object is detected, "No Falling Objects" will never be displayed in the "Detection Results" column. However, in configurations where the detection results are displayed by specifying a time or range, "No Falling Objects" will be displayed if no falling objects are detected.

[0064] The "Time" field stores information about the time when the falling object was detected in the camera footage. The "route" field stores information that identifies the road (e.g., route) where the fallen object was detected.

[0065] The "falling object image" section stores file information for images (referred to as "falling object images") that show the falling object extracted from the camera footage. File information includes, for example, the file name, data format, and a link to the image data. The falling object image 91 shown in Figure 8 shows multiple vehicles traveling on route A and the falling object 12. The falling object image 91 can be, for example, a frame image or an enlarged image.

[0066] "Accuracy" is the degree of certainty of the execution result output by each falling object detection algorithm when a falling object is detected. Accuracy is expressed as 0 to 1 or as a probability (%). For example, in a falling object detection algorithm using object recognition, it represents the degree to which an object in the frame detected from the camera image belongs to the falling object class.

[0067] In the output result 90 of Figure 8, an example is shown where the detection result is "Falling object present", the time is "2024 / 3 / 24 10:01:32", the route is "Route A", the falling object image is "image.png", and the accuracy is "0.8".

[0068] [Computer hardware configuration] Here, the hardware configuration of the computer in the road control system 1 will be explained with reference to Figure 2. Figure 2 is a block diagram showing an example of a computer hardware configuration. The computer 30 shown in Figure 2 is an example of hardware used as a computer. In this embodiment, the road control system 1 realizes road control (falling object detection) through the coordinated operation of each functional block shown in Figure 1 by the computer 30 executing a program.

[0069] Computer 30 includes a CPU (Central Processing Unit) 31, ROM (Read Only Memory) 32, and RAM (Random Access Memory) 33 connected to a system bus. Computer 30 also includes a display device 34, an input device 35, non-volatile storage 36, and a network interface 37.

[0070] The CPU 31 reads the program code for the software that implements each function according to this embodiment from the ROM 32, loads it into the RAM 33, and executes it. Variables and parameters that occur during the calculation process of the CPU 31 are temporarily written to the RAM 33, and these variables and parameters are read out by the CPU 31 as appropriate. The functions of each functional block of the road control system 1 are realized when the CPU 31 executes the program code read from the ROM 32. However, other processors such as an MPU (Micro Processing Unit) may be used instead of the CPU 31.

[0071] The display device 34 is a monitor such as a liquid crystal display, and displays a GUI (Graphical User Interface) screen or the results of calculations performed by the CPU 31. The input device 35 generates input signals in response to user operations and outputs them to the CPU 31. For example, the input device 35 can be a mouse, keyboard, or touch sensor, and the user can input information or instructions by operating the input device 35. The display device 34 and the input device 35 may be configured as an integrated touch panel.

[0072] Non-volatile storage 36 is an example of a recording medium that can store data used by a program or data obtained by executing a program. The tables and data described later are constructed using non-volatile storage 36. In addition, the OS (Operating System) and programs executed by the CPU 31 may be recorded on the non-volatile storage 36. Examples of non-volatile storage 36 include HDDs (Hard Disk Drives), SSDs (Solid State Drives), optical discs that utilize light (and magnetism), or semiconductor memory cards.

[0073] A communication device such as a NIC (Network Interface Card) can be used for the network interface 37. The network interface 37 can send and receive various types of data to and from external devices via a communication network such as a LAN or a dedicated line.

[0074] Terminals used by administrators and air traffic controllers (not shown) can also be implemented using the same configuration as computer 30.

[0075] [Processing in the switching table creation section] Next, the processing of the switching table creation unit 2 according to one embodiment of the present invention will be described with reference to Figure 9. Figure 9 is a flowchart showing an example of the processing procedure of the switching table creation unit 2.

[0076] First, the administrator operates a terminal (not shown) to input a dataset (for example, data 1 to data N) as shown in Figure 3 into the switching table creation unit 2 of the road control system 1 (step S1).

[0077] Next, the switching table creation unit 2 calculates the detection accuracy of each falling object detection algorithm using each falling object video for each falling object video in the dataset (step S2). In step S2, the switching table creation unit 2 calculates the detection accuracy for each falling object detection algorithm using the falling object video of the target data. The switching table creation unit 2 then performs this process for each falling object video of the data included in the dataset. Once the detection accuracy for all falling object videos in the dataset has been calculated for each falling object detection algorithm, the process proceeds to step S3.

[0078] Next, the switching table creation unit 2 creates evaluation results for each falling object detection algorithm for each data point in the dataset (see Figure 4) (step S3).

[0079] Next, the switching table creation unit 2 calculates the average of the evaluation results for each label for each falling object detection algorithm (step S4). In step S4, the switching table creation unit 2 calculates the average value (score) of the evaluation results for each common label of multiple data for the target falling object detection algorithm. The switching table creation unit 2 then performs this process for each common label for each falling object detection algorithm. Once the average value of the evaluation results for each label has been calculated for each falling object detection algorithm, the process proceeds to step S5.

[0080] Next, the switching table creation unit 2 creates a switching table (see Figure 5) from the calculation results of step S4, showing the correspondence between the falling object detection algorithm and the score for the road condition label (step S5).

[0081] Next, the switching table creation unit 2 outputs the created switching table to the switching table management unit 4 for registration (step S6). After step S6 is completed, the switching table creation process is terminated.

[0082] [Overall processing of the road traffic control system] Next, the overall processing of the road control system 1 according to one embodiment of the present invention will be described with reference to Figure 10. Figure 10 is a flowchart showing an example of the overall processing procedure for Road Traffic Control System 1.

[0083] First, the road condition identification unit 6 of the road control system 1 acquires camera images output by camera 11 and identifies the current road conditions of the route shown in the camera images (step S11). Details of this process will be explained using Figure 11, which will be described later.

[0084] Next, the falling object detection algorithm switching unit 7 refers to the switching table based on the current road conditions identified by the road condition identification unit 6 and switches the falling object detection algorithm (step S12). Details of this process will be explained later using Figure 12.

[0085] Next, the falling object detection algorithm execution unit 8 executes the falling object detection algorithm switched by the falling object detection algorithm switching unit 7 and performs falling object detection (step S13). After step S13 is completed, the process returns to step S11. The road control system 1 repeatedly executes the processes from steps S11 to S13.

[0086] As described above, if the camera image is, for example, 30fps, the falling object detection algorithm execution unit 8 will execute step S13 30 times per second. Also, as described above, the road condition identification unit 6 and the falling object detection algorithm switching unit 7 periodically (for example, every minute) at a time longer than the frame cycle, they identify the road conditions and switch the falling object detection algorithm. Therefore, it can be considered that every 1800 times step S13 is executed, the processing of step S11 by the road condition identification unit 6 and the processing of step S12 by the falling object detection algorithm switching unit 7 are each executed once. In this case, the processing of steps S11 and S12 are not executed at any other time.

[0087] [Processing of the falling object detection algorithm switching unit] Next, the processing of the falling object detection algorithm switching unit 7 in step S12 will be explained with reference to Figure 11. Figure 11 is a flowchart showing an example of the processing procedure of the falling object detection algorithm switching unit 7.

[0088] First, the falling object detection algorithm switching unit 7 extracts the score of each falling object detection algorithm for the current road conditions from the switching table (see Figure 5) (step S21).

[0089] Next, the falling object detection algorithm switching unit 7 extracts the degree of danger for the current road conditions from the danger level table (see Figure 6) (step S22).

[0090] Next, the falling object detection algorithm switching unit 7 calculates an evaluation value for each falling object detection algorithm based on the score of each falling object detection algorithm extracted in step S21 and the risk level extracted in step S21 (step S23). Furthermore, if the Risk Level Table Registration Unit 3 and Risk Level Table Management Unit 5 are not present, the processes in steps S22 and S23 can be deleted.

[0091] Next, the falling object detection algorithm switching unit 7 selects the falling object detection algorithm with the highest evaluation value. If there are multiple algorithms with the same evaluation value, it selects multiple falling object detection algorithms (step S24). After the processing in step S24, the falling object detection algorithm switching process is terminated.

[0092] In this way, by reflecting the degree of risk associated with the current road conditions in the evaluation value, the system can switch to a suitable object detection algorithm when the road conditions pose a high risk of falling objects. Therefore, it is possible to select an object detection algorithm that is robust to road conditions with a high probability of collision with falling objects.

[0093] [Calculation of risk-based evaluation values] Here, we will explain how to calculate the evaluation value for each falling object detection algorithm in step S23. In step S23, the evaluation value is calculated based on the degree of risk of falling objects in the road conditions. By reflecting the current risk of road conditions in the evaluation value, the falling object detection algorithm can be switched according to the risk of the current road conditions. There are two main methods for calculating the evaluation value based on the risk of road conditions. The first method is to calculate the weighted sum (dot product) of the risk of each road condition label and the fraction of the score, and use the total value as the evaluation value. The second method is to calculate the weighted sum of the risk of each road condition label and the score, and use the total value as the evaluation value.

[0094] (Weighted sum of risk level and score fraction) Taking the switching table 60 shown in Figure 5 and the risk level table 70 shown in Figure 6 as examples, the first method for detecting falling objects by object identification is expressed by the following formula. Evaluation value = minΣ(score for current road conditions) * (1 / degree of danger for current road conditions) =min(0.8 / 1 (daytime) + 0.3 / 3 (rain) + 0.7 / 4 (heavy traffic) + 0.5 / 1 (low speed) + 0.8 / 1 (no construction))

[0095] In the first method, the sum of the loads (minimum value) of the risk level of each road condition label and the score fraction of the falling object detection algorithm is calculated, and the falling object detection algorithm with the largest sum (minimum value) is selected. In the first method, by maximizing the evaluation value among the minimum values, it is expected that excessive oversights and false detections of falling objects can be prevented.

[0096] (Weighted sum of risk level and score) Similarly, taking the switching table 60 shown in Figure 5 and the risk level table 70 shown in Figure 6 as examples, the second method for detecting falling objects by object identification is expressed by the following formula. Evaluation value = Σ (score relative to current road conditions) * (danger level relative to current road conditions) =0.8*1(daytime)+0.3*3(rain)+0.7*4(heavy traffic)+0.5*1(low speed)+0.8*1(no construction)

[0097] In the second method, the total weight of the risk level and score for each road condition label of the falling object detection algorithm is calculated, and the falling object detection algorithm with the highest total value is selected. The second method is expected to have the greatest falling object detection effect for a given road condition. On the other hand, there is a possibility that the number of missed detections and false detections will increase.

[0098] [Calculation of evaluation values ​​without using risk levels] Here, we will explain how to calculate the evaluation values ​​for each falling object detection algorithm when the danger level of the falling object is not used. The falling object detection algorithm switching unit 7 refers to the switching table 60 and the current road conditions 80, calculates an evaluation value from the score corresponding to the current road conditions, and selects a falling object detection algorithm that satisfies the pre-set conditions.

[0099] For example, the falling object detection algorithm switching unit 7 may calculate the sum of the scores for each label of the road environment as an evaluation value and select the falling object detection algorithm with the highest evaluation value. As an example, in the case of a falling object detection algorithm using object recognition, the evaluation value is calculated as 0.8 (daytime) + 0.3 (rain) + 0.7 (heavy traffic) + 0.5 (low speed) + 0.8 (no construction).

[0100] Furthermore, the falling object detection algorithm switching unit 7 may calculate the product of the scores of each label of the road environment as an evaluation value and select the falling object detection algorithm with the highest evaluation value. For example, in the case of a falling object detection algorithm using object recognition, the evaluation value is calculated as 0.8 (daytime) × 0.3 (rainy) × 0.7 (heavy traffic) × 0.5 (low speed) × 0.8 (no construction).

[0101] Furthermore, the falling object detection algorithm switching unit 7 may select the falling object detection algorithm with the highest minimum score among the scores of each label in the road environment. When the minimum score is high, a certain level of falling object detection accuracy can be expected regardless of road conditions. In the switching table 60, the minimum value for detection by object identification is "0.2", the minimum value for detection by lane change is "0.1", the minimum value for detection by background subtraction is "0.3", the minimum value for detection by likelihood is "0.3", and the minimum value for detection by reconstruction error is also "0.3". Therefore, when this calculation method is applied to the switching table 60 in Figure 5, the falling object detection algorithm by lane change is selected.

[0102] [Processing by the falling object detection algorithm execution unit] Next, the processing of the falling object detection algorithm execution unit 8 in step S13 will be explained with reference to Figure 12. Figure 12 is a flowchart showing an example of the processing procedure of the falling object detection algorithm execution unit 8.

[0103] First, the falling object detection algorithm execution unit 8 calculates the total risk level for the current road conditions (step S31).

[0104] Next, the falling object detection algorithm execution unit 8 raises the detection threshold of each falling object detection algorithm as the total risk level increases, and lowers the detection threshold of each falling object detection algorithm as the total risk level decreases (step S32). As a result, the detection threshold is adjusted according to the risk level of the road conditions for falling objects, so that falling objects can be detected appropriately according to the risk level, and missed detections and false detections can be reduced.

[0105] For example, in an object recognition-based falling object detection algorithm, the probability of which class (vehicle, person, or falling object) an object in the detected frame belongs to is calculated, and if the probability of it being a falling object is high, the frame is detected as a falling object. At this time, the threshold for determining high / low probability is adjusted according to the degree of danger to the current road conditions. In contrast, in a background subtraction-based falling object detection algorithm, an area is detected as a falling object if that area is maintained for a certain period of time. The length of this period is adjusted according to the degree of danger to the current road conditions. For example, if the detection threshold is increased, the length of the period is shortened from 30 seconds to 15 seconds.

[0106] Next, the falling object detection algorithm execution unit 8 executes each falling object detection algorithm (step S33).

[0107] Next, the falling object detection algorithm execution unit 8 determines whether it is executing two or more falling object detection algorithms (step S34).

[0108] If the falling object detection algorithm execution unit 8 is executing two or more falling object detection algorithms (YES determination in step S34), it determines whether the sum of the risk levels is below a threshold (step S35).

[0109] If the total risk level is below the threshold (YES determination in step S35), the falling object detection algorithm execution unit 8 takes the logical AND of the results of each falling object detection algorithm as the detection result (step S36). When the perceived risk to the current road conditions is relatively low, the logical AND of the results from multiple falling object detection algorithms can reduce false positives of falling objects.

[0110] Furthermore, if the total risk level is greater than the threshold (NO determination in step S35), the falling object detection algorithm execution unit 8 takes the logical OR of the results of each falling object detection algorithm as the detection result (step S37). When the current road conditions pose a relatively high risk, employing the logical OR of the results from multiple falling object detection algorithms can reduce the likelihood of missing fallen objects.

[0111] On the other hand, if only one falling object detection algorithm is executed in the determination process of step S34 (NO determination in step S34), the falling object detection algorithm execution unit 8 takes the result of the one falling object detection algorithm as the detection result (step S38).

[0112] After any of steps S36, S37, or S38 is completed, the execution of this falling object detection algorithm is terminated.

[0113] As described above, the falling object detection system (road control system 1) according to this embodiment is a falling object detection system that detects falling objects on a road, comprising a computing device (CPU 31) for executing a program, a storage device (ROM 32 or non-volatile storage 36) for storing the program, and a plurality of different falling object detection algorithms. The computing unit performs at least the following processes: identifying the current road conditions (e.g., current road conditions 80) from information about road conditions obtained by photographing the road (e.g., camera footage) (corresponding to the road condition identification unit 6); selecting and switching a falling object detection algorithm based on the current road conditions (corresponding to the falling object detection algorithm switching unit 7); and executing the selected falling object detection algorithm to detect fallen objects on the road (corresponding to the falling object detection algorithm execution unit 8). Furthermore, in the process of selecting and switching falling object detection algorithms, the computing unit refers to a switching table (for example, switching table 60) that stores the scores of each falling object detection algorithm for each road condition label, calculates an evaluation value from the scores of the falling object detection algorithms corresponding to each label of the current road condition, and selects a falling object detection algorithm based on the evaluation value.

[0114] According to this embodiment, it is possible to identify road conditions in real time from information about road conditions obtained by photographing the road, and to automatically switch to an appropriate fallen object detection algorithm according to those road conditions. As a result, fallen objects can be correctly detected even if road conditions change in real time.

[0115] This embodiment can solve the problems that arise from simply combining multiple falling object detection algorithms. This embodiment identifies changes in road conditions in real time, dynamically calculates an evaluation value for the falling object detection algorithm in relation to the current road conditions, and automatically switches to the appropriate falling object detection algorithm.

[0116] In the embodiments described above, an example of detecting objects that have fallen onto the road was explained, but the present invention can also be used to detect vehicles driving in the wrong direction. Furthermore, the present invention can also be used to detect stopped vehicles on the road. For example, the object detection algorithm can be applied not only to detecting fallen objects but also to detecting stopped vehicles such as vehicles driving in the wrong direction or broken-down vehicles, thereby detecting other vehicles that pose an obstacle to vehicles traveling normally on the road.

[0117] The present invention is not limited to the embodiments described above, and various other modifications and applications are possible as long as they do not depart from the gist of the invention as described in the claims. For example, the embodiments described above are described in detail and specifically in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the components described. Furthermore, it is possible to add, replace, or delete other components in some of the components of the embodiments.

[0118] Furthermore, some or all of the above configurations, functions, and processing units may be implemented in hardware, for example, by designing them as integrated circuits. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may be used as hardware.

[0119] Furthermore, each component of the road control system 1 according to the above embodiment may be implemented on any hardware, as long as the respective hardware can send and receive information from each other via a network. Also, the processing performed by a certain processing unit may be realized by a single piece of hardware, or by distributed processing by multiple pieces of hardware. [Explanation of Symbols]

[0120] 1…Road control system, 2…Switching table creation unit, 3…Risk level table creation unit, 4…Switching table management unit, 5…Risk level table management unit, 6…Road condition identification unit, 7…Falling object detection algorithm switching unit, 8…Falling object detection algorithm execution unit, 10a,10b…Camera image, 11…Camera, 12…Falling object, 21…Weather forecasting system, 22…Road traffic information center, 23…Connected car system, 24…Road emergency dial, 25…Outside world recognition sensor, 30…Computer, 40…Dataset, 50…Evaluation results, 60…Switching table, 70…Risk level table, 80…Current road conditions, 90…Output results, 91…Falling object image

Claims

1. A falling object detection system for detecting falling objects on a road, comprising: an arithmetic unit for executing a program; a storage device for storing the program; and a plurality of different falling object detection algorithms, The computing device performs at least the following processes: identifying the current road conditions from information on road conditions obtained by photographing the road; selecting and switching a falling object detection algorithm based on the current road conditions; and executing the selected falling object detection algorithm to detect falling objects on the road. In the process of selecting and switching the falling object detection algorithm, the aforementioned computing device refers to switching information that stores the scores of each falling object detection algorithm for each road condition label, calculates an evaluation value from the scores of the falling object detection algorithms corresponding to each label of the current road condition, and selects the falling object detection algorithm based on the evaluation value. Falling object detection system.

2. There is a risk level table that sets the risk level of falling objects for each road condition label. The calculation device calculates the evaluation value from the score of the falling object detection algorithm corresponding to each label of the current road conditions and the degree of danger for each label of the current road conditions. The falling object detection system according to claim 1.

3. The computing device calculates the score of each falling object detection algorithm for each road condition label from a dataset that associates the correct information with the road condition label for each piece of road condition information, and then creates the switching information. The falling object detection system according to claim 1.

4. The calculation device selects a falling object detection algorithm if the evaluation values ​​of the multiple falling object detection algorithms are the same. If the risk level of the current road conditions is higher than the threshold, the logical OR of the detection results of the selected multiple falling object algorithms is adopted. If the level of danger to the current road conditions is below a threshold, the logical AND of the detection results of the selected multiple falling object algorithms is adopted. The falling object detection system according to claim 2.

5. The aforementioned computing device adjusts the threshold for detecting falling objects in the falling object detection algorithm according to the degree of danger associated with the current road conditions. The falling object detection system according to claim 2.

6. The information regarding road conditions obtained by photographing the road is information from images taken by a camera installed on the road. A falling object detection system according to any one of claims 1 to 5.

7. The aforementioned road conditions include information on road traffic and information on the road environment. A falling object detection system according to any one of claims 1 to 5.

8. A computer having an arithmetic unit for executing a program, a storage device for storing the program, and a plurality of different falling object detection algorithms, is used to detect falling objects on a road, and the method for detecting falling objects on a road, The computing device performs at least the following processes: identifying the current road conditions from information on road conditions obtained by photographing the road; selecting and switching a falling object detection algorithm based on the current road conditions; and executing the selected falling object detection algorithm to detect falling objects on the road. In the process of selecting and switching the falling object detection algorithm, the aforementioned computing device refers to switching information that stores the scores of each falling object detection algorithm for each road condition label, calculates an evaluation value from the scores of the falling object detection algorithms corresponding to each label of the current road condition, and selects the falling object detection algorithm based on the evaluation value. Method for detecting falling objects.

Images