Area monitoring system and area monitoring method

Abstract

To provide an area monitoring system and an area monitoring method capable of more appropriately monitoring a monitoring target area.SOLUTION: An area monitoring system includes: a sensor device installed at a position where a monitoring target area can be imaged; an analysis section that analyzes an image captured by the sensor device; a management section that manages a movement situation of a mobile object moving on a road included in the monitoring target area on the basis of an analysis result of the analysis section; and a providing section that provides the mobile object with predetermined information. The providing section provides the mobile object with an action instruction for the mobile object when the mobile object is likely to depart from the road on the basis of the movement situation.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an area monitoring system and an area monitoring method.

Background Art

[0002] In recent years, efforts have been actively made to provide access to a sustainable transportation system that takes into account people in vulnerable positions among traffic participants. Toward this realization, research and development focusing on further improving traffic safety and convenience through research and development related to safety prevention technologies have been carried out. In this regard, conventionally, technologies are known in which a warning is output when a vehicle deviates from a driving lane by a sensor mounted on the vehicle, or the start of movement of an object is determined using the detection results of a plurality of detection units (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in technologies related to safety prevention, there were cases where the situation of a moving body moving within an area or the situation of a road could not be accurately grasped due to changes in the situation regarding the monitoring target area or the like. Therefore, there was a problem that the monitoring target area might not be appropriately monitored.

[0005] One object of the present application is to provide an area monitoring system and an area monitoring method that can more appropriately monitor a monitoring target area in order to solve the above problems. And by extension, it contributes to the development of a sustainable transportation system.

Means for Solving the Problems

[0006] The area monitoring system and area monitoring method according to this invention employ the following configuration. (1) An area monitoring system according to one aspect of the present invention comprises a sensor device installed at a position capable of capturing images of a monitoring target area, an analysis unit that analyzes images captured by the sensor device, a management unit that manages the movement status of a moving object moving on a road included in the monitoring target area based on the analysis results by the analysis unit, and a provision unit that provides predetermined information to the moving object, wherein the provision unit provides the moving object with instructions to act when there is a possibility that the moving object will deviate from the road based on the movement status.

[0007] (2) In the embodiment of (1) above, the analysis unit assigns identification information to each moving object based on the shape of the moving object included in the image, and the management unit manages the movement status of each moving object based on the assigned identification information.

[0008] (3) In the embodiment of (1) above, the analysis unit measures the travel time of the moving object in a predetermined section included in the image, and the management unit estimates that an abnormality has occurred in the moving object or the monitored area if the difference between the travel time and a predetermined reference travel time is greater than or equal to a threshold.

[0009] (4) In the embodiment of (1) above, the management unit determines that the moving body is slipping if the amount of change in the angle formed by the extension direction of the road and the direction of travel of the moving body over a predetermined time is greater than or equal to a threshold.

[0010] (5) In the embodiment of (1) above, when the moving body moves in a blind spot area within the monitored area that is not included in the sensor device, the management unit manages the movement status of the moving body using images from another moving body that can capture images of the moving body moving in the blind spot area.

[0011] (6): In the embodiment of (1) above, a plurality of sensor devices are provided in the monitored area, and the management unit predicts that an abnormality has occurred in the moving body when the moving body is expected to move out of the sensor range of the first sensor device and move toward the sensor range of the second sensor device, and the moving body cannot be detected by the second sensor device within a predetermined time.

[0012] (7) In the embodiment of (6) above, the management unit predicts that the moving object has moved to the branching road if the moving object could not be detected by the second sensor device within a predetermined time and there is a branching road between the sensor range of the first sensor device and the sensor range of the second sensor device.

[0013] (8) An area monitoring method according to another aspect of the present invention is an area monitoring method in which a computer analyzes images captured by a sensor device installed at a position where the area to be monitored can be imaged, manages the movement status of a moving object moving on a road included in the area to be monitored based on the results of the analysis, provides predetermined information to the moving object, and provides instructions to the moving object to take action if there is a possibility that the moving object will deviate from the road based on the movement status. [Effects of the Invention]

[0014] According to the embodiments described in (1) to (8) above, the area to be monitored can be monitored more appropriately. [Brief explanation of the drawing]

[0015] [Figure 1] This figure shows an example of the configuration of the area monitoring system 1 according to the embodiment. [Figure 2] This figure shows an example of the configuration of the sensor device 100. [Figure 3] This figure shows an example of the configuration of the information provision device 200. [Figure 4] This figure shows an example of the configuration of vehicle 300. [Figure 5] It is a perspective view of vehicle 300 seen from above. [Figure 6] It is a diagram showing an example of the configuration of area monitoring server 400. [Figure 7] It is a diagram for explaining the first monitoring example. [Figure 8] It is a diagram for explaining the second monitoring example. [Figure 9] It is a diagram for explaining the third monitoring example. [Figure 10] It is a diagram showing an example of characteristic information for sensor types. [Figure 11] It is a diagram showing the first example of information provision. [Figure 12] It is a diagram showing the second example of information provision. [Figure 13] It is a sequence diagram showing an example of the processing flow executed by area monitoring system 1 of the embodiment. [Figure 14] It is a flowchart showing an example of the first processing. [Figure 15] It is a flowchart showing an example of the second processing. [Figure 16] It is a flowchart showing an example of the third processing.

Mode for Carrying Out the Invention

[0016] The following describes embodiments of the area monitoring system and area monitoring method of the present invention with reference to the drawings. In the area monitoring system of the embodiments, "moving objects" include vehicles such as two-wheeled, three-wheeled, or four-wheeled vehicles, bicycles, people (pedestrians), and other objects that can move on a road. "Road" may include not only roads (lanes) exclusively for vehicles, but also paths, passages, road surfaces, areas of a predetermined size and length, etc., on which vehicles and people can move. Vehicles include all vehicles that can carry a person (driver) and move on the road surface, including, for example, single-person vehicles (micromobility). In the following description, vehicles will be described as four-wheeled micromobility. Furthermore, the following description will describe the case where left-hand traffic regulations apply, but if right-hand traffic regulations apply, simply reverse left and right.

[0017] [System Configuration] Figure 1 shows an example of the configuration of an area monitoring system 1 according to an embodiment. The area monitoring system 1 shown in Figure 1 includes, for example, a sensor device 100, an information providing device 200, a vehicle 300, and an area monitoring server 400. These are connected communicably via, for example, a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), a LAN (Local Area Network), provider equipment, a wireless base station, etc. The area monitoring system 1 may include one or more of each of the sensor device 100, information providing device 200, vehicle 300, and area monitoring server 400. Furthermore, the area monitoring system 1 may be configured to not include at least one of the information providing device 200 or vehicle 300. In addition, the area monitoring system 1 may transmit and receive information between each component via one or more relay devices (for example, gateway devices or small servers).

[0018] The sensor device 100 detects objects present in the area to be monitored. The area to be monitored is, for example, an area including roads where moving objects such as vehicles 300, pedestrians, cyclists, and other traffic participants pass. When the area to be monitored is a road, multiple sensor devices 100 are installed along the road at predetermined intervals. The sensor device 100 may include optical sensors and radio wave sensors. Optical sensors are, for example, camera devices (image sensors) such as digital cameras, and specifically include stereo cameras, monocular cameras, fisheye cameras, infrared cameras, etc. Radio wave sensors include radar devices, LIDAR (Light Detection and Ranging), TOF (Time Of Flight) cameras, etc. Radar devices emit radio waves such as millimeter waves around the sensor device 100 and detect radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of the object. LIDAR irradiates light (or electromagnetic waves with wavelengths close to light) around the sensor device 100 and measures the scattered light. The LIDAR detects the distance from the sensor device 100 to the target based on the time from light emission to light reception. The emitted light is, for example, pulsed laser light. If the area to be monitored is a road, the sensor device 100 is installed, for example, to image the area including the road from above the road. The sensor ranges (shooting ranges) of the multiple sensor devices 100 installed may overlap in some or all areas.

[0019] The information providing device 200 is installed near the area under surveillance and provides information to vehicles 300 and other traffic participants passing through the area under surveillance. Alternatively, the information providing device 200 may be a terminal device (for example, a smartphone or tablet) owned by the administrator (maintenance officer) responsible for maintaining (managing) the area under surveillance.

[0020] Vehicle 300 is driven by power output from an internal combustion engine that operates on fuel such as gasoline, diesel, or hydrogen. Alternatively, vehicle 300 may be driven by an electric motor powered by electricity supplied by a battery. The battery may be, for example, a lithium-ion battery (LIB), a nickel-metal hydride battery, or a solid-state battery. Vehicle 300 may also be a hybrid vehicle driven by the aforementioned internal combustion engine or electric motor. The internal combustion engine and electric motor are just examples of power sources mounted on vehicle 300.

[0021] The area monitoring server 400 monitors the status of the monitored area (for example, the status of moving objects within the area, the road surface, and the installation status of the sensor device 100) based on information obtained from the sensor device 100 and the vehicle 300 via the network NW, and outputs information based on the monitoring results via the information providing device 200. The area monitoring server 400 may be implemented, for example, as a server device or storage device incorporated into a cloud computing system. In this case, the functions of the area monitoring server 400 may be implemented by multiple server devices and storage devices in the cloud computing system.

[0022] Next, we will specifically describe the functional configurations of the sensor device 100, the information provision device 200, the vehicle 300, and the area monitoring server 400.

[0023] [Sensor device] Figure 2 shows an example of the configuration of the sensor device 100. In the example in Figure 2, the configuration is shown when the sensor device 100 is a camera device. The sensor device 100 includes, for example, a communication unit 110, an imaging unit 120, and a control unit 130.

[0024] The communication unit 110 communicates with the area monitoring server 400 and other external devices via the network NW. For example, the communication unit 110 transmits image data captured by the imaging unit 120 to the area monitoring server 400. The communication unit 110 may also transmit information received from the area monitoring server 400 to nearby information providing devices 200 or vehicles 300.

[0025] The imaging unit 120 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 120 repeatedly captures an area including the area to be monitored at a predetermined period or timing. The field of view (shooting area) of the imaging unit 120 is fixed. The imaging unit 120 may be a stereo camera, a monocular camera, a fisheye camera, an infrared camera, etc.

[0026] If the sensor device 100 is a radio wave type sensor, a radar device, LIDAR, TOF camera, etc. may be provided in place of (or in addition to) the imaging unit 120. Furthermore, the sensor device 100 may be equipped with a microphone for collecting ambient sounds.

[0027] The control unit 130 controls the entire configuration of the sensor device 100. For example, the control unit 130 transmits sensor data, which includes an image captured by the imaging unit 120 (hereinafter referred to as a camera image), acquisition date and time information, and identification information that identifies the sensor device 100 (e.g., sensor ID), to the area monitoring server 400 via the network NW. In addition to (or instead of) the camera image, the sensor data may include information obtained by various sensors such as radar devices, LIDAR, and microphones.

[0028] [Information providing device] Figure 3 shows an example of the configuration of the information providing device 200. The information providing device 200 includes, for example, a communication unit 210, a display 220, a speaker 230, and a control unit 240.

[0029] The communication unit 210 communicates with the area monitoring server 400 and other external devices via the network NW. For example, the communication unit 210 receives various information transmitted from the area monitoring server 400.

[0030] The display 220 displays images related to the information provided by the area monitoring server 400. The display 220 is, for example, a digital signage system such as an electronic billboard or electronic signboard. The speaker 230 outputs audio related to the information provided by the area monitoring server 400. The information providing device 200 only needs to include at least one of the display 220 and the speaker 230.

[0031] The control unit 240 controls the entire configuration of the information providing device 200. For example, the control unit 240 generates images and audio related to the information provided by the area monitoring server 400 and outputs the generated images and audio from the display 220 or speaker 230. Alternatively, if images or audio are provided by the area monitoring server 400, the control unit 240 may output those images and audio directly to the display 220 or speaker 230.

[0032] The information providing device 200 may be provided with a light-emitting unit in place of (or in addition to) the above-described configuration. The light-emitting unit illuminates or flashes light-emitting elements provided on at least a portion of road markings such as stop lines and pedestrian crossings on roads in the monitored area. The light-emitting unit is, for example, an LED (Light Emitting Diode), but is not limited to this. The light-emitting unit may also illuminate the area around or at least a portion of the display. The light-emitting unit may emit light in a predetermined color, or in a color instructed by the control unit 240. The control unit 240 also illuminates the light-emitting unit in response to instructions from the area monitoring server 400. The light-emitting unit is provided, for example, on pedestrian crossings that cross roads or on stop lines that stop the progress of vehicles.

[0033] [vehicle] Figure 4 shows an example of the configuration of vehicle 300. Vehicle 300 is equipped with, for example, an external detection device 302, a vehicle sensor 304, an operator 306, an internal camera 308, a positioning device 310, a communication device 312, an HMI (Human Machine Interface) 314, a moving mechanism 320, a drive device 330, an external notification device 340, a storage device 350, and a control device 360. Some of these components that are not essential for realizing the functions of the present invention may be omitted.

[0034] The external environment detection device 302 detects the external conditions of the vehicle 300. For example, the external environment detection device 302 is a device whose detection range is at least a part of the area around the vehicle 300 (including the direction of travel). The external environment detection device 302 includes an external camera, radar device, LIDAR, sensor fusion device, etc. The external camera is, for example, a digital camera using a solid-state image sensor such as a CCD or CMOS. The external camera can be mounted at any location on the vehicle 300. When imaging the area in front, the external camera can be mounted on the top of the front windshield or behind the rearview mirror, etc. The external camera, for example, periodically and repeatedly images the area around the vehicle 300 (including the direction of travel). The external camera may be a stereo camera, a monocular camera, a fisheye camera, etc.

[0035] The radar device emits radio waves such as millimeter waves around the vehicle 300 and detects radio waves reflected by objects (reflected waves) to detect at least the position (distance and direction) of an object. The radar device can be mounted at any location on the vehicle 300. The vehicle 300 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR irradiates light (or electromagnetic waves with a wavelength close to light) around the vehicle 300 and measures the scattered light. The LIDAR detects the distance from the vehicle 300 to the target based on the time from emission to reception. The LIDAR can be mounted at any location on the vehicle 300. The external detection device 302 outputs information indicating the detection result (image, object position, etc.) to the control device 360.

[0036] The vehicle sensors 304 include, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, a compass sensor, and a manipulated amount detection sensor attached to the operator 306.

[0037] The control element 306 receives driving operations from the occupants of the vehicle 300. The control element 306 includes, for example, controls for instructing acceleration and deceleration (e.g., an accelerator pedal, a brake pedal, a dial switch, a lever) and controls for instructing steering (e.g., a steering wheel). In this case, the vehicle sensor 304 may include an accelerator opening sensor, a brake pedal pressure sensor, a steering torque sensor, etc. The vehicle 300 may also be equipped with controls other than those described above as the control element 306 (e.g., a non-annular rotary control, a joystick, a button, etc.).

[0038] The internal camera 308 captures images of at least the heads of the occupants of the vehicle 300 from the front. The internal camera 308 is a digital camera that uses an image sensor such as a CCD or CMOS. The internal camera 308 outputs the captured images to the control device 360.

[0039] The positioning device 310 is a device that determines the position of the vehicle 300. The positioning device 310 is, for example, a GNSS (Global Navigation Satellite System) receiver, which determines the position of the vehicle 300 based on signals received from GNSS satellites and outputs it as position information. The position information of the vehicle 300 may be estimated from the position of the Wi-Fi base station to which the communication device 312 is connected. The positioning device 310 may be included in the vehicle sensor 304.

[0040] The communication device 312 communicates with other vehicles in the vicinity using, for example, a cellular network, Wi-Fi network, Bluetooth®, DSRC (Dedicated Short Range Communication), or with various external devices (for example, an area monitoring server 400, a sensor device 100, and an information providing device 200) via a wireless base station.

[0041] The HMI314 presents (or informs, notifies) various information to the occupants of the vehicle 300 and accepts input operations from the occupants. The HMI314 includes various display devices, speakers, microphones, buzzers, touch panels, switches, keys, lamps, etc. The HMI314 is an example of an "internal notification device." For example, the HMI314 informs the occupants of the driving status of the vehicle 300, which is controlled by the control device 360, in different notification modes depending on the driving status. In addition, the HMI314 presents information from the control device 360, or presents information acquired from external devices via the communication device 312.

[0042] The mobility mechanism 320 is a mechanism for moving the vehicle 300 on a road. The mobility mechanism 320 is, for example, a group of wheels including steering wheels and drive wheels. Alternatively, the mobility mechanism 320 may be legs for multi-legged walking.

[0043] The drive unit 330 outputs force to the moving mechanism 320 to move the vehicle 300. For example, the drive unit 330 includes a motor that drives the drive wheels, a battery that stores the power supplied to the motor, and a steering device that adjusts the steering angle of the steering wheels. The drive unit 330 may also be equipped with an internal combustion engine or a fuel cell as a means of outputting driving force or a means of generating power. The drive unit 330 may also be further equipped with a braking device that uses friction or air resistance. The drive unit 330 may control the movement of the vehicle 300 based on information regarding steering control and speed control from the area monitoring server 400, instead of (or in addition to) the operation of the operator 306 (manual driving operation).

[0044] The external notification device 340 is, for example, a lamp, display device, speaker, etc., provided on the outer panel of the vehicle 300 to notify information to the outside of the vehicle 300. The external notification device 340 notifies the surrounding area of ​​the moving object (within a predetermined distance from the vehicle 300) of the driving status of the vehicle 300 controlled by the control device 360, using different notification methods depending on the driving status.

[0045] Figure 5 is a perspective view of vehicle 300 from above. In the figure, FW is the steering wheel, RW is the drive wheel, SD is the steering mechanism, MT is the motor, and BT is the battery. The steering mechanism SD, motor MT, and battery BT are included in the drive unit 330. Also, AP is the accelerator pedal, BP is the brake pedal, WH is the steering wheel, SP is the speaker, and MC is the microphone. The illustrated vehicle 300 is a single-seater four-wheeled vehicle, and the occupant P is seated in the driver's seat DS wearing a seat belt SB. Arrow α1 is the direction of travel (velocity vector) of vehicle 300.

[0046] The external environment detection device 302 is located near the front end of the vehicle 300, and the internal camera 308 is positioned to capture images of the occupant P's head from in front of the occupant P. An external notification device 340, which serves as a display device, is also located near the front end of the vehicle 300. An HMI 314, which serves as a display device, is located in front of the occupant P inside the mobile vehicle. The external notification device 340 may be formed integrally with the speaker SP, and the HMI 314 may be formed integrally with the speaker SP and the microphone MC.

[0047] Returning to Figure 4, the storage device 350 is a non-transient storage device such as an HDD (Hard Disk Drive), flash memory, or RAM (Random Access Memory). The storage device 350 stores map information 352, a program 354 executed by the control device 360, etc. The map information 352 stores, for example, road information associated with location information (road shape (width, curvature, gradient), location of stop lines and pedestrian crossings), POI (Point of Interest) information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. Location information includes latitude and longitude. In the figure, the storage device 350 is shown outside the frame of the control device 360, but the storage device 350 may be included in the control device 360.

[0048] The control device 360 ​​includes, for example, an object recognition unit 362 and a control unit 364. The object recognition unit 362 and the control unit 364 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software) 354. Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in the storage device 350 in advance, or it may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed in the storage device 350 when the storage medium is mounted on a drive device.

[0049] The object recognition unit 362 recognizes the surrounding conditions of the vehicle 300 based on the output of the external detection device 302. For example, the object recognition unit 362 recognizes objects that exist within a predetermined distance from the vehicle 300. Objects include some or all of the following: moving objects such as vehicles, bicycles, and pedestrians; road boundaries such as road markings, steps, guardrails, shoulders, and median strips; structures installed on the road such as road signs and billboards; and obstacles such as fallen objects present (on the road). For example, the object recognition unit 362 acquires information on the presence, location, and type of other moving objects by inputting the image captured by the external camera of the external detection device 302 into a trained model that has been trained to output information such as the presence, location, and type of an object when the image captured by the external camera of the external detection device 302 is input. The type of other moving objects can also be estimated based on the size in the image and the intensity of the reflected waves received by the radar device of the external detection device 302. Furthermore, the object recognition unit 362 may recognize the speed of other moving objects detected by a radar device using, for example, Doppler shift.

[0050] Furthermore, the object recognition unit 362 recognizes the lane markings that define the road on which the vehicle 300 is traveling. For example, the object recognition unit 362 recognizes the lane markings by analyzing images captured by the external camera of the external environment detection device 302. In addition, the output of a radar device, LIDAR, sensor fusion device, etc., may be used as an auxiliary for recognizing the lane markings.

[0051] Furthermore, the object recognition unit 362 may compare the position information of the vehicle 300 obtained by the positioning device 310 with the map information 352 to recognize whether or not the vehicle 300 is traveling on a road (whether or not it has deviated from the road).

[0052] The control unit 364 controls all components of the vehicle 300. For example, based on information recognized by the object recognition unit 362, the control unit 364 causes the HMI 314 to output information for notification to the occupants, or causes information to output to the external notification device 340. The control unit 364 also causes the communication device 312 to transmit vehicle data, including information acquired by the external detection device 302, surrounding conditions recognized by the object recognition unit 362, vehicle 300 position information determined by the positioning device 310, and date and time information (acquisition date and time, recognition date and time, positioning date and time), to the area monitoring server 400. Furthermore, based on information provided by the area monitoring server 400, the control unit 364 causes the HMI 314 to output information, causes information to output to the external notification device 340, or causes the drive unit 330 to execute driving control.

[0053] [Area monitoring server] Figure 6 shows an example of the configuration of an area monitoring server 400. The area monitoring server 400 includes, for example, a server-side communication unit 410, an acquisition unit 420, an analysis unit 430, a management unit 440, a provision unit 450, and a server-side storage unit 460. The acquisition unit 420, the analysis unit 430, the management unit 440, and the provision unit 450 are implemented, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as LSIs, ASICs, FPGAs, and GPUs, or by the cooperation of software and hardware. The program may be stored in advance in a storage device, or it may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed in the storage device when the storage medium is mounted on a drive device.

[0054] The server-side storage unit 460 may be implemented using the various storage devices mentioned above, or by SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), RAM, etc. The server-side storage unit 460 stores, for example, a monitoring information DB (Database) 462, infrastructure equipment information 464, map information 466, programs, and various other information.

[0055] The monitoring information DB 462 stores information such as sensor data transmitted from the sensor device 100 and vehicle data transmitted from the vehicle 300. The infrastructure equipment information 464 associates, for example, the sensor ID of the sensor device 100 or the identification information (information providing device ID) of the information providing device 200 with the installation location, the orientation of the installation (angle of view (shooting range), display direction), and type information. For example, in the case of the sensor device 100, the type information includes identification information that identifies stereo cameras, monocular cameras, fisheye cameras, infrared cameras, TOF cameras, radar devices, LIDAR, microphones, etc. In the case of the information providing device 200, the type information includes identification information that identifies displays, speakers, light-emitting parts, etc. The map information 466 contains the same information as the map information 352. The map information 466 may be updated as needed by the server-side communication unit 410 communicating with an external device.

[0056] The server-side communication unit 410 communicates with the sensor device 100, the information providing device 200, the vehicle 300, and other external devices via the network NW.

[0057] The acquisition unit 420 acquires information from the sensor device 100, the vehicle 300, and other external devices. For example, the acquisition unit 420 acquires sensor data from the sensor device 100 or vehicle data from the vehicle 300, and stores the acquired data in the monitoring information DB 472.

[0058] The analysis unit 430 analyzes the sensor data and vehicle data acquired by the acquisition unit 420. For example, if the sensor data or vehicle data includes image data, the analysis unit 430 analyzes the image data. For example, the analysis unit 430 transforms the image data into an overhead coordinate system and performs image analysis processing based on the transformed coordinate system using well-known methods (binarization, contour extraction, image enhancement, feature extraction, pattern matching, etc.) to recognize the surrounding conditions of the sensor device 100. Alternatively, the analysis unit 430 may perform the above-mentioned image analysis processing without performing coordinate transformation.

[0059] For example, the analysis unit 430 identifies objects included in the image data based on the matching result (degree of match) between the feature information obtained by analyzing the image data and the feature information predetermined for each object. The degree of match may be derived, for example, by determining how many of the multiple feature elements included in the feature information match, or by determining the similarity of each feature element or the entire feature information. Alternatively, the degree of match may be derived, for example, based on the sum of the differences for each element. Furthermore, the degree of match may be derived from two pieces of information being compared using AI (Artificial Intelligence) functions such as machine learning (neural networks) or deep learning, or by using other methods.

[0060] Furthermore, the analysis unit 430 may recognize the position, type, speed, etc., of objects present within the monitored area included in the image data. Objects include, for example, moving objects such as vehicles 300, pedestrians, and bicycles. Objects may also include road structures, etc. Road structures include, for example, road signs, traffic signals, curbs, median strips, guardrails, fences, walls, level crossings, pedestrian crossings drawn on the road surface, stop lines, etc. Objects may also include obstacles that obstruct (or are likely to obstruct) the movement of vehicle 300.

[0061] Furthermore, the analysis unit 430 recognizes the position (relative position) of moving objects within the monitored area and recognizes the state of the moving objects, such as their speed, acceleration, and direction of movement, using time-series image data. The position of an object is recognized as a position in an absolute coordinate system with the origin being, for example, a representative point of the moving object (such as the center of gravity or the center of the drive axis). The "state" of an object may include, for example, the acceleration and jerk of the moving object, or its "action state" (for example, whether or not it is crossing or about to cross a pedestrian crossing). The analysis unit 430 may also recognize objects by processing similar to the object recognition processing performed by the object recognition unit 362.

[0062] Furthermore, the analysis unit 430 may analyze characteristic information such as shape to identify moving objects included in the monitored area. Characteristic information may include not only shape, but also patterns (including symbols, numbers, etc., visible from the outside), color, etc. The analysis unit 430 may also measure the travel time of a moving object in a predetermined section.

[0063] Furthermore, the analysis unit 430 may analyze the error between the position of the road included in the image data and the position of the road included in reference image data (reference image) previously captured by the same sensor device 100. The reference image is an image captured using the same sensor device 100 at the correct position, direction, and field of view (shooting range). For example, the degree of deviation between the road conditions based on the image data and the road conditions based on the predetermined reference image is derived. The degree of deviation is an index value that indicates the magnitude of the deviation, and the larger the deviation, the greater the degree. For example, the analysis unit 430 binarizes both the captured image data and the reference image data, and analyzes the degree of deviation from the amount of deviation (amount of deviation) of the distance and direction of the road markings obtained from the binarized results.

[0064] Furthermore, the analysis unit 430 may extract an image region to be used for analysis from the entire image region of the image data based on the season, weather, or time of day when the image data was acquired, and analyze the amount of deviation using the image of the extracted region (extracted image). This makes it possible to suppress misrecognition of objects included in the image due to the movement of shadows caused by buildings, trees, etc., around the monitored area.

[0065] Furthermore, the analysis unit 430 may monitor the situation in the monitored area using vehicle data obtained by sensors (external detection devices 302) mounted on other moving objects when a moving object is moving outside the sensor range of multiple sensor devices in the monitored area. The analysis unit 430 may also recognize at least one piece of information from the sensor data, which includes information about the movement of the moving object and the reflectivity corresponding to the attributes of the moving object. Information about the movement of the moving object includes various types of information that change with movement (from which the amount of movement can be derived), such as the deviation of the moving object's position and velocity distribution. The deviation of the moving object's position is, for example, the deviation of its position in the lateral direction (road width direction) relative to the extension direction (longitudinal direction) of the road. The deviation includes information such as being to the right of the center of the lane and the amount of change in lateral position over a predetermined time. The velocity distribution is the time-series velocity pattern of the moving object (for example, gradually decelerating or accelerating, or traveling at a constant speed). The attributes of the moving object are, for example, the type of vehicle, pedestrian, bicycle, etc.

[0066] The management unit 440 manages the status of the monitored area based on the analysis results from the analysis unit 430. For example, the management unit 440 manages whether a vehicle 300 traveling on a road included in the monitored area is prone to deviating from the road, whether the vehicle 300 is slipping (skidding), and whether there is a possibility of contact with other objects. The management unit 440 may also determine that there is an abnormality in the vehicle 300 (e.g., poor health of the occupant, vehicle malfunction) or an abnormality in the monitored area (e.g., icy road surface) if the difference between the driving time of the vehicle 300 in a predetermined section measured by the analysis unit 430 and a predetermined standard driving time is greater than or equal to a threshold.

[0067] Furthermore, the management unit 440 may also include, for example, a determination unit 442. The determination unit 442 determines whether the moving objects detected by each of the multiple sensor devices 100 installed near the area to be monitored are the same object, based on the detection results of each of the multiple sensor devices 100. If the management unit 440 determines that they are the same object, it integrates the analysis results of each sensor to track the behavior of the moving object; if it determines that they are not the same object, it tracks the behavior of the moving objects individually. The determination unit 442 may also determine whether the degree of deviation from the reference image analyzed by the analysis unit 430 is greater than or equal to a threshold. If the management unit 440 determines that the degree of deviation is greater than or equal to a threshold, it determines that maintenance of at least one of the surrounding monitoring area or the sensor device 100 is necessary. For example, the management unit 440 determines that maintenance of the sensor device 100 is necessary if the error between the position of the road included in the image captured by the sensor device 100 and the position of the road included in the reference image is greater than or equal to a threshold.

[0068] The information provider 450 provides information to moving entities (vehicles 300, pedestrians, and other traffic participants) moving within the monitored area via vehicles 300 traveling within the monitored area, sensor devices 100 installed near the area, and information provider devices 200. The information provided in this case may include, for example, instructions for action to moving entities (e.g., slow down, stop, etc.). The information provider 450 may also provide information to managers who manage the monitored area (e.g., maintenance personnel). The information provided in this case may include maintenance locations and maintenance details (e.g., adjustment of the installation location of sensor devices, cleaning of the monitored area).

[0069] [Specific examples of area monitoring] Next, we will explain several specific examples of area monitoring in Area Monitoring System 1.

[0070] <First monitoring example> Figure 7 is a diagram illustrating the first monitoring example. The first monitoring example involves comprehensively monitoring the road conditions by integrating information obtained from multiple sensor devices 100 installed on the road (an example of a monitoring area). In the example in Figure 7, vehicles 300-1 and 300-2 are shown traveling in lane L1 demarcated by road markings LR and LL, and sensor devices 100-1 and 100-2 are shown installed near lane L1. Sensor device 100-1 is an example of the "first sensor device," and sensor device 100-2 is an example of the "second sensor device." Vehicle 300-1 is assumed to be traveling at a speed V1 in the direction of extension of lane L1, and vehicle 300-2 is assumed to be traveling behind vehicle 300-1 at a speed V2 in the same direction as vehicle 300-1. Sensor devices 100-1 and 100-2 are installed at predetermined intervals and capture images of an area including at least a portion of the road (lane L1) with fixed field of view (imaging range) AR1 and AR2, respectively. Sensor data, including the captured camera images, is transmitted to the area monitoring server 400 by the communication unit 110. The installation position, direction, and field of view (information regarding the reference image) of sensor devices 100-1 and 100-2 are registered in advance in the infrastructure equipment information 464.

[0071] The acquisition unit 420 of the area monitoring server 400 acquires sensor data transmitted from sensor devices 100-1 and 100-2. The acquired sensor data may be stored in the monitoring information DB 462. The analysis unit 430 analyzes the sensor data and recognizes the positions of vehicles 300-1 and 300-2, the positions of road markings LR and LL, etc. The analysis unit 430 also analyzes the speed of vehicles 300-1 and 300-2 from the amount of movement (change in position) over a predetermined time obtained from the time-series sensor data. Furthermore, the analysis unit 430 identifies the vehicles included in each of the sensor data acquired from multiple sensor devices 100 based on the degree of matching of characteristic information such as the shape of vehicle 300 (for example, the roof portion of vehicle 300). For example, the analysis unit 430 identifies vehicles with a degree of matching above a threshold as the same vehicle (same object), assigns common identification information to those vehicles, and assigns different identification information to vehicles with a degree of matching below the threshold. This allows the management unit 440 to manage the movement status of each vehicle based on the assigned identification information. The analysis unit 430 may also perform similar analysis processing on other moving objects in the monitored area besides the vehicle 300.

[0072] The management unit 440 tracks the driving status of the vehicle 300 based on the analysis results, manages whether vehicles 300-1 and 300-2 tend to deviate from lane L1, and manages whether there are any abnormalities in the behavior of vehicles 300-1 and 300-2 due to slipping or other reasons. For example, the determination unit 442 of the management unit 440 obtains the speed and direction of movement of the vehicle 300 between predetermined points based on the analysis results, and determines that there is a possibility that the vehicle 300 will cross the road markings LR and LL within a predetermined time based on the positional relationship between the obtained speed and direction of movement and the road markings LR and LL, and determines that the vehicle 300 tends to deviate from lane L1.

[0073] Furthermore, the control unit 440 may determine that the vehicle 300 is slipping if the amount of change in the angle formed by the direction of extension of lane L1 (road) and the direction of travel of the vehicle 300 over a predetermined time (the rate of change of the yaw angle (yaw rate (angular velocity)) with respect to the direction of travel over a predetermined time) is greater than or equal to a threshold.

[0074] Here, multiple sensor devices 100 installed in the area to be monitored are installed consecutively at intervals such that their respective sensor ranges (for example, the field of view of a camera device) overlap, allowing for more accurate tracking of the same vehicle or person. However, in reality, due to factors such as road shape and equipment costs, the sensor ranges may not overlap, resulting in blind spots in the area to be monitored. Therefore, in the first monitoring example, when the sensor ranges do not overlap, and it is estimated that a passing vehicle is outside the sensor range of one of the multiple sensor devices 100 and is moving towards the sensor range of the other sensor device 100, information about the section where the sensor ranges do not overlap is obtained from the detection results of the external environment detection device 302 mounted on the vehicle 300.

[0075] For example, when monitoring the driving status of vehicle 300-1 as shown in Figure 7, the behavior of vehicle 300-1 outside the sensor range (field of view AR1, AR2) of sensor devices 100-1 and 100-2 is tracked using image data captured by the external environment detection device 302 mounted on vehicle 300-2. This makes it possible to recognize lane departures, skidding, etc., of vehicle 300-1 in blind spots not included in the field of view AR1, AR2.

[0076] Furthermore, in the first monitoring example, even if the sensor ranges of multiple sensor devices 100 overlap, tracking may be performed using the analysis results of image data captured by the external detection device 302 mounted on the vehicle 300-2, even if a blind spot area occurs, by analyzing images extracted from the entire image area of ​​the image data based on the season, weather, or time of day.

[0077] Thus, according to the first monitoring example, by interpolating the blind spots in the sensor data from the sensor device 100 using vehicle data acquired from the vehicle 300-2, the blind spot area can be reduced, and the situation occurring in the monitored area can be monitored without fail.

[0078] <Second example of monitoring> Figure 8 illustrates a second monitoring example. This second monitoring example involves monitoring the behavior of a vehicle 300 based on the analysis results of images captured by multiple sensor devices 100 installed in the monitoring area and the road shape included in the monitoring area. The example in Figure 8 shows a T-junction road where lane L2 is connected perpendicularly to lane L1.

[0079] In the second monitoring example, sensor devices 100-1 and 100-2 are installed in positions where their sensor ranges (angles AR1 and AR2) do not overlap. The management unit 440 tracks the driving status of vehicle 300-1 based on each of the sensor devices 100-1 and 100-2. Here, if it is estimated that vehicle 300, which was within the angle AR1 of sensor device 100-1, has moved outside the angle AR1 and is heading towards the angle AR2 of sensor device 100-2, and if vehicle 300 is not recognized in the sensor data captured by sensor device 100-2 within a predetermined time, the management unit 440 predicts that there may be an abnormality in vehicle 300 or that vehicle 300 has stopped. Furthermore, based on the positional information of sensor devices 100-1 and 100-2, the management unit 440 refers to map information 466 to obtain the road shape around the installation location of sensor device 100, and predicts that vehicle 300 may have moved into lane L2 if there is a lane (branching lane) L2 that connects to lane L1 between the respective sensor ranges of sensor devices 100-1 and 100-2. Lane L2 is an example of a "branching road".

[0080] Furthermore, the management unit 440 may predict that an abnormality such as an accident may have occurred in the section between field angles AR1 and AR2 if vehicle 300 is not recognized in the sensor data captured by sensor device 100-2 within a predetermined time, and no other vehicles are recognized (or the number of passing vehicles is below a threshold). According to the first and second monitoring examples, the area to be monitored can be monitored more appropriately using multiple sensor devices 100 installed on the road.

[0081] <Third example of monitoring> Figure 9 is a diagram illustrating a third monitoring example. The third monitoring example involves monitoring the installation status of a sensor device 100 installed near the area to be monitored. In the example in Figure 9, a vehicle 300 traveling in the X-axis direction on road RD1, which is also passable by oncoming vehicles, is shown. For example, the orientation and sensor range of the sensor device 100 may deviate from the standard due to the effects of earthquakes, wind and rain, etc. If this deviation is not detected early, it may become impossible to recognize objects or accurate area monitoring may not be possible due to misrecognition, etc. Therefore, in the third monitoring example, the management unit 440 aggregates the positions (road positions) where vehicles 300 traveling in the same direction (for example, in the X-axis direction) on road RD1 in the area to be monitored pass, and monitors the installation status of the sensor device 100 based on the aggregated results.

[0082] For example, the management unit 440 obtains the error (pixel deviation) W1 between the actually recognized road position obtained as a result of the aggregation and the predetermined road position for the sensor device 100 installed on the road, and determines that there is an abnormality in the state (installation state) of the sensor device 100 if the obtained error W1 is greater than or equal to a threshold.

[0083] In the example shown in Figure 9, the determination was made based on the lateral displacement W1 of road RD1. However, instead of this (or in addition to this), the displacement amounts in the longitudinal, diagonal, and rotational directions of road D1 may be obtained, and the presence or absence of an abnormality in the sensor device 100 may be determined based on the obtained multidimensional displacement amounts.

[0084] Furthermore, if the management unit 440 determines that the total number of pixels recognized as road areas from the image data is below a threshold, it may predict that there may be fallen leaves or other obstacles on the road.

[0085] Furthermore, the total number of pixels recognized as roads may change if shadows are expected to be cast on roads within the monitored area due to factors such as the direction of sunlight, the surrounding environment (e.g., trees and buildings), and streetlights installed in the monitored area. Therefore, the management unit 440 may adjust the total number of pixels for each time period, or it may set areas in the image where recognition processing is not performed, taking into account the position and extent of shadows at different times of day, and monitor the road conditions based on the total number of road pixels using images outside of those areas. The management unit 440 may also adjust the time period for each season, and it may determine whether or not to perform the above-mentioned processing depending on the presence or absence of shadows due to the weather.

[0086] Furthermore, in the third monitoring example, the management unit 440 may determine, based on the monitoring results, whether maintenance is required for at least one of the monitored area or the sensor device 100. For example, maintenance may include maintenance of roads included in the monitored area or maintenance of the sensor device 100 installed in the monitored area. Road maintenance may include, for example, cleaning up fallen leaves and garbage on the road, removing obstacles, and draining puddles. Maintenance of the sensor device 100 may include, for example, correcting (readjusting) any misalignment in the installation direction of the sensor device 100, or repairing or replacing the device itself. Hereinafter, specific examples of monitoring for road maintenance and monitoring for the maintenance of the sensor device 100 will be described.

[0087] <Monitoring of road maintenance> In the case of monitoring road maintenance, the management unit 440 determines that road maintenance is necessary for the road included in the camera image if, for example, the first deviation amount between the reference image and the camera image is greater than or equal to a predetermined amount, and prompts the provision unit 450 to provide information encouraging maintenance. The first deviation amount is, for example, the difference in the number of pixels (bits) when the total number of pixels in the part of the camera image recognized as a road is compared to the same part in the reference image. For example, if a camera image of a road with fallen leaves or the like accumulated on the road surface is binarized, the area with the accumulated leaves may not be recognized as a road, and the number of pixels in the part of the camera image recognized as a road may be less than that of the reference image. Also, if there are puddles or the like on or around the road, the number of pixels may be more than that of the reference image depending on the imaging environment. Therefore, the management unit 440 determines that road maintenance is necessary if the degree of deviation based on the first deviation amount is greater than or equal to a threshold (i.e., the deviation is large).

[0088] <Monitoring for the maintenance of the sensor device 100> In the case of monitoring the maintenance of a sensor device 100 (for example, a camera device), for example, the management unit 440 causes the providing unit 450 to provide information prompting maintenance of the sensor device 100 when the error between the position of the road included in the reference image and the position of the road included in the camera image exceeds a threshold. For example, the management unit 440 determines that maintenance of the camera device that captured the camera image is necessary when the second deviation amount between the reference image and the camera image exceeds a predetermined amount, and causes the providing unit 450 to provide information prompting maintenance. The second deviation amount is the difference between the output result obtained by multiplying the portions recognized (set) as roads in the binarized reference image and the binarized camera image, and the number of pixels (bits) recognized as roads in the binarized reference image. "Multiplying and outputting" means, for example, assigning "1" to the parts (pixels) recognized as roads in each binarized image and "0" to the other parts (pixels), multiplying the values ​​(1 or 0) of pixels at the same position in each image together, and then adding the results for all pixels to obtain the total value. The management unit 440 determines that maintenance of the sensor device 100 is necessary if the degree of deviation based on the second deviation amount is greater than or equal to a threshold (i.e., the deviation is large). The first deviation amount and the second deviation amount described above are analyzed by, for example, the analysis unit 430.

[0089] Thus, according to this third monitoring example, abnormalities in the position of the sensor device 100 or abnormalities in the road can be detected early and notified to maintenance personnel, etc.

[0090] <Fourth example of monitoring> Next, we will explain the fourth monitoring example. The fourth monitoring example involves monitoring a target area by combining sensor data obtained from multiple sensor devices of different types (characteristics). Figure 10 shows an example of characteristic information for each sensor type. In the example in Figure 10, the characteristic information for each sensor type includes evaluation results for position accuracy, speed control, detection range, situation understanding, and environmental resistance. In the example in Figure 10, "◎" indicates the highest evaluation, followed by "〇" and then "△" in descending order of evaluation. Figure 10 may also include evaluation results for each sensor type in terms of cost.

[0091] For example, in sections with a predetermined distance, such as roads, it becomes necessary to use multiple sensors to detect objects within the monitoring area. Since multiple sensor devices may not have the same characteristics (performance, function), an appropriate combination is required depending on the monitoring area. Therefore, in the fourth monitoring example, as shown in Figure 10, the optimal type and number of sensors are installed in the monitoring area, taking into account the characteristics of each sensor, based on predetermined characteristics for each sensor type, and the conditions of the monitoring area are monitored. For example, stereo cameras, TOF cameras, LIDAR, etc., are used in areas where the position of objects needs to be monitored more accurately, and radar devices are installed in areas where the speed of objects needs to be monitored more accurately. In addition, radio wave sensors may be installed in sections of the road included in the monitoring area where the curvature is less than a threshold, and optical sensors may be installed in sections where the curvature of the road is greater than or equal to a threshold. The management unit 440 manages the type (combination) and number of sensor devices 100 to be installed near the monitoring area based on characteristic information and road shape, as shown in Figure 10. The management unit 440 may also adjust the type and number of sensors to be installed according to the cost of the sensor devices 100.

[0092] For example, if a radio wave sensor and an optical sensor are installed in the area to be monitored, and after a moving object (e.g., a vehicle 300) is recognized by the optical sensor, the moving object moves out of the sensor range and a radio wave sensor is present at the destination, the determination unit 442 determines whether the objects detected by each sensor are the same object based on at least one piece of information obtainable from the sensor data, which includes information about the movement of the moving object (position deviation, velocity distribution) and the reflectance corresponding to the attributes of the moving object. For example, the determination unit 442 selects at least one piece of information from the position deviation, velocity distribution, and reflectance of the moving object, depending on the sensor type of the first sensor device and the second sensor device, and determines whether the moving objects detected by each sensor are the same object.

[0093] Furthermore, the determination unit 442 may also use the same information to determine whether or not the moving object is the same object even if the moving object moves out of the detection range after being recognized by the radio wave sensor and an optical sensor is present at the destination.

[0094] Furthermore, the control unit 440 may issue action instructions to the vehicle 300 if the aforementioned moving object is the vehicle 300 and there is an object approaching the vehicle 300 (for example, another vehicle, pedestrian, or bicycle) (in other words, there is an object that may come into contact with the vehicle 300). Action instructions may include, for example, instructions relating to speed control (deceleration or stopping) or steering control, or one or both.

[0095] Furthermore, the control unit 440 may vary the content of the action instructions given to the vehicle 300 depending on the degree of impact of the approaching object. For example, if the approaching object is a person or a bicycle, which are vulnerable road users (road users who are likely to cause injuries or other physical harm), the action instructions may be stronger (more forceful actions may be given) compared to when the approaching object is another vehicle. Specifically, when a person is approaching the vehicle 300 (when the person is within a predetermined distance), the control unit 440 may be instructed to decelerate more forcefully than when the vehicle 300 is approaching. This can further improve safety.

[0096] When an information providing device 200 is set in the monitored area, the providing unit 450, under the control of the management unit 440, outputs an image or sound indicating the content of the action instruction to the information providing device 200. As a result, the action instruction output from the information providing device 200 allows the occupants of the vehicle 300 to drive in a way that avoids contact with objects. If a camera device (image sensor) is included in the multiple sensor devices 100, the management unit 440 may also control the providing unit 450 to acquire information regarding the attributes of objects approaching the vehicle 300 included in the camera image of the camera device, and output the acquired information to the information providing device 200. As a result, the information providing device 200 installed near the approaching object can display not only the action instruction but also the attributes of the approaching object (person, vehicle, bicycle, etc.), allowing for a more accurate identification of the target object for which the action instruction was given.

[0097] Furthermore, the supply unit 450 may, under the control of the management unit 440, output information to the vehicle 300 causing the vehicle 300 to execute either speed control or steering control, or both, according to the content of the action instruction given to the vehicle 300. This allows the vehicle 300's driving control to be executed without waiting for the driver's input from the vehicle 300.

[0098] Thus, according to the fourth monitoring example, even if different types of sensor devices 100 are installed in the area to be monitored, the situation can be monitored more appropriately using the data from each sensor. Furthermore, according to the fourth monitoring example, lower-cost sensor devices can be combined depending on the monitoring content, thus reducing equipment costs. Note that each of the first to fourth monitoring examples described above may include some or all of the other monitoring examples.

[0099] [Information provided] Next, an example of information provision based on monitoring results will be explained using a diagram. Figure 11 shows a first example of information provision. In the example in Figure 11, for example, if it is determined that vehicle 300 is deviating from lane L1 based on the first monitoring example, information is provided to the occupants of vehicle 300 to move in a direction that does not deviate. In the example in Figure 11, a sensor device 100 is installed on a curved road (a section where the curvature is greater than a predetermined value) where vehicles are prone to deviating. In addition, an information provision device 200 is installed near lane L1.

[0100] The management unit 440 tracks the movement of the vehicle 300 based on sensor data obtained from the sensor device 100. If, based on the tracking results, it is determined that the direction of travel of the vehicle 300 (arrow A1 in the figure) may deviate from the area of ​​lane L1 (cross the road markings LR), the information provision unit 450 causes the information provision device 200 to display information (action instructions) to prevent the deviation. In the example in Figure 10, text information such as "Turn left" is displayed on the display 220 of the information provision device 200, which is installed in a position visible to the occupants of the vehicle 300. The management unit 440 manages which information provision device 200 displays this text information.

[0101] Furthermore, the supply unit 450 may transmit information (action instructions) to the vehicle 300 via the sensor device 100 for the occupant of the vehicle 300 to perform steering operations and display it on the HMI 314, or it may transmit control information to the vehicle 300 to automatically perform steering control of the vehicle M, causing the drive unit 330 to perform driving control to steer the vehicle 300 to the left. This makes it possible to move the vehicle 300 in the direction of arrow A2 in the figure.

[0102] Furthermore, if the control unit 440 determines that the vehicle 300 is slipping, the supply unit 450 may display warning information such as "Caution: Icy Road Surface" on the display 220 of the information supply device 200, or transmit information (action instructions) to the vehicle 300 to prompt the occupants of the vehicle 300 to perform deceleration operations (or automatic deceleration control).

[0103] In the example shown in Figure 11, information from the area monitoring server 400 is transmitted to the information providing device 200 and the vehicle 300 via the sensor device 100. However, the information may also be transmitted directly from the area monitoring server 400 to the information providing device 200 and the vehicle 300.

[0104] Figure 12 shows a second example of information provision. In the example in Figure 12, notifications are provided to avoid collisions between moving objects within lane L1. In the example in Figure 12, a vehicle 300, a pedestrian P1, and a bicycle P2 are shown moving within lane L1. Near lane L1 shown in Figure 12, a sensor device 100 that images the area including lane L1 and an information provision device 200 are installed. Also, arrows A3, A4, and A5 shown in Figure 12 indicate the directions of movement of the vehicle 300, pedestrian P1, and bicycle P2 obtained from time-series sensor data.

[0105] The management unit 440 of the area monitoring server 400 analyzes the sensor data detected by the sensor device 100 and determines from the analysis results whether there is a possibility that the vehicle 300, pedestrian P1, or bicycle P2 may come into contact with another object. If it is determined that there is a possibility of contact, it provides information (action instructions) to the sensor device 100, information providing device 200, vehicle 300, etc., to avoid contact.

[0106] For example, the information provision unit 450 transmits information to the information provision device 200 via the network NW to display text images such as "Please be careful of contact" on the display 220 of the information provision device 200. As a result, the occupants of the vehicle 300, pedestrians P1, and cyclists P2 who see the text displayed on the display 220 of the information provision device 200 can avoid contact earlier.

[0107] Furthermore, the information provider 450 may transmit information to the vehicle 300 so that it displays the same information on the display 220 as on the HMI 314 of the vehicle 300, or it may transmit information notifying that a pedestrian P1 or bicycle P2 is approaching. In addition, the information provider 450 may transmit control information to the vehicle 300 to stop the vehicle, or it may transmit instruction information to the vehicle 300 so that the external notification device 340 of the vehicle 300 outputs a warning sound (horn, etc.) to inform the occupants of the pedestrian P1 or bicycle P2 that the vehicle 300 is approaching. This makes it possible to provide more appropriate information according to the situation of objects traveling on the road (lane L1), and to further improve safety when traveling in lane L1.

[0108] Furthermore, in the third monitoring example, when providing information, the management unit 440 may provide information requesting maintenance (environmental improvement) to the maintenance personnel's terminal device (not shown). In this case, the location of the sensor device 100 that has malfunctioned, the location of the area to be monitored, and the type of malfunction (for example, movement of the sensor device 100) may be notified. This allows the maintenance personnel to understand the location and content of the maintenance more concretely, and to prepare for and perform the work more appropriately.

[0109] Furthermore, if the information providing device 200 is equipped with a light-emitting unit, the light-emitting unit may be controlled to illuminate in a state corresponding to an action instruction. For example, as shown in Figure 11, if it is determined that the vehicle 300 is tending to deviate from lane L1, the providing unit 450 will, in addition to (or instead of) providing the information described above, illuminate the light-emitting unit installed on the road markings LR. Also, as shown in Figure 12, if two moving objects are approaching each other, the light-emitting unit installed on the road surface will be illuminated. The providing unit 450 may switch between illumination and flashing, or adjust the color and light intensity, depending on the distance between the vehicle 300 and the road markings LR, and the degree of proximity (relative distance) between the moving objects. This allows the moving objects to understand the current situation more appropriately.

[0110] [Processing Sequence] Figure 13 is a sequence diagram showing an example of the processing flow performed by the area monitoring system 1 of the embodiment. In the example of Figure 13, processing using a sensor device 100, a vehicle 300, an area monitoring server 400, and an information providing device 200 will be described. In the example of Figure 13, the sensor device 100 is assumed to be a camera device, the vehicle 300 is assumed to be a vehicle traveling within the field of view (monitoring area) captured by the sensor device 100, and the information providing device 200 is assumed to be installed near the monitoring area of ​​the sensor device 100.

[0111] In the example shown in Figure 13, the sensor device 100 captures an image of the area to be monitored (step S100) and transmits the sensor data, including the captured camera image, to the area monitoring server 400 (step S102). The vehicle 300 also detects the surrounding conditions of the area to be monitored using an external detection device 302, etc. (step S104) and transmits vehicle data, including the detection results, to the area monitoring server 400 (step S106).

[0112] The area monitoring server 400 receives sensor data and vehicle data, analyzes image data and other information contained in the received data (step S108), and manages the monitored area based on the analysis results (step S110). Specifically, at least one of the first to fourth monitoring examples described above is executed. Subsequently, if the area monitoring server 400 determines that it is necessary to provide information to the monitored area, it generates information to be provided (step S112) and transmits the generated information to the information providing device 200 (step S114). The area monitoring server 400 also transmits the generated information to the vehicle 300 (step S116).

[0113] The information providing device 200 receives information transmitted from the area monitoring server 400 and displays an image corresponding to the received information on the display 220 or the like (step S118). Alternatively, instead of (or in addition to) displaying an image on the display 220, the information providing device 200 may output audio corresponding to the received information from the speaker 230. It may also light up (or flash) a light-emitting part installed on the road in a manner corresponding to the received information.

[0114] Vehicle 300 receives information transmitted from area monitoring server 400 and displays an image corresponding to the received information on HMI 314, or performs driving control (e.g., speed control, steering control) corresponding to the received information (step S120). Alternatively, vehicle 300 may output audio corresponding to the received information to external notification device 340 instead of (or in addition to) the above-mentioned control. This completes the processing of this sequence.

[0115] In addition, during the processing in step S112, the area monitoring server 400 may generate information to be provided to the maintenance personnel if maintenance work is required for the monitored area, and transmit the generated information to the terminal device owned by the maintenance personnel.

[0116] Next, the specific processes described in steps S108 to S112 above will be explained using a flowchart.

[0117] <First process> Figure 14 is a flowchart illustrating an example of the first process. In the example in Figure 14, the sensor device 100 installed in the monitored area is assumed to be a camera device, and the example of a moving object is assumed to be a vehicle 300. The process shown in Figure 14 may be executed repeatedly at predetermined intervals or timings. The same applies to the process shown in Figure 15 (the second monitoring process), which will be described later.

[0118] In the example shown in Figure 14, the acquisition unit 420 acquires camera data from the camera device (step S200). Next, the analysis unit 430 analyzes the acquired camera data (step S202). The management unit 440 assigns identification information to each vehicle included in the camera data based on the shape of the vehicle obtained as a result of the analysis (step S204), and manages the driving status of the vehicle 300 within the monitored area (step S206).

[0119] Next, the management unit 440 determines whether the vehicle 300 is deviating from its lane (road) or whether the vehicle 300 is slipping (step S208). If it is determined that the vehicle is deviating from its lane or is slipping, the providing unit 450 generates information to provide to the vehicle 300 (step S210) and outputs the generated information to the information providing device 200 (step S212). In step S212, the information to be provided to the vehicle 300 may also be transmitted to the vehicle 300. In this case, the information to be provided to the vehicle 300 includes information (images and sounds) to be output from the HMI 314, or information for steering control and speed control. This completes the processing of this flowchart. Also, if it is determined in step S208 that the vehicle 300 has not deviated from its lane or is not slipping, the processing of this flowchart also completes. In addition, the processing of step S208 described above may determine whether there is a possibility of contact between moving objects moving within the monitored area included in the camera data.

[0120] <Second process> Figure 15 is a flowchart illustrating an example of the second process. In the example in Figure 15, the acquisition unit 420 acquires camera data captured by a camera device installed in the area to be monitored (step S300). Next, the analysis unit 430 analyzes the acquired camera data (step S302). The management unit 440 derives the degree of discrepancy between the road conditions based on the camera data obtained as an analysis result and the road conditions based on a predetermined reference image (step S304). Next, the determination unit 442 determines whether the degree of discrepancy is above a threshold (step S306). If it is determined that the degree of discrepancy is above a threshold, the provision unit 450 generates information to encourage maintenance regarding at least one of the area to be monitored or the camera device (an example of a sensor device) (step S308), and outputs the generated information to the terminal device of the maintenance worker, which is an example of the information provision device 200 (step S310). This completes the process in this flowchart. Furthermore, if it is determined in step S306 that the degree of deviation is not equal to or greater than the threshold, the process of this flowchart is terminated.

[0121] <Third process> Figure 16 is a flowchart illustrating an example of the third process. In the example in Figure 16, it is assumed that the multiple sensor devices 100 installed in the monitored area may include both radio wave sensors and optical sensors. The process shown in Figure 16 may be repeatedly executed at predetermined intervals or timings. In the example in Figure 16, the acquisition unit 420 acquires sensor data from the sensor devices 100 (step S400). Next, the analysis unit 430 analyzes the acquired sensor data (step S402). Next, the determination unit 442 determines whether the sensor device 100 from which the sensor data was acquired includes sensor data from both radio wave sensors and optical sensors. If it is determined that the sensor data includes both radio wave sensors and optical sensors, the determination unit 442 determines that the objects are the same based on at least one piece of information from the deviation of the moving object's position, the velocity distribution, and the reflection intensity (step S406). Note that the deviation of the moving object's position and the velocity distribution are just examples, and other information related to the movement of the moving object may be used. Furthermore, if it is determined that the sensor data does not include data from either the radio wave sensor or the optical sensor (i.e., only data from one of the sensors is present), the determination unit 442 determines that it is the same object based on the similarity of the analysis results of each sensor (step S408). Next, the management unit 440 manages the movement status of the same object (step S410). This concludes this flowchart. In the third monitoring process, after step S410, the system may monitor for slippage, lane departure, etc., of the same object and provide information based on the monitoring results. It may also monitor whether or not the object comes into contact with another object and provide information based on the monitoring results.

[0122] <Variation> At least some of the components of the area monitoring server 400 described above may be provided in the sensor device 100, the information providing device 200, or the vehicle 300. For example, if the functions of the analysis unit 430 are provided in the sensor device 100 or the vehicle 300, the results of the analysis performed by each device are transmitted to the area monitoring server 400.

[0123] Furthermore, at least a portion of the configuration of the information providing device 200 in the embodiment may be provided in the sensor device 100, and at least a portion of the configuration of the sensor device 100 may be provided in the information providing device 200.

[0124] According to the embodiments described above, the area monitoring system 1 includes a sensor device 100 installed in a position capable of imaging the area to be monitored, an analysis unit 430 that analyzes the images captured by the sensor device 100, a management unit 440 that manages the movement status of moving objects moving on roads included in the area to be monitored based on the analysis results by the analysis unit 430, and a provision unit 450 that provides predetermined information to the moving objects. The provision unit 450 provides instructions to the moving objects to take action when there is a possibility that the moving objects will deviate from the road based on the movement status, thereby enabling more appropriate monitoring of the area to be monitored. Therefore, it can contribute to the development of a sustainable transportation system.

[0125] For example, according to this embodiment, multiple sensor devices 100 installed in the monitored area can be linked to more accurately track moving objects and understand their situation. Also, according to this embodiment, infrastructure equipment such as cameras installed in a predetermined area can be effectively utilized to provide an infrastructure-cooperative monitoring system. Furthermore, since the sensor devices fixed in the monitored area have a fixed sensor range (angle of view, etc.), they can acquire, for example, the position of road shapes and road markings included in the captured images with high precision, thus enabling a more accurate understanding of road conditions and the movement of moving objects on the road. In addition, according to this embodiment, maintenance can be performed on the area monitoring system 1 more appropriately and quickly. This allows the infrastructure equipment to operate properly. Furthermore, according to this embodiment, traffic management can be performed using radio wave sensors and optical sensors. Therefore, considering the characteristics of each sensor, the optimal type and number of sensors can be installed for each monitored area. Consequently, the overall situation of the region can be understood more appropriately by the infrastructure equipment.

[0126] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0127] 1 Area Monitoring System 100 Sensor device 110, 210 Communications Department 120 Imaging Unit 130, 240, 364 Control Unit 200 Information provision device 220 displays 230 speakers 240 Control Unit 300 vehicles 302 External detection device 304 Vehicle Sensors 306 Operator 308 Internal Camera 310 Positioning device 312 Communication equipment 314 HMI 340 External notification device 360 Control System 400 Area Monitoring Servers 410 Server-side communication unit 420 Acquisition Department 430 Analysis Department 440 Management Department 442 Judgment section 450 Providing Department 460 Server-side storage

Claims

1. Multiple sensor devices are installed in positions that allow imaging of the area to be monitored, An analysis unit that integrates and analyzes information including images captured by the aforementioned multiple sensor devices and information acquired from a moving object moving on a road included in the monitored area, Based on the analysis results from the analysis unit, a management unit manages the movement status of the moving object within the entire area of ​​the monitored area, The system comprises a providing unit that provides the mobile body with information based on the movement status of the mobile body managed by the management unit, The providing unit, when it is predicted that the moving body may deviate from the road within a predetermined time based on the movement status, provides the moving body with instructions for action. Area monitoring system.

2. The analysis unit assigns identification information to each moving object based on the shape of the moving object included in the image. The management unit manages the movement status of each mobile body based on the assigned identification information. The area monitoring system according to claim 1.

3. The analysis unit measures the travel time of the moving object in a predetermined section included in the image, The management unit estimates that an abnormality has occurred in the mobile object or the monitored area if the difference between the travel time and a predetermined standard travel time is greater than or equal to a threshold. The area monitoring system according to claim 1.

4. The management unit determines that the moving body is slipping if the amount of change in the angle formed by the extension direction of the road and the direction of travel of the moving body over a predetermined time period is greater than or equal to a threshold. The area monitoring system according to claim 1.

5. The management unit manages the movement status of the moving object when the moving object moves within a blind spot area of ​​the monitored area that is not included in the sensor range of the sensor device, using images from another moving object that can capture images of the moving object moving within the blind spot area. The area monitoring system according to claim 1.

6. The plurality of sensor devices include a first sensor device and a second sensor device, The control unit predicts that an abnormality has occurred in the moving body when the moving body is expected to move out of the sensor range of the first sensor device and move towards the sensor range of the second sensor device, and the moving body cannot be detected by the second sensor device within a predetermined time. The area monitoring system according to claim 1.

7. The management unit predicts that the moving object has moved to the branching road if it is not detected by the second sensor device within a predetermined time and there is a branching road between the sensor range of the first sensor device and the sensor range of the second sensor device. The area monitoring system according to claim 6.

8. A sensor device installed in a position capable of capturing images of the area to be monitored, An analysis unit that analyzes images captured by the aforementioned sensor device, Based on the analysis results from the aforementioned analysis unit, a management unit manages the movement status of moving objects on roads included in the monitored area, The system comprises a providing unit that provides the mobile body with information based on the movement status of the mobile body managed by the management unit, The providing unit provides instructions to the moving body to take action when there is a possibility that the moving body may deviate from the road based on the movement status. The management unit manages the movement status of the moving object when the moving object moves within a blind spot area of ​​the monitored area that is not included in the sensor range of the sensor device, using images from another moving object that can capture images of the moving object moving within the blind spot area. Area monitoring system.

9. A sensor device installed in a position capable of capturing images of the area to be monitored, An analysis unit that analyzes images captured by the aforementioned sensor device, Based on the analysis results from the aforementioned analysis unit, a management unit manages the movement status of moving objects on roads included in the monitored area, The system comprises a providing unit that provides the mobile body with information based on the movement status of the mobile body managed by the management unit, The providing unit provides instructions to the moving body to take action when there is a possibility that the moving body may deviate from the road based on the movement status. Multiple sensor devices are provided in the aforementioned monitoring area. The control unit predicts that an abnormality has occurred in the moving body when the moving body is expected to move out of the sensor range of the first sensor device and move towards the sensor range of the second sensor device, and the moving body cannot be detected by the second sensor device within a predetermined time. Area monitoring system.

10. Computers Information including images captured by multiple sensor devices installed in positions capable of capturing images of the monitored area, and information acquired from a moving object traveling on a road included in the monitored area are integrated and analyzed. Based on the analysis results, the movement status of the moving object within the entire area of ​​the monitored area is managed. Information based on the movement status of the managed mobile body is provided to the mobile body, If, based on the aforementioned movement status, it is predicted that the moving object may deviate from the road within a predetermined time, instructions for action are provided to the moving object. Area monitoring methods.

Images