Area monitoring system and area monitoring method

Abstract

To provide an area monitoring system and an area monitoring method capable of more appropriately monitoring a monitoring target area.SOLUTION: An area monitoring system includes: a plurality of sensor devices each installed in a monitoring target area and configured to detect a mobile object moving in the monitoring target area; and a determination section that determines whether or not mobile objects detected by the plurality of respective sensor devices are the same object on the basis of detection results of the plurality of respective sensor devices. The plurality of sensor devices includes at least a radio wave sensor and an optical sensor. The determination section determines whether or not the mobile objects detected by the radio wave sensor and the optical sensor are the same object on the basis of information of at least one of feature quantities regarding movements of the mobile objects and reflection intensities corresponding to attributes of the mobile objects when the mobile objects have been detected by the radio wave sensor and the optical sensor.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an area monitoring system and an area monitoring method. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have been gaining momentum. To achieve this, efforts are being made to further improve traffic safety and convenience through research and development of safety prevention technologies. In this regard, conventionally, technologies are known that use sensors mounted on vehicles to output a warning when the vehicle deviates from its lane, or that use detection results from multiple detection units to determine when an object has started to move (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-31167 [Patent Document 2] Patent Publication No. 2021-51466 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in safety prevention technologies, there are cases where the conditions of mobile objects moving within a monitored area or road conditions cannot be accurately grasped due to changes in the conditions of the monitored area, etc. Therefore, there is a problem in that the monitored area may not be monitored appropriately.

[0005] In order to solve the above-mentioned problems, one of the objects of the present application is to provide an area monitoring system and an area monitoring method that can more appropriately monitor a target area, thereby contributing to the development of a sustainable transportation system. [Means for solving the problem]

[0006] The area monitoring system and area monitoring method according to the present invention employ the following configuration. (1): An area monitoring system according to one embodiment of the present invention comprises a plurality of sensor devices installed in a monitored area and detecting moving objects moving in the monitored area, and a determination unit that determines whether the moving objects detected by each of the plurality of sensor devices are the same object, wherein the plurality of sensor devices include at least a radio wave sensor and an optical sensor, and when the moving object is detected by each of the radio wave sensor and the optical sensor, the determination unit determines whether the moving objects detected by each of the radio wave sensor and the optical sensor are the same object based on at least one piece of information selected from the group consisting of a feature amount related to the movement of the moving object and a reflection intensity corresponding to an attribute of the moving object.

[0007] (2): In the above aspect (1), when the moving body moves out of the sensor range after being detected by the optical sensor and the radio wave sensor is present at the moving destination, the determination unit determines whether the moving body is the same object based on at least one piece of information among the deviation of the moving body's position, the speed distribution, and the reflection intensity according to the attributes of the moving body.

[0008] (3): In the above aspect (1), when the moving body moves out of the sensor range after being detected by the radio wave sensor and the optical sensor is present at the moving destination, the determination unit determines whether the moving body is the same object based on at least one piece of information among the deviation of the moving body's position, the speed distribution, and the reflection intensity according to the attributes of the moving body.

[0009] (4): In the above aspect (1), a management unit is further provided for managing the situation of the monitored area, and the management unit issues instructions to the vehicle to take action when the moving body is a vehicle and an object is approaching the vehicle.

[0010] (5) In the above aspect (4), the management unit varies the content of the action instruction given to the vehicle depending on the degree of influence on contact of an object approaching the vehicle.

[0011] (6): In the above aspect (4), when an information providing device is set in the monitored area, the management unit causes the information providing device to output an image or sound indicating the content of the action instruction.

[0012] (7): In the above aspect (6), when the plurality of sensor devices includes an image sensor, the information providing device is caused to output information regarding the attributes of an object approaching the vehicle that is included in an image captured by the image sensor.

[0013] (8): In the above aspect (4), the management unit outputs information to the vehicle to cause the vehicle to perform one or both of speed control and steering control according to the content of the behavioral instructions for the vehicle.

[0014] (9): In the above aspect (1), the radio wave sensor is installed in the section of the road included in the monitored area where the curvature is less than a threshold, and the optical sensor is installed in the section of the road where the curvature is greater than or equal to the threshold.

[0015] (10): Another aspect of the present invention is an area monitoring method in which a computer is installed in a monitored area and determines whether the moving objects detected by each of a plurality of sensor devices that detect moving objects moving in the monitored area are the same object based on the detection results of each of the plurality of sensor devices, the plurality of sensor devices including at least a radio wave sensor and an optical sensor, and when the moving object is detected by each of the radio wave sensor and the optical sensor, determines whether the moving objects detected by each of the radio wave sensor and the optical sensor are the same object based on at least one piece of information among a feature amount related to the movement of the moving object and a reflection intensity corresponding to an attribute of the moving object. [Effects of the Invention]

[0016] According to the above aspects (1) to (10), the area to be monitored can be monitored more appropriately. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram showing an example of the configuration of an area monitoring system 1 according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of the configuration of a sensor device 100. FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an information providing device 200. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a vehicle 300. [Figure 5] FIG. 3 is a perspective view of a vehicle 300 seen from above. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of an area monitoring server 400. [Figure 7] FIG. 10 is a diagram illustrating a first monitoring example. [Figure 8] FIG. 10 is a diagram illustrating a second monitoring example. [Figure 9] FIG. 10 is a diagram illustrating a third monitoring example. [Figure 10] FIG. 10 is a diagram illustrating an example of characteristic information for each sensor type. [Figure 11] FIG. 10 is a diagram showing a first example of information provision. [Figure 12] FIG. 10 is a diagram showing a second example of information provision. [Figure 13] 3 is a sequence diagram showing an example of the flow of processing executed by the area monitoring system 1 of the embodiment. FIG. [Figure 14] 10 is a flowchart illustrating an example of a first process. [Figure 15] 10 is a flowchart illustrating an example of a second process. [Figure 16] 10 is a flowchart illustrating an example of a third process. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, with reference to the drawings, embodiments of the area monitoring system and area monitoring method of the present invention will be described. Note that the mobile object in the area monitoring system of the embodiment includes objects capable of moving on roads, such as two-wheeled, three-wheeled, or four-wheeled vehicles, bicycles, and people (pedestrians). The term "road" may include not only roads (lanes) exclusively for vehicles, but also paths, passages, road surfaces, and areas of a certain size or length on which vehicles and people can move. The term "vehicle" includes any vehicle that can carry a person (driver) and move on a road surface, including, for example, a single-seater vehicle (micromobility). In the following description, the vehicle will be described as a four-wheeled micromobility. Furthermore, the following description will be given assuming that laws stipulating left-hand traffic apply; however, if laws stipulating right-hand traffic apply, the terms "left" and "right" can be reversed.

[0019] [System Configuration] FIG. 1 is a diagram illustrating an example of the configuration of an area monitoring system 1 according to an embodiment. The area monitoring system 1 illustrated in FIG. 1 includes, for example, a sensor device 100, an information providing device 200, a vehicle 300, and an area monitoring server 400. These are communicably connected via, for example, a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi network, a wide area network (WAN), a local area network (LAN), a provider device, a wireless base station, and the like. The area monitoring system 1 may include one or more of each of the sensor device 100, the information providing device 200, the vehicle 300, and the area monitoring server 400. The area monitoring system 1 may also be configured without including at least one of the information providing device 200 and the vehicle 300. The area monitoring system 1 may transmit and receive information to and from each component via one or more relay devices (for example, a gateway device or a small-scale server).

[0020] The sensor device 100 detects objects present in a monitoring area. The monitoring area is, for example, an area including roads on which moving objects such as vehicles 300, pedestrians, and bicycles pass. When the monitoring area is a road, a plurality of sensor devices 100 are installed at predetermined intervals along the road. The sensor device 100 may include an optical sensor and a radio wave sensor. The optical sensor is, for example, a camera device (image sensor) such as a digital camera, and specifically includes a stereo camera, a monocular camera, a fisheye camera, and an infrared camera. The radio wave sensor includes a radar device, a LIDAR (Light Detection and Ranging) camera, a TOF (Time of Flight) camera, and the like. A radar device emits radio waves such as millimeter waves around the sensor device 100 and detects radio waves reflected by the object (reflected waves) to detect at least the position (distance and direction) of the object. A LIDAR emits light (or electromagnetic waves with a wavelength similar to light) around the sensor device 100 and measures the scattered light. LIDAR detects the distance from the sensor device 100 to an object based on the time between light emission and light reception. The emitted light is, for example, pulsed laser light. If the area to be monitored is a road, the sensor device 100 is installed, for example, so as to capture an image of an area including the road from above the road. The sensor ranges (image capture ranges) of the installed multiple sensor devices 100 may or may not at least partially overlap.

[0021] The information providing device 200 is installed near the area to be monitored and provides information to vehicles 300 and traffic participants passing through the area to be monitored. The information providing device 200 may also be a terminal device (for example, a smartphone or a tablet terminal) owned by a manager (maintenance officer) who maintains (manages) the area to be monitored.

[0022] The vehicle 300 runs on power output in response to the operation of an internal combustion engine that runs on fuel such as gasoline, diesel, or hydrogen. The vehicle 300 may also run on an electric motor that is driven by power supplied by a battery. The battery may be, for example, a lithium-ion secondary battery (LIB), a nickel-metal hydride battery, or an all-solid-state battery. The vehicle 300 may also be a hybrid vehicle that uses the above-mentioned internal combustion engine or electric motor as a drive source. The internal combustion engine and the electric motor are examples of power sources installed in the vehicle 300.

[0023] The area monitoring server 400 monitors the status of the area to be monitored (for example, the status of mobile objects moving within the area, the status of the road surface, and the installation status of the sensor device 100) based on information obtained from the sensor device 100 and the vehicle 300 via the network NW, and causes the information providing device 200 to output information based on the monitoring results. The area monitoring server 400 may be realized, for example, by a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the area monitoring server 400 may be realized by multiple server devices and storage devices in the cloud computing system.

[0024] Next, the functional configurations of the sensor device 100, the information providing device 200, the vehicle 300, and the area monitoring server 400 will be specifically described.

[0025] [Sensor device] Fig. 2 is a diagram illustrating an example of the configuration of the sensor device 100. Note that the example in Fig. 2 illustrates a configuration in which the sensor device 100 is a camera device. The sensor device 100 includes, for example, a communication unit 110, an imaging unit 120, and a control unit 130.

[0026] The communication unit 110 communicates with the area monitoring server 400 and other external devices via the network NW. For example, the communication unit 110 transmits image data captured by the imaging unit 120 to the area monitoring server 400. The communication unit 110 may also transmit information received from the area monitoring server 400 to an information providing device 200 or a vehicle 300 present in the vicinity.

[0027] The imaging unit 120 is, for example, a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 120 repeatedly captures images of an area including a monitoring target area at a predetermined cycle or at a predetermined timing. The angle of view (image capture area) of the imaging unit 120 is fixed. The imaging unit 120 may be a stereo camera, a monocular camera, a fisheye camera, an infrared camera, or the like.

[0028] If the sensor device 100 is a radio wave sensor, a radar device, a LIDAR, a TOF camera, or the like may be provided instead of (or in addition to) the imaging unit 120. The sensor device 100 may also be provided with a microphone that picks up ambient sounds.

[0029] The control unit 130 controls the overall components of the sensor device 100. For example, the control unit 130 transmits sensor data including an image captured by the imaging unit 120 (hereinafter referred to as a camera image), image capture date and time information, and identification information (e.g., a sensor ID) for identifying the sensor device 100, to the area monitoring server 400 via the network NW. The sensor data may include information obtained by various sensors such as a radar device, a LIDAR, a microphone, etc. in addition to (or instead of) the camera image.

[0030] [Information providing device] 3 is a diagram showing an example of the configuration of the information providing device 200. The information providing device 200 includes, for example, a communication unit 210, a display 220, a speaker 230, and a control unit 240.

[0031] The communication unit 210 communicates with the area monitoring server 400 and other external devices via the network NW. For example, the communication unit 210 receives various types of information transmitted from the area monitoring server 400.

[0032] The display 220 displays an image related to the information provided by the area monitoring server 400. The display 220 is, for example, a digital signage system such as an electronic bulletin board or electronic signboard. The speaker 230 outputs audio related to the information provided by the area monitoring server 400. It is sufficient that the information providing device 200 includes at least one of the display 220 and the speaker 230.

[0033] The control unit 240 controls the overall components of the information providing device 200. For example, the control unit 240 generates images and sounds related to the information provided by the area monitoring server 400, and outputs the generated images and sounds from the display 220 or the speaker 230. Furthermore, when images and sounds are provided from the area monitoring server 400, the control unit 240 may output the images and sounds directly to the display 220 or the speaker 230.

[0034] The information providing device 200 may be provided with a light-emitting unit instead of (or in addition to) the above-described configuration. The light-emitting unit, for example, lights up or flashes light-emitting elements provided on at least a part of road markings such as stop lines and crosswalks provided on roads in the monitored area. The light-emitting unit is, for example, an LED (Light Emitting Diode), but is not limited to this. The light-emitting unit may also illuminate the periphery or at least a part of the display. The light-emitting unit may emit light in a predetermined color, or in a color instructed by the control unit 240. The control unit 240 causes the light-emitting unit to emit light in response to an instruction from the area monitoring server 400. The light-emitting unit is provided, for example, on crosswalks that cross roads or stop lines that stop vehicles from proceeding.

[0035] [vehicle] 4 is a diagram showing an example of the configuration of a vehicle 300. The vehicle 300 is equipped with, for example, an external environment detection device 302, a vehicle sensor 304, an operator 306, an internal camera 308, a positioning device 310, a communication device 312, an HMI (Human Machine Interface) 314, a movement mechanism 320, a drive device 330, an external notification device 340, a storage device 350, and a control device 360. Note that some of these components that are not essential for realizing the functions of the present invention may be omitted.

[0036] The external environment detection device 302 detects the external situation of the vehicle 300. For example, the external environment detection device 302 is a variety of devices whose detection range covers at least a portion of the periphery of the vehicle 300 (including the direction of travel). The external environment detection device 302 includes an external camera, a radar device, a LIDAR, a sensor fusion device, etc. The external camera is, for example, a digital camera that uses a solid-state imaging element such as a CCD or CMOS. The external camera is attached to any location on the vehicle 300. When capturing an image of the front, the external camera is attached to the top of the front windshield, the back of the rearview mirror, etc. The external camera, for example, periodically and repeatedly captures images of the periphery of the vehicle 300 (including the direction of travel). The external camera may be a stereo camera, a monocular camera, a fisheye camera, etc.

[0037] The radar device emits radio waves such as millimeter waves around the vehicle 300 and detects radio waves reflected by the object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device may be attached to any location on the vehicle 300. The vehicle 300 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method. The LIDAR irradiates the area around the vehicle 300 with light (or electromagnetic waves with a wavelength close to light) and measures scattered light. The LIDAR detects the distance from the vehicle 300 to an object based on the time between light emission and light reception. The LIDAR may be attached to any location on the vehicle 300. The external environment detection device 302 outputs information indicating the detection results (images, object positions, etc.) to the control device 360.

[0038] The vehicle sensors 304 include, for example, a speed sensor, an acceleration sensor, a yaw rate (angular velocity) sensor, a direction sensor, and an operation amount detection sensor attached to the operation element 306 .

[0039] Operator 306 accepts driving operations by a passenger of vehicle 300. Operator 306 includes, for example, an operator for instructing acceleration / deceleration (e.g., an accelerator pedal, a brake pedal, a dial switch, a lever) and an operator for instructing steering (e.g., a steering wheel). In this case, vehicle sensor 304 may include an accelerator opening sensor, a brake depression sensor, a steering torque sensor, etc. Vehicle 300 may also be provided with an operator of a type other than those described above as operator 306 (e.g., a non-annular rotary operator, a joystick, a button, etc.).

[0040] Internal camera 308 captures an image of at least the head of an occupant of vehicle 300 from the front. Internal camera 308 is a digital camera that uses an imaging element such as a CCD or CMOS. Internal camera 308 outputs the captured image to control device 360.

[0041] The positioning device 310 is a device that measures the position of the vehicle 300. The positioning device 310 is, for example, a Global Navigation Satellite System (GNSS) receiver, and identifies the position of the vehicle 300 based on signals received from GNSS satellites and outputs the position information. Note that the position information of the vehicle 300 may be estimated from the position of a Wi-Fi base station to which the communication device 312 is connected. The positioning device 310 may be included in the vehicle sensor 304.

[0042] The communication device 312 communicates with other vehicles in the vicinity using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various external devices (for example, the area monitoring server 400, the sensor device 100, and the information providing device 200) via a wireless base station.

[0043] The HMI 314 presents (or notifies or notifies) various information to the occupants of the vehicle 300 and accepts input operations by the occupants. The HMI 314 includes various display devices, speakers, microphones, buzzers, touch panels, switches, keys, lamps, etc. The HMI 314 is an example of an "internal notification device." For example, the HMI 314 notifies the occupants of the driving state of the vehicle 300 controlled by the control device 360 ​​in different notification modes depending on the driving state. The HMI 314 also presents, for example, information from the control device 360 ​​and information obtained from an external device via the communication device 312.

[0044] The locomotion mechanism 320 is a mechanism for moving the vehicle 300 on a road. The locomotion mechanism 320 is, for example, a group of wheels including steering wheels and drive wheels. The locomotion mechanism 320 may also be legs for multi-legged walking.

[0045] The drive device 330 outputs force to the movement mechanism 320 to move the vehicle 300. For example, the drive device 330 includes a motor that drives the drive wheels, a battery that stores power to be supplied to the motor, a steering device that adjusts the steering angle of the steering wheels, and the like. The drive device 330 may also include an internal combustion engine, a fuel cell, or the like as a drive force output means or a power generation means. The drive device 330 may also include a brake device that utilizes frictional force or air resistance. The drive device 330 may control the traveling of the vehicle 300 based on information related to steering control or speed control from the area monitoring server 400 instead of (or in addition to) the operation content (manual driving operation) of the operator 306.

[0046] The external notification device 340 is, for example, a lamp, a display device, a speaker, or the like, provided on an outer panel portion of the vehicle 300, for notifying information to the outside of the vehicle 300. The external notification device 340, for example, notifies the surroundings of the moving body (within a predetermined distance from the vehicle 300) of the running state of the vehicle 300 controlled by the control device 360 ​​in different notification modes depending on the running state.

[0047] 5 is a perspective view of vehicle 300 seen from above. In the figure, FW is the steering wheel, RW is the driving wheel, SD is the steering device, MT is the motor, and BT is the battery. The steering device SD, motor MT, and battery BT are included in the driving device 330. Also, AP is the accelerator pedal, BP is the brake pedal, WH is the steering wheel, SP is a speaker, and MC is a microphone. The vehicle 300 shown in the figure is a one-seater four-wheel vehicle, and an occupant P is seated in the driver's seat DS and wearing a seat belt SB. Arrow α1 indicates the traveling direction (velocity vector) of vehicle 300.

[0048] External environment detection device 302 is provided near the front end of vehicle 300, and internal camera 308 is provided at a position where it can capture an image of the head of occupant P from in front of occupant P. An external notification device 340 serving as a display device is also provided near the front end of vehicle 300. An HMI 314 serving as a display device is also provided in front of occupant P inside the vehicle. External notification device 340 may be formed integrally with speaker SP, and HMI 314 may be formed integrally with speaker SP and microphone MC.

[0049] Returning to FIG. 4 , the storage device 350 is a non-transitory storage device such as a hard disk drive (HDD), flash memory, or random access memory (RAM). The storage device 350 stores map information 352, a program 354 executed by the control device 360, and the like. The map information 352 stores, for example, road information (road shape (width, curvature, gradient), location of stop lines and crosswalks) associated with location information, point of interest (POI) information, traffic regulation information, address information (address and postal code), facility information, telephone number information, and the like. The location information includes latitude and longitude. Although the storage device 350 is illustrated outside the frame of the control device 360 ​​in the figure, the storage device 350 may be included in the control device 360.

[0050] The control device 360 ​​includes, for example, an object recognition unit 362 and a control unit 364. The object recognition unit 362 and the control unit 364 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software) 354. Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in the storage device 350 in advance, or may be stored on a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed in the storage device 350 by inserting the storage medium into a drive device.

[0051] The object recognition unit 362 recognizes the surroundings of the vehicle 300 based on the output of the external environment detection device 302. For example, the object recognition unit 362 recognizes objects present within a predetermined distance from the vehicle 300. The objects include some or all of the following: moving objects such as vehicles, bicycles, and pedestrians; road boundaries such as road markings, steps, guardrails, road shoulders, and medians; road structures such as road signs and billboards; and obstacles such as fallen objects present (or fallen) on the road. The object recognition unit 362 acquires information such as the presence, position, and type of other moving objects by inputting images captured by the external camera of the external environment detection device 302 into a trained model that has been trained to output information such as the presence, position, and type of an object when an image captured by the external camera of the external environment detection device 302 is input. The type of other moving objects can also be estimated based on the size in the image, the intensity of reflected waves received by the radar device of the external environment detection device 302, and the like. Furthermore, the object recognition unit 362 may recognize the speed of another moving object detected by a radar device using Doppler shift or the like, for example.

[0052] The object recognition unit 362 also recognizes lane markings that demarcate the road on which the vehicle 300 is traveling. For example, the object recognition unit 362 recognizes lane markings by analyzing an image captured by an external camera of the external environment detection device 302. Note that the recognition of lane markings may be supplemented by the output of a radar device, a LIDAR device, a sensor fusion device, or the like.

[0053] In addition, the object recognition unit 362 may compare the position information of the vehicle 300 from the positioning device 310 with the map information 352 to recognize whether the vehicle 300 is traveling on a road (whether it has deviated from the road).

[0054] The control unit 364 controls the overall components of the vehicle 300. For example, based on information recognized by the object recognition unit 362, the control unit 364 causes the HMI 314 to output information for notifying the occupants and causes the external alarm device 340 to output information. The control unit 364 also causes the communication unit 312 to transmit vehicle data including information acquired by the external environment detection device 302, the surrounding conditions recognized by the object recognition unit 362, position information of the vehicle 300 measured by the positioning device 310, and date and time information (acquisition date and time, recognition date and time, positioning date and time) to the area monitoring server 400. Based on information provided from the area monitoring server 400, the control unit 364 also causes the HMI 314 to output information, causes the external alarm device 340 to output information, and causes the drive unit 330 to perform driving control.

[0055] [Area monitoring server] FIG. 6 is a diagram illustrating an example of the configuration of the area monitoring server 400. The area monitoring server 400 includes, for example, a server-side communication unit 410, an acquisition unit 420, an analysis unit 430, a management unit 440, a provision unit 450, and a server-side storage unit 460. The acquisition unit 420, the analysis unit 430, the management unit 440, and the provision unit 450 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in a storage device in advance, or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM, and installed in the storage device by inserting the storage medium into a drive device.

[0056] The server-side storage unit 460 may be realized by the various storage devices described above, or a solid-state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a RAM, etc. The server-side storage unit 460 stores, for example, a monitoring information database (DB) 462, infrastructure facility information 464, map information 466, programs, and various other information.

[0057] The monitoring information DB 462 stores the contents of the sensor data transmitted from the sensor device 100, the vehicle data transmitted from the vehicle 300, and the like. In the infrastructure information 464, for example, the sensor ID of the sensor device 100 or the identification information (information providing device ID) of the information providing device 200 is associated with the installation location, the installation orientation (angle of view (photography range), display direction), and type information. For example, in the case of the sensor device 100, the type information includes identification information for identifying a stereo camera, a monocular camera, a fisheye camera, an infrared camera, a TOF camera, a radar device, a LIDAR, a microphone, and the like. In addition, in the case of the information providing device 200, the type information includes identification information for identifying a display, a speaker, a light-emitting unit, and the like. The map information 466 includes information similar to that of the map information 352. The map information 466 may be updated as needed by the server-side communication unit 410 communicating with an external device.

[0058] The server-side communication unit 410 communicates with the sensor device 100, the information providing device 200, the vehicle 300, and other external devices via the network NW.

[0059] The acquisition unit 420 acquires information from the sensor device 100, the vehicle 300, and other external devices. For example, the acquisition unit 420 acquires sensor data from the sensor device 100 or vehicle data from the vehicle 300, and stores the acquired data in the monitoring information DB 472.

[0060] The analysis unit 430 analyzes the sensor data and vehicle data acquired by the acquisition unit 420. For example, when the sensor data and vehicle data include image data, the analysis unit 430 analyzes the image data. For example, the analysis unit 430 converts the image data into a bird's-eye view coordinate system, and performs image analysis processing using well-known methods (such as binarization processing, contour extraction processing, image enhancement processing, feature extraction processing, and pattern matching processing) based on the converted coordinate system to recognize the surrounding situation of the sensor device 100. The analysis unit 430 may also perform the above-described image analysis processing without performing coordinate conversion.

[0061] For example, the analysis unit 430 identifies an object included in the image data based on a matching result (degree of match) between feature information obtained by analyzing the image data and feature information predetermined for each object. The degree of match may be derived, for example, based on how many of the multiple feature elements included in the feature information match, or based on the similarity of each feature element or the entire feature information. The degree of match may also be derived, for example, based on the sum of the differences for each element. The degree of match may also be derived from the two pieces of information being compared using AI (artificial intelligence) functions such as machine learning (neural networks) and deep learning, or may be derived using other techniques.

[0062] The analysis unit 430 may also recognize the position, type, speed, etc. of objects present in the monitored area included in the image data. Examples of the objects include moving bodies such as the vehicle 300, pedestrians, and bicycles. The objects may also include road structures, etc. Examples of road structures include road signs, traffic signals, curbs, medians, guardrails, fences, walls, railroad crossings, crosswalks drawn on the road surface, and stop lines. The objects may also include obstacles that impede (or are likely to impede) the travel of the vehicle 300.

[0063] The analysis unit 430 also recognizes the position (relative position) of a moving object present in the monitored area, and recognizes the state of the moving object, such as its speed, acceleration, and moving direction, using time-series image data. The position of the object is recognized as a position in an absolute coordinate system with a representative point of the moving object (such as the center of gravity or the center of a drive shaft) as the origin. The "state" of the object may include, for example, the acceleration or jerk of the moving object, or its "behavioral state" (for example, whether or not the moving object is crossing a pedestrian crossing or about to cross a pedestrian crossing). The analysis unit 430 may also recognize the object using a process similar to the object recognition process performed by the object recognition unit 362.

[0064] The analysis unit 430 may also analyze characteristic information such as shape to identify a moving object included in the monitoring area. The characteristic information may include, in addition to shape, a pattern (including externally visible symbols, numbers, etc.), a color, etc. The analysis unit 430 may also measure the travel time of the moving object in a predetermined section.

[0065] The analysis unit 430 may also analyze the error between the road position included in the image data and the road position included in reference image data (reference image) captured in advance by the same sensor device 100. The reference image is an image captured using the same sensor device 100 at the correct position, direction, and angle of view (capture range). For example, the analysis unit 430 derives the degree of deviation between the road conditions based on the image data and the road conditions based on a predetermined reference image. The degree of deviation is an index value indicating the magnitude of the deviation, and the greater the deviation, the greater the degree. For example, the analysis unit 430 binarizes the captured image data and the reference image data, and analyzes the degree of deviation from the amount of deviation (deviation amount) of the distance, direction, etc. of the road dividing lines obtained from the binarization results.

[0066] Furthermore, the analysis unit 430 may extract an image area to be used as an analysis target from the entire image area of ​​the image data based on the season, weather, or time period when the image data was acquired, and analyze the amount of deviation using an image of the extracted area (extracted image). This makes it possible to prevent erroneous recognition of objects included in the image due to the movement of shadows caused by buildings, trees, etc. around the area to be monitored.

[0067] Furthermore, when a mobile object moves outside the sensor ranges of multiple sensor devices in the monitored area, the analysis unit 430 may monitor the situation in the monitored area using vehicle data obtained by sensors (external environment detection devices 302) mounted on other mobile objects. Furthermore, the analysis unit 430 may recognize at least one of information related to the movement of the mobile object from the sensor data and reflection intensity according to the attributes of the mobile object. Information related to the movement of the mobile object includes various information that changes with movement (from which the amount of movement can be derived), such as the position deviation and speed distribution of the mobile object. The position deviation of the mobile object is, for example, the deviation of the position in the lateral direction (road width direction) relative to the extension direction (longitudinal direction) of the road. Deviations include information such as being to the right of the center of the lane and the amount of change in lateral position over a predetermined period of time. The speed distribution is the time-series speed pattern of the mobile object (e.g., gradually decelerating or accelerating, or traveling at a constant speed). The attributes of the mobile object include, for example, the type of vehicle, pedestrian, bicycle, etc.

[0068] The management unit 440 manages the status of the monitored area based on the analysis results by the analysis unit 430, etc. For example, the management unit 440 manages whether the vehicle 300 traveling on a road included in the monitored area has a tendency to deviate from the road, whether the vehicle 300 is slipping (sliding), whether there is a possibility of contact with another object, etc. Furthermore, the management unit 440 may determine that there is an abnormality in the vehicle 300 (e.g., poor physical condition of a passenger, vehicle breakdown) or an abnormality in the monitored area (e.g., frozen road surface) when the difference between the traveling time of the vehicle 300 for a predetermined section measured by the analysis unit 430 and a predetermined reference traveling time is equal to or greater than a threshold value.

[0069] The management unit 440 may also include, for example, a determination unit 442. Based on the detection results of the multiple sensor devices 100 installed near the monitored area, the determination unit 442 determines whether the moving objects detected by each of the multiple sensor devices 100 are the same object. If the management unit 440 determines that the moving objects are the same object, it integrates the analysis results of each sensor to track the behavior of the moving objects. If the management unit 440 determines that the moving objects are not the same object, it tracks the behavior of each moving object individually. The determination unit 442 may also determine whether the degree of deviation from the reference image analyzed by the analysis unit 430 is equal to or greater than a threshold. If the management unit 440 determines that the degree of deviation is equal to or greater than a threshold, it determines that at least one of the surrounding monitoring area or the sensor device 100 needs to be maintained. For example, the management unit 440 determines that the sensor device 100 needs to be maintained if the error between the position of a road included in the image captured by the sensor device 100 and the position of a road included in the reference image is equal to or greater than a threshold.

[0070] The providing unit 450 provides information to moving objects (vehicles 300, pedestrians, and other traffic participants) moving through the area to be monitored via the vehicle 300 traveling through the area to be monitored, the sensor device 100 installed near the area, and the information providing device 200. In this case, the provided information includes, for example, instructions to the moving objects to take action (for example, slow down, stop, etc.). The providing unit 450 may also provide information to a manager (for example, a maintenance officer) who manages the area to be monitored. In this case, the provided information includes the location and content of maintenance (for example, adjusting the installation position of the sensor device, cleaning the area to be monitored).

[0071] [Examples of area monitoring] Next, specific examples of area monitoring in the area monitoring system 1 will be explained using several examples.

[0072] <First monitoring example> FIG. 7 is a diagram illustrating a first monitoring example. In the first monitoring example, for example, information obtained from multiple sensor devices 100 installed on a road (an example of a monitored area) is integrated to comprehensively monitor the conditions on the road. The example in FIG. 7 shows vehicles 300-1 and 300-2 traveling on a lane L1 defined by road dividing lines LR and LL, and sensor devices 100-1 and 100-2 installed near lane L1. Sensor device 100-1 is an example of a "first sensor device," and sensor device 100-2 is an example of a "second sensor device." Vehicle 300-1 is traveling in the direction of lane L1 at a speed V1, and vehicle 300-2 is traveling behind vehicle 300-1 at a speed V2 in the same direction as vehicle 300-1. The sensor devices 100-1 and 100-2 are installed at a predetermined distance from each other and capture an image of an area including at least a part of the road (lane L1) with fixed angles of view (imaging ranges) AR1 and AR2, respectively. Sensor data including the captured camera images is transmitted to the area monitoring server 400 by the communication unit 110. The installation positions, directions, and angles of view (information related to the reference image) of the sensor devices 100-1 and 100-2 are registered in advance in the infrastructure information 464.

[0073] The acquisition unit 420 of the area monitoring server 400 acquires sensor data transmitted from the sensor devices 100-1 and 100-2. The acquired sensor data may be stored in the monitoring information DB 462. The analysis unit 430 analyzes the sensor data to recognize the positions of the vehicles 300-1 and 300-2 and the positions of the road dividing lines LR and LL. The analysis unit 430 also analyzes the speed of the vehicles 300-1 and 300-2 from the amount of movement (amount of change in position) over a predetermined time period obtained from the time-series sensor data. The analysis unit 430 also identifies vehicles included in the sensor data acquired from the multiple sensor devices 100 based on the degree of match of characteristic information such as the shape of the vehicle 300 (e.g., the roof portion of the vehicle 300). For example, the analysis unit 430 identifies vehicles whose degree of match is equal to or greater than a threshold as the same vehicle (same object), assigns common identification information to the vehicles, and assigns unique identification information to vehicles whose degree of match is less than the threshold. This allows the management unit 440 to manage the movement status of each vehicle based on the assigned identification information. The analysis unit 430 may also perform similar analysis processing on moving objects other than the vehicle 300 that exist in the monitoring area.

[0074] Based on the analysis results, the management unit 440 tracks the driving conditions of the vehicle 300 and manages whether the vehicles 300-1 and 300-2 have a tendency to deviate from the lane L1, and whether abnormalities are occurring in the behavior of the vehicles 300-1 and 300-2 due to slippage, etc. For example, the determination unit 442 of the management unit 440 acquires the movement speed and movement direction of the vehicle 300 between predetermined points based on the analysis results, and determines that the vehicle 300 has a tendency to deviate from the lane L1 when it determines that there is a possibility that the vehicle 300 will cross the road dividing lines LR and LL within a predetermined time based on the positional relationship between the acquired movement speed and movement direction and the road dividing lines LR and LL.

[0075] In addition, the management unit 440 may determine that the vehicle 300 is slipping if the amount of change in the angle formed by the extension direction of the lane L1 (road) and the traveling direction of the vehicle 300 over a predetermined period of time (the rate of change in the yaw angle (yaw rate (angular velocity)) with respect to the traveling direction within a predetermined period of time) is greater than or equal to a threshold value.

[0076] Here, the multiple sensor devices 100 installed in the monitored area can be installed consecutively at intervals so that their sensor ranges (e.g., the angles of view of the camera devices) overlap, thereby enabling more accurate tracking of the same vehicle or the same person. However, in reality, due to factors such as road shape and equipment costs, the sensor ranges may not overlap, resulting in blind spots in the monitored area. Therefore, in a first monitoring example, when the sensor ranges do not overlap, if a passing vehicle is outside the sensor range of one of the multiple sensor devices 100 and is estimated to be heading toward the sensor range of the other sensor device 100, information on the section where the sensor ranges do not overlap is obtained from the detection results of the external environment detection device 302 mounted on the vehicle 300.

[0077] 7, the behavior of the vehicle 300-1 outside the sensor ranges (angles of view AR1 and AR2) of the sensor devices 100-1 and 100-2 is tracked using image data captured by the external environment detection device 302 mounted on the vehicle 300-2. This makes it possible to recognize lane departure, skidding, and the like of the vehicle 300-1 in blind spot areas not included in the angles of view AR1 and AR2.

[0078] In the first monitoring example, even if the sensor ranges of multiple sensor devices 100 overlap, analysis can be performed using images extracted from the entire image area of ​​the image data based on the season, weather, or time of day, and tracking can be performed using the analysis results of the image data captured by the external environment detection device 302 mounted on the vehicle 300-2 even if a blind spot area occurs.

[0079] Thus, according to the first monitoring example, the blind spot area of ​​the sensor data from the sensor device 100 can be interpolated using the vehicle data acquired from the vehicle 300-2, thereby reducing the blind spot area and enabling the situation occurring in the monitored area to be monitored without any omissions.

[0080] <Second monitoring example> Fig. 8 is a diagram for explaining a second monitoring example. In the second monitoring example, the behavior of a vehicle 300 is monitored based on the analysis results of images captured by a plurality of sensor devices 100 installed in a monitoring target area and the shape of roads included in the monitoring target area. The example in Fig. 8 shows a T-junction road where lane L2 is connected so as to be perpendicular to lane L1.

[0081] In the second monitoring example, the sensor devices 100-1 and 100-2 are installed in positions where their sensor ranges (angles of view AR1 and AR2) do not overlap. The management unit 440 tracks the traveling status of the vehicle 300-1 based on the sensor devices 100-1 and 100-2. Here, when the vehicle 300 that was present within the angle of view AR1 of the sensor device 100-1 is estimated to move outside the angle of view AR1 and head in the direction of the angle of view AR2 of the sensor device 100-2, and the vehicle 300 is not recognized in the sensor data captured by the sensor device 100-2 within a predetermined time, the management unit 440 predicts that an abnormality may have occurred in the vehicle 300 or that the vehicle 300 has stopped. Furthermore, the management unit 440 refers to the map information 466 based on the position information of the sensor devices 100-1 and 100-2, acquires the road shape around the installation position of the sensor device 100, and predicts that the vehicle 300 may have proceeded into the lane L2 if a lane (branching lane) L2 connecting to the lane L1 exists between the sensor ranges of the sensor devices 100-1 and 100-2. Note that the lane L2 is an example of a "branching road."

[0082] Furthermore, if the vehicle 300 is not recognized in the sensor data captured by the sensor device 100-2 within a predetermined time period, and if no other vehicles are recognized (or the number of passing vehicles is less than a threshold), the management unit 440 may predict that an abnormality such as an accident may have occurred in the section between the angles of view AR1 and AR2. According to the first and second monitoring examples, the area to be monitored can be monitored more appropriately using multiple sensor devices 100 installed on the road.

[0083] <Third monitoring example> FIG. 9 is a diagram illustrating a third monitoring example. The third monitoring example monitors the installation status of the sensor device 100 installed near the monitored area. The example in FIG. 9 shows a vehicle 300 traveling in the X-axis direction on a road RD1 that is open to oncoming vehicles. For example, the orientation or sensor range of the sensor device 100 may deviate from the standard due to the influence of an earthquake, wind, rain, or the like. If this deviation is not detected early, there is a possibility that the object will not be recognized or that accurate area monitoring will be hindered due to erroneous recognition, etc. Therefore, in the third monitoring example, the management unit 440 tally up the positions (travel path positions) passed by the vehicles 300 traveling in the same direction (e.g., the X-axis direction) on the road RD1 in the monitored area, and monitors the installation status of the sensor device 100 based on the tallying results.

[0084] For example, the management unit 440 acquires the error (pixel deviation) W1 between the actually recognized roadway position obtained as a result of the aggregation and the predetermined roadway position for the sensor device 100 installed on the road, and if the acquired error W1 is equal to or greater than a threshold value, it determines that there is an abnormality in the state (installation state) of the sensor device 100.

[0085] In the example of Figure 9, the judgment is based on the lateral deviation W1 of road RD1, but instead (or in addition), the deviation amounts in the vertical, diagonal, rotational, etc. directions of road D1 may be obtained, and whether or not there is an abnormality in the state of sensor device 100 may be judged based on the obtained multidimensional deviation amounts.

[0086] In addition, if the management unit 440 determines that the total number of pixels recognized as a road area from the image data is below a threshold, it may predict that there may be fallen leaves or other obstacles on the road.

[0087] Note that the total number of pixels recognized as roads may change if shadows are predicted to be cast on roads within the monitored area due to the direction of sunlight, the surrounding environment (e.g., trees, buildings, etc.), or streetlights installed in the monitored area. Therefore, the management unit 440 may adjust the total number of pixels for each time period, or may set areas within the image where recognition processing is not performed, taking into account the position and range of shadows during each time period, and monitor the road conditions based on the total number of pixels of the road using images outside of those areas. Note that the management unit 440 may adjust the time period for each season, or may determine whether to perform the above-mentioned processing depending on the presence or absence of shadows due to weather.

[0088] In the third monitoring example, the management unit 440 may determine, based on the monitoring results, whether maintenance is required for at least one of the monitored area and the sensor device 100. For example, maintenance includes maintenance of roads included in the monitored area and maintenance of the sensor device 100 installed in the monitored area. Road maintenance includes, for example, cleaning up fallen leaves and garbage on the road, removing obstacles, draining puddles, etc. Maintenance of the sensor device 100 includes, for example, correcting (readjusting) a deviation in the installation direction of the sensor device 100, repairing or replacing the device itself, etc. Specific examples of monitoring road maintenance and monitoring sensor device 100 maintenance are described below.

[0089] <Road maintenance monitoring> In the case of monitoring road maintenance, for example, when a first deviation amount between the reference image and the camera image is equal to or greater than a predetermined amount, the management unit 440 determines that road maintenance is necessary in the camera image and causes the providing unit 450 to provide information prompting maintenance. The first deviation amount is, for example, a difference value in the pixel amount (number of bits) when the total pixel amount of a portion recognized as a road in the camera image is compared with the same portion in the reference image. For example, when a camera image of a road with fallen leaves or the like piled up on the road surface is binarized, the area with the fallen leaves may not be recognized as a road, and the pixel amount of the portion recognized as a road in the camera image may be smaller than that in the reference image. Furthermore, if there are puddles or the like on or around the road, the pixel amount may be larger than that in the reference image depending on the imaging environment. Therefore, the management unit 440 determines that road maintenance is necessary when the degree of deviation based on the first deviation amount is equal to or greater than a threshold (i.e., when the deviation is large).

[0090] <Monitoring the Maintenance of the Sensor Device 100> In the case of monitoring the maintenance of the sensor device 100 (e.g., a camera device), for example, the management unit 440 causes the providing unit 450 to provide information prompting the maintenance of the sensor device 100 when the error between the position of a road included in a reference image and the position of a road included in a camera image is equal to or greater than a threshold. For example, when a second deviation amount between the reference image and the camera image is equal to or greater than a predetermined amount, the management unit 440 determines that the camera device that captured the camera image needs to be maintained, and causes the providing unit 450 to provide information prompting the maintenance. The second deviation amount is a difference value between the output result obtained by multiplying and outputting the portions of the binarized reference image and the binarized camera image that are recognized (set) as roads, and the number of pixels (number of bits) recognized as roads in the binarized reference image. "Multiplying and outputting" means, for example, setting the parts (pixels) recognized as roads in each binarized image to "1" and the other parts (pixels) to "0," multiplying the values ​​(1 or 0) of pixels at the same position in each image, and then adding the results for all pixels to obtain a total value. The management unit 440 determines that maintenance of the sensor device 100 is necessary when the degree of deviation based on the second deviation amount is equal to or greater than a threshold (i.e., when the deviation is large). The above-mentioned first deviation amount and second deviation amount are analyzed by, for example, the analysis unit 430.

[0091] As described above, according to the third monitoring example, it is possible to detect abnormalities in the position of the sensor device 100 or abnormalities in the road at an early stage and notify a maintenance person or the like.

[0092] <Fourth monitoring example> Next, a fourth monitoring example will be described. In the fourth monitoring example, a monitoring target area is monitored by combining sensor data obtained from multiple sensor devices of different types (characteristics). FIG. 10 is a diagram showing an example of characteristic information for each sensor type. In the example of FIG. 10, the characteristic information for each sensor type shows evaluation results for position accuracy, speed control, detection range, situation awareness, and environmental resistance. In the example of FIG. 10, "◎" is the highest evaluation, followed by "◯" and then "△" in decreasing order. FIG. 10 may also include evaluation results for each sensor type in terms of cost.

[0093] For example, in a section of a road or other area over a predetermined distance, it may be necessary to use multiple sensors to detect objects within the monitored area. Because multiple sensor devices may not have the same characteristics (performance, functions), an appropriate combination is required depending on the monitored area. Therefore, in the fourth monitoring example, as shown in FIG. 10 , the characteristics of each sensor are determined based on predetermined sensor type characteristics, and the type and number of sensors optimal for the monitored area are installed to monitor the situation in the monitored area. For example, stereo cameras, TOF cameras, LIDAR, etc. are used in areas where more accurate monitoring of object positions is required, and radar devices are installed in areas where more accurate monitoring of object speeds is required. Furthermore, radio wave sensors may be installed in sections of roads within the monitored area where the curvature is less than a threshold, and optical sensors may be installed in sections where the curvature is greater than or equal to the threshold. The management unit 440 manages the type (combination) and number of sensor devices 100 to be installed near the monitored area based on characteristic information, road shapes, etc., as shown in FIG. 10 . The management unit 440 may also adjust the type and number of sensors to be installed depending on the cost of the sensor devices 100.

[0094] For example, if a radio wave sensor and an optical sensor are installed in a monitoring area, and a moving object (e.g., vehicle 300) is recognized by the optical sensor, and then the moving object moves out of the sensor range and a radio wave sensor is present at the moving destination, the determination unit 442 determines whether the objects detected by each sensor are the same object based on at least one piece of information about the movement of the moving object (position deviation, speed distribution) that can be acquired from each sensor data, and reflection intensity according to the attributes of the moving object. For example, the determination unit 442 selects at least one piece of information from the position deviation, speed distribution, and reflection intensity of the moving object depending on the sensor type of the first sensor device and the second sensor device, and determines whether the moving objects detected by each sensor are the same object.

[0095] In addition, the determination unit 442 may also use similar information to determine whether a moving object is the same object when the moving object is recognized by a radio wave sensor and then moves out of the detection range, and an optical sensor is present at the destination.

[0096] Furthermore, when the above-mentioned moving body is the vehicle 300 and there is an object (for example, another vehicle, a pedestrian, or a bicycle) approaching the vehicle 300 (in other words, when there is an object that may come into contact with the vehicle 300), the management unit 440 may issue an action instruction to the vehicle 300. The action instruction includes, for example, an instruction regarding one or both of speed control (deceleration or stopping) and steering control.

[0097] Furthermore, the management unit 440 may vary the content of the action instruction given to the vehicle 300, for example, depending on the impact of contact with an object approaching the vehicle 300. For example, if the approaching object is a traffic participant that is vulnerable to the vehicle 300, such as a person or a bicycle (a traffic participant that is highly susceptible to physical impact, such as injury), the action content is strengthened (a more drastic action is instructed) compared to when the approaching object is another vehicle. Specifically, when a person is approaching the vehicle 300 (when the person is within a predetermined distance), a stronger deceleration command is given than when the vehicle 300 is approaching. This can further improve safety.

[0098] When an information providing device 200 is set in a monitoring target area, the providing unit 450, under the control of the management unit 440, causes the information providing device 200 to output an image or sound indicating the content of the action instruction. As a result, the action instruction output from the information providing device 200 enables the occupant of the vehicle 300 to drive in a manner that avoids contact with the object. Note that, when a camera device (image sensor) is included in the multiple sensor devices 100, the management unit 440 may acquire information regarding the attributes of an object approaching the vehicle 300 that is included in a camera image of the camera device, and control the providing unit 450 to output the acquired information to the information providing device 200. As a result, the information providing device 200 installed near the approaching object can display not only the action instruction but also the attributes of the approaching object (person, vehicle, bicycle, etc.), thereby enabling the target object for which the action instruction has been given to be performed to be more accurately identified.

[0099] Furthermore, under the control of the management unit 440, the providing unit 450 may output to the vehicle 300 information for causing the vehicle 300 to execute one or both of speed control and steering control according to the content of the action instruction for the vehicle 300. This allows the driving control of the vehicle 300 to be executed without waiting for the driving operation of the occupant of the vehicle 300.

[0100] Thus, according to the fourth monitoring example, even if different types of sensor devices 100 are installed in the area to be monitored, the situation can be monitored more appropriately using the sensor data from each of them. Furthermore, according to the fourth monitoring example, lower cost sensor devices can be combined depending on the monitoring content, etc., thereby reducing equipment costs. Note that each of the above-described first to fourth monitoring examples may include some or all of the other monitoring examples.

[0101] [Information provided] Next, an example of information provision based on the monitoring results will be described with reference to the drawings. FIG. 11 is a diagram showing a first example of information provision. In the example of FIG. 11, for example, when it is determined based on the first monitoring example that the vehicle 300 is about to deviate from lane L1, information is provided to the occupants of the vehicle 300 to encourage them to move in a direction that will prevent the vehicle from deviating. In the example of FIG. 11, the sensor device 100 is installed on a curved road (a section with a curvature equal to or greater than a predetermined value) from which the vehicle is likely to deviate. In addition, an information provision device 200 is installed near lane L1.

[0102] The management unit 440 tracks the travel of the vehicle 300 based on the sensor data obtained from the sensor device 100. Here, if it is determined based on the tracking results that the traveling direction of the vehicle 300 (arrow A1 in the figure) is likely to deviate from the lane L1 (cross over the road dividing line LR), the provision unit 450 causes the information provision device 200 to display information (action instructions) for preventing the deviation. In the example of FIG. 10 , text information such as "Please steer left" is displayed on the display 220 of the information provision device 200 installed in a position visible to the occupants of the vehicle 300. The management unit 440 manages which information provision device 200 should display this text information.

[0103] Furthermore, the providing unit 450 may transmit information (action instructions) for causing the occupant of the vehicle 300 to perform a steering operation to the vehicle 300 via the sensor device 100 and display the information on the HMI 314, or may transmit control information for automatically controlling the steering of the vehicle M to the vehicle 300 and cause the drive device 330 to perform driving control for steering the vehicle 300 to the left. This allows the vehicle 300 to move in the direction of arrow A2 in the figure.

[0104] In addition, when the management unit 440 determines that the vehicle 300 is slipping, the provision unit 450 may display warning information such as "Beware of frozen roads" on the display 220 of the information provision device 200, or may transmit information (action instructions) to the vehicle 300 to cause the occupants of the vehicle 300 to perform deceleration operations (or automatic deceleration control).

[0105] In the example of Figure 11, information from the area monitoring server 400 is shown to be transmitted to the information providing device 200 or the vehicle 300 via the sensor device 100, but it may also be transmitted directly from the area monitoring server 400 to the information providing device 200 or the vehicle 300.

[0106] FIG. 12 is a diagram showing a second example of information provision. In the example of FIG. 12, a notification is given to avoid collision between moving objects moving in lane L1. In the example of FIG. 12, a vehicle 300, a pedestrian P1, and a bicycle P2 moving in lane L1 are shown. A sensor device 100 that captures an image of an area including lane L1 and an information provision device 200 are installed near lane L1 shown in FIG. 12. Arrows A3, A4, and A5 shown in FIG. 12 indicate the movement directions of the vehicle 300, pedestrian P1, and bicycle P2 acquired from time-series sensor data.

[0107] The management unit 440 of the area monitoring server 400 analyzes the sensor data detected by the sensor device 100, and determines from the analysis results whether there is a possibility that the vehicle 300 and either the pedestrian P1 or the bicycle P2 will come into contact with another object. If it is determined that there is a possibility of contact, the management unit 440 provides the sensor device 100, the information providing device 200, the vehicle 300, etc. with information (action instructions) to avoid contact.

[0108] For example, the providing unit 450 transmits to the information providing device 200 via the network NW information to be provided for displaying a text image such as "Be careful of contact" on the display 220 of the information providing device 200. This allows the occupant of the vehicle 300, the pedestrian P1, or the occupant of the bicycle P2 who sees the text displayed on the display 220 of the information providing device 200 to avoid contact early.

[0109] Furthermore, the providing unit 450 may transmit information to the vehicle 300 so as to cause the HMI 314 of the vehicle 300 to display the same information as that displayed on the display 220, or may transmit information notifying that a pedestrian P1 or a bicycle P2 is approaching. Furthermore, the providing unit 450 may transmit control information to the vehicle 300 to stop the vehicle 300, or may transmit instruction information to the vehicle 300 to cause the external alarm device 340 of the vehicle 300 to output a warning sound (horn) or the like to notify the pedestrian P1 or the rider of the bicycle P2 that the vehicle 300 is approaching. This makes it possible to provide more appropriate information according to the status of objects traveling on the road (lane L1), thereby further improving safety when traveling on lane L1.

[0110] Furthermore, when providing information in the third monitoring example, the management unit 440 may provide information requesting maintenance (environmental improvement) to a terminal device (not shown) of a maintenance person. In this case, the location of the sensor device 100 where the abnormality occurred, the location of the area to be monitored, and the type of abnormality (for example, movement of the sensor device 100) may be notified. This allows the maintenance person to more specifically understand the location and content of the maintenance work, and to more appropriately prepare for and perform the work.

[0111] Furthermore, if the information providing device 200 is provided with a light-emitting unit, the light-emitting unit may be controlled to emit light in accordance with the action instruction. For example, as shown in FIG. 11, when it is determined that the vehicle 300 is likely to deviate from the lane L1, the providing unit 450 turns on the light-emitting unit installed on the road dividing line LR in addition to (or instead of) providing the information described above. Also, as shown in FIG. 12, when moving objects are approaching each other, the providing unit 450 turns on the light-emitting unit installed on the road surface. Note that the providing unit 450 may switch between lighting and blinking, or adjust the color or light amount (intensity), depending on the distance between the vehicle 300 and the road dividing line LR or the degree of approach (relative distance) between the moving objects. This allows the moving objects to more appropriately understand the current situation.

[0112] [Processing Sequence] Fig. 13 is a sequence diagram showing an example of the flow of processing executed by the area monitoring system 1 of the embodiment. In the example of Fig. 13, processing using the sensor device 100, the vehicle 300, the area monitoring server 400, and the information providing device 200 will be described. In the example of Fig. 13, it is assumed that the sensor device 100 is a camera device, the vehicle 300 is a vehicle traveling within the angle of view (monitoring target area) captured by the sensor device 100, and the information providing device 200 is installed near the monitoring target area of ​​the sensor device 100.

[0113] 13, the sensor device 100 captures an image of the area to be monitored (step S100) and transmits sensor data including the captured camera image to the area monitoring server 400 (step S102). The vehicle 300 also detects the surrounding conditions of the area to be monitored using the external environment detection device 302 or the like (step S104) and transmits vehicle data including the detection results to the area monitoring server 400 (step S106).

[0114] The area monitoring server 400 receives the sensor data and vehicle data, analyzes image data and the like included in the received data (step S108), and manages the area to be monitored based on the analysis results and the like (step S110). Specifically, at least one of the first to fourth monitoring examples described above is executed. Thereafter, if the area monitoring server 400 determines that information provision for the area to be monitored is necessary, it generates information to be provided (step S112) and transmits the generated information to the information providing device 200 (step S114). The area monitoring server 400 also transmits the generated information to be provided to the vehicle 300 (step S116).

[0115] The information providing device 200 receives the information transmitted from the area monitoring server 400 and displays an image corresponding to the received information on the display 220 or the like (step S118). Note that instead of (or in addition to) displaying an image on the display 220, the information providing device 200 may output audio corresponding to the received information from the speaker 230. It may also light up (or blink) a light-emitting unit provided on the road in a manner corresponding to the received information.

[0116] The vehicle 300 receives the information transmitted from the area monitoring server 400, displays an image corresponding to the received information on the HMI 314, and executes driving control (e.g., speed control, steering control) corresponding to the received information (step S120). Note that instead of (or in addition to) the above-mentioned control, the vehicle 300 may output a sound corresponding to the received information to the external alarm device 340. This completes the processing of this sequence.

[0117] In addition, in the processing of step S112, if maintenance work is required in the monitored area, the area monitoring server 400 may generate information to be provided to the maintenance personnel and transmit the generated information to a terminal device owned by the maintenance personnel.

[0118] Next, the specific processing of steps S108 to S112 described above will be explained using a flowchart.

[0119] <First process> Fig. 14 is a flowchart showing an example of the first processing. In the example of Fig. 14, the sensor device 100 installed in the area to be monitored is a camera device, and an example of a moving body is a vehicle 300. The processing shown in Fig. 14 may be repeatedly executed at a predetermined cycle or at a predetermined timing. The same applies to the processing (second monitoring processing) shown in Fig. 15, which will be described later.

[0120] 14, the acquisition unit 420 acquires camera data from a camera device (step S200). Next, the analysis unit 430 analyzes the acquired camera data (step S202). The management unit 440 assigns identification information to each vehicle included in the camera data based on the shape of the vehicle obtained as a result of the analysis (step S204), and manages the traveling status of the vehicle 300 within the monitored area (step S206).

[0121] Next, the management unit 440 determines whether the vehicle 300 is deviating from the lane (road) on which the vehicle 300 is traveling or whether the vehicle 300 is skidding (step S208). If it is determined that the vehicle 300 is deviating from the lane or skidding, the provision unit 450 generates information to be provided to the vehicle 300 (step S210) and causes the information provision device 200 to output the generated information (step S212). Note that in the process of step S212, the information to be provided to the vehicle 300 may be transmitted to the vehicle 300. In this case, the information to be provided to the vehicle 300 includes information (images and audio) to be output from the HMI 314 or information for steering control and speed control. This ends the process of this flowchart. Also, if it is determined in the process of step S208 that the vehicle 300 is not deviating from the lane or skidding, the process of this flowchart also ends. Note that the process of step S208 described above may determine whether there is a possibility of contact between moving objects moving in the monitored area included in the camera data.

[0122] <Second process> FIG. 15 is a flowchart illustrating an example of the second process. In the example of FIG. 15, the acquisition unit 420 acquires camera data captured by a camera device installed in the monitored area (step S300). Next, the analysis unit 430 analyzes the acquired camera data (step S302). The management unit 440 derives the degree of deviation between the road conditions based on the camera data obtained as the analysis result and the road conditions based on a predetermined reference image (step S304). Next, the determination unit 442 determines whether the degree of deviation is equal to or greater than a threshold (step S306). If it is determined that the degree of deviation is equal to or greater than the threshold, the provision unit 450 generates information prompting maintenance of at least one of the monitored area or the camera device (an example of a sensor device) (step S308), and causes the generated information to be output to a terminal device of a maintenance worker, which is an example of the information provision device 200 (step S310). This completes the process of this flowchart. If it is determined in the process of step S306 that the degree of deviation is not equal to or greater than the threshold value, the process of this flowchart ends.

[0123] <Third Processing> FIG. 16 is a flowchart illustrating an example of the third process. In the example of FIG. 16, it is assumed that multiple sensor devices 100 installed in the monitoring area may include both radio wave sensors and optical sensors. The process illustrated in FIG. 16 may be repeatedly executed at a predetermined interval or timing. In the example of FIG. 16, the acquisition unit 420 acquires sensor data from the sensor device 100 (step S400). Next, the analysis unit 430 analyzes the acquired sensor data (step S402). Next, the determination unit 442 determines whether the sensor device 100 from which the sensor data was acquired includes sensor data from both radio wave sensors and optical sensors. If it is determined that the sensor data includes sensor data from both radio wave sensors and optical sensors, the determination unit 442 determines whether the objects are the same based on at least one of information on the position deviation, speed distribution, and reflection intensity of the moving object (step S406). Note that the position deviation and speed distribution of the moving object are merely examples, and other information related to the movement of the moving object may also be used. Furthermore, if it is determined that the sensor data does not include the radio wave sensor and the optical sensor (i.e., only one of the sensor data is present), the determination unit 442 determines whether the object is the same based on the similarity of the analysis results of each sensor (step S408). Next, the management unit 440 manages the movement state of the same object (step S410). This ends this flowchart. Note that in the third monitoring process, after step S410, the same object may be monitored for slippage, lane departure, etc., and information may be provided based on the monitoring results. Furthermore, it may be monitored whether or not the object comes into contact with another object, and information may be provided based on the monitoring results.

[0124] <Modification> At least a part of the configuration of the area monitoring server 400 described above may be provided in the sensor device 100, the information providing device 200, or the vehicle 300. For example, when the function of the analysis unit 430 is provided in the sensor device 100 or the vehicle 300, the results of analysis performed by each of them are transmitted to the area monitoring server 400.

[0125] Furthermore, at least a part of the configuration of the information providing device 200 in the embodiment may be provided in the sensor device 100, and at least a part of the configuration of the sensor device 100 may be provided in the information providing device 200.

[0126] According to the embodiment described above, the area monitoring system 1 includes a plurality of sensor devices 100 installed in a monitored area to detect moving objects moving within the monitored area, and a determination unit 442 that determines whether the moving objects detected by each of the sensor devices 100 are the same object based on the detection results of each of the sensor devices 100, where the plurality of sensor devices 100 includes at least a radio wave sensor and an optical sensor. When a moving object is detected by each of the radio wave sensor and the optical sensor, the determination unit 442 determines whether the moving objects detected by the radio wave sensor and the optical sensor are the same object based on at least one of information regarding the movement of the moving object and reflection intensity according to the attributes of the moving object, thereby enabling more appropriate monitoring of the monitored area. This can therefore contribute to the development of sustainable transportation systems.

[0127] For example, according to the embodiment, multiple sensor devices 100 installed in a monitored area can be coordinated to more accurately track moving objects and grasp the situation. Furthermore, according to the embodiment, an infrastructure-cooperative monitoring system can be provided by effectively utilizing infrastructure equipment such as cameras installed in a predetermined area. Furthermore, since the sensor devices fixed to the monitored area have a fixed sensor range (angle of view, etc.), they can accurately acquire, for example, the road shape and the position of road dividing lines included in captured images, thereby more accurately grasping road conditions and the movement status of moving objects moving on the road. Furthermore, according to the embodiment, maintenance can be performed more appropriately and quickly on the area monitoring system 1. This allows infrastructure equipment to operate appropriately. Furthermore, according to the embodiment, traffic management can be performed using radio sensors and optical sensors. Therefore, the optimal type and number of sensors can be installed for each monitored area, taking into account the characteristics of each sensor. Therefore, the situation of the entire area can be more appropriately grasped using infrastructure equipment.

[0128] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0129] 1 Area monitoring system 100 Sensor device 110, 210 Communications Department 120 Imaging unit 130, 240, 364 Control section 200 Information provision device 220 Display 230 Speaker 240 Control Unit 300 vehicles 302 External sensing device 304 Vehicle Sensor 306 Operator 308 Internal Camera 310 Positioning Device 312 Communication equipment 314 HMI 340 External alarm device 360 Control Device 400 Area Monitoring Server 410 Server-side communication unit 420 Acquisition Department 430 Analysis Department 440 Management Department 442 Judgment section 450 Providing Department 460 Server-side storage unit

Claims

1. a plurality of sensor devices installed in a monitoring area to detect moving objects moving in the monitoring area; a determination unit that determines whether the moving objects detected by each of the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, when the moving object is detected by each of the radio wave sensor and the optical sensor, the determination unit determines whether the moving objects detected by the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount related to the movement of the moving object and a reflection intensity according to an attribute of the moving object; the determination unit, when the moving object moves out of the sensor range after being detected by the optical sensor and the radio wave sensor is present at the moving destination, determines whether the moving object is the same object based on at least one of information of a position deviation of the moving object, a speed distribution, and a reflection intensity according to an attribute of the moving object; Area surveillance system.

2. a plurality of sensor devices installed in a monitoring area to detect moving objects moving in the monitoring area; a determination unit that determines whether the moving objects detected by each of the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, when the moving object is detected by each of the radio wave sensor and the optical sensor, the determination unit determines whether the moving objects detected by the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount related to the movement of the moving object and a reflection intensity according to an attribute of the moving object; when the moving object moves out of the sensor range after being detected by the radio wave sensor and the optical sensor is present at the moving destination, the determination unit determines whether the moving object is the same object based on at least one of information of a position deviation of the moving object, a speed distribution, and a reflection intensity according to an attribute of the moving object; Area surveillance system.

3. a plurality of sensor devices installed in a monitoring area to detect moving objects moving in the monitoring area; a determination unit that determines whether the moving objects detected by each of the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, when the moving object is detected by each of the radio wave sensor and the optical sensor, the determination unit determines whether the moving objects detected by the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount related to the movement of the moving object and a reflection intensity according to an attribute of the moving object; The radio wave sensor is installed in a section of the road included in the monitoring area where the curvature is less than a threshold, and the optical sensor is installed in a section of the road where the curvature is equal to or greater than the threshold. Area surveillance system.

4. A management unit that manages the status of the monitoring target area is further provided, the management unit issues an action instruction to the vehicle when the moving body is a vehicle and an object is approaching the vehicle; 4. The area monitoring system according to claim 1.

5. the management unit varies the content of the action instruction to the vehicle depending on the degree of influence on contact of an object approaching the vehicle. The area surveillance system of claim 4 .

6. the management unit, when an information providing device is set in the monitoring target area, causes the information providing device to output an image or a sound indicating the content of the action instruction; The area surveillance system of claim 4 .

7. When the plurality of sensor devices includes an image sensor, the management unit causes the information providing device to output information relating to attributes of an object approaching the vehicle that is included in an image captured by the image sensor.

7. The area surveillance system of claim 6.

8. the management unit outputs to the vehicle information for causing the vehicle to execute one or both of speed control and steering control in accordance with the content of the action instruction for the vehicle; The area surveillance system of claim 4 .

9. The computer a plurality of sensor devices are installed in a monitoring area, and the plurality of sensor devices detect moving objects moving in the monitoring area. Based on the detection results of the plurality of sensor devices, it is determined whether the moving objects detected by the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, When the moving object is detected by each of the radio wave sensor and the optical sensor, it is determined whether the moving objects detected by each of the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount relating to a change in the position of the moving object and a reflection intensity according to an attribute of the moving object; When the moving object moves out of the sensor range after being detected by the optical sensor and the radio wave sensor is present at the moving destination, it is determined whether the moving object is the same object or not based on at least one of information of the position deviation of the moving object, the speed distribution, and the reflection intensity according to the attribute of the moving object. Area monitoring methods.

10. The computer a plurality of sensor devices are installed in a monitoring area, and the plurality of sensor devices detect moving objects moving in the monitoring area. Based on the detection results of the plurality of sensor devices, it is determined whether the moving objects detected by the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, When the moving object is detected by each of the radio wave sensor and the optical sensor, it is determined whether the moving objects detected by each of the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount relating to a change in the position of the moving object and a reflection intensity according to an attribute of the moving object; When the moving object moves out of the sensor range after being detected by the radio wave sensor and the optical sensor is present at the moving destination, it is determined whether the moving object is the same object or not based on at least one of information of the position deviation of the moving object, the speed distribution, and the reflection intensity according to the attribute of the moving object. Area monitoring methods.

11. The computer a plurality of sensor devices are installed in a monitoring area, and the plurality of sensor devices detect moving objects moving in the monitoring area. Based on the detection results of the plurality of sensor devices, it is determined whether the moving objects detected by the plurality of sensor devices are the same object; the plurality of sensor devices include at least a radio wave sensor and an optical sensor, When the moving object is detected by each of the radio wave sensor and the optical sensor, it is determined whether the moving objects detected by each of the radio wave sensor and the optical sensor are the same object based on at least one of information of a feature amount relating to a change in the position of the moving object and a reflection intensity according to an attribute of the moving object; The radio wave sensor is installed in a section of the road included in the monitoring area where the curvature is less than a threshold, and the optical sensor is installed in a section of the road where the curvature is equal to or greater than the threshold. Area monitoring methods.

Images