On-site monitoring method and on-site monitoring device

Abstract

To provide a premises monitoring method and a premises monitoring apparatus for reliably monitoring a state of the premises of a facility.SOLUTION: A premises monitoring apparatus 1 for monitoring a state of the premises of a facility includes a virtual map generation unit 15 which generates virtual three-dimensional maps of the premises of the facility, with time, on the basis of information on the surroundings of a vehicle acquired by a sensor 2 arranged on the vehicle which can travel in the premises of the facility, and compares a standard virtual map indicating a standard state of the premises of the facility in the virtual three-dimensional maps with a first virtual map generated after the standard virtual map in chronological order in the virtual three-dimensional maps, to monitor the change in state of the premises of the facility.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a method for monitoring the inside of a facility and a device for monitoring the inside of a facility.

Background Art

[0002] In the data collection system disclosed in Patent Document 1 below, the telematics center includes a data collection condition determination unit, a data collection condition distribution unit, and a probe reception processing unit. The data collection condition determination unit determines data collection conditions including the collection frequency of probe data in a vehicle. The data collection condition distribution unit distributes the data collection conditions to the vehicle. The probe reception processing unit receives probe data transmitted from the vehicle and accumulates it in a storage device. Further, the central processing unit mounted on the vehicle includes a data acquisition unit, a collected data narrowing-down processing unit, and a probe data transmission unit. The data acquisition unit acquires data. The collected data narrowing-down processing unit extracts probe data for each collection frequency of the data collection conditions. The probe data transmission unit transmits the extracted probe data to the telematics center.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when a vehicle is running inside a site such as a logistics terminal, information around the vehicle can be obtained from sensors such as LiDAR or cameras provided on the vehicle. However, the information obtained from the sensors of the vehicle is limited to information indicating the state around the vehicle inside the facility. Therefore, when trying to monitor the state inside the facility, it was sometimes difficult to monitor areas other than the area around the vehicle inside the facility. That is, it was sometimes difficult to monitor the state inside the facility. [Means for solving the problem]

[0005] The on-site monitoring method relating to this disclosure is an on-site monitoring method for monitoring the state of a facility premises, which involves generating a virtual three-dimensional map of the facility premises over time based on information about the surroundings of a vehicle acquired by a sensor installed on the vehicle capable of traveling within the facility premises, and monitoring changes in the state of the facility premises by comparing a reference virtual map, which represents a standard state of the facility premises, with a first virtual map, which is generated later in the time series than the reference virtual map, among the virtual three-dimensional maps.

[0006] In the above-described on-site monitoring method, a first object belonging to a predetermined category is detected from the first virtual map, the first virtual map is displayed on a display unit capable of displaying the virtual three-dimensional map, and the first object may be highlighted during display.

[0007] In the above-described on-site monitoring method, it is determined whether or not there is a difference between the reference virtual map and the first virtual map, and if it is determined that there is a difference, the difference may be highlighted and displayed on a display unit capable of displaying the virtual three-dimensional map.

[0008] In the above-described on-site monitoring method, if it is determined that a difference exists, it is determined whether or not to highlight the difference on the display unit, and if it is determined that the difference should be highlighted, the difference may be highlighted on the display unit.

[0009] In the above-described on-site monitoring method, a first interference determination is performed to determine whether the adjusted first object and other objects in the first virtual map interfere with each other, in response to the adjustment of the position of the first object in the first virtual map within the first virtual map. If the first interference determination determines that the first object and the objects in the first virtual map interfere with each other, the occurrence of interference may be notified.

[0010] In the above-described on-site monitoring method, when a second object is placed in the first virtual map, a second interference determination is performed to determine whether the placed second object and other objects in the first virtual map interfere with each other. If the second interference determination determines that the second object and the objects in the first virtual map interfere with each other, the occurrence of interference may be notified.

[0011] In the above-described on-site monitoring method, the virtual three-dimensional map of the facility premises may be generated over time based on information about the surroundings of each of the multiple vehicles, which is acquired by sensors installed on each of the multiple vehicles that can travel within the facility premises.

[0012] In the above-described on-site monitoring method, the sensor may be a three-dimensional sensor capable of detecting the shape of objects present around the vehicle and their distance from the vehicle.

[0013] The on-site monitoring device according to this disclosure is an on-site monitoring device for monitoring the state of a facility premises, and includes a virtual map generation unit that generates a virtual three-dimensional map of the facility premises over time based on information about the surroundings of a vehicle acquired by a sensor installed on the vehicle capable of traveling within the facility premises, and monitors changes in the state of the facility premises by comparing a reference virtual map that shows a reference state of the facility premises from among the virtual three-dimensional maps, and a first virtual map that is generated later than the reference virtual map in chronological order from among the virtual three-dimensional maps. [Effects of the Invention]

[0014] According to this disclosure, the conditions within the facility premises can be monitored more reliably. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram showing an example of a vehicle that can travel within the facility premises monitored by the on-site monitoring device according to the embodiment. [Figure 2] This is a block diagram showing an example of the overall configuration of the on-site monitoring device according to this embodiment. [Figure 3] It is a schematic diagram showing a reference virtual map generated by a virtual map generation unit of an in-premises monitoring device according to an embodiment. [Figure 4] It is a schematic diagram showing a first virtual map generated by a virtual map generation unit of an in-premises monitoring device according to an embodiment. [Figure 5] It is a diagram for explaining an example of the operation of an object display unit of an in-premises monitoring device according to an embodiment, and is a schematic diagram showing an object highlighting display displayed on a first virtual map. [Figure 6] It is a diagram for explaining an example of the operation of a difference display unit of an in-premises monitoring device according to an embodiment, and is a schematic diagram showing a difference highlighting display displayed on a first virtual map. [Figure 7] It is a schematic diagram for explaining the position adjustment of a first object and the first interference determination in an in-premises monitoring device according to an embodiment. [Figure 8] It is a schematic diagram for explaining the position adjustment of a first object and the first interference determination in an in-premises monitoring device according to an embodiment. [Figure 9] It is a schematic diagram for explaining the arrangement of a second object and the second interference determination in an in-premises monitoring device according to an embodiment. [Figure 10] It is a schematic diagram for explaining the arrangement of a second object and the second interference determination in an in-premises monitoring device according to an embodiment. [Figure 11] It is a flowchart showing an example of the operation of an in-premises monitoring device according to an embodiment. [Figure 12] It is a flowchart showing an example of the process of difference highlighting determination of an in-premises monitoring device according to an embodiment. [Figure 13] It is a flowchart showing an example of the process when a predetermined operation is performed on a first virtual map in an in-premises monitoring device according to an embodiment.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, several exemplary embodiments will be described while referring to the drawings. For elements having the same function, the same reference numerals are assigned, and redundant descriptions are omitted.

[0017] According to the in-facility monitoring method and the in-facility monitoring device 1 according to the embodiment, the state of a predetermined facility interior can be monitored. The facility to be monitored is not particularly limited, and the facility interior may be, for example, within the site of a manufacturing plant that manufactures slabs or the like, or within the site of a cargo terminal where the loading of trucks is transferred. Within the facility interior, a road on which the vehicle V can travel is disposed at a predetermined position. Also, around the road, buildings, sidewalks, parking spaces for parking vehicles, luggage storage areas for placing luggage, parked vehicles, luggage, etc. may be set or disposed.

[0018] The vehicle V illustrated in FIG. 1 is a transport vehicle capable of transporting luggage. FR and RR in the figure indicate the front and rear in the longitudinal direction of the vehicle V, respectively. LH indicates the left side in the width direction of the vehicle V. UP and DN indicate the upper and lower sides in the vertical direction of the vehicle V, respectively.

[0019] The vehicle V is an autonomous vehicle equipped with an autonomous driving control function and can travel within the facility interior by autonomous driving. For example, the vehicle V may travel in a circular route along a plurality of predetermined routes. Note that the vehicle V may travel within the facility interior by driving operations such as steering and pedal operations by an occupant without using the autonomous driving control function.

[0020] The autonomous driving control function is a function that automatically controls the travel of the host vehicle based on the surrounding situation of the host vehicle or the state of the host vehicle, etc., without relying on driving operations such as steering and pedal operations by an occupant. When equipped with the autonomous driving control function, the vehicle V can travel by autonomous driving that automatically performs acceleration, deceleration, and steering without the intervention of an occupant's operation. Also, the autonomous driving may include a case where the vehicle V automatically performs acceleration, deceleration, and steering without the intervention of an occupant's operation in a predetermined driving environment. The predetermined driving environment may be, for example, within the site of a manufacturing plant or a cargo terminal.

[0021] Furthermore, vehicle V is not limited to transport vehicles, but may also be passenger cars, commercial vehicles such as vans, passenger buses, or cargo handling vehicles such as forklift trucks. Also, vehicle V is not limited to autonomous vehicles. For example, vehicle V may be a vehicle driven by the occupants through steering, pedal operation, and other driving operations.

[0022] Vehicle V is equipped with a sensor 2 and a communication device 3. Sensor 2 is a measuring device capable of measuring the shape and positional relationship of objects surrounding vehicle V over time and in three dimensions. In the illustrated example, sensor 2 comprises a LiDAR 2a (Light Detection and Ranging) and a stereo camera 2b. In the illustrated example, the LiDAR 2a is located on the top of the vehicle V's cabin, and the stereo camera 2b is located inside the cabin facing forward, but this is not limited to this configuration. The position, orientation, etc., of sensor 2 can be appropriately set according to the shape of vehicle V, the measurement direction, etc.

[0023] LiDAR2a emits pulsed laser light around the vehicle V and observes the reflected light of the laser at measurement points on objects surrounding the vehicle V, thereby acquiring data indicating the three-dimensional coordinates of the measurement points over time. In other words, LiDAR2a can acquire point cloud data indicating the three-dimensional coordinates of each of multiple measurement points on the surface of the object over time.

[0024] The stereo camera 2b includes two cameras that are facing the same direction, have parallel optical axes, and equal focal lengths. Based on the images captured by each of these two cameras, the stereo camera 2b can acquire the distance from the stereo camera 2b to feature points such as corners of objects surrounding the vehicle V over time. In other words, the stereo camera 2b can acquire the three-dimensional coordinate data of each of multiple feature points on objects surrounding the vehicle V over time.

[0025] The sensor 2 of the vehicle V is not particularly limited and can be any three-dimensional sensor capable of detecting the shape of objects around the vehicle V and their distance from the vehicle V in three dimensions. Sensor 2 may consist of only one of LiDAR 2a and stereo camera 2b. Sensor 2 may also be, for example, a monocular camera or millimeter-wave radar.

[0026] As vehicle V travels within the facility grounds, sensor 2 measures the area around vehicle V, thereby acquiring information about the distance, position, shape, color, etc., of objects in the area surrounding the vehicle V's travel path within the facility grounds. The data regarding the area around vehicle V acquired by sensor 2 is transmitted wirelessly via communication device 3 to the facility monitoring device 1, which will be described later. The data transmission interval does not have to be constant and may be set appropriately according to, for example, processing load, communication load, etc. The data regarding the area around vehicle V may also be temporarily stored in a recording medium (not shown) such as memory installed in vehicle V and transmitted to the facility monitoring device 1 at a predetermined transmission interval.

[0027] Furthermore, multiple vehicles V, each equipped with a sensor 2, may travel within the facility grounds. This allows information regarding the distance, position, shape, color, etc., of objects in the area surrounding each of the multiple vehicles V within the facility grounds to be acquired from each of the multiple vehicles V. The data acquired by each of the sensors 2 of the multiple vehicles V may be transmitted to the facility monitoring device 1 via wireless communication.

[0028] Next, the configuration of the on-site monitoring device 1 according to the embodiment will be described with reference to Figure 2. The on-site monitoring device 1 includes a controller 10 that performs processing necessary for monitoring the facility premises. In the illustrated example, the on-site monitoring device 1 is connected to a sensor 2 and an HMI 5 (Human Machine Interface). The on-site monitoring device 1 is installed, for example, in a control facility that monitors the conditions within the facility premises, but is not limited to this. For example, a vehicle V may be equipped with the on-site monitoring device 1.

[0029] First, let's explain HMI5. HMI5 is a device that can receive output from controller 10 or input to controller 10. For example, HMI5 is installed in a control facility that monitors the conditions within the facility grounds. If the on-site monitoring device 1 is mounted on a vehicle V, the on-site monitoring device 1 and the HMI5 in the control facility may be wirelessly connected. In that case, only the results of data processing necessary for monitoring the facility grounds can be transmitted from the on-site monitoring device 1 to the HMI5. Therefore, the amount of communication data can be further reduced.

[0030] The HMI5 includes a display unit 6, which is, for example, a touch panel display. The display 6 can display a virtual three-dimensional map 50, which will be described later. The HMI5 may also include an operation unit 7. The operation unit 7 is configured to allow predetermined operations to be performed on the virtual three-dimensional map 50 displayed on the display 6. The operation unit 7 may be a switch or the like set on the display surface of the display 6, which is composed of a touch panel display, or it may be a switch or the like arranged independently of the display 6.

[0031] For example, when the first virtual map 51b, described later, is displayed on the display 6, the user of the premises monitoring device 1 can virtually adjust the position of objects within the first virtual map 51b by operating the control unit 7. Furthermore, new objects can be virtually placed within the first virtual map 51b.

[0032] Next, the controller 10 will be described. The controller 10 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, input / output unit, etc. The microcomputer's memory has a computer program installed that includes default rules and instructions for monitoring the facility premises. By executing this computer program, the microcomputer can perform monitoring of the facility premises.

[0033] The controller 10 includes a virtual map generation unit 15.

[0034] The virtual map generation unit 15 can generate and output a virtual three-dimensional map 50 of the facility premises over time based on information about the vehicle V acquired by the sensor 2. The virtual three-dimensional map 50 consists of a three-dimensional virtual space corresponding to the state of the facility premises at each time T. The virtual map generation unit 15 can also display the virtual three-dimensional map 50 on the display 6 of the HMI 5. The display range, display angle, etc., of the virtual three-dimensional map 50 on the display 6 are not particularly limited and may be set as appropriate by, for example, the user of the facility monitoring device 1.

[0035] The three-dimensional virtual space is a space that reflects the structure of the facility grounds, as well as the position and shape of objects placed within the facility grounds, in both the horizontal and vertical directions. Therefore, the virtual three-dimensional map 50 includes virtual objects that virtually represent objects present within the facility grounds. The three-dimensional virtual space is constructed, for example, by applying known techniques to point cloud data obtained by LiDAR2a. As such known techniques, for example, SLAM (Simultaneous Localization and Mapping) may be applied. This makes it possible to generate a three-dimensional virtual space simultaneously with estimating the attitude and position of the vehicle V based on the point cloud data acquired by LiDAR2a. Alternatively, the three-dimensional virtual space may be generated by applying SLAM to the three-dimensional coordinate data of feature points acquired by the stereo camera 2b. Furthermore, the three-dimensional virtual space may be generated by using sensor fusion technology to combine measurement data acquired by LiDAR2a and measurement data acquired by the stereo camera 2b in parallel or complementary, and then applying SLAM.

[0036] Furthermore, the three-dimensional virtual space generated by complementaryly combining the measurement data acquired by LiDAR2a and the measurement data acquired by stereo camera2b may include color information. This allows the virtual map generation unit 15 to generate a virtual three-dimensional map 50 with color information over time. By displaying the virtual three-dimensional map 50 with color information on the display 6, users of the on-site monitoring device 1 can more accurately grasp the state of the facility premises.

[0037] The three-dimensional virtual space generated by the virtual map generation unit 15 is constructed, for example, by arranging voxels. A voxel is a cubic region having a predetermined volume in the three-dimensional virtual space, that is, a three-dimensional region or volume element. The size of the voxels in the three-dimensional virtual space is not particularly limited and can be set appropriately according to the measurement error of the sensor 2, etc. For example, the size of the voxels may be set appropriately so that the measurement error of the sensor 2 is not reflected in the three-dimensional virtual space.

[0038] The virtual map generation unit 15 may update the virtual three-dimensional map 50 at predetermined time intervals. The update interval may be the same as, for example, the data transmission interval of the sensor 2, or an integer multiple thereof. The update interval does not have to be a constant interval. The virtual map generation unit 15 may also generate the virtual three-dimensional map 50 over time by updating a part of the virtual three-dimensional map 50 based on information acquired by the sensor 2. This reduces the processing load when generating the virtual three-dimensional map 50.

[0039] The virtual map generation unit 15 generates at least a reference virtual map 51a and a first virtual map 51b as a virtual three-dimensional map 50. The reference virtual map 51a illustrated in Figure 3 is a virtual three-dimensional map 50 that shows the state of the facility premises at a certain time T0. The virtual objects included in the reference virtual map 51a, namely virtual roads 52a, virtual buildings 53a, virtual sidewalks 54a, virtual parked vehicles 55a1, virtual luggage 56a1, 56a2, 56a3, and virtual pedestrians 57a1, 57a2, correspond to the roads, buildings, sidewalks, parked vehicles, luggage, and pedestrians that exist within the facility premises at time T0, respectively. In addition, the virtual parked vehicle 55a1 is located within area 58a. Area 58a corresponds to the area within the facility premises that has been set up as a parking space. The virtual luggage 56a1, 56a2, 56a3 are located within area 59a. Area 59a corresponds to the area within the facility premises that has been set up as a luggage storage area.

[0040] At time T0, the facility premises are set to a reference state. That is, the reference virtual map 51a is a virtual three-dimensional map 50 that shows the reference state of the facility premises. The reference state of the facility premises refers to the state in which objects present within the facility premises are arranged in their predetermined locations. The user of the facility monitoring device 1 can arbitrarily set what state will be considered the reference state. For example, the reference state may be one in which buildings, vehicles, luggage, etc., present within the facility premises are arranged in their predetermined reference locations.

[0041] The first virtual map 51b illustrated in Figure 4 is a virtual three-dimensional map 50 showing the state of the facility premises at a certain time T1. Time T1 is a later time in the time series than time T0. That is, the first virtual map 51b is a virtual three-dimensional map 50 that was generated after the reference virtual map 51a in the time series. For example, time T1 may be the current time. That is, the first virtual map 51b may be a virtual three-dimensional map 50 showing the latest state of the facility premises. The virtual roads 52b, virtual buildings 53b, virtual sidewalks 54b, virtual parked vehicles 55b1, 55b2, virtual luggage 56b1, 56b2, and virtual pedestrians 57b1 included in the first virtual map 51b correspond to the roads, buildings, sidewalks, parked vehicles, luggage, and pedestrians that exist within the facility premises at time T1, respectively.

[0042] The virtual stationary vehicle 55b1 corresponds to the virtual stationary vehicle 55a1 in the reference virtual map 51a. The virtual cargo 56b1 and 56b2 correspond to the virtual cargo 56a1 and 56a2 in the reference virtual map 51a. The virtual pedestrian 57b1 corresponds to the virtual pedestrian 57a1 in the reference virtual map 51a. Region 58b corresponds to region 58a in the reference virtual map 51a. Region 59b corresponds to region 59a in the reference virtual map 51a. In the examples shown in Figures 4 to 10, for explanatory purposes, the virtual cargo 56a3 or virtual pedestrian 57a2 in the reference virtual map 51a is shown with a dashed line.

[0043] The first virtual map 51b newly includes a virtual stopped vehicle 55b2 that was not included in the reference virtual map 51a. This indicates that between time T0 and time T1, a vehicle corresponding to the virtual stopped vehicle 55b2 was parked in a parking space within the facility premises corresponding to area 58a. Also, the first virtual map 51b does not include the virtual luggage 56a3 that was included in the reference virtual map 51a. This indicates that between time T0 and time T1, luggage corresponding to the virtual luggage 56a3 that was located in the luggage storage area of ​​the manufacturing plant corresponding to area 59a was moved to another location. Furthermore, the first virtual map 51b does not include the virtual pedestrian 57a2 that was included in the reference virtual map 51a. This indicates that between time T0 and time T1, pedestrian corresponding to the virtual pedestrian 57a2 that was on the sidewalk of the manufacturing plant moved to another location. In this way, the premises monitoring device 1 monitors changes in the state of the facility premises by comparing the reference virtual map 51a and the first virtual map 51b.

[0044] When comparing the reference virtual map 51a and the first virtual map 51b, known methods may be used to associate objects included in the reference virtual map 51a with objects included in the first virtual map 51b. For example, the voxels constituting the three-dimensional virtual space of the reference virtual map 51a may be matched with point cloud data obtained from the LiDAR 2a mounted on a moving vehicle. Furthermore, the voxels constituting the reference virtual map 51a may be associated with voxels constituting the first virtual map 51b that can be associated with the voxels constituting the reference virtual map 51a. This allows for a more reliable comparison between the reference virtual map 51a and the first virtual map 51b.

[0045] Furthermore, if multiple vehicles V are traveling within the facility grounds, the virtual map generation unit 15 may generate a virtual three-dimensional map 50 of the facility grounds over time based on information about the surroundings of each of the multiple vehicles V, acquired by the sensors 2 of each of the multiple vehicles V. This allows the facility monitoring device 1 to generate a virtual three-dimensional map 50 that more accurately reflects the state of the facility grounds.

[0046] Note that the reference virtual map 51a is not limited to the virtual three-dimensional map 50 at time T0. The user of the premises monitoring device 1 may set as the reference virtual map 51a any virtual three-dimensional map 50 generated at a desired timing from among the virtual three-dimensional maps 50 generated over time by the virtual map generation unit 15.

[0047] Furthermore, the reference virtual map 51a may have additional displays 60 that show predetermined information. In the example shown in Figure 3, a display 60 indicating the location of a pedestrian crossing is set. The displays 60 may be added to the reference virtual map 51a by operation via the HMI 5's control panel 7, for example. Also, as illustrated in Figure 4, the on-site monitoring device 1 may be configured so that displays equivalent to the displays 60 are automatically added to virtual three-dimensional maps 50 that are generated later in chronological order than the reference virtual map 51a. This allows the user of the on-site monitoring device 1 to more accurately understand which location within the facility grounds corresponds to the area shown by the virtual three-dimensional map 50 displayed on the display 6.

[0048] As illustrated in Figure 2, the controller 10 may include an object detection unit 20 and an object display unit 25.

[0049] The object detection unit 20 detects a first object from the first virtual map 51b. The first object is an object belonging to a predetermined category from among the virtual objects contained in the first virtual map 51b. This category can be set arbitrarily, and may be set to, for example, "vehicle," "luggage," and "person." In that case, virtual stationary vehicles 55b1, 55b2, virtual luggage 56b1, 56b2, and virtual pedestrian 57b1 are detected as the first object from the first virtual map 51b (see Figure 4).

[0050] Furthermore, known methods can be applied when detecting the first object from the first virtual map 51b. For example, by applying object recognition using convolutional deep learning to the first virtual map 51b, the first object belonging to a predetermined category may be detected from within the first virtual map 51b. In this case, a group of voxels recognized as belonging to a predetermined category may be detected as the first object from a three-dimensional virtual space composed of voxels.

[0051] The object display unit 25 can perform object highlighting, which highlights the first object on the display 6 of the HMI 5. Object highlighting may be performed, for example, by displaying the area in which the first object exists within the first virtual map 51b in a more prominent color than other areas, or by displaying the outline of the first object in a more prominent color than other outlines. In the example shown in Figure 5, the virtual stationary vehicles 55b1, 55b2, virtual luggage 56b1, 56b2, and virtual pedestrian 57b1 are highlighted as the first objects. If no first object is detected, the object display unit 25 does not need to perform object highlighting.

[0052] As illustrated in Figure 2, the controller 10 may include a difference extraction unit 30, a difference determination unit 31, a difference emphasis determination unit 32, and a difference display unit 33.

[0053] The difference extraction unit 30 extracts the difference between the reference virtual map 51a and the first virtual map 51b. The difference is the portion of the three-dimensional virtual space that constitutes the first virtual map 51b that has changed from the three-dimensional virtual space that constitutes the reference virtual map 51a. Comparing the reference virtual map 51a (see Figure 3) and the first virtual map 51b (see Figure 4), the number of virtual stationary vehicles 55b2 increases, while the number of virtual cargo 56a3 and virtual pedestrians 57a2 decreases between time T0 and time T1. Therefore, these increased and decreased virtual stationary vehicles 55b2, virtual cargo 56a3, and virtual pedestrians 57a2 are extracted as differences. If there is no change between the reference virtual map 51a and the first virtual map 51b, the difference extraction unit 30 does not extract a difference.

[0054] Furthermore, the difference extraction unit 30 may store information about the extracted differences, such as the position of the difference in the three-dimensional virtual space, the category of the object to which the difference belongs, or the duration of the difference extraction. Note that difference extraction may also be performed by extracting voxels that have been added or removed in the three-dimensional virtual space. For example, differences may be extracted by comparing the voxels constituting the reference virtual map 51a with the voxels constituting the first virtual map 51b and identifying the added or removed voxels.

[0055] The difference determination unit 31 determines whether or not there is a difference between the reference virtual map 51a and the first virtual map 51b. Between the reference virtual map 51a and the first virtual map 51b, virtual stopped vehicles 55b2, etc., are extracted as differences. Therefore, the difference determination unit 31 determines that there is a difference between the reference virtual map 51a and the first virtual map 51b. If no difference is extracted between the reference virtual map 51a and the first virtual map 51b, the difference determination unit 31 determines that there is no difference. In this way, the difference determination unit 31 determines whether or not there is a difference between the reference virtual map 51a and the first virtual map 51b based on the difference extracted by the difference extraction unit 30.

[0056] The difference enhancement determination unit 32 performs a difference enhancement determination in response to the difference determination unit 31 determining that there is a difference between the reference virtual map 51a and the first virtual map 51b. The difference enhancement determination determines whether or not to display the difference with emphasis on the display 6 of the HMI 5. The difference enhancement determination may include at least one of the area determination, category determination, and duration determination described later. In the flowchart illustrated in Figure 12 described later, the difference extraction unit 30 applies area determination, category determination, and duration determination to the difference it extracts to determine whether or not to display the difference with emphasis on the display 6. If the difference extraction unit 30 has extracted multiple differences, the difference enhancement determination unit 32 can perform a difference enhancement determination for each of the multiple differences.

[0057] The difference display unit 33 performs difference highlighting display in accordance with the difference highlighting determination unit 32's determination to highlight the extracted difference. In difference highlighting display, the difference determined to be highlighted is displayed on the display 6 with emphasis. Difference highlighting display may, for example, display the area in which the difference exists within the first virtual map 51b in a more conspicuous color than other areas, or display the outline of the difference with a thicker line or in a more conspicuous color than other outlines.

[0058] For example, if the difference emphasis determination unit 32 determines that the virtual stopped vehicle 55b2 and the virtual cargo 56a3 should be emphasized, the difference display unit 33 performs difference emphasis display for the virtual stopped vehicle 55b2 and the virtual cargo 56a3. In the example shown in Figure 6, difference emphasis display is performed by displaying the outlines of the virtual stopped vehicle 55b2 and the virtual cargo 56a3 with thick lines. Since the virtual cargo 56a3 is the difference that decreased between time T0 and time T1, the outline of the area corresponding to the position of the virtual cargo 56a3 in the reference virtual map 51a is emphasized with a dashed line and displayed.

[0059] The difference display unit 33 may display the difference between the reference virtual map 51a and the first virtual map 51b on the display 6 without determining whether or not to highlight the difference on the display 6. In other words, the difference display unit 33 may highlight the difference on the display 6 in response to the difference determination unit 31 determining that a difference exists. In that case, for example, the virtual stationary vehicle 55b2, virtual luggage 56a3, and virtual pedestrian 57a2 will be displayed on the display 6 with emphasis. In that case, the controller 10 does not need to include the difference emphasis determination unit 32.

[0060] Next, we will explain the region determination, category determination, and duration determination, which are part of the difference emphasis determination performed by the difference emphasis determination unit 32.

[0061] In the region determination, it is determined whether the extracted difference is located in a predetermined region within the virtual three-dimensional map 50. If the difference is located in the predetermined region within the virtual three-dimensional map 50, it is determined that the difference should be highlighted on the display 6. If the difference is not located in the predetermined region within the virtual three-dimensional map 50, it is determined that the difference should not be highlighted on the display 6.

[0062] The predetermined area is not particularly limited, and any desired area can be set in advance. For example, the predetermined area may be set appropriately according to the layout of roads, buildings, sidewalks, parking spaces, luggage storage areas, etc., within the facility grounds. Alternatively, area determination may be performed by determining whether the position of the voxels constituting the extracted difference in the three-dimensional virtual space is located within the virtual three-dimensional map 50. In the first virtual map 51b illustrated in Figure 6, areas 58b and 59b are set as predetermined areas within the virtual three-dimensional map 50. The virtual stopped vehicle 55b2 is located within area 58b, and the virtual luggage 56a3 is located within area 59b. Therefore, it is determined that the virtual stopped vehicle 55b2 and the virtual luggage 56a3 are located within the predetermined areas within the virtual three-dimensional map 50. That is, the difference emphasis determination unit 32 determines to emphasize and display the virtual stopped vehicle 55b2 and the virtual luggage 56a3 on the display 6.

[0063] By performing difference highlighting determination based on area determination, it is possible to determine whether or not to highlight the extracted difference on the display 6 based on its location. Therefore, users of the on-site monitoring device 1 can more easily monitor whether or not a difference has occurred in an area they particularly want to monitor.

[0064] In category determination, it is determined whether the extracted difference belongs to a predetermined category. If the difference belongs to the predetermined category, it is determined to highlight the difference on the display 6. If the difference does not belong to the predetermined category, it is determined not to highlight the difference on the display 6. The predetermined category is the classification of objects to which objects that are predetermined to be highlighted on the display 6 belong. For example, if the extracted difference is in the category "vehicle" or the category "luggage" and you want to highlight that difference on the display 6, you set "vehicle" and "luggage" as the predetermined categories in advance. In that case, of the virtual stationary vehicle 55b2, virtual luggage 56a3, and virtual pedestrian 57a2 extracted as differences, the virtual stationary vehicle 55b2, which belongs to the category "vehicle," and the virtual luggage 56a3, which belongs to the category "luggage," are determined to belong to the predetermined category. That is, the difference highlighting determination unit 32 determines to highlight the virtual stationary vehicle 55b2 and the virtual luggage 56a3 on the display 6.

[0065] By performing difference highlighting determination based on category determination, it is possible to determine whether or not to highlight the extracted difference on the display 6 according to the category to which the difference belongs. Therefore, users of the on-site monitoring device 1 can more easily monitor whether or not a difference has occurred for a type of object they particularly want to monitor.

[0066] The duration determination determines whether the extracted difference has been continuously extracted for a predetermined time or longer. If the difference has been continuously extracted for a predetermined time or longer, it is determined that the difference should be highlighted on the display 6. If the difference has not been continuously extracted for a predetermined time or longer, it is determined that the difference should not be highlighted on the display 6. The value of the predetermined time is not particularly limited and can be set to any desired value in advance. For example, the predetermined time may be set appropriately according to the detection interval of the sensor 2, the transmission interval of the data detected by the sensor 2, the update interval of the virtual three-dimensional map 50 by the virtual map generation unit 15, etc. In one embodiment, the predetermined time may be set to 15 seconds to 90 seconds.

[0067] As vehicle V travels within the facility grounds and sensor 2 acquires information about its surroundings, objects around vehicle V are continuously detected within the detection range of sensor 2. For example, if a moving object such as a pedestrian within the facility grounds is temporarily within the detection range of sensor 2, a virtual moving object corresponding to that object will be included in the virtual three-dimensional map 50 generated based on the information from sensor 2 over time. This virtual moving object is then extracted over time by the difference extraction unit 30. By performing a difference highlighting determination based on the duration of the difference extraction, it is possible to determine whether or not to highlight the difference on the display 6 depending on the duration of the difference extraction. Therefore, for example, if a virtual moving object in the virtual three-dimensional map 50 corresponding to a moving object that was briefly within the detection range of sensor 2 is not continuously extracted for a shorter period than a predetermined time, that virtual moving object can be excluded from the difference highlighting display. In other words, only differences that have been continuously extracted for a predetermined time or longer can be included in the difference highlighting display.

[0068] As illustrated in Figure 2, the controller 10 may include an operation detection unit 40, an interference determination unit 41, and a notification unit 42.

[0069] The operation detection unit 40 detects a predetermined operation on the first virtual map 51b. The operation detection unit 40 may also detect an operation when there is input to the operation unit 7 of the HMI 5. For example, the operation detection unit 40 may detect input to the operation unit 7 set on the display surface of the display 6, which is composed of a touch panel display.

[0070] The operation detection unit 40 can detect when the position of the first object within the first virtual map 51b has been adjusted within the first virtual map 51b. In the example shown in Figure 7, the user of the premises monitoring device 1 performs an operation via the operation unit 7 to adjust the position of the first object, a virtual load 56b1, from position P1 to position P2. When such an operation is performed, the operation detection unit 40 can detect that the position of the first object has been adjusted within the first virtual map 51b. Note that the position adjustment of the first object is not limited to the horizontal direction, but can also be adjusted in the height direction.

[0071] Furthermore, the operation detection unit 40 can detect when a second object has been newly placed in the first virtual map 51b. The second object is an object that virtually represents an object newly placed within the facility premises. For example, the second object is an object that has not been detected within the facility premises at time T1, but virtually represents an object that is scheduled to be newly placed within the facility premises at a time later than time T1. Therefore, the object virtually represented by the second object is different from the object virtually represented by the first object. In other words, the second object virtually represents an object different from the first object. Also, the second object may be an object that virtually represents, for example, a vehicle, luggage, or a person, which has been pre-stored in the controller 10 by the user of the premises monitoring device 1.

[0072] In the example shown in Figure 9, the user of the premises monitoring device 1 performs an operation via the operation unit 7 to place a new virtual cargo 56c as a second object at position P2 in the first virtual map 51b. When such an operation is performed, the operation detection unit 40 can detect that a second object has been placed in the first virtual map 51b.

[0073] The interference determination unit 41 performs a first interference determination in response to the adjustment of the position of the first object within the first virtual map 51b. In the first interference determination, it is determined whether the first object, after its position has been adjusted, interferes with other virtual objects included in the first virtual map 51b.

[0074] In the example shown in Figure 7, at position P2 in the first virtual map 51b, the virtual cargo 56b1 after position adjustment does not interfere with other objects in the first virtual map 51b. Therefore, in the first interference determination, the interference determination unit 41 determines that the virtual cargo 56b1 after position adjustment does not interfere with other objects.

[0075] On the other hand, in the example shown in Figure 8, the user of the premises monitoring device 1 performs an operation via the operation unit 7 to adjust the position of the virtual cargo 56b1 in the first virtual map 51b from position P1 to position P3. Then, the left side of the virtual cargo 56b1 in the diagram, which has been adjusted to position P3, and the right side of the virtual cargo 56b2 in the diagram are interfering with each other. Therefore, in the first interference determination, the interference determination unit 41 determines that the virtual cargo 56b1 after position adjustment and an object other than the virtual cargo 56b1 are interfering with each other.

[0076] Furthermore, the interference determination unit 41 performs a second interference determination in response to the new placement of the second object within the first virtual map 51b. In the second interference determination, it is determined whether the placed second object interferes with any of the other virtual objects included in the first virtual map 51b.

[0077] In the example shown in Figure 9, at position P2 in the first virtual map 51b, the placed virtual cargo 56c does not interfere with other objects in the first virtual map 51b. Therefore, in the second interference determination, the interference determination unit 41 determines that the placed virtual cargo 56c and objects other than the virtual cargo 56c do not interfere with each other.

[0078] On the other hand, in the example shown in Figure 10, the user of the premises monitoring device 1 performs an operation via the operation unit 7 to place the virtual cargo 56c at position P4 within the first virtual map 51b. The left side of the virtual cargo 56c placed at position P4 interferes with the right side of the virtual cargo 56b1 in the diagram. Therefore, in the second interference determination, the interference determination unit 41 determines that the placed virtual cargo 56c and an object other than the virtual cargo 56c interfere with each other.

[0079] In the examples shown in Figures 7 to 10, the interference determination unit 41 determines the interference of objects in the horizontal direction, but it is not limited to this. The virtual three-dimensional map 50 generated by the virtual map generation unit 15 reflects the positions and shapes of objects placed within the facility in both the horizontal and vertical directions. Therefore, the interference determination unit 41 can determine whether or not objects interfere with each other in the vertical direction. For example, in the first interference determination, it is possible to determine whether or not the virtual cargo 56b1, after its position has been adjusted, interferes with a roof, piping, etc. (not shown) placed above area 59b.

[0080] In addition, known interference detection methods or collision detection methods can be applied to determine whether or not an object interferes with another object in the first or second interference detection. For example, the first interference detection may be performed by determining whether or not voxels constituting another object exist in the space occupied by voxels constituting the first object in the three-dimensional virtual space constituting the first virtual map 51b. Alternatively, the second interference detection may be performed using the same method as the first interference detection.

[0081] The notification unit 42 notifies of the occurrence of interference when the first interference determination determines that the first object and other virtual objects included in the first virtual map 51b, excluding the first object, are interfering with each other. In the example shown in Figure 8, the notification unit 42 notifies, using a predetermined notification method, that the virtual luggage 56b1 after position adjustment is interfering with other objects. The notification unit 42 also notifies of the occurrence of interference when the second interference determination determines that the second object and other virtual objects included in the first virtual map 51b, excluding the second object, are interfering with each other. In the example shown in Figure 10, the notification unit 42 notifies, using a predetermined notification method, that the virtual luggage 56c after placement is interfering with other objects.

[0082] The method of notification by the notification unit 42 is not particularly limited. For example, the notification unit 42 may emit a warning sound from a speaker (not shown) provided by the HMI 5, or it may display a warning mark or warning text on the display 6. In addition, the notification unit 42 may notify of interference by indicating the area where the first object or the second object is interfering with another object in a more conspicuous color than other areas, or by indicating the outline of the said area in a more conspicuous color than other outlines.

[0083] Next, an example of the operation of the on-site monitoring device 1 will be explained with reference to the flowchart in Figure 11. By repeating the processes of steps S101 to S108 illustrated in the figure, the status of the facility premises can be monitored more reliably.

[0084] When the on-site monitoring device 1 starts monitoring the facility premises, it first generates a reference virtual map 51a, for example, as shown in Figure 3, based on the information about the vehicle V acquired by the sensor 2 at time T0, and stores it in a storage device (not shown) in the controller 10. Then, by operating the operation unit 7 of the HMI 5, for example, the monitoring of the facility premises is started.

[0085] In step S101, the virtual map generation unit 15 generates a first virtual map 51b, for example, as shown in Figure 4, based on the information about the vehicle V acquired by the sensor 2 at time T1. The first virtual map 51b corresponds to the state of the facility premises at time T1, which is chronologically later than time T0. The process then proceeds to step S102.

[0086] In step S102, the object detection unit 20 detects a first object belonging to a predetermined category from the first virtual map 51b. If "vehicles," "luggage," and "people" have been set as the categories in advance, then virtual stationary vehicles 55b1, 55b2, virtual luggage 56b1, 56b2, and virtual pedestrian 57b1 are detected as first objects from the first virtual map 51b. If the first virtual map 51b does not contain any objects belonging to the category, the object detection unit 20 does not need to detect a first object. The process then proceeds to step S103.

[0087] In step S103, the difference extraction unit 30 extracts the difference between the reference virtual map 51a and the first virtual map 51b. In the example shown in Figures 3 and 4, compared to the reference virtual map 51a, the number of virtual stationary vehicles 55b2 has increased in the first virtual map 51b, while the number of virtual cargo 56a3 and virtual pedestrians 57a2 has decreased. Therefore, the difference extraction unit 30 extracts these increased and decreased virtual stationary vehicles 55b2, virtual cargo 56a3, and virtual pedestrians 57a2 as differences. Note that if there is no change between the reference virtual map 51a and the first virtual map 51b, the difference extraction unit 30 does not need to extract any differences. After that, the process proceeds to step S104.

[0088] In step S104, the virtual map generation unit 15 displays the first virtual map 51b on the display 6 of the HMI 5. At this time, the user of the premises monitoring device 1 may appropriately set the display range, display angle, etc., of the first virtual map 51b on the display 6. After that, the process proceeds to step S105.

[0089] In step S105, the object display unit 25 performs object highlighting, highlighting the first object detected in step S102 on the display 6. For example, as shown in Figure 5, the virtual stationary vehicles 55b1, 55b2, the virtual luggage 56b1, 56b2, and the virtual pedestrian 57b1 are highlighted. If no first object is detected in step S102, step S105 may be omitted. The process then proceeds to step S106.

[0090] In step S106, the difference determination unit 31 determines whether there is a difference between the reference virtual map 51a and the first virtual map 51b. That is, it determines whether the difference extraction unit 30 extracted a difference in step S103. In step S103, the difference extraction unit 30 extracted the virtual stopped vehicle 55b2, virtual luggage 56a3, and virtual pedestrian 57a2 as differences, so the difference determination unit 31 determines Yes. The process then proceeds to step S107. If the difference extraction unit 30 did not extract a difference in step S103, the difference determination unit 31 determines No in step S106, and the series of processes ends.

[0091] In step S107, the difference emphasis determination unit 32 performs a difference emphasis determination. That is, it determines whether or not the extracted difference is subject to difference emphasis display. If the difference emphasis determination determines that the extracted difference should be displayed with emphasis on the display 6 (Yes in step S107), the process proceeds to step S108. On the other hand, if the difference emphasis determination determines that the extracted difference should not be displayed with emphasis on the display 6 (No in step S107), the series of processes ends.

[0092] If the difference extraction unit 30 has extracted multiple differences, the difference emphasis determination unit 32 performs a difference emphasis determination for each of the multiple differences. Then, it determines which differences are to be displayed with difference emphasis. For example, it performs a difference emphasis determination for each of the differences extracted in step S103: the virtual stationary vehicle 55b2, the virtual luggage 56a3, and the virtual pedestrian 57a2. Then, for example, it determines that the virtual stationary vehicle 55b2 and the virtual luggage 56a3 are to be displayed with difference emphasis.

[0093] In step S108, the difference display unit 33 performs difference highlighting, which highlights the differences determined to be highlighted in step S107 on the display 6. In the example shown in Figure 6, the virtual stopped vehicle 55b2 and the virtual cargo 56a3 are the targets of difference highlighting, so they are highlighted. After that, the series of processes ends.

[0094] The difference display unit 33 may highlight and display the difference between the reference virtual map 51a and the first virtual map 51b on the display 6 without performing a difference highlighting determination. In other words, the difference display unit 33 may perform difference highlighting display in response to the difference determination unit 31 determining that a difference exists. In that case, step S107 may be omitted.

[0095] Note that the timing of displaying the first virtual map 51b in step S104, highlighting objects in step S105, or highlighting differences in step S108 is not limited to the example shown in Figure 11. For example, steps S104, S105, and S108 may be performed after the processing in step S107.

[0096] Next, with reference to the flowchart in Figure 12, an example of the process performed in the difference enhancement determination in step S107 will be explained in more detail. The difference enhancement determination can be performed by steps S201 to S205 shown in the figure.

[0097] In step S201, the difference enhancement determination unit 32 determines, by region determination, whether or not to display the differences extracted by the difference extraction unit 30 with emphasis on the display 6. In region determination, it is determined whether or not the extracted differences are located in a predetermined region within the virtual three-dimensional map 50. In the example shown in Figure 6, regions 58b and 59b are set as predetermined regions within the virtual three-dimensional map 50. Therefore, the difference enhancement determination unit 32 determines whether or not each of the differences extracted in step S103—the virtual stationary vehicle 55b2, the virtual luggage 56a3, and the virtual pedestrian 57a2—are located in region 58b or region 59b.

[0098] In the illustrated example, the virtual stationary vehicle 55b2 and the virtual cargo 56a3 are located within either region 58b or region 59b. Therefore, the difference enhancement determination unit 32 determines that the virtual stationary vehicle 55b2 and the virtual cargo 56a3 are located within the predetermined region (Yes in step S201). The process then proceeds to step S202. On the other hand, the virtual pedestrian 57a2 is not located within either region 58b or region 59b. Therefore, the difference enhancement determination unit 32 determines that the virtual pedestrian 57a2 is not located within the predetermined region (No in step S201). The process then proceeds to step S204.

[0099] In step S202, the difference emphasis determination unit 32 determines, by category determination, whether or not to emphasize and display the difference extracted by the difference extraction unit 30 on the display 6. In category determination, it is determined whether or not the extracted difference belongs to a predetermined category.

[0100] For example, if "vehicle" and "luggage" are pre-set as predetermined categories, then for each of the virtual stopped vehicle 55b2 and virtual luggage 56a3, which were determined to be Yes in step S201, it is determined whether or not they belong to the category "vehicle" or the category "luggage". Here, the virtual stopped vehicle 55b2 belongs to the category "vehicle", and the virtual luggage 56a3 belongs to the category "luggage". Therefore, the difference emphasis determination unit 32 determines that the virtual stopped vehicle 55b2 and virtual luggage 56a3 belong to the predetermined categories (Yes in step S202). The process then proceeds to step S203. If the difference that was the subject of the category determination does not belong to the predetermined category, the difference emphasis determination unit 32 determines that the extracted difference does not belong to the predetermined category (No in step S202). The process then proceeds to step S204.

[0101] In step S203, the difference emphasis determination unit 32 determines, based on the duration determination, whether or not to emphasize and display the difference extracted by the difference extraction unit 30 on the display 6. The duration determination determines whether or not the extracted difference has been continuously extracted for a predetermined time or longer.

[0102] For example, the difference emphasis determination unit 32 determines whether each of the virtual stopped vehicle 55b2 and virtual cargo 56a3, which were determined to be Yes in step S202, were extracted continuously for a predetermined time or longer. For example, if the extraction of the virtual stopped vehicle 55b2 and virtual cargo 56a3 continued for a predetermined time or longer, the determination for each of the virtual stopped vehicle 55b2 and virtual cargo 56a3 is made in step S203 as Yes. The process then proceeds to step S205. On the other hand, if the extracted difference was not extracted continuously for a predetermined time or longer, the determination for that difference is made in step S203 as No. The process then proceeds to step S204.

[0103] In step S204, the difference emphasis determination unit 32 excludes from the difference emphasis display targets any differences extracted in step S103 that are determined not to be emphasized on the display 6. For example, virtual pedestrian 57a2, which was determined to be No in the area determination in step S201, is excluded from the difference emphasis display targets.

[0104] In step S205, the difference emphasis determination unit 32 determines the differences to be highlighted. For example, it is determined that the virtual stationary vehicle 55b2 and virtual cargo 56a3 shown in Figure 6 will be the targets for difference emphasis display.

[0105] In the flowchart illustrated in Figure 12, the difference enhancement determination is performed by a combination of region determination, type determination, and duration determination, but it is not limited to this. The difference enhancement determination may be performed using at least one of the region determination, type determination, and duration determination.

[0106] Next, with reference to the flowchart in Figure 13, an example of the processing that occurs when a predetermined operation is performed on the first virtual map 51b will be explained.

[0107] For example, when monitoring the premises using the premises monitoring device 1, the user can adjust the position of a first object within the first virtual map 51b, or place a second object within the first virtual map 51b, via the operation unit 7 of the HMI 5. The operations that can be performed via the operation unit 7 are not limited to this, and the premises monitoring device 1 may be configured to allow the user to perform other operations.

[0108] In step S301, the operation detection unit 40 detects that a predetermined operation has been performed on the first virtual map 51b. The process then proceeds to step S302.

[0109] In step S302, the operation detection unit 40 determines whether the operation detected in step S301 was an operation to adjust the position of the first object within the first virtual map 51b. If the operation was an adjustment of the position of the first object within the first virtual map 51b, the result is determined to be Yes. The process then proceeds to step S303. On the other hand, if the operation was not an adjustment of the position of the first object within the first virtual map 51b, the result is determined to be No. The process then proceeds to step S304.

[0110] In step S303, the interference determination unit 41 performs a first interference determination. In the first interference determination, it is determined whether the first object, after its position has been adjusted, interferes with any other object in the first virtual map 51b.

[0111] In the example shown in Figure 7, at position P2 in the first virtual map 51b, the first object, virtual luggage 56b1, after position adjustment, does not interfere with any other objects in the first virtual map 51b. Therefore, the interference determination unit 41 determines No in step S303. The process then ends. On the other hand, in the example shown in Figure 8, the virtual luggage 56b1, which has been adjusted to position P3, interferes with virtual luggage 56b2. Therefore, the interference determination unit 41 determines Yes in step S303. The process then proceeds to step S306.

[0112] In step S304, the operation detection unit 40 determines whether the operation detected in step S301 was an operation to place a second object in the first virtual map 51b. If the operation was the placement of a second object in the first virtual map 51b, the result is determined to be Yes. The process then proceeds to step S305. On the other hand, if the operation was not the placement of a second object in the first virtual map 51b, the result is determined to be No. The series of processes then ends.

[0113] In step S305, the interference determination unit 41 performs a second interference determination. In the second interference determination, it is determined whether the second object, after being placed, interferes with other objects in the first virtual map 51b.

[0114] In the example shown in Figure 9, the second object, virtual cargo 56c after placement, does not interfere with other objects in the first virtual map 51b. Therefore, the interference determination unit 41 determines No in step S305. The process then ends. On the other hand, in the example shown in Figure 10, the virtual cargo 56c placed at position P4 interferes with the virtual cargo 56b1. Therefore, the interference determination unit 41 determines Yes in step S305. The process then proceeds to step S306.

[0115] In step S306, the notification unit 42 notifies of the occurrence of interference. For example, if Yes is determined in step S303, the notification unit 42 notifies of interference between virtual luggage 56b1, which has been adjusted to position P3, and virtual luggage 56b2 using a predetermined notification method. Also, if Yes is determined in step S305, the notification unit 42 notifies of interference between virtual luggage 56c, which is located at position P4, and virtual luggage 56b1 using a predetermined notification method.

[0116] Next, the operation and effects of the on-site monitoring method and on-site monitoring device according to the embodiment will be described.

[0117] (1) The on-site monitoring method according to the embodiment is an on-site monitoring method for monitoring the state of a facility site, which generates a virtual three-dimensional map 50 of the facility site over time based on information about the surroundings of a vehicle V acquired by a sensor 2 installed on a vehicle that can travel within the facility site, and monitors changes in the state of the facility site by comparing a reference virtual map 51a that shows a reference state of the facility site from the virtual three-dimensional map 50 and a first virtual map 51b that is generated later than the reference virtual map 51a in chronological order from the virtual three-dimensional map 50.

[0118] According to the on-site monitoring method of the embodiment, by comparing a reference virtual map 51a and a first virtual map 51b among the virtual three-dimensional maps 50 of the facility site that are generated over time, changes from a reference state within the facility site can be monitored. Therefore, the area around the vehicle V equipped with the sensor 2, and the area outside the area around the vehicle V, can be monitored more reliably within the facility site. In other words, the state of the facility site can be monitored more reliably. Furthermore, in the on-site monitoring method of the embodiment, the virtual three-dimensional map 50 is generated based on information acquired by the sensor 2 installed on the vehicle V. Therefore, compared to the case where the virtual three-dimensional map 50 is generated based on information acquired by sensors 2 installed at multiple locations within the facility site, it is possible to generate a virtual three-dimensional map 50 that reflects the state of the facility site more easily.

[0119] (2) The system may detect first objects 55b1, 55b2, 56b1, 56b2, and 57b1 belonging to predetermined categories from the first virtual map 51b, display the first virtual map 51b on the display unit 6 capable of displaying the virtual three-dimensional map 50, and highlight the first objects 55b1, 55b2, 56b1, 56b2, and 57b1.

[0120] As a result, as illustrated in Figure 5, the first objects 55b1, 55b2, 56b1, 56b2, and 57b1 contained within the first virtual map 51b can be highlighted and displayed. Therefore, the positions of the first objects 55b1, 55b2, 56b1, 56b2, and 57b1 within the facility premises can be monitored more reliably.

[0121] (3) It is determined whether or not there is a difference between the reference virtual map 51a and the first virtual map 51b, and if it is determined that there is a difference, the difference may be highlighted and displayed on the display unit 6 capable of displaying the virtual three-dimensional map 50.

[0122] This allows the changes between the state of the facility premises shown in the reference virtual map 51a and the state of the facility premises shown in the first virtual map 51b to be displayed as differences on the display 6. Therefore, changes in the state of the facility premises can be monitored more reliably.

[0123] (4) Depending on whether a difference is determined to exist, it may be determined whether or not to highlight the difference on the display unit 6, and depending on whether it is determined to highlight the difference, the difference may be highlighted on the display unit 6.

[0124] This allows only the differences between the reference virtual map 51a and the first virtual map 51b that meet predetermined conditions to be highlighted and displayed on the display 6. Therefore, changes in the state of the facility can be grasped more easily.

[0125] (5) In response to the adjustment of the positions of the first objects 55b1, 55b2, 56b1, 56b2, 57b1 within the first virtual map 51b, a first interference determination is performed to determine whether the adjusted first objects 55b1, 55b2, 56b1, 56b2, 57b1 and other objects in the first virtual map 51b interfere with each other. If the first interference determination determines that the first objects 55b1, 55b2, 56b1, 56b2, 57b1 and other objects in the first virtual map 51b interfere with each other, the occurrence of interference may be notified.

[0126] By performing a first interference determination within the first virtual map 51b, for example, when it is planned to adjust the position of luggage within the facility grounds, it is possible to know in advance whether or not the luggage will interfere with other objects within the facility grounds at the destination location.

[0127] (6) In response to the placement of the second object 56c in the first virtual map 51b, a second interference determination is performed to determine whether the placed second object 56c and other objects in the first virtual map 51b interfere with each other, and if the second interference determination determines that the second object 56c and the objects in the first virtual map 51b interfere with each other, the occurrence of interference may be notified.

[0128] By performing a second interference check within the first virtual map 51b, it is possible to determine in advance whether, for example, when a new package is placed within the facility grounds, the new package will interfere with an object already present within the facility grounds.

[0129] (7) A virtual three-dimensional map 50 of the facility premises may be generated over time based on information about the surroundings of each of the multiple vehicles V, which are acquired by sensors 2 installed on each of the multiple vehicles V that are capable of traveling within the facility premises.

[0130] This allows each of the multiple vehicles V traveling within the facility to acquire information about the surroundings of each vehicle V using its respective sensor 2. Based on the acquired information, a virtual three-dimensional map 50 can be generated over time. As a result, a first virtual map 51b that more accurately shows the state of the facility can be generated, and the state of the facility can be monitored more reliably.

[0131] (8) Sensor 2 may be a three-dimensional sensor capable of detecting the shape of objects present around the vehicle V and their distance from the vehicle V in three dimensions.

[0132] As a result, the virtual map generation unit 15 can generate a three-dimensional virtual space that reflects the structure of the facility grounds, as well as the positions and shapes of objects placed within the facility grounds, in both the horizontal and vertical directions. Therefore, the virtual map generation unit 15 can generate a reference virtual map 51a and a first virtual map 51b that reflect the horizontal and vertical state of the facility grounds. This allows for more accurate monitoring of the state of the facility grounds.

[0133] (9) The on-site monitoring device according to the embodiment is an on-site monitoring device 1 for monitoring the state of the facility premises, and includes a virtual map generation unit 15 that generates a virtual three-dimensional map 50 of the facility premises over time based on information about the surroundings of the vehicle V acquired by a sensor 2 installed on the vehicle V that can travel within the facility premises, and monitors changes in the state of the facility premises by comparing a reference virtual map 51a that shows a reference state of the facility premises from among the virtual three-dimensional map 50 and a first virtual map 51b that is generated later than the reference virtual map 51a in chronological order from among the virtual three-dimensional map 50.

[0134] According to the on-site monitoring device 1 of this embodiment, the virtual map generation unit 15 generates a virtual three-dimensional map 50 of the facility premises over time. By comparing a reference virtual map 51a and a first virtual map 51b within the virtual three-dimensional map 50, changes in the state of the facility premises can be monitored. Therefore, the area around the vehicle V equipped with the sensor 2, and the area outside the area around the vehicle V, can be monitored more reliably within the facility premises. In other words, the state of the facility premises can be monitored more reliably. Furthermore, in the on-site monitoring device 1 of this embodiment, the virtual map generation unit 15 generates the virtual three-dimensional map 50 based on information acquired by the sensor 2 installed on the vehicle V. Therefore, compared to the case where the virtual three-dimensional map 50 is generated based on information acquired by sensors 2 installed at multiple locations within the facility premises, the on-site monitoring device 1 can provide a virtual three-dimensional map 50 that more easily reflects the state of the facility premises.

[0135] This disclosure can contribute, for example, to Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0136] Although several embodiments have been described above, it is possible to modify or transform the embodiments based on the above disclosure. All components of the above embodiments and all features described in the claims may be individually selected and combined, provided that they do not contradict each other. [Explanation of Symbols]

[0137] V Vehicle 1. On-site monitoring device 2 sensors 6. Display (Display Unit) 15. Virtual Map Generation Unit 50 Virtual 3D Maps 51a Reference virtual map 51b First Virtual Map 55b1, 55b2 Virtual stationary vehicle (first object) 56b1, 56b2 Virtual cargo (first object) 57b1 Virtual pedestrian (first object) 56c Virtual cargo (second object)

Claims

1. A method for monitoring the conditions within a facility premises, Based on information about the surroundings of a vehicle acquired by sensors installed on the vehicle capable of traveling within the facility grounds, a virtual three-dimensional map of the facility grounds is generated over time. By comparing a reference virtual map, which represents a standard state within the facility premises, with a first virtual map, which is generated later than the reference virtual map in chronological order, changes in the state within the facility premises are monitored. A first object belonging to a predetermined category is detected from the first virtual map, The first virtual map is displayed on a display unit capable of displaying the virtual three-dimensional map, and the first object is displayed with emphasis. In response to the adjustment of the position of the first object within the first virtual map, a first interference determination is performed to determine whether the adjusted first object and other objects in the first virtual map interfere with each other. In the first interference determination, if it is determined that the first object and the object in the first virtual map interfere with each other, the occurrence of interference is notified. On-site monitoring methods.

2. Determine whether or not there is a difference between the reference virtual map and the first virtual map. The on-site monitoring method according to claim 1, wherein, in response to the determination that the aforementioned difference exists, the difference is highlighted and displayed on a display unit capable of displaying the virtual three-dimensional map.

3. In response to the determination that the aforementioned difference exists, a determination is made as to whether or not to highlight the aforementioned difference on the display unit. The on-site monitoring method according to claim 2, wherein, in response to a determination that the difference should be emphasized, the display unit displays the difference with emphasis.

4. In response to the placement of the second object within the first virtual map, a second interference determination is performed to determine whether the placed second object and other objects within the first virtual map interfere with each other. The on-site monitoring method according to claim 1, wherein, in response to the determination in the second interference determination that the second object and the object in the first virtual map interfere with each other, the method notifies of the occurrence of interference.

5. The facility monitoring method according to claim 1, wherein a virtual three-dimensional map of the facility is generated over time based on information about the surroundings of each of the multiple vehicles, which is acquired by sensors installed on each of the multiple vehicles that can travel within the facility premises.

6. The on-site monitoring method according to claim 1, wherein the sensor is a three-dimensional sensor capable of detecting the shape of objects present around the vehicle and their distance from the vehicle.

7. A facility monitoring device that monitors the conditions within the facility grounds, The facility includes a virtual map generation unit that generates a virtual three-dimensional map of the facility premises over time based on information about the surroundings of a vehicle acquired by sensors installed on the vehicle capable of traveling within the facility premises, By comparing a reference virtual map, which represents a standard state within the facility premises, with a first virtual map, which is generated later than the reference virtual map in chronological order, changes in the state within the facility premises are monitored. A first object belonging to a predetermined category is detected from the first virtual map, The first virtual map is displayed on a display unit capable of displaying the virtual three-dimensional map, and the first object is displayed with emphasis. In response to the adjustment of the position of the first object within the first virtual map, a first interference determination is performed to determine whether the adjusted first object and other objects in the first virtual map interfere with each other. In the first interference determination, if it is determined that the first object and the object in the first virtual map interfere with each other, the occurrence of interference is notified. On-site monitoring system.

Images