Situation awareness support device, situation awareness support method, and program

Abstract

To provide a situation determination aiding device that can aid in determination of a disaster situation.SOLUTION: A situation determination aiding device according to an aspect of the present disclosure comprises: acquisition means for acquiring a camera image capturing a disaster site; identification means for identifying a partial region corresponding to the captured disaster site from model data indicating a three-dimentional model in which a real space is represented by aligning the camera image with the model data; detection means for detecting an object on the basis of a difference between the camera image and an image of the identified partial region; generation means for generating a current model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object; and display control means for displaying the generated current model data on a display.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a technology for grasping the situation at a disaster site.

Background Art

[0002] When a disaster such as a fire occurs, the on-site disaster response team members and their back-up supporters (e.g., operators who give instructions to the team members) are required to grasp the situation at the disaster site.

[0003] Regarding grasping the situation at a fire site, in Patent Document 1, there is a technology for detecting dangerous locations. For example, in Patent Document 1, it is disclosed that a dangerous location at the site is identified from images taken by a camera module worn by a firefighter, and an image indicating the position of the dangerous location is displayed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technology disclosed in Patent Document 1, the position of the dangerous location is merely shown on a flat map. In this case, there is a risk that the back-up supporters who are not directly at the site may not be able to fully grasp the situation. There is room for improvement in this regard.

[0006] The present disclosure has been made in view of the above problems, and one of the objectives is to provide a situation grasping support device or the like that can assist in grasping the disaster situation.

Means for Solving the Problems

[0007] A situation awareness support device according to an aspect of the present disclosure includes an acquisition unit that acquires a captured image of a disaster site, and a specific unit that specifies a partial region corresponding to the captured disaster site among the model data by aligning the captured image with model data showing a three-dimensional model representing the real space, a detection unit that detects an object based on a difference between the captured image and an image of the specified partial region, a generation unit that generates current situation model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object, and a display control unit that causes the generated current situation model data to be displayed on a display.

[0008] A situation awareness support method according to an aspect of the present disclosure includes acquiring a captured image of a disaster site, specifying a partial region corresponding to the captured disaster site among the model data by aligning the captured image with model data showing a three-dimensional model representing the real space, detecting an object based on a difference between the captured image and an image of the specified partial region, generating current situation model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object, and causing the generated current situation model data to be displayed on a display.

[0009] A program according to an aspect of the present disclosure causes a computer to execute a process of acquiring a captured image of a disaster site, a process of specifying a partial region corresponding to the captured disaster site among the model data by aligning the captured image with model data showing a three-dimensional model representing the real space, a process of detecting an object based on a difference between the captured image and an image of the specified partial region, a process of generating current situation model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object, and a process of causing the generated current situation model data to be displayed on a display.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to assist in grasping a disaster situation.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] <First Embodiment> The outline of the situation grasping support device according to the first embodiment will be described.

[0014] FIG. 1 is a diagram schematically showing an example of the configuration of a situation grasping support system 1000. The situation grasping support system 1000 includes at least a situation grasping support device 100. In the example of FIG. 1, the situation grasping support system 1000 includes a situation grasping support device 100, a team member device 200, and a terminal device 300. The situation grasping support device 100 is communicably connected to the team member device 200 and the terminal device 300 via a wireless or wired network.

[0015] The situation grasping support system 1000 in the present disclosure is used in the response when a disaster occurs. As an example of the situation where the system is used, there is a situation where team members carry out rescue activities or fire extinguishing activities at the disaster site. Examples of team members include emergency team members, firefighters, and rescue team members. At this time, the back supporter grasps the situation of the disaster site based on reports from the team members and gives instructions to the team members. The back supporter may also be called an operator or a commander.

[0016] The on-site team members carry the team member device 200. The team member device 200 has a function of performing predetermined measurements. For example, the team member device 200 may be a device worn by a team member. That is, the location of the measurement by the team member device 200 may change as the team member moves. The terminal device 300 is a device used by a back-end supporter. The terminal device 300 has at least a display. The situation awareness support device 100 performs various processes using the information measured by the team member device 200. Then, the situation awareness support device 100 causes information regarding the situation at the disaster site according to the results of various processes to be displayed on the display of the terminal device 300. In this way, the situation awareness support system 1000 is a system capable of enabling a back-end supporter to grasp the situation at the disaster site by using the information at the disaster site acquired by the team member device 200.

[0017] Note that in FIG. 1, each of the situation awareness support device 100, the team member device 200, and the terminal device 300 is described as being configured by one device for convenience. However, the configuration of each of the situation awareness support device 100, the team member device 200, and the terminal device 300 is not limited to this example. That is, each of the situation awareness support device 100, the team member device 200, and the terminal device 300 may be configured by a plurality of devices. Also, there may be a plurality of team member devices 200 and terminal devices 300.

[0018] The team member device 200 has a function of performing predetermined measurements. An example of the predetermined measurement is a photographing function. That is, the team member device 200 is equipped with a camera module capable of photographing the surroundings.

[0019] The terminal device 300 may be a personal computer. The terminal device 300 may have various input / output devices. Examples of the input / output devices are a keyboard, a microphone, a camera, a display, and a speaker, etc. An example of the terminal device 300 is a personal computer and a head-mounted display connectable to the personal computer.

[0020] The situation awareness support device 100 may be a server device.

[0021] Next, an example of the functional configuration of the situation awareness support device 100 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the situation awareness support device 100. As shown in FIG. 2, the situation awareness support device 100 includes an acquisition unit 110, a specification unit 120, a detection unit 130, a generation unit 140, and a display control unit 150.

[0022] The acquisition unit 110 acquires various types of information from the team member device 200. For example, the team member device 200 takes pictures of the surroundings at the disaster site. The acquisition unit 110 acquires the captured images taken by the team member device 200. In this way, the acquisition unit 110 acquires the captured images of the disaster site. The acquisition unit 110 is an example of an acquisition means.

[0023] The specification unit 120 performs processing on the model data using the captured images. The model data is a three-dimensional model in which the real space is represented. More specifically, the model data is a three-dimensional model in which existing buildings, terrain, etc. in the real world are reproduced in a virtual space. The model data may be stored in a storage device possessed by the situation awareness support device 100, or may be stored in an external storage device that is communicably connected to the situation awareness support device 100.

[0024] The specification unit 120 specifies the area of the model data corresponding to the photographed disaster site based on the captured images. For example, the specification unit 120 uses the features of the buildings shown in the captured images to align the captured images with the model data. The partial area of the model data specified by this alignment becomes the area corresponding to the disaster site.

[0025] In this way, the specification unit 120 specifies the partial area of the model data corresponding to the disaster site by aligning the captured images with the model data showing the three-dimensional model in which the real space is represented. The specification unit 120 is an example of a specification means.

[0026] The detection unit 130 detects an object. Specifically, the detection unit 130 calculates the difference between the captured image and the image of the partial region of the model data specified by the specifying means. Then, the detection unit 130 detects an object for the calculated difference portion. Examples of the object include a person such as a person in need of first aid, an obstacle, a fire, and smoke.

[0027] In this way, the detection unit 130 detects an object based on the difference between the captured image and the image of the specified partial region. The detection unit 130 is an example of a detection means.

[0028] The generation unit 140 generates model data reflecting information about the detected object. Specifically, the generation unit 140 specifies the position on the model data corresponding to the position on the captured image of the object detected by the detection unit 130. Then, the generation unit 140 reflects the information indicating the detected object at the specified position on the model data. For example, assume that a fire is detected by the detection unit 130. At this time, the generation unit 140 specifies the position on the model data corresponding to the position on the captured image of the detected fire. Then, the generation unit 140 adds an icon indicating the fire to the specified position on the model data. The icon may be an image displayed in a virtual space. For example, the icon is an object image. Thereby, the information of the detected object is reflected in the model data. The model data in which the information about the detected object is reflected is referred to as the current model data. Note that the method of reflecting the information about the object in the model data is not limited to this example.

[0029] In this way, the generation unit 140 generates the current model data in which an icon corresponding to the object is arranged at the position on the model data corresponding to the position of the detected object. The generation unit 140 is an example of a generation means.

[0030] The display control unit 150 causes the current model data to be displayed. Specifically, the display control unit 150 causes the current model data to be displayed on the display included in the terminal device 300. The current model data may be displayed as a VR (Virtual Reality) image.

[0031] Next, an example of the operation of the situation awareness support device 100 will be described with reference to FIG. 3. In the present disclosure, each step of the flowchart is represented using the number assigned to each step, such as "S1".

[0032] FIG. 3 is a flowchart for explaining an example of the operation of the situation awareness support device 100. The acquisition unit 110 acquires a captured image of the disaster site (S1). The identification unit 120 identifies a partial region corresponding to the captured disaster site in the model data by aligning the captured image with model data indicating a three-dimensional model representing the real space (S2). The detection unit 130 detects an object based on the difference between the captured image and the image of the identified partial region (S3). The generation unit 140 generates current situation model data in which icons corresponding to the object are arranged at positions on the model data corresponding to the positions of the detected objects (S4). The display control unit 150 causes the generated current situation model data to be displayed on the display (S5).

[0033] As described above, the situation awareness support device 100 according to the first embodiment acquires a captured image of the disaster site, and identifies a partial region corresponding to the captured disaster site in the model data by aligning the captured image with model data indicating a three-dimensional model representing the real space. Further, the situation awareness support device 100 detects an object based on the difference between the captured image and the image of the identified partial region. Then, the situation awareness support device 100 generates current situation model data in which icons corresponding to the object are arranged at positions on the model data corresponding to the positions of the detected objects, and causes the generated current situation model data to be displayed on the display.

[0034] As a result, the situation awareness support device 100 can, for example, enable a rear supporter to grasp the situation at the disaster site using a three-dimensional model. Therefore, the situation awareness support device 100 can enable the situation at the disaster site to be grasped more intuitively than when showing the situation at the disaster site using characters or a flat map. That is, the situation awareness support device 100 according to the first embodiment can support grasping the disaster situation.

[0035] <Second Embodiment> Next, the situation awareness support system according to the second embodiment will be described. In the second embodiment, a further example regarding the situation awareness support system described in the first embodiment will be described. Note that descriptions of content overlapping with the first embodiment will be partially omitted.

[0036] In the second embodiment, an application example of the situation awareness support system in the case where a disaster has occurred and team members are operating inside a predetermined building will be described, but this does not limit the applicable situations of the system.

[0037] FIG. 4 is a diagram schematically showing an example of the configuration of the situation awareness support system 1000. FIG. 4 is a specific example showing the situation awareness support system 1000 shown in FIG. 1. In the example of FIG. 4, there are team member devices 200-1, 200-2, ···, 200-n (n is a natural number). Hereinafter, when the team member devices 200-1, 200-2, ···, 200-n are not distinguished, they are simply referred to as team member devices 200. Also, as shown in FIG. 4, the situation awareness support system 1000 may include a communication device. The communication device is communicably connected to the team member device 200 and the situation awareness support device 100. There may be a plurality of communication devices. Details of the communication device will be described later.

[0038] Also, in the example of FIG. 4, the terminal device 300 includes a computer and a head-mounted display. In this case, the rear supporter wears the head-mounted display. By outputting the current situation model data generated by the situation awareness support device 100 to the head-mounted display, the rear supporter can virtually experience the situation at the disaster site.

[0039] [Details of the Situation Awareness Support System 1000] FIG. 5 is a block diagram showing an example of the configuration of the situation awareness support system 1000.

[0040] The team member device 200 includes a visible light imaging unit 210, an infrared imaging unit 220, and a position information communication unit 230. The visible light imaging unit 210 performs imaging and generates a captured image. The visible light imaging unit 210 is realized by a camera module that uses visible light. In short, the visible light imaging unit 210 receives visible light and generates a visible image. In the present disclosure, the visible image captured by the visible light imaging unit 210 is simply referred to as a captured image. The visible light imaging unit 210 transmits the captured image to the situation awareness support device 100.

[0041] The infrared imaging unit 220 performs imaging and generates an infrared image. The infrared imaging unit 220 is realized by a camera module that uses infrared light. In short, the infrared imaging unit 220 receives infrared light and generates an infrared image. At this time, the infrared image is captured at substantially the same angle of view as the captured image. That is, the infrared imaging unit 220 generates an infrared image corresponding to the captured image. The infrared imaging unit 220 transmits the infrared image to the situation awareness support device 100.

[0042] Note that the visible light imaging unit 210 and the infrared imaging unit 220 may transmit the images captured at the same time to the situation awareness support device 100 in association with each other. That is, the team member device 200 may transmit the captured image and the infrared image captured at the same time to the situation awareness support device 100 in association with each other.

[0043] The location information communication unit 230 transmits and receives signals for acquiring location information. As an example, the location information communication unit 230 uses a beacon to receive signals related to location information. In this case, the communication device is a beacon terminal. Then, a plurality of communication devices are installed in the buildings at the disaster site. For example, the communication device transmits a unique ID (Identification), which is an example of a signal related to location information, by short-range wireless communication such as Bluetooth (registered trademark). The unique ID is different information for each communication device and is associated with information indicating the location where the communication device is installed. The location information communication unit 230 receives signals related to location information. Then, the location information communication unit 230 transmits the received signals related to location information to the situation awareness support device 100. The situation awareness support device 100 acquires the location information of the team member device 200 based on the signals related to the location information acquired from the team member device 200. Note that the location information communication unit 230 may use Wi-Fi (registered trademark) instead of a beacon to receive signals related to location information. In this case, the communication device is an access point.

[0044] Moreover, not limited to this example, the location information communication unit 230 may receive signals related to location information using a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System). In this case, the situation awareness support device 100 acquires the signals received by the location information communication unit 230 as the location information of the team member device 200.

[0045] Note that the team member device 200 may be a wearable device worn by each team member. For example, the team member device 200 is worn on the clothes of the team member. Also, the team member device 200 may be worn on the mask portion of the self-contained breathing apparatus worn by the team member. Thereby, when the team member is at the disaster site, the captured images and infrared images captured by the team member device 200 are images showing the disaster site.

[0046] The situation awareness support device 100 includes an acquisition unit 110, a specification unit 120, a detection unit 130, a generation unit 140, and a display control unit 150. Further, the situation awareness support device 100 may have a storage device 190. The storage device 190 may be an external device communicably connected to the situation awareness support device 100.

[0047] The acquisition unit 110 acquires a photographed image from the team member device 200. Further, the acquisition unit 110 may acquire an infrared image from the team member device 200. At this time, the acquisition unit 110 acquires an infrared image photographed at the same time with a viewing angle substantially the same as that of the photographed image. That is, the acquisition unit 110 may acquire an infrared image corresponding to the photographed image.

[0048] Furthermore, the acquisition unit 110 acquires a signal related to position information from the team member device 200. When the signal related to position information is a signal obtained using a beacon or Wi-Fi, the acquisition unit 110 acquires the position information of the team member based on the signal related to position information. In this case, map information in which the signal related to position information is associated with information indicating the location where the communication device is installed is stored in the storage device 190 in advance. The acquisition unit 110 uses the map information to specify the location corresponding to the acquired signal related to position information. Then, the acquisition unit 110 acquires the specified location as the position information of the team member.

[0049] When the signal related to position information is a signal obtained using GNSS, the acquisition unit 110 acquires the signal as the position information of the team member.

[0050] In this way, the acquisition unit 110 may acquire the position information of the person (team member) who photographed the photographed image.

[0051] The specification unit 120 uses the photographed image acquired by the acquisition unit 110 to specify a partial area of the model data corresponding to the disaster site shown in the photographed image.

[0052] Here, as described above, the model data is a three-dimensional model in which the real space is represented. The model data is stored, for example, in the storage device 190. Also, the model data may be open data that is generally available.

[0053] The model data may be data constructed by a digital twin. A digital twin means reproducing the real space on a virtual space. The position information indicating the real space and the position information on the model data are associated with each other. That is, the model data includes information indicating the correspondence relationship between the model data and the real space such that it is possible to know which position in the real space each position on the model data corresponds to.

[0054] Also, the model data may be data used, for example, for the design data of each building. The model data may be data realized in, for example, the format of BIM (Building Information Modeling). That is, the model data may include not only information indicating the shape of a building or the like, but also information regarding the facilities of a building or the like. For example, when the building is a building, examples of information regarding facilities include the evacuation route of the building, the position of fire extinguishing equipment, the position of air conditioning equipment, the position of sprinklers, the position of fire shutters, and the position of installed cabinets. Further, the model data may include information regarding environmental simulation of the building, such as airflow analysis data and thermal analysis data of the building. Note that this is not limited to this example, and the model data may include information such as the material of the building and parts used for the building or the like.

[0055] When the specific processing by the specific part 120 is performed, first, it is necessary to specify the model data corresponding to the building at the disaster site. At this time, for example, a user (such as a back supporter) using the terminal device 300 may perform an input operation to specify the model data corresponding to the building at the disaster site.

[0056] Then, the specifying unit 120 aligns the specified model data with the captured image. The alignment is a process of specifying which part of the model data corresponds to the location shown in the captured image. The specifying unit 120 may perform the alignment between the specified model data and the captured image by using image processing techniques such as pattern matching. For example, the specifying unit 120 extracts the edges of the feature parts shown in the captured image, and specifies, as a partial region, a region including a feature part that matches the extracted edges among the specified model data. At this time, the specifying unit 120 searches for from which viewpoint of the specified model data an edge that matches the extracted edge appears, and may specify, as a partial region, a region visible from the viewpoint when the search hits. When the disaster site is inside a building, an example of the feature part is a wall, a ceiling, a pillar, etc. That is, in this case, the specifying unit 120 extracts the edges of the wall, the ceiling, the pillar, etc. shown in the captured image, and specifies, as a partial region, a region on the model data where there are a wall, a ceiling, a pillar, etc. that match the extracted edges. The specified partial region is a region on the model data corresponding to the photographed disaster site.

[0057] Note that the specifying unit 120 may use the position information of the team members during the specifying process. In this case, the specifying unit 120 uses the position information of the team members when the captured image is taken. Specifically, the specifying unit 120 searches for the position on the model data corresponding to the position information acquired by the acquisition unit 110. Then, the specifying unit 120 performs alignment with the captured image for the region around the position on the model data that hits by the search. Thereby, the computational load of the specifying process can be reduced. In this way, the specifying unit 120 may specify a partial region by aligning the captured image with the region of the model data corresponding to the acquired position information.

[0058] The detection unit 130 first obtains an image of the specified partial region. The image of this partial region is, for example, an image showing the partial region as seen from the viewpoint when the above-described search hits. That is, the image of the partial region is an image showing a location on the model data corresponding to the disaster site shown in the captured image with an angle of view substantially the same as that of the captured image.

[0059] The detection unit 130 calculates the difference between such an image of the partial region and the captured image. For example, the detection unit 130 generates a difference image showing the difference. Here, the difference image may be an image showing only the difference part, or may be an image obtained by cutting out a predetermined region including the difference from the captured image. Also, the detection unit 130 may generate a plurality of difference images.

[0060] Then, the detection unit 130 detects an object from the generated difference image. The difference between the image of the partial region and the captured image basically indicates an object existing at the disaster site.

[0061] As a method for detecting an object from the difference image, existing object detection techniques may be used. For example, the detection unit 130 extracts feature amounts from the difference image. The feature amounts may be image feature amounts derived using HOG (Histograms of Oriented Gradients), SIFT (Scaled Invarience Feature Transform), machine learning, etc. Then, the detection unit 130 may detect an object using a detection model that takes the extracted feature amounts as input. The detection model is a model that has learned the relationship between the feature amounts extracted from the difference image and the object. The detection unit 130 may input the extracted feature amounts into the detection model and detect the information output from the detection model as an object included in the difference image. Note that the example of object detection is not limited to this example. For example, the detection model may take an image as input. In this case, the detection unit 130 detects an object by inputting the difference image into the detection model. Examples of the objects to be detected are a person, an obstacle, a fire, and smoke, etc.

[0062] [First specific example of detection processing] Next, an example of the object detection process by the detection unit 130 will be described. FIG. 6 is a diagram showing an example of a captured image. Assume that the acquisition unit 110 has acquired a captured image as shown in FIG. 6. In the example of FIG. 6, the interior of the building is reflected in the captured image. And in the captured image, a scattered cabinet and a fallen person are reflected. This fallen person corresponds to a person in need of first aid.

[0063] The detection unit 130 acquires an image of the specified partial region. FIG. 7 is a diagram showing an example of an image of the partial region. That is, the image shown in FIG. 7 is an image showing model data. The detection unit 130 generates a difference image showing the difference between the image of the partial region and the captured image. FIG. 8 is a diagram showing a first example of the difference image. The difference image in FIG. 8 shows the difference portion between the captured image in FIG. 6 and the image of the partial region in FIG. 7. In the example of FIG. 8, the difference image includes a scattered cabinet and a fallen person.

[0064] Then, the detection unit 130 performs an object detection process on the difference image. In the case of the example in FIG. 8, the detection unit 130 detects a person and a cabinet as an obstacle.

[0065] [Second specific example of detection process] Next, another example of the object detection process by the detection unit 130 will be described. FIG. 9 is a diagram showing another example of a captured image. Assume that the acquisition unit 110 has acquired a captured image as shown in FIG. 9. The same location as the interior of the building shown in the captured image of FIG. 6 is reflected in the captured image of FIG. 9. And in the captured image of FIG. 9, flames and smoke are reflected.

[0066] The detection unit 130 acquires the image of the partial region in FIG. 7. Then, the detection unit 130 generates a difference image showing the difference between the image of the partial region in FIG. 7 and the captured image in FIG. 9. At this time, the detection unit 130 may generate a difference image as shown in FIGS. 10 and 11. FIG. 10 is a second example of the difference image. The difference image in FIG. 10 includes smoke. The detection unit 130 may detect the smoke by performing the above-described object detection process on the difference image.

[0067] At this time, the detection unit 130 may detect smoke from the color and edges of the difference image. When smoke is included in the captured image, the portion where the smoke exists in the captured image includes white, gray, black, and colors close to these. Also, it is difficult to detect an edge from the portion where the smoke exists in the captured image. Therefore, the detection unit 130 may detect smoke using the color of the difference image and information on whether an edge has been detected in the difference image. Specifically, in the difference image, when a location where no edge is detected is white, gray, black, or a color close to these, smoke exists at that location. In this way, the detection unit 130 may detect smoke based on the color of the difference location and information on whether an edge has been detected at the difference location in the captured image.

[0068] FIG. 11 is a third example of the difference image. The difference image in FIG. 11 includes a flame. The detection unit 130 may detect the flame by performing the above-described object detection process on the difference image.

[0069] At this time, the detection unit 130 may further use the infrared image to detect the flame. Temperature information can be extracted from the infrared image. Therefore, when the location where the flame exists is captured by the infrared imaging unit 220, high-temperature temperature information can be extracted from the infrared image. Therefore, the detection unit 130 may detect the difference location corresponding to the location where a temperature of a predetermined value or more is indicated by the infrared image as a flame. Also, when a flame is included in the captured image, the portion where the flame exists in the captured image includes red and colors close to red. Therefore, the detection unit 130 may detect, as a flame, a location in the difference image that shows red and colors close to red and where a temperature of a predetermined value or more is indicated.

[0070] The processes of the acquisition unit 110, the specification unit 120, and the detection unit 130 may be performed for each team member device 200 (that is, for each team member).

[0071] Returning to the description of the functional configuration of the situation awareness support device 100.

[0072] The generation unit 140 generates current state model data according to the detection result of the object by the detection unit 130. Specifically, when an object is detected by the detection unit 130, the generation unit 140 reflects information about the detected object at a position on the model data corresponding to the position of the detected object in the captured image. For example, assume that a captured image as shown in FIG. 9 is obtained. And assume that a person, an obstacle (scattered cabinet), a fire, and smoke are detected by the detection unit 130. In this case, the generation unit 140 arranges, for example, an icon indicating a person at a position on the model data corresponding to the position of the detected person. Here, the icon indicating a person is an example of information about the object. The icon is, for example, an object image represented in a virtual space. The icon may be an object image of a two-dimensional model or an object image of a three-dimensional model. Similarly, the generation unit 140 arranges icons indicating obstacles, fire, and smoke in the model data. In this way, by reflecting information about the detected object in the model data, current state model data is generated. By reflecting information about the detected object in the model data, the situation at the disaster site is reproduced in the virtual space. That is, the current state model data can be said to be a three-dimensional model in which the situation at the disaster site is reproduced. Note that the icon may be character information. For example, characters indicating the object may be arranged at a position on the model data corresponding to the position of the detected object.

[0073] In addition, the generation unit 140 may reflect an icon corresponding to the team member at a position on the model data corresponding to the position of the team member based on the obtained position information.

[0074] When there are a plurality of team member devices 200, information about the detected object detected based on the information obtained from each team member device 200 is reflected in the model data. When the detected object overlaps with the object detected based on the information obtained from another team member device 200, the generation unit 140 may reflect any of the detected objects in the model data.

[0075] Further, the generation unit 140 may generate a map based on the current situation model data. FIG. 12 is a diagram showing an example of a map based on the current situation model data. In the example of FIG. 12, a map based on the current situation model data in which information from a plurality of team member devices 200 is reflected is shown. Further, in the example of FIG. 12, the map shows not only the positions of the detected objects but also the positions of the team members.

[0076] The display control unit 150 causes the terminal device 300 to display the current situation model data. For example, the display control unit 150 causes the current situation model data to be displayed as a VR image on the head-mounted display of the terminal device 300. Since the current situation model data reproduces the situation at the disaster site, the user of the terminal device 300 can experience the situation at the disaster site in a virtual manner.

[0077] [Operation Example of Situation Awareness Support Device 100] Next, an example of the operation of the situation awareness support device 100 will be described with reference to FIG. 13. In this operation example, an example of generating current situation model data based on information obtained from one team member device 200 will be described. Also, in this operation example, model data corresponding to the buildings at the disaster site is specified in advance.

[0078] FIG. 13 is a flowchart for explaining an example of the operation of the situation awareness support device 100. First, the acquisition unit 110 acquires a photographed image, an infrared image, and the position information of the team member device 200 photographed by the team member device 200 (S101). For example, the acquisition unit 110 acquires signals related to the photographed image, the infrared image, and the position information from the team member device 200. Then, the acquisition unit 110 acquires the position information of the team member device 200 based on the signal related to the position information.

[0079] The specifying unit 120 searches for the position on the model data corresponding to the acquired position information (S102). Then, the specifying unit 120 specifies a partial area of the model data corresponding to the disaster site shown in the photographed image (S103). At this time, the specifying unit 120 specifies the partial area by performing alignment between the photographed image and a region within a predetermined range from the position on the model data hit by the search.

[0080] The detection unit 130 acquires an image of the specified partial region (S104). Further, the detection unit 130 generates a difference image between the image of the partial region and the captured image (S105). Then, the detection unit 130 detects an object based on the difference image (S106).

[0081] The generation unit 140 generates current state model data (S107). Specifically, the generation unit 140 generates the current state model data by reflecting information about the detected object in the model data.

[0082] The display control unit 150 causes the generated current state model data to be displayed on the terminal device 300 (S108).

[0083] When there are a plurality of team member devices 200, the situation awareness support device 100 may perform the processes of S101 to S107 for each team member device 200. As a result, information on a plurality of locations at the disaster site is reflected in one model data based on the information obtained from the plurality of team member devices 200. Further, the processes of S101 to S107 may be repeatedly performed. Thereby, the current state model data is updated at any time. Note that this operation example is merely an example, and the operation of the situation awareness support device 100 is not limited to this example.

[0084] As described above, the situation awareness support device 100 according to the second embodiment acquires a captured image of the disaster site, and identifies a partial region corresponding to the captured disaster site in the model data by aligning the captured image with the model data indicating the three-dimensional model representing the real space. Further, the situation awareness support device 100 detects an object based on the difference between the captured image and the image of the identified partial region. Then, the situation awareness support device 100 generates current state model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object, and causes the generated current state model data to be displayed on the display.

[0085] As a result, the situation awareness support device 100 can, for example, enable a rear supporter to understand the situation at the disaster site based on the three-dimensional model. Therefore, the situation awareness support device 100 can more intuitively enable the rear supporter to understand the situation at the disaster site compared to showing the situation at the disaster site by text or a flat map. That is, the situation awareness support device 100 according to the second embodiment can support the understanding of the disaster situation.

[0086] In addition, the situation awareness support device 100 according to the second embodiment may acquire the position information of the team member who took the captured image, and identify the partial area by aligning the captured image with the area of the model data corresponding to the acquired position information. As a result, the situation awareness support device 100 can reduce the computational load compared to the case of performing alignment with the captured image for all parts of the model data.

[0087] In addition, the situation awareness support device 100 according to the second embodiment may acquire an infrared image corresponding to the captured image, and detect, as a flame, a difference area corresponding to a location in the captured image where a temperature of a predetermined value or higher is indicated by the infrared image. As a result, the situation awareness support device 100 can detect the flame with higher accuracy compared to detecting the flame only from the captured image.

[0088] In addition, the situation awareness support device 100 according to the second embodiment may detect smoke based on the color of the difference area in the captured image and information on whether or not the edge of the difference area has been detected. As a result, the situation awareness support device 100 can detect smoke.

[0089] [Modification Example 1] When more information about the disaster site is available, the situation awareness support system 1000 may utilize such information.

[0090] Assume that the disaster site includes a predetermined building. At this time, if various sensors are installed in the building, the situation awareness support device 100 acquires the information detected by the sensors.

[0091] For example, assume that a smoke sensor is installed in a building. An example of the smoke sensor is a fire alarm. At this time, the acquisition unit 110 acquires sensor data from the smoke sensor. Then, the detection unit 130 detects smoke by using the sensor data of the smoke sensor existing within a predetermined range from the position where the captured image was taken, the color of the difference image, and information on whether an edge was detected in the difference image. For example, when detecting smoke when a predetermined value based on the color of the difference image and information on whether an edge was detected in the difference image is equal to or greater than a threshold value, the threshold value may be changed according to the value of the sensor data.

[0092] Also, for example, assume that a temperature sensor is installed in a building. At this time, the acquisition unit 110 acquires sensor data from the temperature sensor. Then, the detection unit 130 detects a flame by using the sensor data of the temperature sensor existing within a predetermined range from the position where the captured image was taken and the difference image. For example, the detection unit 130 may detect a flame for a location where there is a location showing red or a color close to red in the difference image and the temperature indicated by the sensor data is equal to or greater than a predetermined value.

[0093] <Third Embodiment> Next, the situation awareness support system according to the third embodiment will be described. In the third embodiment, an example of estimating the situation at the disaster site from the obtained information will be described. Note that descriptions of content overlapping with the first and second embodiments will be partially omitted.

[0094] In the third embodiment, an application example of the situation awareness support system in the case where a disaster accompanied by a fire occurs and team members are active in a predetermined building will be described, but this does not limit the applicable situations of the system.

[0095] [Details of Situation Awareness Support System 1001] FIG. 14 is a block diagram showing an example of the configuration of the situation awareness support system 1001 according to the third embodiment. The situation awareness support system 1001 includes at least the situation awareness support device 101. In the example of FIG. 14, the situation awareness support system 1001 includes the situation awareness support device 101, the team member device 201, the terminal device 300, and the communication device. Each of the situation awareness support device 101 and the team member device 201 may have functions to be described below in addition to the respective functions of the situation awareness support device 100 and the team member device 200.

[0096] The team member device 201 includes a visible light imaging unit 210, an infrared imaging unit 220, a position information communication unit 230, and an output unit 240. A plurality of team member devices 201 may exist in the same manner as the team member device 200. That is, team member devices 201-1, 201-2, ···, 201-n (n is a natural number) may exist.

[0097] The output unit 240 has a function of outputting information generated by the situation awareness support device 101. The output unit 240 is, for example, a display and a projector.

[0098] FIG. 15 is a diagram showing an example of the team member device 201. In the example of FIG. 15, the team member device 201 is shown as a full-face mask-type device. For example, a team member wears such a mask-type device. As shown in FIG. 15, the output unit 240 may be a display that also serves as the function of the goggles of the mask. Without being limited to this, the output unit 240 may be a projector that projects an image onto the goggle portion of the mask. Also, in the example of FIG. 15, the team member device 201 may have the function of an air respirator mask.

[0099] The situation awareness support device 101 includes an acquisition unit 111, a specification unit 120, a detection unit 130, a generation unit 141, a display control unit 151, an estimation unit 160, an activity information output unit 170, and a route calculation unit 180. The situation awareness support device 101 also has a storage device 190.

[0100] The situation awareness support device 101 estimates the origin of a fire from the information obtained from the team member device 201 and the current situation model data. Specifically, the estimation unit 160 estimates the position of the origin of the fire at the disaster site. At this time, the estimation unit 160 uses the position of the flame detected by the detection unit 130, the temperature data at the disaster site, and the airflow analysis data and heat analysis data of the building.

[0101] An example of the temperature data at the disaster site is a temperature map of the disaster site. The temperature data is acquired, for example, by the acquisition unit 111. The temperature data may be information based on an infrared image taken by the team member device 201, for example. In this case, the temperature data may be information generated based on the temperature information extracted from the infrared images acquired from each of the team member devices 201. When temperature sensors are installed in the building, the temperature data may be information based on the sensor data acquired by the temperature sensors. Also, in order to manage the physical condition of the team members operating at the disaster site, temperature sensors may be attached to the clothing of the team members. In such a case, the temperature data may be information based on the sensor data acquired by the attached temperature sensors. The temperature data is reflected in the current situation model data by the generation unit 141.

[0102] The airflow analysis data and the heat analysis data are information included in the model data. The airflow analysis data is simulation data regarding the flow of air in the building. When designing the building, the installation positions of ventilation fans, air conditioning equipment, etc. may be considered in view of the airflow in the building. The airflow analysis data may be data created in advance at the time of designing such a building. The heat analysis data is simulation data regarding the transfer of heat in the building. For example, the heat analysis data may be data capable of simulating changes in the temperature distribution inside the building when there is a heat source. Similar to the airflow analysis data, the heat analysis data may also be data created in advance at the time of designing the building.

[0103] The estimation unit 160 uses the airflow analysis data and the heat analysis data to simulate, from the current position of the flame and the current temperature data, how the fire has spread to reach the current fire situation. Through this simulation, the position of the fire source is estimated.

[0104] In this way, the estimation unit 160 estimates the position of the fire source at the disaster site based on the detected position of the flame, the temperature data, the airflow analysis data, and the heat analysis data. The estimation unit 160 is an example of an estimation means.

[0105] The generation unit 141 generates current situation model data in which an icon indicating the fire source is arranged at the position on the model data corresponding to the estimated position of the fire source. At this time, the generation unit 141 reflects and updates the information indicating the fire source in the current situation model data generated so far.

[0106] In addition, the situation grasping support device 101 estimates how the fire will spread in the future. Specifically, after estimating the position of the fire source, the estimation unit 160 estimates how the fire will spread at the disaster site. The ways the fire spreads include the degree of progress of the flame at the scene over time and the degree of smoke filling at the scene over time.

[0107] For example, the estimation unit 160 uses the airflow analysis data and the heat analysis data to simulate how the fire will spread in the future from the position of the fire source, the current position of the flame, and the current temperature data. Through this simulation, the degree of progress of the flame at the scene over time, such as in which direction and at what speed the scene will burn and spread, is estimated. Also, the degree of smoke filling at the scene over time, such as in which direction and at what speed the smoke will fill the scene, is estimated.

[0108] In this way, the estimation unit 160 estimates how the fire will spread at the disaster site based on the estimated position of the fire source, the detected position of the flame, the temperature data, the airflow analysis data, and the heat analysis data.

[0109] Furthermore, the estimation unit 160 may estimate the passage of time of the person detected at the disaster site. If it is possible to simulate how the fire has spread to the current fire situation, it is possible to estimate the time elapsed from the occurrence of the fire to the current situation. And the person detected at the disaster site is highly likely to be left behind due to slow evacuation. Therefore, the estimation unit 160 can estimate the time (i.e., the passage of time) that the detected person has been left behind at the disaster site from such a simulation.

[0110] In addition, the situation awareness support device 101 has a navigation function for the team members at the disaster site. For example, the situation awareness support device 101 shows the evacuation route or the position where firefighting should be carried out to the team members.

[0111] First, the activity information output unit 170 outputs activity information. The activity information is information indicating the activities that the team members should perform at the disaster site. Examples of activities are firefighting activities, rescue activities, and evacuation.

[0112] The activity information output unit 170 outputs the activity information based on the estimated spread of the fire and the current situation model data. At this time, the activity information output unit 170 outputs the activity information using a learning model in which predetermined learning has been performed. The learning model is a learning model that has learned the relationship between the situation of the disaster site and the spread of the fire and the guidelines for activities at the disaster site. The guidelines for activities at the disaster site may indicate the activities to be performed according to various situations.

[0113] For example, the learning model may output information such as whether fire extinguishing activities should be carried out based on the fire situation, such as the position of the fire and the way the fire spreads, and if fire extinguishing activities should be carried out, at which position the fire extinguishing activities should be carried out (fire extinguishing position). Specifically, the activity information output unit 170 uses the current situation model data and the information indicating the way the fire spreads estimated by the estimation unit 160 as inputs to the learning model, and outputs the information output by the learning model as activity information. In this case, the current situation model data includes information indicating the position of the fire source, information indicating the position of the flame, temperature data, and the like.

[0114] In addition, the learning model may output information such as whether evacuation should be carried out based on the fire situation. Similarly in this case, the activity information output unit 170 uses the current situation model data and the information indicating the way the fire spreads estimated as inputs to the learning model, and outputs the information output by the learning model as activity information.

[0115] In addition, the learning model may output information such as whether people should be rescued based on the fire situation and the passage of time of the people at the disaster site (i.e., the people left behind). Similarly in this case, the activity information output unit 170 uses the current situation model data and the information indicating the way the fire spreads estimated as inputs to the learning model, and outputs the information output by the learning model as activity information.

[0116] In this way, the activity information output unit 170 uses a learning model that has learned the relationship between the situation at the disaster site and the way the fire spreads and the guidelines for activities at the disaster site, and based on the current situation model data and the estimated way the fire spreads, outputs activity information indicating the activities that the team members should carry out at the photographed disaster site. The activity information output unit 170 is an example of activity information output means.

[0117] The route calculation unit 180 calculates the routes that the team members can take based on the activity information. The route calculation unit 180 is an example of route calculation means.

[0118] For example, assume that the fire extinguishing position is indicated in the activity information. In this case, the route calculation unit 180 calculates a route from the position of the team member to the fire extinguishing position using the current model data. The fire extinguishing activity may be required to be performed against the fire source. In that case, the fire extinguishing position is near the fire source. At this time, the route calculation unit 180 calculates a route from the position of the team member to the position of the fire source. The route to the fire extinguishing position is referred to as the fire extinguishing route. When an obstacle is detected by the detection unit 130, the route calculation unit 180 may calculate a fire extinguishing route that avoids the obstacle.

[0119] Also, for example, assume that the activity information includes information indicating that evacuation should be carried out. In this case, the route calculation unit 180 calculates an evacuation route from the position of the team member based on the current model data and information indicating the evacuation route of the building included in the model data in advance. Assume that there are places where people cannot pass due to obstacles and flames detected here. In such a case, the route calculation unit 180 may identify a place where people cannot pass based on the detected object and calculate an evacuation route that avoids the impassable place from the position of the team member.

[0120] When the building at the disaster site has multiple floors, it may be possible to enter and exit through windows on the second floor and above using a ladder truck or the like. In such a case, the route calculation unit 180 may calculate an evacuation route to the accessible window. The information indicating the accessible window is shown in the current model data. For example, the information indicating the accessible window is input by the user using the terminal device 300. Then, the information indicating the accessible window is reflected in the current model data by the generation unit 141.

[0121] Also, assume that the activity information includes information indicating that a person should be rescued. In this case, the route calculation unit 180 calculates, as a rescue route, an evacuation route from the team member via the person to be rescued (the person in need of first aid) using the current model data.

[0122] FIG. 16 is a diagram showing an example of a route. Specifically, FIG. 16 is a diagram showing various routes from team members in a map based on current model data. For example, the map is generated by the generation unit 141. For example, in the case of a fire extinguishing route, the route from each team member to the position of the fire source, which is the fire extinguishing position, is shown.

[0123] The generation unit 141 generates instruction information indicating the direction for heading to the place to which the team member should head. At this time, the generation unit 141 generates corresponding instruction information for each team member. For example, it is assumed that the fire extinguishing position is shown in the activity information. In this case, the generation unit 141 generates, for each team member, instruction information indicating the moving direction from the perspective of the team member to the fire extinguishing position based on the fire extinguishing route from the position of the team member.

[0124] Also, it is assumed that the activity information includes information indicating that evacuation should be carried out. In this case, the generation unit 141 generates, for each team member, instruction information indicating the moving direction for evacuation from the perspective of the team member based on the evacuation route from the position of the team member.

[0125] Also, it is assumed that the activity information includes information indicating that a person should be rescued. In this case, the generation unit 141 generates, for each team member, instruction information indicating the moving direction for rescuing the person from the perspective of the team member based on the rescue route from the position of the team member. Note that the generation unit 141 may generate instruction information indicating the moving direction for rescuing the person only for team members whose distance from the person in need of first aid is less than a predetermined value based on the position information of each team member.

[0126] The display control unit 151 causes the instruction information to be displayed on the goggle portion of the mask worn by the team member. Specifically, when the output unit 240 is a projector, the display control unit 151 causes the output unit 240 to project the instruction information onto the goggle portion. Also, when the output unit 240 is a display arranged on the goggle portion, the display control unit 151 causes the output unit 240 to display the instruction information.

[0127] FIG. 17 is a diagram showing an example of instruction information. In FIG. 17, the view through the goggles from a team member is shown. Instruction information is displayed on the goggles part. In the example of FIG. 17, if it is instruction information regarding evacuation, a downward arrow is shown together with the character information "Evacuation Route". This indicates that the evacuation route is in the forward direction as seen from the team member. Similarly, if it is instruction information regarding fire extinguishing activities, a leftward arrow is shown together with the character information "Fire Extinguishing Location". Also, if it is instruction information regarding rescue activities, a graphic that emphasizes the person in need of rescue and an arrow are shown together with the character information "Rescue Route" and "Person in Need of First Aid". In this way, by displaying the instruction information on the goggles part, it appears to the team member that the instruction information is superimposed on the real space. Thereby, the team member can recognize the direction to go.

[0128] Note that in the example of FIG. 17, the instruction information for each of the fire extinguishing activity, evacuation, and rescue activity is shown, but only any one of the instruction information may be displayed. The instruction information is information corresponding to each team member. Therefore, for example, only the instruction information regarding the fire extinguishing activity may be displayed for one team member, and only the instruction information regarding the rescue activity may be displayed for another team member.

[0129] Also, the activity information may include information indicating not to rescue the detected person. This is because the detected person may already be dead. In such a case, the generation unit 141 generates instruction information including information indicating that the rescue of the person is unnecessary. Then, the display control unit 151 causes the fact that the rescue of the person is unnecessary to be displayed.

[0130] [Operation Example of Situation Awareness Support Device 101] Next, an example of the operation of the situation awareness support device 101 will be described with reference to FIG. 18. In this operation example, it is assumed that the processing described in FIG. 13 has already been performed. That is, the current situation model data has been generated. Further, the model data preliminarily includes at least information indicating the evacuation route of the building, airflow analysis data, and heat analysis data. Further, the current situation model data shows the position of the fire, the position of the smoke, and the position of the team members.

[0131] FIG. 18 is a flowchart for explaining an example of the operation of the situation awareness support device 101. The acquisition unit 111 acquires temperature data (S201). For example, the acquisition unit 111 acquires a temperature map of the disaster site based on an infrared image captured by the team member device 201. The estimation unit 160 estimates the position of the fire source (S202). For example, the estimation unit 160 estimates the position of the fire source by simulating how the current fire situation has developed based on the current position of the fire and the current temperature data using the airflow analysis data and the heat analysis data. At this time, the generation unit 141 reflects the position of the fire source in the current situation model data.

[0132] Then, the estimation unit 160 estimates the spreading pattern of the fire (S203). For example, the estimation unit 160 performs a simulation of how the fire will spread in the future based on the position of the fire source, the current position of the fire, and the current temperature data using the airflow analysis data and the heat analysis data. As a result, the estimation unit 160 estimates the spreading pattern of the fire, including, for example, the progress of the fire at the scene over time and the degree of smoke filling at the scene over time.

[0133] The activity information output unit 170 outputs activity information based on the current situation model data and the estimated spreading pattern of the fire (S204). At this time, the activity information output unit 170 uses, for example, a learning model that has learned the relationship between the situation at the disaster site and the spreading pattern of the fire and the guidelines for activities at the disaster site.

[0134] The route calculation unit 180 calculates the routes that the team members can take based on the activity information (S205). For example, when the activity information includes information indicating that evacuation should be carried out, the route calculation unit 180 calculates the evacuation route from the positions of the team members based on the current situation model data and the information indicating the evacuation routes of the buildings.

[0135] The generation unit 141 generates instruction information for each team member (S206). For example, the generation unit 141 generates, for each team member, instruction information indicating the moving direction for evacuation from the perspective of the team member based on the evacuation route from the position of the team member.

[0136] Then, the display control unit 151 causes the output unit 240 of each team member device 201 to display the instruction information. For example, the display control unit 151 causes the goggle portion of the mask worn by the team member to display the instruction information.

[0137] Note that the processes of S201 to S207 may be repeated. As a result, the current situation model data is updated at any time, and the instruction information displayed to the team members is also updated. This operation example is merely an example, and the operation of the situation awareness support device 101 is not limited to this example.

[0138] As described above, the situation awareness support device 101 according to the third embodiment estimates the position of the fire source at the disaster site based on the position of the flame, the temperature data, the airflow analysis data, and the heat analysis data. Then, the situation awareness support device 101 generates the current situation model data in which an icon indicating the fire source is arranged at the position on the model data corresponding to the estimated position of the fire source. Thereby, even when the position of the fire source is unknown, the situation awareness support device 101 can show the estimated location of the fire source to the user.

[0139] In addition, the situation awareness support device 101 according to the third embodiment can also estimate the spreading manner of the fire at the disaster site based on the estimated position of the fire source, the detected position of the flame, the temperature data, the airflow analysis data, and the heat analysis data. Thereby, the situation awareness support device 101 can show the user the future spreading manner of the fire.

[0140] In addition, the situation awareness support device 101 of the third embodiment outputs activity information indicating the activities that team members should perform at the disaster site, calculates the possible routes that team members can take based on the activity information, generates instruction information indicating the direction for the team members to head towards the place they should go from their positions, and displays the instruction information on the goggle part of the mask worn by the team members. Thereby, the situation awareness support device 101 can intuitively show the actions that should be taken to the team members at the disaster site.

[0141] At this time, for example, assuming that activity information indicating evacuation from a building is output, the situation awareness support device 101 may identify, on the current situation model data, locations where people cannot pass based on the detected objects, and calculate an evacuation route that avoids the impassable locations from the positions of the team members. Then, the situation awareness support device 101 may generate instruction information indicating the direction for evacuation from the positions of the team members according to the calculated evacuation route.

[0142] <Example of the hardware configuration of the situation awareness support device> The hardware constituting the situation awareness support devices of the above-described first, second, and third embodiments will be described. FIG. 19 is a block diagram showing an example of the hardware configuration of a computer device constituting the situation awareness support device in each embodiment. In the computer device 90, the situation awareness support device and the situation awareness support method described in each embodiment and each modification are realized. For example, each of the situation awareness support device, the team member device, and the terminal device described in each embodiment and each modification may have the hardware configuration shown in FIG. 19.

[0143] As shown in FIG. 19, the computer device 90 includes a processor 91, a RAM (Random Access Memory) 92, a ROM (Read Only Memory) 93, a storage device 94, an input / output interface 95, a bus 96, and a drive device 97. Note that the situation awareness support system may be realized by a plurality of electric circuits.

[0144] The memory device 94 stores a program (computer program) 98. The processor 91 executes the program 98 of this situation grasping support system using the RAM 92. Specifically, for example, the program 98 includes programs that cause a computer to execute the processes shown in FIGS. 3, 13, and 18. When the processor 91 executes the program 98, the functions of each component of this situation grasping support system are realized. The program 98 may be stored in the ROM 93. Also, the program 98 may be recorded on the storage medium 80 and read out using the drive device 97, or may be transmitted from an external device (not shown) to the computer device 90 via a network (not shown).

[0145] The input / output interface 95 exchanges data with peripheral devices (such as a keyboard, a mouse, and a display device) 99. The input / output interface 95 functions as a means for acquiring or outputting data. The bus 96 connects each component.

[0146] Note that there are various modification examples for the method of realizing the situation grasping support system. For example, each component included in the situation grasping support system can be realized as a dedicated device. Also, each component included in the situation grasping support system can be realized based on a combination of a plurality of devices.

[0147] A processing method of recording a program for realizing each component in the functions of each embodiment on a storage medium, reading out the program recorded on the storage medium as code, and executing it on a computer is also included in the scope of each embodiment. That is, a computer-readable storage medium is also included in the scope of each embodiment. Also, the storage medium on which the above-described program is recorded and the program itself are included in each embodiment.

[0148] The memory medium is, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD (Compact Disc)-ROM, a magnetic tape, a non-volatile memory card, or a ROM, but is not limited to this example. Also, the program recorded on the memory medium is not limited to a program that executes processing alone, and programs that operate on an OS (Operating System) and execute processing in cooperation with other software and the functions of expansion boards are also included in the scope of each embodiment.

[0149] As described above, the present invention has been described with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0150] Also, the above embodiments and modifications can be combined as appropriate.

Explanation of Signs

[0151] 100, 101 Situation Grasping Support Device 110, 111 Acquisition Unit 120 Identification Unit 130 Detection Unit 140, 141 Generation Unit 150, 151 Display Control Unit 160 Estimation Unit 170 Activity Information Output Unit 180 Route Calculation Unit 190 Storage Device 200, 201 Team Member Device 300 Terminal Device

Claims

1. An acquisition means for acquiring a captured image of a disaster site; A specifying means for specifying a partial area corresponding to the photographed disaster site among the model data by aligning the captured image with model data showing a three-dimensional model of the real space; A detection means for detecting an object based on the difference between the captured image and the image of the specified partial area; A generation means for generating current situation model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object; A display control means for causing the generated current situation model data to be displayed on a display, A situation understanding support device.

2. The acquisition means acquires position information of a team member who captured the captured image, The specifying means specifies the partial area by aligning the captured image with the area of the model data corresponding to the acquired position information. The situation understanding support device according to Claim 1.

3. The acquisition means acquires an infrared image corresponding to the captured image, The detection means detects a portion of the difference corresponding to a location in the captured image that exhibits a temperature of a predetermined value or higher according to the infrared image as a flame. The situation understanding support device according to Claim 1.

4. The detection means detects smoke based on the color of the portion of the difference in the captured image and information on whether or not an edge of the portion of the difference has been detected. The situation understanding support device according to Claim 1.

5. Comprising an estimation means, The disaster site includes inside a building, The model data includes airflow analysis data which is simulation data regarding the flow of air in a building and heat analysis data which is simulation data regarding the transfer of heat in a building, The acquisition means acquires temperature data of the photographed disaster site, The detection means detects a flame, The estimation means estimates the position of the origin of the fire at the disaster site based on the position of the detected flame, the temperature data, the airflow analysis data, and the heat analysis data, The generation means generates the current situation model data in which an icon indicating the origin of the fire is arranged at a position on the model data corresponding to the estimated position of the origin of the fire. The situation understanding support device according to Claim 1.

6. The estimation means estimates the way the fire spreads at the disaster site based on the estimated position of the fire origin, the detected position of the flame, the temperature data, the airflow analysis data, and the heat analysis data. The situation awareness support device according to claim 5.

7. The current situation model data includes at least information indicating the position of the fire origin, the position of the flame, the temperature data, and the position of the team members. Using a learning model that has learned the relationship between the situation of the disaster site and the way the fire spreads, and the guidelines for activities at the disaster site, based on the current situation model data and the estimated way the fire spreads, activity information output means for outputting activity information indicating the activities that team members should perform at the photographed disaster site. Path calculation means for calculating the paths that team members can take based on the activity information. The generation means generates instruction information indicating the direction for heading to the place to be headed from the position of the team member. The display control means displays the instruction information on the goggle part of the mask worn by the team member. The situation awareness support device according to claim 6.

8. The photographed disaster site is inside a building. The model data includes information indicating the evacuation route of the building. When the activity information output means outputs the activity information indicating evacuation from the building. The path calculation means, on the current situation model data. Based on the detected object, identify the locations where people cannot pass through. Calculate an evacuation route that avoids the impassable locations from the position of the team member. The generation means generates the instruction information indicating the direction for evacuation from the position of the team member according to the calculated evacuation route. The situation awareness support device according to claim 7.

9. Obtain a photographed image of the disaster site. By aligning the photographed image with the model data showing the three-dimensional model representing the real space, identify the partial area corresponding to the photographed disaster site among the model data. Detect an object based on the difference between the photographed image and the image of the identified partial area. Generate current situation model data in which icons corresponding to the object are arranged at the positions on the model data corresponding to the positions of the detected objects. Display the generated current situation model data on a display. Situation awareness support method.

10. A process of obtaining a photographed image of the disaster site. A process of identifying a partial region corresponding to the photographed disaster site among the model data by aligning the photographed image with model data showing a three-dimensional model in which the real space is represented. A process of detecting an object based on the difference between the photographed image and the image of the identified partial region. A process of generating current state model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object. A process of causing a computer to execute a process of displaying the generated current state model data on a display. Program.

Images