Situation determination aiding device, situation determination aiding method, and program

Abstract

To provide a situation determination aiding device that can aid in determination of a disaster situation.SOLUTION: A situation determination aiding device according to an aspect of the present disclosure comprises: acquisition means for acquiring a camera image capturing a disaster site; identification means for identifying a partial region corresponding to the captured disaster site from model data indicating a three-dimentional model in which a real space is represented by aligning the camera image with the model data; detection means for detecting an object on the basis of a difference between the camera image and an image of the identified partial region; generation means for generating a current model data in which an icon corresponding to the object is arranged at a position on the model data corresponding to the position of the detected object; and display control means for displaying the generated current model data on a display.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a technology for understanding the situation at a disaster site. [Background technology]

[0002] When a disaster such as a fire occurs, the disaster response team and their supporters (for example, operators who give instructions to the disaster response team) are required to understand the situation at the disaster site.

[0003] In relation to understanding the situation at a fire scene, Patent Document 1 discloses a technology for detecting dangerous areas. For example, Patent Document 1 discloses a technology for identifying dangerous areas at the scene from images captured by a camera module worn by a firefighter and displaying an image showing the location of the dangerous areas. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-169837 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology disclosed in Patent Document 1, the locations of dangerous areas are simply shown on a two-dimensional map. In this case, there is a risk that rear support personnel who are not directly observing the site may not be able to fully grasp the situation. This leaves room for improvement.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and one of its objectives is to provide a situation understanding support device etc. that can support understanding of a disaster situation. [Means for solving the problem]

[0007] A situation understanding support device according to one aspect of the present disclosure includes an acquisition means for acquiring a photographed image of a disaster site, an identification means for identifying a partial area of the model data that corresponds to the photographed disaster site by aligning the photographed image with model data that indicates a three-dimensional model that represents real space, a detection means for detecting an object based on the difference between the photographed image and an image of the identified partial area, a generation means for generating current model data in which an icon corresponding to the detected object is placed in the model data at a position that corresponds to the position of the detected object, and a display control means for displaying the generated current model data on a display.

[0008] A situation understanding support method according to one aspect of the present disclosure acquires an image of a disaster site, aligns the image with model data indicating a three-dimensional model that represents real space, identifies a partial area of the model data that corresponds to the image of the disaster site, detects an object based on the difference between the image of the image and an image of the identified partial area, generates current-state model data in which an icon corresponding to the object is placed at a position on the model data that corresponds to the position of the detected object, and displays the generated current-state model data on a display.

[0009] A program according to one aspect of the present disclosure causes a computer to perform the following processes: acquiring an image of a disaster site; identifying a partial area of the model data that corresponds to the image of the disaster site by aligning the image with model data that indicates a three-dimensional model that represents real space; detecting an object based on the difference between the image of the image and an image of the identified partial area; generating current-state model data in which an icon corresponding to the detected object is placed in the model data at a position that corresponds to the position of the detected object; and displaying the generated current-state model data on a display. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to assist in understanding the disaster situation. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of a situation understanding support system according to a first embodiment; [Figure 2] 1 is a block diagram illustrating an example of a functional configuration of a situation grasp assistance device according to a first embodiment. [Figure 3] 4 is a flowchart illustrating an example of an operation of the situation grasp support device according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of a situation grasping support system according to a second embodiment. [Figure 5] FIG. 10 is a block diagram illustrating an example of the configuration of a situation understanding support system according to a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of a captured image according to the second embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of an image of a partial region according to the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a first example of a difference image according to the second embodiment. [Figure 9] FIG. 10 is a diagram showing another example of a captured image according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating a second example of a difference image according to the second embodiment. [Figure 11] FIG. 10 is a diagram illustrating a third example of a difference image according to the second embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a map based on current model data according to the second embodiment. [Figure 13] 10 is a flowchart illustrating an example of an operation of the situation grasp support device according to the second embodiment. [Figure 14] FIG. 10 is a block diagram illustrating an example of the configuration of a situation understanding support system according to a third embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of a worker device according to a third embodiment. [Figure 16] FIG. 11 is a diagram illustrating an example of a route according to a third embodiment. [Figure 17] FIG. 13 is a diagram illustrating an example of instruction information according to the third embodiment. [Figure 18] 10 is a flowchart illustrating an example of an operation of the situation grasp assistance device according to the third embodiment. [Figure 19] 1 is a block diagram showing an example of a hardware configuration of a computer device that realizes a situation grasp assistance device according to a first, second, and third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] First Embodiment An overview of the situation recognition support device according to the first embodiment will be described.

[0014] Fig. 1 is a diagram schematically illustrating an example of the configuration of a situation understanding support system 1000. The situation understanding support system 1000 includes at least a situation understanding support device 100. In the example of Fig. 1, the situation understanding support system 1000 includes the situation understanding support device 100, a team member device 200, and a terminal device 300. The situation understanding support device 100 is communicably connected to the team member device 200 and the terminal device 300 via a wireless or wired network.

[0015] The situation understanding support system 1000 according to the present disclosure is used in response to the occurrence of a disaster. One example of a situation in which the system is used is when rescue workers perform rescue operations or firefighting operations at a disaster site. Examples of such workers include ambulance crews, firefighters, and rescue workers. In this case, a rear supporter understands the situation at the disaster site based on reports from the workers and gives instructions to the workers. The rear supporter is also called an operator or a commander.

[0016] A team member at the scene carries a team member device 200. The team member device 200 has the function of performing predetermined measurements. For example, the team member device 200 may be a device worn by the team member. That is, the location of measurements by the team member device 200 may change as the team member moves. The terminal device 300 is a device used by a rear supporter. The terminal device 300 has at least a display. The situation assessment support device 100 performs various processes using information measured by the team member device 200. The situation assessment support device 100 then displays information about the situation at the disaster site according to the results of the various processes on the display of the terminal device 300. In this way, the situation assessment support system 1000 is a system that can allow a rear supporter to understand the situation at the disaster site by using information at the disaster site acquired by the team member device 200.

[0017] 1, for the sake of convenience, the situation understanding support device 100, the team member device 200, and the terminal device 300 are each depicted as being configured as a single device. However, the configurations of the situation understanding support device 100, the team member device 200, and the terminal device 300 are not limited to this example. That is, each of the situation understanding support device 100, the team member device 200, and the terminal device 300 may be configured as a plurality of devices. Furthermore, there may be a plurality of team member devices 200 and a plurality of terminal devices 300.

[0018] The worker device 200 has a function of performing predetermined measurements. One example of the predetermined measurements is a photography function. That is, the worker device 200 is equipped with a camera module that can photograph the surroundings.

[0019] The terminal device 300 may be a personal computer. The terminal device 300 may have various input / output devices. Examples of the input / output devices include a keyboard, a microphone, a camera, a display, and a speaker. Examples of the terminal device 300 include a personal computer and a head-mounted display connectable to a personal computer.

[0020] The situation understanding support device 100 may be a server device.

[0021] Next, a description will be given of an example of the functional configuration of the situation understanding support device 100. Fig. 2 is a block diagram showing an example of the functional configuration of the situation understanding support device 100. As shown in Fig. 2, the situation understanding support device 100 includes an acquisition unit 110, an identification unit 120, a detection unit 130, a generation unit 140, and a display control unit 150.

[0022] The acquisition unit 110 acquires various types of information from the team member device 200. For example, the team member device 200 captures images of the surroundings at the disaster site. The acquisition unit 110 acquires the images captured by the team member device 200. In this way, the acquisition unit 110 acquires the images of the disaster site. The acquisition unit 110 is an example of an acquisition means.

[0023] The identification unit 120 performs processing on the model data using the captured image. The model data is a three-dimensional model that represents a real space. More specifically, the model data is a three-dimensional model in which buildings and terrain that exist in reality are reproduced in a virtual space. The model data may be stored in a storage device included in the situation understanding support device 100, or may be stored in an external storage device that is communicably connected to the situation understanding support device 100.

[0024] The identification unit 120 identifies an area of the model data that corresponds to the disaster site based on the captured image. For example, the identification unit 120 aligns the captured image with the model data by using the features of buildings shown in the captured image. The partial area of the model data identified by this alignment becomes the area that corresponds to the disaster site.

[0025] In this way, the identification unit 120 identifies a partial area corresponding to the disaster site in the model data by aligning the captured image with the model data indicating a three-dimensional model that expresses real space. The identification unit 120 is an example of an identification means.

[0026] The detection unit 130 detects an object. Specifically, the detection unit 130 calculates the difference between the captured image and the image of the partial region of the model data identified by the identification means. The detection unit 130 then detects an object in the calculated difference portion. Examples of objects include a person in need of rescue, an obstacle, flames, and smoke.

[0027] In this way, the detection unit 130 detects an object based on the difference between the captured image and the image of the specified partial region. The detection unit 130 is an example of a detection means.

[0028] The generation unit 140 generates model data that reflects information about the detected object. Specifically, the generation unit 140 identifies a position in the model data that corresponds to the position in the captured image of the object detected by the detection unit 130. Then, the generation unit 140 reflects information indicating the detected object at the identified position in the model data. For example, assume that the detection unit 130 detects a flame. At this time, the generation unit 140 identifies a position in the model data that corresponds to the position of the detected flame in the captured image. Then, the generation unit 140 adds an icon indicating the flame to the identified position in the model data. The icon may be an image displayed in a virtual space. For example, the icon is an object image. In this way, information about the detected object is reflected in the model data. The model data that reflects information about the detected object is referred to as current model data. Note that the method of reflecting information about the object in the model data is not limited to this example.

[0029] In this way, the generating unit 140 generates current model data in which an icon corresponding to the detected object is placed at a position on the model data that corresponds to the position of the object. The generating unit 140 is an example of a generating means.

[0030] The display control unit 150 displays the current model data. Specifically, the display control unit 150 displays the current model data on a display included in the terminal device 300. The current model data may be displayed as a VR (Vertial Reality) image.

[0031] Next, an example of the operation of the situation grasp support device 100 will be described with reference to Fig. 3. In this disclosure, each step in a flowchart will be represented by a number assigned to the step, such as "S1".

[0032] FIG. 3 is a flowchart illustrating an example of the operation of the situation understanding support device 100. The acquisition unit 110 acquires a photographed image of a disaster site (S1). The identification unit 120 identifies a partial area of the model data corresponding to the photographed disaster site by aligning the photographed image with model data indicating a three-dimensional model that represents real space (S2). The detection unit 130 detects an object based on the difference between the photographed image and an image of the identified partial area (S3). The generation unit 140 generates current model data in which an icon corresponding to the object is placed in a position on the model data that corresponds to the position of the detected object (S4). The display control unit 150 displays the generated current model data on a display (S5).

[0033] In this way, the situation understanding support device 100 of the first embodiment acquires a photographed image of a disaster site, and identifies a partial area of the model data corresponding to the photographed disaster site by aligning the photographed image with model data that indicates a three-dimensional model that represents real space. The situation understanding support device 100 also detects an object based on the difference between the photographed image and an image of the identified partial area. The situation understanding support device 100 then generates current state model data in which an icon corresponding to the detected object is placed in the model data at a position corresponding to the position of the detected object, and displays the generated current state model data on a display.

[0034] As a result, the situation understanding support device 100 can allow, for example, a rear supporter to grasp the situation at a disaster site using a three-dimensional model. Therefore, the situation understanding support device 100 can allow the situation at a disaster site to be grasped intuitively, compared to when the situation at a disaster site is shown using text or a two-dimensional map. In other words, the situation understanding support device 100 of the first embodiment can support the grasping of the disaster situation.

[0035] <Second embodiment> Next, a situation awareness support system according to a second embodiment will be described. In the second embodiment, a further example of the situation awareness support system described in the first embodiment will be described. Note that some of the content overlapping with the first embodiment will not be described.

[0036] In the second embodiment, an example of application of the situation awareness support system when a disaster occurs and personnel are operating within a designated building is described, but this does not limit the situations in which the system can be applied.

[0037] FIG. 4 is a diagram schematically illustrating an example of the configuration of the situation understanding support system 1000. FIG. 4 is a specific example of the situation understanding support system 1000 shown in FIG. 1. In the example of FIG. 4, there are team member devices 200-1, 200-2, . . . , 200-n (n is a natural number). Hereinafter, when there is no need to distinguish between the team member devices 200-1, 200-2, . . . , 200-n, they will simply be referred to as team member devices 200. Furthermore, as shown in FIG. 4, the situation understanding support system 1000 may include a communication device. The communication device is communicably connected to the team member device 200 and the situation understanding support device 100. There may be multiple communication devices. Details of the communication devices will be described later.

[0038] 4, the terminal device 300 includes a computer and a head-mounted display. In this case, a rear supporter wears the head-mounted display. The current state model data generated by the situation understanding support device 100 is output to the head-mounted display, allowing the rear supporter to simulate a disaster site situation.

[0039] [Details of the Situation Awareness Support System 1000] FIG. 5 is a block diagram showing an example of the configuration of the situation understanding support system 1000. As shown in FIG.

[0040] The team member device 200 includes a visible light imaging unit 210, an infrared imaging unit 220, and a position information communication unit 230. The visible light imaging unit 210 captures images and generates captured images. The visible light imaging unit 210 is implemented by a camera module that uses visible light. In short, the visible light imaging unit 210 receives visible light and generates visible images. In this disclosure, the visible image captured by the visible light imaging unit 210 is simply referred to as a captured image. The visible light imaging unit 210 transmits the captured image to the situation understanding support device 100.

[0041] The infrared imaging unit 220 captures an image and generates an infrared image. The infrared imaging unit 220 is realized by a camera module that uses infrared rays. In short, the infrared imaging unit 220 receives infrared light and generates an infrared image. At this time, the infrared image is captured at approximately the same angle of view as the captured image. In other words, the infrared imaging unit 220 generates an infrared image corresponding to the captured image. The infrared imaging unit 220 transmits the infrared image to the situation understanding support device 100.

[0042] The visible light imaging unit 210 and the infrared imaging unit 220 may associate images captured at the same time and transmit them to the situation understanding support device 100. In other words, the team member device 200 may associate an image and an infrared image captured at the same time and transmit them to the situation understanding support device 100.

[0043] The position information communication unit 230 transmits and receives signals for acquiring position information. As an example, the position information communication unit 230 receives signals related to position information using a beacon. In this case, the communication device is a beacon terminal. Then, multiple communication devices are installed in buildings at the disaster site. For example, the communication devices transmit unique IDs (IDentification), which are an example of signals related to position information, via short-range wireless communication such as Bluetooth (registered trademark). The unique ID is information that differs for each communication device and is associated with information indicating the location where the communication device is installed. The position information communication unit 230 receives the signals related to position information. Then, the position information communication unit 230 transmits the received signals related to position information to the situation understanding support device 100. The situation understanding support device 100 acquires the position information of the team member device 200 based on the signals related to position information acquired from the team member device 200. Note that the position information communication unit 230 may receive signals related to position information using Wi-Fi (registered trademark) instead of a beacon. In this case, the communication devices are access points.

[0044] Furthermore, without being limited to this example, the position information communication unit 230 may receive a signal related to position information using a global navigation satellite system (GNSS) such as a global positioning system (GPS). In this case, the situation understanding support device 100 acquires the signal received by the position information communication unit 230 as the position information of the team member device 200.

[0045] The team member device 200 may be a wearable device that is attached to each team member. For example, the team member device 200 is attached to the clothing of the team member. The team member device 200 may also be attached to the mask portion of the air respirator worn by the team member. As a result, when the team member is at the disaster site, the photographic images and infrared images taken by the team member device 200 will show the disaster site.

[0046] The situation understanding support device 100 includes an acquisition unit 110, an identification unit 120, a detection unit 130, a generation unit 140, and a display control unit 150. The situation understanding support device 100 may further include a storage device 190. The storage device 190 may be an external device communicably connected to the situation understanding support device 100.

[0047] The acquisition unit 110 acquires a captured image from the team member device 200. The acquisition unit 110 may also acquire an infrared image from the team member device 200. At this time, the acquisition unit 110 acquires an infrared image captured at the same time with approximately the same angle of view as the captured image. In other words, the acquisition unit 110 may acquire an infrared image corresponding to the captured image.

[0048] Furthermore, the acquisition unit 110 acquires a signal related to location information from the team member device 200. If the signal related to location information is a signal obtained using a beacon or Wi-Fi, the acquisition unit 110 acquires the team member's location information based on the signal related to location information. In this case, map information in which the signal related to location information and information indicating the location where the communication device is installed are associated is stored in advance in the storage device 190. The acquisition unit 110 uses the map information to identify a location corresponding to the acquired signal related to location information. Then, the acquisition unit 110 acquires the identified location as the team member's location information.

[0049] If the signal relating to the location information is a signal obtained using GNSS, the acquisition unit 110 acquires the signal as the location information of the team member.

[0050] In this way, the acquisition unit 110 may acquire the location information of the person (team member) who captured the captured image.

[0051] The identifying unit 120 uses the photographed image acquired by the acquiring unit 110 to identify a partial area of the model data corresponding to the disaster site shown in the photographed image.

[0052] As described above, the model data is a three-dimensional model that represents a real space. The model data is stored, for example, in the storage device 190. The model data may be open data that is generally available.

[0053] The model data may be data constructed by a digital twin. A digital twin refers to a reproduction of a real space in a virtual space. Position information indicating the real space is associated with position information in the model data. In other words, the model data includes information indicating the correspondence between positions in the model data and the real space, so that each position in the model data corresponds to a position in the real space.

[0054] Furthermore, the model data may be, for example, data used for the design data of each building. The model data may be, for example, data realized in a BIM (Building Information Modeling) format. That is, the model data may include not only information indicating the shape of a building, etc., but also information about the building's equipment. For example, if the building is a building, examples of information about the equipment include the building's evacuation routes, the location of fire extinguishing equipment, the location of air conditioning equipment, the location of sprinklers, the location of fire shutters, and the location of built-in cabinets. Furthermore, the model data may include information about the building's environmental simulation, such as airflow analysis data and thermal analysis data for the building. Note that, without being limited to this example, the model data may also include information about the building's materials and parts used in the building, etc.

[0055] When the identification process is performed by the identification unit 120, it is first necessary to specify model data corresponding to the structures at the disaster site. At this time, for example, a user (such as a rear supporter) using the terminal device 300 may perform an input operation to specify the model data corresponding to the structures at the disaster site.

[0056] The identification unit 120 then aligns the specified model data with the captured image. Alignment is a process of identifying which part of the model data corresponds to a location in the captured image. The identification unit 120 may align the specified model data with the captured image using image processing techniques such as pattern matching. For example, the identification unit 120 extracts edges of characteristic parts in the captured image and identifies, as a partial region, an area of the specified model data that includes characteristic parts that match the extracted edges. The identification unit 120 may then search for a viewpoint from which the specified model data is viewed that identifies an edge that matches the extracted edge, and identify, as a partial region, an area visible from the viewpoint that results in a hit. If the disaster site is inside a building, examples of characteristic parts include walls, ceilings, and pillars. In other words, in this case, the identification unit 120 extracts edges of walls, ceilings, pillars, and the like that appear in the captured image and identifies, as a partial region, an area on the model data where walls, ceilings, pillars, and the like that match the extracted edges. The identified partial area is an area on the model data that corresponds to the photographed disaster site.

[0057] The identification unit 120 may use the location information of the team members when performing the identification process. In this case, the identification unit 120 uses the location information of the team members when the photographed image was captured. Specifically, the identification unit 120 searches for a position in the model data that corresponds to the location information acquired by the acquisition unit 110. Then, the identification unit 120 aligns the captured image with the area around the position in the model data that is found by the search. This reduces the calculation load of the identification process. In this way, the identification unit 120 may identify a partial area by aligning the captured image with the area of the model data that corresponds to the acquired location information.

[0058] The detection unit 130 first acquires an image of the identified partial area. This image of the partial area is, for example, an image showing the partial area as seen from the viewpoint when the above-mentioned search result was found. That is, the image of the partial area has approximately the same angle of view as the photographed image and is an image showing the location in the model data corresponding to the disaster site shown in the photographed image.

[0059] The detection unit 130 calculates the difference between the image of such a partial region and the captured image. For example, the detection unit 130 generates a difference image that indicates the difference. Here, the difference image may be an image that indicates only the difference portion, or may be an image obtained by cutting out a predetermined region that includes the difference from the captured image. Furthermore, the detection unit 130 may generate multiple difference images.

[0060] The detection unit 130 then detects an object from the generated difference image. The difference between the image of the partial region and the captured image basically indicates an object that exists at the disaster site.

[0061] An existing object detection technique may be used to detect an object from the difference image. For example, the detection unit 130 extracts features from the difference image. The features may be image features derived using HOG (Histograms of Oriented Gradients), SIFT (Scaled Invariance Feature Transform), machine learning, or the like. The detection unit 130 may then detect the object using a detection model that inputs the extracted features. The detection model is a model that learns the relationship between the features extracted from the difference image and the object. The detection unit 130 may input the extracted features to the detection model and detect information output from the detection model as the object included in the difference image. Note that the example of object detection is not limited to this example. For example, the detection model may input an image. In this case, the detection unit 130 detects the object by inputting the difference image to the detection model. Examples of objects that can be detected include a person, an obstacle, flame, smoke, etc.

[0062] [First specific example of detection processing] Next, an example of object detection processing by the detection unit 130 will be described. Fig. 6 is a diagram showing an example of a captured image. Assume that the acquisition unit 110 has acquired a captured image as shown in Fig. 6. In the example of Fig. 6, the captured image shows the interior of a building. The captured image also shows scattered cabinets and a collapsed person. The collapsed person corresponds to a person in need of rescue.

[0063] The detection unit 130 acquires an image of the identified partial region. FIG. 7 is a diagram showing an example of the image of the partial region. That is, the image shown in FIG. 7 is an image showing model data. The detection unit 130 generates a difference image showing the difference between the image of the partial region and the captured image. FIG. 8 is a diagram showing a first example of the difference image. The difference image in FIG. 8 shows the difference between the captured image in FIG. 6 and the image of the partial region in FIG. 7. In the example of FIG. 8, the difference image includes a scattered cabinet and a fallen person.

[0064] Then, the detection unit 130 performs an object detection process on the difference image. In the example of Fig. 8, the detection unit 130 detects a person and a cabinet as an obstacle.

[0065] [Second specific example of detection processing] Next, another example of the object detection process by the detection unit 130 will be described. Fig. 9 is a diagram showing another example of a captured image. Assume that the acquisition unit 110 acquires a captured image as shown in Fig. 9. The captured image in Fig. 9 shows the same location as the interior of the building shown in the captured image in Fig. 6. Furthermore, the captured image in Fig. 9 shows flames and smoke.

[0066] The detection unit 130 acquires the image of the partial region in FIG. 7. Then, the detection unit 130 generates a difference image that indicates the difference between the image of the partial region in FIG. 7 and the captured image in FIG. 9. At this time, the detection unit 130 may generate a difference image as shown in FIGS. 10 and 11. FIG. 10 is a second example of a difference image. The difference image in FIG. 10 includes smoke. The detection unit 130 may detect smoke by performing the above-described object detection process on the difference image.

[0067] At this time, the detection unit 130 may detect smoke from the color and edges of the difference image. When smoke is included in the captured image, the portion of the captured image where the smoke exists contains white, gray, black, and colors similar to these. Furthermore, edges are difficult to detect from the portion of the captured image where the smoke exists. Therefore, the detection unit 130 may detect smoke using the color of the difference image and information on whether an edge has been detected in the difference image. Specifically, if a portion of the difference image where no edge has been detected is white, gray, black, or colors similar to these, smoke exists in that portion. In this way, the detection unit 130 may detect smoke based on the color of the difference portion of the captured image and information on whether an edge has been detected in the difference portion.

[0068] Fig. 11 is a third example of a difference image. A flame is included in the difference image of Fig. 11. The detection unit 130 may detect the flame by performing the object detection process described above on the difference image.

[0069] At this time, the detection unit 130 may further use the infrared image to detect the flame. Temperature information can be extracted from the infrared image. Therefore, when a location where a flame exists is photographed by the infrared photographing unit 220, high temperature information can be extracted from the infrared image. Therefore, the detection unit 130 may detect, as a flame, a difference portion corresponding to a portion in the infrared image where a temperature equal to or higher than a predetermined value is indicated. Furthermore, when a flame is included in the photographed image, the portion in the photographed image where the flame exists includes red and colors close to red. Therefore, the detection unit 130 may detect, as a flame, a portion in the difference image that shows red and colors close to red and where a temperature equal to or higher than a predetermined value is indicated.

[0070] The processing by the acquisition unit 110, the identification unit 120, and the detection unit 130 may be performed for each team member device 200 (that is, for each team member).

[0071] Returning to the description of the functional configuration of the situation understanding assistance device 100, the following will be described.

[0072] The generation unit 140 generates current-state model data according to the object detection result by the detection unit 130. Specifically, when an object is detected by the detection unit 130, the generation unit 140 reflects information about the detected object at a position in the model data corresponding to the position of the detected object in the captured image. For example, assume that a captured image such as that shown in FIG. 9 is acquired. Then, assume that the detection unit 130 detects a person, an obstacle (littered cabinets), flames, and smoke. In this case, the generation unit 140 places, for example, an icon representing a person at a position in the model data corresponding to the position of the detected person. Here, the icon representing a person is an example of information about an object. The icon is, for example, an object image represented in a virtual space. The icon may be a two-dimensional model object image or a three-dimensional model object image. Similarly, the generation unit 140 places icons representing the obstacle, flames, and smoke in the model data. In this way, current-state model data is generated by reflecting information about the detected object in the model data. By reflecting information about the detected object in the model data, the situation at the disaster site is reproduced in the virtual space. In other words, the current state model data can be considered a three-dimensional model that recreates the situation at the disaster site. Note that the icon may be text information. For example, text representing the object may be placed at a position on the model data that corresponds to the position of the detected object.

[0073] The generation unit 140 may reflect an icon corresponding to the team member at a position on the model data that corresponds to the team member's position based on the acquired position information.

[0074] When there are multiple team member devices 200, the model data reflects information about the object detected based on information obtained from each team member device 200. If the detected object overlaps with an object detected based on information obtained from another team member device 200, the generation unit 140 may reflect one of the detected objects in the model data.

[0075] Furthermore, the generation unit 140 may generate a map based on the current state model data. Fig. 12 is a diagram showing an example of a map based on the current state model data. In the example of Fig. 12, a map based on the current state model data reflecting information from a plurality of team member devices 200 is shown. In the example of Fig. 12, the map shows not only the positions of detected objects but also the positions of team members.

[0076] The display control unit 150 displays the current state model data on the terminal device 300. For example, the display control unit 150 displays the current state model data as a VR image on a head-mounted display of the terminal device 300. The current state model data reproduces the situation at the disaster site, allowing the user of the terminal device 300 to have a simulated experience of the situation at the disaster site.

[0077] [Example of operation of situation understanding support device 100] Next, an example of the operation of the situation understanding support device 100 will be described with reference to Fig. 13. In this operation example, an example will be described in which current state model data is generated based on information obtained from one of the team member devices 200. In this operation example, model data corresponding to buildings at the disaster site is specified in advance.

[0078] 13 is a flowchart illustrating an example of the operation of the situation understanding support device 100. First, the acquisition unit 110 acquires a captured image and an infrared image captured by the team member device 200, as well as position information of the team member device 200 (S101). For example, the acquisition unit 110 acquires signals related to the captured image, infrared image, and position information from the team member device 200. Then, the acquisition unit 110 acquires the position information of the team member device 200 based on the signals related to the position information.

[0079] The identification unit 120 searches for a position on the model data that corresponds to the acquired position information (S102). Then, the identification unit 120 identifies a partial area of the model data that corresponds to the disaster site shown in the photographed image (S103). At this time, the identification unit 120 identifies the partial area by aligning the photographed image with an area within a predetermined range from the position on the model data that was found by the search.

[0080] The detection unit 130 acquires an image of the specified partial region (S104). The detection unit 130 also generates a difference image between the image of the partial region and the captured image (S105). The detection unit 130 then detects an object based on the difference image (S106).

[0081] The generation unit 140 generates current model data (S107). Specifically, the generation unit 140 generates the current model data by reflecting information about the detected object in the model data.

[0082] The display control unit 150 causes the terminal device 300 to display the generated current model data (S108).

[0083] When there are multiple team member devices 200, the situation assessment support device 100 may perform the processes of S101 to S107 for each team member device 200. As a result, information on multiple locations in the disaster site is reflected in one model data based on information obtained from the multiple team member devices 200. Furthermore, the processes of S101 to S107 may be performed repeatedly. As a result, the current model data is updated as needed. Note that this operation example is merely an example, and the operation of the situation assessment support device 100 is not limited to this example.

[0084] In this way, the situation understanding support device 100 of the second embodiment acquires a photographed image of a disaster site, and identifies a partial area of the model data corresponding to the photographed disaster site by aligning the photographed image with model data that indicates a three-dimensional model that represents real space. The situation understanding support device 100 also detects an object based on the difference between the photographed image and an image of the identified partial area. The situation understanding support device 100 then generates current state model data in which an icon corresponding to the detected object is placed in the model data at a position corresponding to the position of the object, and displays the generated current state model data on a display.

[0085] As a result, the situation understanding support device 100 can allow, for example, a rear supporter to grasp the situation at a disaster site using a three-dimensional model. Therefore, the situation understanding support device 100 can allow the situation at a disaster site to be grasped intuitively compared to when the situation at a disaster site is shown using text or a two-dimensional map. In other words, the situation understanding support device 100 of the second embodiment can support the grasping of the disaster situation.

[0086] Furthermore, the situation understanding support device 100 of the second embodiment may acquire location information of the team member who captured the captured image, and identify a partial area by aligning the captured image with an area of model data corresponding to the acquired location information. This allows the situation understanding support device 100 to reduce the calculation load compared to when aligning all parts of the model data with the captured image.

[0087] Furthermore, the situation understanding support device 100 of the second embodiment may acquire an infrared image corresponding to the captured image, and detect as a flame a difference portion of the captured image that corresponds to a portion of the infrared image that shows a temperature equal to or higher than a predetermined value. This allows the situation understanding support device 100 to detect a flame more accurately than detecting a flame from the captured image alone.

[0088] Furthermore, the situation understanding support device 100 of the second embodiment may detect smoke based on the color of the difference portion in the captured image and information on whether the edge of the difference portion has been detected, thereby enabling the situation understanding support device 100 to detect smoke.

[0089] [Variation 1] If further information about the disaster site is available, the situation understanding support system 1000 may use that information.

[0090] If a disaster site includes the inside of a specific building, and various sensors are installed in the building, the situation understanding support device 100 acquires information detected by the sensors.

[0091] For example, suppose a smoke sensor is installed in a building. An example of a smoke sensor is a fire alarm. In this case, the acquisition unit 110 acquires sensor data from the smoke sensor. The detection unit 130 then detects smoke using the sensor data from the smoke sensor located within a predetermined range from the position where the captured image was captured, the color of the difference image, and information on whether an edge has been detected in the difference image. For example, if smoke is detected when a predetermined value based on the color of the difference image and the information on whether an edge has been detected in the difference image is equal to or greater than a threshold, the threshold may be changed depending on the value of the sensor data.

[0092] Also, for example, assume that a temperature sensor is installed in a building. In this case, the acquisition unit 110 acquires sensor data from the temperature sensor. Then, the detection unit 130 detects a flame using the sensor data from the temperature sensor located within a predetermined range from the position where the captured image was taken and the difference image. For example, if there is a location in the difference image that shows red or a color close to red, and if the temperature indicated in the sensor data is equal to or higher than a predetermined value, the detection unit 130 may detect a flame in that location.

[0093] <Third embodiment> Next, a situation understanding support system according to a third embodiment will be described. In the third embodiment, an example of estimating the situation at a disaster site from obtained information will be described. Note that some of the contents overlapping with the first and second embodiments will not be described.

[0094] In the third embodiment, an example of application of the situation awareness support system will be described in which a disaster involving a fire occurs and rescue workers are operating within a designated building, but this does not limit the situations in which the system can be applied.

[0095] [Details of Situation Assessment Support System 1001] Fig. 14 is a block diagram showing an example of the configuration of a situation understanding support system 1001 according to the third embodiment. The situation understanding support system 1001 includes at least a situation understanding support device 101. In the example of Fig. 14, the situation understanding support system 1001 includes the situation understanding support device 101, a team member device 201, a terminal device 300, and a communication device. Each of the situation understanding support device 101 and the team member device 201 may have the functions described below in addition to the functions of the situation understanding support device 100 and the team member device 200, respectively.

[0096] The worker device 201 includes a visible light imaging unit 210, an infrared imaging unit 220, a position information communication unit 230, and an output unit 240. There may be a plurality of worker devices 201, just like the worker device 200. That is, there may be worker devices 201-1, 201-2, ..., 201-n (n is a natural number).

[0097] The output unit 240 has a function of outputting information generated by the situation grasp assistance device 101. The output unit 240 is, for example, a display and a projector.

[0098] FIG. 15 is a diagram showing an example of a team member device 201. In the example of FIG. 15, the team member device 201 is shown as a full-face mask-type device. For example, a team member wears such a mask-type device. As shown in FIG. 15, the output unit 240 may be a display that also functions as goggles for the mask. However, the output unit 240 may be a projector that projects an image onto the goggle portion of the mask. Also, in the example of FIG. 15, the team member device 201 may have the function of a mask for an air respirator.

[0099] The situation understanding support device 101 includes an acquisition unit 111, an identification unit 120, a detection unit 130, a generation unit 141, a display control unit 151, an estimation unit 160, an activity information output unit 170, and a path calculation unit 180. The situation understanding support device 101 also includes a storage device 190.

[0100] The situation understanding support device 101 estimates the source of the fire from the information obtained from the firefighter device 201 and the current model data. Specifically, the estimation unit 160 estimates the location of the source of the fire at the disaster site. At this time, the estimation unit 160 uses the location of the flame detected by the detection unit 130, temperature data at the disaster site, and airflow analysis data and thermal analysis data of the building.

[0101] An example of temperature data at a disaster site is a temperature map of the disaster site. The temperature data is acquired by, for example, the acquisition unit 111. The temperature data may be information based on an infrared image captured by the disaster site team member device 201. In this case, the temperature data may be information generated based on temperature information extracted from the infrared image acquired by each disaster site team member device 201. If a temperature sensor is installed in a building, the temperature data may be information based on sensor data acquired by the temperature sensor. Furthermore, in some cases, temperature sensors are attached to the clothing of disaster site team members to monitor their physical condition. In such cases, the temperature data may be information based on sensor data acquired by the attached temperature sensor. The temperature data is reflected in the current state model data by the generation unit 141.

[0102] Airflow analysis data and thermal analysis data are information included in the model data. Airflow analysis data is simulation data related to the flow of air in a building. When designing a building, the installation locations of ventilation fans, air conditioning equipment, etc. may be considered taking into account the airflow in the building. Airflow analysis data may be data created in advance at the time of designing such a building. Thermal analysis data is simulation data related to the transfer of heat in a building. For example, thermal analysis data may be data that can simulate changes in temperature distribution within a building when a heat source is present. Like airflow analysis data, thermal analysis data may also be data created in advance at the time of designing a building.

[0103] The estimation unit 160 uses the airflow analysis data and the thermal analysis data to simulate how the fire will spread and how the current fire situation will develop, based on the current flame position and current temperature data. The location of the fire source is estimated by this simulation.

[0104] In this way, the estimation unit 160 estimates the location of the source of the fire at the disaster site based on the detected flame location, temperature data, airflow analysis data, and thermal analysis data. The estimation unit 160 is an example of an estimation means.

[0105] The generation unit 141 generates current model data in which an icon indicating the source of the fire is placed at a position on the model data corresponding to the estimated location of the source of the fire. At this time, the generation unit 141 updates the current model data generated up to that point by reflecting the information indicating the source of the fire.

[0106] The situation understanding support device 101 also estimates how the fire will spread in the future. Specifically, the estimation unit 160 estimates how the fire will spread at the disaster site after estimating the location of the fire source. The way the fire will spread includes the progress of the flames at the site over time and the degree to which the site is filled with smoke over time.

[0107] For example, the estimation unit 160 uses airflow analysis data and thermal analysis data to simulate how the fire will spread in the future based on the location of the fire source, the current flame location, and current temperature data. The simulation estimates the progress of the fire at the scene over time, such as in which direction and at what speed the fire will spread. The simulation also estimates the degree to which smoke will fill the scene over time, such as in which direction and at what speed the smoke will fill the scene.

[0108] In this way, the estimation unit 160 estimates how the fire will spread at the disaster site based on the estimated location of the fire source, the detected location of the flame, temperature data, airflow analysis data, and thermal analysis data.

[0109] Furthermore, the estimation unit 160 may estimate the passage of time for a person detected at the disaster site. If it is possible to simulate how the fire spreads and how the current fire situation develops, it is possible to estimate the time that has passed since the fire broke out. There is a high possibility that a person detected at the disaster site is left behind because they were too late to escape. Therefore, the estimation unit 160 can estimate the time (i.e., the passage of time) that the detected person has been left behind at the disaster site from such a simulation.

[0110] The situation awareness support device 101 also has a navigation function for disaster relief personnel at the disaster site. For example, the situation awareness support device 101 shows the disaster relief personnel evacuation routes and locations where firefighting should be carried out.

[0111] First, the activity information output unit 170 outputs activity information. The activity information is information indicating the activities that team members should perform at the disaster site. Examples of activities include firefighting, rescue, and evacuation.

[0112] The activity information output unit 170 outputs activity information based on the estimated fire spread and the current state model data. At this time, the activity information output unit 170 outputs the activity information using a learning model that has undergone predetermined learning. The learning model is a learning model that has learned the relationship between the situation at the disaster site, the fire spread, and guidelines for activities at the disaster site. The guidelines for activities at the disaster site may indicate activities that should be performed according to various situations.

[0113] For example, the learning model may output information such as whether or not firefighting activities should be performed based on the fire situation, such as the location of the flames and how the fire is spreading, and if so, where the firefighting activities should be performed (firefighting location). Specifically, the activity information output unit 170 inputs the current model data and information indicating how the fire is spreading estimated by the estimation unit 160 to the learning model, and outputs the information output by the learning model as activity information. In this case, the current model data includes information indicating the location of the source of the fire, information indicating the location of the flames, temperature data, etc.

[0114] The learning model may also output information such as whether evacuation is necessary based on the fire situation. In this case as well, the activity information output unit 170 inputs the current model data and information indicating the estimated spread of the fire to the learning model, and outputs the information output by the learning model as activity information.

[0115] The learning model may also output information such as whether or not people should be rescued based on the status of the fire and the time elapsed for people at the disaster site (i.e., people left behind). In this case as well, the activity information output unit 170 inputs the current model data and information indicating the estimated spread of the fire to the learning model, and outputs the information output by the learning model as activity information.

[0116] In this way, the activity information output unit 170 uses a learning model that has learned the relationship between the situation at the disaster site, the spread of the fire, and guidelines for activities at the disaster site, and outputs activity information that indicates activities that team members should perform at the photographed disaster site based on the current model data and the estimated spread of the fire. The activity information output unit 170 is an example of an activity information output means.

[0117] The route calculation unit 180 calculates a route that the team member can take based on the activity information. The route calculation unit 180 is an example of a route calculation means.

[0118] For example, suppose the activity information indicates a fire extinguishing position. In this case, the path calculation unit 180 uses the current model data to calculate a path from the position of the firefighter to the fire extinguishing position. There are cases where fire extinguishing activities are required to be carried out at the source of the fire. In that case, the fire extinguishing position will be near the source of the fire. In this case, the path calculation unit 180 calculates a path from the position of the firefighter to the position of the source of the fire. The path to the fire extinguishing position is called the fire extinguishing path. If an obstacle is detected by the detection unit 130, the path calculation unit 180 may calculate a fire extinguishing path that avoids the obstacle.

[0119] Furthermore, for example, it is assumed that the activity information includes information indicating that evacuation should be performed. In this case, the route calculation unit 180 calculates an evacuation route from the position of the rescue team member based on the current model data and information indicating the evacuation route of the building that is previously included in the model data. It is assumed that there are areas where people cannot pass due to detected obstacles, flames, etc. In such a case, the route calculation unit 180 may identify the areas where people cannot pass based on the detected objects, and calculate an evacuation route from the position of the rescue team member that avoids the areas where people cannot pass.

[0120] If the building at the disaster site is multi-story, it may be possible to enter or exit through a window on the second or higher floor using a ladder truck or the like. In such a case, the path calculation unit 180 may calculate an evacuation route to an accessible window. Information indicating accessible windows is indicated in the current model data. For example, information indicating accessible windows is input by a user using the terminal device 300. Then, the generation unit 141 reflects the information indicating accessible windows in the current model data.

[0121] Furthermore, it is assumed that the activity information includes information that a person should be rescued. In this case, the route calculation unit 180 uses the current state model data to calculate, as a rescue route, an evacuation route from the rescue team member via the person to be rescued (person in need of rescue).

[0122] Fig. 16 is a diagram showing an example of a route. Specifically, Fig. 16 is a diagram showing various routes from team members on a map based on current model data. For example, the map is generated by the generation unit 141. For example, in the case of a fire extinguishing route, the route from each team member to the location of the fire source, which is the fire extinguishing position, is shown.

[0123] The generation unit 141 generates instruction information indicating the direction for moving from the location of the firefighter to the location where the firefighter should go. At this time, the generation unit 141 generates corresponding instruction information for each firefighter. For example, assume that the activity information indicates a fire extinguishing location. In this case, the generation unit 141 generates instruction information for each firefighter indicating the movement direction for moving from the firefighter's viewpoint to the fire extinguishing location based on the fire extinguishing route from the location of the firefighter.

[0124] In addition, it is assumed that the activity information includes information indicating that evacuation should be performed. In this case, the generation unit 141 generates, for each team member, instruction information indicating a moving direction for evacuation from the team member's viewpoint, based on the evacuation route from the team member's position.

[0125] Furthermore, it is assumed that the activity information includes information indicating that a person should be rescued. In this case, the generation unit 141 generates, for each team member, instruction information indicating a movement direction to rescue the person from the team member's viewpoint, based on a rescue route from the team member's position. Note that the generation unit 141 may generate instruction information indicating a movement direction to rescue the person only for team members whose distance from the person in need of rescue is less than a predetermined value, based on the position information of each team member.

[0126] The display control unit 151 displays the instruction information on the goggle portion of the mask worn by the soldier. Specifically, when the output unit 240 is a projector, the display control unit 151 causes the output unit 240 to project the instruction information onto the goggle portion. Furthermore, when the output unit 240 is a display disposed in the goggle portion, the display control unit 151 causes the output unit 240 to display the instruction information.

[0127] FIG. 17 is a diagram showing an example of instruction information. FIG. 17 shows the view from a firefighter through the goggles. The instruction information is displayed on the goggles. In the example of FIG. 17, if the instruction information is related to evacuation, a downward arrow is displayed along with the text information "Evacuation Route." This indicates that the evacuation route is toward the firefighter. Similarly, if the instruction information is related to firefighting activities, a leftward arrow is displayed along with the text information "Firefighting Position." Furthermore, if the instruction information is related to rescue activities, a graphic highlighting the person in need of rescue and an arrow are displayed along with the text information "Rescue Route" and "Person in Need of First Aid." In this way, by displaying the instruction information on the goggles, the instruction information appears to the firefighter as being superimposed on real space. This allows the firefighter to recognize the direction in which to head.

[0128] 17, instruction information for each of firefighting activities, evacuation, and rescue activities is shown, but only instruction information for one of them may be displayed. Instruction information is information corresponding to each team member. Therefore, for example, only instruction information related to firefighting activities may be displayed for one team member, and only instruction information related to rescue activities may be displayed for another team member.

[0129] The activity information may also include information indicating that the detected person will not be rescued. This is because the detected person may already be dead. In such a case, the generation unit 141 generates instruction information including information indicating that rescuing the person is not necessary. Then, the display control unit 151 displays that rescuing the person is not necessary.

[0130] [Example of operation of the situation awareness support device 101] Next, an example of the operation of the situation understanding support device 101 will be described with reference to Fig. 18. In this operation example, it is assumed that the processing described in Fig. 13 has already been performed. That is, current state model data has been generated. The model data also includes in advance information indicating at least the evacuation routes of the building, airflow analysis data, and thermal analysis data. The current state model data also indicates the positions of the flames, smoke, and rescue workers.

[0131] FIG. 18 is a flowchart illustrating an example of the operation of the situation understanding support device 101. The acquisition unit 111 acquires temperature data (S201). For example, the acquisition unit 111 acquires a temperature map of the disaster site based on infrared images captured by the firefighter device 201. The estimation unit 160 estimates the location of the fire source (S202). For example, the estimation unit 160 estimates the location of the fire source by using airflow analysis data and thermal analysis data to simulate how the fire spreads and results in the current fire situation based on the current flame location and current temperature data. At this time, the generation unit 141 reflects the location of the fire source in the current model data.

[0132] Then, the estimation unit 160 estimates how the fire will spread (S203). For example, the estimation unit 160 uses the airflow analysis data and the thermal analysis data to perform a simulation of how the fire will spread in the future based on the location of the fire origin, the current flame location, and the current temperature data. In this way, the estimation unit 160 estimates how the fire will spread, including, for example, the progress of the flames at the scene over time and the degree to which the scene is filled with smoke over time.

[0133] The activity information output unit 170 outputs the activity information based on the current model data and the estimated fire spread (S204). At this time, the activity information output unit 170 uses, for example, a learning model that has learned the relationship between the situation at the disaster site, the fire spread, and guidelines for activities at the disaster site.

[0134] The route calculation unit 180 calculates possible routes that the team members can take based on the activity information (S205). For example, if the activity information includes information that evacuation should be performed, the route calculation unit 180 calculates evacuation routes from the team members' positions based on the current state model data and information indicating evacuation routes for buildings.

[0135] The generation unit 141 generates instruction information for each team member (S206). For example, the generation unit 141 generates instruction information for each team member that indicates a moving direction for evacuation from the team member's viewpoint, based on an evacuation route from the team member's position.

[0136] Then, the display control unit 151 causes the output unit 240 of each team member device 201 to display the instruction information. For example, the display control unit 151 causes the instruction information to be displayed on the goggles of the mask worn by the team member.

[0137] The processes of S201 to S207 may be repeated. As a result, the current model data is updated as needed, and the instruction information displayed to the team members is also updated. This operation example is merely an example, and the operation of the situation understanding support device 101 is not limited to this example.

[0138] In this way, the situation understanding support device 101 of the third embodiment estimates the location of the fire source at the disaster site based on the flame position, temperature data, airflow analysis data, and thermal analysis data. Then, the situation understanding support device 101 generates current model data in which an icon indicating the fire source is placed at a position on the model data corresponding to the estimated location of the fire source. In this way, the situation understanding support device 101 can show the user the estimated location of the fire source even if the location of the fire is unknown.

[0139] Furthermore, the situation awareness support device 101 of the third embodiment can estimate how the fire will spread at the disaster site based on the estimated location of the fire source, the detected location of the flame, temperature data, airflow analysis data, and thermal analysis data. This allows the situation awareness support device 101 to show the user how the fire will spread in the future.

[0140] Furthermore, the situation understanding support device 101 of the third embodiment outputs activity information indicating activities that team members should perform at the disaster site, and calculates possible routes that the team members can take based on the activity information.The situation understanding support device 101 then generates instruction information indicating the direction in which the team members should travel from their current locations to their destinations, and displays the instruction information on the goggles of the masks worn by the team members.In this way, the situation understanding support device 101 can intuitively show team members at the disaster site what actions they should take.

[0141] In this case, for example, if activity information indicating evacuation from a building is output, the situation understanding support device 101 may identify an impassable location for a person based on the detected object in the current model data, and calculate an evacuation route that avoids the impassable location from the position of the rescue team member.The situation understanding support device 101 may then generate instruction information indicating the direction to evacuate from the position of the rescue team member according to the calculated evacuation route.

[0142] <Example of hardware configuration for situation awareness support device> The hardware constituting the situation understanding support device of the first, second, and third embodiments described above will be described. Fig. 19 is a block diagram showing an example of the hardware configuration of a computer device constituting the situation understanding support device in each embodiment. The situation understanding support device and situation understanding support method described in each embodiment and each modified example are realized in a computer device 90. For example, each of the situation understanding support device, team member device, terminal device, etc. described in each embodiment and each modified example may have the hardware configuration shown in Fig. 19.

[0143] 19, a computer device 90 includes a processor 91, a RAM (Random Access Memory) 92, a ROM (Read Only Memory) 93, a storage device 94, an input / output interface 95, a bus 96, and a drive device 97. The situation awareness assistance system may be realized by a plurality of electric circuits.

[0144] The storage device 94 stores a program (computer program) 98. The processor 91 executes the program 98 of the situation recognition support system using the RAM 92. Specifically, for example, the program 98 includes a program that causes a computer to execute the processes shown in FIGS. 3, 13, and 18. The processor 91 executes the program 98 to realize the functions of each component of the situation recognition support system. The program 98 may be stored in the ROM 93. Alternatively, the program 98 may be recorded in the storage medium 80 and read out using the drive device 97, or may be transmitted to the computer device 90 from an external device (not shown) via a network (not shown).

[0145] The input / output interface 95 exchanges data with peripheral devices (such as a keyboard, a mouse, and a display device) 99. The input / output interface 95 functions as a means for acquiring or outputting data. The bus 96 connects each component.

[0146] There are various variations in the method for realizing the situation awareness support system. For example, each component included in the situation awareness support system can be realized as a dedicated device. Furthermore, each component included in the situation awareness support system can be realized based on a combination of multiple devices.

[0147] The scope of each embodiment also includes a processing method for recording a program for realizing each configuration of the function of each embodiment on a storage medium, reading the program recorded on the storage medium as code, and executing it on a computer. That is, a computer-readable storage medium is also included in the scope of each embodiment. Furthermore, the storage medium on which the above-mentioned program is recorded and the program itself are also included in each embodiment.

[0148] The storage medium may be, but is not limited to, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD (Compact Disc)-ROM, a magnetic tape, a non-volatile memory card, or a ROM. The programs recorded on the storage medium are not limited to standalone programs that execute processes, but also include programs that run on an OS (Operating System) in cooperation with other software and functions of an expansion board.

[0149] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0150] Furthermore, the above-described embodiments and modifications can be combined as appropriate. [Explanation of symbols]

[0151] 100, 101 Situational awareness support device 110, 111 Acquisition Department 120 Specific section 130 Detector 140, 141 generation section 150, 151 Display control unit 160 Estimation Department 170 Activity Information Output Unit 180 Route calculation unit 190 Storage device 200, 201 personnel equipment 300 Terminal Device

Claims

1. An acquisition means for acquiring photographed images of the disaster site; an identification means for identifying a partial area of the model data corresponding to the photographed disaster site by aligning the photographed image with model data representing a three-dimensional model that expresses real space; a detection means for detecting an object based on a difference between the captured image and an image of the specified partial region; a generating means for generating current model data in which an icon corresponding to the detected object is placed at a position on the model data corresponding to the position of the object; and a display control means for displaying the generated current state model data on a display. Situational awareness support device.

2. The acquisition means acquires location information of the member who captured the photographed image, the specifying means specifies the partial area by aligning the captured image with an area of the model data corresponding to the acquired position information. The situation awareness support device according to claim 1 .

3. the acquisition means acquires an infrared image corresponding to the captured image, The detection means detects, as a flame, a portion of the captured image where the difference corresponds to a portion of the captured image where the infrared image indicates a temperature equal to or higher than a predetermined value. The situation awareness support device according to claim 1 .

4. the detection means detects smoke based on a color of the difference portion in the captured image and information on whether an edge of the difference portion has been detected. The situation awareness support device according to claim 1 .

5. An estimation means is provided, The disaster site includes the inside of a building. the model data includes airflow analysis data, which is simulation data regarding airflow in the building, and thermal analysis data, which is simulation data regarding heat transfer in the building; The acquisition means acquires temperature data of the photographed disaster site, the detection means detects a flame; the estimation means estimates the location of a fire source in the disaster site based on the detected flame location, the temperature data, the airflow analysis data, and the thermal analysis data; the generating means generates the current model data in which an icon indicating the source of the fire is placed at a position on the model data corresponding to the estimated location of the source of the fire. The situation awareness support device according to claim 1 .

6. the estimation means estimates how the fire will spread at the disaster site based on the estimated location of the fire source, the detected location of the flame, the temperature data, the airflow analysis data, and the thermal analysis data; The situation grasping assistance device according to claim 5.

7. The current state model data includes at least information indicating the location of the fire source, the location of the flame, the temperature data, and the location of the firefighters; an activity information output means for outputting activity information indicating the activities that team members should perform at the photographed disaster site based on the current model data and the estimated fire spread, using a learning model that has learned the relationship between the situation at the disaster site, the spread of the fire, and guidelines for activities at the disaster site; a route calculation means for calculating a route that a team member can take based on the activity information; the generating means generates instruction information indicating a direction for the team member to travel from the team member's location to a destination; The display control means displays the instruction information on a goggle portion of a mask worn by the soldier. The situation grasping assistance device according to claim 6.

8. The disaster site photographed is inside a building, the model data includes information indicating evacuation routes for the building; When the activity information output means outputs the activity information indicating that the person is to evacuate from the building, The route calculation means performs the following on the current model data: Identifying a location where a person cannot pass based on the detected object; Based on the location of the team members, an evacuation route that avoids the impassable area is calculated. the generating means generates the instruction information indicating a direction for evacuation from the position of the rescue team member in accordance with the calculated evacuation route. The situation understanding assistance device according to claim 7.

9. Acquire photographic images of the disaster site, By aligning the photographed image with model data indicating a three-dimensional model that expresses real space, a partial area of the model data that corresponds to the photographed disaster site is identified; detecting an object based on a difference between the captured image and an image of the specified partial region; generating current model data in which an icon corresponding to the detected object is placed at a position on the model data corresponding to the position of the detected object; Displaying the generated current state model data on a display. Methods to assist in understanding the situation.

10. A process of acquiring photographed images of the disaster site; A process of identifying a partial area of the model data corresponding to the photographed disaster site by aligning the photographed image with model data indicating a three-dimensional model that expresses real space; a process of detecting an object based on a difference between the captured image and an image of the identified partial region; a process of generating current model data in which an icon corresponding to the detected object is placed at a position on the model data corresponding to the position of the detected object; and displaying the generated current state model data on a display. program.

Images