State confirmation method and state confirmation device

Abstract

To provide a state confirmation method and a state confirmation device for grasping the state of an actual site.SOLUTION: A actual-site management device 20 which manages the state of an actual site comprises a difference extraction part 213 which specifies differences based upon a first image group at a first point of time and a second image group at a second point of time after the first point of view. The management part 20 further comprises a difference display part 214 which specifies and displays a variation part of the actual site in a second free viewpoint image visualized by a free viewpoint image generation technique using the second image group.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a situation confirmation method and a situation confirmation device for grasping the situation at a site. [Background technology]

[0002] A situation assessment system for assessing the progress of construction work, etc. using images has been studied (see, for example, Patent Document 1). The situation assessment system disclosed in Patent Document 1 uses a management server equipped with a building element information storage unit that records building elements used in processes at a construction site, a site image information recording unit that records images taken of the construction site, and a control unit connected to a manager's terminal. The control unit then identifies the building elements contained in the captured images recorded in the site image information recording unit. Next, the building element information storage unit is used to identify the process associated with the building element. The identified process is then output to the manager's terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-107443 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, architectural elements are identified using two-dimensional images. In this case, it may be difficult to grasp the situation on-site using two-dimensional images. [Means for solving the problem]

[0005] The situation confirmation method and situation confirmation device for solving the above problems specifically display, as a changed portion, a difference between first free viewpoint data captured at a first time point and visualized using a free viewpoint image generation technique, and second free viewpoint data captured at a second time point after the first time point and visualized using the free viewpoint image generation technique. In this way, by specifically displaying the difference between the first time point and the second time point in the free viewpoint image, it is possible to accurately grasp the changed portion in the free viewpoint image.

[0006] In the status confirmation method and status confirmation device configured as described above, the first free viewpoint data is an image of the construction status viewed from a predetermined viewpoint of a predetermined area visualized by a free viewpoint image generation technique using a first image group taken at the first time point, and the second free viewpoint data is an image of the construction status viewed from the predetermined viewpoint of a predetermined area visualized by a free viewpoint image generation technique using a second image group taken at the second time point. In this way, by using the images of the construction status, it is possible to grasp the progress of the construction work.

[0007] A situation confirmation method and a situation confirmation device for solving the above problem generate a third free-viewpoint image by using a free-viewpoint image generation technique to visualize a third image group in which the difference between a first image group taken at a first time point and a second image group taken at a second time point after the first time point is specifically displayed as a progress portion. In this way, by displaying the third image group in which the difference is identified in the free-viewpoint image, it is possible to grasp the progress portion in a 3D image.

[0008] In the situation confirmation method and situation confirmation device configured as described above, the second image group is visualized by a free-viewpoint image generation technique to generate a second free-viewpoint image, and the third free-viewpoint image is associated with the second free-viewpoint image. By associating the third free-viewpoint image with the second free-viewpoint image in this way, the two can be compared to confirm the current state and progress.

[0009] A situation confirmation method and a situation confirmation device for solving the above problem identify, as a changed portion, a difference between a first point cloud generated from a first image group captured at a first time point and a second point cloud generated from a second image group captured at a second time point after the first time point, visualize the second image group using a free-viewpoint image generation technique to generate a second free-viewpoint image, and specifically display the difference on the second free-viewpoint image. In this way, by identifying the difference using the point cloud, progress can be grasped in the second free-viewpoint image.

[0010] In the situation confirmation method having the above configuration, the specific display is a highlight display of the difference. By highlighting in this way, it is possible to grasp the changed part in the free viewpoint image.

[0011] In the situation confirmation method having the above configuration, the specific display is an image caption obtained by natural language processing of the difference. By adding an image caption in this way, the changed portion can be easily grasped in text. [Effects of the Invention]

[0012] According to the present invention, the situation at the site can be grasped by efficiently identifying and displaying the changed portion. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is an explanatory diagram of a system according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram of a hardware configuration of the first embodiment. [Figure 3] FIG. 2 is an explanatory diagram of a processing procedure according to the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a display screen according to the first embodiment. [Figure 5] FIG. 2 is an explanatory diagram of a display screen according to the first embodiment. [Figure 6] FIG. 2 is an explanatory diagram of a display screen according to the first embodiment. [Figure 7] FIG. 10 is an explanatory diagram of a processing procedure according to a second embodiment. [Figure 8] FIG. 11 is an explanatory diagram of a processing procedure according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (First embodiment) An embodiment of a status confirmation method and a status confirmation device will be described below with reference to Figures 1 to 6. In this embodiment, a progress management device will be described that supports progress management of a construction project using images of the construction status captured at a construction site. Here, progress areas are specifically displayed (for example, highlighted) in a free viewpoint image of a new date. In this embodiment, as shown in FIG. 1, an image capturing device C1, a user device 10, and a management device 20 (situation confirmation device) are used, which are connected via a network.

[0015] (Hardware configuration description) 2, the hardware configuration of the information processing device H10 that constitutes the user device 10 and the management device 20 will be described. The information processing device H10 includes a communication device H11, an input device H12, a display device H13, a storage device H14, and a processor H15. Note that this hardware configuration is an example, and it can also be realized by other hardware.

[0016] The communication device H11 is an interface that establishes a communication path with other devices and executes data transmission and reception, and is, for example, a network interface or a wireless interface.

[0017] The input device H12 is a device that accepts input of various information, such as a mouse or a keyboard. The display device H13 is a display that displays various information. In this embodiment, the input device H12 and the display device H13 function as an interface unit.

[0018] The storage device H14 stores data and various programs for executing various functions of the user device 10 and the management device 20. Examples of the storage device H14 include a ROM, a RAM, and a hard disk.

[0019] The processor H15 uses programs and data stored in the storage device H14 to control each process in the user device 10 and the management device 20. Examples of the processor H15 include a CPU and an MPU. The processor H15 loads programs stored in a ROM or the like into a RAM and executes various processes for each process.

[0020] The processor H15 is not limited to a processor that performs all of its processing using software. For example, the processor H15 may include a dedicated hardware circuit (e.g., an application-specific integrated circuit (ASIC)) that performs hardware processing for at least some of the processing it performs. That is, the processor H15 may be configured with the following:

[0021] [1] One or more processors that operate according to a computer program (software). [2] One or more dedicated hardware circuits that perform at least some of the various processes [3] Circuits containing combinations of these The processor includes a CPU and memory, such as RAM and ROM, that stores program code or instructions configured to cause the CPU to perform processes. Memory, or computer-readable media, includes any available media that can be accessed by a general-purpose or special-purpose computer.

[0022] (System Configuration) Next, each function of the situation confirmation device will be explained using FIG. The photographing device C1 is a photographing means such as a camera that photographs the construction site. Using this photographing device C1, a plurality of photographed images including the same subject (construction materials, etc.) are generated at the construction site.

[0023] The user device 10 is a computer terminal used by a manager of a construction site. The management device 20 is a computer system that evaluates the progress of a process. The management device 20 includes a control unit 21, an image information storage unit 22, and a progress information storage unit .

[0024] The control unit 21 supports progress management of the construction process. To this end, the control unit 21 functions as an information acquisition unit 211, an image generation unit 212, a difference extraction unit 213, and a difference display unit 214 by executing a management program that executes each stage (information acquisition stage, image generation stage, difference extraction stage, difference display stage, etc.).

[0025] The information acquisition unit 211 acquires from the image information storage unit 22 a plurality of images captured by the image capture device C1. The image generation unit 212 estimates and visualizes the structure of a 3D scene from multiple captured images using a free-viewpoint image generation technique. For example, a structure from motion (SfM) algorithm or the like can be used for this free-viewpoint image generation technique. In this embodiment, the SfM algorithm is used to identify the same subject in multiple captured images by detecting feature points in the captured images, detecting edges in the captured images, and so on, and acquire a correspondence between the captured images. Next, the image generation unit 212 estimates the position and orientation of the image capture device C1 using parallax information corresponding to this correspondence. Next, the image generation unit 212 uses a machine learning model to learn a 3D model including the structure (object placement) and color distribution of a 3D space in which objects exist. Then, the image generation unit 212 calculates a viewpoint projection matrix at the viewpoint position of the free viewpoint specified on the user device 10. This projection matrix is used to convert the positions of points in the 3D space to the positions of the 2D image to be displayed. The image generation unit 212 generates a 2D free-viewpoint image in which color information is assigned to the positions of the 2D image using techniques such as rendering and index mapping. In this embodiment, data generated by the free viewpoint image generation technique is called free viewpoint data (including a three-dimensional model and a free viewpoint image).

[0026] The difference extraction unit 213 extracts a difference region in the second image based on a comparison between a first image at a preceding first time point and a second image at a subsequent second time point. The difference extraction unit 213 selectively extracts a difference region related to the progress of the process. For this selective extraction, a prediction model that recognizes a difference region related to the progress of the process can be used. For example, this prediction model is generated by machine learning using, as training data, images that include, as annotations, difference regions related to the progress of the process. Note that the extraction of the difference region is not limited to a method using machine learning.

[0027] The difference display unit 214 displays an area extracted as a progress portion (change portion) of the process based on the difference in the free viewpoint data at the second time point. Furthermore, the difference display unit 214 performs object recognition on the extracted progress portion to identify the process corresponding to the component. In this case, a correspondence table or a prediction model for identifying the process from the component can be used.

[0028] Image data captured using the photographing device C1 is recorded in the image information storage unit 22. This image data is recorded when the image data is acquired from the photographing device C1. The image data includes information about the date and time of photographing and the location of photographing for the photographed image of the subject.

[0029] The captured images are images of a construction site, and the multiple images are images of the same subject taken from multiple viewpoints. The photographing date and time information is information relating to the date and time when the photograph was taken. The photography location information is information that identifies the location where the photography was performed (construction site, photography position, etc.).

[0030] The progress information storage unit 23 stores progress management data on the progress of construction work. This progress management data is recorded when the extracted difference components are identified. The progress management data stores information on the location and process for each date and time.

[0031] The date and time information is information relating to the date and time when the progress was identified. The location information is information about the location where the progress is identified. The process information is information relating to the process corresponding to the component recognized at this location.

[0032] (Progress management processing) The progress management process will be described with reference to FIGS. First, the control unit 21 of the management device 20 executes a process for specifying a processing target (step S10). Specifically, the administrator specifies a construction site, a first time point, and a second time point for which the administrator wishes to check progress, using the user device 10. In this case, the information acquisition unit 211 of the control unit 21 acquires information about the specified construction site, the first time point, and the second time point from the user device 10.

[0033] Next, the control unit 21 of the management device 20 executes a process for acquiring two-dimensional images at a first time point (step S11). Specifically, the information acquisition unit 211 of the control unit 21 acquires a plurality of images (first image group) taken at the first time point for the designated construction site from the image information storage unit 22. A common period, such as the same date, is used as the first time point. Here, as shown in Fig. 4, a plurality of photographed images including an image 500 are acquired. This image 500 is an image of the construction situation at a first point in time.

[0034] Next, the control unit 21 of the management device 20 executes a process for generating a 3D model (first time point) (step S12). Specifically, the image generation unit 212 of the control unit 21 learns a 3D model using a plurality of captured images (first image group) at the first time point by a free viewpoint image generation method. By using this 3D model, each point in space can be represented by color information (RGB, etc.) and transmittance, and a 2D first free viewpoint image of the construction situation in which the visualized predetermined area is viewed from a predetermined viewpoint (free viewpoint position) can be generated.

[0035] Furthermore, the control unit 21 of the management device 20 executes a process for acquiring two-dimensional images at a second time point (step S13). Specifically, the information acquisition unit 211 of the control unit 21 acquires a plurality of captured images (second image group) at the second time point from the image information storage unit 22. The second images are images captured at the same construction site (designated construction site) as the first images, at a common time point, such as on the same date subsequent to the preceding first time point. In this case, it is assumed that part of the subject included in the first image is also included in the second image. Here, as shown in Fig. 5, a plurality of photographed images including an image 510 are acquired. This image 510 is an image of the construction situation at a second time point.

[0036] Next, the control unit 21 of the management device 20 executes a process for generating a 3D model (second time point) (step S14). Specifically, the image generation unit 212 of the control unit 21 learns a 3D model using a plurality of captured images (second image group) at the second time point using a free viewpoint image generation method. By using this 3D model, each point in space can be represented by color information and transmittance, and a 2D second free viewpoint image of the construction situation in which the visualized specified area is viewed from a specified viewpoint (free viewpoint position) can be generated.

[0037] Next, the control unit 21 of the management device 20 executes a comparison process of the free viewpoint data (step S15). Specifically, the difference extraction unit 213 of the control unit 21 compares the first free viewpoint data with the second free viewpoint data. Here, the comparison may be based on the three-dimensional spatial structures of the two three-dimensional models, or may be based on two-dimensional free viewpoint images taken from a common viewpoint position. Then, the difference extraction unit 213 identifies a difference area (progress portion) in the second free viewpoint data from the first free viewpoint data.

[0038] Next, control unit 21 of management device 20 executes a process of displaying the difference in the free viewpoint image (step S16). Specifically, difference display unit 214 of control unit 21 sets a highlight display for the difference region in the second free viewpoint image. Then, difference display unit 214 outputs the free viewpoint image including the highlight display to user device 10.

[0039] Here, as shown in FIG. 6, a free viewpoint image 520 is generated, which is an image of the construction status viewed from a specific viewpoint. A difference region 521 is highlighted in this free viewpoint image 520. For comparison, the free viewpoint image 520 is an image from the same viewpoint as images 500 and 510, but the construction status can be viewed from any viewpoint specified in three-dimensional space on the user device 10. Note that the free viewpoint image 520 also includes equipment and materials 522, but these are not highlighted because they are not related to the progress of the work.

[0040] Next, the control unit 21 of the management device 20 executes a process identification process according to the difference (step S17). Specifically, the difference display unit 214 of the control unit 21 recognizes the component included in the difference area and identifies the process that uses this component. Then, the difference display unit 214 generates progress management data including the date and time of the second point in time, the location where the second image was taken, and information about the process, and records the data in the progress information storage unit 23.

[0041] (Operation of the first embodiment) Since the free viewpoint data at different points in time generated by the free viewpoint image generation technique are compared, the location of the difference region in the free viewpoint image is identified.

[0042] (Effects of the first embodiment) According to the first embodiment, the following effects can be obtained. (1-1) In this embodiment, the control unit 21 of the management device 20 executes a process of acquiring a 2D image at a first time point (step S11) and a process of generating a 3D model (at the first time point) (step S12). The control unit 21 of the management device 20 also executes a process of acquiring a 2D image at a second time point (step S13) and a process of generating a 3D model (at the second time point) (step S14). The control unit 21 of the management device 20 then executes a process of comparing the free-viewpoint images (step S15). This allows the free-viewpoint images to be used to identify differences related to progress. For example, even if the shooting positions of the 2D image at the first time point and the 2D image at the second time point are different, the viewpoint position can be adjusted using the free-viewpoint images to identify the differences at a position that facilitates comparison.

[0043] (1-2) In this embodiment, the control unit 21 of the management device 20 executes a process for displaying the difference in the free viewpoint data (step S16). This allows the user to efficiently grasp the highlighted difference area in the free viewpoint image at a subsequent time. Furthermore, by changing the viewpoint position in the free viewpoint image, the sense of realism is improved, allowing the user to accurately confirm the difference area.

[0044] (1-3) In this embodiment, the control unit 21 of the management device 20 executes a process identification process in accordance with the difference (step S17), thereby enabling efficient management of the progress status.

[0045] (Second embodiment) Next, a second embodiment of the progress management method and progress management device will be described with reference to Fig. 7. In the first embodiment, the differential region is identified by a comparison process of free viewpoint data (step S15). In the second embodiment, the configuration is modified so that the differential region is identified by comparing two-dimensional images. The same parts as in the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

[0046] First, the control unit 21 of the management device 20 executes the process of specifying the processing target, similar to step S10 (step S20). Next, the control unit 21 of the management device 20 executes the process of acquiring a two-dimensional image at the first time point, similar to step S11 (step S21).

[0047] Next, the control unit 21 of the management device 20 executes the process of acquiring a two-dimensional image at a second time point, similar to step S13 (step S22). Next, the control unit 21 of the management device 20 executes a comparison process of the two-dimensional images (step S23). Specifically, the difference extraction unit 213 of the control unit 21 compares the two-dimensional image (first time point) with the two-dimensional image (second time point). In this case, the same subject included in the two-dimensional image (first time point) and the two-dimensional image (second time point) is identified using feature points.

[0048] Next, the control unit 21 of the management device 20 executes a differential region extraction process (step S24). Specifically, the difference display unit 214 of the control unit 21 identifies differential regions in the second image group that have changed from the first image group. In this case, the difference extraction unit 213 also selectively extracts only differential regions related to the progress of the process.

[0049] Next, the control unit 21 of the management device 20 executes a process of displaying the difference in the free viewpoint image (step S25). Specifically, the difference display unit 214 of the control unit 21 generates a three-dimensional model (third free viewpoint data) using a third image group that indicates the difference region in the second image group. In this case, the second image group may be visualized using a free viewpoint image generation method to generate a second free viewpoint image, and the third free viewpoint image may be displayed in correspondence with the second free viewpoint image. Next, the control unit 21 of the management device 20 executes a process identification process in accordance with the difference, similar to step S17 (step S26).

[0050] (Operation of the second embodiment) Since two-dimensional images are compared, the location of difference regions in the images is identified.

[0051] (Effects of the second embodiment) According to the second embodiment, in addition to the effects (1-2) and (1-3), the following effects can be obtained.

[0052] (2-1) In this embodiment, the control unit 21 of the management device 20 executes a comparison process of two-dimensional images (step S23) and a process of extracting a difference region (step S24). This makes it possible to extract differences related to progress using the two-dimensional images.

[0053] (Third embodiment) Next, a third embodiment of the situation confirmation method and situation confirmation device will be described with reference to Fig. 8. In the first embodiment, the difference region is identified by a comparison process of free viewpoint data (step S15). In the second embodiment, the difference region is identified by a comparison process of two-dimensional images (step S23). The third embodiment has a modified configuration in which point clouds are compared to identify the difference region, and parts similar to those in the above embodiments are given the same reference numerals and detailed description thereof will be omitted.

[0054] First, the control unit 21 of the management device 20 executes the process of specifying the processing target, similar to step S10 (step S30). Next, the control unit 21 of the management device 20 executes the process of acquiring a two-dimensional image at a first time point, similar to step S11 (step S31).

[0055] Next, the control unit 21 of the management device 20 executes a process of acquiring a point cloud at a first time point (step S32). Specifically, the image generation unit 212 of the control unit 21 uses a plurality of 2D images (a first image group at a first time point) to generate a first point cloud (3D) at a first time point as intermediate data for generating free viewpoint data of a 3D scene. This point cloud is generated by arranging feature points included in the 2D images in a 3D space according to the camera position obtained by an SfM algorithm or the like.

[0056] Next, the control unit 21 of the management device 20 executes the process of acquiring a two-dimensional image at a second time point, similar to step S13 (step S33). Next, the control unit 21 of the management device 20 executes a process of acquiring a point cloud at a second time point (step S34). Specifically, the image generation unit 212 of the control unit 21 uses the multiple 2D images (second image group at the second time point) to generate a second point cloud (3D) at the second time point as intermediate data for generating free viewpoint data of a 3D scene.

[0057] Next, the control unit 21 of the management device 20 executes a comparison process of the point cloud data (step S35). Specifically, the difference extraction unit 213 of the control unit 21 compares the first point cloud and the second point cloud. Here, common feature points are identified between the first point cloud and the second point cloud arranged in three-dimensional space, and the first point cloud and the second point cloud are aligned and compared.

[0058] Next, the control unit 21 of the management device 20 executes an extraction process for the three-dimensional position of the difference (step S36). Specifically, the difference extraction unit 213 of the control unit 21 identifies an area in the second point cloud that includes a difference area (added point cloud) that is not in the first point cloud. In this case, the difference extraction unit 213 selectively extracts only the area related to the progress of the process.

[0059] Next, the control unit 21 of the management device 20 executes a process for generating a 3D model (second time point) (step S37). Specifically, the image generation unit 212 of the control unit 21 generates the 3D model (second time point) using the point cloud (3D) at the second time point. Here, the image generation unit 212 generates a free viewpoint image to which color and brightness information of the point cloud is assigned using techniques such as rendering and index map for the second point cloud.

[0060] Next, the control unit 21 of the management device 20 executes a process of displaying the difference in the free viewpoint image (step S38). Specifically, the difference display unit 214 of the control unit 21 displays the position range of the difference region identified in the point cloud in the free viewpoint image (second time point). Next, the control unit 21 of the management device 20 executes a process identification process in accordance with the difference, similar to step S17 (step S39).

[0061] (Operation of the third embodiment) By comparing point clouds, the location of difference regions in three-dimensional space is identified.

[0062] (Effects of the third embodiment) According to the third embodiment, in addition to the effects (1-2) and (1-3), the following effects can be obtained.

[0063] (3-1) In this embodiment, the control unit 21 of the management device 20 executes a comparison process of point cloud data (step S35) and an extraction process of the three-dimensional position of the difference (step S36). This makes it possible to identify the location of the difference area in three-dimensional space using the point cloud.

[0064] This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility. In the above embodiment, the present invention is applied to the case of managing progress at a construction site. However, the application of the present invention is not limited to progress management at a construction site. For example, the present invention can be applied to displaying damage caused by an earthquake or changes in scenery.

[0065] In the above embodiment, the image capturing device C1, the user device 10, and the management device 20 are connected via a network. However, the hardware configuration is not limited to this. For example, the user device 10 and the management device 20 may be configured as an integrated device. In the above embodiment, a plurality of captured images taken by the image capture device C1 are used. The plurality of captured images may be generated by cutting out a plurality of frame images from a video captured by the image capture device C1. In the above embodiment, the SfM algorithm is used as the elemental technology for generating free viewpoint data, but the elemental technology for generating free viewpoint data is not limited to this. For example, SLAM (Simultaneous Localization and Mapping) and MVS (Multi-View Stereo) may be used. Furthermore, NeRF (Neural Radiance Fields) technology may be used to reconstruct the structure of a 3D scene.

[0066] In the first and third embodiments, a process for displaying the differences is executed in the free viewpoint image (steps S16, S25, S38). In addition, a text description of the difference region (image caption) may be displayed. In this case, the free viewpoint image at the second time point, in which the difference region is highlighted (annotated), is input to the generation AI, which uses natural language processing to generate text describing the difference region.

[0067] In the above embodiment, progress is identified by object recognition of the differential region. In addition to or instead of this, progress may be managed using design information. This design information may be, for example, BIM (Building Information Modeling). By using this BIM, various information on structural design and facility design is managed for the 3D model. In this case, progress is managed by identifying added components in the BIM information using position information of the differential region.

[0068] In the second embodiment, the control unit 21 of the management device 20 executes a process for displaying the differences in the free viewpoint images (step S25). Here, a 3D model is generated using a third image group that indicates the difference regions in the second image group. Alternatively, the difference regions may be highlighted in the free viewpoint images (at the second time point) generated using the second image group. In this case, an area corresponding to the difference region in the 2D image is identified in the free viewpoint images (at the second time point). [Explanation of symbols]

[0069] C1...imaging device, 10...user device, 20...management device, 21...control unit, 211...image acquisition unit, 212...image generation unit, 213...difference extraction unit, 214...difference display unit, 22...image information storage unit, 23...progress information storage unit.

Claims

1. A situation confirmation method that specifically displays, as a changed portion, the difference between first free viewpoint data that is captured at a first point in time and visualized using a free viewpoint image generation technique, and second free viewpoint data that is captured at a second point in time after the first point in time and visualized using the free viewpoint image generation technique.

2. the first free viewpoint data is an image of a construction situation in which a predetermined area is visualized from a predetermined viewpoint using the free viewpoint image generation method in a first image group captured at the first time point, 2. The status confirmation method according to claim 1, wherein the second free viewpoint data is an image of the construction status viewed from the specified viewpoint of the specified area visualized by the free viewpoint image generation method using a second group of images taken at the second time point.

3. A situation confirmation method in which a third image group, in which the differences between a first image group taken at a first point in time and a second image group taken at a second point in time after the first point in time are specifically displayed as changed parts, is visualized using a free viewpoint image generation technique to generate a third free viewpoint image.

4. The situation confirmation method according to claim 3 , wherein the second image group is visualized by the free viewpoint image generation technique to generate a second free viewpoint image, and the third free viewpoint image is made to correspond to the second free viewpoint image.

5. A situation confirmation method in which the difference between a first point cloud generated from a first image group taken at a first time point and a second point cloud generated from a second image group taken at a second time point after the first time point is identified as a changed portion, the second image group is visualized using a free viewpoint image generation method to generate a second free viewpoint image, and the difference is specifically displayed on top of the second free viewpoint image.

6. 6. The status checking method according to claim 1, wherein the specific display is a highlight display of the difference.

7. The situation confirmation method according to any one of claims 1 to 5, wherein the specific display is an image caption obtained by natural language processing of the difference.

8. A situation confirmation device for managing the situation at a site, a difference extraction unit that identifies a difference based on a first image group at a first time point and a second image group at a second time point after the first time point; and a difference display unit that uses the second image group to specifically display the difference as a changed portion of the situation in second free viewpoint data visualized by a free viewpoint image generation technique.

9. 9. The situation confirmation device according to claim 8, wherein the difference extraction unit identifies a difference between first free viewpoint data visualized by the first image group and second free viewpoint data visualized by the second image group.

10. The situation confirmation device according to claim 8 , wherein the difference extraction unit identifies differences between the first image group and the second image group.

11. The situation confirmation device according to claim 8 , wherein the difference extraction unit identifies differences between a first point cloud generated from the first image group and a second point cloud generated from the second image group.

Images