Site situation management system, site situation management method, and site situation management program

Abstract

The system reconstructs the 3D space of the site from on-site data acquired using any data acquisition device during routine patrols, thereby understanding the site conditions. [Solution] The field situation management system 1 includes one or more computers 2 that perform the following processes: extracting feature points that indicate characteristic parts included in the field data; estimating the acquisition position and orientation of the data acquisition device when the field data was acquired based on the correspondence between feature points in the field data; estimating the distribution of feature points in 3D space based on the feature points, acquisition position and orientation; reconstructing the 3D structure of the subject from the distribution of feature points in 3D space; generating a free-viewpoint image including an image of the subject as seen from a viewpoint other than the acquisition position from the 3D structure and information from at least one of the pixels or points of the surrounding field data corresponding to it; and displaying the free-viewpoint image.

Description

Technical Field

[0001] Embodiments of the present invention relate to on-site situation management technology.

Background Art

[0002] The construction plan in the construction of a new plant needs to be reviewed at any time according to the daily work progress. On-site, regular patrol inspections are carried out for work progress management or abnormality confirmation of the construction. However, conventional maintenance inspections are carried out by manual recording work, and cannot be accurately shared with the relevant personnel at the base based on the latest situation on site, and construction is carried out without the relevant personnel at the base being able to grasp it, and there is a risk of occurrence of a backward process. Therefore, technologies aiming at labor saving and mechanization of a series of inspection operations are demanded. Among these technologies, there is a restoration technology for converting the on-site situation into 3D data using images taken by a camera during patrol inspections. By this technology, it is possible to convert the on-site space of the patrol location into 3D data from the moving images taken while moving during patrol, and it is possible to grasp the latest equipment arrangement in a power plant or a factory three-dimensionally. For example, if this reconstructed 3D data is stored in a server, the situation on site at the time of inspection can be confirmed from any location. Also, it can be utilized for grasping changes from the time of construction or daily changes in renovation work. By converting to 3D data, it becomes possible to grasp the three-dimensional dimensional information of the space, so it can be used for grasping the dimensions of the location to be confirmed and for confirming interference when arranging equipment.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional technologies include systems that convert images into 3D data of the on-site space. This technology extracts distance images by performing stereoscopic viewing in overlapping regions of images obtained from two imaging devices. For example, it performs 3D sensing of the entire horizontal direction to ensure that distance accuracy does not change with the horizontal angle. However, the distance information required for 3D conversion is based on stereoscopic viewing and cannot be performed with a monocular camera. Another known technology is a system that monitors the status of equipment using 3D shape and images. This conventional technology grasps the overall situation from any viewpoint by mapping images onto a shape model in a 3D composite image. Furthermore, it enables detailed situational awareness using actual monitoring images and facilitates seamless monitoring by combining these. However, the shape model needs to be acquired by a large-scale sensor such as a laser.

[0005] The embodiments of the present invention have been made in consideration of these circumstances, and aim to reconstruct the three-dimensional space of the site from on-site data acquired by any data acquisition device during routine patrols, and to understand the on-site situation. [Means for solving the problem]

[0006] The field situation management system according to an embodiment of the present invention is subject to the reconstruction of a three-dimensional structure. One or more computers that perform the process of generating a free-viewpoint image when the user views the three-dimensional structure of the subject at the site as a viewpoint, in order for the user to manage the site where the subject is located, the The system inputs two or more field data sets that contain information about the subject, are either image or point cloud data, and are acquired by any data acquisition device at two or more different acquisition locations and arbitrary acquisition orientations. It extracts feature points that indicate characteristic parts included in the field data, estimates the acquisition location and orientation of the data acquisition device at the time the field data was acquired based on the correspondence between the feature points in the field data, and then, based on the feature points, acquisition location, and acquisition orientation, By determining the intersection points of lines projected onto feature points from different external parameters as 3D coordinates, The distribution of the feature points in three-dimensional space is estimated, the three-dimensional structure of the subject is reconstructed from the distribution of the feature points in three-dimensional space, and information from the three-dimensional structure and at least one of the pixels or points of the surrounding field data corresponding thereto, As any coordinate of the aforementioned three-dimensional structure The system includes one or more computers that perform processing to generate a free-viewpoint image, which includes an image of the subject as seen from a viewpoint other than the acquisition position, and to display the free-viewpoint image. [Effects of the Invention]

[0007] According to embodiments of the present invention, it is possible to reconstruct the three-dimensional space of a site and understand the site conditions from on-site data acquired by any data acquisition device during routine patrols. However, this effect is not limited to the present invention. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing the configuration of the field situation management system according to the first embodiment. [Figure 2] A block diagram showing the hardware configuration of the processing server. [Figure 3] A functional block diagram showing the flow of on-site situation management processing. [Figure 4] Plan view showing each viewpoint to explain free-viewpoint images. [Figure 5] A side view illustrating each viewpoint to explain free-viewpoint images. [Figure 6] An explanatory diagram showing a panoramic image. [Figure 7] An explanatory diagram showing a free-viewpoint image of the site from an overhead perspective. [Figure 8] An explanatory diagram showing a free-viewpoint image of a state as seen from any arbitrary viewpoint. [Figure 9] A functional block diagram showing the flow of the on-site situation management process in the second embodiment. [Figure 10] An explanatory diagram showing the process of excluding areas that are not to be restored. [Figure 11] An explanatory diagram showing a free-viewpoint image of a state as seen from any arbitrary viewpoint. [Figure 12] An explanatory diagram showing the inspection targets color-coded. [Figure 13] An explanatory diagram showing the display excluding items that are not subject to inspection. [Modes for carrying out the invention]

[0009] (First Embodiment) Hereinafter, embodiments of a site status management system, a site status management method, and a site status management program will be described in detail with reference to the drawings.

[0010] Reference numeral 1 in FIG. 1 is the site status management system of the first embodiment. A site status management method is implemented using this site status management system 1.

[0011] This site status management system 1 is a system for the user U to manage the site where inspection targets such as plants are placed. The site status management system 1 provides a large number of images taken at the site to the user U. Also, when there are multiple users U, these users U can share the images of the site with each other.

[0012] For example, the site status management system 1 is a system capable of sharing the latest information of the site that can be utilized for on-site construction work, and restores the extensive three-dimensional structure of the site from the omnidirectional images taken during daily patrols. The user U can grasp and manage the site status based on the restored three-dimensional structure.

[0013] The site status management system 1 includes a processing server 2, an inspection robot 3, and a management terminal 4. These are connected to each other via a predetermined network.

[0014] The user U is, for example, a plant manager. The management terminal 4 is a predetermined computer and is operated by the user U. The user U can access the processing server 2 via the management terminal 4 and grasp the site status by checking the images stored in the processing server 2.

[0015] The sites include power plants, chemical plants, and factories. These plants contain numerous pieces of equipment and structures that are subject to inspection. These pieces of equipment and structures are the subjects for 3D structural reconstruction and the subjects for anomaly detection. The sites may also be designated commercial or public facilities with large buildings.

[0016] The inspection robot 3 photographs the equipment and structures at the site. The inspection robot 3 is equipped with a camera 5 that captures a 360-degree panoramic view and travels around the plant using a predetermined mobile device including wheels 6. The area in which the inspection robot 3 moves may be indoors or outdoors. The inspection robot 3 photographs the site with the camera 5 while moving. The images captured by the inspection robot 3 are sent to the processing server 2. The images acquired by the camera 5 may be videos or still images.

[0017] Camera 5 constitutes the data acquisition equipment. In other words, the data acquisition equipment includes Camera 5, which is capable of capturing images in all directions. The field data also includes the all-around images captured by Camera 5. The all-around images include panoramic images that capture the entire horizontal circumference. Camera 5 may be a single unit capable of capturing images in all directions, or it may be a system that generates all-around images by combining images captured by multiple units.

[0018] Inspection robot 3 is exemplified as a robot that travels on floors or the ground. However, inspection robot 3 may take other forms. For example, inspection robot 3 may be a drone that flies over the site, or a drone that moves on or underwater. In other words, the data acquisition equipment is mounted on the drone or mobile robot that moves around the site.

[0019] Furthermore, data acquisition equipment may be carried or equipped by personnel moving around the site. For example, camera 5, which serves as data acquisition equipment, may be mounted on the helmet of a worker patrolling the site. Alternatively, a worker may carry camera 5 in their hand while patrolling the site.

[0020] Camera 5 may be a dedicated, general-purpose, or consumer-grade product. The omnidirectional image is a full-sphere panoramic image obtained by converting the image captured by Camera 5 into a flat, all-around viewable image. This conversion can be achieved, for example, by projection using equirectangular projection. The output resolution can be arbitrarily determined by the user U. For example, the output resolution may be 8K, 4K, or FullHD.

[0021] As shown in Figure 6, for example, a 360-degree panoramic image is an image in which the subject is distorted.

[0022] The image data may be in the form of a 3-channel color image composed of R, G, and B images. Alternatively, the image data may be in the form of a monochrome image, where each channel information of the color image is multiplied by a predetermined coefficient to create a 1-channel image. The pixel Y that makes up the monochrome image can be determined, for example, by Y = 0.2126*R + 0.7512*G + 0.0722*B.

[0023] The field data acquired by the data acquisition device is at least one of either image data or point cloud data. Images are acquired by camera 5. Point cloud data is generated by extracting feature points from the images acquired by camera 5. The data acquisition device may also be one that acquires only point cloud data, such as a laser scanner. The data acquisition device may also be one that acquires color images, such as a LiDAR or depth sensor.

[0024] The management terminal 4 outputs predetermined information. The field situation management system 1 includes a device for displaying images, such as a display that outputs analysis results. In other words, the management terminal 4 controls the images displayed on the display. The display may be separate from the computer main unit or integrated into it.

[0025] Although a display is given as an example of a device for displaying images, other forms may also be used. For example, images may be displayed using a head-mounted display or a projector. Furthermore, a printer that prints information on paper may be used instead of a display. In other words, the objects controlled by the management terminal 4 may include a head-mounted display, a projector, or a printer.

[0026] The site condition management system 1 generates free-viewpoint images based on images acquired at the site. For example, as shown in Figures 4 and 5, suppose camera 5 photographs a predetermined subject, inspection target Q, at two shooting points V1 and V2 at the site. When camera 5 photographs one inspection target Q at at least two shooting points V1 and V2, depth information of inspection target Q can be obtained. Based on this information, the three-dimensional structure of inspection target Q can be reconstructed. In this way, an image of inspection target Q as seen from a virtual viewpoint W can be obtained. Furthermore, the overall picture of the virtual site can be reproduced with the reconstructed three-dimensional structure. The reconstructed three-dimensional structure is called a free-viewpoint image because the user U can freely change their viewpoint.

[0027] For example, as shown in Figure 4, if camera 5 takes images at two horizontally offset shooting points V1 and V2, it can obtain an image as seen from a virtual viewpoint W located at a different horizontal position from these shooting points V1 and V2. Also, as shown in Figure 5, if camera 5 takes images at two vertically offset shooting points V1 and V2, it can obtain an image as seen from a virtual viewpoint W located at a different height from these shooting points V1 and V2.

[0028] A free-viewpoint image is, for example, an image that reproduces the three-dimensional structure of an object at the site, as shown in Figure 7. A free-viewpoint image can reproduce an image viewed from an overhead position different from the actual shooting position by camera 5. Then, as shown in Figure 8, user U can obtain an image based on the free-viewpoint image, viewing the three-dimensional structure at the site from any coordinate as the viewpoint.

[0029] Next, the hardware configuration of processing server 2 will be explained with reference to the block diagram shown in Figure 2.

[0030] As shown in Figure 2, the processing server 2 comprises an input unit 10, an output unit 11, a communication unit 12, a processing circuit 13, and a storage unit 14. The management terminal 4 has the same hardware configuration as the processing server 2.

[0031] Processing Server 2 is a computer equipped with hardware resources such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and SSD (Solid State Drive). The CPU executes various programs, enabling software-based information processing using these hardware resources. Furthermore, the field situation management method is implemented by having the computer execute various programs.

[0032] Each component of the Field Situation Management System 1 does not necessarily have to be installed on a single computer. For example, one Field Situation Management System 1 may be implemented using multiple computers connected to each other via a network. For instance, the image processing function and the machine learning function of the Field Situation Management System 1 may each be installed on separate computers.

[0033] Furthermore, the field situation management system 1 is implemented as a cloud service. In other words, the computer that constitutes the field situation management system 1 is the processing server 2 on the cloud. For example, it is also possible to have a configuration where not only the configuration that processes memory, but also the configuration that processes the main processing, all reside on the cloud, and user U only configures the field situation management system 1 and checks its input / output via an API (Application Programming Interface) or a web browser.

[0034] The input unit 10 receives predetermined information. This input unit 10 includes input devices such as a mouse, keyboard, and touch panel. In other words, predetermined information is input in accordance with the operation of these input devices.

[0035] The output unit 11 outputs predetermined information. The output unit 11 includes a device for displaying images, such as a display, which outputs the analysis results.

[0036] The communication unit 12 communicates with other computers via communication lines such as the internet. For example, images of the site acquired by the inspection robot 3 are input to the processing server 2 via the communication unit 12. Also, images output by the management terminal 4 are sent via the communication unit 12.

[0037] The memory unit 14 stores a predetermined program to be executed by the processing circuit 13. The memory unit 14 also stores various information necessary for performing predetermined processing. Furthermore, the memory unit 14 stores various information necessary for generating trained models. For example, the memory unit 14 stores images sent from the inspection robot 3.

[0038] The processing circuit 13 is, for example, a circuit equipped with a CPU, GPU, or a dedicated or general-purpose processor. This processor realizes various functions by executing various programs stored in the memory unit 14. The processing circuit 13 may also be composed of hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Various functions can also be realized by this hardware. Furthermore, the processing circuit 13 can realize various functions by combining software processing by the processor and programs with hardware processing.

[0039] Next, the flow of the on-site situation management process will be explained with reference to the functional block diagram shown in Figure 3. Processing server 2 may include components other than those shown in Figure 3, and some of the components shown in Figure 3 may be omitted.

[0040] The processing server 2 comprises a data acquisition processing unit 20, an estimation processing unit 21, a restoration processing unit 22, a generation processing unit 23, a display processing unit 24, and a registration processing unit 25. These are realized by the CPU executing a program stored in memory or on the HDD.

[0041] Furthermore, the processing server 2 includes a predetermined database 26 in which images are registered. This database 26 is a collection of information that is stored in memory, HDD, or cloud computing resources and organized so that it can be searched or stored.

[0042] Processing server 2 receives two or more field data sets acquired by any data acquisition device at two or more different acquisition locations and arbitrary acquisition orientations. The field data contains information about the subject to be reconstructed as a 3D structure, and is at least one of image or point cloud data. If the data acquisition device is camera 5, the acquisition location is the shooting location of camera 5, and the acquisition orientation is the orientation of camera 5 at the time of shooting. The acquisition orientation includes the field of view of camera 5.

[0043] The data acquisition processing unit 20 acquires on-site data sent from the inspection robot 3. The on-site data is, for example, an omnidirectional image. For each subject, the data acquisition processing unit 20 acquires at least two omnidirectional images of that subject.

[0044] The estimation processing unit 21 extracts feature points that indicate characteristic parts included in the field data. Here, the estimation processing unit 21 extracts a 3D structure of the subject from the field data, which is a point cloud of first density and composed of sparse points. Furthermore, the estimation processing unit 21 estimates the acquisition position and acquisition orientation of the data acquisition equipment when the field data was acquired, based on the correspondence between feature points in the field data.

[0045] For example, if the data acquisition device is a camera 5, the estimation processing unit 21 extracts feature points that indicate characteristic parts present in the image, and estimates the shooting position and the orientation of the camera 5 at the time of shooting based on the correspondence between feature points in the images.

[0046] A feature point is a pixel in a given image that represents a corner. A corner is a point where the pixel value changes significantly in all directions. For feature extraction, a method such as ORB (Oriented Fast and Rotated Brief) may be used.

[0047] The estimation processing unit 21 extracts identical feature points by comparing them with the recorded feature points. The correspondence between the extracted feature points can be calculated based on similarity calculation. The similarity calculation method can be, for example, Brute Force, Brute Force-L1, Brute Force-Hamming, or Flann Based.

[0048] The position and orientation of camera 5 can be calculated, for example, by simultaneously estimating the external parameters (R,t) of camera 5 and the feature point P, such that the projected feature point P and its back-projected point P' are minimized using a nonlinear least squares method. Here, R represents the rotational movement with respect to a reference point (e.g., the starting point for capturing omnidirectional images), and t represents the translational movement with respect to the reference point.

[0049] The reference point may be the origin of the local coordinate system. Furthermore, if the absolute scale of the field is known, the external parameters and feature points may be converted from local coordinates to global coordinates. Here, the local coordinates are, for example, the virtual coordinates used by the estimation processing unit 21. The global coordinates are, for example, the coordinates of the field in real space. In other words, the external parameters and feature points may be corrected to match the known scale of the field.

[0050] The reconstruction processing unit 22 estimates the distribution of feature points in three-dimensional space based on the feature points, acquisition position, and acquisition orientation. Furthermore, the reconstruction processing unit 22 reconstructs the three-dimensional structure of the subject from the distribution of feature points in three-dimensional space.

[0051] The reconstruction of a 3D structure can be performed by estimating the 3D distribution of feature points based on the correspondence between feature points in multiple omnidirectional images taken at different positions and orientations, and the external parameters (R,t) in those omnidirectional images. The 3D distribution of feature points can be determined by finding the intersection points of lines projected from different external parameters to the feature points as 3D coordinates. The reconstruction processing unit 22 performs this process on multiple feature points to reconstruct the 3D structure of the target.

[0052] The generation processing unit 23 generates a free-viewpoint image that includes an image of the subject as seen from a viewpoint W (Figure 4) other than the acquisition position, using information from the 3D structure and at least one of the pixels or points of the surrounding field data corresponding to it. For example, the generation processing unit 23 generates a free-viewpoint image that includes viewpoints W other than the shooting points V1 and V2 (Figure 4) using pixel information of the image around the feature points corresponding to the 3D structure.

[0053] Free-viewpoint images are generated by a technique that uses multiple images taken from different positions and orientations to generate images from positions that were not actually captured. This technique involves machine learning. For example, it can be achieved by simulating the changes in light intensity and radiance characteristics c that occur when light passes through an object before it enters camera 5, using deep learning. Light intensity is, for example, the absorptivity σ. Radiance characteristics c are, for example, color.

[0054] The technology used to generate free-viewpoint images can be implemented, for example, by using NeRF (Neural Radiance Field). In this case, the position x=(x,y,z) of camera 5 in 3D space and the viewpoint direction (θ,φ) are input, and a multilayer neural network is used that outputs the color and the light intensity at that position, thereby representing the luminance field. The luminance field is a function that represents what color and in what direction every point in space emits.

[0055] The technique for generating free-viewpoint images from a luminance field can also be volume rendering. Here, volume rendering is a method of drawing data with a three-dimensional extent onto a two-dimensional screen.

[0056] Here, let σ(x) be the light intensity at a certain position x. Let c(x,d) be the color corresponding to the viewpoint direction d. Let tn be the distance to the Near plane in the space that camera 5 renders in the image. Let tf be the distance to the Far plane.

[0057] The color of the pixel corresponding to the camera ray r(t) traversing space is calculated according to the volume rendering method. In this case, T(t) is the cumulative transmittance of the camera ray from tn to t. Discretizing this, it can be approximated by the following equation 1.

[0058]

number

[0059] Following this series of processes, an image is generated from the given shooting position as training data, the squared error between this image and the actually captured image is calculated, and the weights of the multilayer neural network are updated using backpropagation.

[0060] Another technique that can be used to generate free-viewpoint images is 3D-GS (3D Gaussian Splatting).

[0061] 3D-GS is similar to NeRF in that it takes the position x=(x,y,z) of camera 5 in 3D space and the viewpoint direction (θ,φ) as input. However, instead of calculating the color and light intensity at that position based on a multilayer neural network, it represents it as a Gaussian blur and outputs a point cloud with Gaussian information attached.

[0062] This Gaussian has learnable parameters such as position, size, and color. During training, each parameter is optimized so that the rendered image reproduces the actual image, and the density of the Gaussian is adaptively controlled at the same time. In addition, techniques may be used in which depth information corresponding to the pixels that make up the image is estimated and this information is used as a constraint.

[0063] Furthermore, the generation processing unit 23 generates a three-dimensional structure of the subject composed of a second density point cloud, which is denser than the first density point cloud, from the three-dimensional structure of the subject composed of a first density point cloud, through specific processing. In addition, the generation processing unit 23 generates a free-viewpoint image, including an image of the subject, from the three-dimensional structure composed of the second density point cloud.

[0064] The specific processing includes processing using a pre-trained model obtained through deep learning. Furthermore, the specific processing includes processing that performs at least one of the following on the first density point cloud: increasing or decreasing points, deforming points, changing the transparency of points, and changing the color of points. In addition, the specific processing includes processing that generates an image of the subject in a normal state so that the similarity, which is the degree to which the image of the subject is similar to the image of the subject during inspection, increases.

[0065] The generation processing unit 23 includes a trained model. The trained model includes an input layer, an intermediate layer, and an output layer. Input data is input to the input layer. The parameters of the intermediate layer are pre-machine-trained using training data. The output layer outputs output data that shows the results processed in the intermediate layer in response to the input data input to the input layer.

[0066] A trained model is machine-learned using training data that consists of at least one of the input data or data that mimics it as input, and at least one of the output data or data that mimics it as output.

[0067] The input data is, for example, a point cloud of first density, including a 3D structure of an object composed of sparse points. The input data is a 3D structure, which may be a coarse point cloud or an image. The output data includes, for example, a 3D structure of an object composed of a point cloud of second density, which is denser than the first density. The output data is an ellipse formed from the points of the input data, with the degree of light reflection and color set by deep learning.

[0068] The registration processing unit 25 registers the free-viewpoint image in the database 26 along with the acquisition time of the field data. The acquisition time includes the time of shooting. For example, if the field data that generated the free-viewpoint image consists of multiple images, the acquisition time is the average of the multiple shooting times. Note that the time also includes the meaning of the date. The field data can be saved in a format such as PLY (Polygon File Format).

[0069] The registration processing unit 25 executes a process to register the field data that generated the free-viewpoint image in the database 26, associating it with the time of acquisition.

[0070] The display processing unit 24 displays the free-viewpoint image generated by the generation processing unit 23 on the screen of the management terminal 4 (Figure 1). Alternatively, the display processing unit 24 may display free-viewpoint images registered in the database 26.

[0071] This display can be achieved by installing general-purpose viewer software on the management terminal 4. Alternatively, the processing server 2 can be set up as a web server, and the port can be forwarded on the management terminal 4 to display the information in a browser.

[0072] According to the first embodiment, the three-dimensional space of the site is reconstructed from images of the site taken by data acquisition equipment during daily inspections. Then, construction information such as the site conditions is shared among multiple users U who are involved, enabling labor savings in construction planning and a reduction in rework processes.

[0073] (Second Embodiment) Next, a second embodiment will be described using Figures 9 to 13. Note that components identical to those shown in the previously described embodiment are denoted by the same reference numerals, and redundant descriptions are omitted. Furthermore, the previously described drawings may be referenced as appropriate.

[0074] The flow of the on-site situation management process in the second embodiment will be explained with reference to the functional block diagram shown in Figure 9.

[0075] The processing server 2 of the second embodiment, like the first embodiment described above, includes a data acquisition processing unit 20, an estimation processing unit 21, a restoration processing unit 22, a generation processing unit 23, a display processing unit 24, a registration processing unit 25, and a database 26. Furthermore, the processing server 2 of the second embodiment includes a conversion processing unit 30, a noise reduction processing unit 31, a region extraction processing unit 32, and a change extraction processing unit 33. These are realized by the CPU executing a program stored in memory or on the HDD.

[0076] Database 26 contains free-viewpoint images linked to their acquisition time. Processing server 2 divides the free-viewpoint image into regions and extracts the 3D region of the subject to be inspected. Furthermore, processing server 2 compares a free-viewpoint image acquired at an arbitrary time with past free-viewpoint images already registered in database 26 that were generated before the arbitrary acquisition time. Processing server 2 calculates the difference in the 3D region of the subject between the free-viewpoint images at a predetermined position and direction to extract the changes over time. Then, processing server 2 displays the results of this change extraction on the screen of management terminal 4 (Figure 1).

[0077] The conversion processing unit 30 extracts an arbitrary range from the field data and generates a perspective projection image that has been corrected to eliminate distortion. For example, a perspective projection image without distortion in a predetermined direction is generated from an omnidirectional image (Figure 6).

[0078] A perspective projection image is, for example, an image of the distribution panel 40 as seen from the front, when the object being inspected is the distribution panel 40 (Figure 11).

[0079] Furthermore, the conversion processing unit 30 calculates the relative position and relative orientation of the data acquisition equipment at the time of acquiring the field data, by relatively converting them so that they match the perspective projection image. In addition, the conversion processing unit 30 projects feature points onto the perspective projection image based on the relative position and relative orientation.

[0080] The noise reduction processing unit 31 excludes areas of the field data that are not to be restored. For example, the noise reduction processing unit 31 excludes areas of the 3D structure of a subject included in the omnidirectional image (Figure 6) that are not to be restored from the input field data.

[0081] The noise reduction processing unit 31, for example as shown in Figure 10, removes the area containing an unwanted person 41 from the image if the image is field data. The noise reduction processing unit 31 also removes other areas that constitute noise. For example as shown in Figure 13, if the object being inspected is a distribution board 40, the noise reduction processing unit 31 blacks out the area other than the distribution board 40.

[0082] Furthermore, the reconstruction processing unit 22 uses the principle of triangulation to reconstruct the three-dimensional structure, including the subject and its surrounding structures, based on the perspective projection image from which unnecessary areas have been removed, the relative position and relative orientation, and the feature points projected onto the perspective projection image.

[0083] The region extraction processing unit 32 divides the free-viewpoint image into multiple regions. The region extraction processing unit 32 then extracts a specific region corresponding to the subject from the divided regions. Here, the multiple regions are, for example, multiple pixels. This process uses, for example, semantic segmentation.

[0084] The change extraction processing unit 33 calculates the difference between a specific region of the subject captured in a free-viewpoint image at an arbitrary acquisition time and a specific region of the subject captured in a free-viewpoint image at a past acquisition time. Then, the change extraction processing unit 33 extracts the time-series changes of the subject based on the difference. Here, the free-viewpoint images at past acquisition times are, for example, those registered in the database 26. The change extraction processing unit 33 extracts the difference in the 3D region of the subject that is the target of inspection between free-viewpoint images at a predetermined position and direction.

[0085] The display processing unit 24 displays a free-viewpoint image from an arbitrary acquisition time, a free-viewpoint image from a past acquisition time, and the change extraction results, which extract the time-series changes of the subject, on the screen of the management terminal 4 (Figure 1).

[0086] The display processing unit 24 may, for example, display a portion of the distribution board 40 in a different color if the object being inspected is the distribution board 40 (Figure 12). The change in color may then indicate a change in the object being inspected.

[0087] Next, a specific example of processing will be described. The conversion processing unit 30 generates, for example, a distortion-free perspective projection image (Figure 11) in a predetermined direction from an omnidirectional image (Figure 6). Here, the conversion processing unit 30 calculates the relative position and relative orientation of the perspective projection image obtained by relatively transforming the acquisition position and acquisition orientation of the data acquisition device.

[0088] The conversion processing unit 30 projects the feature points onto the perspective projection image using the calculated relative position and relative orientation. Here, the angle of perspective projection may be freely set in the horizontal and vertical directions. Alternatively, an angle may be set by a combination of both directions. Furthermore, the internal parameter K of the camera 5 corresponding to the perspective projection image is expressed by the following equation 2.

[0089]

number

[0090] Here, the focal length f and the optical center (Cx,Cy) can be freely set. Also, the distortion coefficient may be set to, for example, 0.

[0091] Furthermore, the relative position and relative orientation of the perspective projection image, where the orientation (Rp,tp) corresponding to the perspective projection image is determined by the following formula, where R is the relative rotation component of the perspective projection image with respect to the direction of travel of camera 5.

[0092] (Rp,tp)=(R)^(-1)(R,t)

[0093] Projecting feature points onto a perspective projection image can be achieved by determining the coordinates of the feature points using their relative position and relative orientation.

[0094] The noise reduction processing unit 31 removes, for example, areas included in the omnidirectional image that are not subject to 3D structure reconstruction from the input omnidirectional image. Noise includes, for example, people 41 (Figure 10), such as the photographer operating the camera 5. Also, parts of the housing of the inspection robot 3 equipped with the camera 5 are also considered noise.

[0095] The noise reduction method can be, for example, an algorithm that detects areas belonging to the YOLO category, such as person 41 (Figure 10), or an algorithm that detects specific noise. Alternatively, the noise reduction method can be implemented by extracting the noisy areas from the image and deleting the image containing these areas. Another method is to create a mask that explicitly identifies the noisy areas and generate a free-viewpoint image that excludes these masked areas from the calculation. For example, if the object being inspected is a power distribution panel 40, an image (Figure 13) can be generated with all areas except the power distribution panel 40 masked in black. The image from which the noisy areas are extracted can be an omnidirectional image or a perspective projection image.

[0096] The region extraction processing unit 32 divides the region of the free-viewpoint image and extracts the 3D region of the subject to be inspected. The extraction method can be achieved by dividing the region of the omnidirectional image or perspective projection image and projecting that information into 3D space.

[0097] Next, we will explain an example of segmenting a 360-degree image into regions. Segmenting a 360-degree image into regions can be done, for example, by performing semantic segmentation. The semantic segmentation model may be a pre-trained model that has been machine-learned to segment specific regions, or it may be a pre-trained model that has been machine-learned using a large database and is capable of segmenting an unspecified number of regions. Labels may then be assigned to the segmented regions. Here, the labels are training labels that indicate that they are the subject of the extraction target.

[0098] Projection into three-dimensional space can be achieved by displaying each pixel corresponding to each region of the omnidirectional image in its corresponding three-dimensional coordinates. The display method may, for example, involve using color information to clearly indicate each region.

[0099] The change extraction processing unit 33 extracts from the database 26 free-viewpoint images that show the same subject as the free-viewpoint image taken at an arbitrary acquisition time, but which were taken at an earlier acquisition time than the arbitrary acquisition time. Furthermore, the change extraction processing unit 33 calculates the difference between the subject shown in the free-viewpoint image taken at the arbitrary acquisition time and the subject shown in the free-viewpoint image taken at an earlier acquisition time, and extracts the time-series changes of the subject based on the difference.

[0100] Furthermore, the change extraction processing unit 33 projects the subject captured in the free-viewpoint image into a virtual three-dimensional space. The change extraction processing unit 33 also calculates the difference based on the viewpoints of the subject at the same location viewed from multiple different positions and directions, and extracts the time-series changes of the subject based on the difference. In this way, since the difference can be calculated from multiple positions and directions, the accuracy of extracting changes in the subject can be improved.

[0101] For example, if the object to be inspected is a distribution panel 40, the system generates a first field-of-view image (Figure 11) showing the distribution panel 40 from the front, and a second field-of-view image (Figure 13) showing the distribution panel 40 from an oblique side view. The change extraction processing unit 33 compares the first field-of-view image at an arbitrary acquisition time with a first field-of-view image from a past acquisition time and calculates the difference. Furthermore, the change extraction processing unit 33 compares the second field-of-view image at an arbitrary acquisition time with a second field-of-view image from a past acquisition time and calculates the difference. Based on these multiple differences, the change extraction processing unit 33 extracts the time-series changes of the object.

[0102] According to the second embodiment, if there is an abnormality in the object being inspected, that abnormality can be detected. Furthermore, false detection of abnormalities can be suppressed, improving inspection efficiency and enabling inspections to be performed with less or no human intervention.

[0103] Although the present invention has been described above based on the first and second embodiments, a configuration applied in any one embodiment may be applied to another embodiment, or the configurations applied in each embodiment may be combined.

[0104] In the example described above, the computer constituting the field situation management system 1 performs various processes, but other configurations are also possible. For example, user U may perform some of the processes described above, and the computer may receive the results of those processes as input and use them in its own processing.

[0105] Note that the arrows in the functional block diagram are just one example of a processing flow, and there may be other processing flows besides those indicated by the arrows. Also, the order of each process is not necessarily fixed, and the order of some processes may be reversed. Furthermore, some processes may be executed in parallel with other processes.

[0106] The aforementioned field situation management system 1 comprises a control device, a storage device, an output device, an input device, and a communication interface. Here, the control device includes a highly integrated processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or a dedicated chip. The storage device includes ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The output device includes a display panel, head-mounted display, projector, printer, etc. The input device includes a mouse, keyboard, touch panel, etc. This field situation management system 1 can be implemented with a hardware configuration using a standard computer.

[0107] Furthermore, the program or learned model executed by the aforementioned field situation management system 1 is provided pre-loaded into ROM or similar media. Additionally or alternatively, this program or learned model is provided as an installable or executable file stored on a computer-readable, non-temporary storage medium. This storage medium includes CD-ROMs, CD-Rs, memory cards, DVDs, flexible disks (FDs), and the like.

[0108] Furthermore, the program or trained model executed by this field situation management system 1 may be stored on a computer connected to a network such as the Internet and provided for download via the network. In other words, the program or trained model may be provided from cloud computing resources. Alternatively, a server on the cloud may execute the program or trained model, and only the processing results may be provided via the cloud. Moreover, this system can also be configured by connecting and combining separate modules, each independently performing the functions of its components, via a network or dedicated line.

[0109] The network in the aforementioned embodiment is a communication line such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), or mobile communication network.

[0110] Furthermore, the field situation management system 1 may include a computer equipped with artificial intelligence (AI) for machine learning. The field situation management system 1 may also include a deep learning unit that extracts specific patterns from multiple patterns based on deep learning.

[0111] The aforementioned computer-based analysis can utilize analytical techniques based on artificial intelligence learning. For example, trained models generated by machine learning using neural networks, trained models generated by other machine learning methods, deep learning algorithms, and mathematical algorithms such as regression analysis can be used. Furthermore, forms of machine learning include clustering and deep learning.

[0112] The field situation management system 1 includes a computer equipped with artificial intelligence that performs machine learning. For example, this field situation management system 1 may consist of one computer equipped with a neural network, or it may consist of multiple computers equipped with neural networks.

[0113] Here, a neural network is a mathematical model that represents the characteristics of brain function through computer simulation. For example, it shows a model in which artificial neurons (nodes) that form a network through synaptic connections change the strength of their synaptic connections through learning and acquire problem-solving abilities. Furthermore, neural networks acquire problem-solving abilities through deep learning.

[0114] For example, a neural network may have multiple layers, each consisting of several units. By pre-training a multi-layer neural network with training data (supervised data), it is possible to automatically extract features from patterns of changes in the state of a circuit or system. Furthermore, the number of hidden layers, units, learning rate, number of training iterations, and activation function of a multi-layer neural network can be set arbitrarily via the user interface.

[0115] Furthermore, a reward function may be set for each information item to be learned, and deep reinforcement learning, in which the information item with the highest value is extracted based on the reward function, may be used in the neural network.

[0116] One method used to estimate the position of camera 5 from the image is Visual-SLAM (Visual Simultaneous Localization and Mapping), which extracts feature points from the video and simultaneously creates a map of the camera 5's position and its surrounding environment.

[0117] Visual-SLAM extracts feature points from objects in an image and calculates the shooting position and direction based on these feature points. For example, by using a predetermined location with a known position as the starting point (shooting start point) and calculating the movement trajectory of camera 5 from that starting point, the shooting position and direction can be calculated. Furthermore, Visual-SLAM does not require prior acquisition of the 3D structure of the site; it can calculate the shooting position and direction from images alone.

[0118] According to at least one embodiment described above, the field situation management system 1 generates free-viewpoint images that include images of subjects visible from viewpoints other than the acquisition position of the data acquisition device. This makes it possible to reconstruct the three-dimensional space of the site from the field data acquired by any data acquisition device during daily patrols and to understand the field situation.

[0119] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, modifications, and combinations are possible without departing from the spirit of the invention. These embodiments or their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. Where there is a singular noun, it does not exclude plural nouns unless the context clearly indicates otherwise. Furthermore, conjunctions such as "and" and "or" are inclusive unless the context clearly indicates otherwise. [Explanation of Symbols]

[0120] 1...Site situation management system, 2...Processing server, 3...Inspection robot, 4...Management terminal, 5...Camera, 6...Wheels, 10...Input unit, 11...Output unit, 12...Communication unit, 13...Processing circuit, 14...Storage unit, 20...Data acquisition processing unit, 21...Estimation processing unit, 22...Restoration processing unit, 23...Generation processing unit, 24...Display processing unit, 25...Registration processing unit, 26...Database, 30...Conversion processing unit, 31...Noise reduction processing unit, 32...Region extraction processing unit, 33...Change extraction processing unit, 40...Distribution board, 41...Person, Q...Inspection target, U...User, V1,V2...Shooting point, W...Viewpoint.

Claims

1. One or more computers that perform a process to generate free-viewpoint images when an arbitrary coordinate of the three-dimensional structure of an object located at a site is used as the viewpoint, in order for a user to manage the site where the object is located. The data includes information about the subject, is at least one of image or point cloud data, and inputs two or more field data acquired by any data acquisition device at two or more different acquisition locations and any acquisition orientation. Extract feature points that indicate characteristic parts included in the aforementioned field data, Based on the correspondence between the feature points in the aforementioned field data, the acquisition position and acquisition orientation of the data acquisition device at the time the field data was acquired are estimated. Based on the feature points, the acquisition position, and the acquisition orientation, the distribution of the feature points in three-dimensional space is estimated by determining the intersection points of lines projected onto the feature points from different external parameters as three-dimensional coordinates. From the distribution of the feature points in the three-dimensional space, the three-dimensional structure of the subject is reconstructed. From the information of the three-dimensional structure and at least one of the pixels or points of the surrounding field data corresponding thereto, a free-viewpoint image is generated that includes an image of the subject as seen from a viewpoint other than the acquisition position, as an arbitrary coordinate of the three-dimensional structure. Display the aforementioned free viewpoint image. A system comprising one or more computers that perform processing, Site situation management system.

2. The aforementioned computer, A perspective projection image is generated by extracting an arbitrary range from the aforementioned field data and correcting it to eliminate distortion. The relative position and relative orientation are calculated by relatively transforming the acquisition position and acquisition orientation so that they fit the perspective projection image. Based on the relative position and relative orientation, the feature points are projected onto the perspective projection image. Based on the perspective projection image, the relative position and relative orientation, and the feature points projected onto the perspective projection image, the three-dimensional structure, including the subject and its surrounding structure, is reconstructed using the principle of triangulation. It is what executes the process. The on-site situation management system according to claim 1.

3. The aforementioned computer performs a process to exclude areas of the field data that are not to be restored from the field data. A field situation management system according to claim 1 or claim 2.

4. The computer performs the process of registering the field data that generated the free-viewpoint image into a database, associating it with the acquisition time. A field situation management system according to claim 1 or claim 2.

5. The aforementioned computer, From the database, extract free-viewpoint images that show the same subject as the free-viewpoint image at any given acquisition time, and which were acquired at an earlier time than the given acquisition time. The difference between the subject captured in the free-viewpoint image at any given acquisition time and the subject captured in the free-viewpoint image at a past acquisition time is calculated. Based on the aforementioned difference, the time-series changes of the subject are extracted. It is what executes the process. The on-site situation management system according to claim 4.

6. The aforementioned computer, The subject captured in the free-viewpoint image is projected onto the virtual three-dimensional space. The difference is calculated based on viewpoints from which the subject at the same location is viewed from multiple different positions and directions. Based on the aforementioned difference, the time-series changes of the subject are extracted. It is what executes the process. The field situation management system according to claim 5.

7. The aforementioned computer, The aforementioned free-viewpoint image is divided into multiple regions, A specific region corresponding to the subject is extracted from the multiple divided regions. The difference between the specific region of the subject captured in the free-viewpoint image at any given acquisition time and the specific region of the subject captured in the free-viewpoint image at a past acquisition time is calculated. Based on the aforementioned difference, the time-series changes of the subject are extracted. It is what executes the process. The field situation management system according to claim 5.

8. The computer performs a process to display the free-viewpoint image at any acquisition time, the free-viewpoint image at past acquisition times, and the change extraction result which extracts the time-series changes of the subject. The field situation management system according to claim 5.

9. The aforementioned data acquisition device includes a camera capable of capturing images in all directions. The aforementioned on-site data includes an all-around image captured by the aforementioned camera. The field situation management system according to claim 5.

10. A process for generating a free-viewpoint image when an arbitrary coordinate of the three-dimensional structure of an object located at a site is used as the viewpoint, in order for the user to manage the site where the object, which is the subject of the three-dimensional structure restoration, is located. The data includes information about the subject, is at least one of image or point cloud data, and inputs two or more field data acquired by any data acquisition device at two or more different acquisition locations and any acquisition orientation. Extract feature points that indicate characteristic parts included in the aforementioned field data, Based on the correspondence between the feature points in the aforementioned field data, the acquisition position and acquisition orientation of the data acquisition device at the time the field data was acquired are estimated. Based on the feature points, the acquisition position, and the acquisition orientation, the distribution of the feature points in three-dimensional space is estimated by determining the intersection points of lines projected onto the feature points from different external parameters as three-dimensional coordinates. From the distribution of the feature points in the three-dimensional space, the three-dimensional structure of the subject is reconstructed. From the information of the three-dimensional structure and at least one of the pixels or points of the surrounding field data corresponding thereto, a free-viewpoint image is generated that includes an image of the subject as seen from a viewpoint other than the acquisition position, as an arbitrary coordinate of the three-dimensional structure. Display the aforementioned free viewpoint image. The process is performed by one or more computers. Methods for managing on-site conditions.

11. A process for generating a free-viewpoint image when an arbitrary coordinate of the three-dimensional structure of an object located at a site is used as the viewpoint, in order for the user to manage the site where the object, which is the subject of the three-dimensional structure restoration, is located. The data includes information about the subject, is at least one of image or point cloud data, and inputs two or more field data acquired by any data acquisition device at two or more different acquisition locations and any acquisition orientation. Extract feature points that indicate characteristic parts included in the aforementioned field data, Based on the correspondence between the feature points in the aforementioned field data, the acquisition position and acquisition orientation of the data acquisition device at the time the field data was acquired are estimated. Based on the feature points, the acquisition position, and the acquisition orientation, the distribution of the feature points in three-dimensional space is estimated by determining the intersection points of lines projected onto the feature points from different external parameters as three-dimensional coordinates. From the distribution of the feature points in the three-dimensional space, the three-dimensional structure of the subject is reconstructed. From the information of the three-dimensional structure and at least one of the pixels or points of the surrounding field data corresponding thereto, a free-viewpoint image is generated that includes an image of the subject as seen from a viewpoint other than the acquisition position, as an arbitrary coordinate of the three-dimensional structure. Display the aforementioned free viewpoint image. To have one or more computers perform the process. A site situation management program.

Images