Information processing device, method and program

Abstract

To provide a novel technique for recognizing unit structures constituting a structure in an image in which the structure is captured.SOLUTION: An information processing device 1 comprises a feature detecting unit 28 and a unit structure estimating unit 29. The feature detecting unit detects features relating to an external structure from a structure image obtained by capturing a structure including a plurality of unit structures each of which has an identical or similar external structure by an imaging device. The unit structure estimation unit estimates a potion relating to each of the plurality of unit structures included in the structure in the structure image on the basis of the plurality of detected features.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to image processing techniques. [Background technology]

[0002] Conventionally, an image processing device has been proposed that is capable of acquiring a panoramic image of a subject that is wider than the shooting range of a camera by including an object extraction unit that extracts a portion of each captured image that includes a predetermined velocity vector as part of the subject, a relative position calculation unit that calculates the relative position of each captured image, and a creation unit that arranges each captured image based on the relative positions to create a wide-area image that includes a range wider than the shooting range (see Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Patent Document 1] JP 2020-182048 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, digital transformation (DX) has been promoted for monitoring and inspection of structures from the viewpoint of improving safety and optimizing costs. Here, when a structure to be monitored or inspected includes multiple unit structures having the same or similar external structure (for example, a structure in which reinforcing bars or the like cross in an X shape), it is necessary to understand the correspondence between the elements in the image of the structure and the positions of the unit structures in the structure. However, the conventionally proposed technology has an issue in the process of recognizing the unit structures that make up the structure in the image of the structure.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide a novel technique for recognizing unit structures that make up a structure in an image of the structure. [Means for solving the problem]

[0006] One example of the present disclosure is an information processing device that includes a feature detection means that detects features related to an external structure from a structure image obtained by imaging a structure including a plurality of unit structures each having the same or similar external structure using an imaging device, and a unit structure estimation means that estimates portions of the structure image related to each of the plurality of unit structures included in the structure based on the detected features.

[0007] The present disclosure can be understood as an information processing device, a system, a method executed by a computer, or a program executed by a computer. The present disclosure can also be understood as such a program recorded on a recording medium readable by a computer or other device, machine, etc. Here, a recording medium readable by a computer, etc. refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. Effect of the Invention

[0008] According to the present disclosure, it is possible to provide a novel technique for recognizing unit structures that make up a structure in an image of the structure. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing a configuration of a system according to an embodiment. [Diagram 2] 1 is a diagram illustrating an outline of a functional configuration of an information processing device according to an embodiment. [Diagram 3] FIG. 2 is a schematic diagram showing a flight pattern when imaging a structure in an embodiment. [Figure 4] FIG. 11 is a diagram showing a graph in which pixels are arranged in ascending order of magnitude in an embodiment. [Diagram 5] FIG. 11 is a diagram showing an example of a histogram showing the distribution of magnitude in a captured image, which is created in the embodiment. [Figure 6] FIG. 1A is a diagram showing an overview of unit estimation in an embodiment. [Figure 7] FIG. 13B is a diagram showing an outline of unit estimation in the embodiment. [Figure 8] FIG. 11 is a diagram showing how components detected from a structure image are clustered in the embodiment. [Figure 9] FIG. 11 is a diagram showing a state in which components detected from design data are clustered in the embodiment. [Figure 10] 10 is a flowchart showing a flow of a panoramic image generating process according to the embodiment. [Figure 11] 1 is a flowchart showing a flow of an inspection process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of an information processing device, a method, and a program according to the present disclosure will be described with reference to the drawings. However, the embodiment described below is an example of an embodiment, and the information processing device, a method, and a program according to the present disclosure are not limited to the specific configuration described below. In carrying out the present disclosure, a specific configuration according to the embodiment may be appropriately adopted, and various improvements and modifications may be made.

[0011] In this embodiment, an embodiment will be described in which the technology according to the present disclosure is implemented for monitoring and inspection of structures such as steel towers, etc. However, the technology according to the present disclosure can be widely used for processing images of captured objects, and the application of the present disclosure is not limited to the examples shown in the embodiment.

[0012] Traditionally, monitoring and inspection of structures such as steel towers that are expected to be used as base stations for communication networks have generally been done visually, but in recent years, digital transformation (DX) for the inspection of such structures has been promoted, and various technologies have been proposed to replace the previous visual monitoring and inspection with monitoring and inspection using flying objects such as drones. For example, an inspection technology that makes full use of image processing and image analysis of structures has been proposed, and it is expected that this inspection technology will reduce the effort and burden of manual work involved in inspecting structures, as well as easily identify each unit structure and component in the structure and detect abnormalities. However, the previously proposed technologies have had various issues.

[0013] For example, various techniques have been proposed for extracting image elements showing a specific part of a structure (e.g., the structure closest to the imaging position) from a captured image in order to monitor or inspect a structure based on the captured image. However, there are problems with the extraction accuracy when using machine learning or the like to extract the image elements, and when it is desired to improve the accuracy, an additional sensor (e.g., a ToF (Time of Flight) sensor) is required to detect the three-dimensional shape of the structure. Thus, there remain problems in improving the accuracy when extracting image elements showing a specific part of the imaging target from a captured image.

[0014] In addition, in this embodiment, in order to monitor or inspect a long structure such as a steel tower, an image processing technique is adopted in which a group of captured images of the structure captured by an air vehicle such as a drone is synthesized to generate a single panoramic image showing the entire structure. However, in the conventionally proposed technique, the air vehicle such as a drone, which is the imaging means, may be affected by an accidental event such as a gust of wind, causing the velocity vector of the air vehicle itself to change. When such an accidental event occurs, it may be difficult to identify the relative positions of the captured images, and it may be difficult to generate a suitable panoramic image.

[0015] In addition, when monitoring or inspecting a long structure such as a steel tower, it is necessary to know which part of the structure corresponds to the captured image of the structure. In addition, considering that a structure such as a steel tower is composed of many parts, it is necessary to know which part of the structure is indicated by any one captured image that corresponds to only a part of the structure. However, the conventionally proposed technology has room for improvement in terms of comprehensively recognizing the unit structures and parts that constitute the structure in the captured image of the structure.

[0016] In consideration of the above-mentioned problems, the information processing device, method, and program of this embodiment make it possible to improve the accuracy of extracting image elements in which a specific portion of a structure is captured from a captured image, generate a suitable panoramic image even when the image of the structure is affected by an accidental event, and provide a new technology for comprehensively recognizing the unit structures and components that make up a structure in an image of the structure.

[0017] <System configuration> 1 is a schematic diagram showing the configuration of a system according to the present embodiment. The system according to the present embodiment includes an information processing device 1, a drone 8, and a user terminal 9, which are connected to a network and capable of communicating with each other.

[0018] The information processing device 1 is a computer including a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a storage device 14 such as an electrically erasable and programmable read only memory (EEPROM) or a hard disk drive (HDD), a communication unit 15 such as a network interface card (NIC), etc. However, the specific hardware configuration of the information processing device 1 can be omitted, replaced, or added as appropriate depending on the embodiment. In addition, the information processing device 1 is not limited to a device consisting of a single housing. The information processing device 1 may be realized by multiple devices using so-called cloud or distributed computing technology, etc.

[0019] The drone 8 is a small unmanned aerial vehicle whose flight is controlled according to an external input signal and / or a program recorded in the device, and includes a propeller, a motor, a CPU, a ROM, a RAM, a storage device, a communication unit, an input device, an output device, and the like (not shown). However, the specific hardware configuration of the drone 8 can be omitted, replaced, or added as appropriate depending on the embodiment. The drone 8 according to this embodiment also includes an imaging device 81, and when flying around a target structure, images of the structure are captured according to an external input signal and / or a program recorded in the device.

[0020] The user terminal 9 is a terminal device used by a user. The user terminal 9 is a computer equipped with a CPU, a ROM, a RAM, a storage device, a communication unit, an input device, an output device, and the like (not shown). However, the specific hardware configuration of the user terminal 9 can be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the user terminal 9 is not limited to a device consisting of a single housing. The user terminal 9 may be realized by a plurality of devices using so-called cloud or distributed computing technology, etc. The user uses various services provided by the system according to this embodiment via these user terminals 9.

[0021] 2 is a diagram showing an outline of the functional configuration of the information processing device according to this embodiment. The information processing device 1 functions as an information processing device including a captured image acquisition unit 21, a movement acquisition unit 22, a threshold determination unit 23, a histogram creation unit 24, a range estimation unit 25, an image element identification unit 26, an image connection unit 27, a feature detection unit 28, a unit structure estimation unit 29, a member detection unit 30, a cluster creation unit 31, a cluster correspondence unit 32, a member correspondence unit 33, a position alignment unit 34, and a construction state determination unit 35, by a program recorded in the storage device 14 being read into the RAM 13 and executed by the CPU 11, and each hardware included in the information processing device 1 is controlled. Note that in this embodiment and other embodiments described later, each function included in the information processing device 1 is executed by the CPU 11, which is a general-purpose processor, but some or all of these functions may be executed by one or more dedicated processors.

[0022] The captured image acquisition unit 21 acquires a plurality of captured images. Here, the method of acquiring the captured images is not limited, but in this embodiment, an example will be described in which an image captured by a drone 8 equipped with an imaging device 81 is acquired via a user terminal 9. The captured images may be a plurality of still images cut out from a video, or a plurality of still images captured continuously. Note that the plurality of captured images acquired in this embodiment are captured images obtained by imaging a predetermined portion (in this embodiment, one side of the structure) of a structure (for example, a steel tower) that is an imaging target, while changing the positional relationship between the imaging target and the imaging position by an imaging device provided on a flying object that flies substantially vertically while maintaining a substantially constant distance and a substantially constant speed during imaging, and a plurality of unit structures and a plurality of predetermined members included in the structure are simultaneously imaged in the captured images.

[0023] FIG. 3 is a schematic diagram showing a flight pattern when imaging a structure in this embodiment. Taking a case where the structure is a steel tower as an example, in this embodiment, the worker captures a video while flying the drone from the bottom to the top of the steel tower in a substantially vertical manner with the imaging device mounted on the drone facing the side of the steel tower to be inspected, thereby obtaining a video in which the entire side of the steel tower to be inspected is sequentially captured from the bottom to the top. In this way, it is possible to obtain a video in which all of the unit structures (in this embodiment, units in which bars such as reinforcing bars constituting the steel tower cross in an X shape) and the plurality of predetermined members (in this embodiment, fastening materials) included in the structure that can be captured from the side of the inspection target are captured. Note that the flight pattern of the drone is not limited to the substantially vertical flight exemplified in this embodiment, and may be set appropriately according to the shape of the structure.

[0024] The motion acquisition unit 22 acquires a vector (so-called optical flow) indicating the motion of the imaging target in the image for each of the multiple image elements constituting the captured image based on the multiple captured images. The optical flow acquired here includes a parameter (hereinafter referred to as "magnitude") indicating the magnitude of the optical flow for each image element. Here, the image element may be a pixel included in the captured image, a set of pixels in which the imaging target is captured in the captured image, or a feature point indicating the imaging target specified in the captured image, but in this embodiment, an example in which pixels included in the captured image are used as the image element will be described. More specifically, the motion acquisition unit 22 acquires an optical flow for each of the multiple image elements constituting the captured image by comparing images that are adjacent in time series based on multiple captured images (for example, multiple frames constituting a video) that are acquired by the captured image acquisition unit 21. Note that in this embodiment, an example in which dense optical flow analysis that analyzes the motion of the entire pixels is used will be described, but a specific method for optical flow analysis is not limited.

[0025] The threshold determination unit 23 determines a threshold for excluding image elements that are not included in a histogram showing the distribution of the magnitudes of a plurality of image elements based on the magnitudes of the image elements. For example, the threshold determination unit 23 may arrange image elements included in at least a series of images captured consecutively among a plurality of captured images in ascending / descending order of magnitude, identify a point in the arrangement where a change in magnitude exceeds a predetermined criterion, and determine the magnitude corresponding to the point as the threshold.

[0026] FIG. 4 is a diagram showing a graph in which pixels are arranged in ascending order of magnitude in this embodiment. In this figure, the horizontal axis shows pixels arranged in ascending order of magnitude, and the vertical axis shows the magnitude of each pixel. The threshold determination unit 23 creates a graph in which pixels are arranged in ascending order of magnitude, and identifies a "steep gradient" portion where the change in magnitude exceeds a predetermined standard based on the graph. There is no specific limit to the method for identifying the "steep gradient" portion. For example, the threshold determination unit 23 can identify a "steep gradient" portion where the change in magnitude exceeds a predetermined standard by using a method of calculating the gradient (slope) of a tangent to a line graph connecting magnitudes on the graph and identifying a portion where the gradient exceeds a predetermined threshold, or a method of identifying a portion where the difference or ratio of magnitudes between adjacent or nearby pixels on the graph exceeds a predetermined threshold. Note that, although an example in which pixels are arranged in ascending order of magnitude is shown in FIG. 4, the magnitudes may be arranged in descending order.

[0027] Then, the threshold determination unit 23 determines, as a percentage for specifying the threshold, the ratio of the number of pixels having a magnitude equal to or less than (or less than) the magnitude of the specified location to the number of all pixels arranged on the graph, or the ratio of the number of pixels having a magnitude greater than (or equal to or greater than) the magnitude of the specified location to the number of all pixels arranged on the graph (i.e., the ratio of the number of pixels on either the left or right side of the location where the steep gradient is identified in the graph of FIG. 4) as a percentage for specifying the threshold. More specifically, in this embodiment, the ratio of the number of pixels having a magnitude equal to or less than the magnitude of the specified location to the number of all pixels arranged on the graph is determined as a percentile for specifying the threshold. Then, the threshold determination unit 23 determines the magnitude corresponding to the percentile for each of the multiple captured images, and determines the determined magnitude as a threshold for creating a histogram for each captured image.

[0028] By determining the magnitude threshold via percentile in this way, it is possible to determine the threshold based on percentile as a relative degree that can accommodate a wide variety of pixel movements that differ for each image in a group of images captured using a drone. However, instead of the method of determining the magnitude threshold via percentile, a method of directly determining the magnitude corresponding to the identified location as the magnitude threshold, a method of omitting step S103 by adopting a preset value as the threshold, a method of identifying pixels where the magnitude increases rapidly for each image and setting the threshold based on the pixels, etc. may be adopted.

[0029] The histogram creation unit 24 creates a histogram showing the distribution of the magnitudes of the multiple image elements, for image elements (in other words, image elements having magnitudes equal to or less than the threshold) excluding image elements having magnitudes exceeding the threshold determined by the threshold determination unit 23 among the multiple image elements. More specifically, the histogram creation unit 24 creates a histogram showing the distribution of the magnitudes of the multiple image elements by dividing image elements having magnitudes equal to or less than the threshold determined by the threshold determination unit 23 for each of the multiple captured images into magnitude intervals (bins) having a predetermined width and calculating the number of image elements for each bin. In this way, it is possible to create a histogram excluding unnecessary pixels such as noise, and it is possible to improve the accuracy of range estimation by the range estimation unit 25 described later.

[0030] The range estimation unit 25 refers to the histogram and estimates a magnitude range corresponding to an image element in which a predetermined portion of the imaging target that is within a predetermined distance range from the imaging position is captured. In this embodiment, the foreground structure (hereinafter referred to as "foreground") of the structure that is the imaging target is specified as the predetermined portion.

[0031] FIG. 5 is a diagram showing an example of a histogram showing the distribution of magnitude in a captured image, which is created in this embodiment. In this embodiment, the range estimation unit 25 identifies a bin in the histogram where the distribution is concentrated, and estimates the range of magnitude related to the foreground based on the identified bin. The estimated range of magnitude may be the same as or different from the interval of the identified bin. For example, the range estimation unit 25 can estimate a predetermined range around the identified bin as the range of magnitude corresponding to the image element of which a predetermined part is captured. The width of this range may be determined based on the accuracy of the captured image, the distance between the imaging device 81 and the structure at the time of capturing the image, and the like.

[0032] The image element identification unit 26 identifies image elements that belong to the estimated magnitude range, and thereby identifies an image element in which a specific portion (in this embodiment, the foreground, which is the structure that is the foremost among the structures) is captured among the multiple image elements that make up the captured image.

[0033] The image connection unit 27 connects the multiple captured images so that the image elements identified by the image element identification unit 26 included in each of the multiple captured images match, thereby obtaining a connected image (e.g., a panoramic image, etc.) in which a range wider than the range that can be captured simultaneously from one imaging position is captured. Specifically, for example, the image connection unit 27 obtains a connected image by matching feature points commonly detected in each of the multiple captured images captured consecutively and connecting each image. At this time, the image connection unit 27 can obtain a connected image in which a predetermined part (the foreground in this embodiment) of the captured images is connected so as to match by matching feature points related to the image elements (i.e., image elements related to a predetermined part such as the foreground) identified by the image element identification unit 26 and connecting each image. Note that the detection / extraction of feature points is a conventional technique, so a detailed description is omitted, but the feature points can be detected / extracted by using, for example, a standard module (feature2d) of OpenCV.

[0034] In addition, when connecting images, the image connection unit 27 may calculate a homography matrix (perspective transformation matrix) of a group of captured images based on a plurality of captured images (frames in a video), and convert the plurality of captured images onto one connected image using the calculated homography matrix to obtain a connected image. More specifically, in this embodiment, the image connection unit 27 calculates a homography matrix for the plurality of captured images by matching feature points related to image elements related to the foreground identified by the image element identification unit 26 between the plurality of captured images. Then, the image connection unit 27 uses the homography matrix to transform the plurality of captured images so that planes related to the foreground in the plurality of captured images match on one connected image, and arranges the transformed images on one connected image to obtain a connected image in which the foreground is arranged on one plane as a whole. The image linking unit 27 may calculate the homography matrix calculated in the group of captured images using a captured image showing the center of the structure to be captured as a reference, where the image showing the center of the structure may be determined according to the timing of capturing the captured image, for example, the (n / 2)th captured image in a group of n captured images. The image linking unit 27 may also calculate the homography matrix calculated in the group of captured images using a captured image showing the center of a unit of the identified structure as a reference.

[0035] The feature detection unit 28 detects features related to the external structure that are approximately common among a plurality of units from a structure image obtained by imaging a structure including a plurality of unit structures (hereinafter referred to as "units") each having the same or similar external structure with an imaging device. In this embodiment, a connected image obtained by the image connection unit 27 is used as the structure image. However, the structure image to be processed in the inspection process is not limited to the connected image obtained by the image connection unit 27, and may be a structure image obtained by other methods. In this embodiment, a structure in which bars such as reinforcing bars constituting the structure cross diagonally (X-shaped) is used as the external structure, and features obtained based on edges or points where line segments cross diagonally (intersections of straight lines or edges) detected by image recognition are used as the features related to the external structure.

[0036] In addition, in this embodiment, the feature detection unit 28 detects edges or lines in the structure image based on image elements in which a predetermined portion (in this embodiment, the foreground identified by the image element identification unit 26) identified by the image element identification unit 26 is captured in the structure image in which the inspection target side of the inspection target structure is captured, and detects features related to the external structure by referring to the detected edges or lines. More specifically, the feature detection unit 28 detects straight lines in the structure image, extracts straight lines included in the predetermined portion by filtering (masking) that extracts only image elements related to the predetermined portion identified by the image element identification unit 26, and detects intersections of diagonally extending members, thereby detecting features related to the external structure in which bars such as reinforcing bars that constitute a steel tower cross diagonally (X-shaped).

[0037] The unit structure estimation unit 29 estimates, for each of the multiple features detected by the feature detection unit 28, a predetermined range including the feature as a portion of the structure image relating to each of the multiple units included in the structure.

[0038] 6 and 7 are diagrams showing an overview of unit estimation in this embodiment. The target structure in this embodiment is a steel tower, which is a structure in which units, each having an external structure in which bars such as reinforcing bars cross diagonally (in an X-shape), are lined up in the vertical direction. An example of a method for estimating a unit structure for such a structure will be described with reference to FIGS. 6 and 7. First, the unit structure estimation unit 29 creates a graph (a graph showing the density of feature points in the vertical direction; see FIG. 6) that counts the number of feature points in the vertical direction based on a plurality of feature points detected by the feature detection unit 28, and identifies the central region of the steel tower (the region through which the center of the unit passes) by identifying the range of pixel rows with a high density of feature points. Next, the unit structure estimation unit 29 creates a graph (a graph showing the density of feature points in the horizontal direction; see Figure 7) by counting feature points horizontally based on a number of feature points in the central region of the tower among the multiple feature points identified by the feature detection unit 28, and identifies the center of each unit (the position where the bars that make up the unit intersect in an X shape) by identifying multiple ranges of pixel rows with a high density of feature points.

[0039] When the center of each unit is specified, the unit structure estimation unit 29 specifies the region of each unit including the specified center of the unit. When specifying the region of each unit, the end of the region can also be specified in the same manner as the center of the unit. That is, the unit structure estimation unit 29 refers to a graph (a graph showing the density of feature points in the vertical direction, see FIG. 6) that is created based on the multiple feature points detected by the feature detection unit 28, in which the number of feature points is counted in the vertical direction, and specifies the range of pixel columns with high density of feature points, excluding the range related to the center of the unit, as the left and right end regions of the unit. Then, the unit structure estimation unit 29 refers to a graph (a graph showing the density of feature points in the horizontal direction, see FIG. 7) that is created based on the multiple feature points in the left and right end regions of the unit among the multiple feature points specified by the feature detection unit 28, in which the feature points are counted in the horizontal direction, and specifies multiple ranges of pixel rows with high density of feature points, thereby specifying the end of each unit.

[0040] When the region (predetermined range) of each unit is specified, the unit structure estimation unit 29 assigns a label to each specified unit to uniquely identify the unit. There are no limitations on the method of labeling, and for example, labels consisting of numbers and / or letters may be assigned in ascending order from the top unit to the bottom unit.

[0041] The component detection unit 30 detects a plurality of predetermined components (e.g., fasteners for screw fastening, such as bolts, nuts, and washers) included in the unit from the structure image. In this embodiment, the component detection unit 30 detects a plurality of predetermined components included in the unit using a machine learning model created in advance by subjecting a predetermined component captured in the structure image to deep learning or the like. However, a specific method by which the component detection unit 30 detects a predetermined component from the structure image is not limited. For example, the component detection unit 30 can detect a plurality of predetermined components included in the unit by a method of matching an image of a predetermined component prepared in advance with the structure image, a method of detecting a pattern specific to a predetermined component prepared in advance by feature detection, or the like.

[0042] The cluster creation unit 31 creates multiple clusters of components that are constructed closely together by clustering the detected components based on the distance between the components. In this embodiment, the cluster creation unit 31 performs clustering of the components using density-based clustering such as DBScan. However, the specific method for clustering the components is not limited, and other clustering methods may be adopted.

[0043] Fig. 8 is a diagram showing how components detected from a structure image are clustered in this embodiment. Predetermined components such as fastening materials are detected from the structure image shown on the left side of Fig. 8, and the detected components are clustered based on their positions, as shown on the right side of Fig. 8. Here, the dashed circles indicate the respective clusters.

[0044] The cluster correspondence unit 32 compares the position of each of the multiple clusters with the position of each of the multiple clusters of components shown in the design data of the structure prepared in advance, thereby corresponding the clusters in the structure image to the clusters in the design data. In this embodiment, the center of the cluster (more specifically, for example, the center of gravity or the center of a circumscribing rectangle) is used as the position of the cluster. For this reason, the cluster correspondence unit 32 calculates the center of the cluster created by the cluster creation unit 31, calculates the center of the cluster of the components shown in the design data, compares the center positions, and corresponds the clusters that match or are similar. At this time, the cluster of the components shown in the design data may also be created by the cluster creation unit 31 described above.

[0045] Fig. 9 is a diagram showing how components detected from design data are clustered in this embodiment. From the design data shown on the left side of Fig. 9, specific components such as fastening materials are detected, and the right side of Fig. 9 shows how the detected components are clustered based on their positions. Here, dashed circles indicate each cluster. Note that in this embodiment, an example is shown and explained in which a design image is used as design data for a structure, but the design data for a structure is not limited to an image. Any data that allows the relative positional relationship of fastening materials to be grasped can be used as design data for a structure.

[0046] The component matching unit 33 matches the components in the structure image with the components shown in the design data by comparing the positions of the components belonging to the cluster in the structure image with the positions of the components belonging to the cluster in the design data matched to the cluster by the cluster matching unit 32. A specific method of matching may be adopted similar to the method of matching clusters by the cluster matching unit 32. The component matching unit 33 compares the positions of the components belonging to the clusters between the clusters in the matched structure image and the clusters in the design data, and matches matching or similar components.

[0047] The alignment unit 34 aligns the structure image with the design data based on the result of the association between the components in the structure image and the components shown in the design data by the component association unit 33. Specifically, the alignment unit 34 aligns the structure image with the design data by transforming the structure image so that the positions of the components in the structure image associated by the component association unit 33 match (or are as close as possible to) the positions of the components in the design data.

[0048] The construction state determination unit 35 determines the construction state of the components based on the images of the components such as fastening parts included in the captured image. Here, the specific determination method can be appropriately adopted depending on the embodiment, and is not limited, so detailed description will be omitted. For example, the construction state determination unit 35 compares the image of each unit divided in the structure image with the design data (design image) of the unit to detect differences in the total number of components such as fastening parts.

[0049] <Processing flow> Next, a flow of processing executed by the information processing device according to the present embodiment will be described. Note that the specific contents and processing order of the processing described below are an example for implementing the present disclosure. The specific contents and processing order may be appropriately selected according to the embodiment of the present disclosure.

[0050] 10 is a flowchart showing the flow of the panoramic image generating process according to the present embodiment. The process shown in this flowchart is executed when a captured image is input or an instruction to execute the panoramic image generating process is input by an operator during monitoring or inspection during or after the construction of a structure.

[0051] In step S101, an image is acquired. The worker uses the imaging device 81 to capture an image of the side of the structure to be inspected, and inputs image data of the captured image to the information processing device 1. Although the imaging method and the method of inputting the image data to the information processing device 1 are not limited, in this embodiment, the side of the structure to be inspected is captured using a drone 8 equipped with the imaging device 81, and the image data transferred from the imaging device 81 to the user terminal 9 via communication or a recording medium is further transferred to the information processing device 1 via a network, whereby the image data of the captured image is input to the information processing device 1, and the captured image acquisition unit 21 acquires this. After that, the process proceeds to step S102.

[0052] In step S102, the optical flow is estimated. The motion acquisition unit 22 acquires the optical flow for each of the pixels constituting the captured image by comparing images that are adjacent in time series based on the captured images (frames that constitute the video) that are acquired in step S101. As described above, the optical flow acquired here includes magnitude, which is a parameter that indicates the magnitude of the optical flow for each pixel. Then, the process proceeds to step S103.

[0053] In steps S103 and S104, a histogram of the optical flow, which is a partially flattened histogram, is created. The threshold determination unit 23 identifies a location where the change in magnitude exceeds a predetermined standard when the pixels included in the series of captured images are arranged in ascending / descending order of magnitude, and determines the magnitude threshold for each of the captured images based on the percentile corresponding to the location (step S103). Then, the histogram creation unit 24 creates a histogram showing the distribution of the magnitude of multiple pixels for each of the captured images, excluding pixels associated with a magnitude exceeding the threshold determined in step S103 (in other words, pixels associated with a magnitude equal to or less than the threshold) (step S104). After that, the process proceeds to step S105.

[0054] In steps S105 and S106, the foreground in the captured image group is divided based on the histogram of the optical flow. The range estimation unit 25 refers to the histogram created in step S104 and identifies a magnitude section (bin) in which the distribution is concentrated, thereby estimating a magnitude range corresponding to pixels in which a predetermined part (in this embodiment, the foreground) of the imaging target within a predetermined distance range from the imaging position is captured (step S105). Then, the image element identification unit 26 identifies pixels that belong to the estimated magnitude range, thereby identifying pixels in which the foreground is captured among the multiple pixels constituting the captured image (step S106). Thereafter, the process proceeds to step S107.

[0055] In step S107, a panoramic image of the structure is generated based on the captured image group. Here, the panoramic image is generated such that the foreground group divided in step S106 in each captured image group is joined together with consistency. The image connection unit 27 generates a connected still image (panoramic image) by connecting the multiple captured images such that the pixels related to the foreground identified in step S106 are consistent. Note that, as described above in the description of the image connection unit 27, the connected image may be obtained by perspective transformation of the captured images using a homography matrix. Thereafter, the processing shown in this flowchart ends.

[0056] Fig. 11 is a flowchart showing the flow of the inspection process according to this embodiment. The process shown in this flowchart is executed when a structure image is input or an instruction to execute the inspection process is input by an operator during monitoring or inspection during or after the construction of a structure. Note that, although the case where the combined image generated by the above-mentioned panoramic image generation process is input as the structure image will be mainly described here, the structure image may be acquired by other methods.

[0057] In step S201, features related to the external structure are detected from the structure image based on pixels captured in a predetermined portion (foreground in this embodiment) of the structure image in which the side surface of the structure to be inspected (a steel tower in this embodiment) is captured. The feature detection unit 28 extracts straight lines included in the foreground (in other words, straight lines related to the front bar) by filtering (masking) that extracts only pixels related to the foreground from among the straight lines in the structure image, and detects the intersection points of diagonally extending members to detect features related to the external structure in which bars such as reinforcing bars that make up the steel tower cross diagonally (X-shaped). After that, the process proceeds to step S202.

[0058] In step S202, the multiple units constituting the structure are divided. The unit structure estimation unit 29 estimates a predetermined range including each of the multiple features detected in step S201 as a portion of the structure image that corresponds to each of the multiple units included in the structure, and assigns a label to each estimated unit to identify the unit. Then, the process proceeds to step S204.

[0059] In steps S204 to S208, the fastening material is detected from the structure image, and the fastening material detected from the image is associated with and aligned to the fastening material position in the design data. The component detection unit 30 detects a plurality of predetermined components (fastening materials in this embodiment) included in the unit from the structure image (step S204). The cluster creation unit 31 then clusters the detected components based on the distance between the components to create a plurality of clusters of components that are installed closely together (step S205). The cluster association unit 32 compares the positions of the plurality of clusters created in step S205 with the positions of the plurality of clusters of components shown in the design data of the structure prepared in advance, thereby associating the clusters in the structure image with the clusters in the design data (step S206).

[0060] The component matching unit 33 matches the components in the structure image with the components shown in the design data by comparing, for each associated cluster, the positions of the components belonging to that cluster with the positions of the components belonging to the cluster in the design data associated with that cluster in step S206 (step S207). Then, the alignment unit 34 aligns the structure image with the design data based on the result of the association between the components in the structure image and the components shown in the design data in step S207. Then, the process proceeds to step S209.

[0061] In step S209, the construction state is determined. The construction state determination unit 35 determines the construction state, such as the total number of members (fastening parts) included in each unit, and detects any abnormality. The determination result may be notified to the user terminal 9. After that, the process shown in this flowchart ends.

[0062] <Variations> In the embodiment described above, an example has been described in which the input structure image (when a panoramic image generated by the panoramic image generation process is used, the panoramic image) is used as is in the processes from step S204 onwards, but a concatenated image created for each unit may be used for the structure image. In this case, a concatenated image for each unit is generated based on the input captured images. The image connection unit 27 generates a concatenated image for each identified unit by connecting multiple captured images for each unit so that pixels related to the foreground are consistent. Note that at this time, the concatenated image may be obtained by perspective transformation of the captured images using a homography matrix, similar to the generation of a panoramic image for the entire structure described in step S107 above. [Explanation of symbols]

[0063] 1. Information processing device

Claims

1. a motion acquisition means for acquiring a magnitude of motion of the imaging target within the image for each of a plurality of image elements constituting the captured image based on a plurality of captured images obtained by capturing images while changing a positional relationship between the imaging target and the imaging position; a histogram generating means for generating a histogram indicating a distribution of the magnitude of motion of the plurality of image elements; a range estimation means for estimating a range of a magnitude of a movement corresponding to an image element in which a predetermined portion of the imaging target within a predetermined distance range from the imaging position is imaged, by referring to the histogram; an image element specifying means for specifying an image element in which the predetermined portion is captured among the plurality of image elements constituting the captured image by specifying the image element that belongs to a range of the magnitude of the estimated motion; an image linking means for linking the plurality of captured images such that image elements identified by the image element identifying means included in each of the plurality of captured images are consistent with each other, thereby obtaining a structure image in which a wider range is captured than a range that can be simultaneously captured from one imaging position; a feature detection means for detecting features relating to an external structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar external structure with an imaging device; a unit structure estimation means for estimating portions of the structure image relating to each of the unit structures included in the structure based on the detected features; An information processing device comprising:

2. a feature detection means for detecting features relating to an external structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar external structure with an imaging device; a unit structure estimation means for estimating portions of the structure image relating to each of the unit structures included in the structure based on the detected features; a member detection means for detecting a plurality of predetermined members included in the unit structure from the structure image; A cluster creating means for creating a plurality of clusters of components that are constructed closely together by clustering the detected plurality of components based on the distance between the components; a cluster matching means for matching clusters in the structure image with clusters in the design data by comparing positions of the plurality of clusters with positions of a plurality of clusters of components shown in design data prepared in advance for the structure; a member matching means for matching members in the structure image with members shown in the design data by comparing members belonging to the cluster with members belonging to a cluster of members shown in the design data that has been matched to the cluster by the cluster matching means; An information processing device comprising:

3. the feature detection means detects edges or line segments in the structure image based on image elements in which the predetermined portion identified by the image element identification means is captured, and detects the features by referring to the detected edges or line segments. The information processing device according to claim 1 .

4. the external structure is a diagonally crossing bar included in the structure, the feature detection means detects the features based on points where the detected edges or line segments diagonally intersect; The information processing device according to claim 3 .

5. the unit structure estimation means estimates, for each of the detected features, a predetermined range including the feature as a portion in the structure image relating to each of the unit structures included in the structure; The information processing device according to claim 4.

6. a threshold value determining unit for determining a threshold value for excluding image elements not to be targeted by the histogram based on a magnitude of motion of the image elements; the histogram creation means creates the histogram for image elements excluding image elements related to a magnitude of motion exceeding the threshold value among the plurality of image elements. The information processing device according to claim 1 , 2 or 3 .

7. and a positioning unit that performs positioning between the structure image and the design data based on a result of the member matching between the structure image and the member shown in the design data by the member matching unit. The information processing device according to claim 2 .

8. The computer a motion acquisition step of acquiring a magnitude of motion of the imaging target within the image for each of a plurality of image elements constituting the captured image based on a plurality of captured images obtained by capturing images while changing a positional relationship between the imaging target and the imaging position; a histogram creation step of creating a histogram indicating a distribution of the magnitude of motion of the plurality of image elements; a range estimating step of estimating a range of a magnitude of a movement corresponding to an image element in which a predetermined portion of the imaging target that is within a predetermined distance range from the imaging position is imaged, with reference to the histogram; an image element identifying step of identifying an image element in which the predetermined portion is captured among the plurality of image elements constituting the captured image by identifying the image element that belongs to a range of the magnitude of the estimated motion; an image linking step of linking the plurality of captured images so that image elements identified in the image element identifying step included in each of the plurality of captured images are consistent with each other, thereby obtaining a structure image in which a wider range is captured than a range that can be simultaneously captured from one imaging position; a feature detection step of detecting features related to an appearance structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar appearance structure with an imaging device; a unit structure estimating step of estimating portions in the structure image relating to each of the plurality of unit structures included in the structure based on the plurality of detected features; How to do it.

9. The computer a feature detection step of detecting features related to an appearance structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar appearance structure with an imaging device; a unit structure estimating step of estimating portions in the structure image relating to each of the plurality of unit structures included in the structure based on the plurality of detected features; a member detection step of detecting a plurality of predetermined members included in the unit structure from the structure image; A cluster creation step of creating a plurality of clusters of components that are constructed closely together by clustering the detected plurality of components based on the distance between the components; a cluster matching step of matching clusters in the structure image with clusters in the design data by comparing positions of the plurality of clusters with positions of a plurality of clusters of components shown in design data of the structure that has been prepared in advance; a member matching step of matching the members in the structure image with the members shown in the design data by comparing the members belonging to the cluster with the members belonging to the cluster of the members shown in the design data that have been matched to the cluster in the cluster matching step; How to do it.

10. Computer, a motion acquisition means for acquiring a magnitude of motion of the imaging target within the image for each of a plurality of image elements constituting the captured image based on a plurality of captured images obtained by capturing images while changing a positional relationship between the imaging target and the imaging position; a histogram generating means for generating a histogram indicating a distribution of the magnitude of motion of the plurality of image elements; a range estimation means for estimating a range of a magnitude of a movement corresponding to an image element in which a predetermined portion of the imaging target within a predetermined distance range from the imaging position is imaged, by referring to the histogram; an image element specifying means for specifying an image element in which the predetermined portion is captured among the plurality of image elements constituting the captured image by specifying the image element that belongs to a range of the magnitude of the estimated motion; an image linking means for linking the plurality of captured images such that image elements identified by the image element identifying means included in each of the plurality of captured images are consistent with each other, thereby obtaining a structure image in which a wider range is captured than a range that can be simultaneously captured from one imaging position; a feature detection means for detecting features relating to an external structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar external structure with an imaging device; a unit structure estimation means for estimating portions of the structure image relating to each of the unit structures included in the structure based on the detected features; A program that functions as a

11. Computer, a feature detection means for detecting features relating to an external structure from a structure image obtained by capturing an image of a structure including a plurality of unit structures each having the same or similar external structure with an imaging device; a unit structure estimation means for estimating portions of the structure image relating to each of the unit structures included in the structure based on the detected features; a member detection means for detecting a plurality of predetermined members included in the unit structure from the structure image; A cluster creating means for creating a plurality of clusters of components that are constructed closely together by clustering the detected plurality of components based on the distance between the components; a cluster matching means for matching clusters in the structure image with clusters in the design data by comparing positions of the plurality of clusters with positions of a plurality of clusters of components shown in design data of the structure that has been prepared in advance; a member matching means for matching members in the structure image with members shown in the design data by comparing members belonging to the cluster with members belonging to a cluster of members shown in the design data that has been matched to the cluster by the cluster matching means; A program that functions as a

Images