Information processing systems, information processing methods, and programs

Abstract

To provide an information processing system, information processing method, and program related to the inspection of structures. [Solution] An information processing system comprising: an imaging data acquisition unit that acquires an image of at least a part of a structure; an inversion processing unit that inverts the image around a straight line along the image plane to generate an inverted image of the image; and an anomaly detection unit that detects anomalies in the target part shown in the image based on a superimposed image obtained by superimposing the original image before inversion and the inverted image.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program related to inspection of structures.

Background Art

[0002] As shown in Patent Document 1, a method for determining the damage state of a structure from an image of the structure is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above Patent Document 1, the damage state of a structure is evaluated by calculating the difference between a past captured image and a most recent captured image. In such an evaluation method, it is necessary to align the past captured image and the most recent captured image, but it is difficult to acquire a captured image at exactly the same position as at the time of past inspection, and due to environmental changes during imaging (weather, time zone, season, light conditions, and other changes in objects within the environment), image parameters (color tone, contrast, saturation, brightness, etc.) also change, making it difficult to compare with past captured images and there may be cases where sufficient evaluation accuracy cannot be obtained. In particular, for minute changes such as slight distortion (deformation) of members constituting the structure, they cannot be detected by an evaluation method based on comparison with past captured images as in Patent Document 1.

[0005] An object of an exemplary embodiment of the present disclosure is to provide an information processing system, an information processing method, and a program related to inspection of structures.

Means for Solving the Problems

[0006] The information processing system relating to one aspect of this disclosure is: An imaging data acquisition unit that acquires images of at least a part of the structure, An inversion processing unit that inverts the captured image around a straight line along the image plane to generate an inverted image of the captured image, The system includes an anomaly detection unit that detects an anomaly in the target area depicted in the captured image based on a superimposed image obtained by overlaying the original captured image before inversion and the inverted image.

[0007] The information processing system, having the above-described features, can determine whether or not there is an abnormality in the target part contained within the structure. Further details regarding the problems disclosed in this application and their solutions will be clarified in the section on embodiments and drawings of this disclosure. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a diagram illustrating the configuration of an information processing system according to one embodiment of the present disclosure. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of the management server shown in Figure 1. [Figure 3] Figure 3 is a block diagram showing the hardware configuration of the mobile device shown in Figure 1. [Figure 4] Figure 4 is a block diagram illustrating the software configuration of the management server shown in Figure 1. [Figure 5] Figure 5 is a conceptual diagram showing an example of the operation of the information processing system shown in Figure 1. [Figure 6] Figure 6 is a flowchart showing an example of information processing related to anomaly detection. [Figure 7] Figure 7 is a diagram that simulates an example of a captured image. [Figure 8] Figure 8 is a diagram illustrating the process of identifying the target area. [Figure 9] Figure 9 is a diagram illustrating the process of generating inverted images. [Figure 10] Figure 10 is a diagram illustrating the process of generating superimposed images. [Modes for carrying out the invention]

[0009] The information processing method, information processing system, and program disclosed herein have, for example, the following configuration. [Item 1] An imaging data acquisition unit that acquires images of at least a part of the structure, An inversion processing unit that inverts the captured image around a straight line along the image plane to generate an inverted image of the captured image, An information processing system comprising: an anomaly detection unit that detects an anomaly in a target area depicted in a captured image based on a superimposed image obtained by overlaying the original captured image before inversion and the inverted image. [Item 2] The information processing system according to item 1, further comprising a target area identification unit that identifies the target area that is the target for determining an abnormality from the captured image by segmentation. [Item 3] The information processing system according to item 1 or 2, wherein the anomaly detection unit translates and / or rotates the inverted image on the image plane to calculate the maximum overlapping area, which is the largest area in which the target part in the captured image and the target part in the inverted image overlap. [Item 4] The information processing system according to item 3, wherein the abnormality detection unit determines whether or not there is an abnormality in the target area based on the ratio of the maximum superimposed area to the area of ​​the target area in the captured image. [Item 5] The information processing system according to item 1 or 2, wherein the target part included in the structure is a member having mirror image symmetry, or a combination of members. [Item 6] To obtain images of at least a part of the structure, The captured image is inverted around a straight line along the image plane to generate an inverted image of the captured image, An information processing method executed by a computer, comprising: detecting an abnormality of a target part imaged in the captured image based on a superimposed image obtained by superimposing the original captured image before inversion and the inverted image. [Item 7] Obtaining a captured image that captures at least a part of a structure Inverting the captured image about a straight line along the image plane to generate an inverted image of the captured image A program for causing a computer to perform: detecting an abnormality of a target part imaged in the captured image based on a superimposed image obtained by superimposing the original captured image before inversion and the inverted image.

[0010] <Details of the Embodiment> An information processing system according to an embodiment of the present disclosure will be described with reference to the drawings. In each of the attached drawings, the same or similar elements are given the same or similar reference numerals and names, and redundant descriptions regarding the same or similar elements may be omitted in the description of the embodiment. Note that the content shown in each drawing is merely an example for explaining the present embodiment, and is merely a schematic example shown for ease of explaining the present embodiment. The content of each drawing may be modified or changed within a range where no technical problems occur.

[0011] <System Overview> The information processing system according to this embodiment is a system for inspecting structures that uses images of the structure to detect abnormalities in predetermined parts of the structure. Examples of structures to be inspected include, but are not limited to, power transmission equipment such as steel towers, radio towers, communication towers, bridges, plant-related equipment (e.g., plant support structures, towers and cooling towers of oil refineries), various buildings (e.g., steel frame structures, dome frameworks, observation decks), and other industrial equipment (cranes, lighthouses, wind power generation equipment, etc.). Furthermore, the method of photographing the structure is not particularly limited. For example, the structure may be photographed using a camera mounted on an unmanned mobile vehicle such as a drone, unmanned aerial vehicle (UAV), or unmanned ground vehicle (UGV), or on a manned mobile vehicle. Alternatively, a person such as a worker may operate a camera (e.g., a smartphone, tablet, digital camera, etc.) to photograph the target object.

[0012] The "designated part (hereinafter referred to as the target part)" for determining whether or not there is an abnormality may be, for example, a component of a structure (hereinafter sometimes referred to as a structural member), or a part consisting of a combination of members (hereinafter sometimes referred to as a unit), and in particular, a member or unit having mirror image symmetry. Here, "mirror image symmetry" means the property that the geometric shape of a part or unit in a normal state (specifically, the geometric shape that is captured when imaged from a predetermined direction) is symmetrical with respect to a predetermined straight line. "Members or units having mirror image symmetry" are not limited to those in which the overall shape of the part or unit has mirror image symmetry, but also include those that have partial mirror image symmetry. Furthermore, even if the geometric shape when imaged from a predetermined direction does not have mirror image symmetry, if at least a part of the geometric shape when imaged from another direction has mirror image symmetry, it shall be included as a "member or unit having mirror image symmetry".

[0013] The captured image only needs to show at least a part of the structural member of the target area, or at least a part of a unit made up of members; the entire target area does not need to be imaged. Furthermore, the abnormalities detected by the information processing system are not necessarily limited and include, for example, deformation abnormalities such as bending, distortion, twisting, deflection, denting, expansion, and partial bulging; abnormalities related to geometric shape such as defects, wear, peeling of the surface layer, loosening or detachment of members, and misalignment of joints. The information processing system of this embodiment is particularly suitable for detecting minute deformation abnormalities such as extremely small bends.

[0014] In this embodiment, as shown in Figure 5, the details of the information processing system will be explained using an example where power transmission equipment such as transmission towers is inspected by photographing them with a camera 42 mounted on an unmanned mobile body 4. The drive of the mobile body 4 may be autonomously controlled based on a pre-set travel path, or it may be controlled by remote operation based on instructions from a user terminal 2 owned by the user.

[0015] Regarding transmission towers, if abnormalities such as bending occur in the structural members that make up the tower (for example, the steel members that make up the steel frame structure), there is a risk that the tower will eventually collapse. Therefore, it is necessary to inspect transmission towers regularly and detect abnormalities in the structural members at an early stage. For example, in the inspection of deformation of structural members in transmission towers, a resolution that can detect deformation on the order of millimeters for a straight structural member with a length of 5m is required.

[0016] One possible method for inspecting deformation abnormalities in structural members is to extract edges from images of the structural members and verify the linearity of the extracted edges. However, it is expected that various objects other than the structural member being inspected will be captured in the images of the structure. In edge extraction processing, numerous edges originating from the contours of other objects will be detected, making it difficult to selectively extract only the edges corresponding to the contour of the structural member being inspected. Even if it were possible to selectively extract the edges of the structural member, it would be difficult to detect minute deformation abnormalities due to the influence of discontinuities (breaks) and noise (e.g., unwanted edges such as textures) in the extracted edges.

[0017] Furthermore, as disclosed in Patent Document 1, it is conceivable to inspect structural members for abnormalities by comparing past images with most recent images. However, it is difficult to acquire images at the exact same location as during past inspections, and image parameters (color tone, contrast, saturation, brightness, etc.) change due to environmental changes during shooting (weather, time of day, season, lighting conditions, and changes in objects in the environment). Therefore, comparison with past images is difficult, and even with this method, it is difficult to detect minute deformation abnormalities.

[0018] In this embodiment of the information processing system, an image of at least a portion of a structure is inverted around a straight line along the image plane to generate an inverted image of the captured image. Based on a superimposed image created by overlapping the original image before inversion with the inverted image, abnormalities in the target area shown in the captured image are detected. This method allows for the efficient detection of abnormalities in target areas such as structural members and units made up of members, without referring to past captured images. In particular, it becomes possible to detect deformation abnormalities on the order of millimeters for structural members with lengths on the order of meters. The information processing system of this embodiment will be described in detail below, with an example of tower inspection using a UAV (mobile vehicle 4).

[0019] <System Configuration> As shown in Figure 1, the information processing system of this embodiment may include a management server 1, one or more user terminals 2, and one or more mobile devices 4. The management server 1, user terminals 2, and mobile devices 4 are connected to each other via a network NW so that they can communicate with one another. Note that the illustrated configuration is just an example and is not limited thereto. For example, the mobile device 4 does not have to be connected to the network NW. In that case, the mobile device 4 may be operated by a transmitter (so-called remote control) operated by the user. Alternatively, image data acquired by the camera of the mobile device 4 may be stored in an auxiliary storage device connected to the mobile device 4 (for example, a memory card such as an SD card and / or a USB memory), and subsequently read from the auxiliary storage device to the user terminal 2 and / or management server 1 and stored by the user. The mobile device 4 may be connected to the network NW only for the purpose of operation or for the purpose of storing image data, or for only one of these purposes.

[0020] <Management Server 1> Figure 2 shows the hardware configuration of management server 1. Note that the configuration shown is just one example, and management server 1 may have a different configuration.

[0021] The management server 1 comprises at least a processor 10, memory 11, storage 12, a transceiver 13, an input / output unit 14, etc., which are electrically connected to each other via a bus 15. The management server 1 may be a general-purpose computer such as a workstation or personal computer, or it may be logically implemented by cloud computing.

[0022] The processor 10 is a computing unit that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and loaded into the memory 11 to perform various information processing tasks.

[0023] Memory 11 includes main memory composed of volatile storage devices such as DRAM (Dynamic Random Access Memory) and auxiliary memory composed of non-volatile storage devices such as flash memory and HDD (Hard Disk Drive). Memory 11 is used as a work area for the processor 10 and also stores the BIOS (Basic Input / Output System) executed when the management server 1 starts up, as well as various configuration information.

[0024] Storage 12 stores various programs, such as application programs. A database containing data used for each process may be built in storage 12. For example, the memory unit 120 described later may be provided in a part of the storage area of ​​storage 12.

[0025] The transmitting / receiving unit 13 is a communication interface for the management server 1 to communicate with user terminals 2 and mobile devices 4, etc., via a communication network. The transmitting / receiving unit 13 may further include short-range communication interfaces such as Bluetooth® and BLE (Bluetooth Low Energy) and / or USB (Universal Serial Bus) terminals, etc.

[0026] The input / output section 14 consists of information input devices such as keyboards and mice, and output devices such as displays.

[0027] Bus 15 is connected in common to all of the above elements and transmits, for example, address signals, data signals, and various control signals.

[0028] <User Terminal 2> The user terminal 2 also includes a processor 20, memory 21, storage 22, a transceiver 23, an input / output unit 24, etc., which are electrically connected to each other via a bus 25. The user terminal 2 may be a general-purpose computer such as a workstation or a personal computer. The functions of each element of the user terminal 2 can be configured in the same way as the management server 1 described above, and a detailed explanation of each element of the user terminal 2 is omitted.

[0029] <Mobile Unit 4> Figure 3 is a block diagram showing the hardware configuration of the mobile unit 4. The flight controller 41 may have one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)).

[0030] The flight controller 41 may also have a memory 411, which is accessible. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412 such as inertial sensors (accelerometers, gyroscopes), GPS sensors, and proximity sensors (e.g., LiDAR).

[0031] The memory 411 may include, for example, a separable medium such as an SD card and random access memory (RAM) or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still images and video data captured by the camera may be recorded in the internal memory or external memory, but is not limited to this; the data may also be recorded from the camera / sensors 42 or internal memory via the network NW to at least one of the management server 1 and the user terminal 2. The camera 42 is mounted on the mobile body 4 via a gimbal 43.

[0032] The flight controller 41 includes a control module (not shown) configured to control the state of the mobile body 4. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ). x , θ y and θ z To adjust the spatial arrangement, speed, and / or acceleration of the mobile body 4, which has a motor 4, the propulsion mechanism (motor 45, etc.) of the mobile body 4 is controlled via the ESC 44 (Electric Speed ​​Controller). The motor 45, powered by the battery 48, rotates the propeller 46, generating lift for the mobile body 4. The control module of the flight controller 41 may have functions to control the drive of the camera 42 and shooting conditions (e.g., sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). The control module can also control one or more of the mounted parts and the state of the sensors.

[0033] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transmitter / receiver (RC) 49, a terminal, a display device, or other remote controller). The transceiver 49 may use any suitable means of communication, such as wired or wireless communication.

[0034] For example, the transmitting / receiving unit 47 may utilize one or more of the following: local area network (LAN), wide area network (WAN), infrared, wireless, Wi-Fi, point-to-point (P2P) network, telecommunications network, cloud communication, etc.

[0035] The transmitting / receiving unit 47 can transmit and / or receive one or more of the following: data acquired by cameras / sensors 42, processing results generated by the flight controller 41, predetermined control data, and user commands from a terminal or remote controller.

[0036] The camera / sensor array 42 may include inertial sensors (accelerometers, gyroscopes), GPS sensors, proximity sensors (e.g., LiDAR), or vision / image sensors (e.g., cameras).

[0037] <Functions of Management Server 1> Figure 4 is a block diagram illustrating the functions implemented in the management server 1. In this embodiment, the management server 1 may include an imaging data acquisition unit 101, a target area identification unit 102, an inversion processing unit 103, and an anomaly detection unit 104. The storage unit 120 of the management server 1 may also include various databases such as an information / image storage unit 121. Although the various functional units shown in Figure 4 are illustrated as functional units realized by the processor 10 of the management server 1, some or all of the various functional units may be realized in the processor 20 of the user terminal 2 or the flight controller 41 of the mobile unit 4, depending on the capabilities of the processor 20 and / or the flight controller 41.

[0038] The imaging data acquisition unit 101 acquires images of at least a part of the structure. For example, the imaging data acquisition unit 101 acquires images captured by the camera 42 mounted on the mobile body 4 via wireless communication through a communication interface. The captured images show at least a part of the structural members and / or units which are combinations of members included in the structure, and do not necessarily have to show the entire structural member or unit. Image 50 shown in Figure 7 is an example of a simulated captured image, in which a part of the steel frame structure captured by the camera 42 is shown as the target part 60.

[0039] The image data acquisition unit 101 may acquire captured images in real time via wireless communication from the mobile body 4 or within the mobile body 4. The captured images acquired by the image data acquisition unit 101 may be moving images or still images. If the captured images are moving images, the moving images may be divided into still images for each frame, and these still images may be used in the various functional units described later. Alternatively, still images may be extracted at predetermined intervals from the still images divided into frames, and these extracted still images may be used in the various functional units. The captured images taken by the mobile body 4, etc., may be color images, grayscale images, or black and white images.

[0040] The target area identification unit 102 receives the captured image acquired by the imaging data acquisition unit 101 as input information and performs a process to identify the target area that is the target for abnormality determination from the captured image by segmentation. Segmentation is a method of dividing the captured image into regions according to the type of object depicted in the image by determining which segment (for example, a category such as power lines, transmission towers, or background) each pixel constituting the captured image belongs to and labeling each pixel. The target area identification unit 102 may be constructed using an AI model that has been subjected to machine learning (e.g., deep learning) capable of extracting a predetermined target area (such as a component or unit constituting a structure) from the captured image. The segmentation architecture performed by the target area identification unit 102 is not particularly limited, and for example, LR-ASPP (Lite Reduced Atrous Spatial Pyramid Pooling) may be used.

[0041] The target area identification unit 102 may identify a region determined by segmentation as a segment representing a structure to be inspected as the target area. Alternatively, the target area identification unit 102 may estimate a region with mirror symmetry from among the regions determined by segmentation as segments representing a structure to be inspected, and identify the estimated region as the target area. In this case, high accuracy is not required for the estimation of mirror symmetry, and it may be estimated that a region has mirror symmetry if it satisfies some simply defined predetermined conditions (for example, that it is a linear object, that the smallest circumscribed figure is a line-symmetric figure such as a rectangle or a circle, etc.).

[0042] The target area identification unit 102 may generate a mask image that visualizes and shows the segmented areas in the captured image based on the segmentation results. For example, as shown in Figure 8, the target area identification unit 102 may generate a mask image 50a that makes the area corresponding to the structure to be inspected (target area 60) visible in the captured image and covers up and makes latent the areas other than the structure to be inspected (the area indicated by reference numeral 65 in Figure 8). The target area identification unit 102 may also generate a mask image that identifies the target area 60 in a manner that allows identification within the captured image, or it may generate a primary processed image in which the area corresponding to the target area 60 is extracted from the captured image.

[0043] Furthermore, if the target area is clear within the captured image, segmentation processing is not necessarily required, and the target area within the captured image may be identified using other known methods instead of segmentation processing. The process of identifying the target area by the target area identification unit 102 may be performed before the process of generating the inverted image by the inversion processing unit 103 described later, or the process of identifying the target area from within the image may be performed after the inversion process for both the original captured image and the inverted image. Alternatively, the process of identifying the target area from within the superimposed image generated during the information processing performed by the anomaly detection unit 104 may be performed.

[0044] The inversion processing unit 103 inverts the captured image around a predetermined straight line along the image plane to generate an inverted image of the captured image. The captured image that serves as the source for the inverted image may be an unprocessed captured image acquired by the imaging data acquisition unit 101, or it may be a captured image that has undergone primary processing, such as a mask image reflecting the segmentation results.

[0045] Figure 9 is a diagram illustrating the process of generating an inverted image. The axis used to invert the captured image is not particularly limited. For example, the center line of the image perpendicular to the width direction of the captured image (the dashed line L1 or L2 shown in Figure 9(a)) may be set as the inversion axis, or the diagonal line of the captured image (the dashed line L3 or L4 shown in Figure 9(a)) may be set as the inversion axis. Alternatively, the inversion processing unit 103 may identify a bisector (line of symmetry) that divides the region identified as the target area into two equal parts, as shown by the double-dotted line L5 in Figure 9(a), and set this bisector as the inversion axis. In this case, it is not necessary to accurately identify the bisector, and errors may occur. From the viewpoint of reducing the computational load during the inversion process, it is not necessary to identify the bisector, and any one reference line that is uniformly set for the captured image regardless of the subject, such as the center line of the image (L1 or L2), may be used as the inversion axis.

[0046] The inversion processing unit 103 generates an inverted image by rotating the captured image by 180 degrees around a predetermined straight line along the image plane as described above. Figure 9(b) illustrates an inverted image 52 obtained by inverting the captured image 50 shown in Figure 9(a) around the center line L1.

[0047] The anomaly detection unit 104 detects anomalies in the target area captured in the captured image based on a superimposed image obtained by overlapping the original captured image before inversion with the inverted image. For example, the anomaly detection unit 104 may determine whether or not an anomaly related to the geometric shape exists in the target area based on the degree of overlap between the target area in the captured image and the target area in the inverted image, as shown in the superimposed image.

[0048] The anomaly detection unit 104 may translate and / or rotate the inverted image on the plane of the superimposed image to determine the degree of overlap, and calculate the maximum overlap area, which is the maximum area in which the target part in the captured image and the target part in the inverted image overlap. The superimposed image 54 illustrated in Figure 10(a) is an image generated by superimposing the captured image 50 in Figure 9(a) and the inverted image 52 in Figure 9(b) so that their outer edges coincide, with the captured image 50 shown as a solid line and the inverted image 52 as a dashed line. If the captured image 50 is inverted around a reference line (such as a center line) that is different from the bisector (line of symmetry) of the target part, the images may be superimposed with the positions of the target part in the captured image and the target part in the inverted image shifted, as shown in Figure 9(a). In such cases, the maximum overlap area can be calculated by calculating the overlap area between the target area in the captured image and the target area in the inverted image, and then translating and / or rotating the inverted image on the plane of the superimposed image (the captured image may also be moved). Figure 10(b) illustrates the superimposed image 54 when the overlap area is maximized by translating the inverted image 52. As described above, the difference between the inversion axis and the symmetry line when generating the inverted image can be absorbed by translating and / or rotating the image on the plane of the superimposed image.

[0049] The anomaly detection unit 104 may calculate the ratio of the maximum superimposed area to the area of ​​the target part in the captured image (expressed as maximum superimposed area / area of ​​the target part in the captured image, and may hereafter be referred to as the superimposed ratio), and determine whether or not there is an anomaly in the target part based on this ratio. The steel tower is constructed by combining straight steel members, and if the steel members, which are structural members, are in a normal state, the superimposed ratio will be 1.0 (100%) or a value very close to 1.0. On the other hand, if deformation anomalies such as bending occur in the steel members, the superimposed ratio will decrease according to the degree of deformation. The anomaly detection unit 104 may compare the superimposed ratio calculated from the superimposed image with a predetermined threshold, and if the superimposed ratio falls below the predetermined threshold, it may determine that there is an anomaly such as deformation in the target part. In the example shown in Figure 10(b), there is a discrepancy between the solid line and the dashed line at the location corresponding to member A, and the superimposed ratio has decreased, so it can be determined that bending deformation has occurred in member A. Even when superimposed images are generated in units of combined components, the superimposition ratio may be calculated for each component constituting the unit, and the presence or absence of abnormalities may be determined for each component.

[0050] Furthermore, the anomaly detection unit 104 may determine not only the presence or absence of an anomaly, but also the degree of the anomaly, which is indicated by a stepped or continuous level (anomaly level). For example, in the case of a deformation anomaly, multiple thresholds corresponding to the degree of deformation may be set. The anomaly detection unit 104 may output the level of the anomaly by identifying a level corresponding to the superposition ratio identified from the superimposed image. The threshold for detecting anomalies may be set, for example, based on preliminary experiments (which may be simulation experiments) and / or empirical rules.

[0051] As described above, by quantifying the state of the target area captured in the captured image using the area in the superimposed image, even minute abnormalities such as slight bends on the order of millimeters can be reliably detected. The abnormality detection unit 104 may, when performing the abnormality detection described above, remove the edge portion of the target area in the superimposed image (for example, a predetermined range from the longitudinal end) and determine the presence or absence of an abnormality based on the maximum superimposed area of ​​the target area after removal. Alternatively, the abnormality detection unit 104 may divide the target area in the superimposed image into multiple sections and perform the abnormality determination described above for each divided section.

[0052] The information / image storage unit 121 stores the captured images sent from the mobile body 4, as well as information associated with those captured images. The information / image storage unit 121 may also store information used by each of the functional units 101-104. Examples of information used by each of the functional units 101-104 include setting information related to the reference line when generating an inverted image, and setting information related to the threshold when determining the presence or absence of an abnormality and / or the level of the abnormality.

[0053] <An example of an anomaly detection method> Next, with reference to Figure 6, an example of an abnormality detection method during structural inspection using the information processing system according to this embodiment will be provided.

[0054] First, the imaging data acquisition unit 101 acquires an image captured by the camera 42 mounted on the mobile body 4, which includes at least a part of the structure (step SQ101). Next, the target part identification unit 102 uses the image acquired by the imaging data acquisition unit 101 as input information, performs segmentation on the image, and identifies the target part within the image (step SQ102). Note that the identification of the target part shown in step SQ102 may be performed by a known method other than segmentation.

[0055] The inversion processing unit 103 inverts the captured image around a straight line along the image plane to generate an inverted image of the captured image (step SQ103). In step SQ103, the axis (inversion axis) used to invert the captured image is not particularly limited, and the inverted image may be generated using the center line of the image as the inversion axis. After the generation of the inverted image, the anomaly detection unit 104 detects an anomaly in the target area depicted in the captured image based on a superimposed image obtained by superimposing the original captured image before inversion and the inverted image (step SQ104). In step SQ104, the anomaly detection unit 104 may translate and / or rotate the inverted image (which may be the captured image) on the plane of the superimposed image to calculate the maximum superimposed area, which is the maximum area in which the target area in the captured image and the target area in the inverted image overlap. The abnormality detection unit 104 may then calculate the maximum overlapping area, calculate the ratio of the maximum overlapping area to the area of ​​the target area in the captured image (overlap ratio), and determine whether or not there is an abnormality in the target area based on the said overlapping ratio.

[0056] The method described above (shown in the flowchart of Figure 6) can reliably detect even minute anomalies, such as slight bends on the order of millimeters. Note that the flowchart in Figure 6 is merely an example, and the anomaly detection method by the information processing system of this embodiment is not limited to the example shown in Figure 6. For example, the process of identifying the target area within the image may be performed after step SQ103 on both the captured image and the inverted image, or it may be performed on the superimposed image in step SQ104.

[0057] The embodiments described above are merely illustrative to facilitate understanding of this disclosure and are not intended to limit it. This disclosure may be modified and improved without departing from its intent, and its equivalents are included.

[0058] For example, some or all of the functions of the image data acquisition unit 101, target part identification unit 102, inversion processing unit 103, and anomaly detection unit 104 described in the above embodiment may be executed by the processor 20 of the user terminal 2, or by the flight controller 41 of the mobile body 4. Furthermore, although the above embodiment illustrates a system for inspecting structures with an unmanned mobile body 4, the information processing system of this disclosure may also be applied when photographing structures with a camera mounted on a manned mobile body, or when a person operates a shooting device such as a smartphone, tablet terminal, or digital camera to sequentially photograph structures.

[0059] Furthermore, while the above embodiments primarily illustrate photography for the purpose of inspecting power transmission equipment such as transmission towers, the structures to be inspected are not limited to the examples in the embodiments and may include radio towers, communication towers, bridges, plant-related facilities, various buildings, and other industrial facilities. Also, while the examples shown in Figures 7-10 mainly illustrate the detection of deformation abnormalities, the abnormalities detected by the information processing system of this disclosure are not limited to deformation abnormalities and may include various abnormalities such as defects, wear, peeling of the surface layer, loosening or detachment of members, and misalignment of joints. [Explanation of Symbols]

[0060] 1. Management Server 2 User terminals 4 Mobile Units

Claims

1. An imaging data acquisition unit that acquires images of at least a part of the structure, An inversion processing unit that inverts the captured image around a straight line along the image plane to generate an inverted image of the captured image, An information processing system comprising: an anomaly detection unit that detects an anomaly in a target area depicted in a captured image based on a superimposed image obtained by overlaying the original captured image before inversion and the inverted image.

2. The information processing system according to claim 1, further comprising a target area identification unit that identifies the target area that is the target for determining an abnormality from the captured image by segmentation.

3. The information processing system according to claim 1 or 2, wherein the anomaly detection unit translates and / or rotates the inverted image on the image plane to calculate the maximum overlapping area, which is the largest area in which the target portion in the captured image and the target portion in the inverted image overlap.

4. The information processing system according to claim 3, wherein the abnormality detection unit determines whether or not there is an abnormality in the target area based on the ratio of the maximum superimposed area to the area of ​​the target area in the captured image.

5. The information processing system according to claim 1 or 2, wherein the target portion included in the structure is a member having mirror image symmetry, or a combination of members.

6. To obtain images of at least a part of the structure, The captured image is inverted around a straight line along the image plane to generate an inverted image of the captured image, An information processing method in which a computer performs the following: detecting an abnormality in a target area depicted in a captured image based on a superimposed image obtained by overlaying the original captured image before inversion and the inverted image.

7. To obtain images of at least a part of the structure, The captured image is inverted around a straight line along the image plane to generate an inverted image of the captured image, A program to cause a computer to perform the following actions: detect an abnormality in a target area shown in a captured image based on a superimposed image obtained by overlaying the original captured image before inversion and the inverted image.

Images