Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program related to the inspection of a structure. [Solution] An information processing system comprising an imaging data acquisition unit that acquires a captured image of at least a portion of a structure, an inversion processing unit that inverts the captured image along a straight line that runs through the image plane to generate an inverted image of the captured image, and an abnormality detection unit that detects abnormalities in a target area depicted in the captured image based on a superimposed image formed by superimposing the original captured image before inversion and the inverted image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program related to the inspection of a structure. [Background technology]

[0002] As disclosed in Patent Document 1, a method for determining the damage state of a structure from an image of the structure is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2022 / 114006 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above Patent Document 1, the damage state of a structure is evaluated by calculating the difference between a past image and a most recent image. In such an evaluation method, it is necessary to align the past image with the most recent image, but it is difficult to obtain an image at exactly the same position as in the past inspection, and image parameters (color tone, contrast, saturation, brightness, etc.) change due to environmental changes at the time of shooting (weather, time of day, season, light conditions, other changes in objects in the environment, etc.), making it difficult to compare with the past image, and sufficient evaluation accuracy may not be obtained. In particular, minute changes such as slight distortion (deformation) of members constituting a structure cannot be detected by an evaluation method based on comparison with past images such as that of Patent Document 1.

[0005] An object of an exemplary embodiment of the present disclosure is to provide an information processing system, an information processing method, and a program related to the inspection of a structure. [Means for solving the problem]

[0006] An information processing system according to one embodiment of the present disclosure includes: an imaging data acquisition unit that acquires an image of at least a part of the structure; an inversion processing unit that inverts the captured image about a straight line along an image plane to generate an inverted image of the captured image; The imaging device further includes an abnormality detection unit that detects abnormalities in a target area captured in the captured image based on a superimposed image obtained by superimposing the original captured image before inversion and the inverted image.

[0007] The information processing system has the above-mentioned features, and thus it is possible to determine the presence or absence of an abnormality in a target portion contained in a structure. Other problems and means for solving the problems disclosed in the present application will be made clear by the embodiments and drawings of the present disclosure. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of the management server illustrated in FIG. [Diagram 3] FIG. 3 is a block diagram showing a hardware configuration of the moving object shown in FIG. [Figure 4] FIG. 4 is a block diagram illustrating an example of a software configuration of the management server illustrated in FIG. [Diagram 5] FIG. 5 is a conceptual diagram showing an example of operation of the information processing system shown in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of information processing related to anomaly detection. [Figure 7] FIG. 7 is a diagram showing an example of a captured image. [Figure 8] FIG. 8 is a diagram for explaining the target portion identification process. [Figure 9] FIG. 9 is a diagram for explaining the process of generating a reversed image. [Figure 10] FIG. 10 is a diagram for explaining the process of generating a superimposed image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The information processing method, information processing system, and program disclosed herein have, for example, the following configuration. [Item 1] an imaging data acquisition unit that acquires an image of at least a part of the structure; an inversion processing unit that inverts the captured image about a straight line along an image plane to generate an inverted image of the captured image; and an abnormality detection unit that detects abnormalities in a target area depicted in the captured image based on a superimposed image obtained by superimposing the original captured image before inversion and the inverted image. [Item 2] 2. The information processing system according to item 1, further comprising a target part identification unit that identifies, by segmentation, the target part that is the target for determining an abnormality from within the captured image. [Item 3] 3. The information processing system according to item 1 or 2, wherein the abnormality detection unit translates and / or rotates the inverted image on an image plane to calculate a maximum overlap area, which is a maximum area where a target area in the captured image overlaps with a target area in the inverted image. [Item 4] 4. The information processing system according to item 3, wherein the abnormality detection unit determines the presence or absence of an abnormality in the target part based on a ratio of the maximum overlap area to an area of ​​the target part in the captured image. [Item 5] 3. The information processing system according to item 1 or 2, wherein the target portion included in the structure is a component or a combination of components having mirror symmetry. [Item 6] Obtaining an image of at least a portion of a structure; Inverting the captured image about a straight line along an image plane to generate an inverted image of the captured image; and detecting abnormalities in a target area depicted in the captured image based on a superimposed image formed by superimposing the original captured image before inversion and the inverted image. [Item 7] Obtaining an image of at least a portion of a structure; Inverting the captured image about a straight line along an image plane to generate an inverted image of the captured image; and detecting abnormalities in a target area depicted in the captured image based on a superimposed image formed by superimposing the original captured image before inversion and the inverted image.

[0010] <Details of the embodiment> An information processing system according to an embodiment of the present disclosure will be described with reference to the drawings. In each of the attached drawings, identical or similar elements are given identical or similar reference symbols and names, and duplicate descriptions of identical or similar elements may be omitted in the description of the embodiment. Note that the contents shown in each drawing are merely examples for explaining the present embodiment, and are merely schematic examples for easy explanation of the present embodiment. The contents of each drawing may be modified or changed within the scope of no technical problem.

[0011] <System Overview> The information processing system according to the present embodiment is a system for inspecting a structure, which detects abnormalities in a specific part of the structure using an image of the structure. Examples of structures to be inspected include, but are not limited to, power transmission equipment such as steel towers, radio towers, communication towers, bridges, plant-related equipment (e.g., plant support structures, towers of oil refining plants, cooling towers, etc.), various buildings (e.g., steel structural frames, dome frameworks, observation decks, etc.), and other industrial equipment (cranes, lighthouses, wind power generation equipment, etc.). In addition, the method of photographing the structure is also not particularly limited. For example, the structure may be photographed by an unmanned moving body such as a drone, an unmanned aerial vehicle (UAV), an unmanned ground vehicle (UGV), or a camera mounted on a manned moving body, or a person such as a worker may operate a photographing device (e.g., various cameras such as a smartphone, a tablet terminal, and a digital camera) to photograph the target object.

[0012] The "predetermined portion (hereinafter referred to as the target portion)" of the object to be determined for the presence or absence of an abnormality may be, for example, a member constituting a structure (hereinafter sometimes referred to as a structural member), or a portion consisting of a combination of members (hereinafter sometimes referred to as a unit), and in particular may be a member or unit having mirror symmetry. Here, "mirror symmetry" means the property that the geometric shape of a portion or unit in a normal state (specifically, the geometric shape captured when imaged from a predetermined direction) is linearly symmetrical about a predetermined straight line. "Members or units having mirror symmetry" are not limited to those in which the entire shape of a portion or unit has mirror symmetry, but also include those that have partial mirror symmetry. In addition, even if the geometric shape when imaged from a predetermined direction does not have mirror symmetry, if at least a part of the geometric shape when imaged from another direction has mirror symmetry, it is considered to be included in the "members or units having mirror symmetry".

[0013] The captured image may include at least a part of the structural member of the target site, or at least a part of the unit of the combined members, and may not include the entire target site. The abnormalities detected by the information processing system are not necessarily limited, and may include, for example, deformation abnormalities such as bending, distortion, twisting, flexure, dents, expansion, and partial bulging, defects, wear, peeling of the surface layer, loosening and falling off of members, and positional deviation of joints, etc., and other abnormalities related to geometric shapes. The information processing system of this embodiment is particularly suitable for detecting minute deformation abnormalities such as extremely small bending.

[0014] In this embodiment, the information processing system will be described in detail with reference to an example in which a power transmission facility such as a steel tower is photographed by a camera 42 mounted on an unmanned mobile body 4 to inspect the power transmission facility, as shown in Fig. 5. The driving of the mobile body 4 may be autonomously controlled based on a preset moving route, or may be remotely controlled based on an instruction from a user terminal 2 owned by a user.

[0015] When structural components of a steel tower (such as the steel materials that make up the steel frame structure) become bent or otherwise abnormal, it can eventually lead to the collapse of the tower. For this reason, it is necessary to inspect steel towers regularly and find abnormalities in the structural components early on. For example, when inspecting the deformation of structural components of steel towers, a resolution is required that can detect deformations on the order of millimeters for linear structural components that are 5m long.

[0016] One possible method for inspecting deformation abnormalities in structural members is, for example, to extract edges from a captured image of the structural member and verify the linearity of the extracted edges. However, it is expected that various objects other than the structural member to be inspected will be captured in the captured image of the structure. In the edge extraction process, many edges originating from the contours of other objects are detected, making it difficult to selectively extract only the edges corresponding to the contour of the structural member to be inspected. Even if it were possible to selectively extract the edges of the structural member, it would be difficult to detect minute deformation abnormalities due to the influence of discontinuity (discontinuity) of the extracted edges, noise (e.g., unnecessary edges such as textures), etc.

[0017] Also, as disclosed in Patent Document 1, it is possible to inspect structural members for abnormalities by comparing previously captured images with the most recently captured images. However, it is difficult to capture images in exactly the same positions as those used during previous inspections, and image parameters (color tone, contrast, saturation, brightness, etc.) change depending on environmental changes at the time of capture (weather, time of day, season, lighting conditions, changes in objects in the environment, etc.). Therefore, it is difficult to compare with previously captured images, and even with this method, it is difficult to detect minute deformation abnormalities.

[0018] In the information processing system of this embodiment, a photographed image of at least a part of a structure is inverted about a straight line along the image plane to generate an inverted image of the photographed image, and an abnormality of a target part shown in the photographed image is detected based on a superimposed image of the original photographed image before inversion and the inverted image. With this method, it is possible to efficiently detect abnormalities in target parts such as structural members and units combining members without referring to past photographed images. In particular, it is possible to detect deformation abnormalities on the order of millimeters for structural members having lengths on the order of meters. The information processing system of this embodiment will be described in detail below, taking as an example a steel tower inspection using a UAV (mobile body 4).

[0019] <System configuration> As shown in FIG. 1, the information processing system of the present embodiment may have a management server 1, one or more user terminals 2, and one or more mobile objects 4. The management server 1, the user terminal 2, and the mobile object 4 are connected to each other via a network NW so that they can communicate with each other. The illustrated configuration is an example, and is not limited to this. For example, the mobile object 4 may not be connected to the network NW. In that case, the operation of the mobile object 4 may be performed by a transmitter (so-called radio control) operated by a user. In addition, the image data acquired by the camera of the mobile object 4 may be stored in an auxiliary storage device (for example, a memory card such as an SD card and / or a USB memory) connected to the mobile object 4, and may be read out from the auxiliary storage device to the user terminal 2 and / or the management server 1 and stored by the user after the fact. The mobile object 4 may be connected to the network NW only for the purpose of operation or for the purpose of storing image data.

[0020] <Administration Server 1> 2 is a diagram showing a hardware configuration of the management server 1. Note that the illustrated configuration is just an example, and the management server 1 may have other configurations.

[0021] The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, an input / output unit 14, etc., which are electrically connected to each other via a bus 15. The management server 1 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0022] The processor 10 is a computing device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing required for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs stored in the storage 12 and deployed in the memory 11 to perform each information processing.

[0023] The memory 11 includes a main memory configured with a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory configured with a non-volatile storage device such as a flash memory, an HDD (Hard Disc Drive), etc. The memory 11 is used as a work area for the processor 10, and also stores a BIOS (Basic Input / Output System) that is executed when the management server 1 is started, various setting information, and the like.

[0024] The storage 12 stores various programs such as application programs. A database storing data used in each process may be constructed in the storage 12. For example, a storage unit 120 described later may be provided in a part of the storage area of ​​the storage 12.

[0025] The transmission / reception unit 13 is a communication interface for the management server 1 to communicate with the user terminal 2, the mobile object 4, etc. via a communication network. The transmission / reception unit 13 may further include a short-range communication interface such as Bluetooth (registered trademark) and BLE (Bluetooth Low Energy) and / or a USB (Universal Serial Bus) terminal, etc.

[0026] The input / output unit 14 includes information input devices such as a keyboard and a mouse, and output devices such as a display.

[0027] A bus 15 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals and various control signals.

[0028] <User device 2> The user terminal 2 also includes a processor 20, a memory 21, a storage 22, a transmission / reception unit 23, an input / output unit 24, etc., which are electrically connected to each other via a bus 25. The user terminal 2 may be a general-purpose computer such as a workstation or a personal computer. The functions of each element of the user terminal 2 can be configured in the same way as the above-mentioned management server 1, and detailed description of each element of the user terminal 2 will be omitted.

[0029] <Mobile unit 4> 3 is a block diagram showing a hardware configuration of the moving object 4. The flight controller 41 may have one or more processors, such as a programmable processor (for example, a central processing unit (CPU)).

[0030] The flight controller 41 may also have or have access to a memory 411. The memory 411 stores logic, code, and / or program instructions that the flight controller can execute to perform one or more steps. The flight controller 41 may also include sensors 412, such as inertial sensors (accelerometers, gyro sensors), GPS sensors, and proximity sensors (e.g., lidar).

[0031] The memory 411 may include, for example, a separable medium such as an SD card and a random access memory (RAM) or an external storage device. Data acquired from the camera / sensors 42 may be directly transmitted to and stored in the memory 411. For example, still image / video data captured by a camera or the like may be recorded in an internal memory or an external memory, but is not limited thereto, and may be recorded in at least one of the management server 1 and the user terminal 2 from the camera / sensor 42 or the internal memory via the network NW. The camera 42 is installed on the moving body 4 via a gimbal 43.

[0032] The flight controller 41 includes a control module (not shown) configured to control the state of the moving body 4. For example, the control module has six degrees of freedom (translational motion x, y, and z, and rotational motion θ x , θ y and θ z ), the flight controller 41 controls the propulsion mechanism (motor 45, etc.) of the moving body 4 via an ESC 44 (Electric Speed ​​Controller) in order to adjust the spatial arrangement, speed, and / or acceleration of the moving body 4 having a rotor 44a. The propeller 46 rotates by the motor 45 powered by a battery 48, thereby generating lift for the moving body 4. The control module of the flight controller 41 may have a function of controlling the driving of the camera 42 and the shooting conditions (for example, sharpness, focus (focal length), exposure value, shutter speed, ISO sensitivity, lens aperture, etc.). The control module can also control one or more of the states of the mounted parts and sensors.

[0033] The flight controller 41 can communicate with a transceiver 47 configured to transmit and / or receive data from one or more external devices (e.g., a transceiver 49, a terminal, a display device, or other remote control). The transceiver 49 may use any suitable communication means, such as wired or wireless communication.

[0034] For example, the transceiver 47 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.

[0035] The transceiver unit 47 can transmit and / or receive one or more of the following: data acquired by the cameras / sensors 42, processing results generated by the flight controller 41, specified control data, user commands from a terminal or a remote controller, etc.

[0036] The cameras / sensors 42 may include inertial sensors (accelerometers, gyro sensors), GPS sensors, proximity sensors (e.g., lidar), or vision / image sensors (e.g., cameras).

[0037] <Management Server 1 Function> FIG. 4 is a block diagram illustrating functions implemented in the management server 1. In this embodiment, the management server 1 may include an imaging data acquisition unit 101, a target part identification unit 102, an inversion processing unit 103, and an abnormality detection unit 104. The storage unit 120 of the management server 1 may include various databases such as an information / image storage unit 121. Note that the various functional units illustrated in FIG. 4 are illustrated as functional units realized by the processor 10 of the management server 1, but some or all of the various functional units may be realized in the processor 20 of the user terminal 2 or the processor 20 and / or the flight controller 41 according to the capabilities of the flight controller 41 of the moving object 4, etc.

[0038] The imaging data acquisition unit 101 acquires a captured image of at least a part of a structure. For example, the imaging data acquisition unit 101 acquires a captured image captured by a camera 42 mounted on a moving body 4 by wireless communication via a communication interface. The captured image shows at least a part of a structural member included in the structure and / or a unit which is a combination of members, and does not necessarily show the entire structural member or unit. An image 50 shown in FIG. 7 is an example of a simulated captured image, and a part of a steel structure captured by the camera 42 is shown as a target portion 60 in the image 50.

[0039] The imaging data acquisition unit 101 may acquire captured images in real time by wireless communication from the moving object 4 or within the moving object 4. The captured images acquired by the imaging data acquisition unit 101 may be moving images or still images. When the captured images are moving images, the moving images may be divided into still images for each frame, and the still images may be used in each functional unit described later. Still images may be extracted at predetermined intervals from the still images divided into frames, and the extracted still images may be used in each functional unit. The captured images captured by the moving object 4 or the like may be color images, grayscale images, or black and white images.

[0040] The target part identification unit 102 receives the captured image acquired by the image data acquisition unit 101 as input information, and executes a process of identifying a target part to be judged for anomaly from the captured image by segmentation. Segmentation is a method of dividing the captured image into regions for each type of object depicted in the captured image by determining which segment (e.g., category such as power line, steel tower, background, etc.) each pixel constituting the captured image belongs to and labeling each pixel. The target part identification unit 102 may be constructed by an AI model that has been subjected to machine learning (e.g., deep learning) that can extract a predetermined target part (component, unit, etc. constituting a structure) from the captured image. The architecture of the segmentation executed by the target part identification unit 102 is not particularly limited, and for example, LR-ASPP (Lite Reduced Atrous Spatial Pyramid Pooling) may be used.

[0041] The target part identification unit 102 may identify, as the target part, an area determined by segmentation as a segment indicating the structure to be inspected. The target part identification unit 102 may also estimate an area having mirror symmetry from among the areas determined by segmentation as a segment indicating the structure to be inspected, and identify the estimated area as the target part. In this case, high accuracy is not required for the estimation of mirror symmetry, and the object may be estimated to have mirror symmetry when it satisfies a predetermined condition (for example, the object is linear, the minimum circumscribing figure is a line-symmetric figure such as a rectangle or a circle, etc.) that is simply set.

[0042] The target part identification unit 102 may generate a mask image that visualizes the divided segments in the captured image based on the segmentation result. For example, as shown in Fig. 8, the target part identification unit 102 may generate a mask image 50a in which an area corresponding to the structure to be inspected (target part 60) in the captured image is made visible and an area other than the structure to be inspected (area indicated by reference numeral 65 in Fig. 8) is made hidden. The target part identification unit 102 is not limited to the example shown in Fig. 8, and may generate a mask image that specifies the target part 60 in an identifiable manner in the captured image, or may generate a primary processed image in which an area corresponding to the target part 60 is extracted from the captured image.

[0043] If the target part is clear in the photographed image, it is not necessary to perform segmentation processing, and the target part in the photographed image may be identified by other known methods instead of segmentation processing. The process of identifying the target part by the target part identification unit 102 may be performed before the process of generating an inverted image by the inversion processing unit 103 described later, or a process of identifying the target part from within the image may be performed for each of the original photographed image and the inverted image after the inversion process. Alternatively, a process of identifying the target part from within the image may be performed in a superimposed image generated in the course of information processing performed by the abnormality detection unit 104.

[0044] The inversion processing unit 103 inverts the captured image about a predetermined straight line along the image plane to generate an inverted image of the captured image. The captured image that is the source of the inverted image may be a captured image acquired by the imaging data acquisition unit 101 that has not been subjected to primary processing, or may be a captured image that has been subjected to primary processing such as a mask image that reflects the results of segmentation.

[0045] FIG. 9 is a diagram for explaining the generation process of an inverted image. The axis for inverting the captured image is not particularly limited. For example, the center line of the image (dash-dotted line L1 or L2 shown in FIG. 9(a)) perpendicular to the width direction of the captured image may be set as the inversion axis, or the diagonal line of the captured image (dash-dotted line L3 or L4 shown in FIG. 9(a)) may be set as the inversion axis. Alternatively, the inversion processing unit 103 may specify a bisector (symmetric line) that divides the region specified as the target part into two equal parts, such as the two-dot chain line L5 shown in FIG. 9(a), and then set the bisector as the inversion axis. In this case, it is not necessary to specify the bisector accurately, and an error may occur. From the viewpoint of reducing the calculation load during the inversion process, it is not necessary to specify the bisector, and any one of the reference lines that is uniformly set for the captured image regardless of the subject, such as the center line of the image (L1 or L2), may be set as the inversion axis.

[0046] The inversion processing unit 103 generates an inverted image by rotating the captured image by 180 degrees around a predetermined straight line along the image plane as described above. Fig. 9(b) illustrates an inverted image 52 obtained by inverting the captured image 50 shown in Fig. 9(a) around the center line L1.

[0047] The abnormality detection unit 104 detects an abnormality in the target part shown in the photographed image based on a superimposed image obtained by superimposing the original photographed image before inversion and the inverted image. For example, the abnormality detection unit 104 may determine whether or not an abnormality related to a geometric shape exists in the target part based on the degree of overlap between the target part in the photographed image and the target part in the inverted image shown in the superimposed image.

[0048] In order to specify the degree of overlap, the abnormality detection unit 104 may translate and / or rotate the inverted image on the plane of the superimposed image to calculate a maximum overlapping area, which is the maximum area where the target part in the photographed image and the target part in the inverted image overlap. The superimposed image 54 illustrated in FIG. 10(a) is an image generated by superimposing the photographed image 50 in FIG. 9(a) and the inverted image 52 in FIG. 9(b) so that the outer edges of the images match, and the photographed image 50 is indicated by a solid line and the inverted image 52 by a dashed line. When the photographed image 50 is inverted about a reference line (such as a center line) different from the bisector (line of symmetry) of the target part, the images may be superimposed with the target part in the photographed image and the target part in the inverted image misaligned, as shown in FIG. 9(a). In such a case, the maximum overlapping area can be calculated by translating and / or rotating the inverted image on the plane of the superimposed image (the photographed image may be moved) while calculating the overlapping area between the target part in the photographed image and the target part in the inverted image. Fig. 10(b) shows an example of a superimposed image 54 when the overlapping area is maximized by translating the inverted image 52. As described above, the difference between the inversion axis and the line of symmetry when generating the inverted image can be absorbed by translating and / or rotating the image on the plane of the superimposed image.

[0049] The abnormality detection unit 104 may calculate the ratio of the maximum overlap area to the area of ​​the target part in the captured image (represented by maximum overlap area / area of ​​the target part in the captured image, hereinafter sometimes referred to as overlap ratio), and may determine the presence or absence of an abnormality in the target part based on the ratio. A steel tower is constructed by combining straight steel materials, and if the steel materials, which are structural members, are in a normal state, the overlap ratio is 1.0 (100%) or a value very close to 1.0. On the other hand, if the steel materials are deformed abnormally, such as bent, the overlap ratio decreases according to the degree of deformation. The abnormality detection unit 104 may compare the overlap ratio calculated from the superimposed image with a predetermined threshold value, and determine that an abnormality such as deformation abnormality exists in the target part when the overlap ratio falls below the predetermined threshold value. In the example shown in FIG. 10(b), a deviation occurs between the solid line and the broken line at a location corresponding to member A, and the overlap ratio decreases, and it can be determined that member A is bent. Even if a superimposed image is generated for each unit made up of a combination of members, the superimposition ratio may be calculated for each member constituting the unit, and the presence or absence of an abnormality may be determined for each member.

[0050] Furthermore, the anomaly detection unit 104 may determine not only the "presence or absence of anomaly" but also the degree of the anomaly indicated by a stepped or continuous level (anomaly level). For example, in the case of a deformation anomaly, a plurality of thresholds may be set according to the degree of deformation. The anomaly detection unit 104 may output the level of the anomaly by specifying a level corresponding to the superimposition ratio specified from the superimposed image. The threshold for detecting an anomaly may be set based on, for example, a preliminary experiment (which may be an experiment by simulation) and / or an empirical rule.

[0051] As described above, by quantifying the state of the target part shown in the captured image by the area in the superimposed image, it becomes possible to reliably detect even a minute abnormality such as a slight bend on the order of millimeters. When detecting the abnormality described above, the abnormality detection unit 104 may remove the edge portion (e.g., a predetermined range from the longitudinal end) of the target part in the superimposed image, and determine the presence or absence of the abnormality based on the maximum superimposed area of ​​the target part after the removal. Furthermore, the abnormality detection unit 104 may divide the target part in the superimposed image into a plurality of sections, and perform the above-mentioned determination of the presence or absence of the abnormality for each divided section.

[0052] The information / image storage unit 121 stores the captured images sent from the moving object 4 and information associated with the captured images. The information / image storage unit 121 may also store information used by the functional units 101-104. Examples of the information used by the functional units 101-104 include setting information related to a reference line when generating an inverted image, and setting information related to a threshold value when determining the presence or absence of an abnormality and / or the level of the abnormality.

[0053] <An example of an anomaly detection method> Next, with reference to FIG. 6, an example of an abnormality detection method during structure inspection by the information processing system according to this embodiment will be described.

[0054] First, the imaging data acquisition unit 101 acquires a captured image including at least a part of a structure captured by the camera 42 mounted on the moving object 4 (step SQ101). Next, the target part identification unit 102 executes segmentation on the captured image using the captured image acquired by the imaging data acquisition unit 101 as input information, and identifies a target part in the captured image (step SQ102). Note that the identification of the target part shown in step SQ102 may be executed by a known method other than segmentation.

[0055] The inversion processing unit 103 inverts the photographed image about a straight line along the image plane to generate an inverted image of the photographed image (step SQ103). In this step SQ103, the axis (inversion axis) for inverting the photographed image is not particularly limited, and the inverted image may be generated about the center line of the image or the like. After the inverted image is generated, the abnormality detection unit 104 detects an abnormality in the target area shown in the photographed image based on a superimposed image obtained by superimposing the original photographed image before inversion and the inverted image (step SQ104). In this step SQ104, the abnormality detection unit 104 may translate and / or rotate the inverted image (which may be the photographed image) on the plane of the superimposed image to calculate a maximum overlapping area, which is the maximum area where the target area in the photographed image and the target area in the inverted image overlap. Then, the abnormality detection unit 104 may calculate the maximum overlapping area, and then calculate the ratio (overlap ratio) of the maximum overlapping area to the area of ​​the target part in the captured image, and determine whether or not there is an abnormality in the target part based on the overlapping ratio.

[0056] The above method (the method shown in the flowchart in FIG. 6) can reliably detect even minute abnormalities such as slight bending on the order of millimeters. Note that the flowchart in FIG. 6 is merely an example, and the abnormality detection method by the information processing system of this embodiment is not limited to the example in FIG. 6. For example, the process of identifying the target area in the image may be performed after step SQ103 for each of the captured image and the inverted image, or may be performed for the superimposed image in step SQ104.

[0057] The above-described embodiment is merely an example for facilitating understanding of the present disclosure, and is not intended to limit the present disclosure. The present disclosure can be modified or improved without departing from the spirit thereof, and it goes without saying that the present disclosure includes equivalents thereof.

[0058] For example, some or all of the functional units of the imaging data acquisition unit 101, the target part identification unit 102, the inversion processing unit 103, and the anomaly detection unit 104 described in the above embodiment may be executed by the processor 20 of the user terminal 2, or may be executed by the flight controller 41 of the moving body 4. Furthermore, while the above embodiment illustrates a system in which an unmanned moving body 4 inspects a structure, the information processing system of the present disclosure may be applied to a case in which a structure is photographed by a camera or the like mounted on a manned moving body, or a case in which a person sequentially photographs a structure by operating an imaging device such as a smartphone, a tablet terminal, or a digital camera.

[0059] In the above embodiment, the photographing is mainly for the purpose of inspecting power transmission facilities such as steel towers, but the structures to be inspected are not limited to the examples in the embodiment, and may be radio towers, communication towers, bridges, plant-related facilities, various buildings, and other industrial facilities. In addition, in the examples shown in Figures 7 to 10, the cases where deformation abnormalities are mainly detected are illustrated, but the abnormalities detected by the information processing system of the present disclosure are not limited to deformation abnormalities, and may be various abnormalities such as defects, wear, peeling of the surface layer, loosening and falling off of members, and misalignment of joints, etc. [Explanation of symbols]

[0060] 1 Management Server 2. User terminal 4. Mobile

Claims

1. an imaging data acquisition unit that acquires an image of at least a part of the structure; an inversion processing unit that inverts the captured image about a straight line along an image plane to generate an inverted image of the captured image; and an abnormality detection unit that detects abnormalities in the target area contained in the structure based on a superimposed image obtained by superimposing the target area before inversion in the original captured image on the target area after inversion in the inverted image.

2. The information processing system according to claim 1 , further comprising a target portion identification unit that identifies, by segmentation, the target portion that is a target for determining an abnormality from within the captured image.

3. 3. The information processing system according to claim 1, wherein the abnormality detection unit translates and / or rotates the inverted image on an image plane to calculate a maximum overlap area, which is the maximum area of ​​overlap between the target area in the captured image and the target area in the inverted image.

4. The information processing system according to claim 3 , wherein the abnormality detection unit determines the presence or absence of deformation of the target part based on a ratio of the maximum overlapping area to an area of ​​the target part in the captured image.

5. The information processing system according to claim 1 , wherein the target portion included in the structure is a member having mirror symmetry or a combination of members.

6. An information processing system as described in claim 1 or 2, wherein the abnormality detection unit detects abnormalities related to the geometric shape of the target portion contained in the structure based on the degree of overlap between the target portion before inversion and the target portion after inversion, calculated using the superimposed image.

7. Obtaining an image of at least a portion of a structure; Inverting the captured image about a straight line along an image plane to generate an inverted image of the captured image; and detecting an abnormality in the target area contained in the structure based on a superimposed image in which the target area before inversion in the original captured image is superimposed on the target area after inversion in the inverted image.

8. Obtaining an image of at least a portion of a structure; Inverting the captured image about a straight line along an image plane to generate an inverted image of the captured image; and detecting abnormalities in the target area contained in the structure based on a superimposed image obtained by superimposing the target area before inversion in the original captured image on the target area after inversion in the inverted image.

Images