Structure condition determination device, machine learning device, inference device, structure condition determination method, and machine learning method

The structure state determination device uses machine learning to classify and predict damage progression in structures, enhancing maintenance planning by accurately assessing damage type, degree, and time, thereby improving maintenance efficiency.

JP7586486B2Active Publication Date: 2024-11-19SKYSTER CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021047156
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-22
Publication Date
2024-11-19
Estimated Expiration
2041-03-22

AI Technical Summary

Technical Problem

Existing systems fail to accurately determine the time required for damage progression in structures, making it difficult to plan inspections and maintenance based on the extent of damage, rather than the impairment of structural function.

Method used

A structure state determination device and method that utilize machine learning to analyze images of structures, classifying damage type, progression degree, and time required for progression using a trained model, incorporating environmental data for improved accuracy.

Benefits of technology

Enables precise determination of damage state and progression time, facilitating targeted inspections and maintenance planning based on damage extent.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586486000001
    Figure 0007586486000001
  • Figure 0007586486000002
    Figure 0007586486000002
  • Figure 0007586486000003
    Figure 0007586486000003
Patent Text Reader

Abstract

To provide a structure state determination device which facilitates implementation of inspection and maintenance works according to the damage progression degree.SOLUTION: A structure state determination device for determining the damage state of a structure comprises: a determination data acquisition unit 500 which acquires determination data including a determination image in which at least a portion of a structure is imaged; a learned model storage unit 52 which stores a learned model 2 that has machine-learned a correlation between input data including a learning image and output data including the damage type, damage progression degree and damage progression time included in the learning image; and an inference unit 501 which inputs the determination data acquired by the determination data acquisition unit 500 to the learning model 2 and infers the damage type, damage progression degree and damage progression time as the damage state of the structure captured in the determination image.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a structure state determination device, a machine learning device, an inference device, a structure state determination method, and a machine learning method. [Background technology]

[0002] Conventionally, when determining whether or not to perform inspection and maintenance work on various structures, or when formulating an implementation plan for the inspection and maintenance work, the surface of the structure is imaged by an imaging device, and the damage state of the structure is evaluated based on the image. For example, Patent Document 1 discloses an apparatus that detects the depth of a crack from an image of the surface of an inspection target, and evaluates the life of the structure according to the depth of the crack. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-175110 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 discloses that the lifespan of a structure is evaluated based on the depth of cracks that have appeared on the surface of the object to be inspected, but the lifespan of a structure is considered to represent the time until the function of the structure is impaired.

[0005] Here, when a structure is damaged due to its deterioration over time, the degree of damage, which indicates the extent of the damage, gradually progresses. Therefore, the manager of the structure needs to determine whether or not to carry out inspection and maintenance work and to formulate an implementation plan for the inspection and maintenance work, anticipating not the life of the structure, i.e., the time until the function of the structure is impaired, but the time required for the damage degree to progress to the next stage. However, the device disclosed in Patent Document 1 cannot obtain the time required for the damage degree to progress to the next stage, making it difficult to carry out inspection and maintenance work according to the damage degree.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a structure condition determination device, a machine learning device, an inference device, a structure condition determination method, and a machine learning method that facilitate the implementation of inspection and maintenance work according to the degree of damage progression. [Means for solving the problem]

[0007] In order to achieve the above object, a structure state determination device according to one aspect of the present invention comprises: A structure condition determination device for determining a damage condition of a structure, comprising: a determination data acquisition unit that acquires determination data including a determination image in which at least a portion of the structure is captured; a trained model storage unit that stores a trained model that has been machine-learned to obtain a correlation between input data including a training image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the training image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; and The method includes an inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers the damage type, the damage progression level, and the damage progression time as the damage state of the structure captured in the judgment image. Effect of the Invention

[0008] According to one embodiment of the structure condition determination device of the present invention, determination data including a determination image in which at least a portion of a structure is captured is input into a learning model, and the damage type, damage progression, and damage progression time are inferred as the damage state of the structure captured in the determination image, thereby making it easier to carry out inspection and maintenance work according to the damage progression.

[0009] Other objects, configurations and effects will become apparent from the detailed description of the invention described below. [Brief description of the drawings]

[0010] [Figure 1] 1 is an overall configuration diagram showing an example of a structure management system 1 according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a machine learning device 4 according to the first embodiment. [Diagram 3] FIG. 2 is a data configuration diagram showing an example of learning data according to the first embodiment. [Figure 4] 1 is a schematic diagram showing an example of a neural network model 20 applied to a learning model 2 according to a first embodiment. FIG. [Diagram 5] 1 is a block diagram showing an example of a structure state determination device 5 according to a first embodiment. [Figure 6] FIG. 2 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501 according to the first embodiment. [Figure 7] 1 is a hardware configuration diagram showing an example of a computer 900 constituting each of the devices 3 to 6 of the structure management system 1. FIG. [Figure 8] 4 is a flowchart showing an example of a machine learning method performed by the machine learning device 4 according to the first embodiment. [Figure 9] 5 is a flowchart showing an example of a structure state determination method performed by the structure state determination device 5 according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of a machine learning device 4 according to a second embodiment. [Figure 11]FIG. 11 is a schematic diagram showing an example of a neural network model 20b applied to a learning model 2a according to the second embodiment. [Figure 12] FIG. 11 is a block diagram showing an example of a structure state determination device 5 according to a second embodiment. [Figure 13] FIG. 11 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501a according to the second embodiment. [Figure 14] FIG. 11 is a block diagram showing an example of a machine learning device 4 according to a third embodiment. [Figure 15] FIG. 11 is a schematic diagram showing an example of a neural network model 20b applied to a learning model 2b according to the third embodiment. [Figure 16] FIG. 11 is a block diagram showing an example of a structure state determination device 5 according to a third embodiment. [Figure 17] FIG. 13 is a functional explanatory diagram showing an example of an inference process performed by an inference unit 501b according to the third embodiment. [Figure 18] 1 is a graph showing the relationship between the damage progression level and damage progression time for each damage type. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. In the following, the scope necessary for the explanation to achieve the object of the present invention will be shown in a schematic manner, and the scope necessary for the explanation of the relevant part of the present invention will be mainly explained, and the parts that are omitted will be explained as being related to the publicly known technology.

[0012] (First embodiment) 1 is an overall configuration diagram showing an example of a structure management system 1 according to a first embodiment. The structure management system 1 is a system that manages the damage state of a structure 10 that is a management target.

[0013] The structure management system 1 comprises an imaging device 3 that images at least a portion of a structure 10 to be managed, a machine learning device 4 that operates as a subject of the learning phase of the machine learning, a structure state determination device 5 that operates as a subject of the inference phase of the machine learning, and a terminal device 6 used by a manager or worker of the structure management system 1. The devices 3 to 6 of the structure management system 1 are connected to each other via a network 7 so that they can communicate with each other.

[0014] The structure 10 is constructed in various locations by being formed from exterior materials or paving materials such as concrete, fiber-reinforced plastic (FRP), etc. Specific examples of the structure 10 include, but are not limited to, buildings, bridges, solar power generation facilities, wind power generation facilities, dams, railway facilities, roads, chimneys, tunnels, etc.

[0015] Various types of damage 100 may occur in each part of the structure 10 due to deterioration over time, and when damage 100 occurs, the degree of damage 100 gradually progresses as the damage progression level. Therefore, the structure management system 1 operates as a system for managing the damage state of the structure 10 and providing information regarding the need for inspection and maintenance work, and information regarding the formulation of an implementation plan for inspection and maintenance work.

[0016] The imaging device 3 includes a camera configured with an image sensor such as a CMOS sensor or a CCD sensor, and captures images of each part of the structure 10. The imaging device 3 may be a handheld or fixed camera that is operated by an operator to capture images, or may automatically capture images when a predetermined imaging condition is satisfied. The imaging device 3 may be a camera attached to a drone, a robot, or the like, which is remotely controlled or autonomously moved to each imaging position of the structure 10 to capture images.

[0017] The imaging device 3 captures an image of the structure 10 within the angle of view of the imaging device 3, and outputs the image as digital data based on a predetermined image format. As shown in Fig. 1, the imaging device 3 may be provided separately as an imaging device 3 connected to the machine learning device 4 and an imaging device 3 connected to the structure state determination device 5, or one imaging device 3 may be connected to both the machine learning device 4 and the structure state determination device 5 and used in common. Also, in Fig. 1, for the sake of simplicity, one imaging device 3 is connected to each of the machine learning device 4 and the structure state determination device 5, but multiple imaging devices 3 may be connected to each.

[0018] The machine learning device 4 is composed of a general-purpose or dedicated computer (see FIG. 7 described later) or the like, and operates as a main player in the learning phase of machine learning. The machine learning device 4 performs machine learning of the learning model 2 using learning data including images (learning images) captured by the imaging device 3. The machine learning device 4 provides the trained learning model 2 to the structure state determination device 5 via a network 7, a recording medium, or the like. The details of the machine learning device 4 will be described later.

[0019] The structure state determination device 5 is composed of a general-purpose or dedicated computer (see FIG. 7 described later) or the like, and operates as a main part of the inference phase in machine learning. The structure state determination device 5 determines the damage state of the structure 10 from an image (determination image) captured by the imaging device 3 using a learning model 2 trained by the machine learning device 4. The structure state determination device 5 will be described in detail later.

[0020] The terminal device 6 is configured with a general-purpose or dedicated computer (see FIG. 7 described later) or the like. The terminal device 6 is used for preparing learning data, machine learning of the learning model 2, and In order to determine the damage state of the structure 10, various operational inputs are accepted via the input screen, and various information is displayed via a display screen such as an app or a browser. Also, for the sake of simplicity, one terminal device 6 is illustrated in Fig. 1, but multiple terminal devices 6 may be used.

[0021] (Machine Learning Device 4) 2 is a block diagram showing an example of the machine learning device 4 according to the first embodiment. The machine learning device 4 includes a control unit 40, a communication unit 41, a training data storage unit 42, and a trained model storage unit 43.

[0022] The control unit 40 functions as a learning data acquisition unit 400 and a machine learning unit 401. The communication unit 41 is connected to external devices (e.g., the imaging device 3, the structure state determination device 5, and the terminal device 6) via the network 7, and functions as a communication interface for transmitting and receiving various data.

[0023] The learning data acquisition unit 400 is connected to an external device via the communication unit 41 and the network 7, and acquires learning data configured by associating input data with output data.

[0024] The learning data storage unit 42 is a database that stores a plurality of sets of learning data acquired by the learning data acquisition unit 400. The specific configuration of the database that constitutes the learning data storage unit 42 may be designed as appropriate.

[0025] The machine learning unit 401 performs machine learning using the learning data stored in the learning data storage unit 42. That is, the machine learning unit 401 generates the learning model 2 by inputting a plurality of sets of learning data to the learning model 2 and having the learning model 2 learn by machine learning the correlation between the input data and the output data that constitute the learning data.

[0026] The trained model storage unit 43 is a database that stores the trained learning model 2 trained by the machine learning unit 401. The trained model 2 stored in the trained model storage unit 43 is provided to a real system (e.g., a structure state determination device 5) via a network 7, a recording medium, or the like. The trained model 2 may be provided to an external computer (e.g., a server-type computer or a cloud-type computer) and stored in a storage unit of the external computer. Also, although the trained data storage unit 42 and the trained model storage unit 43 are shown as separate storage units in FIG. 2, they may be configured as a single storage unit.

[0027] 3 is a data configuration diagram showing an example of learning data according to the first embodiment. The learning data is composed of input data including learning images 30, and output data including the damage type, damage progression level, and damage progression time of the damaged portion included in the learning images 30.

[0028] The learning image 30 included in the input data is an image of at least a part of the structure 10 captured by the imaging device 3.

[0029] The output data is, for example, called a correct answer label in supervised learning, and is assigned according to the damage state of the structure 10 included in the learning image 30.

[0030] The damage type included in the output data indicates the type of the state when the damage 100 occurs in the structure 10, for example, when classified by the shape or distribution of the damage 100. When the structure 10 is made of concrete, the damage type is classified into a plurality of classes, such as "no damage," "irregular crack," "parallel crack," "lattice crack," and "peeling," including the normal state when the damage 100 does not occur, as shown in FIG. 3. When the structure 10 is made of FRP, the damage type is classified into a plurality of classes, such as "no damage," "irregular crack," "parallel crack," "lattice crack," and "peeling," including the normal state when the damage 100 does not occur, as shown in FIG. 3. The damage types are classified into a plurality of classes, such as "no damage", "wear", and "delamination". Note that the damage types are not limited to the definitions and numbers of the above classifications.

[0031] The damage progression included in the output data indicates the progression of the damaged portion when the progression of the damage 100 is classified into a plurality of stages. When damage 100 occurs in the structure 10 from a normal state, the damage progression is classified into three stages, such as mild, moderate, and severe, according to the progression of the damage 100. In this case, the damage progression is classified into a plurality of classes, such as "mild," "moderate," and "severe." Note that the damage progression is not limited to the definition and number of the above classifications.

[0032] The damage progression time included in the output data indicates the time required for the damage 100 progression level to progress to the next stage. For example, if the current damage 100 progression level is mild, the damage 100 progression time indicates the time required for the damage 100 progression level to progress to moderate. In this case, the unit of damage progression time may be any of days, weeks, months, and years.

[0033] 3, the output data is associated with the learning image 30 included in the input data as a damage type "parallel crack", a damage progression level "medium", and a damage progression time "3 (months)" (in this embodiment, the unit is months). Note that one learning image 30 may include multiple damaged portions.

[0034] 4 is a schematic diagram showing an example of a neural network model 20 applied to the learning model 2 according to the first embodiment. The neural network model 20 employs a convolutional neural network (CNN) as a specific method of machine learning. The neural network model 20 includes an input layer 21, an intermediate layer 22, and an output layer 23.

[0035] The input layer 21 has neurons whose number corresponds to the number of pixels of the learning image 30 as input data, and the pixel value of each pixel is input to each neuron.

[0036] The intermediate layer 22 is composed of a convolutional layer 220, a pooling layer 221, and a fully connected layer 222. The convolutional layer 220 and the pooling layer 221 are provided, for example, in a plurality of alternating layers. The convolutional layer 220 and the pooling layer 221 extract features from an image input via the input layer 21. The fully connected layer 222 converts the two-dimensional array of features extracted from the image by the convolutional layer 220 and the pooling layer 221, for example, by an activation function, and outputs it as a one-dimensional array of feature vectors. Note that the fully connected layer 222 may be provided in a plurality of layers.

[0037] The output layer 23 outputs output data including a determination result of a damaged portion included in the learning image 30 based on the feature vector output from the fully connected layer 222. In the example of Fig. 4, the output layer 23 has five classes of damage type, three classes of damage progression degree, and neurons in numbers corresponding to damage progression time.

[0038] Synapses are laid between each layer of the neural network model 20, connecting the neurons between the layers, and weights are assigned to each synapse in the convolution layer 220 and the fully connected layer 222 of the intermediate layer 22.

[0039] The machine learning unit 401 inputs the learning data to the neural network model 20, and causes the neural network model 20 to machine-learn the correlation between the input data (learning images 30) and the output data (damage type, damage progression level, and damage progression time). Specifically, the machine learning unit 401 uses the learning images 30 constituting the learning data as input data, and The learning images 30 are input to the input layer 21 of the network model 20. Note that the machine learning unit 401 may perform predetermined image adjustments (e.g., image format, image size, image filter, image mask, etc.) on the learning images 30 as preprocessing before inputting the learning images 30 to the input layer 21.

[0040] The machine learning unit 401 iteratively adjusts the weights associated with each synapse (backpropagation) so that the evaluation value of the error function is reduced, using an error function that compares the output data (determination results of the damage type, damage progress level, and damage progress time) output from the output layer 23 as an inference result with the output data (correct labels of the damage type, damage progress level, and damage progress time) that constitutes the learning data. Then, when the machine learning unit 401 determines that a predetermined learning termination condition, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than an allowable value, is satisfied, the machine learning is terminated, and the neural network model 20 at that time (a weight parameter group consisting of all the weights associated with each synapse) is stored in the trained model storage unit 43 as the trained learning model 2. In addition, when performing machine learning of the learning model 2, the machine learning unit 401 may adopt, for example, online learning, batch learning, mini-batch learning, etc. as a method for adjusting weights, and may adopt, for example, a hold-out method, cross-validation, etc. as a method for dividing multiple sets of learning data into training data and test data and evaluating the learning model 2, and may determine that the misclassification rate is minimized as a predetermined learning termination condition.

[0041] (Structure condition determination device 5) 5 is a block diagram showing an example of a structure state determination device 5 according to the first embodiment. The structure state determination device 5 includes a control unit 50, a communication unit 51, and a trained model storage unit 52.

[0042] The control unit 50 functions as a judgment data acquisition unit 500, an inference unit 501, and an output processing unit 502. The communication unit 51 is connected to external devices (e.g., the imaging device 3, the machine learning device 4, and the terminal device 6) via a network 7, and functions as a communication interface for transmitting and receiving various data.

[0043] The determination data acquisition unit 500 is connected to an external device via the communication unit 51 and the network 7, and acquires determination data including a determination image 31 in which at least a part of the structure 10 is captured.

[0044] The inference unit 501 inputs the judgment data acquired by the judgment data acquisition unit 500 into the learning model 2, and performs an inference process (see Figure 6 described later) to infer the damage type, damage progression degree, and damage progression time of the damaged part contained in the judgment image 31 as the damage state of the structure 10.

[0045] The trained model storage unit 52 is a database that stores trained learning models 2 used in the inference process of the inference unit 501. The number of trained models 2 stored in the trained model storage unit 52 is not limited to one, and for example, a plurality of trained models with different conditions, such as machine learning methods and types of exterior materials or coating materials forming the structure 10, may be stored and selectively usable. The trained model storage unit 52 may be substituted by a storage unit of an external computer (for example, a server-type computer or a cloud-type computer), and in that case, the inference unit 501 may perform the above-mentioned inference process by accessing the external computer.

[0046] The output processing unit 502 performs an output process for outputting the judgment result of the damage state of the structure 10 inferred by the inference unit 501. As a specific output means for outputting the judgment result, various means can be adopted. For example, the output processing unit 502 outputs the judgment result to a terminal device The determination result may be transmitted to the structure state determination device 5, and further displayed on a screen, or the determination result may be stored in a storage unit of the structure state determination device 5.

[0047] FIG. 6 is a functional explanatory diagram showing an example of an inference process by the inference unit 501 according to the first embodiment.

[0048] The judgment image 31 included in the judgment data is an image of at least a part of the structure 10 captured by the imaging device 3. The judgment image 31 corresponds to the learning image 30 when the machine learning device 4 trains the learning model 2 by machine learning.

[0049] The learning model 2 here is a learning model 2 in which a weight parameter group has been adjusted (trained) by machine learning the correlation between the learning image 30 and the damage type, damage progression level, and damage progression time of the damaged portion included in the learning image 30 using multiple sets of learning data in the machine learning device 4. Therefore, the inference unit 501 inputs the judgment image 31 included in the judgment data to the learning model 2, and infers the damage type, damage progression level, and damage progression time of the damaged portion included in the judgment image 31.

[0050] At this time, the damage type is output as a score (confidence) for each class, and when the damage type classification is expressed as five classes (no damage, irregular crack, parallel crack, lattice crack, and peeling), the scores for each class are output, for example, as "0.02", "0.10", "0.15", "0.95", and "0.12". Any method may be adopted for using the scores, and as shown in Fig. 6, the class with the highest score (in the above example, the lattice crack with a score of "0.95") may be used as the judgment result, or when the score of a predetermined class exceeds a predetermined score reference value (in the above example, the score of the lattice crack class, "0.95", exceeds the score reference value "0.80"), the class may be used as the judgment result.

[0051] The damage progression is output as a score (confidence) for each class, and when the damage progression is classified into three classes (mild, moderate, and severe), the scores for each class are output, for example, as "0.92", "0.14", and "0.08". Any method may be adopted for using the scores, and as shown in FIG. 6, the class with the highest score (in the above example, mild with a score of "0.92") may be used as the judgment result, or when the score of a predetermined class exceeds a predetermined score reference value (in the above example, the score of the mild class, "0.92", exceeds the score reference value of "0.75"), the class may be used as the judgment result.

[0052] The damage progression time is output as a numerical value in a specified unit (days, weeks, months, or years). If the damage type determination result is "lattice crack" and the damage progression determination result is "moderate," the time required for the lattice crack to progress from "moderate" to "severe" is output as "6 (months)" (in this embodiment, in months).

[0053] In addition, it is preferable that the damage state judgment results for the judgment image 31 are stored in the trained model memory unit 52 or other memory device (not shown), and past judgment results may be used as learning data to be used for online learning or re-learning, for example, to further improve the inference accuracy of the trained learning model 2.

[0054] (Configuration of computer 900) FIG. 7 is a hardware configuration diagram showing an example of a computer 900 constituting each of the devices 3 to 6 of the structure management system 1. As shown in FIG.

[0055] Each of the devices 3 to 6 of the structure management system 1 is configured by a general-purpose or dedicated computer 900. The computer 900 includes, as its main components, a bus 910, a processor 912, a memory 914, an input device 916, an output device 917, a display device 918, a storage device 920, a communication I / F (interface) unit 922, an external device I / F unit 924, an I / O (input / output) device I / F unit 926, and a media input / output unit 928. Note that the above components may be omitted as appropriate depending on the application of the computer 900.

[0056] The processor 912 is composed of one or more arithmetic processing devices (such as a central processing unit (CPU), a micro-processing unit (MPU), a digital signal processor (DSP), or a graphics processing unit (GPU)) and operates as a control unit that controls the entire computer 900. The memory 914 stores various data and programs 930, and is composed of, for example, a volatile memory (such as a DRAM or SRAM) that functions as a main memory, and a non-volatile memory (such as a ROM or flash memory).

[0057] The input device 916 is composed of, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc., and functions as an input unit. The output device 917 is composed of, for example, a sound (audio) output device, a vibration device, etc., and functions as an output unit. The display device 918 is composed of, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc., and functions as an output unit. The input device 916 and the display device 918 may be integrally configured, such as a touch panel display. The storage device 920 is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), etc., and functions as a storage unit. The storage device 920 stores various data necessary for the execution of the operating system and the program 930.

[0058] The communication I / F unit 922 is connected to a network 940 (which may be the same as the network 7 in FIG. 1) such as the Internet or an intranet by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from other computers according to a predetermined communication standard. The external device I / F unit 924 is connected to an external device 950 such as a camera, a printer, a scanner, a reader / writer by wire or wirelessly, and functions as a communication unit that transmits and receives data to and from the external device 950 according to a predetermined communication standard. The I / O device I / F unit 926 is connected to an I / O device 960 such as various sensors and actuators, and functions as a communication unit that transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, between the I / O device 960. The media input / output unit 928 is composed of a drive device such as a DVD (Digital Versatile Disc) drive or a CD (Compact Disc) drive, and reads and writes data from and to a medium (non-temporary storage medium) 970 such as a DVD or CD.

[0059] In the computer 900 having the above configuration, the processor 912 calls up a program 930 stored in the storage device 920 into the memory 914, executes the program, and controls each unit of the computer 900 via the bus 910. The program 930 may be stored in the memory 914 instead of the storage device 920. The program 930 may be recorded in the medium 970 in an installable file format or an executable file format, and provided to the computer 900 via the media input / output unit 928. The program 930 may be provided to the computer 900 by downloading it via the network 940 through the communication I / F unit 922. In addition, the computer 900 may implement various functions, which are realized by the processor 912 executing the program 930, using hardware such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). It may also be realized by software.

[0060] The computer 900 is, for example, a stationary computer or a portable computer, and is an electronic device of any type. The computer 900 may be a client computer or an edge computer, or may be a server computer or a cloud computer, depending on the use of the computer 900.

[0061] (Machine learning methods) FIG. 8 is a flowchart showing an example of a machine learning method performed by the machine learning device 4 according to the first embodiment.

[0062] First, in step S100, the learning data acquisition unit 400 prepares a desired number of pieces of learning data as a preliminary preparation for starting machine learning, and stores the prepared learning data in the learning data storage unit 42. The number of pieces of learning data to be prepared here may be set in consideration of the inference accuracy required for the learning model 2 to be finally obtained.

[0063] Several methods can be adopted for preparing the learning data. For example, an operator uses the imaging device 3 to capture the learning image 30 at an arbitrary imaging location of the structure 10, and uses the terminal device 6 to record the imaging date and time and the imaging location, and if damage 100 occurs, records the damage type and damage progression of the damage 100. Then, when the operator captures a new learning image 30 at the same imaging location, if the operator determines that the damage progression of the damage 100 has progressed to the next stage, the operator prepares learning data by associating the damage progression time (e.g., the elapsed time from the previous imaging date and time to the current imaging date and time) with the previously captured learning image 30. Then, by repeating such operations, it is possible to prepare multiple sets of learning data. As another method, for example, it is also possible to obtain learning data by performing an acceleration test in which damage 100 is intentionally generated in a test specimen of the structure 10. Furthermore, as the learning data, multiple sets of learning data are prepared, which are composed of input data and output data when no damage 100 has occurred, i.e., when the structure 10 is in a normal state (for example, the damage type of the output data in this case is “no damage”), as well as when damage 100 has occurred.

[0064] Next, in step S110, the machine learning unit 401 prepares a pre-learning learning model 2 to start machine learning. The pre-learning learning model 2 prepared here employs the convolutional neural network illustrated in FIG. 4, and the weights of each synapse are set to an initial value. Each neuron in the input layer 21 is associated with each pixel of the learning image 30 as input data constituting the learning data. Each neuron in the output layer 23 is associated with a damage type, a damage progression level, and a damage progression time as output data constituting the learning data.

[0065] Next, in step S120, the machine learning unit 401 acquires, for example, one set of learning data randomly from the multiple sets of learning data stored in the learning data storage unit .

[0066] Next, in step S130, the machine learning unit 401 inputs the input data included in the set of learning data to the input layer of the prepared learning model 2 before learning (or during learning). As a result, output data is output as an inference result from the output layer of the learning model 2, but the output data is generated by the learning model 2 before learning (or during learning). Therefore, in the state before learning (or during learning), the output data output as an inference result indicates information different from the output data (correct answer label) included in the learning data.

[0067] Next, in step S140, the machine learning unit 401 The output data (correct label) included in the obtained set of learning data is compared with the output data output from the output layer as an inference result in step S130, and the weight of each synapse is adjusted, thereby performing machine learning. In this way, the machine learning unit 401 makes the learning model 2 learn the correlation between the input data and the output data.

[0068] Next, in step S150, the machine learning unit 401 determines whether a predetermined learning termination condition has been satisfied, for example, based on an evaluation value of an error function based on the inference result and the output data (correct label) included in the learning data, or the remaining number of unlearned learning data stored in the learning data storage unit 42.

[0069] In step S150, if the machine learning unit 401 determines that the learning end condition is not satisfied and machine learning is to be continued (No in step S150), the process returns to step S120, and the process of steps S120 to S140 is performed multiple times on the learning model 2 being trained using untrained learning data. On the other hand, in step S150, if the machine learning unit 401 determines that the learning end condition is satisfied and machine learning is to be terminated (Yes in step S150), the process proceeds to step S160.

[0070] Then, in step S160, the machine learning unit 401 stores the trained learning model 2 (adjusted weight parameter group) trained by adjusting the weights associated with each synapse in the trained model storage unit 43, and ends the series of machine learning methods shown in FIG. 8. In the machine learning method, step S100 corresponds to a training data storage step, steps S110 to S150 correspond to machine learning steps, and step S160 corresponds to a trained model storage step. Note that, in the series of machine learning methods shown in FIG. 8, the case where online learning is adopted as a method for adjusting the weights has been described, but batch learning or mini-batch learning (for example, 100 sets of training data units) may be adopted. In addition, when multiple sets of training data are divided into training data and test data, the series of machine learning methods shown in FIG. 8 may be executed using the training data. Furthermore, it may be determined whether a predetermined learning end condition is satisfied based on an erroneous determination rate.

[0071] As described above, the machine learning device 4 and machine learning method of this embodiment can provide a learning model 2 that can infer (determine) the damage state of a structure 10 with high accuracy from a learning image 30 in which at least a portion of the structure 10 is captured by an imaging device 3.

[0072] (Method of determining the condition of a structure) Fig. 9 is a flowchart showing an example of a structure state determination method by the structure state determination device 5 according to the first embodiment. Note that the series of structure state determination methods shown in Fig. 9 are repeatedly executed at a predetermined timing by the structure state determination device 5. The predetermined timing may be any timing, and may be, for example, when the imaging device 3 captures a new determination image 31, or when a predetermined event occurs (when a determination operation by a manager or worker is accepted from the terminal device 6, when a determination command is received from the structure management system 1, etc.).

[0073] First, in step S200, the structure 10 is imaged by the imaging device 3, and the determination image 31 is transmitted to the structure state determination device 5, whereby the determination data acquisition unit 500 acquires determination data including the determination image 31.

[0074] Next, in step S210, the inference unit 501 performs a predetermined preprocessing on the judgment data and inputs it to the input layer of the trained learning model 2, and obtains the scores for each class of damage type, the scores for each class of damage progression, and the numerical values ​​of damage progression time output as inference results from the output layer of the learning model 2.

[0075] Next, in step S220, as an example of post-processing of supervised learning, the inference unit 501 determines the damage type, damage progression and damage progression time of the damaged part contained in the judgment image 31 by, for example, determining the highest class among the scores for each damage type class as the damage type and determining the highest class among the scores for each damage progression class as the damage progression.

[0076] Next, in step S230, the output processing unit 502 determines whether the result of the damage type determination by the inference unit 501 is “no damage” or not. If it determines that it is “no damage”, the process proceeds to step S240, and if it does not determine that it is “no damage”, the process proceeds to step S250.

[0077] Then, in step S240, the output processing unit 502 outputs information indicating “no damage.” Note that step S240 may be omitted.

[0078] Meanwhile, in step S250, the output processing unit 502 outputs information indicating the damage type, the damage progression level, and the damage progression time as information indicating "damaged", and ends the series of structure state determination methods shown in Fig. 9. In the structure state determination method, step S200 corresponds to a determination data acquisition step, steps S210 to S220 correspond to an inference step, and steps S230 to S250 correspond to an output processing step.

[0079] As described above, according to the structure condition determination device 5 and structure condition determination method of this embodiment, determination data including the determination image 31 in which at least a portion of the structure 10 is captured is input to the learning model 2, and the damage type, damage progression, and damage progression time are inferred as the damage state of the structure 10 captured in the determination image 31, making it easier to carry out inspection and maintenance work according to the damage progression.

[0080] Second embodiment Fig. 10 is a block diagram showing an example of a machine learning device 4 according to the second embodiment. Fig. 11 is a schematic diagram showing an example of a neural network model 20a applied to a learning model 2a according to the second embodiment. Fig. 12 is a block diagram showing an example of a structure state determination device 5 according to the second embodiment. Fig. 13 is a functional explanatory diagram showing an example of an inference process by an inference unit 501a according to the second embodiment.

[0081] A learning model 2a used in a machine learning device 4 and a structure state determination device 5 according to the second embodiment is obtained by changing the contents of input data with respect to the learning model 2 according to the first embodiment. Specifically, in the learning model 2a according to the second embodiment, the input data constituting the learning data further includes learning environment information 80 indicating the environment of the imaging location where the learning image 30 was captured, and accordingly, the determination data further includes determination environment information 81 indicating the environment of the imaging location where the determination image 31 was captured. Since the basic configuration and operation of the machine learning device 4 and the structure state determination device 5 according to the second embodiment are similar to those of the first embodiment, the following description will focus on the differences from the first embodiment.

[0082] The learning environment information 80 and the judgment environment information 81 are information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, acidity value of acid rain, amount of substances contained in acid rain, amount of airborne salt, and amount of radiation. The learning environment information 80 and the judgment environment information 81 are preferably calculated as average values ​​or cumulative values ​​for a predetermined observation period, for example.

[0083] The learning environment information 80 and the judgment environment information 81 are observation data observed by various environmental sensors provided in the environment observation device 8A. The various environmental sensors may be installed in the structure 10, or may be installed in each predetermined area and observed as representative values ​​representing each area. The learning environment information 80 and the judgment environment information 81 may be provided data provided by an environmental information providing device 8B operated by the Japan Meteorological Agency, an environmental information provider, or the like, and may be provided data corresponding to the location or region of the structure 10.

[0084] The learning data acquisition unit 400a of the machine learning device 4 acquires learning data consisting of input data including learning images 30 and learning environment information 80, and output data including the damage type, damage progression, and damage progression time of the damaged parts included in the learning images 30.

[0085] The machine learning unit 401a inputs multiple sets of learning data into the learning model 2a, and causes the learning model 2a to machine-learn the correlation between the input data (learning images 30 and learning environment information 80) that constitute the learning data and the output data (damage type, damage progression, and damage progression time).

[0086] In the neural network model 20a (see FIG. 11) applied to the learning model 2a, the learning image 30 is input to the input layer 21, and the learning environment information 80 is input to the fully connected layer 222. Therefore, the fully connected layer 222 further has at least one neuron to which the learning environment information 80 (determination environment information 81) is input. Note that the learning environment information 80 (determination environment information 81) may be a part or all of the information exemplified above, or may further include other information, and the number of neurons in the fully connected layer 222 is set according to the number of pieces of learning environment information 80 (determination environment information 81). In addition, when the learning environment information 80 (determination environment information 81) is input to the fully connected layer 222, the value may be input as it is, or a normalized value may be input, or, for example, a categorical variable or one-hot encoding may be introduced and the converted value may be input.

[0087] The judgment data acquisition unit 500a of the structure state judgment device 5 acquires judgment data including the judgment image 31 and the judgment environment information 81.

[0088] The inference unit 501a inputs the judgment data (the judgment image 31 and the judgment environment information 81) acquired by the judgment data acquisition unit 500a to the learning model 2a, and thereby infers the damage type, damage progression degree, and damage progression time of the damaged portion included in the judgment image 31 as the damage state of the structure 10. At that time, the judgment image 31 is input to the input layer 21, and the judgment environment information 81 is input to the fully connected layer 222.

[0089] As described above, according to this embodiment, by adding learning environment information 80 and judgment environment information 81 to the learning image 30 and judgment image 31, respectively, and performing machine learning of the learning model 2a, the influence of the learning environment information 80 and the judgment environment information 81 on the damage state of the structure 10 is taken into account, thereby improving the judgment accuracy of the learning model 2a.

[0090] (Third embodiment) Fig. 14 is a block diagram showing an example of a machine learning device 4 according to the third embodiment. Fig. 15 is a schematic diagram showing an example of a neural network model 20b applied to a learning model 2b according to the third embodiment. Fig. 16 is a block diagram showing an example of a structure state determination device 5 according to the third embodiment. Fig. 17 is a functional explanatory diagram showing an example of an inference process by an inference unit 501b according to the third embodiment.

[0091] The learning model 2b used in the machine learning device 4 and structure state determination device 5 according to the third embodiment is obtained by changing the contents of the output data of the learning model 2 according to the first embodiment. Specifically, in the learning model 2b according to the third embodiment, the output data constituting the learning data includes a plurality of damage progression times indicating the times required for the damage progression level to progress to each of the following stages. The basic configuration and operation of the structure state determination device 5 are similar to those of the first embodiment, so the following description will focus on the differences from the first embodiment.

[0092] The learning data acquisition unit 400b of the machine learning device 4 acquires learning data consisting of input data including a learning image 30 and output data including the damage type, damage progression degree, and multiple damage progression times of the damaged portion included in the learning image 30.

[0093] The machine learning unit 401b inputs multiple sets of learning data into the learning model 2b, and causes the learning model 2b to machine-learn the correlation between the input data (learning images 30) that constitute the learning data and the output data (damage type, damage progression level, and multiple damage progression times).

[0094] At that time, in the neural network model 20b (see FIG. 15) applied to the learning model 2b, the multiple damage progression times include, for example, the damage progression time required for the damage progression level to progress from "start" to "mild", the damage progression time required for the damage progression level to progress from "mild" to "moderate", and the damage progression time required for the damage progression level to progress from "moderate" to "severe", and are output from each neuron of the output layer 23. Therefore, the output layer 23 has neurons in numbers corresponding to five classes of damage types, three classes of damage progression levels, and three damage progression times.

[0095] Fig. 18 is a graph showing the relationship between the damage progression and the damage progression time for each damage type. In Fig. 18, the horizontal axis shows time and the vertical axis shows the damage progression for four classes of damage types: "irregular crack", "parallel crack", "lattice crack" and "peeling". The three damage progression times as the progression progresses are illustrated in the time degradation graph.

[0096] As described above, the learning data is composed of input data including the learning image 30 and output data including the damage type, damage progression degree, and multiple damage progression times of the damaged parts included in the learning image 30, so the machine learning unit 401b trains the learning model 2b to machine learn the above-mentioned time-dependent deterioration graphs for each damage type.

[0097] The inference unit 501b of the structure state judgment device 5 inputs the judgment data (judgment image 31) acquired by the judgment data acquisition unit 500 into the learning model 2b, and infers the damage type, damage progression degree, and multiple damage progression times of the damaged portion contained in the judgment image 31 as the damage state of the structure 10.

[0098] As described above, according to this embodiment, by performing machine learning of the learning model 2b using learning data including multiple damage progression times as output data, the damage type, damage progression level, and multiple damage progression times are inferred as the damage state of the structure 10 captured in the assessment image 31, so that the situation in which the damage progression level progresses can be determined in more detail.

[0099] (Other embodiments) The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention, all of which are included in the technical concept of the present invention.

[0100] For example, the second embodiment and the third embodiment may be combined, and the output data constituting the learning data according to the second embodiment may include a plurality of damage progression times as in the third embodiment. In this case, the machine learning unit of the machine learning device 4 machine-learns the correlation between the input data (learning images 30 and learning environment information 80) and the output data (damage type, damage progression level, and a plurality of damage progression times) into a learning model. Also, the inference unit of the structure state determination device 5 uses the determination data (determination images 31 and determination environment information 81) as the learning model. By inputting the data into the model, the damage type, damage progression level, and multiple damage progression times of the damaged portion included in the assessment image 31 are inferred as the damage state of the structure 10.

[0101] In the above embodiment, the learning models 2, 2a, and 2b are configured by neural network models (including deep learning) 20, 20a, and 20b that treat the determination of the damage state of the structure 10 as a classification problem as a learning method of machine learning. However, the learning models may be configured by neural network models that treat it as a regression problem. Furthermore, as long as the correlation between the input data and the output data in the above embodiment is machine-learned from the learning data, other learning methods may be adopted without being limited to the above example. For example, the learning model may use a recurrent neural network (RNN) or ensemble learning. In addition, when treating it as a regression problem, the learning model may use a statistical learning method, for example, a statistical model such as an autoregressive integrated moving average (ARIMA) model or Bayesian estimation.

[0102] Furthermore, each unit of the control unit 40 of the machine learning device 4 or the control unit 50 of the structure state determination device 5 in the above embodiment may be realized by having the processor 912 of the computer 900 shown in Figure 7 execute a program. Cut.

[0103] Also, instead of the aspect of the structure state determination device 5 according to the above embodiment, the inference device may be provided in the form of an inference device (inference method or inference program) used to determine the damage state of the structure 10. In this case, the inference device includes a memory and a processor, and the processor executes a series of processes. The series of processes includes a data acquisition process for acquiring determination data including the determination image 31, and an inference process for inferring the damage type, damage progression level, and damage progression time of the damaged portion included in the determination image 31 as the damage state of the structure 10 captured in the determination image 31 when the determination data is acquired in the data acquisition process. By providing the inference device (inference method or inference program) in the form of the above, it becomes possible to apply the inference device to various devices more easily than when the structure state determination device 5 is implemented. It should be obvious to those skilled in the art that when an inference device (inference method or inference program) determines the damage state of a structure 10, it may apply an inference method implemented by the inference units 501, 501a, 501b of the structure state determination device 5 using the trained learning models 2, 2a, 2b generated by the machine learning device 4 according to the above embodiment. [Explanation of symbols]

[0104] 1... structure management system, 2, 2a, 2b... learning model, 3... imaging device, 4...machine learning device, 5...structure condition determination device, 6...terminal device, 7...network, 8A...environmental observation device, 8B...environmental information providing device, 10...structure, 20, 20a, 20b…Neural network model, 21...input layer, 22...intermediate layer, 23...output layer, 30...Learning image, 31...Judgment image, 40: control unit, 41: communication unit, 42: learning data storage unit, 43: learned model storage unit, 50: control unit, 51: communication unit, 52: learned model storage unit, 80...Learning environment information, 81...Judgment environment information, 100...Damage, 220...convolutional layer, 221...pooling layer, 222...fully connected layer, 400, 400a, 400b...learning data acquisition unit, 401, 401a, 401b...Machine Learning Department, 500, 500a...judgment data acquisition unit, 501, 501a, 501b...inference unit, 502: output processing unit, 900: computer

Claims

1. A structure condition determination device for determining a damage condition of a structure, comprising: a determination data acquisition unit that acquires determination data including a determination image in which at least a portion of the structure is captured; a trained model storage unit that stores a trained model that has been machine-learned to obtain a correlation between input data including a training image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the training image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; and an inference unit that inputs the judgment data acquired by the judgment data acquisition unit into the learning model and infers the damage type, the damage progression level, and the damage progression time as the damage state of the structure captured in the judgment image, Structural condition determination device.

2. The output data is The damage type, the damage progression level, and a plurality of damage progression times indicating times required for the damage progression level to progress to each of a plurality of subsequent stages, The inference unit is Inferring the damage type, the damage progression degree, and a plurality of damage progression times as the damage state; The structure state determination device according to claim 1 .

3. The determination data is Further, the determination-use environment information indicates an environment of an imaging location where the determination-use image is captured. The input data is Further, learning environment information indicating an environment of an imaging location where the learning image was captured is included. The structure state determination device according to claim 1 or 2.

4. The judgment environment information and the learning environment information are The information is information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, the acidity value of acid rain, the amount of substances contained in the acid rain, the amount of airborne salt, and the amount of radiation. The structure state determination device according to claim 3 .

5. A machine learning device that generates a learning model used in a structure condition determination device that determines a damage state of a structure, a learning data storage unit that stores a plurality of sets of learning data, each set including input data including a learning image in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; a machine learning unit that inputs a plurality of sets of the learning data into the learning model to machine-learn the correlation between the input data and the output data; and a trained model storage unit that stores the trained model trained by the machine learning unit. Machine learning device.

6. The output data is The damage type, the damage progression level, and a plurality of damage progression times indicating each time required for the damage progression level to progress to each of the next or subsequent stages, The structure state determination device according to claim 5 .

7. The input data is Further, learning environment information indicating an environment of an imaging location where the learning image was captured is included. The machine learning device according to claim 5 or 6.

8. The learning environment information is The information is information on at least one of the amount of ultraviolet rays, temperature, humidity, wind speed, ocean waves, lightning, climate, the acidity value of acid rain, the amount of substances contained in the acid rain, the amount of airborne salt, and the amount of radiation. The machine learning device according to claim 7.

9. An inference device used to determine a damage state of a structure, comprising: The inference device comprises: A memory and a processor, The processor, A data acquisition process for acquiring determination data including a determination image in which at least a portion of the structure is captured; When the determination data is acquired in the data acquisition process, an inference process is executed to infer, as the damage state of the structure captured in the determination image, a damage type of the damaged portion included in the determination image, a damage progression indicating the progression of the damaged portion when the progression of the damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage. Reasoning device.

10. A method for determining a damage state of a structure, comprising: a determination data acquisition step of acquiring determination data including a determination image in which at least a portion of the structure is captured; an inference process for inputting the judgment data acquired by the judgment data acquisition process into a learning model that has been machine-learned to learn a correlation between input data including a learning image in which at least a part of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage, and inferring the damage type, the damage progression, and the damage progression time as the damage state of the structure captured in the judgment image. Method for determining the condition of a structure.

11. A machine learning method for generating a learning model used in a structure condition determination method for determining a damage state of a structure, comprising: a learning data storage process for storing in a learning data storage unit a plurality of sets of learning data, the learning data being composed of input data including learning images in which at least a portion of the structure is captured, and output data including a damage type of a damaged portion included in the learning image, a damage progression indicating the progression of the damaged portion when the progression of damage is classified into a plurality of stages, and a damage progression time indicating the time required for the damage progression to progress to the next stage; a machine learning process for inputting a plurality of sets of the learning data into the learning model, thereby making the learning model learn the correlation between the input data and the output data by machine learning; and a learned model storage step of storing the learned model machine-learned by the machine learning step in a learned model storage unit. Machine learning methods.

Citation Information

Patent Citations

  • Method for evaluating lifetime

    JP2009175110A

  • Method for estimating corrosion and deterioration process of reinforced concrete structure

    JP2012008152A

  • Deterioration status identification device, deterioration status identification method and program

    JP2019157535A

  • Information processing device and information processing method

    WO2017110278A1