Condition definition apparatus, condition definition method, and condition definition program

Abstract

To contribute to machine learning for defining a condition of an inspection object, such as a damage, and to define a condition of an inspection object, such as a damage, using learning results of this machine learning.SOLUTION: A condition definition apparatus includes: accepting first inspection object data indicating an image of a first inspection object, summary text data explaining a summary of a condition of the inspection object in text, and detailed text data explaining details of the condition of the inspection object in text; generating an image feature vector indicating features of the image indicated by the first inspection object data from the accepted first inspection object data, generating a summary feature vector indicating summary of features of the condition of the inspection object indicated by the summary text data from the accepted summary text data; generating an integrated feature vector by integrating the generated summary feature vector and the detailed feature vector; and generating inspection object class data used to define the condition of the inspection object from the integrated feature vector.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a state definition device, a state definition method, and a state definition program. [Background technology]

[0002] In transportation infrastructure such as bridges and tunnels (hereinafter, "infrastructure" will be abbreviated as "infrastructure"), early detection of damage such as cracks and corrosion is important for long-term maintenance. However, visual inspection can lead to oversights due to differences in the ability of inspectors. Patent Document 1 discloses a method for quantitatively detecting damage from images captured by a camera.

[0003] Another method known as CLIP (Contrastive Language-Image Pre-training) is to learn images and summaries that describe them, and then classify the object classes in the images. It is known that slight differences in the summaries can have a significant impact on classification accuracy in CLIP. For example, if an image of a dog is given the summaries "a photo of a dog" and "an image of a dog," the image classification results will be significantly different, even if the summaries have essentially the same meaning. Furthermore, a method known as DualCoOp (Dual Context Optimization) is also known for such applications. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-168548 Summary of the Invention [Problem to be solved by the invention]

[0005] The disclosures of the above prior art documents are incorporated herein by reference.The following analysis was conducted by the inventors.

[0006] However, the method disclosed in Patent Document 1 defines damage by processing images (damage images) of damage such as cracks and corrosion. However, such methods do not necessarily define damage correctly because they do not use definitions of damage relevant to actual inspections when processing images. Furthermore, CLIP and DualCoOp are sensitive to slight differences in the summary text attached to the image, and even if a summary text with substantially the same meaning is attached to the same damage, the definition of damage may change significantly. Therefore, these methods are not suitable for classifying images to define damage.

[0007] In view of the above-mentioned problems, the object of the present invention is to contribute to machine learning for defining the state of damage, etc. of an inspection object, and to defining the state of damage, etc. of an inspection object using the learning results of this machine learning. [Means for solving the problem]

[0008] In a first aspect of the present invention, there is provided a state definition device for defining a state of an inspection target, comprising one or more processors. The one or more processors are configured to accept first inspection target data showing an image of a first inspection target, summary sentence data describing in sentences an overview of the state of the inspection target, and detailed sentence data describing in sentences the details of the state of the inspection target, generate an image feature vector from the accepted first inspection target data showing features of the image shown by the first inspection target data, generate a summary feature vector from the accepted summary sentence data showing features of the overview of the state of the inspection target shown by the summary sentence data, generate a detail feature vector from the accepted detailed sentence data showing features of the details of the state of the inspection target shown by the detailed sentence data, integrate the generated summary feature vector and detail feature vector to generate an integrated feature vector, and generate inspection target class data from the integrated feature vector to be used in defining the state of the inspection target.

[0009] In a second aspect of the present invention, there is provided a state definition method including the steps of: accepting first inspection object data showing an image of a first inspection object, summary sentence data describing in sentences an overview of the state of the inspection object, and detailed sentence data describing in sentences the details of the state of the inspection object; generating an image feature vector from the accepted first inspection object data that shows the features of the image shown by the first inspection object data; generating a summary feature vector from the accepted summary sentence data that shows the features of the overview of the state of the inspection object that is indicated by the summary sentence data; generating a detail feature vector from the accepted detailed sentence data that shows the features of the details of the state of the inspection object that is indicated by the detailed sentence data; integrating the generated summary feature vector and detailed feature vector to generate an integrated feature vector; and generating inspection object class data from the integrated feature vector that is used to define the state of the inspection object.

[0010] In a third aspect of the present invention, there is provided a state definition program that causes a computer to execute the following processes: a process for accepting first inspection object data showing an image of a first inspection object, summary sentence data that provides a sentence summary of the state of the inspection object, and detailed sentence data that provides a sentence summary of the state of the inspection object; a process for generating, from the accepted first inspection object data, an image feature vector that indicates the characteristics of the image shown by the first inspection object data; a process for generating, from the accepted summary sentence data, a summary feature vector that indicates the characteristics of the summary of the state of the inspection object that is indicated by the summary sentence data; a process for generating, from the accepted detailed sentence data, a detailed feature vector that indicates the characteristics of the details of the state of the inspection object that are indicated by the detailed sentence data; a process for integrating the generated summary feature vector and detailed feature vector to generate an integrated feature vector; and a process for generating inspection object class data from the integrated feature vector that is used to define the state of the inspection object. The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, etc. The present invention can be embodied as a computer program product. [Effects of the Invention]

[0011] Each aspect of the present invention can contribute to machine learning for defining the state of an inspection object, such as damage, and to defining the state of an inspection object, such as damage, using the learning results of this machine learning. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example configuration of a state definition device according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device that executes the program when the state definition device shown in FIG. 1 is realized as a program. [Figure 3] FIG. 3 is a flowchart illustrating an example of a machine learning operation of the state definition device shown in FIG. [Figure 4] FIG. 4 is a flowchart illustrating an example of an operation for outputting input damage image data in association with damage class data. [Figure 5] FIG. 5 is a diagram illustrating another configuration example of a state definition device according to the present disclosure. [Figure 6] FIG. 6 is a flowchart illustrating another example of the machine learning operation of the state definition device shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] [First embodiment] A first embodiment of the present disclosure will be described below with reference to the drawings. However, the present disclosure is not limited to the embodiment described below. Furthermore, each drawing is schematic, and the same or corresponding elements, processes, and communications are appropriately designated by the same reference numerals.

[0014] 1 is a diagram illustrating an example configuration of a state definition device 10 according to the present disclosure. As shown in FIG. 1, the state definition device 10 includes an image feature extraction unit 100, a general feature extraction unit 102, a detailed feature extraction unit 104, an integrated feature vector generation unit 106, a similarity data generation unit 108, a machine learning unit 110, a damage class database (damage class DB (Data Base)) 112, and an image / class display unit 114.

[0015] All or some of the components of the state definition device 10 may be realized by dedicated hardware, or by a combination of hardware and software such as a program running on a computer's operating system (OS), or by software running on an information processing device's OS.

[0016] For the sake of specificity and clarity of the following description, a specific example will be given in which the state definition device 10 defines damage that has occurred to an inspection object, such as a concrete wall or bolt included in transportation infrastructure such as a bridge or tunnel. However, in addition to defining such damage, the state definition device 10 can be applied to defining various events, such as defining states other than damage to an inspection object, defining meteorological phenomena, defining engineering and scientific events, defining biological events, and defining geological events.

[0017] FIG. 2 illustrates an example hardware configuration of an information processing device (computer) 12 that executes the state definition device 10 shown in FIG. 1 when the state definition device 10 is implemented as a program. When all of the components of the state definition device 10 are implemented in software as a program running on the OS of the information processing device, such a program can be executed, for example, by the information processing device 12 illustrated in FIG. 2. However, the hardware configuration of the information processing device 12 illustrated in FIG. 2 is merely an example and does not limit the hardware configuration of the information processing device 12. Furthermore, the information processing device 12 may further include components not illustrated in FIG. 2. As shown in FIG. 2, the information processing device 12 includes a CPU (Central Processing Unit; processor) 120, a main memory device 122, an auxiliary memory device 124, a network interface (network IF (Interface)) 126, and an input / output interface (input / output IF) 128, which are connected to each other via a bus, wiring, etc. so as to be able to input and output data.

[0018] In the information processing device 12, the CPU 120 executes instructions contained in a program for realizing the functions of each component of the state definition device 10. The main memory device 122 includes storage elements such as RAM (Random Access Memory) and ROM (Read Only Memory). The main memory device 122 stores in its storage elements the program including instructions to be executed by the CPU 120, as well as data required for executing the program.

[0019] The network interface 126 has a function for connecting the CPU 120 of the information processing device 12 to a network 130 such as a local area network (LAN), a wide area network (WAN), or a mobile wireless system such as 5G (5th Generation) so as to enable data communication. The CPU 120 can mutually communicate data with other information processing devices (not shown) connected to the network 130 via the network interface 126 and the network 130.

[0020] The auxiliary storage device 124 includes a nonvolatile storage device (not shown), such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The auxiliary storage device 124 stores programs for realizing the functions of the components of the state definition device 10, data required for the execution of the programs, and the like, in the nonvolatile storage device for the medium to long term. The auxiliary storage device 124 may include a connector, such as a universal serial bus (USB) connector, to which a nonvolatile memory and a cable can be connected.

[0021] A program for realizing the functions of each component of the state definition device 10 may be provided to the information processing device 12 from another information processing device via the network 130 and the network interface 126. Alternatively, this program may be provided to the information processing device 12 from another information processing device connected by cable via a connector of the auxiliary storage device 124. Alternatively, this program may be provided to the information processing device 12 as a program product recorded on a non-transitory computer-readable storage medium such as a USB memory, a CD-ROM, or a DVD (none of which are shown). Furthermore, data may be written to a non-volatile storage medium such as a USB memory via the connector of the auxiliary storage device 124.

[0022] An input device 132, such as a keyboard or a mouse, may be connected to the input / output interface 128, which receives information indicating a user's operation in response to the user's operation of the state definition device 10. An output device 134, including a display 136 that displays information generated by the state definition device 10, may also be connected to the input / output interface 128. The input / output interface 128 outputs the information received by the input device 132 to the CPU 120, and outputs or displays information generated by the state definition device 10 to the user. In the following description, a case will be exemplified in which the state definition device 10 is realized by a program executed on an OS running on the information processing device 12.

[0023] Referring again to FIG. 1 , the components of the state definition device 10 will be described. The image feature extraction unit 100 of the state definition device 10 receives damage image data indicating damage and receives image feature extraction parameters from the machine learning unit 110. The damage includes cracks, corrosion, fissures, peeling, exposed rebar, and the like that have occurred in each of multiple inspection targets included in infrastructure facilities. The damage images are still images, moving images, or videos of the damage recorded by an inspector using a camera. The damage image data is generated, for example, by capturing images of the damage using a digital camera connected as an input device 132 to the input / output interface 128 ( FIG. 2 ). The damage image data may be collected from the network 130 by the image feature extraction unit 100.

[0024] The image feature extraction unit 100 performs image processing on the damage image data, for example, using a neural network with image feature extraction parameters, converts the damage image data into a numerical vector, and extracts the image feature vector. To generate the image feature vector from the damage image data, global features using a Residual Network (ResNet) or a Histogram of Oriented Gradients (HoG) may also be used. Furthermore, the image feature extraction unit 100 outputs the image feature vector and the damage image data to the machine learning unit 110.

[0025] The summary feature extraction unit 102 receives summary sentence data indicating a summary explanatory sentence corresponding to the damage image data received by the image feature extraction unit 100, and receives summary feature extraction parameters from the machine learning unit 110. The summary sentence data is generated in response to, for example, a user's operation on a keyboard included in the input device 132 connected to the input / output interface 128 (FIG. 2). Alternatively, the summary sentence data may be collected by the summary feature extraction unit 102 from the network 130, for example.

[0026] The summary description is a natural language sentence prepared for inspectors by an expert on damage to infrastructure inspection objects, qualitatively describing the outline of the damage shown in the damage image data. For example, when the damage image shows "cracks" as the damage, the summary description may be written as "a photo of cracks." However, in this summary description, "a photo of" may be written as "an image of." For example, when the damage image shows "vertical cracks" as the damage, the summary description may be written as "a photo of a vertical crack." For example, when the damage image shows "tortoiseshell cracks" as the damage, the summary description may be written as "a photo of alligator cracks."

[0027] The summary feature extraction unit 102 generates a summary feature vector by converting the character strings indicated by the received summary sentence data into a numerical vector. The summary feature extraction unit 102 outputs the generated summary feature vector to the integrated feature vector generation unit 106. Note that the conversion from the character strings of the summary sentence data to the numerical vector is performed, for example, by processing using a neural network to which summary feature extraction parameters are applied. Alternatively, this conversion can be performed by a text encoder or a learning device such as an SVM.

[0028] The detailed feature extraction unit 104 receives detailed sentence data indicating detailed explanatory sentences corresponding to the damage image data received by the image feature extraction unit 100, and receives detailed feature extraction parameters from the machine learning unit 110. The detailed sentence data is generated in response to a user's operation on a keyboard included in the input device 132 connected to the input / output interface 128 (FIG. 2). Alternatively, the detailed sentence data may be collected by the detailed feature extraction unit 104 from the network 130.

[0029] The detailed description is written in natural language and is prepared for inspectors by experts on damage to infrastructure objects to be inspected. It qualitatively describes the details of the damage shown in the damage image data. For example, if the damage image shows a "crack" as the damage, the detailed description might be something like "Elongated and narrow zigzag line. Clearly darker compared to the surrounding area or black." If the damage image shows a "hexagonal crack" as the damage, the detailed description might be something like "Many branched cracks. Mostly arbitrarily oriented. Usually with a small crack width." When the damage image shows "concrete corrosion" as the damage, the detailed description could be, "Includes the visually similar defects: Washouts, Concrete corrosion and generally all kinds of planar corrosion / erosion / abrasion of concrete. Concrete corrosion can appear as a result of frost-thaw cycles, loss in succession to chemical attacks or abrasion (mechanical or action of acid and salt solutions)."

[0030] The detailed feature extraction unit 104 generates a detailed feature vector by converting the character string indicated by the received detailed sentence data into a numerical vector. The detailed feature extraction unit 104 outputs the generated detailed feature vector to the integrated feature vector generation unit 106. Note that the conversion from the character string indicated by the detailed sentence data to a numerical vector is performed, for example, by processing using a neural network to which detailed feature extraction parameters are applied, similar to the conversion from character string to numerical vector in the summary feature extraction unit 102. Alternatively, this conversion can be performed by a text encoder or a learning device such as an SVM.

[0031] The integrated feature vector generation unit 106 receives a general feature vector from the general feature extraction unit 102 and a detailed feature vector from the detailed feature extraction unit 104. The integrated feature vector generation unit 106 generates an integrated feature vector by integrating the received general feature vector and the received detailed feature vector, and outputs the integrated feature vector to the similarity data generation unit 108 and the image / class display unit 114. Note that the integration of the general feature vector and the detailed feature vector involves converting these vectors into a single integrated feature vector, and this integration can be achieved, for example, by simply concatenating the vectors together, or by processing using a transformer-type neural network (multi-head attention module).

[0032] The similarity data generation unit 108 receives the image feature vector from the image feature extraction unit 100 and receives the integrated feature vector from the integrated feature vector generation unit 106. The similarity data generation unit 108 calculates the similarity between the received image feature vector and the integrated feature vector, generates similarity data indicating the calculated similarity, and outputs the data to the machine learning unit 110.

[0033] The machine learning unit 110 receives the integrated feature vector from the integrated feature vector generation unit 106 and similarity data from the similarity data generation unit 108. The image feature extraction unit 100 performs learning by adjusting the values ​​of the image feature extraction parameters, general feature extraction parameters, and detailed feature extraction parameters so as to maximize the similarity between the integrated feature vector indicated by the similarity data and the image feature vector. The machine learning unit 110 maximizes the similarity between the integrated feature vector indicated by the similarity data and the image feature vector by using processes such as stochastic gradient descent (SGD), adaptive moment estimation (Adam), or adaptive gradient algorithm (AdaGrad).

[0034] The machine learning unit 110 outputs the image feature extraction parameters with their adjusted values ​​to the image feature extraction unit 100. The machine learning unit 110 also outputs the general feature extraction parameters with their adjusted values ​​to the general feature extraction unit 102. The machine learning unit 110 also outputs the detailed feature extraction parameters with their adjusted values ​​to the detailed feature extraction unit 104. The machine learning unit 110 further processes the image feature vectors to generate damage class data indicating numbers, symbols, etc. for defining the damage corresponding to the image feature vectors. The machine learning unit 110 associates the generated damage class data with the integrated feature vectors and image feature vectors and outputs them to the damage class database 112. Note that the number of damage class data is equal to the number of integrated feature vectors and the number of image feature vectors.

[0035] The damage class database 112 stores the damage class data input from the machine learning unit 110 in association with the integrated feature vector corresponding to the damage class data. To generate the damage class data from the image feature vector, a neural network such as a trained MLP (Multi-Layer Perceptron) or Transformer is used. This training is performed by adjusting parameters used to generate the damage class data from the image feature vector so that the similarity between the integrated feature vector and the feature vector generated from the damage class data is maximized, for example.

[0036] The image / class display unit 114 receives the integrated feature vector from the integrated feature vector generation unit 106. The image / class display unit 114 searches the damage class database 112 using the received integrated feature vector, and outputs the damage class data and damage image data associated with the integrated feature vector to the display 136 of the output device 134 via the input / output interface 128 (FIG. 2). The display 136 displays the input damage class data and damage image data to the user.

[0037] Next, we will explain the machine learning operation of the state defining device 10. Figure 3 is a flowchart illustrating an example of the machine learning operation (S10) of the state defining device 10 shown in Figure 1. Note that in the machine learning operation of the state defining device 10, the image / class display unit 114 does not function.

[0038] In S100, the state definition device 10 determines whether learning using all of the damage image data, summary sentence data, and detailed sentence data has been completed. If learning using all of the damage image data, summary sentence data, and detailed sentence data has been completed (Y in the process of S100), the state definition device 10 proceeds to the process of S122. If learning using all of the damage image data, summary sentence data, and detailed sentence data has not been completed (N in the process of S100), the state definition device 10 proceeds to the process of S102.

[0039] In S102, the image feature extraction unit 100 accepts damage image data that has not previously been the subject of machine learning. In S104, the summary feature extraction unit 102 accepts summary sentence data corresponding to the damage image data accepted in the process of S102. In S106, the detailed feature extraction unit 104 accepts detailed sentence data corresponding to the damage image data accepted in the process of S102.

[0040] In S108, the image feature extraction unit 100 extracts an image feature vector from the damage image data based on the image feature extraction parameters input from the machine learning unit 110, and outputs the image feature vector to the integrated feature vector generation unit 106, the similarity data generation unit 108, and the machine learning unit 110. In S110, the summary feature extraction unit 102 extracts a summary feature vector from the summary sentence data based on the summary feature extraction parameters input from the machine learning unit 110, and outputs the summary feature vector to the integrated feature vector generation unit 106. In S112, the detailed feature extraction unit 104 extracts a detailed feature vector from the detailed sentence data based on the detailed feature extraction parameters input from the machine learning unit 110, and outputs the detailed feature vector to the integrated feature vector generation unit 106.

[0041] In S114, the integrated feature vector generation unit 106 generates an integrated feature vector by integrating the features of the summary description sentence indicated by the summary feature vector and the features of the detailed description sentence indicated by the detailed feature vector, and outputs the integrated feature vector to the similarity data generation unit 108. In S116, the similarity data generation unit 108 calculates the similarity between the image feature vector input from the image feature extraction unit 100 and the integrated feature vector input from the integrated feature vector generation unit 106, and outputs similarity data indicating the calculated similarity to the machine learning unit 110.

[0042] In S118, learning is performed by adjusting the values ​​of the image feature extraction parameters, summary feature extraction parameters, and detailed feature extraction parameters so as to maximize the similarity between the image feature vector and the integrated feature vector. In S120, the machine learning unit 110 outputs these parameters to the image feature extraction unit 100, summary feature extraction unit 102, and detailed feature extraction unit 104, respectively.

[0043] The processing loop of S100 to S118 ends, for example, when processing of all damage image data, their summary text data, and their detailed text data is completed. Alternatively, the processing loop may end when the processing loop has been repeated a predetermined number of times. Furthermore, the processing loop may end when the change in the similarity between the image feature vector and the integrated feature vector becomes minimal and it is determined that the similarity between the image feature vector and the integrated feature vector has converged.

[0044] In S122, the machine learning unit 110 generates damage class data, associates the generated damage class data with the integrated feature vector, and outputs the associated data to the damage class database 112. The damage class database 112 stores the damage class data, integrated feature vector, and image feature vector input from the machine learning unit 110 in association with each other, and then ends the processing.

[0045] Next, the operation of the state definition device 10 to associate damage class data with input damage image data and output the data will be described. In this operation, the similarity data generation unit 108 and the machine learning unit 110 do not function, but the image / class display unit 114 functions. FIG. 4 is a flowchart illustrating an example of the operation (S14) to associate damage class data with input damage image data and output the data. As shown in FIG. 4, first, the state definition device 10 performs S102 and S110 described with reference to FIG. 3 to generate an image feature vector corresponding to the input damage image data.

[0046] In S140, the image / class display unit 114 uses the integrated feature vector generated by the processes of S102 to S114 to search for damage class data and image feature vectors stored in the damage class database 112. This search obtains damage class data associated with the image feature vector that has the highest similarity to the generated image feature vector. In S142, the image / class display unit 114 outputs the damage class data and damage image data obtained by the search in the process of S140 to the output device 134 via the input / output interface 128. That is, for example, the display 136 of the output device 134 displays the damage image data output from the image / class display unit 114 and the definition of damage given by the damage class data, and shows them to the user.

[0047] According to the state definition device 10 of the present disclosure described above, damage can be defined using numbers and symbols indicating damage classes based on a damage image, a summary description sentence that summarizes the damage, and a detailed description sentence that explains the damage in detail. That is, the state definition device 10 can perform a general classification of the damage indicated by the damage image using the summary feature extraction unit 102, and a detailed classification of the damage indicated by the damage image using the detailed feature extraction unit 104. In this way, the state definition device 10 can qualitatively learn the damage indicated by the damage image through machine learning using summary data related to the damage and detailed data related to the damage, that is, through machine learning using multiple data with different granularities.

[0048] [Second embodiment] A second embodiment of the present disclosure will be described below. Fig. 5 is a diagram illustrating an example configuration of a state defining device 18 according to the present disclosure. The state defining device 18 has a configuration in which the integrated feature vector generating unit 106, similarity data generating unit 108, machine learning unit 110, and damage class database 112 of the state defining device 10 shown in Fig. 1 are replaced with an integrated feature vector generating unit 180, a similarity data generating unit 182, a machine learning unit 184, and an inspection target class database 186.

[0049] As shown in Figure 5, in place of the damage image data input to the state defining device 10, inspection target data that is different from the damage image data that was the subject of machine learning shown in Figure 3 is input to the image feature extraction unit 100 of the state defining device 18. However, the inspection target data and the inspection target class data described below differ from the damage image data and damage class data only in names that have been changed for convenience of explanation and in whether or not they were the subject of machine learning shown in Figure 3. These data are essentially the same, and the processing of the inspection target data and inspection target class data is the same as the processing of the damage image data and damage class data.

[0050] For the sake of specificity and clarity, the second embodiment will be described as a case in which the state defining device 18 performs machine learning on one piece of inspection target data and N sets of summary sentence data and detailed sentence data corresponding to the inspection target data. Therefore, in the second embodiment, the number of summary sentence data and the number of detailed sentence data input to the state defining device 18 are the same, but the number of inspection target data and the number of summary sentence data and detailed sentence data are not necessarily the same. It goes without saying that the state defining device 18 can perform machine learning on each of multiple pieces of inspection target data and N sets of summary sentence data and detailed sentence data corresponding to the inspection target data. It also goes without saying that the state defining device 18 can perform machine learning on multiple pieces of inspection target data obtained by photographing a single inspection target over time.

[0051] The inspection target data indicates, for example, images of a plurality of inspection targets captured after machine learning is performed on the damage image data, summary sentence data, and detailed sentence data by machine learning shown in Fig. 3. Therefore, the inspection target data has not been subjected to machine learning by the state defining device 18 shown in Fig. 3 before being input to the state defining device 18.

[0052] The integrated feature vector generation unit 180 performs the same processing as the integrated feature vector generation unit 106 (FIG. 1). The feature vector generation unit 180 further receives N sets of general feature vectors and detailed feature vectors corresponding to one piece of inspection target data from the general feature extraction unit 102. The integrated feature vector generation unit 180 integrates the general feature vector and detailed feature vector of each of the received N sets to generate N types of integrated feature vectors and outputs them to the similarity data generation unit 182.

[0053] The similarity data generation unit 182 performs the same processing as the similarity data generation unit 108 (FIG. 1). The similarity data generation unit 182 further calculates the similarity between N types of integrated feature vectors and one piece of inspection target data corresponding to these N types of integrated feature vectors, generates similarity data indicating the calculated similarity, and outputs the data to the machine learning unit 184.

[0054] The machine learning unit 184 performs the same processing as the machine learning unit 110 (FIG. 1). In the processing of the machine learning unit 110 by the machine learning unit 184, damage image data and damage class data, etc., are replaced with inspection target data and inspection target class data, etc. The machine learning unit 184 further receives N types of integrated feature vectors corresponding to one piece of inspection target data from the integrated feature vector generation unit 180, and receives similarity data from the similarity data generation unit 182. The image feature extraction unit 100 performs machine learning by adjusting the values ​​of the numerical values ​​included in the image feature extraction parameters, general feature extraction parameters, and detailed feature extraction parameters so as to maximize the similarity between the N types of integrated feature vectors indicated by the similarity data and the image feature vector corresponding to the inspection target data.

[0055] The inspection object class database 186 performs the same processing as the damage class database 112 on the image feature vectors obtained from the inspection object data, the integrated feature vectors obtained from the general feature vectors and detailed feature vectors corresponding to the inspection object data, and the inspection object class data used to define the state of the inspection object indicated by the inspection object data. That is, the inspection object class database 186 stores the image feature vectors, integrated feature vectors, and inspection object class data in association with each other. The inspection object class database 186 further stores the N types of integrated feature vectors input from the feature vector generation unit 180 and the inspection object class data input from the machine learning unit 184 in association with each other.

[0056] Next, the machine learning operation of the state defining device 18 will be described. FIG. 6 is a flowchart illustrating another example (S20) of the machine learning operation of the state defining device 18 shown in FIG. 4. Note that the image / class display unit 114 does not function in the machine learning operation of the state defining device 18. Furthermore, the operation of associating and outputting inspection target data (damage image data) with inspection target class data (loss class data) used to define the state of the inspection target based on the results of the machine learning processing shown in FIG. 6 is the same as that described with reference to FIG. 4, and therefore will not be described here. However, in this case, in the description of FIG. 4, it is necessary to replace the integrated feature vector generation unit 106 with the integrated feature vector generation unit 180, the similarity data generation unit 108 with the similarity data generation unit 182, and the machine learning unit 110 with the machine learning unit 184. Furthermore, it is necessary to replace the damage class data with the inspection target class data, and the damage image data with the inspection target image data.

[0057] As shown in Fig. 6, in S200, the state defining device 18 determines whether learning has been completed using one piece of inspection target data and all of the N sets of summary sentence data and detailed sentence data corresponding to this inspection target data. When learning has been completed using all of the damage image data, summary sentence data, and detailed sentence data (Y in the process of S200), the state defining device 18 terminates the machine learning operation after processing in S222 below. When learning has not been completed using all of the damage image data, summary sentence data, and detailed sentence data (N in the process of S200), the state defining device 18 proceeds to processing in S202.

[0058] In S202, the image feature extraction unit 100 accepts inspection target data that has previously been the subject of the machine learning shown in Fig. 3 and has not previously been the subject of the machine learning shown in Fig. 6. In S204, the summary feature extraction unit 102 accepts N types of summary sentence data corresponding to the inspection target data accepted in the processing of S202. In S206, the detailed feature extraction unit 104 accepts N types of detailed sentence data corresponding to the inspection target data accepted in the processing of S202.

[0059] In S208, the image feature extraction unit 100 extracts an image feature vector from the inspection target data based on the image feature extraction parameters input from the machine learning unit 184, and outputs the image feature vector to the integrated feature vector generation unit 180, the similarity data generation unit 182, and the machine learning unit 184. In S210, the summary feature extraction unit 102 extracts N types of summary feature vectors from the N types of summary sentence data based on the summary feature extraction parameters input from the machine learning unit 184, and outputs the N types of summary feature vectors to the integrated feature vector generation unit 180. In S212, the detailed feature extraction unit 104 extracts N types of detailed feature vectors from the N types of detailed sentence data based on the detailed feature extraction parameters input from the machine learning unit 184, and outputs the N types of detailed feature vectors to the integrated feature vector generation unit 180.

[0060] In S214, the integrated feature vector generation unit 180 integrates the N sets of summary feature vectors and detailed feature vectors, generates N types of integrated feature vectors indicating the integrated features, and outputs them to the similarity data generation unit 182. In S216, the similarity data generation unit 182 calculates the similarity between the image feature vector generated from the inspection target data by the image feature extraction unit 100 and the N types of integrated feature vectors input from the integrated feature vector generation unit 180, and outputs N types of similarity data indicating the calculated similarities to the machine learning unit 184.

[0061] In S218, learning is performed by adjusting the values ​​of the image feature extraction parameters, summary feature extraction parameters, and detailed feature extraction parameters so as to maximize the similarity between the image feature vector and the N types of integrated feature vectors. In S220, the machine learning unit 184 outputs these parameters to the image feature extraction unit 100, summary feature extraction unit 102, and detailed feature extraction unit 104, respectively.

[0062] In S222, the machine learning unit 184 generates inspection object class data, associates the generated inspection object class data with the integrated feature vector, and outputs the associated data to the inspection object class database 186. The inspection object class database 186 stores the damage class data and the integrated feature vector input from the machine learning unit 184, associating them with each other. By performing the machine learning shown in Fig. 6, an event such as damage occurring in one inspection object is defined from multiple perspectives, and the versatility of the inspection object class database 186 is increased.

[0063] [Variations] A modified example of an embodiment of the present disclosure will be described below. The image / class display unit 114 can read damage class data (inspection target class data) associated with an integrated feature vector input from the similarity data generation unit 108 (182) that has a similarity equal to or greater than a predetermined threshold (e.g., 80%) from the damage class database 112 (inspection target class database 186) and display the damage class data (inspection target class data) to the user along with the damage image data (inspection target data). Note that there is no substantial difference between "above threshold" and "greater than threshold." By modifying the operation of the image / class display unit 114 in this way, the condition definition device 10 (18) can obtain one or more types of damage class data (inspection target class data) as a search result based on one input damage image data (inspection target data) and display the damage class data (inspection target class data) to the user along with the input damage image data (inspection target data).

[0064] 4 may be executed in advance by the state definition devices 10, 18, and then the processes of S102, S110, and S142 may be executed. For example, by inputting a large amount of damage image data into the state definition devices 10, 18 in advance and searching for damage class data (inspection target class data), the amount of processing in S102, S110, and S142 can be reduced, and the processing time can also be shortened.

[0065] As described above, the condition definition devices 10, 18 (FIGS. 1 and 5) use damage image data (inspection target data) as input to easily and quickly obtain damage class data corresponding to the damage image data. Therefore, the damage (inspection target) indicated by the damage image data (inspection target data) can be easily and quickly defined.

[0066] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. [Appendix 1] A state definition device for defining a state of an inspection object, comprising one or more processors, wherein the one or more processors are configured to accept first inspection object data showing an image of a first inspection object, summary sentence data describing in sentences an overview of the state of the inspection object, and detailed sentence data describing in sentences the details of the state of the inspection object, generate an image feature vector from the accepted first inspection object data showing features of the image shown by the first inspection object data, generate a summary feature vector from the accepted summary sentence data showing features of the overview of the state of the inspection object indicated by the summary sentence data, generate a detail feature vector from the accepted detailed sentence data showing features of the details of the state of the inspection object indicated by the detailed sentence data, integrate the generated summary feature vector and detail feature vector to generate an integrated feature vector, and generate inspection object class data from the integrated feature vector to be used in defining the state of the inspection object. [Appendix 2] A state definition device as described in Appendix 1, which accepts first inspection object data indicating an image of a first inspection object and one or more sets of summary sentence data and detailed sentence data, and generates one or more sets of summary feature vectors and detailed feature vectors corresponding to the first inspection object data. [Appendix 3] 3. The state definition device of claim 1, wherein the one or more processors are further configured to adjust an image feature extraction parameter used to generate the summary feature vector, a summary feature extraction parameter used to generate the summary feature vector, and a detailed feature extraction parameter used to generate the detailed feature vector, so as to maximize a similarity between the generated image feature vector and the integrated feature vector. [Appendix 4] The state definition device according to any one of appendices 1 to 3, wherein the one or more processors are further configured to store the integrated feature vector, the inspection object class data, and the image feature vector in association with each other. [Appendix 5] 5. A state definition device according to any one of appendices 1 to 4, which accepts second inspection object data indicating an image of a second inspection object, generates an image feature vector indicating characteristics of an image indicated by the second inspection object data from the accepted second inspection object data, and obtains the inspection object class data corresponding to the generated image feature vector. [Appendix 6] 6. A state definition device according to any one of appendices 1 to 5, which obtains an image feature vector indicating a feature of an image indicated by the second inspection object data, and one or more inspection object class data corresponding to the image feature vector having a similarity equal to or greater than a predetermined threshold. [Appendix 7] 7. A state definition device according to any one of appendices 1 to 6, wherein the state of the inspection object is damage to the inspection object. [Appendix 8] 8. A state definition device according to any one of appendices 1 to 7, wherein the summary text data is created by an expert on the inspection object and qualitatively indicates characteristics of damage to the inspection object. [Appendix 9] A step of receiving first inspection target data showing an image of a first inspection target, summary sentence data explaining in sentences an overview of the state of the inspection target, and detailed sentence data explaining in sentences the details of the state of the inspection target, and a step of generating an image feature vector showing features of the image shown by the first inspection target data from the received first inspection target data; A state definition method comprising the steps of: generating, from the received summary sentence data, a summary feature vector indicating the summary characteristics of the state of the inspection object indicated by the summary sentence data; generating, from the received detailed sentence data, a detailed feature vector indicating the detailed characteristics of the state of the inspection object indicated by the detailed sentence data; integrating the generated summary feature vector and detailed feature vector to generate an integrated feature vector; and generating, from the integrated feature vector, inspection object class data used to define the state of the inspection object. [Appendix 10] A state definition program that causes a computer to execute the following processes: a process for accepting first inspection object data showing an image of a first inspection object, summary sentence data that provides a sentence summary of the state of the inspection object, and detailed sentence data that provides a sentence summary of the state of the inspection object; a process for generating, from the accepted first inspection object data, an image feature vector that indicates the characteristics of the image shown by the first inspection object data; a process for generating, from the accepted summary sentence data, a summary feature vector that indicates the characteristics of the summary of the state of the inspection object that is indicated by the summary sentence data; a process for generating, from the accepted detailed sentence data, a detailed feature vector that indicates the characteristics of the details of the state of the inspection object that are indicated by the detailed sentence data; a process for integrating the generated summary feature vector and detailed feature vector to generate an integrated feature vector; and a process for generating inspection object class data from the integrated feature vector that is used to define the state of the inspection object. It goes without saying that combinations of the various forms described in the appendix of this disclosure, or any combination of the elements described in each aspect and embodiment (including the non-selection of some elements), can be made at any time by those skilled in the art in accordance with the basic concept of this disclosure.

[0067] The disclosures of the above-cited patent documents and other documents are incorporated herein by reference. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concept thereof. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including elements of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by those skilled in the art in accordance with the entire disclosure and technical concept, including the claims. In particular, the numerical ranges set forth herein should be construed as specifically describing any numerical value or subrange within the range, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents are deemed to be included in the disclosure of this application, in part or in whole, which may be used in combination with the disclosures herein, as part of the disclosure of the present invention, in accordance with the spirit of the present invention, as necessary. [Explanation of symbols]

[0068] 10,18 State definition device 100 Image feature extraction unit 102 Overview feature extraction unit 104 Detailed feature extraction unit 106,180 Integrated feature vector generation unit 108,182 Similarity data generation part 110,184 Machine Learning Department 112 Damage Class Database 114 Image / Class Display Section 186 Inspection Class Database

Claims

1. A state definition device including one or more processors for defining a state of an inspection target, the one or more processors including: receiving first inspection target data showing an image of a first inspection target, summary sentence data describing in sentences an overview of the state of the inspection target, and detailed sentence data describing in sentences the details of the state of the inspection target; generating an image feature vector indicating features of an image indicated by the first inspection target data from the received first inspection target data; generating a summary feature vector indicating a summary feature of the state of the inspection object indicated by the summary sentence data from the received summary sentence data; generating a detailed feature vector from the received detailed sentence data, the detailed feature vector indicating the detailed features of the state of the inspection object indicated by the detailed sentence data; Integrating the generated summary feature vector and the generated detailed feature vector to generate an integrated feature vector; Generate inspection object class data used to define the state of the inspection object from the integrated feature vector. A state definition device configured as follows.

2. receiving first inspection target data representing an image of a first inspection target and one or more sets of the summary sentence data and the detailed sentence data; generating one or more sets of the general feature vector and the detailed feature vector corresponding to the first inspection target data; The state definition device according to claim 1 .

3. The one or more processors further An image feature extraction parameter used to generate the summary feature vector, a summary feature extraction parameter used to generate the summary feature vector, and a detailed feature extraction parameter used to generate the detailed feature vector are adjusted so that the similarity between the generated image feature vector and the integrated feature vector is maximized.

2. The state definition device according to claim 1, configured to:

4. The one or more processors further The integrated feature vector, the inspection object class data, and the image feature vector are stored in association with each other.

4. The state definition device according to claim 3, which is configured as follows:

5. receiving second inspection object data representing an image of the second inspection object; generating an image feature vector indicating features of an image indicated by the second inspection target data from the received second inspection target data; Obtain the inspection target class data corresponding to the generated image feature vector.

5. A state definition device according to claim 4.

6. An image feature vector indicating the characteristics of an image indicated by the second inspection object data is obtained, and one or more inspection object class data corresponding to the image feature vector having a similarity equal to or greater than a predetermined threshold value is obtained.

6. A state definition device according to claim 5.

7. The condition of the inspection object is damage to the inspection object. The state definition device according to claim 1 .

8. The summary text data is prepared by an expert on the inspection object and qualitatively indicates characteristics of damage to the inspection object. The state definition device according to claim 1 .

9. A step of receiving first inspection target data showing an image of a first inspection target, summary sentence data describing in sentences an overview of a state of the inspection target, and detailed sentence data describing in sentences the details of the state of the inspection target; generating an image feature vector indicating features of an image indicated by the first inspection target data from the received first inspection target data; generating a summary feature vector indicating summary features of the state of the inspection object indicated by the summary sentence data from the received summary sentence data; generating a detailed feature vector from the received detailed sentence data, which indicates the detailed features of the state of the inspection object indicated by the detailed sentence data; generating an integrated feature vector by integrating the generated summary feature vector and the generated detailed feature vector; generating inspection object class data used to define the state of the inspection object from the integrated feature vector; A state definition method including:

10. a process for receiving first inspection target data showing an image of a first inspection target, summary sentence data describing in sentences an overview of the state of the inspection target, and detailed sentence data describing in sentences the details of the state of the inspection target; a process of generating an image feature vector indicating features of an image represented by the first inspection target data from the received first inspection target data; a process of generating, from the received summary sentence data, a summary feature vector indicating summary features of the state of the inspection object indicated by the summary sentence data; a process of generating a detailed feature vector from the received detailed sentence data, which indicates the detailed features of the state of the inspection object indicated by the detailed sentence data; a process of integrating the generated summary feature vector and the generated detailed feature vector to generate an integrated feature vector; a process of generating inspection object class data used to define the state of the inspection object from the integrated feature vector; A state definition program that causes a computer to execute the above.

Images