Inspection device, inspection method, and image data generation method

Abstract

To provide an inspection system with improved reliability.SOLUTION: An inspection device 10 includes an image generation unit 11 that generates image data in which a target is written together with a load-displacement relationship obtained in a loading test, and an evaluation unit 12 that evaluates the inspection target by inputting the image data to a learned deep learning model. The deep learning model performs deep learning using training data including teacher image data obtained by converting a test result and a reference for training and a label indicating an evaluation so as to output the evaluation when image data is input.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an inspection apparatus, an inspection method, and a method for generating image data. [Background technology]

[0002] In order to improve earthquake resistance, the introduction of seismic isolation and vibration control devices into buildings is increasing. However, in recent years, the falsification of inspection data for seismic isolation and vibration control devices has become a problem.

[0003] Inspection of seismic isolation and vibration control devices involves carrying out load tests on the devices. From the load-displacement relationship obtained from the test, rigidity, energy, etc. are calculated, and pass / fail is judged based on the ratio of the calculated value to the standard value. In this inspection, evaluation by a third party with no vested interest in the manufacturer is important in order to maintain objectivity.

[0004] Patent Document 1 discloses a technique for reading inspection conditions based on identification information of a specimen used for seismic isolation and vibration control of a structure and then carrying out the inspection.

[0005] Patent Document 2 discloses a technique for efficiently inspecting a seismic isolation device by taking an image including the seismic isolation device and a reference scale (dimensions), as a method for inspecting the dimensions of the device. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-018449 [Patent Document 2] Japanese Patent Publication No. 2020-003295 [Non-patent literature]

[0007] [Non-Patent Document 1] Karsten R., Latha, P., Joaquin, Z., Bernhard, S., Thomas B., and Peter, G., “Towards Total Recall in Industrial Anomaly Detection”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 14318-14328, 2022. [Non-patent document 2] T. Fukasawa, S. Okamura, T. Somaki, T. Miyagawa, T. Yamamoto, T. Watakabe, T. Morohishi, R. Fujita, S. Fujita, “Study on analytical model of oil damper focusing on high frequency and low amplitude”, Journal of Structural and Mechanical Engineering, Architectural Institute of Japan, Vol. 83, No. 754 (2018), pp. 1777-1787. [Non-patent document 3] T. Fukasawa, S. Okamura, T. Somaki, T. Miyagawa, T. Yamamoto, T. Watakabe, T. Morohishi, R. Fujita, S. Fujita, “Proposal of a Hysteresis Model Using Differential Equations”, Journal of Structural and Mechanical Engineering, Architectural Institute of Japan, Vol. 84, No. 763 (2019), pp. 1231-1241. Summary of the Invention [Problem to be solved by the invention]

[0008] Conventional inspections involve human judgment in determining pass / fail, which poses the risk of human error and fraud.

[0009] Furthermore, because inspectors are required to have specialized skills, there is also the problem that inspection accuracy depends on the experience and skills of the inspectors. Furthermore, there are concerns about a shortage of inspector human resources.

[0010] The present disclosure has been made in view of the above, and aims to provide an inspection system with improved reliability. [Means for solving the problem]

[0011] An inspection device according to one aspect of the present disclosure includes an image generation unit that generates image data that includes test results obtained from a test on an object to be inspected, along with standards that the object must meet in the test, and an evaluation unit that inputs the image data into a trained deep learning model to evaluate the object to be inspected. The deep learning model is deep-learned using the teacher image data that includes the teacher test results and the standards.

[0012] An inspection method according to one aspect of the present disclosure is an inspection method using an inspection device, which generates image data in which test results obtained from a test on an object to be inspected are written together with standards that the object to be inspected must meet in the test, and inputs the image data into a trained deep learning model to evaluate the object to be inspected. The deep learning model is deep-trained using teacher image data in which the teacher test results and standards are written together. [Effects of the Invention]

[0013] According to the present disclosure, an inspection system with improved reliability can be provided. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an inspection system. [Figure 2] FIG. 2 is a diagram showing an example of image data obtained by converting test results and targets. [Figure 3] FIG. 3 is a diagram showing an example of image data obtained by converting test results and targets. [Figure 4] FIG. 4 is a diagram showing an example of image data obtained by converting test results and targets. [Figure 5] FIG. 5 is a diagram illustrating an example of a heat map. [Figure 6] FIG. 6 is a diagram illustrating an example of a heat map. [Figure 7] FIG. 7 is a diagram illustrating an example of a heat map. [Figure 8] FIG. 8 is a diagram illustrating an example of a heat map. [Figure 9]FIG. 9 is a flowchart showing an example of the processing flow of the inspection device. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of the learning process. [Figure 11] FIG. 11 is a diagram showing an example of image data of the load-displacement relationship of an oil damper that has been detected as being abnormal. [Figure 12] FIG. 12 is a diagram showing an example of a heat map of an oil damper that has been detected as abnormal. [Figure 13] FIG. 13 is a diagram showing an example of image data of the load-displacement relationship of a laminated rubber member that has been detected as being abnormal. [Figure 14] FIG. 14 is a diagram showing an example of a heat map of a laminated rubber bearing that has been detected as abnormal. DETAILED DESCRIPTION OF THE INVENTION

[0015] [Inspection system configuration] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0016] An example of the configuration of an inspection system according to this embodiment will be described with reference to Fig. 1. The inspection system shown in the figure includes an inspection device 10, a load test device 30, and a data storage device 50. Each device is connected to each other via a network so that they can communicate with each other.

[0017] The load test device 30 performs a load test on the test object to obtain test result data. A load test is a test to confirm whether the required rigidity, damping force, etc., for which the product or material is designed are exhibited and whether it can appropriately withstand the load. The test object is, for example, a vibration control device or seismic isolation device, and the load test is used to check the performance of oil dampers, laminated rubber, etc. The load test device 30 performs a load test on each test object to obtain time-series data on the load-displacement relationship. Test conditions may be changed to obtain test result data for each test condition.

[0018] The data storage device 50 receives the test result data from the load test device 30 and stores the test result data for each inspection object. The data storage device 50 transmits the test result data to the inspection device 10.

[0019] The inspection device 10 inputs test result data for each inspection object, converts the test result data into image data, and also writes a reference value (target) for pass / fail judgment on the image data.The image data is input into a convolutional neural network (hereinafter referred to as a CNN or deep learning model) to judge whether the inspection object passes or fails.

[0020] The inspection device 10 includes an image generation unit 11, an evaluation unit 12, an output unit 13, a learning unit 14, and a storage unit 15. Each unit included in the inspection device 10 may be configured with at least one computer equipped with an arithmetic processing unit, a storage device, etc., and the processing of each unit may be executed by a program. This program is stored in a storage device included in the inspection device 10, and can also be recorded on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, or semiconductor memory, or provided via a network. Each unit included in the inspection device 10 will be described below.

[0021] The image generation unit 11 converts the test result data for each inspection object into image data and draws a target on the image data. Specifically, the image generation unit 11 generates an image of a graph plotting the relationship between load and displacement from the load and displacement time-series data obtained in the load test, and draws a target graph on that image. The target is a value set by the manufacturer of the inspection object and is the performance standard that the inspection object must meet in the load test. The target data may be stored in the data storage device 50 along with the test result data, and the image generation unit 11 may receive the test result data and the target data from the data storage device 50, or the memory unit 15 may store the target data and the image generation unit 11 may read it from the memory unit 15. The inspector may input the target into the inspection device 10.

[0022] 2 to 4 show examples of image data generated by the image generation unit 11. In this embodiment, the load is plotted on the vertical axis and the displacement on the horizontal axis, with the test results in blue and the targets in green, on the same RGB image. In FIGS. 2 to 4, the test results are shown with solid lines and the targets in dashed lines. FIG. 2 shows an image of a linear-type oil damper, FIG. 3 shows an image of a bilinear-type oil damper, and FIG. 4 shows an image of a target and a hysteresis loop showing the load-displacement relationship of a laminated rubber bearing. Note that the colors of each line may be selected arbitrarily as long as the test result line and the target line can be distinguished. The target may be drawn above the test results on the image, or the test results may be drawn above the target. Although axes and scales are drawn in FIGS. 2 to 4, the axes and scales may not be drawn, and numerical values ​​may be drawn on the scales.

[0023] The evaluation unit 12 inputs image data depicting the test results and targets into a trained deep learning model to obtain a pass / fail evaluation of the test object. The deep learning model used here is one that has undergone deep learning using teacher image data, which is obtained by converting the teacher's test results and teacher's targets into image data, and teacher data including labels indicating the evaluation, so that an evaluation is output when the image data is input. The deep learning model can use a CNN, which can extract and classify features from the input image.

[0024] The output unit 13 outputs the evaluation of whether the inspection object passes or fails obtained by the evaluation unit 12. The output unit 13 may store the evaluation of the inspection object in the data storage device 50.

[0025] If the test object fails, the output unit 13 may output a heat map showing highly abnormal areas superimposed on the image data of the test results as the basis for the failure judgment. The heat map can be generated, for example, by extracting the output of the convolution layer or pooling layer of the CNN as a feature map, selecting channels that react strongly to abnormal areas from each channel of the feature map, and emphasizing the strength of the reaction.

[0026] Figures 5 to 8 show examples of heat maps output by the output unit 13. In the heat maps in Figures 5 to 8, areas with a high degree of abnormality are highlighted. Figures 5 and 6 show the test results and heat maps for an oil damper that was determined to be unacceptable. In Figure 5, the positive and negative loads exceed the allowable values. In Figure 6, the positive and negative loads are within the allowable values, but a large free-running section (slip region) near the maximum amplitude was detected. Currently, there is no evaluation formula for slip shape, and the slip characteristics are inspected based on the skill of the inspector. Figures 7 and 8 show the test results and heat maps for a laminated rubber bearing that was determined to be unacceptable. In Figure 7, the load (stiffness) at 100% shear strain exceeds the allowable value. In Figure 8, an abnormality in the load (stiffness) was detected near shear strain ±100%, including the hardening region (not subject to inspection in general delivery inspections).

[0027] The learning unit 14 learns the deep learning model using teacher image data in which the teacher test results and targets are converted into images, and teacher data including labels indicating evaluations of the test results. The inspection device 10 may not be equipped with the learning unit 14, and the deep learning model may be learned by a separate device, and parameters of the learned deep learning model may be stored in the memory unit 15.

[0028] The memory unit 15 stores parameters of the deep learning model. The memory unit 15 may store data generated in the process of evaluating the test object, such as image data, heat maps, and evaluation results.

[0029] In this embodiment, the load-displacement relationship is used, but the stress-strain relationship may also be used as the load-displacement relationship, or in the case of an oil damper, the pressure-flow rate relationship may also be used as the load-displacement relationship.

[0030] The inspection target of the inspection device 10 is not limited to seismic isolation and vibration control devices. It can be applied to any object that can perform a test, convert the cause-and-effect relationship obtained from the test results into image data, and evaluate the test results against a reference value (threshold value).

[0031] The test result data may be transmitted from the load test device 30 to the inspection device 10. In this case, the inspection device 10 may store the test result data and the evaluation of the inspection target in the data storage device 50.

[0032] [Operation of inspection equipment] Next, an example of the processing flow of the inspection device 10 will be described with reference to the flowchart of Fig. 9. It is assumed that a load test has already been carried out on the inspection object, and test result data has been obtained.

[0033] In step S11, the inspection device 10 inputs the test result data and converts the test result into an image.

[0034] In step S12, the inspection device 10 draws a target on the image into which the test results have been converted.

[0035] In step S13, the inspection device 10 inputs the test results and the image data on which the target is depicted into the deep learning model.

[0036] In step S14, the inspection device 10 obtains an evaluation of the inspection object from the deep learning model and outputs the evaluation.

[0037] If the evaluation is unacceptable, the inspection device 10 outputs a heat map in step S15.

[0038] [Deep learning model training example] Next, an example of learning a deep learning model will be described.

[0039] First, a data set of the load-displacement relationship for the oil damper and laminated rubber bearing was generated using an analytical model. Specifically, as training test result data, a random deviation of ±20% was given to the target, and data with added noise was generated to mimic the noise that occurs during testing. For the oil damper, a slip region was given to the load near the maximum amplitude. The size of the slip region was randomly sampled using random numbers. For the laminated rubber bearing, hardening characteristics and load reduction due to repetition were taken into account.

[0040] Loads and stiffness within ±10% of the target were labeled as normal, and loads and stiffness exceeding ±10% of the target were labeled as abnormal. For oil dampers, even if the load and stiffness were normal, those with a large slip region were labeled as abnormal. 500 pieces of training data were prepared for each oil damper and laminated rubber.

[0041] The training data and target were converted into 500 images each (for example, the images shown in Figures 2 and 4), and transfer learning was performed on a binary classification problem between normal and abnormal using ResNet101 as the deep learning model.

[0042] Here, the learning dataset was generated using an analytical model, but when implementing the inspection system, actual measured data is used as the training data.

[0043] [Another example of deep learning model training] Generally, for industrial products, there is more normal data than abnormal data, and it is easier to obtain normal data. Therefore, we use Patch Core (Non-Patent Document 1) to train a deep learning model using only normal data.

[0044] An example of the process flow for training a deep learning model using only normal data will be described with reference to the flowchart in Figure 10.

[0045] In step S101, the learning unit 14 calculates an input image I∈R H×W×C Let N(H) × N(W) be the patch P i,j ∈R H×W×C Divide into.

[0046]

number

[0047] where I is the input image, H is the height of the image, W is the width of the image, C is the number of channels, i, j are the positions of the patches, N is the size of the patches (small regions) into which the image is divided, and R H×W×C is the set of all real numbers. The input image is an image obtained by converting normal data into image data and drawing a target on the image data.

[0048] In step S102, the learning unit 14 uses CNN to i,j from the feature vector f i,j ∈R d Extract.

[0049]

number

[0050] Here, φ is the feature extraction function of the CNN, and d is the dimensionality of the feature vector. CNN can use a pre-trained model that has learned general image recognition tasks using a large dataset and has acquired the ability to extract various features from images. Alternatively, normal image data can be used to fine-tune the weights of the pre-trained model.

[0051] In step S103, the learning unit 14 constructs a feature space F using feature vectors obtained from normal image data.

[0052]

number

[0053] where K is the total number of normal image data, f i,j k is the feature vector of the k-th normal image data position (i, j). The feature vector may be the output of the final layer of the CNN or the output of an intermediate layer of the CNN.

[0054] Through the above process, a feature space F is constructed using normal data. In this embodiment, the input image is 224 pixels x 224 pixels (RGB), and feature vectors are extracted using ResNet50. The number of dimensions d is 2048. The patch size (H x W) is 16 pixels x 16 pixels.

[0055] [Evaluation method] When evaluating the test results, the evaluation unit 12 generates a patch P i,j test and then use CNN to create a feature vector f i,j test and compare it with the feature space F. Using K-nearest neighbor (K-NN), the anomaly score s i,j is calculated by the following formula:

[0056]

number

[0057] where ||·|| is the Euclidean distance, KNN(f i,j test , F) is the feature space F i,j test is the set of k nearest neighbors of

[0058] Based on the Euclidean distance of the feature points, the distance of the data point (i, j) in the test results (test data) from the normal data is evaluated. The greater the distance, the more abnormal the data point is judged to be. This makes it possible to detect anomalies from the image data of the test results.

[0059] [Validity verification] To verify the effectiveness of the method, load-displacement data was analytically created for oil dampers and laminated rubber bearings, and the accuracy of anomaly detection was confirmed. Specifically, discrete data for the load-displacement relationship of seismic isolation oil dampers (bilinear type) and natural rubber systems was created using an analytical model, and data within ±10% of the reference value was treated as normal, while data that did not conform to this was treated as abnormal. 800 pieces of normal data for both seismic isolation oil dampers and laminated rubber bearings were used to train the deep learning model. The accuracy of anomaly detection was determined from the consistency relationship obtained by inputting 100 pieces of normal data and 100 pieces of abnormal data into the trained deep learning model. The abnormal data included variations within a range of ±20% of the reference value.

[0060] The discrete data for the load-displacement relationship of the seismic isolation oil damper used for verification was created using the following procedure.

[0061] Bilinear damping characteristics are often used for seismic isolation oil dampers. Generally, the damping specifications for bilinear oil dampers define two types of damping coefficients (C1 and C2) and a relief speed (the speed at which the damping coefficient switches). These damping coefficients and loads are evaluated in loading tests. Here, for bilinear damping characteristics, data within ±10% of the reference value is treated as normal. Furthermore, even if they meet the reference value, if the amount of slip (the amount of free running without damping force being expressed) is large, it is treated as abnormal.

[0062] The data required for learning was created using the analytical model (Maxwell Model with switchable damping coefficient) in Non-Patent Document 2. The damping specifications of the target seismic isolation oil damper were: maximum load: 1000 kN, maximum speed: 1.5 m / s, damping coefficient C1: 2500 kN / m, damping coefficient C2: 169.5 kN / m, relief speed: 0.32 m / s.

[0063] A sine wave was created with the above damping specifications, with an amplitude of ±0.5 m or less and a speed of 1.5 m / s or less. This was input as a forced displacement input into the analysis model of Non-Patent Document 2 to obtain discrete data on the load-displacement relationship. Gaussian noise was added to the load obtained by the analysis model to simulate noise during a loading test.

[0064] Figure 11 shows image data (including test results and targets) of the load-displacement relationship of an oil damper that was detected as abnormal, and Figure 12 shows a heat map visualizing the degree of abnormality (the higher the degree of abnormality, the higher the brightness). The verification data for the oil damper includes data that is ±20% of the reference value, but loads that are greater than 1.1 times the reference value and less than 0.9 times the reference value (abnormal values) were detected. Furthermore, even when the reference value is met, loads with large amounts of slip are judged to be abnormal. In the case of a bilinear oil damper, the hysteresis loop shape of the load-displacement relationship differs before and after the relief speed, but regardless of these hysteresis loop shapes, abnormal data was detected with 98% accuracy.

[0065] The discrete load-displacement relationship data for the laminated rubber used in the verification was created using the following procedure.

[0066] Using the analytical model described in Non-Patent Document 2, discrete data on the load-displacement relationship of a natural rubber laminate was generated. The target laminate was G6, whose linear limit is approximately 226%. Inspection of laminated rubber is generally performed below the linear limit. However, to verify the applicability of anomaly detection in the nonlinear region above this limit, the maximum shear strain in this example was set to 400%. The analytical data was gradually increased up to the maximum shear strain, and this data was input into the analytical model to determine the load-displacement relationship. The load-displacement relationship also takes into account hardening characteristics and the decrease in load (stress) due to repeated loading using the hysteresis model described in Non-Patent Document 3. Gaussian noise was added to the load obtained from the analytical model to simulate noise during loading tests.

[0067] Figure 13 shows image data (including test results and targets) of the load-displacement relationship of the laminated rubber that was detected as abnormal, and Figure 14 shows a heat map visualizing the degree of abnormality (the higher the degree of abnormality, the higher the brightness). For the laminated rubber, a tendency for the degree of abnormality to increase is observed as the slope of the load-displacement relationship (stress-strain relationship) deviates from the reference value. As with the oil damper, in the case of the laminated rubber, loads exceeding 1.1 times the reference value or below 0.9 times the reference value are treated as abnormal, and abnormal data for these loads was detected. Furthermore, as shear strain increases, hardening characteristics are exhibited, and although the stress-strain relationship is not directly proportional, abnormal data can be detected even when the stress-strain relationship progresses into the hardening region. The accuracy of detecting abnormalities in the laminated rubber was 100%.

[0068] In this example, the deep learning model was trained using only normal data, and anomalies could be detected with high accuracy. Furthermore, by treating the load-displacement relationship as two-dimensional information when evaluating the load, it became possible to detect anomalies in items that are difficult to evaluate using only load data (evaluation of its maximum and minimum values), such as the slip phenomenon of an oil damper. Furthermore, it was confirmed that the degree of anomaly could be visualized through a heat map of the load-displacement relationship.

[0069] As described above, the inspection device 10 of this embodiment includes an image generation unit 11 that generates image data in which a target is displayed alongside the load-displacement relationship obtained in a loading test, and an evaluation unit 12 that inputs the image data into a trained deep learning model to evaluate the inspection object. The deep learning model performs deep learning using training image data that converts training test results and standards, and training data that includes labels indicating the evaluation, so that an evaluation is output when the image data is input. In this way, by inputting images converted from two-dimensional information on the load-displacement relationship into the deep learning model and evaluating the inspection object, it is possible to provide objective and highly reliable inspections.

[0070] In this embodiment, normal data is used as a training test result, and a deep learning model including a feature space constructed using feature vectors obtained from each patch obtained by dividing the training image data is used to evaluate the inspection object according to the distance between the feature vectors obtained from each patch obtained by dividing the inspection object image data and the feature vectors in the feature space. This allows the deep learning model to be trained using only normal data, and enables highly accurate anomaly detection.

[0071] In this embodiment, by using an image in which the test results are shown alongside the target, it is possible to train a model with a smaller amount of training data than when using an image in which only the test results are drawn, and by using this model, it is possible to obtain a more accurate evaluation.

[0072] The inspection device 10 of this embodiment is equipped with an output unit 13 that displays a heat map showing areas with high abnormality levels superimposed on image data converted from the test results. This allows the user to confirm the basis for the inspection device 10's failure judgment, ensuring transparency of the inspection. [Explanation of symbols]

[0073] 10 Inspection equipment 11 Image generation unit 12 Evaluation Section 13 Output section 14 Learning Department 15 Storage section 30 Load test equipment 50 Data storage device

Claims

1. an image generating unit that generates image data in which test results obtained from a test on an inspection object are written together with standards that the inspection object must satisfy in the test; an evaluation unit that inputs the image data into a trained deep learning model to evaluate the inspection object; Equipped with The deep learning model was deep-learned using teacher image data in which the test results for teachers were written along with the standards. Inspection equipment.

2. The inspection device according to claim 1, The deep learning model uses normal data as a training test result and includes a feature space constructed using feature vectors obtained from each patch obtained by dividing the training image data; The evaluation unit evaluates the inspection object according to the distance between a feature vector obtained from each patch obtained by dividing the image data of the inspection object and a feature vector in the feature space. Inspection equipment.

3. 3. The inspection device according to claim 1 or 2, The test results are data that show the relationship between cause and effect. Inspection equipment.

4. The inspection device according to claim 3, the image generation unit generates image data that includes both the relationship between load and displacement obtained in the load test and a reference for the relationship between load and displacement obtained in the load test; The evaluation unit evaluates whether the inspection object passes or fails. Inspection equipment.

5. 3. The inspection device according to claim 1 or 2, and an output unit that displays a heat map showing areas with high abnormality superimposed on image data obtained by converting the test results. Inspection equipment.

6. An inspection method using an inspection device, generating image data in which test results obtained from a test on the test object are written together with standards that the test object must satisfy in the test; inputting the image data into a trained deep learning model to evaluate the inspection object; The deep learning model was deep-learned using teacher image data in which the test results for teachers were written along with the standards. Testing method.

7. The inspection method according to claim 6, The deep learning model uses normal data as a training test result and includes a feature space constructed using feature vectors obtained from each patch obtained by dividing the training image data; The inspection object is evaluated according to the distance between a feature vector obtained from each patch obtained by dividing the image data of the inspection object and a feature vector in the feature space. Testing method.

8. The inspection method according to claim 6 or 7, The test results are data that show the relationship between cause and effect. Testing method.

9. The inspection method according to claim 8, The relationship between the load and displacement obtained in the load test and the reference for the relationship between the load and displacement obtained in the load test are converted into image data; Evaluate the pass / fail of the inspection object Testing method.

10. The inspection method according to claim 6 or 7, A heat map showing areas with high abnormalities is superimposed on the image data converted from the test results. Testing method.

11. The computer generating image data depicting test results obtained from testing the test object; A standard that the inspection object must satisfy in the test is depicted on the image data in the same format as the test result; The image data is used for training or inference of a deep learning model. How image data is generated.

Images