Predictive maintenance system

Abstract

To provide a predictive maintenance system capable of determining propriety of an operating state of a monitoring target based on generated sound or vibration.SOLUTION: A predictive maintenance system 10c includes: a sound source visualization device 500 that has a microphone 502 for identifying a sound source generated from a monitoring target and generates a sound source visualization image in which the sound source is visualized; a storage unit 443d that stores a machine-learning model constructed based on training data generated from the sound source visualization images in which the sound source changing due to a sign of a malfunction occurring in the monitoring target 600 is visualized; and a determination unit that determines propriety of an operating state of the monitoring target based on the sound source visualization image and the machine-learning model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a pass / fail determination system, a pass / fail determination method, and an instruction data group generation program.

Background Art

[0002] Patent Document 1 describes a service providing system that provides a service using machine learning by artificial intelligence. This service providing system is a service providing system that provides a service using machine learning by artificial intelligence, and includes a machine learning means for inputting learning data based on information sent from a user and generating a general model modeled by machine learning, a personalization means for personalizing the general model into a model suitable for the user based on the information sent from the user, and a service providing means for providing a personalized service to the user using the personalized model, and uses the information sent from the user for both the machine learning and the personalization.

[0003] Patent Document 2 describes a sound source position visualization display method for visualizing the position of an arbitrary sound source in real time by associating it with the real space. This visualization display method is characterized by detecting one or more sounds, determining the position of each sound source, converting information on the sound source including at least the position of the sound source into visible information, and displaying it in real time by superimposing it on a real image around the sound source.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present invention aims to provide a quality determination system, a quality determination method, and a teaching data group generation program that can determine the quality of an object under inspection with higher accuracy. [Means for solving the problem]

[0006] The invention described in claim 1 is a quality determination system comprising: a teaching data group generation unit that classifies images containing defects in a plurality of objects to be inspected according to the magnitude of a parameter that distinguishes and characterizes the defects from non-defective parts and generates a plurality of classified teaching data groups; a storage unit that stores a plurality of machine learning models that have learned about the quality of an object to be inspected based on each of the plurality of teaching data groups; a camera unit that captures images of the objects to be inspected; a determination unit that determines the quality of the objects to be inspected as captured by the camera unit, based on the plurality of machine learning models stored in the storage unit; and an optimal model selection unit that evaluates the result of the determination by the determination unit and selects the optimal machine learning model from the plurality of machine learning models.

[0007] The invention described in claim 2 is a quality determination system according to claim 1, wherein the teaching data group generation unit performs the following processes for an image containing the plurality of defects: finding reference values ​​that fall within the range of the magnitude of the parameter characterizing the defect from a predetermined set of reference values; and duplicating the same number of images containing the defect as the number of feature values ​​obtained, and classifying the duplicated images into different teaching data groups corresponding to the feature values.

[0008] The invention described in claim 3 is the quality determination system described in claim 2, wherein the parameter is brightness, hue, saturation, lightness, R value, G value, or B value.

[0009] The invention described in claim 4 is a method for determining whether an object is good or bad, comprising the steps of: classifying images containing defects in a plurality of objects to be inspected according to the magnitude of a parameter that distinguishes and characterizes the defects from non-defective parts, and generating a plurality of classified teaching data sets; preparing a plurality of machine learning models that have learned about the goodness or badness of an object to be inspected based on each of the plurality of teaching data sets; determining whether an object to be inspected is good or bad based on each of the plurality of machine learning models; evaluating the result of the determination and selecting the optimal machine learning model from the plurality of machine learning models; and determining whether an object to be inspected is good or bad using the selected optimal machine learning model.

[0010] The invention described in claim 5 is a teaching data group generation program that classifies images containing defects in a plurality of objects under inspection according to the magnitude of a parameter that distinguishes and characterizes the defects from non-defective parts, and generates a plurality of classified teaching data groups, wherein the teaching data group generation program causes a computer to function as a means for causing a computer to perform a process of determining, for the images containing defects, a reference value that falls within the range of the magnitude of the parameter that characterizes the defects from a plurality of predetermined reference values, as a feature value; and a means for causing a computer to perform a process of duplicating the same number of images containing defects as the number of obtained feature values, and classifying the duplicated images into different teaching data groups according to the corresponding feature values. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a quality determination system, a quality determination method, and a teaching data group generation program that can determine the quality of an object under inspection with higher accuracy. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram showing the configuration of a quality determination system according to the first embodiment of the present invention. [Figure 2] This is an explanatory diagram of the teaching data generation device included in the same quality judgment system. [Figure 3] This is a diagram illustrating the server used in the quality judgment system. [Figure 4] It is an explanatory diagram of a teaching data group generated by a pass / fail determination system and a plurality of machine learning models respectively constructed by these teaching data groups. [Figure 5] It is an explanatory diagram of an image including a defect of an inspection object. [Figure 6] It is an explanatory diagram of a horizontal gray value profile graph schematically shown for a part of a scanning position. [Figure 7A] It is a horizontal gray value profile graph of Image A schematically shown for a part of a scanning position. [Figure 7B] It is a horizontal gray value profile graph of Image B schematically shown for a part of a scanning position. [Figure 7C] It is a horizontal gray value profile graph of Image C schematically shown for a part of a scanning position. [Figure 8] It is an explanatory diagram of preprocessing by a pass / fail determination system. [Figure 9] It is an explanatory diagram of a terminal and an imaging unit included in a pass / fail determination system. [Figure 10] It is a flowchart showing the operation of a pass / fail determination system. [Figure 11] It is a configuration diagram of a pass / fail determination system according to the second embodiment of the present invention. [Figure 12] It is an explanatory diagram of a terminal and an imaging unit included in a pass / fail determination system. [Figure 13] [[ID=三五]]It is a configuration diagram of a predictive maintenance system according to the third embodiment of the present invention. [Figure 14] It is an explanatory diagram of a server included in the predictive maintenance system. [Figure 15] It is an explanatory diagram of a terminal and a sound source visualization device included in the predictive maintenance system. [Figure 16] It is a flowchart showing the operation of the predictive maintenance system. [Figure 17] It is a configuration diagram of a predictive maintenance system according to the fourth embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0013] Next, embodiments of the present invention will be described with reference to the attached drawings to facilitate understanding of the invention. Note that parts of the drawings that are not relevant to the explanation may be omitted.

[0014] [First Embodiment] The quality determination system 10a (see Figure 1) according to the first embodiment of the present invention can determine the quality of the appearance of an object to be inspected 14, which is a product manufactured by the user, using machine learning. The quality of the appearance is determined by the presence or absence of defects such as scratches or foreign matter. The items to be inspected 14 include, for example, parts of transportation machinery such as automobiles and aircraft, and food products. However, the items to be inspected 14 are not limited to these parts and food products. This quality determination service using the quality determination system 10a is provided to users by the service provider.

[0015] As shown in Figure 1, the pass / fail judgment system 10a includes a teaching data creation device 20, a server 30, and an inspection system 40a. The teaching data creation device 20, the server 30, and the inspection system 40a are connected to each other via the Internet N.

[0016] The teaching data creation device 20 is, for example, a personal computer. The teaching data creation device 20 is managed by a service provider or a user using the inspection system 40a, and has a teaching data creation unit 202, as shown in Figure 2. The teaching data creation unit (an example of a teaching data creation means) 202 can, for example, acquire an image of the object to be inspected 14 from a camera 460 installed in the inspection system 40a and create an image of the object to be inspected 14 as teaching data for building a machine learning model. The teaching data creation device 20 functions as a teaching data creation means through a program executed on the teaching data creation device 20. The teaching data creation device 20 may be a mobile device such as a smartphone.

[0017] Server 30 is managed by the service provider and, as shown in Figure 3, has a teaching data group generation unit 302, an optimal model selection unit 306, and a management unit 308. As shown in Figure 4, the teaching data group generation unit (an example of teaching data group generation means) 302 preprocesses the images IMG1, IMG2, IMG3, ... (teaching data group TD) of the object under inspection 14 containing multiple different defects, created by the teaching data creation device 20, to create a set of preprocessed teaching data groups TDg, i.e., teaching data groups TD1, TD2, TD3, TD4, ....

[0018] Here, when images IMG1, IMG2, IMG3, ... of the inspected object 14 containing the defect D centered at position (Xd, Yd) are captured by, for example, a 1.3-megapixel camera, their size will be 1024px vertically (Y-axis direction) and 1280px horizontally (X-axis direction), as shown in Figure 5. This image is represented by a horizontal grayscale profile graph, as shown in Figure 6. Here, the horizontal grayscale profile graph is a graph that shows the brightness corresponding to the scanning position as the pixels of the image are scanned sequentially, with the horizontal axis representing the scanning position and the vertical axis representing the brightness. The scanning direction of the pixels is, for example, from the top left to the right of the image (positive X-axis direction), as shown by the arrow in Figure 5, and this is repeated downwards (positive Y-axis direction) until all pixels are scanned.

[0019] The horizontal grayscale profile graph shown in Figure 6 indicates that the brightness of defect D is brighter than that of the non-defective area. However, the brightness of defect D is not always brighter than that of the non-defective area; it can sometimes be darker. Thus, the brightness of the pre-identified defect D has different characteristics from the brightness of the non-defective areas, and the brightness (an example of a parameter) distinguishes and characterizes the defect D from the non-defective areas. Note that the horizontal Gray value profile graph shown in Figure 6 only shows a portion of the scanning area (near defect D), with the omitted area indicated by a dashed line. The same applies to Figures 7A to 7C, which will be discussed later.

[0020] The preprocessing performed by the teaching data group generation unit 302 includes the following processes P1 and P2. As shown in Figure 6, processing P1 is a process in which the teaching data group generation unit 302, for each of the images IMG1, IMG2, IMG3, ... (teaching data group TD) of the multiple objects under inspection 14 that have been previously captured by the teaching data creation device 20, compares the brightness of the pixels that characterize the defects D whose positions have been previously identified with a plurality of predetermined thresholds of different sizes (an example of a reference value that serves as a basis for determining the range of brightness) TH1 to TH10, and determines the plurality of thresholds that fall within the range of brightness of the pixels that characterize the defects D as feature values ​​that represent the brightness that characterizes the defects D. Process P2 is a process in which the teaching data group generation unit 302 classifies each image IMG1, IMG2, IMG3, ... (teaching data group TD) into teaching data groups TD1, TD2, TD3, TD4, ... according to the multiple feature values ​​obtained in process P1. However, in this process, the teaching data group generation unit 302 does not classify a single image into any one of the teaching data groups TD1, TD2, TD3, TD4, ..., but rather prepares the same number of images as the number of feature values ​​by copying the original image, and classifies them into teaching data groups TD1, TD2, TD3, TD4, ... corresponding to each threshold. Therefore, this preprocessing generates teaching data sets TD1, TD2, TD3, TD4, ... from the teaching data set TD, as shown in Figure 4.

[0021] Next, specific examples of this preprocessing will be explained based on images A, B, C, etc., which are examples of images IMG1, IMG2, and IMG3 of the object to be inspected 14. Image A, corresponding to image IMG1, includes a defect D1 whose location has been predetermined, as shown in Figure 7A, and the brightness range that characterizes defect D1 is 155 to 205. Image B, which corresponds to image IMG2, includes a defect D2 whose location has been predetermined, as shown in Figure 7B, and the brightness range that characterizes defect D2 is 165 to 215. Image C, which corresponds to image IMG3, includes a defect D3 whose location has been predetermined, as shown in Figure 7C, and the brightness range that characterizes defect D3 is 155 to 230.

[0022] In the aforementioned process P1, first, several thresholds (examples of reference values) 20, 40, 60, 80, 100, 120, 140, 160, 180, 200, 220, and 240 are set. For image A (see Figure 7A), three thresholds 160, 180, and 200, which fall within the brightness range 155-205 that characterizes defect D1, are obtained as feature values.

[0023] In the aforementioned process P2, by copying image A, the same number of images as the number of feature values, i.e., three images of image A, are prepared, and each image is classified into teaching data group TD1 corresponding to feature value 160, teaching data group TD2 corresponding to feature value 180, and teaching data group TD3 corresponding to feature value 200, as shown in Figure 8.

[0024] The processes P1 and P2 described above are also performed on the remaining images B, C, etc. Specifically, for image B (see Figure 7B), two thresholds 180 and 200, which fall within the brightness range of 165-215 that characterizes defect D2, are obtained as feature values, and image B is classified into teaching data group TD2 corresponding to feature value 180 and teaching data group TD3 corresponding to feature value 200 (see Figure 8). For image C (see Figure 7C), four thresholds 160, 180, 200, and 220, which fall within the brightness range of 155-230 that characterizes defect D3, are obtained as feature values. Image C is then classified into teaching data group TD1 corresponding to feature value 160, teaching data group TD2 corresponding to feature value 180, teaching data group TD3 corresponding to feature value 200, and teaching data group TD4 corresponding to feature value 220 (see Figure 8). Furthermore, the remaining images included in the instruction data set TD (images other than images A, B, and C) are also classified into corresponding instruction data sets TD1, TD2, TD3, TD4, ... according to the obtained feature values, and a set of instruction data sets TDg (see Figure 4) is generated.

[0025] These teaching data sets TD1, TD2, TD3, TD4, ..., classified according to the brightness that characterizes the defects, serve as teaching data sets for constructing machine learning models M1, M2, M3, M4, ..., which determine whether the object under inspection 14 is good or bad.

[0026] In preprocessing, images IMG1, IMG2, IMG3, ... (teaching data group TD) are not limited to being classified into teaching data groups TD1, TD2, TD3, TD4, ... based on brightness, but may also be classified based on parameters other than brightness. Examples of parameters other than brightness include hue, saturation, and value that constitute the HSV color space, and R (Red), G (Green), and B (Blue) expressed in gradation in the RGB color model. In other words, the parameters can be arbitrary as long as they can characterize the defective parts in distinction from the non-defective parts. Furthermore, pre-processing may include filtering to make the defective areas stand out more than the non-defective areas.

[0027] Here, the machine learning model building service 80 shown in Figures 1 and 4 is provided as a cloud computing service and can build a trained machine learning model based on uploaded training data. This machine learning model building service 80 is, for example, Cloud AutoML Vision provided on Google Cloud Platform (GCP).

[0028] The optimal model selection unit (an example of an optimal model selection means) 306 (see Figure 3) evaluates the results of determining whether the object under inspection 14 is good or bad based on multiple trained machine learning models constructed by the machine learning model construction service 80, and can select the optimal machine learning model.

[0029] The management unit (an example of a management means) 308 can manage the status of the inspection system 40a used by the user. Specifically, the management unit 308 can record operational information regarding the operating status of the imaging unit 460 (see Figure 1) of the inspection system 40a. This operational information will be described later.

[0030] Furthermore, the server 30 functions as a means for generating teaching data sets, an optimal model selection means, and a management means, through a program executed within the server 30.

[0031] As shown in Figure 1, the inspection system 40a includes a terminal 440, a PLC (Programmable Logic Controller) 450, and an imaging unit (an example of a judgment device) 460 that images the object to be inspected 14 and determines whether it is good or bad. The terminal 440, PLC 450, and imaging unit 460 are connected to each other by wired or wireless communication.

[0032] Terminal 440 is managed by the user. Terminal 440 may be a mobile device such as a personal computer or smartphone, or it may be a higher-level controller of PLC450. As shown in Figure 9, terminal 440 includes a machine learning model receiving unit 440a, a control unit 440b, and an operating status output unit 440c.

[0033] The machine learning model receiving unit (an example of a receiving means) 440a can download each machine learning model constructed by the machine learning model construction service 80. The download of each machine learning model is performed via secure communication.

[0034] The control unit (an example of a control means) 440b can control the PLC 450 and the imaging unit 460.

[0035] The operating status output unit (an example of an operating status output means) 440c can output operating information relating to the operating status of the imaging unit 460. This operating information is, for example, information about the time from when the imaging unit 460 starts capturing images until it finishes capturing them. The operating information may also be information about the number of images captured by the imaging unit 460.

[0036] Furthermore, terminal 440 functions as a receiving means, a control means, and an operating status output means, depending on the program executed within terminal 440. Furthermore, it is not limited to the fact that a single terminal 440 has all of the machine learning model receiving unit 440a, control unit 440b, and operating status output unit 440c; each unit may be divided and exist in multiple terminals connected to each other. Furthermore, terminal 440 may have a teaching data creation unit 202 instead of the teaching data creation device 20 shown in Figure 2.

[0037] The PLC450 is a controller managed by the user, which controls the inspection device 470 that inspects the object to be inspected 14, as shown in Figure 1.

[0038] The imaging unit 460 (see Figure 9) is managed by the user and can capture images of the object under inspection 14. Furthermore, the imaging unit 460 can determine the quality of each image of the object under inspection 14 based on multiple machine learning models. The imaging unit 460 is, for example, a camera equipped with a GPU (Graphics Processing Unit). However, the imaging unit 460 may also be a mobile device with a camera, such as a smartphone. The imaging unit 460 includes a camera unit 460a, a storage unit 460b, and a determination unit 460c.

[0039] The camera unit 460a can capture an image of the object to be inspected 14 and acquire image data. The memory unit 460b can store multiple machine learning models downloaded by the machine learning model receiver unit 440a. The determination unit 460c can determine whether or not the object under inspection 14 has defects, i.e., whether the object under inspection 14 is good or bad, based on the image data of the object under inspection 14 captured by the camera unit 460a and the machine learning model stored in the storage unit 460b. The determination unit 460c can perform calculation processing using the machine learning model at high speed and is configured, for example, with a GPU.

[0040] Next, the operation of the quality determination system 10a (method for determining the quality of the inspected object 14) will be explained based on Figure 10. The quality determination system 10a operates according to the following steps S1 to S9. Of steps S1 to S9, steps S1 to S7 are preparatory steps necessary before the actual quality determination is made, while steps S8 and S9 are the actual quality determination operations in the inspection process before shipment of parts, etc. Furthermore, if possible, steps S1 to S7 may be performed in a different order or in parallel.

[0041] (Step S1) The teaching data creation unit 202 (see Figure 2) of the teaching data creation device 20 (see Figure 1) creates multiple image data of the object under inspection 14, which will become the teaching data group TD shown in Figure 4, based on the image data of the object under inspection 14 captured by the imaging unit 460. The teaching data set TD consists of multiple image data of the object under inspection 14 containing a specified defect and multiple image data sets of the object under inspection 14 that do not contain the specified defect. Examples of defect types include scratches, voids, stains, and foreign matter contamination. However, the defects to be inspected will vary depending on the object under inspection 14. The images of the multiple objects to be inspected 14 that have been created are stored in a storage means (not shown) and transmitted to the server 30 shown in Figure 3. The teaching data set TD may be created manually based on pre-prepared image data of the object under inspection 14, instead of being created by the teaching data creation unit 202.

[0042] (Step S2) As shown in Figure 4, the teaching data group generation unit 302 of the server 30 preprocesses multiple images (teaching data group TD) of the object under inspection 14 generated by the teaching data creation device 20, and generates multiple teaching data groups TD1, TD2, TD3, TD4, ... classified according to feature values. In the preprocessing, for example, multiple thresholds TH1 to TH10 (see Figure 6) for brightness are used. Alternatively, each threshold can be determined by finding the maximum brightness of defect D and dividing this maximum value. Subsequently, the server 30 uploads the pre-processed training data sets TD1, TD2, TD3, TD4, ... to the machine learning model building service 80, either through an operation by the service provider or the user. In this way, since the service provider, rather than the user, performs the preprocessing of the training data which requires expertise, the user can easily introduce the machine learning model-based quality judgment system 10a.

[0043] (Step S3) Based on the uploaded training data set, the machine learning model building service 80 constructs machine learning models M1, M2, M3, M4, etc. The constructed and trained machine learning models are each validated by the optimal model selection unit 306 of server 30 (see Figure 3) to check the accuracy of each machine learning model M1, M2, M3, M4, etc. If the accuracy is worse than a predetermined standard, the process returns to the previous step S2, and the teaching data group generation unit 302 performs further preprocessing, such as applying filtering.

[0044] (Step S4) Upon user operation, the machine learning model receiving unit 440a (see Figure 9) of terminal 440 downloads each trained machine learning model constructed by the machine learning model construction service 80. Each downloaded trained machine learning model is transmitted to the imaging unit 460 via terminal 440.

[0045] (Step S5) The memory unit 460b of the imaging unit 460 stores each machine learning model downloaded by the machine learning model receiving unit 440a.

[0046] (Step S6) The control unit 440b of terminal 440 controls the PLC 450 and the imaging unit 460 to experimentally determine the quality of the inspected object 14 manufactured by the inspection device 470 (see Figure 1). In detail, as a preparatory step for determining whether the inspected item 14 is good or bad before shipment, the camera unit 460a of the imaging unit 460 (see Figure 9) captures an image of the inspected item 14, and the determination unit 460c inspects whether or not defects exist in the inspected item 14 based on the captured image and multiple machine learning models stored in the storage unit 460b. If there are no defects, it is determined to be a good product, and if there are defects, it is determined to be a defective product.

[0047] (Step S7) The experimental judgment results from the previous step S6 are transmitted from the imaging unit 460 to the server 30 via the terminal 440. The optimal model selection unit 306 of the server 30 shown in Figure 3 evaluates the quality of each machine learning model based on the experimental judgment results and selects the optimal machine learning model (hereinafter referred to as the "optimal model"). Information about the selected optimal model is sent from server 30 to terminal 440.

[0048] As mentioned above, steps S7 are preparatory steps necessary before the actual pass / fail judgment can be made. The next step, S8, and subsequent steps involve the actual pass / fail judgment process in the pre-shipment inspection of parts and other components.

[0049] (Step S8) The control unit 440b (see Figure 9) of terminal 440 controls the PLC 450 and the imaging unit 460, and the imaging unit 460 determines whether the object to be inspected 14, which is transported on the conveyor of the inspection device 470 (see Figure 1), is good or bad based on the selected optimal model. In detail, the camera unit 460a (see Figure 9) of the imaging unit 460 captures an image of the object to be inspected 14, and the determination unit 460c inspects whether or not there are defects in the object to be inspected 14 based on the captured image and the optimal model stored in the storage unit 460b. If there are no defects, it is determined to be a good product, and if there are defects, it is determined to be a defective product.

[0050] (Step S9) The terminal 440's operating status output unit 440c (see Figure 9) transmits operating information regarding the operating status of the imaging unit 460 to the server 30. The transmitted operating information is stored in the server 30's memory unit (not shown), and the operating status of the pass / fail judgment system 10a is centrally managed by the server 30.

[0051] Thus, the quality determination system 10a according to this embodiment determines the quality of the object to be inspected 14 using the optimal machine learning model selected from among the multiple machine learning models constructed, thereby obtaining a more accurate determination result. Furthermore, depending on the type of camera unit 460a, the quality determination system 10a can determine the quality of the inspected object 14 in aspects other than its external appearance. For example, if the camera unit 460a is an infrared camera, it is also possible to determine the quality of the internal condition of the inspected object 14.

[0052] [Second Embodiment] Next, a quality determination system 10b according to a second embodiment of the present invention will be described. Components having the same function as those in the quality determination system 10a according to the first embodiment may be denoted by the same reference numerals and their detailed descriptions may be omitted. As shown in Figure 11, the pass / fail judgment system 10b includes a teaching data creation device 20, a server 30, and an inspection system 40b. The inspection system 40b includes a terminal 442, a PLC (Programmable Logic Controller) 450, and an imaging unit 462 for imaging the object to be inspected 14.

[0053] Terminal 442 (an example of a judgment device) is, for example, a personal computer, a smartphone, an MR (Mixed Reality) device for realizing MR, or an AR (Augmented Reality) device for realizing AR. As shown in Figure 12, terminal 442 has a machine learning model receiving unit (an example of receiving means) 440a, a control unit (an example of control means) 440b, an operating status output unit (an example of operating status output means) 440c, a storage unit (an example of storage means) 460b, and a determination unit (an example of determination means) 460c, and can determine the quality of the inspected object based on multiple machine learning models. Furthermore, terminal 442 functions as a receiving means, control means, operating status output means, storage means, and determination means, depending on the program executed within terminal 442.

[0054] The imaging unit 462 has a camera unit 460a.

[0055] In other words, in this quality determination system 10b, the terminal 442 has the storage unit 460b and the determination unit 460c that were present in the imaging unit 460 according to the first embodiment. Furthermore, terminal 442 may have some of the machine learning model receiving unit 440a, control unit 440b, operating status output unit 440c, storage unit 460b, and determination unit 460c, while PLC 450 may have the others. In other words, the inspection system 40b as a whole only needs to have the machine learning model receiving unit 440a, control unit 440b, operating status output unit 440c, storage unit 460b, and determination unit 460c. Furthermore, the machine learning model receiving unit 440a, storage unit 460b, and determination unit 460c may not be located in the inspection system 40b, but rather in the server 30 shown in Figure 11.

[0056] Comparing the quality determination system 10b according to this embodiment with the quality determination system 10a according to the first embodiment, the only difference is that the storage unit 460b and the determination unit 460c, which were located in the imaging unit 460, are located in the terminal 442, as shown in Figures 9 and 12. Therefore, the operation of the pass / fail judgment system 10b is essentially the same as the operation of the pass / fail judgment system 10a (steps S1 to S9), so its explanation will be omitted.

[0057] [Third Embodiment] Next, a predictive maintenance system (an example of a pass / fail judgment system) 10c according to a third embodiment of the present invention will be described. Components having the same function as those in the pass / fail judgment system 10b according to the second embodiment may be denoted by the same reference numerals and their detailed descriptions may be omitted.

[0058] The predictive maintenance system 10c according to this embodiment can determine whether the operational status of the monitored object is good or bad by measuring the sound emitted by the monitored object, and can be applied to predictive maintenance that predicts malfunctions in the monitored object. The objects under monitoring are, for example, mechanical devices, specifically press machines. However, the objects under monitoring are not limited to press machines, as long as they are devices or equipment whose malfunctions can be predicted by sound.

[0059] As shown in Figure 13, the predictive maintenance system includes a sound source visualization device (an example of a visualization device) 500, a teaching data creation device 20, a server 33, and a terminal 443.

[0060] The sound source visualization device 500 has a camera (not shown) for imaging the monitored object 600 and multiple microphones 502 for identifying sound sources emanating from the monitored object 600. It can output a sound source visualization image that visualizes the sound source by superimposing the sound intensity distribution onto a real image of the area around the sound source in real time. This sound intensity distribution is represented as heatmap-like visualized information using different colors corresponding to the magnitude of the sound pressure. The sound source visualization device 500 is sometimes referred to as an acoustic camera.

[0061] As shown in Figure 2, the teaching data creation device 20 has a teaching data creation unit 202 that can capture a sound source visualization image and create an image as teaching data.

[0062] Server 33 is managed by the service provider and, as shown in Figure 14, has a teaching data group generation unit 302, a determination unit 334, an optimal model selection unit 306, and a management unit 308. As shown in Figure 4, the teaching data group generation unit (an example of teaching data group generation means) 302 preprocesses the teaching data group TD created by the teaching data creation device 20 and creates a set of preprocessed teaching data groups TDg, i.e., teaching data groups TD1, TD2, TD3, TD4, ...

[0063] The determination unit (an example of a determination means) 334 can virtually determine whether the operating status of the monitored target 600 is good or bad based on the teaching data group TD and multiple trained machine learning models constructed by the machine learning model construction service 80.

[0064] The optimal model selection unit (an example of an optimal model selection means) 306 evaluates the determination result from the determination unit 334 and can select the optimal machine learning model.

[0065] The management unit (an example of a management means) 308 can manage the status of the terminal 443 or the sound source visualization device 500. More specifically, the management unit 308 can record operational information regarding the operating status of the terminal 443 or the sound source visualization device 500. Furthermore, the server 33 functions as a means for generating teaching data sets, a means for making decisions, a means for selecting the optimal model, and a means for managing the data, through a program executed within the server 33.

[0066] As shown in Figure 15, terminal (an example of a judgment device) 443 is connected to the sound source visualization device 500. Terminal 443 has a machine learning model receiving unit 440a, a control unit 443b, an operating status output unit 443c, a storage unit 443d, and a judgment unit 443e, and can determine whether the monitored object 600 is good or bad based on the optimal model.

[0067] The machine learning model receiving unit (an example of a receiving means) 440a can receive the optimal model selected by the optimal model selection unit 306 from the server 33. The reception of the optimal model is performed via secure communication.

[0068] The control unit (an example of a control means) 443b can control the PLC 450 and the sound source visualization device 500.

[0069] The operating status output unit (an example of an operating status output means) 443c can output operating information relating to the operating status of the terminal 443 or the sound source visualization device 500. This operating information is, for example, information about the time from when the sound source visualization device 500 starts capturing images until it finishes capturing them. The operating information may also be information about the number of images output from the sound source visualization device 500.

[0070] The memory unit (an example of a memory means) 443d can store the optimal model received by the machine learning model receiving unit 400a.

[0071] The determination unit (an example of a determination means) 443e can determine whether the operating status of the monitored object 600 is good or bad based on the multiple sound source visualization images output by the sound source visualization device 500 and the optimal model stored in the storage unit 443d. Furthermore, terminal 443 functions as a receiving means, control means, operating status output means, storage means, and determination means, depending on the program executed within terminal 443.

[0072] Next, the operation of the predictive maintenance system 10c (method for determining whether the operating status of the monitored object 600 is good or bad) will be explained based on Figure 16. The predictive maintenance system 10c operates according to the following steps S3-1 to S3-9. Of steps S3-1 to S3-9, steps S3-1 to S3-7 are preparatory steps, and the following steps S3-8 and S3-9 are the operations for determining whether the operating status of the monitored object 600 is good or bad. Furthermore, if possible, steps S3-1 to S3-7 may be performed in a different order or in parallel.

[0073] (Step S3-1) The teaching data creation unit 202 (see Figure 2) of the teaching data creation device 20 (see Figure 13) takes in the sound source visualization image generated by the sound source visualization device 500 and uses it as teaching data to create the teaching data group TD shown in Figure 4. The teaching data set TD is stored in a storage means (not shown) and transmitted to the server 33 shown in Figure 14. The teaching data set TD may be created manually based on pre-prepared sound source visualization images of the monitored object 600, instead of being created by the teaching data creation unit 202.

[0074] (Step S3-2) As shown in Figure 4, the teaching data group generation unit 302 of server 33 preprocesses the teaching data group TD generated by the teaching data creation device 20 and generates multiple teaching data groups TD1, TD2, TD3, TD4, ... classified according to feature values. In the preprocessing, for example, multiple thresholds TH1 to TH10 (see Figure 6) for brightness are used. Alternatively, each threshold can be determined by finding the maximum brightness of defect D and dividing this maximum value. Subsequently, the server 33 uploads the pre-processed training data sets TD1, TD2, TD3, TD4, ... to the machine learning model building service 80, either through an operation by the service provider or the user. In this way, since the service provider, rather than the user, performs the preprocessing of the teaching data, which requires expertise, users can easily implement a predictive maintenance system using machine learning models.

[0075] (Step S3-3) Based on the uploaded training data set, the machine learning model building service 80 constructs machine learning models M1, M2, M3, M4, etc. The accuracy of each constructed and trained machine learning model is verified by the optimal model selection unit 306 of server 33 (see Figure 14). If the accuracy is poor, the process returns to the previous step S3-2, and the teaching data group generation unit 302 preprocesses the sound source visualization image (teaching data group TD) in a different way, such as by applying a different filter.

[0076] (Step S3-4) Upon user input, server 33 (see Figure 14) downloads each trained machine learning model constructed by the machine learning model building service 80. Each downloaded trained machine learning model is stored in a memory unit (not shown).

[0077] (Steps S3-5) The determination unit 334 of the server 33 inputs the teaching data group TD into each machine learning model stored in the storage unit (not shown) and virtually determines whether the operating status of the monitored object 600 is good or bad.

[0078] (Steps S3-6) The optimal model selection unit 306 evaluates the determination result of the judgment unit 334 in the previous step S3-5 regarding the goodness or badness of the operating status of the monitored object 600, and selects the optimal model from among multiple machine learning models.

[0079] (Steps S3-7) The optimal model selected by the optimal model selection unit 306 is transmitted from the server 33 to the terminal 443. The transmitted optimal model is received by the machine learning model receiver 440a (see Figure 15) and stored in the storage unit 443d.

[0080] (Steps S3-8) Step S3-8 is a step in which the monitored object 600 is monitored. The control unit 443b of terminal 443 controls the PLC 450 to operate the monitored device 600. Meanwhile, the sound source visualization device 500 measures the sound generated from the monitored device 600 and outputs a sound source visualization image at a predetermined interval. Terminal 443 determines whether the operational status of the monitored device 600 is good or bad based on the outputted sound source visualization image and the optimal model stored in the storage unit 443d.

[0081] (Step S3-9) The operational status output unit 443c of terminal 440 transmits operational information regarding the operational status of the sound source visualization device 500 to server 33. The transmitted operational information is stored in the memory unit (not shown) of server 33, and the operational status of the predictive maintenance system 10c is centrally managed by server 33.

[0082] Thus, according to the predictive maintenance system 10c of this embodiment, the operating status of the monitored object 600 is determined using the optimal machine learning model selected from among the multiple machine learning models constructed, thereby enabling predictive maintenance with higher accuracy. Alternatively, instead of the sound source visualization system, any visualization device may be used that has a detector for measuring physical quantities that change due to signs of malfunction occurring in the monitored object 600, and that can generate multiple visualization images of those physical quantities.

[0083] [Fourth Embodiment] Next, a predictive maintenance system (an example of a good / bad judgment system) 10d according to the fourth embodiment of the present invention will be described. Components having the same function as those in the predictive maintenance system 10c according to the third embodiment (see Figure 13) may be given the same reference numerals and their detailed descriptions may be omitted.

[0084] The predictive maintenance system according to this embodiment can determine the operational status of a monitored object by measuring vibrations emitted by the object, and can be applied to predictive maintenance that predicts malfunctions in the monitored object. The monitored objects are, for example, machinery and equipment, specifically presses and conveying devices. However, the monitored objects can be any devices or equipment whose malfunctions can be predicted by vibration.

[0085] As shown in Figure 17, the predictive maintenance system 10d includes a vibration visualization device (an example of a visualization device) 700, a teaching data creation device 20, a server 33, and a terminal 443 (an example of a judgment device). The vibration visualization device 700 has multiple vibration sensors 702 for detecting vibrations generated from the monitored object 600, and can output multiple vibration visualization images that visualize the vibrations detected by each vibration sensor 702. These vibration visualization images are images that represent visualized information, for example, by using different colors corresponding to at least one of the vibration magnitude and frequency.

[0086] Here, in the predictive maintenance system 10d, the vibration visualization device 700 and the vibration visualization image correspond to the sound source visualization device 500 and the sound source visualization image in the third embodiment, respectively.

[0087] By using such a predictive maintenance system 10d and performing the aforementioned operation steps S3-1 to S3-9 (see Figure 16), the determination unit 443e (see Figure 15) of the terminal 443 can determine that a malfunction has occurred in the monitored object 600, and that this is abnormal.

[0088] Thus, according to the predictive maintenance system 10d of this embodiment, the condition of the monitored object 600 is determined using the optimal machine learning model selected from among the multiple machine learning models constructed, thereby enabling predictive maintenance with higher accuracy. Alternatively, instead of the vibration visualization system, any visualization device that has a detector for measuring physical quantities that change due to signs of malfunction in the monitored object 600, and that can generate multiple visualization images of those physical quantities, may be used.

[0089] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and any changes to the conditions, etc., that do not depart from the gist of the invention are all within the scope of application of the present invention. [Explanation of symbols]

[0090] 10a, 10b Quality judgment system 10c, 10d Predictive Maintenance System 14. Items to be inspected 20. Teaching data creation device 30 servers 40a, 40b Inspection System 450 PLC 80 Machine Learning Model Building Service 202 Instructional Data Creation Department 302 Instructional Data Group Generation Unit 306 Optimal Model Selection Section 308 Management Department 334 Judgment section 440 devices 440a Machine Learning Model Receiver 440b Control Unit 440c Operating Status Output Unit Terminals 442 and 443 443b Control Unit 443c Operating status output section 443d Storage section 443e Judgment section 460 Imaging Unit 460a Camera Unit 460b Storage section 460c Judgment part 462 Imaging Unit 470 Inspection equipment 500 Sound source visualization device 502 Microphone 600 Subjects under surveillance 700 Vibration visualization device 702 Vibration Sensor N Internet

Claims

1. A teaching data group generation unit that classifies images containing defects in multiple objects under inspection according to the magnitude of a parameter that distinguishes and characterizes the defects from non-defective parts, and generates multiple classified teaching data groups, A storage unit that stores multiple machine learning models that have learned about the quality of an object under inspection based on each of the aforementioned sets of training data, A camera unit for capturing an image of the object to be inspected, A determination unit that determines whether the object to be inspected, as captured by the camera unit, is good or bad based on the plurality of machine learning models stored in the memory unit, A pass / fail judgment system comprising: an optimal model selection unit that evaluates the result of the judgment by the judgment unit and selects the optimal machine learning model from among the plurality of machine learning models.

2. In the quality determination system according to claim 1, The teaching data group generation unit performs a process to determine, for an image containing the multiple defects, a reference value that falls within the range of the magnitude of the parameter characterizing the defect from a predetermined set of reference values, as a feature value. A quality determination system that performs the following processes: duplicates an image containing the same number of defects as the number of required feature values, and classifies each duplicated image into different teaching data sets corresponding to the feature values.

3. In the quality determination system according to claim 2, A quality determination system in which the aforementioned parameters are brightness, hue, saturation, lightness, R value, G value, or B value.

4. The process involves classifying images containing defects in multiple objects under inspection according to the magnitude of parameters that distinguish and characterize the defects from non-defective areas, and generating multiple classified teaching data sets. The steps include: preparing multiple machine learning models that have learned about the quality of the object being inspected based on each of the aforementioned sets of training data; The steps include determining whether the object under inspection is good or bad based on each of the aforementioned multiple machine learning models, The steps include evaluating the results of the above determination and selecting the optimal machine learning model from among the multiple machine learning models, A method for determining whether an object to be inspected is good or bad, comprising the step of determining whether the object is good or bad using the selected optimal machine learning model.

5. A teaching data generation program that classifies images containing defects in multiple objects under inspection according to the magnitude of a parameter that distinguishes and characterizes the defects from non-defective parts, and generates multiple classified teaching data sets, Computers, A means for performing a process to determine, with respect to an image containing the aforementioned defect, a reference value that falls within the range of the magnitude of the parameter characterizing the defect from a predetermined set of reference values, as a feature value. A teaching data set generation program that functions as a means for performing a process of duplicating the same number of images containing the defects as the number of required feature values, and classifying each of the duplicated images into different teaching data sets corresponding to the feature values.

Images