Predictive Maintenance System

Abstract

A quality determination system, quality determination method, server, and program are provided that can determine the quality of an object to be inspected with higher accuracy. [Solution] The pass / fail judgment system 10a includes a teaching data group generation unit that classifies images of multiple inspected objects containing defects according to the brightness that characterizes the defects and generates multiple classified teaching data groups; a memory unit that stores multiple machine learning models constructed by a machine learning model construction service 80 that constructs machine learning models based on the multiple teaching data groups; a camera unit 460 that captures images of the inspected object; a judgment unit that judges the pass / fail of each inspected object 14 imaged by the camera unit 460 based on the multiple machine learning models stored in the memory unit; and an optimal model selection unit that evaluates the judgment results by the judgment unit and selects the optimal machine learning model from the multiple machine learning models.

Description

[Technical Field]

[0001] The present invention provides Predictive Maintenance Regarding the system. [Background technology]

[0002] Patent Document 1 describes a service provision system that provides services using machine learning based on artificial intelligence. This service provision system includes machine learning means for inputting learning data based on information sent from a user and generating a general model modeled by machine learning, personalization means for personalizing the general model to a model suitable for the user based on the information sent from the user, and service provision means for providing a personalized service to the user using the personalized model, and the information sent from the user is used for both the machine learning and the personalization.

[0003] Patent Document 2 describes a method for visualizing the position of a sound source, which visualizes the position of any sound source in real time by associating it with a real space. This method detects one or more sounds, locates each of the sound sources, converts information about the sound source, including at least the sound source position, into visible information, and displays it superimposed on a real image of the area around the sound source in real time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-48417 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-77277 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention occurs To the sound based on mechanical equipment It is possible to determine whether the operating status of Predictive Maintenance The purpose is to provide a system. [Means for solving the problem]

[0006] Book The invention is A camera that captures images of machinery and measures the sounds generated by the machinery having a microphone, An acoustic camera that outputs a sound source visualization image in which the sound intensity distribution is superimposed in real time on an actual image of the area around the sound source of the machine. and, a teaching data generation unit that generates teaching data from the sound source visualization images output by the acoustic camera in a state where a malfunction has occurred in the mechanical device and a state where no malfunction has occurred; The aforementioned Teaching Data It is constructed from the above and learns whether the operating condition of the machine is good or bad. a memory unit that stores the machine learning model; before Based on the above machine learning model, The sound source visualization image of the machine in operation output by the acoustic camera is used to Judgment to determine whether the operating status is good or bad Device and equipped with Predictive Maintenance It is a system.

[0007] [Effects of the Invention]

[0008] According to the present invention, To the sound based on mechanical equipment It is possible to determine whether the operating status of Predictive Maintenance The system can be provided. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram of a quality determination system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is an explanatory diagram of a teaching data creation device provided in the quality determination system. [Figure 3] FIG. 2 is an explanatory diagram of a server provided in the quality determination system. [Figure 4] 1 is an explanatory diagram of a group of teaching data generated by the quality determination system and a plurality of machine learning models constructed from the teaching data groups. FIG. [Figure 5] FIG. 1 is an explanatory diagram of an image including a defect in an object to be inspected. [Figure 6]FIG. 10 is a diagram illustrating a horizontal gray value profile graph schematically illustrating a portion of the scanning position. [Figure 7A] 10 is a horizontal gray value profile graph of image A shown schematically for some of the scan positions. [Figure 7B] 10 is a horizontal gray value profile graph of image B shown schematically for some of the scan positions; [Figure 7C] 10 is a horizontal gray value profile graph of image C shown schematically for some of the scan positions; [Figure 8] FIG. 2 is an explanatory diagram of preprocessing by the quality determination system. [Figure 9] 2 is an explanatory diagram of a terminal and an imaging unit provided in the quality determination system. FIG. [Figure 10] FIG. 2 is a flow chart showing the operation of the quality determination system. [Figure 11] FIG. 10 is a configuration diagram of a quality determination system according to a second embodiment of the present invention. [Figure 12] 2 is an explanatory diagram of a terminal and an imaging unit provided in the quality determination system. FIG. [Figure 13] FIG. 10 is a configuration diagram of a predictive maintenance system according to a third embodiment of the present invention. [Figure 14] FIG. 2 is an explanatory diagram of a server included in the predictive maintenance system. [Figure 15] FIG. 2 is an explanatory diagram of a terminal and a sound source visualization device provided in the predictive maintenance system. [Figure 16] FIG. 2 is a flowchart showing the operation of the predictive maintenance system. [Figure 17] FIG. 10 is a configuration diagram of a predictive maintenance system according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Next, embodiments of the present invention will be described with reference to the accompanying drawings to facilitate understanding of the present invention. Note that in the drawings, parts that are not relevant to the description may be omitted.

[0011] [First embodiment] A quality determination system 10a (see FIG. 1) according to a first embodiment of the present invention can use machine learning to determine whether the appearance of an object to be inspected 14, which is a product manufactured by a user, is good or bad. The quality of this appearance is determined based on the presence or absence of defects such as scratches or foreign matter. The object 14 to be inspected is, for example, a part of a transport machine such as an automobile or an aircraft, or food, but the object 14 to be inspected is not limited to these parts and food. The quality determination service provided by this quality determination system 10a is provided to users by a service provider.

[0012] As shown in FIG. 1, the quality determination system 10a includes a teaching data creation device 20, a server 30, and an inspection system 40a. The teaching data generating device 20, the server 30, and the inspection system 40a are connected to each other via the Internet N.

[0013] The teaching data creation device 20 is, for example, a personal computer. The teaching data creation device 20 is managed by a service provider or a user who uses the inspection system 40a, and includes a teaching data creation unit 202 as shown in FIG. The teaching data creation unit (an example of teaching data creation means) 202 can acquire an image of the object to be inspected 14, for example, from a camera 460 provided in the inspection system 40a, and create the image of the object to be inspected 14 as teaching data for constructing a machine learning model. The teaching data creating device 20 functions as a teaching data creating means by a program executed by the teaching data creating device 20. The teaching data creation device 20 may be a mobile terminal such as a smartphone.

[0014] The server 30 is managed by a service provider, and includes a teaching data group generation unit 302, an optimum model selection unit 306, and a management unit 308, as shown in FIG. The teaching data group generation unit (an example of a teaching data group generation means) 302 preprocesses images IMG1, IMG2, IMG3, ... (teaching data groups TD) of the object to be inspected 14 containing multiple different defects created by the teaching data creation device 20, as shown in Figure 4, to create a set TDg of preprocessed teaching data groups, i.e., teaching data groups TD1, TD2, TD3, TD4, ...

[0015] Here, when each of images IMG1, IMG2, IMG3, ... of the object to be inspected 14 including defect D centered at position (Xd, Yd) is captured by, for example, a 1.3 megapixel camera, the size of each image will be 1024px vertically (Y-axis direction) and 1280px horizontally (X-axis direction), as shown in Fig. 5. This image is represented by a horizontal gray value profile graph as shown in Fig. 6. Here, the horizontal gray value profile graph is a graph that shows the brightness corresponding to the scanning position when the pixels of the image are scanned in order, with the horizontal axis representing the scanning position and the vertical axis representing the brightness. The pixel scanning direction is, for example, as shown by the arrow in Figure 5, from the upper left of the image to the right (positive direction of the X axis), and this is repeated downward (positive direction of the Y axis) until all the pixels are scanned.

[0016] The horizontal gray value profile graph shown in Figure 6 shows that the brightness of defect D is brighter than that of non-defective areas. However, the brightness of defect D is not always brighter than that of non-defective areas, and there are cases where it is darker than that of non-defective areas. In this way, the brightness of the pre-specified defect D has characteristics different from the brightness of the non-defective portion, and the defect D is distinguished from and characterized by the brightness (an example of a parameter). 6 shows only a portion of the scanning position (near the defect D), and the omitted range is indicated by a dashed line. The same applies to FIGS. 7A to 7C described later.

[0017] The pre-processing performed by the teaching data group generation unit 302 includes the following processes P1 and P2. As shown in Figure 6, process P1 is a process in which the teaching data group generation unit 302 compares the brightness of pixels characterizing a defect D, the position of which has been previously identified, for each of images IMG1, IMG2, IMG3, ... (teaching data group TD) of multiple inspection objects 14 previously captured by the teaching data creation device 20 with multiple thresholds TH1 to TH10 of different predetermined magnitudes (examples of reference values ​​that serve as standards for determining the range of brightness), and obtains multiple thresholds included in the range of brightness of the pixels characterizing the defect D as feature values ​​representing the brightness characterizing the defect D. In process P2, the teaching data group generation unit 302 classifies each image IMG1, IMG2, IMG3, ... (teaching data group TD) into teaching data groups TD1, TD2, TD3, TD4, ... according to the plurality of feature values ​​obtained in process P1. However, in this case, the teaching data group generation unit 302 does not classify one image into one of the teaching data groups TD1, TD2, TD3, TD4, ..., but instead copies the original image to prepare the same number of images as the number of feature values, and classifies them into teaching data groups TD1, TD2, TD3, TD4, ... corresponding to each threshold value. Therefore, by this preprocessing, teaching data groups TD1, TD2, TD3, TD4, . . . are generated from the teaching data group TD, as shown in FIG.

[0018] Next, a specific example of this pre-processing will be described based on images A, B, C, etc., which are examples of images IMG1, IMG2, IMG3 of the object 14 to be inspected. Image A corresponding to image IMG1 includes defect D1 whose position has been specified in advance as shown in FIG. 7A, and the brightness range characterizing defect D1 is 155-205. Image B corresponding to image IMG2 includes defect D2 whose position has been specified in advance as shown in FIG. 7B, and the brightness range characterizing defect D2 is 165-215. Image C corresponding to image IMG3 includes defect D3 whose position has been specified in advance as shown in FIG. 7C, and the brightness range characterizing defect D3 is 155-230.

[0019] In the above-mentioned process P1, first, multiple thresholds (examples of reference values) 20, 40, 60, 80, 100, 120, 140, 160, 180, 200, 220, and 240 are set, and for image A (see Figure 7A), three thresholds 160, 180, and 200, which fall within the brightness range 155 to 205 that characterize defect D1, are obtained as feature values.

[0020] In the above-mentioned process P2, image A is copied to prepare three images A, the same number as the number of feature values, and each image is classified into a teaching data group TD1 corresponding to the feature value 160, a teaching data group TD2 corresponding to the feature value 180, and a teaching data group TD3 corresponding to the feature value 200, as shown in Figure 8.

[0021] The above-described processes P1 and P2 are also performed on the remaining images B, C, etc. That is, for image B (see Figure 7B), two thresholds 180 and 200, which are within the brightness range 165 to 215 that characterize defect D2, are obtained as feature values, and image B is classified into teaching data group TD2 corresponding to feature value 180 and teaching data group TD3 corresponding to feature value 200 (see Figure 8). For image C (see Figure 7C), four thresholds 160, 180, 200, and 220, which are included in the brightness range 155 to 230 that characterize defect D3, are obtained as feature values, and image C is classified into teaching data group TD1 corresponding to feature value 160, teaching data group TD2 corresponding to feature value 180, teaching data group TD3 corresponding to feature value 200, and teaching data group TD4 corresponding to feature value 220 (see Figure 8). Furthermore, the remaining images included in the teaching data group TD (images other than images A, B, and C) are classified into corresponding teaching data groups TD1, TD2, TD3, TD4, ... according to the obtained feature values, and a set of teaching data groups TDg (see Figure 4) is generated.

[0022] These teaching data groups TD1, TD2, TD3, TD4, ... classified according to the brightness that characterizes the defects, become teaching data groups for constructing machine learning models M1, M2, M3, M4, ... that determine the pass / fail of the object 14 to be inspected.

[0023] In the preprocessing, the images IMG1, IMG2, IMG3, ... (teaching data groups TD) are not limited to being classified into teaching data groups TD1, TD2, TD3, TD4, ... based on brightness, but may be classified based on parameters other than brightness. Examples of parameters other than brightness include hue, saturation, and value that constitute the HSV color space, and R (Red), G (Green), and B (Blue) expressed as gradations in the RGB color model. In other words, any parameters may be used as long as they can characterize defective portions and distinguish them from non-defective portions. Pre-processing may also include filtering to make defective parts stand out more than non-defective parts.

[0024] 1 and 4 is provided as a cloud computing service, and can build a trained machine learning model based on uploaded teaching data. This machine learning model building service 80 is, for example, Cloud AutoML Vision provided by Google Cloud Platform (GCP).

[0025] The optimal model selection unit (an example of an optimal model selection means) 306 (see Figure 3) can evaluate the pass / fail judgment results of the inspected object 14 based on multiple trained machine learning models constructed by the machine learning model construction service 80, and select the optimal machine learning model.

[0026] The management unit (an example of a management means) 308 can manage the status of the inspection system 40a used by the user. In detail, the management unit 308 can record operation information relating to the operation status of the imaging unit 460 (see FIG. 1) included in the inspection system 40a. This operation information will be described later.

[0027] The server 30 functions as a teaching data group generating means, an optimum model selecting means, and a managing means by means of a program executed within the server 30.

[0028] 1, the inspection system 40a includes a terminal 440, a PLC (Programmable Logic Controller) 450, and an imaging unit (an example of a determination device) 460 that captures an image of the inspection object 14 and determines whether it is good or bad. The terminal 440, the PLC 450, and the imaging unit 460 are connected to each other via wired communication or wireless communication.

[0029] The terminal 440 is managed by a user. The terminal 440 is, for example, a personal computer, a mobile terminal such as a smartphone, or the like, and may be a higher-level controller of the PLC 450. As shown in FIG. 9, the terminal 440 includes a machine learning model receiving unit 440a, a control unit 440b, and an operating status output unit 440c.

[0030] The machine learning model receiving unit (an example of a receiving means) 440a can download each machine learning model built by the machine learning model building service 80. Note that each machine learning model is downloaded via secure communication.

[0031] The control unit (an example of a control means) 440b can control the PLC 450 and the imaging unit 460.

[0032] The operating status output unit (an example of an operating status output means) 440c can output operating information relating to the operating status of the imaging unit 460. This operating information is, for example, information on the time from when the imaging unit 460 starts capturing images to when it finishes capturing images. The operating information may also be information on the number of images captured by the imaging unit 460.

[0033] The terminal 440 functions as a receiving means, a control means, and an operating status output means by a program executed inside the terminal 440. Furthermore, one terminal 440 is not limited to having all of the machine learning model receiving unit 440a, control unit 440b, and operating status output unit 440c, and each unit may exist separately on multiple terminals connected to each other. Furthermore, the terminal 440 may have a teaching data creating unit 202 instead of the teaching data creating device 20 shown in FIG.

[0034] The PLC 450 is a controller that is managed by a user and controls an inspection device 470 that inspects the object under inspection 14, as shown in FIG.

[0035] The imaging section 460 (see FIG. 9) is managed by the user and can capture images of the inspection object 14. The imaging section 460 can also determine the acceptability of each image of the inspection object 14 based on a plurality of machine learning models. The imaging unit 460 is, for example, a camera equipped with a GPU (Graphics Processing Unit), but the imaging unit 460 may also be a mobile terminal with a camera, such as a smartphone. The imaging unit 460 includes a camera unit 460a, a storage unit 460b, and a determination unit 460c.

[0036] The camera section 460a can capture an image of the inspection object 14 and acquire image data. The storage unit 460b can store a plurality of machine learning models downloaded by the machine learning model receiving unit 440a. The determination unit 460c can determine whether or not the object 14 has a defect, i.e., whether or not the object 14 is good, based on the image data of the object 14 captured by the camera unit 460a and the machine learning model stored in the storage unit 460b. The determination unit 460c can perform calculation processing using the machine learning model at high speed, and is configured by, for example, a GPU.

[0037] Next, the operation of the quality determination system 10a (a method for determining the quality of the object 14 to be inspected) will be described with reference to Fig. 10. The quality determination system 10a operates in accordance with the following steps S1 to S9. Of steps S1 to S9, steps S1 to S7 are operations as preparatory steps required before the actual quality determination is made, and the subsequent steps S8 and S9 are operations for the actual quality determination in the inspection process before shipping of parts, etc. If possible, steps S1 to S7 may be performed in a different order or in parallel.

[0038] (Step S1) The teaching data creation unit 202 (see Figure 2) of the teaching data creation device 20 (see Figure 1) creates multiple image data of the object to be inspected 14, which become the teaching data group TD shown in Figure 4, based on image data of the object to be inspected 14 captured by the imaging unit 460. The teaching data group TD is a group of a plurality of image data of the object 14 to be inspected that includes a specified defect and a plurality of image data of the object 14 to be inspected that does not include the specified defect. Note that, for example, types of defects include scratches, voids, stains, and foreign matter contamination. However, the defects to be inspected differ depending on the object 14 to be inspected. The created images of the plurality of objects to be inspected 14 are stored in a storage means (not shown) and transmitted to the server 30 shown in FIG. Instead of being created by the teaching data creating unit 202, the teaching data group TD may be created manually based on image data of the inspection object 14 that has been prepared in advance.

[0039] (Step S2) 4, the teaching data group generation unit 302 of the server 30 preprocesses a plurality of images (teaching data group TD) of the inspection object 14 generated by the teaching data creation device 20, and generates a plurality of teaching data groups TD1, TD2, TD3, TD4, ... classified according to feature values. In the preprocessing, for example, a plurality of threshold values ​​TH1 to TH10 for brightness (see FIG. 6) are used. Each threshold value may be obtained by finding the maximum value of the brightness of the defect D and dividing this maximum value. Thereafter, the server 30 uploads the preprocessed teaching data groups TD1, TD2, TD3, TD4, . . . to the machine learning model building service 80 in response to an operation by the service provider or the user. In this way, the preprocessing of teaching data, which requires know-how, is performed by the service provider, not the user, so the user can easily introduce the pass / fail judgment system 10a using a machine learning model.

[0040] (Step S3) Based on the uploaded teaching data groups, machine learning models M1, M2, M3, M4, . . . are constructed by the machine learning model construction service 80. The accuracy of each of the constructed trained machine learning models M1, M2, M3, M4, . . . is verified by the optimal model selection unit 306 of the server 30 (see FIG. 3). If the accuracy is lower than a predetermined standard, the process returns to the previous step S2, and the teaching data group generating unit 302 performs preprocessing by further applying filtering or the like.

[0041] (Step S4) In response to a user operation, the machine learning model receiving unit 440a (see FIG. 9) of the terminal 440 downloads each trained machine learning model constructed by the machine learning model construction service 80. Each downloaded trained machine learning model is transmitted to the imaging unit 460 via the terminal 440.

[0042] (Step S5) The storage unit 460b of the imaging unit 460 stores each machine learning model downloaded by the machine learning model receiving unit 440a.

[0043] (Step S6) The control unit 440b of the terminal 440 controls the PLC 450 and the imaging unit 460, and judges experimentally whether the inspection object 14 manufactured by the inspection device 470 (see FIG. 1) is good or bad. In detail, as a preparation stage for determining whether the object 14 to be inspected is good or bad when it is shipped, the camera unit 460a (see Figure 9) of the imaging unit 460 captures an image of the object 14 to be inspected, and the judgment unit 460c inspects whether or not there are defects in each object 14 to be inspected based on the captured image and multiple machine learning models stored in the memory unit 460b, and judges the object 14 to be good if there is no defect, and judges the object to be defective if there is a defect.

[0044] (Step S7) The test determination results in the previous step S6 are transmitted from the imaging unit 60 to the server 30 via the terminal 440. The optimum model selection unit 306 included in the server 30 shown in FIG. 3 evaluates the quality of each machine learning model based on each test judgment result, and selects the optimum machine learning model (hereinafter referred to as the "optimum model"). Information on the selected optimum model is transmitted from the server 30 to the terminal 440 .

[0045] As described above, the operations up to step S7 are preparatory steps required before the actual pass / fail determination is made. The next step S8 and subsequent steps are the operation for actually determining whether the parts are good or bad in the inspection process before shipping.

[0046] (Step S8) The control unit 440b (see Figure 9) of the terminal 440 controls the PLC 450 and the imaging unit 460, and the imaging unit 60 judges the quality of the inspection object 14 transported on the conveyor of the inspection device 470 (see Figure 1) based on the selected optimal model. In detail, the camera unit 460a (see FIG. 9) of the imaging unit 460 captures an image of the object 14 to be inspected, and the judgment unit 460c inspects whether or not there is a defect in the object 14 to be inspected based on the captured image and the optimal model stored in the memory unit 460b, and judges the object 14 to be a good product if there is no defect, and judges the object 14 to be a defective product if there is a defect.

[0047] (Step S9) An operating status output unit 440c (see FIG. 9) of the terminal 440 transmits operating information relating to the operating status of the imaging unit 460 to the server 30. The transmitted operating information is stored in a storage unit (not shown) of the server 30, and the operating status of the quality determination system 10a is managed in an integrated manner by the server 30.

[0048] In this way, the pass / fail judgment system 10a according to this embodiment judges the pass / fail of the inspection object 14 using the optimal machine learning model selected from the multiple machine learning models that have been constructed, thereby obtaining more accurate judgment results. Depending on the type of the camera unit 460a, the quality determination system 10a can determine the quality of the state of the object 14 other than the external appearance. For example, if the camera unit 460a is an infrared camera, it is also possible to determine the quality of the internal state of the object 14.

[0049] Second Embodiment Next, a quality determination system 10b according to a second embodiment of the present invention will be described. Components having the same functions as those of the quality determination system 10a according to the first embodiment will be assigned the same reference numerals, and detailed descriptions thereof will be omitted. As shown in FIG. 11, the quality determination system 10b includes a teaching data generating device 20, a server 30, and an inspection system 40b. The inspection system 40b includes a terminal 442, a PLC (Programmable Logic Controller) 450, and an imaging unit 462 that captures an image of the object 14 to be inspected.

[0050] The terminal 442 (an example of a determination device) is, for example, a personal computer, a smartphone, an MR device for realizing MR (Mixed Reality), or an AR device for realizing AR (Augmented Reality). As shown in FIG. 12, the terminal 442 has a machine learning model receiving unit (an example of a receiving means) 440a, a control unit (an example of a control means) 440b, an operating status output unit (an example of an operating status output means) 440c, a memory unit (an example of a memory means) 460b, and a judgment unit (an example of a judgment means) 460c, and can judge the pass / fail of each inspected object based on multiple machine learning models. The terminal 442 functions as a receiving means, a control means, an operating status output means, a storage means, and a determination means by a program executed inside the terminal 442.

[0051] The imaging unit 462 has a camera unit 460a.

[0052] That is, in this quality determination system 10b, the terminal 442 has the storage unit 460b and the determination unit 460c that the imaging unit 460 according to the first embodiment has. The terminal 442 may have some of the machine learning model receiving unit 440a, the control unit 440b, the operating status output unit 440c, the memory unit 460b, and the determination unit 460c, and the PLC 450 may have the rest. That is, it is sufficient if the inspection system 40b as a whole has the machine learning model receiving unit 440a, the control unit 440b, the operating status output unit 440c, the memory unit 460b, and the determination unit 460c. Furthermore, the machine learning model receiving unit 440a, the storage unit 460b, and the determination unit 460c may be included in the server 30 shown in FIG. 11, rather than in the inspection system 40b.

[0053] Comparing the quality determination system 10b according to this embodiment with the quality determination system 10a according to the first embodiment, the only difference is that the memory unit 460b and the determination unit 460c, which were provided in the imaging unit 460, are provided in the terminal 442, as shown in FIGS. 9 and 12. Therefore, the operation of the quality determination system 10b is substantially the same as the operation of the quality determination system 10a (steps S1 to S9), and therefore a description thereof will be omitted.

[0054] Third Embodiment Next, a predictive maintenance system (an example of a quality determination system) 10c according to a third embodiment of the present invention will be described. Components having the same functions as those in the quality determination system 10b according to the second embodiment will be assigned the same reference numerals, and detailed descriptions thereof may be omitted.

[0055] The predictive maintenance system 10c according to this embodiment can determine whether the operating condition of the monitored object is good or bad by measuring the sound emitted by the monitored object, and can be applied to predictive maintenance that predicts malfunctions in the monitored object. The monitored object is, for example, a mechanical device, specifically a press machine. However, the monitored object is not limited to a press machine, as long as it is a device or equipment whose malfunction can be predicted by sound.

[0056] As shown in FIG. 13, the predictive maintenance system includes a sound source visualization device (an example of a visualization device) 500, a teaching data creation device 20, a server 33, and a terminal 443.

[0057] The sound source visualization device 500 has a camera (not shown) that captures the monitored object 600 and multiple microphones 502 that identify the sound source generated from the monitored object 600, and can output a sound source visualization image that visualizes the sound source by superimposing the sound intensity distribution on an actual image of the area around the sound source in real time. This sound intensity distribution is expressed as visualized information in the form of a heat map, with different colors corresponding to the magnitude of the sound pressure. The sound source visualization device 500 may be called an acoustic camera.

[0058] As shown in FIG. 2, the teaching data creating device 20 has a teaching data creating unit 202, and can create an image as teaching data by importing a sound source visualization image.

[0059] The server 33 is managed by a service provider, and includes a teaching data group generation unit 302, a determination unit 334, an optimum model selection unit 306, and a management unit 308, as shown in FIG. The teaching data group generation unit (an example of a teaching data group generation means) 302 preprocesses the teaching data group TD created by the teaching data creation device 20, as shown in Figure 4, and creates a set TDg of preprocessed teaching data groups, i.e., teaching data groups TD1, TD2, TD3, TD4, ...

[0060] The judgment unit (an example of a judgment means) 334 can virtually judge whether the operating status of each monitored object 600 is good or bad based on the teaching data group TD and multiple trained machine learning models constructed by the machine learning model construction service 80.

[0061] The optimum model selection unit (an example of optimum model selection means) 306 evaluates the determination result by the determination unit 334 and can select the optimum machine learning model.

[0062] The management unit (an example of a management means) 308 can manage the state of the terminal 443 or the sound source visualization device 500. In detail, the management unit 308 can record operation information relating to the operation state of the terminal 443 or the sound source visualization device 500. The server 33 functions as a teaching data group generating means, a determining means, an optimum model selecting means, and a managing means by means of a program executed within the server 33.

[0063] 15, the terminal (an example of a determination device) 443 is connected to the sound source visualization device 500. The terminal 443 has a machine learning model receiving unit 440a, a control unit 443b, an operating status output unit 443c, a storage unit 443d, and a determination unit 443e, and can determine whether the monitored object 600 is good or bad based on the optimal model.

[0064] The machine learning model receiving unit (an example of a receiving means) 440a can receive the optimum model selected by the optimum model selecting unit 306 from the server 33. The optimum model is received via secure communication.

[0065] The control unit (an example of a control means) 443b can control the PLC 450 and the sound source visualization device 500.

[0066] The operation status output unit (an example of an operation status output means) 443c can output operation information relating to the operation status of the terminal 443 or the sound source visualization device 500. This operation information is, for example, information on the time from when the sound source visualization device 500 starts capturing images to when it finishes capturing images. The operation information may also be information on the number of images output from the sound source visualization device 500.

[0067] The storage unit (an example of a storage means) 443d can store the optimum model received by the machine learning model receiving unit 400a.

[0068] The determining unit (an example of a determining means) 443e can determine whether the operating state of the monitored object 600 is good or bad, based on a plurality of sound source visualization images output by the sound source visualization device 500 and the optimum model stored in the storage unit 443d. The terminal 443 functions as a receiving means, a control means, an operating status output means, a storage means, and a determination means by a program executed inside the terminal 443.

[0069] Next, the operation of the predictive maintenance system 10c (a method for determining whether the operating status of the monitored object 600 is good or bad) will be described with reference to Fig. 16. The predictive maintenance system 10c operates in accordance with the following steps S3-1 to S3-9. Of steps S3-1 to S3-9, steps S3-1 to S3-7 are preparatory operations, and the subsequent steps S3-8 and S3-9 are operations for determining whether the operating status of the monitored object 600 is actually good or bad. If possible, steps S3-1 to S3-7 may be performed in a different order or in parallel.

[0070] (Step S3-1) The teaching data creation unit 202 (see FIG. 2) of the teaching data creation device 20 (see FIG. 13) imports the sound source visualization image generated by the sound source visualization device 500 as teaching data, and creates the teaching data group TD shown in FIG. 4. teaching Shown data group T D is , is stored in a storage means (not shown) and is transmitted to the server 33 shown in FIG. Instead of being created by the teaching data creating unit 202, the teaching data group TD may be created manually based on a sound source visualization image of the monitored object 600 that is prepared in advance.

[0071] (Step S3-2) The teaching data group generating unit 302 of the server 33 receives the teaching data generated by the teaching data creating device 20 as shown in FIG. Takyo Shown data group T D Preprocessing is performed to generate a plurality of teaching data groups TD1, TD2, TD3, TD4, ... classified according to feature values. Note that in the preprocessing, for example, a plurality of threshold values ​​TH1 to TH10 for brightness (see FIG. 6) are used. Each threshold value may be obtained by finding the maximum value of the brightness of the defect D and dividing this maximum value. Thereafter, the server 33 uploads the preprocessed teaching data groups TD1, TD2, TD3, TD4, . . . to the machine learning model building service 80 in response to an operation by the service provider or the user. In this way, the preprocessing of teaching data, which requires know-how, is performed by the service provider rather than the user, allowing users to easily introduce a predictive maintenance system based on a machine learning model.

[0072] (Step S3-3) Based on the uploaded teaching data groups, machine learning models M1, M2, M3, M4, . . . are constructed by the machine learning model construction service 80. The accuracy of each constructed trained machine learning model is verified by the optimal model selection unit 306 of the server 33 (see FIG. 14). If the accuracy is poor, the process returns to the previous step S3-2, and the teaching data group generation unit 302 pre-processes the sound source visualization image (teaching data group TD) using a different method, such as by applying a different filter process.

[0073] (Step S3-4) In response to a user operation, the server 33 (see FIG. 14) downloads each trained machine learning model constructed by the machine learning model construction service 80. Each downloaded trained machine learning model is stored in a storage unit (not shown).

[0074] (Step S3-5) The determination unit 334 of the server 33 inputs the teaching data group TD into each machine learning model stored in a storage unit (not shown), and virtually determines whether the operating state of the monitored object 600 is good or bad.

[0075] (Step S3-6) The optimum model selection unit 306 evaluates the result of the judgment made by the judgment unit 334 in the previous step S3-6 as to whether the operating state of the monitored object 600 is good or bad, and selects an optimum model from among a plurality of machine learning models.

[0076] (Step S3-7) The optimum model selected by the optimum model selection unit 306 is transmitted from the server 33 to the terminal 442 . The transmitted optimum model is received by machine learning model receiving unit 440a (see FIG. 15) and stored in storage unit 443d.

[0077] (Step S3-8) This step S3-8 is a step for monitoring the monitored object 600. The control unit 443b of the terminal 443 controls the PLC 450 to operate the monitored object 600. Meanwhile, the sound source visualization device 500 measures the sound generated from the monitored object 600 and outputs a sound source visualization image at a predetermined period. The terminal 442 judges whether the operating state of the monitored object 600 is good or bad based on the output sound source visualization image and the optimum model stored in the storage unit 443d.

[0078] (Step S3-9) The operating status output unit 443c of the terminal 440 transmits operating information relating to the operating status of the sound source visualization device 500 to the server 33. The transmitted operating information is stored in a storage unit (not shown) of the server 33, and the operating status of the predictive maintenance system 10c is managed in an integrated manner by the server 33.

[0079] In this way, according to the predictive maintenance system 10c of this embodiment, the operating status of the monitored object 600 is determined to be good or bad using the optimal machine learning model selected from the multiple machine learning models that have been constructed, thereby enabling predictive maintenance with higher accuracy. Instead of the sound source visualization system, any visualization device may be used that has a detector for measuring a physical quantity that changes due to signs of a malfunction occurring in the monitored object 600 and that can generate multiple visualization images that visualize the physical quantity.

[0080] [Fourth embodiment] Next, a predictive maintenance system (an example of a quality determination system) 10d according to a fourth embodiment of the present invention will be described. Components having the same functions as those in the predictive maintenance system 10c according to the third embodiment (see FIG. 13) will be assigned the same reference numerals, and detailed descriptions thereof will be omitted.

[0081] The predictive maintenance system according to this embodiment can determine whether the operating condition of the monitored object is good or bad by measuring the vibrations emitted by the monitored object, and can be applied to predictive maintenance that predicts malfunctions in the monitored object. The monitored object may be, for example, a mechanical device, specifically a press or a conveying device, but may be any device or equipment that can predict malfunctions due to vibrations.

[0082] As shown in FIG. 17, the predictive maintenance system 10d includes a vibration visualization device (an example of a visualization device) 700, a teaching data creation device 20, a server 33, and a terminal 443 (an example of a determination device). The vibration visualization device 700 has a plurality of vibration sensors 702 for detecting vibrations generated from the monitored object 600, and can output a plurality of vibration visualization images that visualize the vibrations detected by each vibration sensor 702. These vibration visualization images are images that are expressed as visualized information using different colors according to at least one of the magnitude and frequency of the vibration, for example.

[0083] Here, in the predictive maintenance system 10d, the vibration visualization device 700 and the vibration visualization image correspond to the sound source visualization device 500 and the sound source visualization image in the third embodiment, respectively.

[0084] By using such a predictive maintenance system 10d and performing the aforementioned operational steps S3-1 to S3-9 (see Figure 16), the judgment unit 443e (see Figure 15) of the terminal 443 can determine that an abnormality has occurred when a malfunction occurs in the monitored object 600.

[0085] In this way, according to the predictive maintenance system 10d of this embodiment, the quality of the monitored object 600 is determined using the optimal machine learning model selected from the multiple machine learning models that have been constructed, thereby enabling predictive maintenance with higher accuracy. In addition, instead of a vibration visualization system, any visualization device may be used that has a detector for measuring a physical quantity that changes due to signs of a malfunction occurring in the monitored object 600 and is capable of generating multiple visualization images that visualize the physical quantity.

[0086] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and all changes in conditions that do not depart from the gist of the present invention are within the scope of application of the present invention. [Explanation of symbols]

[0087] 10a, 10b Good / bad judgment system 10c, 10d Predictive Maintenance System 14 Inspection object 20 Teaching data creation device 30 servers 40a, 40b Inspection System 450 PLC 80 Machine learning model building services 202 Teaching Data Creation Department 302 Teaching data group generation unit 306 Optimal Model Selection Section 308 Management Department 334 Judgment section 440 terminals 440a Machine learning model receiver 440b control section 440c Operation status output section 442, 443 terminals 443b Control section 443c Operation status output section 443d Storage section 443e Judgment section 460 Imaging unit 460a Camera section 460b Storage section 460c Judgment part 462 Imaging unit 470 Inspection Equipment 500 Sound source visualization device 502 Microphone 600 Monitored 700 Vibration visualization device 702 Vibration Sensor N Internet

Claims

[Claim 1] An acoustic camera having a camera for capturing images of a mechanical device and a microphone for measuring sounds generated from the mechanical device, and outputting a sound source visualization image in which the sound intensity distribution is superimposed in real time on an actual image of the area around the sound source of the mechanical device; a teaching data generation unit that generates teaching data from the sound source visualization images output by the acoustic camera in a state where a malfunction has occurred in the mechanical device and a state where no malfunction has occurred; a memory unit that stores a machine learning model that is constructed from the teaching data and that has learned whether the operating state of the machine device is good or bad; and a determination device that determines whether the operating status of the mechanical device is good or bad from the sound source visualization image of the mechanical device in operation output by the acoustic camera based on the machine learning model.

Images