Analysis device, analysis system, analysis program, and analysis method

Abstract

[Problem] To provide an analysis device capable of improving detection accuracy and versatility in, i.e., identifying the cause of a defect in a product through machine learning, while suppressing the amount of training data. [Solution] This analysis device comprises: an acquisition unit that acquires input data; an identification unit that inputs the input data to a machine learning model trained in advance with training data to identify any of predetermined recognition classes; a database that stores a correspondence relationship between the recognition classes and predetermined information; an analysis unit that analyzes the correspondence relationship between a recognition class and predetermined information by using the database; and an output unit that outputs an analysis result.

Description

[Technical Field]

[0001] The present invention relates to an analysis device, an analysis system, an analysis program, and an analysis method. [Background technology]

[0002] 2. Description of the Related Art In industrial production sites, defective products are sorted out by detecting defects in the products through visual inspection or the like.

[0003] Furthermore, development is underway to identify the device or equipment that caused the defect based on information about the defect detected by visual inspection or the like, and to deal with the identified device or equipment.

[0004] The following technology is disclosed in Patent Document 1: In a semiconductor device manufacturing line, a visual inspection device inspects defects in circuit patterns on wafers, and a defect mode database showing the correspondence between defect modes and equipment types is used to identify the equipment type that caused the defects from the inspection results.Then, the equipment that caused the defects is identified from the identified equipment type and manufacturing history. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-197437 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned inspection has a problem in that if there is a mistake in the defect inspection, the database will identify the wrong device. When attempting to identify the type of device that caused the defect in an end-to-end manner using machine learning using photographed images of inspection targets such as industrial products, there is a problem in that when the number of inspection targets is expanded, there is a shortage of training data for machine learning, resulting in a decrease in inspection accuracy. The above-mentioned prior art cannot address this problem.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide an analysis system, an analysis program, an analysis device, and an analysis method that can improve detection accuracy and versatility while reducing the amount of training data in identifying the cause of product defects using machine learning. [Means for solving the problem]

[0008] The above-mentioned problems of the present invention are solved by the following means.

[0009] (1) An analysis device having an acquisition unit that acquires input data, an identification unit that inputs the acquired input data into a machine learning model that has been previously trained using training data to identify one of predetermined recognition classes, a database that stores correspondences between the recognition classes and predetermined information, an analysis unit that uses the database to analyze the correspondences between the recognition classes and the predetermined information, and an output unit that outputs the analysis results by the analysis unit.

[0010] (2) The input data is an image, the database stores a correspondence between the recognition class of the defect and the detection result of the equipment type or manufacturing process, and the analysis unit uses the database to analyze the correspondence between the recognition class and the equipment type or the manufacturing process, in the analysis device described in (1) above.

[0011] (3) The input data is an image, the database stores a correspondence between the recognition class of behavior and the monitoring location, and the analysis unit uses the database to analyze the correspondence between the recognition class of behavior and the monitoring location, in the analysis device described in (1) above.

[0012] (4) The input data is an image, the database stores a correspondence between the recognition class of the behavior and its applicability to the behavior to be detected, and the analysis unit uses the database to analyze the correspondence between the recognition class and its applicability to the behavior to be detected, in the analysis device described in (1) above.

[0013] (5) The analysis device described in (1) above, wherein the machine learning model is a neural network, and the identification unit inputs the input data into the neural network and converts the input data into one of the recognition classes and a confidence level of the recognition class.

[0014] (6) The analysis device described in (1) above, wherein the input data is an image, and further includes a data control unit that switches the database used by the analysis unit for analysis or limits it to a portion of the database depending on product information contained in the image or information on the location of a person contained in the image.

[0015] (7) The analysis device according to (5) above, wherein the analysis unit does not perform analysis when the certainty of the recognition class converted by the identification unit is less than a predetermined threshold.

[0016] (8) The analysis device described in (5) above, wherein the analysis unit weights the certainty factor according to the importance set for each of the recognition classes converted by the identification unit, and analyzes the correspondence between the recognition class with the highest certainty factor after weighting and the specified information using the database.

[0017] (9) The analysis device described in (2) above, wherein the analysis unit accumulates the identification results by the identification unit for each of the images of different analysis targets of the same product, and detects the device type by analysis based on the accumulated identification results.

[0018] (10) An analysis system comprising the analysis device according to any one of (1) to (9) above, and a storage unit that stores the machine learning model and the database.

[0019] (11) An analysis program for causing a computer to execute a process comprising the steps of: (a) acquiring detected input data; (b) inputting the input data acquired in step (a) into a machine learning model previously trained with training data to identify one of predetermined recognition classes; (c) analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and (d) outputting the analysis results of step (c).

[0020] (12) The analysis program according to (11) above, wherein the input data is an image, the database stores a correspondence between the recognition class of the defect and the equipment type or the manufacturing process, and step (c) uses the database to analyze the correspondence between the recognition class and the equipment type or the manufacturing process.

[0021] (13) The analysis program described in (11) above, wherein the input data is an image, the database stores a correspondence between the recognition class of behavior and the monitoring location, and step (c) uses the database to analyze the correspondence between the recognition class of behavior and the monitoring location.

[0022] (14) The analysis program described in (11) above, wherein the input data is an image, the database stores a correspondence between the recognition class of the behavior and its applicability to the behavior to be detected, and step (c) uses the database to analyze the correspondence between the recognition class and its applicability to the behavior to be detected.

[0023] (15) The analysis program described in (11) above, wherein the machine learning model is a neural network, and step (b) inputs the input data into the neural network and converts the input data into one of the recognition classes and a confidence level of the recognition class.

[0024] (16) The analysis program described in (11) above, wherein the input data is an image, and the processing further includes a step (d) of switching the database used for analysis in step (c) or limiting it to a part of the database depending on product information contained in the image or information on the location of a person contained in the image.

[0025] (17) The analysis program according to (15) above, wherein the step (c) does not perform analysis if the certainty of the recognition class converted in the step (b) is less than a predetermined threshold.

[0026] (18) The analysis program described in (15) above, in which step (c) weights the certainty according to the importance set for each of the recognition classes converted in step (b), and analyzes the correspondence between the recognition class with the highest certainty after weighting and the specified information using the database.

[0027] (19) The analysis program according to (12) above, wherein step (c) accumulates the identification results in step (b) for each of the images of different analysis targets of the same product, and detects the device type by analysis based on the accumulated identification results.

[0028] (20) An analysis method comprising: a step (a) of acquiring detected input data; a step (b) of inputting the input data acquired in step (a) into a machine learning model that has been previously trained using training data to identify one of predetermined recognition classes; a step (c) of analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and a step (d) of outputting the analysis results of step (c). [Effects of the Invention]

[0029] From the input data, one of the predetermined recognition classes is identified using a trained machine learning model, and the correspondence between the recognition class and the predetermined information is analyzed using a database that stores the correspondence between the recognition class and the predetermined information. This makes it possible to improve detection accuracy and versatility while suppressing the amount of training data when using machine learning to identify the cause of product defects, etc. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an analysis system. [Figure 2] FIG. 2 is a block diagram of an analysis device included in the analysis system. [Figure 3] FIG. 2 is a functional block diagram of a control unit. [Figure 4] 10 is a flowchart showing the operation of the analysis system. [Figure 5] FIG. 2 is a functional block diagram of a control unit. [Figure 6] 10 is a flowchart showing the operation of the analysis system. [Figure 7] FIG. 2 is a functional block diagram of a control unit. [Figure 8] 10 is a flowchart showing the operation of the analysis system. [Figure 9] FIG. 2 is a functional block diagram of a control unit. [Figure 10] 10 is a flowchart showing the operation of the analysis system. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, an analysis device, an analysis system, an analysis program, and an analysis method according to embodiments of the present invention will be described with reference to the drawings. In the drawings, identical elements are designated by the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0032] (First embodiment) Fig. 1 is a diagram showing the configuration of analysis system 10. Fig. 2 is a block diagram of analysis device 100 included in analysis system 10. Analysis device 100 may be made up of multiple devices.

[0033] The analysis system 10 may include an analysis device 100 and an imaging device 200 .

[0034] The image capturing device 200 captures an image 201 (hereinafter simply referred to as "image 201") of an analysis target 202 (see FIG. 3). The image 201 constitutes input data input to the analysis device 100. The image 201 may be an image of all or part of the analysis target 202. The image 201 may also be an image that includes things other than the analysis target 202. The image capturing device 200 is, for example, a camera. The analysis target 202 is, for example, a product, and products include not only finished products such as automobiles and semiconductor chips, but also parts such as automobile bodies, bolts, nuts, and flat plates. The image capturing device 200 can be replaced by a scanner, a microphone, a voice recognition device that converts voice into text data, an odor sensor, a temperature sensor, etc. Data detected (acquired) by these devices also constitutes input data.

[0035] The image 201 may be, for example, a black and white image or a color image, and may be a 128 pixel by 128 pixel image. The image capturing device 200 transmits the image 201 to the analysis device 100.

[0036] Analysis device 100 detects abnormalities in the analysis target by analyzing the image. Abnormalities include, for example, scratches, chips, creases, bends, stains, discoloration (changes in color), abnormal temperatures, strange odors, etc. As will be described later, analysis device 100 can detect an abnormality (defect) in the analysis target as one of predetermined recognition classes that indicate the type of abnormality.

[0037] As will be described in a third embodiment later, the analysis target 202 may be a person. In this case, the abnormality includes, for example, abnormal behavior of a person on a product manufacturing line.

[0038] Analysis device 100 further performs an analysis to detect (identify) the type of device that caused the abnormality from the type of abnormality detected (fault recognition class), and outputs the analysis results.

[0039] 2, analysis device 100 includes a control unit 110, a storage unit 120, a communication unit 130, and an operation display unit 140. These components are connected to one another via a bus 150. Analysis device 100 is configured by, for example, a computer terminal.

[0040] Control unit 110 is configured with a CPU (Central Processing Unit) and memories such as RAM (Random Access Memory) and ROM (Read Only Memory), and controls and performs arithmetic processing on each part of analysis device 100 according to a program. The functions of control unit 110 will be described in detail later.

[0041] The storage unit 120 is configured by a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various programs and various data.

[0042] The communication unit 130 is an interface circuit (for example, a LAN card) for communicating with an external device via a network.

[0043] The operation and display unit 140 may be configured with, for example, a touch panel. The operation and display unit 140 accepts various inputs from the user. The operation and display unit 140 displays various information including the analysis results of the test subject.

[0044] The function of the control unit 110 will now be described.

[0045] 3 is a functional block diagram of the control unit 110. The control unit 110 functions as an acquisition unit 111, a conversion unit 112, an analysis unit 113, and an output unit 114. Fig. 3 also shows a trained model 121 and a knowledge database 122 stored in the storage unit 120.

[0046] The acquisition unit 111 acquires the image 201 by receiving it from the image capturing device 200. When the image is stored in the storage unit 120, the acquisition unit 111 can acquire the image 201 by reading it from the storage unit 120.

[0047] In FIG. 3, an analysis object 202 is a flat plate, which is a component, and an image 201 is shown as an example in which there is a black dot stain at the bottom right of the flat plate.

[0048] The conversion unit 112 inputs the image 201 into the trained model read from the storage unit 120 and identifies one of the predetermined recognition classes. Identification includes conversion and classification. The conversion unit 112 constitutes an identification unit. For simplicity, the following description will be given assuming that identification is performed by conversion. Specifically, for example, the conversion unit 112 converts the image 201 into a vector of certainty for each recognition class (certainty for each recognition class). The certainty is a likelihood indicating the likelihood that the image 201 corresponds to the corresponding recognition class. The predetermined recognition class may be a defective (abnormal) recognition class. The conversion unit 112 converting the image 201 into a vector of certainty for each recognition class makes it possible to identify a recognition class with a high degree of certainty, which is equivalent to converting the image 201 into a predetermined identification class. Note that the conversion unit 112 may convert the image 201 into a recognition class with the highest degree of certainty (more specifically, data identifying the recognition class with the highest degree of certainty) instead of a vector of certainty.

[0049] The trained model 121 is an example of a pre-trained machine learning model, and may be a trained model of a neural network. The trained model 121 may also be a trained model of a support vector machine or a random forest, which is a model other than a neural network. For ease of explanation, the following description will be given assuming that the trained model 121 is a trained model of a neural network.

[0050] The trained model 121 can be generated in advance by training a neural network using a relatively large amount of teacher data (training data) of combinations of images 201 and recognition classes that are correct labels corresponding to the images 201, and can be stored in the storage unit 120. Specifically, the neural network is trained by backpropagation so as to reduce the difference (loss) between the recognition classes (specifically, the vector of certainty of each recognition class) and the correct labels (specifically, the correct solution of the vector of certainty of each recognition class) when the image 201 is input.

[0051] The training data is preferably a combination of the defective product image 201 and the correct label of the defect recognition class, which is the recognition class corresponding to the defective product image 201. This can efficiently improve the accuracy of detecting abnormalities (defects) by the conversion unit 112.

[0052] The predetermined recognition classes are types of anomalies, and include, for example, "scratches of a predetermined length or more," "scratches of a predetermined length or less," "scratches of a predetermined thickness or more," "scratches of a predetermined thickness or less," "stains of a predetermined area or more," "stains of a predetermined area or less," and "discoloration." Furthermore, for example, "stains of a predetermined area or more" and "stains of a predetermined area or less" can be further subdivided into multiple recognition classes based on the location of the stain on image 201. That is, the recognition classes can include, for example, "stains in the upper right corner of a predetermined area or more," "stains in the lower right corner of a predetermined area or more," "stains in the upper left corner of a predetermined area or more," "stains in the upper right corner less than a predetermined area," "stains in the lower right corner less than a predetermined area," "stains in the upper left corner less than a predetermined area," and "stains in the lower left corner less than a predetermined area." The recognition classes can be further subdivided as needed. The predetermined length, predetermined thickness, and predetermined area can be appropriately set through experiments, etc., from the perspective of analytical accuracy in detecting device types from the recognition classes. The recognition classes may also include classes other than anomaly types (e.g., "good product"). If the analysis target includes a person, the recognition classes may also be behavioral recognition classes.

[0053] The analysis unit 113 performs an analysis to detect the type of device that has caused the abnormality, using the conversion result (i.e., the recognition class) by the conversion unit 112 and the knowledge database 122 stored in the storage unit 120. That is, the analysis unit 113 uses the knowledge database 122 to analyze the correspondence between the recognition class and the predetermined information (detection result). In this way, the type of device that has caused the abnormality is detected. The device type detected by the analysis by the analysis unit 113 constitutes the predetermined information. The predetermined information may be a manufacturing process, a monitoring location, or the like. The knowledge database 122 may be a data group (or device) that indicates (stores) the correspondence between the recognition class and the predetermined information. The predetermined information is preferably information different from the recognition class and not obtained from a machine learning model, but may also be information obtained from a machine learning model. The predetermined information may be a detection result detected by a detection device, or input information input to an input device such as the operation display unit 140. Specifically, the predetermined information is preferably information regarding the cause or location of the occurrence of a defect or abnormality, and more preferably information regarding the device type, the manufacturing process, or the monitoring location. For simplicity's sake, the following description will be given assuming that the predetermined information is a device type unless otherwise specified. Specifically, for example, if the conversion result by the conversion unit 112 has the highest confidence level for the recognition class of "a stain on the bottom right that is smaller than a predetermined area," the analysis unit 113 detects the device type corresponding to the "a stain on the bottom right that is smaller than a predetermined area" by identifying it using a knowledge database.

[0054] The correspondence between the recognition classes and equipment types contained in the knowledge database 122 may be based on the knowledge of, for example, experienced workers or engineers in a production factory. This is because, based on their own experience, workers or the like are likely to be able to more accurately identify the process (equipment type) that caused the abnormality (defect) simply by looking at an image of a flat plate with a stain of less than a predetermined area in the lower right corner.

[0055] The correspondence between the recognition classes and the device types contained in the knowledge database 122 may be set based on the past repair history of the device.

[0056] In this way, instead of detecting the type of device that caused the abnormality from the image 201 in an end-to-end manner using machine learning, the type of abnormality is estimated using machine learning, and information on the type of device that caused the abnormality (e.g., linguistic information) is identified from information on the type of abnormality (e.g., linguistic information) using the knowledge database 122. This makes it possible to improve detection accuracy and versatility while reducing the amount of training data. Since the detection of abnormalities based on images, etc. is easily affected by shooting conditions (e.g., shooting environment), when detecting the type of device that caused the abnormality from images, etc. in an end-to-end manner using machine learning, it is necessary to retrain the trained model every time the production factory changes. In this embodiment, the information in the knowledge database 122 is knowledge of the correspondence between the type of abnormality and the type of device that caused the abnormality, so the trained model 121 can be reused at different production factories, eliminating the need for retraining.

[0057] The analysis unit 113 may detect the type of device that has caused the abnormality by using multiple recognition classes whose certainty is equal to or greater than a predetermined threshold and the knowledge database 122. For example, if the recognition classes whose certainty is equal to or greater than a predetermined threshold are "stain in the lower right corner less than a predetermined area" and "stain in the upper left corner more than a predetermined area," the device type corresponding to these two recognition classes can be identified using the knowledge database. Therefore, the knowledge database 122 may contain data indicating the correspondence between multiple recognition classes and one device type. The predetermined threshold may be appropriately set through experiments or the like in terms of the analytical accuracy of the analysis that identifies the device type from the recognition class.

[0058] If the certainty of the recognition class converted from image 201 (i.e., for example, the highest certainty among the certainty levels included in the vector of certainty levels for each recognition class) is less than a predetermined threshold, the analysis unit 113 does not need to perform analysis to detect the device type.

[0059] The importance of the type of anomaly can be set for each recognition class. The analysis unit 113 may weight the certainty of the recognition class indicating the type of anomaly by multiplying the importance, and may select the recognition class with the highest certainty after weighting as the analysis result the corresponding device type in the knowledge database 122. The importance can be appropriately set from any viewpoint through experiments, etc. For example, the importance can be appropriately set from the viewpoint of the magnitude of the impact of the type of anomaly on the entire production line.

[0060] The analysis unit 113 can detect the device type by accumulating the conversion results by the conversion unit 112 of each image 201 of different analysis targets 202 of the same product in the storage unit 120 or the like, and performing analysis based on the accumulated multiple conversion results. Specifically, for example, the analysis unit 113 can acquire each of the multiple images 201 of the same part flowing down the production line, convert each into a recognition class, accumulate the conversion results, calculate the average value of the vector of certainty of the recognition class, which is the accumulated multiple conversion results, and use the knowledge database 122 to detect the device type corresponding to the recognition class with the highest average certainty value.

[0061] The output unit 114 outputs the analysis result by the analysis unit 113. Specifically, the output unit 114 outputs the device type detected by the analysis by the analysis unit 113 by displaying it on the operation display unit 140. The output unit 114 may output the device type detected by the analysis unit 113 by transmitting it to an external device via the communication unit 130.

[0062] 4 is a flowchart showing the operation of analysis system 10. This flowchart can be executed by control unit 110 of analysis device 100 in accordance with a program.

[0063] The control unit 110 acquires the image 201 by receiving it from the photographing device 200 (S101).

[0064] The control unit 110 converts the image 201 into a vector of confidence levels for each recognition class using the trained model 121 (S102).

[0065] The control unit 110 uses the knowledge database 122 to perform an analysis to detect the device type from the conversion result in step S102 (S103).

[0066] The control unit 110 outputs the device type detected by the analysis by displaying it on the operation display unit 140 (S104).

[0067] (Second embodiment) The second embodiment will be described. This embodiment differs from the first embodiment in the following respects. In this embodiment, the analysis unit 113 switches the knowledge database 122 used for analysis, or limits the knowledge database 122 used for analysis to a part, based on information about the manufacturing process of the product that is the analysis target 202 contained in the image 201. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0068] 5 is a functional block diagram of the control unit 110. The control unit 110 functions as an acquisition unit 111, a conversion unit 112, an analysis unit 113, an output unit 114, and a data control unit 115.

[0069] The acquisition unit 111 acquires the image 201 by receiving it from the image capturing device 200 or the like.

[0070] The data control unit 115 acquires information about the manufacturing process of the product included in the image 201. The information about the manufacturing process of the product is, for example, information indicating the painting process of a flat plate, which is a component. The information about the manufacturing process of the product can be registered by being associated with each of the imaging devices 200 that capture the image 201 of the product in the manufacturing process and stored in advance in the storage unit 120. The data control unit 115 can acquire the information about the manufacturing process of the product, associated with the imaging device 200 that is the recipient of the image 201, by reading it from the storage unit 120, for example.

[0071] The data control unit 115 generates a control signal for the knowledge database 122 based on the information on the manufacturing process of the product included in the image 201. The control signal for the knowledge database 122 is a signal that the analysis unit 113 uses to switch the knowledge database 122 used for analysis, etc.

[0072] In response to the control signal from knowledge database 122, analysis unit 113 switches the knowledge database used by analysis unit 113 for analysis to knowledge database 122 corresponding to the manufacturing process of the product included in image 201. For example, if knowledge database 122 is a comprehensive one that corresponds to the manufacturing processes of all analysis targets (products, parts) in the manufacturing line, data control unit 115 may limit the database used by analysis unit 113 for analysis to a part of knowledge database 122 in response to the control signal from knowledge database 122.

[0073] The analysis unit 113 performs the analysis using the knowledge database 122 after switching in accordance with the control signal of the knowledge database 122. The analysis unit 113 may perform the analysis using only a part of the knowledge database 122 in accordance with the control signal of the knowledge database 122.

[0074] 6 is a flowchart showing the operation of analysis system 10. This flowchart can be executed by control unit 110 of analysis device 100 in accordance with a program.

[0075] The control unit 110 receives and acquires the image 201 and information on the manufacturing process of the product included in the image 201 from the photographing device 200 (S201).

[0076] The control unit 110 converts the image 201 into a vector of confidence levels for each recognition class using the trained model 121 (S202).

[0077] The control unit 110 switches the knowledge database 122 used for analysis to detect the type of device that caused the abnormality based on the information on the manufacturing process of the product (S203).

[0078] The control unit 110 uses the knowledge database 122 after the switch to perform an analysis to detect the device type from the conversion result in step S202 (S204).

[0079] The control unit 110 outputs the device type detected by the analysis by displaying it on the operation display unit 140 (S205).

[0080] (Third embodiment) The third embodiment will be described. This embodiment differs from the first embodiment in the following respects. In the first embodiment, the analysis target is a part or the like, the predetermined recognition class into which the image 201 is converted is the type of abnormality, the knowledge database 122 indicates the relationship between the type of abnormality and the type of device, and the type of device is detected by analysis. On the other hand, in this embodiment, the analysis target 202 is a person, the predetermined recognition class into which the image 201 is converted is human behavior, the knowledge database 122 indicates the relationship between the human behavior and the applicability to the detection target behavior (for example, whether the abnormal behavior is appropriate), and the applicability to the detection target behavior is detected by analysis. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0081] 7 is a functional block diagram of the control unit 110. The control unit 110 functions as an acquisition unit 111, a conversion unit 112, an analysis unit 113, and an output unit 114.

[0082] The acquisition unit 111 acquires the image 201 by receiving it from the photographing device 200, for example. The image 201 includes an image of a person (e.g., a worker on a production line) who is the analysis target 202. The image 201 may include multiple people who are the analysis target 202. The image 201 may also include images of equipment, floors, chairs, etc.

[0083] The conversion unit 112 inputs multiple images 201 (e.g., video images) into the trained model read from the storage unit 120 and converts them into a predetermined recognition class, that is, human behavior. Specifically, the conversion unit 112 converts the multiple images 201 into a vector of confidence levels for each recognition class. The reason for converting the multiple images 201 into a predetermined recognition class is that, since the recognition class is human behavior, it is generally considered that the time progression (temporal change) of the images 201 is necessary to identify the behavior. Note that the conversion unit 112 may convert a single image 201 into a recognition class. For example, behavior such as one worker hitting another worker can be identified from a single image 201.

[0084] The trained model 121 can be generated in advance by training a neural network using a relatively large amount of training data that combines multiple images 201 with recognition classes that are correct labels corresponding to the multiple images 201, and stored in the memory unit 120.

[0085] The neural network training using training data of combinations of multiple images 201 and recognition classes that are correct labels corresponding to the multiple images 201, and the conversion of the confidence levels of each recognition class of the multiple images 201 into a vector using the trained model 121 can be performed using, for example, LSTM (Long Short-Term Memory), a well-known technology.

[0086] The predetermined recognition class is a human behavior, and includes, for example, "lying down and not moving," "sitting down and not moving," "hitting someone," etc. The training data used to generate the trained model 121 preferably uses a combination of images 201 of abnormal behavior such as "lying down and not moving" and "hitting someone," and the correct answer label of the recognition class of the abnormal behavior. This can efficiently improve the detection accuracy of abnormal behavior by the conversion unit 112. The predetermined recognition class may also include "normal behavior."

[0087] The analysis unit 113 performs an analysis to detect applicability to the detection target behavior using the conversion result by the conversion unit 112 and the knowledge database 122 stored in the storage unit 120. That is, the analysis unit 113 uses the knowledge database 122 to analyze the correspondence between the recognition class and the predetermined information (detection result). In this way, applicability to the detection target behavior is detected. The applicability to the detection target behavior detected by the analysis by the analysis unit 113 constitutes the predetermined information. Applicability to the detection target behavior includes whether or not it is an abnormal behavior (whether or not it is an abnormal behavior). For simplicity of explanation, the following description will be given assuming that applicability to the detection target behavior is whether or not it is an abnormal behavior. The knowledge database 122 can be a data group (or a device) that indicates (stores) the correspondence between the recognition class and the predetermined information. The predetermined information includes information on whether or not it is an abnormal behavior. Specifically, when the certainty of the recognition class of "lying down and not moving" is highest in the conversion results by the conversion unit 112, the analysis unit 113 detects whether or not the behavior corresponding to "lying down and not moving" is abnormal (whether or not the behavior of "lying down and not moving" is abnormal) by using a knowledge database.

[0088] The correspondence between the recognition class contained in the knowledge database 122 and whether or not the behavior is abnormal can be appropriately set by determining, for example, based on past care records and monitoring records, whether it is appropriate to judge that the recognition class corresponds to abnormal behavior and issue an alert, etc.

[0089] The correspondence between the recognition classes included in the knowledge database 122 and whether or not the behavior corresponds to abnormal behavior can be set according to the location where the image 201 is captured (hereinafter referred to as the "analysis environment"). For example, if the analysis environment is each process on a production line, the recognition class "lying down and not moving" can be set to correspond to the detection result "corresponding to abnormal behavior." Furthermore, if the analysis environment is a hospital bed, the recognition class "lying down and not moving" can be set to correspond to the detection result "not corresponding to abnormal behavior (normal)."

[0090] The importance of the type of abnormality can be set for each recognition class. The analysis unit 113 may weight the certainty of the recognition class indicating the type of abnormality by multiplying the importance, and the recognition class with the highest certainty after weighting may be used as the analysis result to determine whether the corresponding abnormal behavior in the knowledge database 122 is relevant. The importance can be appropriately set from any viewpoint through experiments, etc. For example, the importance can be appropriately set from the viewpoint of the urgency of care (response) for the behavior of the recognition class.

[0091] The output unit 114 outputs the analysis result by the analysis unit 113. Specifically, the output unit 114 outputs the abnormal behavior detected by the analysis by the analysis unit 113 by displaying it on the operation display unit 140. The output unit 114 may output the abnormal behavior detected by the analysis unit 113 by transmitting it to an external device via the communication unit 130.

[0092] 8 is a flowchart showing the operation of analysis system 10. This flowchart can be executed by control unit 110 of analysis device 100 in accordance with a program.

[0093] The control unit 110 acquires a plurality of images 201 by receiving them from the image capturing device 200 (S301).

[0094] The control unit 110 converts the multiple images 201 into a vector of confidence levels for each recognition class using the trained model 121 (S302).

[0095] The control unit 110 uses the knowledge database 122 to perform an analysis to detect whether or not abnormal behavior occurs from the conversion result in step S302 (S303).

[0096] The control unit 110 outputs whether or not the abnormal behavior detected by the analysis is true by displaying it on the operation display unit 140 (S304).

[0097] (Fourth embodiment) The fourth embodiment will be described. This embodiment differs from the third embodiment in the following respects. In this embodiment, the analysis unit 113 switches the knowledge database 122 used for analysis or limits the knowledge database 122 used for analysis to a part based on information about the analysis environment, which is information about the location of a person who is the analysis target 202 included in the image 201. In other respects, this embodiment is similar to the third embodiment, so duplicated explanations will be omitted or simplified.

[0098] 9 is a functional block diagram of the control unit 110. The control unit 110 functions as an acquisition unit 111, a conversion unit 112, an analysis unit 113, an output unit 114, and a data control unit 115.

[0099] The acquisition unit 111 acquires a plurality of images 201 by receiving them from the image capturing device 200 or the like.

[0100] The data control unit 115 acquires information about the analysis environment, which is information about the location where the image 201 is captured and where the person being analyzed 202 is located. Possible examples of the information about the analysis environment include a work location for painting flat plates on a production line, or a hospital bed. The information about the analysis environment can be registered by being associated with the image capturing device 200 that captures the image 201 and stored in advance in the storage unit 120. The data control unit 115 can acquire the information about the analysis environment, associated with the image capturing device 200 that is the recipient of the image 201, by reading it from the storage unit 120.

[0101] Based on the information on the analysis environment, the data control unit 115 generates a control signal for the knowledge database 122. The control signal for the knowledge database 122 is a signal for the analysis unit 113 to switch the knowledge database 122 used for analysis.

[0102] In response to the control signal from the knowledge database 122, the analysis unit 113 switches the knowledge database used by the analysis unit 113 for analysis to the knowledge database 122 corresponding to the analysis environment where the image 201 was captured. For example, if the knowledge database 122 is a comprehensive one that supports multiple analysis environments, the data control unit 115 may use the control signal from the knowledge database 122 to limit the database used by the analysis unit 113 for analysis to a part of the knowledge database 122 for use in the analysis environment corresponding to the imaging device 200 where the person to be analyzed was captured.

[0103] The analysis unit 113 performs the analysis using the knowledge database 122 after switching in accordance with the control signal of the knowledge database 122. The analysis unit 113 may perform the analysis using only a part of the knowledge database 122 in accordance with the control signal of the knowledge database 122.

[0104] Whether or not abnormal behavior is present may vary depending on the location of the photograph in which the person being analyzed 202 is photographed, but by switching the knowledge database 122 that shows the correspondence between behavior and whether or not abnormal behavior is present depending on the analysis environment, it is possible to respond to a variety of situations involving monitored subjects.

[0105] 10 is a flowchart showing the operation of analysis system 10. This flowchart can be executed by control unit 110 of analysis device 100 in accordance with a program.

[0106] The control unit 110 acquires information about a plurality of images 201 and the analysis environment where the plurality of images 201 were taken (S401).

[0107] The control unit 110 converts the image 201 into a vector of confidence levels for each recognition class using the trained model 121 (S402).

[0108] The control unit 110 switches the knowledge database 122 used for the analysis to detect whether or not abnormal behavior is present, based on the information on the analysis environment (S403).

[0109] The control unit 110 uses the knowledge database 122 after the switch to perform an analysis to detect whether or not abnormal behavior occurs from the conversion result in step S402 (S404).

[0110] The control unit 110 outputs whether or not the abnormal behavior detected by the analysis is true by displaying it on the operation display unit 140 (S405).

[0111] The above-described embodiment provides the following advantages.

[0112] From the input data, one of the predetermined recognition classes is identified using a trained machine learning model, and the correspondence between the recognition class and the predetermined information is analyzed using a database that stores the correspondence between the recognition class and the predetermined information. This makes it possible to improve detection accuracy and versatility while suppressing the amount of training data when using machine learning to identify the cause of product defects, etc.

[0113] Furthermore, the input data is an image, and the database stores the correspondence between the defect recognition class and the equipment type or manufacturing process. The database is then used to analyze the correspondence between the recognition class and the equipment type or manufacturing process. This allows for improved detection accuracy and versatility while reducing the amount of training data when using machine learning to identify the equipment or manufacturing process that caused the product defect.

[0114] Furthermore, the input data is an image, and the database stores the correspondence between the behavior recognition classes and the monitoring locations. The database is then used to analyze the correspondence between the behavior recognition classes and the monitoring locations. This makes it possible to improve detection accuracy and versatility while reducing the amount of training data when using machine learning to identify the monitoring locations where product defects occur.

[0115] Furthermore, the input data is an image, and the database stores the correspondence between the recognition class of the behavior and the detection result of the applicability to the detection target behavior.The database is then used to analyze the correspondence between the recognition class and the applicability to the detection target behavior.This enables the detection accuracy and versatility to be improved while reducing the amount of training data in the detection of abnormal behavior using machine learning.

[0116] Furthermore, the machine learning model is a neural network, and input data is input to the neural network to convert the input data into one of the recognition classes and the confidence level of the recognition class. This makes it possible to more easily improve detection accuracy and versatility while suppressing the amount of training data when using machine learning to identify the cause of product defects, etc.

[0117] Furthermore, depending on the product information contained in the image or the location of a person contained in the image, the database used for analysis can be switched or limited to a portion of the database, which can improve the accuracy of detection, such as identifying the cause of product defects using machine learning.

[0118] Furthermore, if the confidence of the converted recognition class is less than a predetermined threshold, analysis is not performed, thereby reducing the occurrence of false alarms and missed alarms when conversion of input data to a recognition class fails.

[0119] Furthermore, the reliability is weighted according to the importance level set for each converted recognition class, and the correspondence between the recognition class with the highest reliability after weighting and the predetermined information is analyzed using a database, thereby further reducing the occurrence of false reports.

[0120] Furthermore, the system accumulates the identification results from the identification sections of images of different analysis targets for the same product, and detects the device type through analysis based on the accumulated identification results. This further improves the accuracy of machine learning-based detection, such as identifying the cause of product defects.

[0121] The analysis system, analysis program, analysis device, and analysis method described above are the main configurations described in order to explain the features of the above-mentioned embodiments, but are not limited to the above configurations and can be modified in various ways within the scope of the claims. Furthermore, configurations provided in general anomaly detection systems, etc. are not excluded.

[0122] For example, some steps in the above-described flowcharts may be omitted, other steps may be added, some of the steps may be executed simultaneously, or one step may be divided into multiple steps and executed.

[0123] In the embodiment, the conversion unit 112 inputs the image 201 to the trained model 121 of the neural network and converts it into one of the predetermined recognition classes by a convolution operation. However, the conversion unit 112 may input a comparison result (difference data) obtained by comparing an image of a non-defective product reproduced from an image including the analysis target 202 using an autoencoder with an image including the analysis target 202 into the trained model 121 of the neural network, thereby converting the image into one of the predetermined recognition classes.

[0124] Furthermore, the conversion unit 112 may estimate the joint points of a person from the image 201 using an hourglass network, and detect the behavior of the person based on the estimated joint points.

[0125] Furthermore, if the image capturing device 200 is replaced by a microphone, the conversion unit 112 converts the audio data into a predetermined recognition class. If the image capturing device 200 is replaced by an odor sensor, the conversion unit 112 converts the odor data into a predetermined recognition class. If the image capturing device 200 is replaced by a temperature sensor, the conversion unit 112 converts the temperature data into a predetermined recognition class. In these cases, the predetermined recognition class may be, for example, a high-pitched abnormal sound (above a certain frequency), a low-pitched abnormal sound (below a certain frequency), a burning smell, a chemical smell, a high temperature above a certain threshold, and a low temperature below a certain threshold.

[0126] The means and methods for performing various processes in the above-described system can be realized by either dedicated hardware circuits or a programmed computer. The above-described program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The above-described program may be provided as standalone application software, or may be incorporated as a function into the software of a device such as an anomaly detection device.

[0127] This application is based on a Japanese patent application (Patent Application No. 2021-002033) filed on January 8, 2021, the disclosure of which is incorporated by reference in its entirety. [Explanation of symbols]

[0128] 10 analysis systems, 100 analysis equipment, 110 control section, 111 Acquisition Department; 112 conversion unit, 113 Analysis Department, 114 output section, 120 storage section, 121 trained models, 122 knowledge database, 130 Communications Department, 140 Operation display section, 200 imaging devices, 201 images, 202 Subject of analysis.

Claims

1. an acquisition unit for acquiring input data; an identification unit that inputs the acquired input data into a machine learning model that has been trained in advance using training data to identify one of predetermined recognition classes; a database that stores correspondence between the recognition class and predetermined information; an analysis unit that analyzes the correspondence between the recognition class and the predetermined information using the database; an output unit that outputs the analysis result by the analysis unit, the machine learning model is a neural network; the identification unit inputs the input data to the neural network and converts the input data into one of the recognition classes and a certainty of the recognition class; The analysis device, wherein the analysis unit does not perform analysis when the certainty of the recognition class converted by the identification unit is less than a predetermined threshold.

2. an acquisition unit for acquiring input data; an identification unit that inputs the acquired input data into a machine learning model that has been trained in advance using training data to identify one of predetermined recognition classes; a database that stores correspondence between the recognition class and predetermined information; an analysis unit that analyzes the correspondence between the recognition class and the predetermined information using the database; an output unit that outputs the analysis result by the analysis unit, the input data is an image, the database stores a correspondence between the recognition class of the defect and a device type or a manufacturing process; the analysis unit analyzes a correspondence relationship between the recognition class and the device type or the manufacturing process using the database; The analysis unit accumulates the identification results by the identification unit for each of the images of different analysis targets of the same product, and detects the device type by analysis based on the accumulated identification results.

3. the input data is an image, the database stores a correspondence between the recognition class of the defect and a device type or a manufacturing process; The analysis device according to claim 1 , wherein the analysis unit uses the database to analyze the correspondence between the recognition class and the device type or the manufacturing process.

4. the input data is an image, the database stores a correspondence between the recognition class of the behavior and a monitoring location; The analysis device according to claim 1 , wherein the analysis unit uses the database to analyze the correspondence between the recognition class of the behavior and the monitoring location.

5. the input data is an image, the database stores a correspondence between the recognition class of the behavior and its applicability to the behavior to be detected; The analysis device according to claim 1 , wherein the analysis unit uses the database to analyze a correspondence relationship between the recognition class and the relevance to the target behavior.

6. the machine learning model is a neural network; the identification unit inputs the input data to the neural network and converts the input data into one of the recognition classes and a certainty of the recognition class; The analysis device according to claim 2 .

7. the input data is an image, The analysis device according to claim 1, further comprising a data control unit that switches the database used by the analysis unit for analysis or limits it to a portion of the database depending on product information contained in the image or information on the location of a person contained in the image.

8. The analysis device according to claim 1, wherein the analysis unit weights the certainty factor according to the importance level set for each of the recognition classes converted by the identification unit, and analyzes the correspondence between the recognition class with the highest certainty factor after weighting and the specified information using the database.

9. The analysis device according to any one of claims 1 to 8, a storage unit that stores the machine learning model and the database; An analysis system having:

10. (a) obtaining input data; a step (b) of inputting the input data acquired in the step (a) into a machine learning model that has been trained in advance using training data to identify one of predetermined recognition classes; a step (c) of analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and step (d) of outputting the analysis result of step (c), the machine learning model is a neural network; The step (b) inputs the input data to the neural network and converts the input data into one of the recognition classes and a confidence level of the recognition class; The step (c) is an analysis program for causing a computer to execute a process in which analysis is not performed if the certainty of the recognition class converted in the step (b) is less than a predetermined threshold.

11. (a) obtaining input data; a step (b) of inputting the input data acquired in the step (a) into a machine learning model that has been trained in advance using training data to identify one of predetermined recognition classes; a step (c) of analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and step (d) of outputting the analysis result of step (c), the input data is an image, the database stores a correspondence between the recognition class of the defect and a device type or a manufacturing process; The step (c) analyzes a correspondence relationship between the recognition class and the device type or the manufacturing process using the database; The step (c) is an analysis program for causing a computer to execute a process in which the identification results in the step (b) of each of the images of different analysis targets of the same product are accumulated, and the device type is detected by analysis based on the accumulated identification results.

12. the input data is an image, the database stores a correspondence between the recognition class of the defect and a device type or a manufacturing process; 11. The analysis program according to claim 10, wherein said step (c) analyzes the correspondence between said recognition class and said device type or said manufacturing process using said database.

13. the input data is an image, the database stores a correspondence between the recognition class of the behavior and a monitoring location; 11. The analysis program according to claim 10, wherein said step (c) uses said database to analyze a correspondence relationship between said recognition class of behavior and said monitoring location.

14. the input data is an image, the database stores a correspondence between the recognition class of the behavior and its applicability to the behavior to be detected; The analysis program according to claim 10 , wherein the step (c) analyzes a correspondence relationship between the recognition class and the applicability to the detection target behavior using the database.

15. the machine learning model is a neural network; The step (b) inputs the input data to the neural network and converts the input data into one of the recognition classes and a confidence level of the recognition class. The analysis program according to claim 11.

16. the input data is an image, The analysis program according to claim 10, wherein the processing further comprises a step (d) of switching the database used for the analysis in step (c) or limiting it to a part of the database depending on information about a product included in the image or information about a location of a person included in the image.

17. 11. The analysis program according to claim 10, wherein the step (c) weights the certainty factor according to the importance level set for each of the recognition classes converted in the step (b), and analyzes the correspondence between the recognition class with the highest certainty factor after weighting and the specified information using the database.

18. (a) obtaining input data; (b) inputting the input data obtained in step (a) into a machine learning model pre-trained with training data to identify one of predetermined recognition classes; (c) analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and step (d) of outputting the analysis results of step (c), the machine learning model is a neural network; The step (b) inputs the input data to the neural network and converts the input data into one of the recognition classes and a confidence level of the recognition class; The analysis method, wherein the step (c) does not perform analysis if the confidence of the recognition class converted in the step (b) is less than a predetermined threshold.

19. (a) obtaining input data; (b) inputting the input data obtained in step (a) into a machine learning model pre-trained with training data to identify one of predetermined recognition classes; (c) analyzing the correspondence between the recognition class and the predetermined information using a database that stores the correspondence between the recognition class and the predetermined information; and step (d) of outputting the analysis results of step (c), the input data is an image, the database stores a correspondence between the recognition class of the defect and a device type or a manufacturing process; The step (c) analyzes the correspondence between the recognition class and the device type or the manufacturing process using the database; The step (c) is an analysis method in which the identification results in the step (b) for each of the images of different analysis targets of the same product are accumulated, and the device type is detected by analysis based on the accumulated identification results.

Images