Artificial intelligence model-based anomaly diagnosis method, anomaly diagnosis device using the same, and factory monitoring system

Abstract

An abnormality diagnosis device according to one embodiment of the present invention is an abnormality diagnosis device located within a factory monitoring system, and may include at least one processor; and a memory that stores at least one instruction executed through the at least one processor. Here, the at least one instruction may include an instruction to receive image data of an object to be inspected from an image sensor; and an instruction to diagnose the presence or absence of an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model.

Description

Technical Field

[0001] This application claims the benefit of the filing dates of Korean Patent Application No. 10-2022-0109750, filed with the Korean Intellectual Property Office on August 31, 2022, and Korean Patent Application No. 10-2023-0090949, filed with the Korean Intellectual Property Office on July 13, 2023, and all of the content disclosed in the documents of the Korean patent applications is incorporated herein.

[0002] The present invention relates to an abnormality diagnosis method, an abnormality diagnosis apparatus, and a factory monitoring system, and more specifically, to an abnormality diagnosis method for diagnosing an inspection object using a diagnosis model based on an artificial intelligence model, an abnormality diagnosis apparatus using the same, and a factory monitoring system.

Background Art

[0003] A secondary battery is a battery that can be reused through charging even after discharge, and can be used as an energy source for small devices such as mobile phones, tablet PCs, and vacuum cleaners, and can also be used as a medium- to large-sized energy source for personal mobility, automobiles, smart grid ESS (Energy Storage System), etc.

[0004] Secondary batteries are used in the form of assemblies such as battery modules in which a number of battery cells are connected in series and parallel according to the requirements of the system, or battery packs in which battery modules are connected in series and parallel.

[0005] Various processes are required to produce battery cells or battery assemblies. Here, automated equipment for performing each process is installed on the production line, and by organically connecting the automated equipment, mass production of battery cells or battery assemblies becomes possible.

[0006] To ensure that the quality of manufactured goods meets required standards, monitoring is performed at key processing points on the production line. Generally, a video-based monitoring system (FMVS; Factory Monitoring Visual System) using cameras installed at key processing points on the production line is utilized. In the case of FMVS, operators must visually inspect the images output through a display and judge whether there are any abnormalities in specific equipment or processing points, making real-time process control impossible and limiting the accuracy of the diagnosis.

[0007] To address the problems described above, there is a need for appropriate factory monitoring technology that enables real-time process control and significantly improves diagnostic accuracy. [Overview of the project] [Problems that the invention aims to solve]

[0008] The objective of the present invention, in order to solve the above-mentioned problems, is to provide an artificial intelligence model-based anomaly diagnosis device.

[0009] Another objective of the present invention, in order to solve the problems described above, is to provide a method for diagnosing abnormalities using such an abnormality diagnosis device.

[0010] Another objective of the present invention, in order to solve the problems described above, is to provide a factory monitoring system that includes such an anomaly diagnosis device. [Means for solving the problem]

[0011] An abnormality diagnosis device according to one embodiment of the present invention for achieving the above objective is an abnormality diagnosis device located in a factory monitoring system, and may include at least one processor; and a memory for storing at least one instruction executed through the at least one processor.

[0012] Here, at least one of the above commands may include a command to receive image data of the object to be inspected from an image sensor, and a command to diagnose whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model.

[0013] The command to receive the above image data may include a command to receive the above-mentioned object to be inspected at predetermined intervals.

[0014] The above diagnostic model can be pre-trained to receive image data of the object being inspected and output diagnostic result data that includes one or more of the following: whether or not there is an abnormality in the object being inspected, the location of the abnormality, and the type of abnormality.

[0015] The command for diagnosing the presence or absence of abnormalities in the above-mentioned object may include a command to output diagnostic result data based on the comparison result between predefined normal pattern data and the received image data.

[0016] Here, the command for diagnosing the presence or absence of abnormalities in the object under inspection may include a command to calculate an error score based on the comparison result between the normal pattern data and the image data; and a command to determine that an abnormal condition has occurred in the object under inspection if the calculated error score is equal to or greater than a predefined threshold.

[0017] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include: a command to compare pre-stored standard image data with the received image data; a command to correct the position of the region of interest, defined as at least a portion of the received image data, based on the comparison result; and a command to diagnose the presence or absence of abnormalities in the object being inspected based on the corrected position of the region of interest.

[0018] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include: a command to compare the received image data with pre-stored standard image data; a command to derive a correction function to convert the received image data back to the standard image data based on the comparison result; a command to correct the received image data using the derived correction function; and a command to input the corrected image data into the diagnostic model.

[0019] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include a command to confirm the operating status of the object being inspected, and, if the object being inspected is in a stopped state, a command to either not input the received image data into the diagnostic model or to invalidate the diagnostic result data output by the diagnostic model.

[0020] The command for diagnosing the presence or absence of abnormalities in the above-mentioned object may include a command to determine that the object is in a normal state, even if the calculated error score is above a predefined threshold, if it falls into a predefined exception handling type.

[0021] The above at least one instruction may further include an instruction to collect multiple image data determined to be abnormal as a result of the diagnosis; an instruction to extract feature data for each of the above image data; and an instruction to cluster the above image data into multiple clusters based on the extracted feature data.

[0022] Here, at least one of the above instructions may further include an instruction to label each of the clusters based on defect type information entered by the user, and an instruction to retrain the diagnostic model using the labeled clusters.

[0023] An abnormality diagnosis method according to an embodiment of the present invention for achieving the above another object is an abnormality diagnosis method by an abnormality diagnosis device located in a factory monitoring system, including: receiving image data of an inspection object from an image sensor; and diagnosing whether there is an abnormality in the inspection object using the received image data and a pre-learned artificial intelligence-based diagnosis model.

[0024] The step of receiving the image data may include receiving the image data of the inspection object at preset unit time intervals.

[0025] The diagnosis model can be pre-learned to output diagnosis result data including one or more of whether there is an abnormality in the inspection object, the location of the abnormality occurrence, and the type of abnormality by receiving the image data of the inspection object.

[0026] The step of diagnosing whether there is an abnormality in the inspection object may include outputting diagnosis result data based on the comparison result between the pre-defined normal pattern data and the received image data.

[0027] Here, the step of diagnosing whether there is an abnormality in the inspection object includes calculating an error score based on the comparison result between the normal pattern data and the image data; and determining that an abnormal state has occurred in the inspection object when the calculated error score is greater than or equal to a pre-defined threshold.

[0028] The step of diagnosing whether there is an abnormality in the inspection object includes comparing the pre-stored standard image data with the received image data; correcting the positions of at least some regions of the received image data and the defined regions of interest based on the comparison result; and diagnosing whether there is an abnormality in the inspection object based on the corrected positions of the regions of interest.

[0029] The step of diagnosing the presence or absence of abnormality in the inspection object includes: a step of comparing the pre-stored standard image data with the received image data; a step of deriving a correction function for converting the received image data into the standard image data based on the comparison result; a step of correcting the received image data using the derived correction function; and a step of inputting the corrected image data into the diagnostic model.

[0030] The step of diagnosing the presence or absence of abnormality in the inspection object can include: a step of confirming the operating state of the inspection object; and, when the inspection object is in a stopped state, a step of not inputting the received image data into the diagnostic model or invalidating the diagnostic result data output by the diagnostic model.

[0031] The step of diagnosing the presence or absence of abnormality in the inspection object can include: a step of determining that the inspection object is in a normal state even when the calculated error score is greater than or equal to a pre-defined threshold value and is classified into a pre-defined exception handling type.

[0032] The abnormality diagnosis method can further include: a step of collecting a plurality of image data determined to be in an abnormal state as a diagnosis result; a step of extracting feature data for each of the image data; and a step of clustering the image data into a plurality of clusters based on the extracted feature data.

[0033] Here, the abnormality diagnosis method can further include: a step of labeling each of the clusters based on the defect type information input by the user; and a step of re-training the diagnostic model using the labeled clusters.

[0034] A factory monitoring system according to one embodiment of the present invention for achieving the above-mentioned or other objectives may include an image sensor that generates image data for at least a portion of an object to be inspected, and an abnormality diagnosis device that receives the image data from the image sensor and diagnoses whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model. [Effects of the Invention]

[0035] According to the embodiments of the present invention described above, real-time monitoring of objects to be inspected located within a factory is possible, and the diagnostic accuracy of the objects to be inspected can be significantly improved. [Brief explanation of the drawing]

[0036] [Figure 1] This is a block diagram of a factory monitoring system according to an embodiment of the present invention. [Figure 2] This is a block diagram of a factory monitoring system according to another embodiment of the present invention. [Figure 3] This is a reference diagram illustrating a diagnostic model according to an embodiment of the present invention. [Figure 4] This is a reference diagram illustrating the operation of a diagnostic model according to an embodiment of the present invention. [Figure 5] This is an example screen of a monitoring terminal used to illustrate a GUI according to an embodiment of the present invention. [Figure 6] ~ [Figure 9] This is a reference diagram illustrating an abnormality diagnosis method according to an embodiment of the present invention. [Figure 10] This is a flowchart of the abnormality diagnosis method according to an embodiment of the present invention. [Figure 11] This is a block diagram of an abnormality diagnosis device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0037] The present invention can be modified in various ways and has many embodiments; therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this should be understood not as limiting the present invention to specific embodiments, but rather as including all modifications, equivalents, or substitutes that fall within the spirit and technical scope of the present invention. Similar reference numerals are used for similar components in the description of each drawing.

[0038] Terms such as First, Second, A, B, etc., may be used to describe various components, but the components should not be limited by such terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the First component may be named the Second component, and similarly, the Second component may be named the First component. The term "and / or" includes a combination of multiple related items or one of multiple related items.

[0039] When it is stated that one component is "linked" or "connected" to another component, it should be understood that this may mean that it is directly linked or connected to that other component, but that there may also be another component in between. Conversely, when it is stated that one component is "directly linked" or "directly connected" to another component, it should be understood that there is no other component in between.

[0040] The terms used in this application are used solely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless they are clearly different in context. In this application, terms such as “includes” or “having” are intended to specify the presence of features, figures, steps, actions, components, parts, or combinations thereof as described in the specification, and should not be understood to preemptively exclude the presence or possibility of adding one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless explicitly defined herein.

[0042] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. On the other hand, while the factory monitoring system according to the present invention will be described as being installed in a factory that produces or assembles batteries to perform diagnostics on batteries or battery process equipment, the scope of the present invention is not limited to such types of objects to be inspected.

[0043] Figure 1 is a block diagram of a factory monitoring system according to an embodiment of the present invention.

[0044] Referring to Figure 1, the factory monitoring system 100 according to the present invention can be configured to include one or more image sensors 110 and an anomaly diagnosis device 120.

[0045] The image sensor 110 is a device that generates image data for one or more objects 10 located within a factory. For example, the image sensor 110 may correspond to a camera that is fixed in a specific position.

[0046] The image sensor 110 is connected to the anomaly diagnosis device 120 via a network and can transmit the generated image data to the anomaly diagnosis device 120.

[0047] The image sensor 110 can generate image data for a specific area of ​​the object to be inspected 10. For example, the image sensor 110 can be installed at a specific location in assembly equipment and configured to photograph the fastening area of ​​a specific bolt. In another example, the image sensor 110 can be installed in peripheral equipment of an electrolyte injection device and configured to photograph the inside of the electrolyte injection device.

[0048] The image sensor 110 can generate image data of the object to be inspected 10 at preset unit time intervals and transmit it to the anomaly diagnosis device 120. For example, the image sensor 110 can generate 60 image data points per second (60fps) and transmit them to the anomaly diagnosis device 120.

[0049] The factory monitoring system 100 may be configured to include a plurality of image sensors 110. Here, each image sensor may be positioned at a predefined location and configured to generate image data for the corresponding object under inspection and to transmit the generated image data to an anomaly diagnosis device 120.

[0050] The image sensor 110 can be configured to have a pre-set field of view (FOV) depending on the installation location, the type of object being inspected, and the diagnostic purpose. For example, an image sensor installed for the diagnosis of multiple bolt fastening areas may be configured with a sensor having a wide field of view of 160 degrees. As another example, an image sensor installed for the diagnosis of mold wear may be configured with a sensor having a narrow field of view of 50 degrees.

[0051] The abnormality diagnosis device 120 receives image data from the image sensor 110 and diagnoses whether or not there is an abnormality in the object under inspection based on the image data. Here, the abnormality diagnosis device 120 can diagnose whether or not there is an abnormality in the object under inspection using the received image data and a predefined diagnostic model.

[0052] The abnormality diagnosis device 120 can use the received image data and a diagnostic model to generate diagnostic result data that includes one or more of the following: whether or not there is an abnormality in the object under inspection 10, the location where the abnormality occurred, and the type of abnormality. For example, the abnormality diagnosis device 120 can be configured to detect whether or not an abnormality has occurred, such as positional fluctuations, play, fastening failures, or wear, and to identify the object under inspection where the abnormal condition occurred, or to detect the specific location where the abnormal condition occurred.

[0053] Figure 2 is a block diagram of a factory monitoring system according to another embodiment of the present invention.

[0054] Referring to Figure 2, the factory monitoring system 100 can be configured to include one or more image sensors 110, an anomaly diagnosis device 120, a diagnostic model 130, and a monitoring terminal 140. In other words, the factory monitoring system 100 can be configured to include the diagnostic model 130 and the monitoring terminal 140 in addition to the configuration of the factory monitoring system shown in Figure 1.

[0055] The image sensor 110 can generate image data for one or more objects 10 located within the factory and transmit the generated image data to the anomaly diagnosis device 120.

[0056] The abnormality diagnosis device 120 receives image data from the image sensor 110 and can diagnose whether or not there is an abnormality in the object being inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model 130.

[0057] The diagnostic model 130 can be configured to receive image data of the object under inspection and output diagnostic result data. Here, the diagnostic model 130 can be an artificial intelligence-based learning model that has been pre-trained to receive image data of the object under inspection as input data and output diagnostic result data that includes one or more of the following as output data: whether or not there is an abnormality in the object under inspection, the location of the abnormality, and the type of abnormality.

[0058] In the embodiment, the diagnostic model 130 can be generated by the diagnostic model generation device 200. Here, the diagnostic model generation device 200 can pre-train the diagnostic model according to the present invention using a large amount of pre-collected training data, and provide the completed diagnostic model to the factory monitoring system 100.

[0059] The monitoring terminal 140 is a device that outputs one or more of the following: an image of the object being inspected and / or a diagnostic result. Here, the monitoring terminal 140 can be configured to receive image data from the image sensor 110 and diagnostic result data from the anomaly diagnosis device 120, and to display the image data and diagnostic result data via a predefined GUI (Graphical User Interface).

[0060] The monitoring terminal 140 outputs image data received hourly through a display device in real time, and if an abnormal condition occurs in a specific object being inspected, it can output the diagnostic results for that object through the display device.

[0061] In other words, the monitoring terminal 140 is a computing device that outputs video and abnormal information of the object being inspected in real time, enabling the operator to intuitively grasp the abnormal state of the object being inspected, the location of the abnormality, and the type of abnormality, and to quickly take corrective action.

[0062] Figure 3 is a reference diagram illustrating a diagnostic model according to an embodiment of the present invention.

[0063] Referring to Figure 3, the diagnostic model 130 can be configured to receive image data 111 received from the image sensor 110 at unit time intervals as input data, and to output diagnostic result data 131 that includes one or more of the following: whether or not there is an abnormality in the object under inspection, the location where the abnormality occurred, and the type of abnormality.

[0064] Here, the diagnostic model 130 can compare predefined normal pattern data with received image data 111 and output diagnostic result data based on the comparison result. Specifically, the diagnostic model generation device 200 can generate normal pattern data based on image data of an object under inspection that is in a normal state. Here, the diagnostic model 130 can be pre-trained to output diagnostic result data by comparing defined normal pattern data with input image data. For example, the diagnostic model 130 can compare image data sequentially input from an image sensor with a normal bolt fastening pattern image derived from an image of a properly bolted object, and output diagnostic result data 131 including whether or not there is a bolt fastening defect and the location of the defect based on the comparison result.

[0065] Figure 4 is a reference diagram illustrating the operation of a diagnostic model according to an embodiment of the present invention.

[0066] The diagnostic model 130 can calculate an error score based on the comparison between normal pattern data and image data. Here, the error score can be defined as data that quantifies the difference between the normal pattern data and the input image data.

[0067] If the calculated error score is above a predefined threshold, the diagnostic model 130 can determine that an abnormal condition has occurred in the object being inspected.

[0068] Referring to Figure 4, the diagnostic model 130 sequentially compares the image data input at each time unit with normal pattern data to calculate an error score. If the error score is above a predefined threshold, it can be determined that an abnormal condition has occurred. For example, as shown in Figure 4, if the threshold for determining the presence or absence of an abnormality is predefined as 1.18, it can be determined that an abnormal condition has occurred in the object being inspected when the image data for the 8700th frame is generated.

[0069] Figure 5 is an example of a monitoring terminal screen for illustrating a GUI according to an embodiment of the present invention.

[0070] The monitoring terminal 140 can display image data received from the image sensor 110 and diagnostic result data received from the anomaly diagnosis device 120 via a predefined GUI.

[0071] As shown in Figure 5, the monitoring terminal 140 can output diagnostic result data item by item through the diagnostic information output window 510 and output an image of the object being inspected through the image output window 520.

[0072] The diagnostic information output window 510 can be defined to output information regarding the presence or absence of an abnormality, the location of the abnormality (e.g., an identifier for the object being inspected or the image sensor), the time of the abnormality, and the type of abnormality, according to a predefined layout.

[0073] The image output window 520 can be defined to output an image of the object under inspection. Here, the monitoring terminal 140 can visualize and display the abnormality area (A) of the object under inspection so that it is separated from other areas (for example, displayed as a rectangular box).

[0074] According to an embodiment of the present invention, it is possible to support operators in intuitively understanding the abnormal state of an object under inspection, the location of the abnormality, and the type of abnormality through the diagnostic information output window 510 and the image output window 520, and in quickly taking corrective action.

[0075] Figures 6 to 9 are reference diagrams illustrating an abnormality diagnosis method according to an embodiment of the present invention. Hereinafter, various embodiments that can further improve the diagnostic accuracy of the abnormality diagnosis device will be described with reference to Figures 6 to 9.

[0076] At least a portion of the image data received from the image sensor can be predefined as a region of interest. The diagnostic model can then be configured to output diagnostic result data by comparing the region of interest with predefined normal pattern data.

[0077] During the operation of process equipment, the angle of the image sensor may change slightly due to equipment vibration or operator error. In this case, the area captured by the image sensor changes, causing the region of interest in the image data to shift. As a result, the diagnostic model may misdiagnose that an abnormal condition has occurred even though no abnormal condition has occurred. The abnormality diagnosis device according to the embodiment of the present invention can correct the position of the region of interest by a predefined process in order to prevent misdiagnosis due to the angular shift of the image sensor.

[0078] Specifically, the anomaly diagnosis device can compare pre-stored standard image data with image data received from an image sensor. Here, the standard image data may correspond to image data of an object under inspection in a normal state. The standard image data is pre-stored in a memory device, and the anomaly diagnosis device can sequentially compare the received image data with the pre-stored standard image data.

[0079] The anomaly diagnosis device may be configured to compare feature points between standard image data and received image data, or to compare the locations of objects identified using a predefined object identification algorithm.

[0080] If the positional difference value obtained from the comparison is greater than or equal to a predefined threshold, the anomaly diagnosis device can correct the position of the region of interest in the image data based on the comparison results. Here, the anomaly diagnosis device can correct the position of the region of interest in the image data to the correct position based on the positional difference of feature points or the positional difference of identified objects.

[0081] Subsequently, the anomaly diagnosis device can diagnose the presence or absence of an anomaly in the object under inspection based on the corrected location of the region of interest. In other words, the anomaly diagnosis device can input the location-corrected region of interest into a diagnostic model and output diagnostic result data.

[0082] For example, referring to Figure 6, the anomaly diagnosis device can compare image data 610 received from the image sensor with pre-stored standard image data 620. Here, the anomaly diagnosis device can identify the bolt fastening area and calculate a position difference value relative to the bolt fastening position by comparing the feature points of the standard image data 620 and the image data 610, or by using a pre-defined object recognition algorithm. Subsequently, the anomaly diagnosis device can change the position of the region of interest (ROI) on the image data 610 using the calculated position difference value, as shown in Figure 6. After that, the anomaly diagnosis device can input the position-corrected region of interest into the diagnostic model and output diagnostic result data.

[0083] During the operation of process equipment, the characteristics of image data may change due to changes in the surrounding environment of the object being inspected. For example, if the lighting around the object being inspected is turned off or the brightness of the lighting changes, characteristic values ​​such as hue, brightness, and saturation of the image data may change. As a result, the diagnostic model may misdiagnose that an abnormal condition has occurred even though no abnormal condition has occurred. The abnormality diagnosis device according to the embodiment of the present invention can correct the image data by a predefined process in order to prevent misdiagnosis due to changes in the surrounding environment of the object being inspected.

[0084] Specifically, the anomaly diagnosis device can compare pre-stored standard image data with image data received from an image sensor. Here, the standard image data may correspond to image data of an object under inspection in a normal state. The standard image data is pre-stored in a memory device, and the anomaly diagnosis device can sequentially compare the received image data with the pre-stored standard image data.

[0085] The anomaly diagnosis device may be configured to compare characteristic values ​​(e.g., one or more of hue, brightness, and saturation) between standard image data and received image data.

[0086] If the difference between the standard image data and the received image data exceeds a predefined threshold, the anomaly diagnosis device can derive a correction function for the image data based on the comparison result. Here, the correction function can be embodied as a data conversion code that converts the received image data back to the standard image data.

[0087] Subsequently, the anomaly diagnosis device can diagnose the presence or absence of abnormalities in the object being inspected based on the image data corrected by the correction function. In other words, the anomaly diagnosis device can input the corrected image data into a diagnostic model and output diagnostic result data.

[0088] For example, referring to Figure 7, once image data (B or C) is received from the image sensor, the anomaly diagnosis device can compare the characteristic values ​​of the received image data (B or C) with standard image data (A) pre-stored in the memory. If the difference in characteristic values ​​is greater than or equal to a pre-defined threshold, the anomaly diagnosis device can derive a correction function that can convert the received image data (B or C) to standard image data (A). Subsequently, the anomaly diagnosis device can correct the image data (B or C) based on the derived correction function, and then input the corrected image data into the diagnostic model to output diagnostic result data.

[0089] When process equipment is switched to a stopped state during operation, the image data generated by the image sensor may include objects other than the object being inspected. For example, when process equipment stops, as shown in Figure 8, a part of the worker's body may be captured along with the object being inspected. This can cause the diagnostic model to misdiagnose that an abnormal condition has occurred when no abnormal condition has occurred. The abnormality diagnosis device according to an embodiment of the present invention can correct the diagnostic results or invalidate the diagnostic results by a predefined process in order to prevent misdiagnosis due to the switching of the operating state of the object being inspected.

[0090] Specifically, an abnormality detection device can confirm the operating status of the object being inspected.

[0091] In the embodiment, the abnormality diagnosis device can check the operating status of the object being inspected in conjunction with the object being inspected or the control device of the object being inspected.

[0092] In other embodiments, the anomaly diagnosis device can check the operating state of the object under inspection using a predefined operating state sensing algorithm. For example, the diagnostic model can be configured to be pre-trained to learn from the operating video of the object under inspection and determine whether the object is operating at a normal operating speed. Here, the anomaly diagnosis device can determine whether the object under inspection has been switched to a stopped state by inputting the image data received from the image sensor into the pre-trained diagnostic model and determining whether the object is operating below a normal speed.

[0093] If it is confirmed that the object under inspection has been switched to a stopped state, the anomaly diagnosis device can either not input image data received after the point in time when the object was switched to the stopped state into the diagnostic model, or invalidate the diagnostic result data from the diagnostic model.

[0094] Subsequently, once it is confirmed that the object under inspection has switched from a stopped state to an operating state, the abnormality diagnosis device can diagnose whether or not there is an abnormality in the object under inspection based on the received image data.

[0095] If, during the operation of process equipment, raw materials are not supplied to the equipment due to a defect detection in a previous process stage, the image data generated by the image sensor may not include the shape of the raw materials. As a result, the diagnostic model may misdiagnose that an abnormal condition has occurred when no abnormal condition has occurred. Such misdiagnosis cases occur very infrequently during the process.

[0096] The anomaly diagnosis device according to an embodiment of the present invention can correct or invalidate the diagnosis results by a predefined exception handling process in order to prevent low-frequency misdiagnoses.

[0097] Specifically, the diagnostic model may be configured by pre-training it using supervised learning to classify image data subject to exception handling into exception handling types. Here, if the diagnostic model classifies the data into exception handling types, even if the calculated error score is above a predefined threshold, the anomaly diagnostic device can determine that the object under inspection is in a normal state or invalidate the diagnostic result data.

[0098] As diagnostic data using the diagnostic model according to the present invention accumulates, the diagnostic model can be retrained and updated to improve diagnostic accuracy. Here, the diagnostic model may be updated by retraining on image data labeled based on the diagnostic results.

[0099] First, the anomaly diagnosis device can collect multiple image data sets that have been determined to be abnormal by the diagnostic model.

[0100] Subsequently, the anomaly diagnosis device can extract feature data for each of the collected image data and cluster the image data into multiple clusters based on the extracted feature data. For example, referring to Figure 9, a two-dimensional map can be generated based on multiple features extracted from the collected image data, and the image data can be clustered into multiple clusters (C1 to C5) according to the location of the features on the two-dimensional map. Here, each cluster can be labeled based on defect type information entered by the operator. For example, C1 can be labeled as [Misdiagnosis], C2 as [Positional Error], C3 as [Exceptional Handling], C4 as [Bolt Fastening Error], and C5 as [Wear Error].

[0101] The anomaly diagnosis device can retrain its diagnostic model using image data labeled for each cluster. The anomaly diagnosis device can transmit the labeled image data to the diagnostic model generation device, causing the device to retrain the diagnostic model. Once retraining is complete, the anomaly diagnosis device can receive the retrained diagnostic model from the diagnostic model generation device and update its pre-stored diagnostic model with the received model.

[0102] Figure 10 is a flowchart of an abnormality diagnosis method according to an embodiment of the present invention.

[0103] The image sensor 110 can generate image data 111 of the object to be inspected 10 at unit time intervals and transmit it to the abnormality diagnosis device 120 (S1010).

[0104] The anomaly diagnosis device 120 can sequentially input the image data 111 received from the image sensor 110 into a pre-learned diagnostic model 130 (S1020).

[0105] The abnormality diagnosis device 120 can use the diagnostic model 130 to diagnose in real time one or more of the following regarding the presence or absence of abnormalities in the object under inspection, the location of the abnormality, and the type of abnormality (S1030).

[0106] If it is determined that an abnormality has occurred in the object under inspection 10, the abnormality diagnosis device 120 can output diagnostic result data that includes one or more of the following: whether or not there is an abnormality in the object under inspection, the location of the abnormality, and the type of abnormality (S1040). The abnormality diagnosis device 120 can then transmit the diagnostic result data to the monitoring terminal 140.

[0107] The monitoring terminal 140 can synchronize the diagnostic result data transmitted from the anomaly diagnosis device 120 with the image data received from the image sensor 110 and output it through the display device (S1050).

[0108] Figure 11 is a block diagram of an abnormality diagnosis device according to an embodiment of the present invention.

[0109] An abnormality diagnosis device 1100 according to an embodiment of the present invention may include at least one processor 1110, a memory 1120 that stores at least one instruction executed through the processor, and a transceiver 1130 that is connected to a network and performs communication.

[0110] At least one of the above commands may include a command to receive image data of the object to be inspected from an image sensor; and a command to diagnose whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model.

[0111] The command to receive the above image data may include a command to receive the above-mentioned object to be inspected at predetermined intervals.

[0112] The above diagnostic model can be pre-trained to receive image data of the object being inspected and output diagnostic result data that includes one or more of the following: whether or not there is an abnormality in the object being inspected, the location of the abnormality, and the type of abnormality.

[0113] The command for diagnosing the presence or absence of abnormalities in the above-mentioned object may include a command to output diagnostic result data based on the comparison result between predefined normal pattern data and the received image data.

[0114] Here, the command for diagnosing the presence or absence of abnormalities in the object under inspection may include a command to calculate an error score based on the comparison result between the normal pattern data and the image data; and a command to determine that an abnormal condition has occurred in the object under inspection if the calculated error score is equal to or greater than a predefined threshold.

[0115] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include: a command to compare pre-stored standard image data with the received image data; a command to correct the position of the region of interest, defined as at least a portion of the received image data, based on the comparison result; and a command to diagnose the presence or absence of abnormalities in the object being inspected based on the corrected position of the region of interest.

[0116] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include: a command to compare the received image data with pre-stored standard image data; a command to derive a correction function to convert the received image data back to the standard image data based on the comparison result; a command to correct the received image data using the derived correction function; and a command to input the corrected image data into the diagnostic model.

[0117] The command for diagnosing the presence or absence of abnormalities in the object being inspected may include a command to confirm the operating status of the object being inspected, and, if the object being inspected is in a stopped state, a command to either not input the received image data into the diagnostic model or to invalidate the diagnostic result data output by the diagnostic model.

[0118] The command for diagnosing the presence or absence of abnormalities in the above-mentioned object may include a command to determine that the object is in a normal state, even if the calculated error score is above a predefined threshold, if it falls into a predefined exception handling type.

[0119] The above at least one instruction may further include an instruction to collect multiple image data determined to be abnormal as a result of the diagnosis; an instruction to extract feature data for each of the above image data; and an instruction to cluster the above image data into multiple clusters based on the extracted feature data.

[0120] Here, at least one of the above instructions may further include an instruction to label each of the clusters based on defect type information entered by the user, and an instruction to retrain the diagnostic model using the labeled clusters.

[0121] The anomaly diagnosis device 1100 may further include an input interface device 1140, an output interface device 1150, a storage device 1160, and the like. Each component included in the anomaly diagnosis device 1100 is connected by a bus 1170 and can communicate with one another. The storage device 1160 can store pre-trained diagnostic models.

[0122] Here, processor 1110 can mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which the method according to the embodiment of the present invention is performed. Memory (or storage device) can consist of at least one of a volatile storage medium and a non-volatile storage medium. For example, memory can consist of at least one of a read-only memory (ROM) and a random access memory (RAM).

[0123] The operation of the method according to an embodiment of the present invention can be embodied as a computer-readable program or code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Furthermore, computer-readable recording media can be distributed across networked computer systems, allowing computer-readable programs or code to be stored and executed in a distributed manner.

[0124] Some aspects of the present invention have been described in the context of apparatus, but they can also be described by corresponding methods, where a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method can be described by corresponding blocks or items or features of corresponding apparatus. Some or all of the method steps can be carried out by (or using) hardware devices such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps can be carried out by such devices.

[0125] While preferred embodiments of the present invention have been described above with reference to the present invention, those skilled in the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and scope of the invention as set forth in the following claims. [Explanation of symbols]

[0126] 10: Items to be inspected 100: Factory Monitoring System 110: Image sensor 111: Image data 120: Anomaly Diagnosis Device 130: Diagnostic Model 131: Diagnostic results data 140: Monitoring terminal 200: Diagnostic Model Generator 510: Diagnostic Information Output Window 520: Image output window

Claims

1. An anomaly diagnostic device located within a factory monitoring system, At least one processor; Includes a memory for storing at least one instruction executed through the aforementioned at least one processor; The at least one instruction is, A command to receive image data of the object to be inspected from the image sensor; and The command includes a command to diagnose whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model. The command to diagnose whether or not there is an abnormality in the object to be inspected is, A command to confirm the operating state of the object to be inspected; and If the object to be inspected is in a stopped state, the command includes either not inputting the received image data into the diagnostic model, or invalidating the diagnostic result data output by the diagnostic model. Anomaly detection device.

2. The command to receive the aforementioned image data is: The abnormality diagnosis device according to claim 1, which includes a command to receive image data of the object to be inspected at predetermined unit time intervals.

3. The aforementioned diagnostic model, The abnormality diagnosis device according to claim 1, which is pre-trained to output diagnostic result data including one or more of the following: whether or not there is an abnormality in the object to be inspected, the location where the abnormality occurred, and the type of abnormality, by receiving image data of the object to be inspected.

4. The command to diagnose whether or not there is an abnormality in the object to be inspected is, An abnormality diagnosis device according to claim 1, comprising a command to output diagnostic result data based on the result of comparing predefined normal pattern data with the received image data.

5. The command to diagnose whether or not there is an abnormality in the object to be inspected is, A command to calculate an error score based on the comparison result between the normal pattern data and the image data; and The abnormality diagnosis device according to claim 4, which includes an instruction to determine that an abnormal condition has occurred in the object to be inspected if the calculated error score is equal to or greater than a predefined threshold.

6. The command to diagnose whether or not there is an abnormality in the object to be inspected is, An instruction to compare pre-stored standard image data with the received image data; Based on the comparison results, an instruction to correct the position of the region of interest defined as at least a portion of the received image data; and An abnormality diagnosis device according to claim 1, comprising an instruction to diagnose whether or not there is an abnormality in the object to be inspected based on the position of the corrected region of interest.

7. The command to diagnose whether or not there is an abnormality in the object to be inspected is, An instruction to compare pre-stored standard image data with the received image data; An instruction to derive a correction function that converts the received image data into the standard image data based on the comparison results; A command to correct the received image data using the derived correction function; and The abnormality diagnosis device according to claim 1, comprising a command to input the corrected image data into the diagnostic model.

8. The command to diagnose whether or not there is an abnormality in the object to be inspected is, An abnormality diagnosis device according to claim 5, which includes an instruction to determine that the object to be inspected is in a normal state if the calculated error score is above a predefined threshold but is classified into a predefined exception handling type.

9. A command to collect multiple image data that have been determined to be abnormal as a result of the diagnosis; Commands to extract feature data for each of the aforementioned image data; and The anomaly diagnosis device according to claim 1, further comprising an instruction to cluster the image data into a plurality of clusters based on extracted feature data.

10. An instruction to label each of the clusters based on defect type information entered by the user; and The anomaly diagnosis device according to claim 9, further comprising an instruction to retrain the diagnostic model using the labeled clusters.

11. An anomaly diagnosis method using an anomaly diagnosis device located within a factory monitoring system, The steps of receiving image data of the object to be inspected from an image sensor; and The step includes diagnosing whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: A step of confirming the operating state of the object to be inspected; and If the object to be inspected is in a stopped state, the step includes either not inputting the received image data into the diagnostic model, or invalidating the diagnostic result data output by the diagnostic model. Methods for diagnosing abnormalities.

12. The step of receiving the aforementioned image data is: The abnormality diagnosis method according to claim 11, further comprising the step of receiving image data of the object to be inspected at predetermined unit time intervals.

13. The aforementioned diagnostic model, The abnormality diagnosis method according to claim 11, wherein the system is pre-trained to output diagnostic result data that includes one or more of the following: whether or not there is an abnormality in the object to be inspected, the location where the abnormality occurred, and the type of abnormality, by receiving image data of the object to be inspected.

14. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: The abnormality diagnosis method according to claim 11, further comprising the step of outputting diagnostic result data based on the result of comparing predefined normal pattern data with the received image data.

15. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: A step of calculating an error score based on the comparison result between the normal pattern data and the image data; and The abnormality diagnosis method according to claim 14, further comprising the step of determining that an abnormal condition has occurred in the object to be inspected if the calculated error score is equal to or greater than a predefined threshold.

16. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: A step of comparing pre-stored standard image data with the received image data; A step of correcting the location of the region of interest defined as at least a portion of the received image data based on the comparison results; and The abnormality diagnosis method according to claim 11, further comprising the step of diagnosing whether or not there is an abnormality in the object to be inspected based on the position of the corrected region of interest.

17. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: A step of comparing pre-stored standard image data with the received image data; A step of deriving a correction function that converts the received image data into the standard image data based on the comparison results; The steps of correcting the received image data using the derived correction function; and The abnormality diagnosis method according to claim 11, further comprising the step of inputting the corrected image data into the diagnostic model.

18. The step of diagnosing whether or not there is an abnormality in the object to be inspected is: The abnormality diagnosis method according to claim 15, further comprising the step of determining that the object under inspection is in a normal state if the calculated error score is above a predefined threshold but is classified into a predefined exception handling type.

19. The step of collecting multiple image data that have been determined to be abnormal as a result of the diagnosis; A step of extracting feature data for each of the aforementioned image data; and The anomaly diagnosis method according to claim 11, further comprising the step of clustering the image data into a plurality of clusters based on extracted feature data.

20. A step of labeling each of the clusters based on defect type information entered by the user; and The anomaly diagnosis method according to claim 19, further comprising the step of retraining the diagnostic model using the labeled clusters.

21. An image sensor that generates image data for at least a portion of the object being inspected; and The device includes an abnormality diagnosis device that receives image data from the image sensor and diagnoses whether or not there is an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model. The aforementioned abnormality diagnosis device is, The operating state of the object to be inspected is confirmed, If the object to be inspected is in a stationary state, the received image data is either not input to the diagnostic model, or the diagnostic result data output by the diagnostic model is invalidated. It is configured to Factory monitoring system.

Images