Artificial intelligence model-based abnormality diagnosis method, abnormality diagnosis device using the same, and factory monitoring system

Abstract

An abnormality diagnosis device according to one embodiment of the present invention is an abnormality diagnosis device located within a factory monitoring system, and may include at least one processor; and a memory that stores at least one instruction executed through the at least one processor. Here, the at least one instruction may include an instruction to receive image data of an object to be inspected from an image sensor; and an instruction to diagnose the presence or absence of an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model.

Description

[Technical field]

[0001] This application claims the benefit of the filing dates of Korean Patent Application No. 10-2022-0109750 filed with the Korean Intellectual Property Office on August 31, 2022, and Korean Patent Application No. 10-2023-0090949 filed with the Korean Intellectual Property Office on July 13, 2023, and all of the contents disclosed in the documents of said Korean patent applications are incorporated herein by reference.

[0002] The present invention relates to an abnormality diagnosis method, an abnormality diagnosis device, and a factory monitoring system, and more specifically, to an abnormality diagnosis method for diagnosing an object to be inspected using an artificial intelligence model-based diagnostic model, and an abnormality diagnosis device and a factory monitoring system using the same. [Background technology]

[0003] Secondary batteries are batteries that can be reused by charging after discharging. They can be used as energy sources for small devices such as mobile phones, tablet PCs, and vacuum cleaners, and are also used as medium- to large-sized energy sources for personal mobility, automobiles, and energy storage systems (ESS) for smart grids.

[0004] Secondary batteries are used in the form of assemblies such as a battery module in which a number of battery cells are connected in series and parallel, or a battery pack in which battery modules are connected in series and parallel, depending on the requirements of the system.

[0005] Various processes are required to produce a battery cell or a battery assembly. Here, automated equipment for each process is installed on a production line, and the automated equipment is systematically connected to enable mass production of battery cells or battery assemblies.

[0006] To ensure that the quality of finished products meets the required standards, monitoring is performed at major processing points on the production line. Generally, a video-based monitoring system (FMVS: Factory Monitoring Visual System) is used, using cameras installed at major processing points on the production line. In the case of FMVS, workers must visually check the images output on a display to determine whether there are any abnormalities in a specific piece of equipment or processing point, making real-time process management impossible and limiting the accuracy of diagnosis.

[0007] In order to solve the above problems, a suitable factory monitoring technology is required that can enable real-time process management and greatly improve diagnostic accuracy. Summary of the Invention [Problem to be solved by the invention]

[0008] SUMMARY OF THE PRESENT EMBODIMENT In order to solve the above problems, an object of the present invention is to provide an abnormality diagnosis device based on an artificial intelligence model.

[0009] Another object of the present invention to solve the above problems is to provide an abnormality diagnosis method using such an abnormality diagnosis device.

[0010] Another object of the present invention to solve the above problems is to provide a factory monitoring system including such an abnormality diagnosis device. [Means for solving the problem]

[0011] To achieve the above objective, an abnormality diagnosis device according to one embodiment of the present invention is an abnormality diagnosis device located within a factory monitoring system, and may include at least one processor; and a memory that stores at least one instruction executed through the at least one processor.

[0012] Here, the at least one instruction may include an instruction to receive image data of an object to be inspected from an image sensor; and an instruction to diagnose the presence or absence of an abnormality in the object to be inspected using the received image data and a pre-trained artificial intelligence-based diagnostic model.

[0013] The command to receive the image data may include a command to receive image data of the inspection object at a preset unit time.

[0014] The diagnostic model may be pre-trained to receive image data of the object to be inspected and output diagnostic result data including one or more of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality.

[0015] The command to diagnose the presence or absence of an abnormality in the inspection target may include a command to output diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.

[0016] Here, the command to diagnose the presence or absence of an abnormality in the inspection object may include a command to calculate an error score based on a comparison result between the normal pattern data and the image data; and a command to determine that an abnormal state has occurred in the inspection object if the calculated error score is equal to or greater than a predefined threshold value.

[0017] The command to diagnose the presence or absence of an abnormality in the inspection object may include a command to compare the received image data with pre-stored standard image data; a command to correct a position of a region of interest defined as at least a portion of the received image data based on the comparison result; and a command to diagnose the presence or absence of an abnormality in the inspection object based on the position of the corrected region of interest.

[0018] The command to diagnose the presence or absence of an abnormality in the object to be inspected may include a command to compare the received image data with pre-stored standard image data; a command to derive a correction function for converting the received image data to the standard image data based on the comparison result; a command to correct the received image data using the derived correction function; and a command to input the corrected image data to the diagnostic model.

[0019] The command to diagnose whether or not there is an abnormality in the object to be inspected may include a command to check the operating state of the object to be inspected; and, if the object to be inspected is in a stopped state, a command to not input the received image data to the diagnostic model or to invalidate the diagnostic result data output by the diagnostic model.

[0020] The command to diagnose whether the inspection object is abnormal may include a command to determine that the inspection object is in a normal state if the calculated error score is equal to or greater than a predefined threshold but is classified as a predefined exception handling type.

[0021] The at least one instruction may further include an instruction for collecting a plurality of image data determined to be in an abnormal state as a result of the diagnosis; an instruction for extracting feature data for each of the image data; and an instruction for clustering the image data into a plurality of clusters based on the extracted feature data.

[0022] Here, the at least one instruction may further include an instruction to label each of the clusters based on defect type information input by a user; and an instruction to re-train the diagnostic model using the labeled clusters.

[0023] To achieve the above-mentioned another object, an abnormality diagnosis method according to one embodiment of the present invention is an abnormality diagnosis method by an abnormality diagnosis device located in a factory monitoring system, and includes a step of receiving image data of an inspection object from an image sensor; and a step of diagnosing the presence or absence of an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model.

[0024] The step of receiving the image data may include receiving the image data of the object to be inspected at predetermined unit time intervals.

[0025] The diagnostic model may be pre-trained to receive image data of the object to be inspected and output diagnostic result data including one or more of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality.

[0026] The step of diagnosing the presence or absence of an abnormality in the inspection target may include the step of outputting diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.

[0027] Here, the step of diagnosing the presence or absence of an abnormality in the inspection object may include the steps of: calculating an error score based on a comparison result between the normal pattern data and the image data; and, if the calculated error score is equal to or greater than a predefined threshold, determining that an abnormal state has occurred in the inspection object.

[0028] The step of diagnosing the presence or absence of an abnormality in the object to be inspected may include the steps of: comparing the received image data with pre-stored standard image data; correcting a position of a region of interest defined as at least a portion of the received image data based on the comparison result; and diagnosing the presence or absence of an abnormality in the object to be inspected based on the position of the corrected region of interest.

[0029] The step of diagnosing the presence or absence of an abnormality in the inspection object may include the steps of: comparing the received image data with pre-stored standard image data; deriving a correction function for converting the received image data into the standard image data based on the comparison result; correcting the received image data using the derived correction function; and inputting the corrected image data into the diagnostic model.

[0030] The step of diagnosing whether or not there is an abnormality in the object to be inspected may include a step of checking the operating state of the object to be inspected; and, if the object to be inspected is in a stopped state, a step of not inputting the received image data to the diagnostic model or invalidating the diagnostic result data output by the diagnostic model.

[0031] The step of diagnosing whether or not the inspection object is abnormal may include a step of determining that the inspection object is in a normal state if the calculated error score is equal to or greater than a predefined threshold value but is classified as a predefined exception handling type.

[0032] The abnormality diagnosis method may further include the steps of: collecting a plurality of image data determined to be in an abnormal state as a result of the diagnosis; extracting feature data for each of the image data; and clustering the image data into a plurality of clusters based on the extracted feature data.

[0033] Here, the abnormality diagnosis method may further include the steps of labeling each of the clusters based on defect type information input by a user; and re-learning the diagnosis model using the labeled clusters.

[0034] In order to achieve the above-mentioned still another object, a factory monitoring system according to one embodiment of the present invention includes an image sensor that generates image data for at least a portion of an inspection object; and an anomaly diagnosis device that receives the image data from the image sensor and diagnoses the presence or absence of an anomaly in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model. Effect of the Invention

[0035] According to the above-described embodiment of the present invention, it is possible to perform real-time monitoring of the test objects located in the factory, and the accuracy of diagnosis of the test objects can be significantly improved. [Brief description of the drawings]

[0036] [Figure 1] 1 is a block diagram of a factory monitoring system according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a block diagram of a factory monitoring system according to another embodiment of the present invention. [Diagram 3] FIG. 2 is a reference diagram for explaining a diagnostic model according to an embodiment of the present invention. [Figure 4] FIG. 2 is a reference diagram for explaining the operation of a diagnostic model according to an embodiment of the present invention. [Diagram 5] 11 is an example of a screen of a monitoring terminal for explaining a GUI according to an embodiment of the present invention. [Figure 6] ~ [Figure 9] 1 is a reference diagram for explaining an abnormality diagnosis method according to an embodiment of the present invention; [Figure 10] 1 is a flowchart of an abnormality diagnosis method according to an embodiment of the present invention. [Figure 11] 1 is a block diagram of an abnormality diagnosis device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Since the present invention can be modified in various ways and has various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, it is understood that this is not intended to limit the present invention to the specific embodiment, but includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. Similar reference numerals are used for similar components while describing each drawing.

[0038] Terms such as first, second, A, B, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, a first component may be named a second component, and similarly, a second component may be named a first component, without departing from the scope of the present invention. The term "and / or" includes a combination of multiple associated listed items or any of multiple associated listed items.

[0039] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, but there may also be other components in between. In contrast, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no other components in between.

[0040] The terms used in this application are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and are not intended to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Terms as defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined in this application.

[0042] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. Meanwhile, the factory monitoring system according to the present invention will be described as being installed in a battery manufacturing or assembling factory to diagnose batteries or battery process equipment, but the scope of the present invention is not limited to such types of inspection objects.

[0043] FIG. 1 is a block diagram of a factory monitoring system according to an embodiment of the present invention.

[0044] Referring to FIG. 1, a factory monitoring system 100 according to the present invention may include one or more image sensors 110 and an abnormality diagnosis device 120 .

[0045] The image sensor 110 is a device that generates image data for one or more inspection targets 10 located in a factory. For example, the image sensor 110 may correspond to a camera that is fixed at a specific position.

[0046] The image sensor 110 is connected to the abnormality diagnosis device 120 through a network, and can transmit the generated image data to the abnormality diagnosis device 120 .

[0047] The image sensor 110 can generate image data for a specific region of the inspection target 10. For example, the image sensor 110 can be installed at a specific position of an assembly facility and configured to capture an image of a specific bolt fastening region. As another example, the image sensor 110 can be installed in a peripheral facility of an electrolyte injection device and configured to capture an image of the inside of the electrolyte injection device.

[0048] The image sensor 110 can generate image data of the inspection target 10 at preset unit times and transmit the image data to the abnormality diagnosis device 120. For example, the image sensor 110 can generate 60 pieces of image data per second (60 fps) and transmit the image data to the abnormality diagnosis device 120.

[0049] The factory monitoring system 100 may be configured to include a plurality of image sensors 110. Here, each of the image sensors may be arranged at a predefined position, configured to generate image data for a corresponding inspection object, and configured to transmit the generated image data to the abnormality diagnosis device 120.

[0050] The image sensor 110 may be configured to have a field of view (FOV) that is preset depending on the installation position, the type of the inspection target, the diagnosis purpose, etc. For example, an image sensor installed for diagnosis of a plurality of bolt fastening regions may be configured as a sensor having a wide field of view of 160 degrees. As another example, an image sensor installed for diagnosis of die wear may be configured as a sensor having a narrow field of view of 50 degrees.

[0051] The abnormality diagnosis device 120 is a device that receives image data from the image sensor 110 and diagnoses whether or not an abnormality exists in an inspection object based on the image data. Here, the abnormality diagnosis device 120 can diagnose whether or not an abnormality exists in an inspection object using the received image data and a predefined diagnosis model.

[0052] The abnormality diagnosis device 120 may generate diagnosis result data including one or more of the presence or absence of an abnormality, the position where the abnormality occurs, and the type of the abnormality, for the inspection target 10, using the received image data and the diagnosis model. For example, the abnormality diagnosis device 120 may be configured to detect the presence or absence of an abnormality for positional variation, occurrence of play, fastening failure, occurrence of wear, etc., and to identify the inspection target where an abnormal state occurs or to detect a specific position where an abnormal state occurs.

[0053] FIG. 2 is a block diagram of a factory monitoring system according to another embodiment of the present invention.

[0054] 2, a factory monitoring system 100 may be configured to include one or more image sensors 110, an abnormality diagnosis device 120, a diagnosis model 130, and a monitoring terminal 140. That is, the factory monitoring system 100 may be configured by further including the diagnosis model 130 and the monitoring terminal 140 in addition to the configuration of the factory monitoring system shown in FIG.

[0055] The image sensor 110 can generate image data for one or more inspection objects 10 located in a factory, and transmit the generated image data to the abnormality diagnosis device 120 .

[0056] The abnormality diagnosis device 120 receives image data from the image sensor 110 and can diagnose whether or not an abnormality exists in an object to be inspected using the received image data and a previously trained AI-based diagnosis model 130 .

[0057] The diagnostic model 130 may be configured to receive image data of an object to be inspected and output diagnostic result data. Here, the diagnostic model 130 may correspond to an artificial intelligence-based learning model that is pre-trained to receive image data of an object to be inspected as input data and output diagnostic result data including at least one of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality as output data.

[0058] In the embodiment, the diagnostic model 130 may be generated by the diagnostic model generating device 200. Here, the diagnostic model generating device 200 may pre-train the diagnostic model according to the present invention using a large amount of pre-collected training data, and provide the diagnostic model after the training process to the factory monitoring system 100.

[0059] The monitoring terminal 140 is a device that outputs at least one of an image of an inspection target and a diagnosis result. Here, the monitoring terminal 140 may be configured to receive image data from the image sensor 110 and diagnosis result data from the abnormality diagnosis device 120, and display the image data and the diagnosis result data through a predefined GUI (Graphical User Interface).

[0060] The monitoring terminal 140 outputs the image data received every time unit in real time through a display device, and if an abnormal state occurs in a particular inspection object, the monitoring terminal 140 can output a diagnosis result for the corresponding inspection object through the display device.

[0061] That is, the monitoring terminal 140 is a computing device that outputs images and abnormality information of the object to be inspected in real time, and can assist an operator in intuitively grasping the abnormal state of the object to be inspected, the location where the abnormality has occurred, and the type of the abnormality, and in quickly taking follow-up measures therefor.

[0062] FIG. 3 is a reference diagram for explaining a diagnostic model according to an embodiment of the present invention.

[0063] Referring to FIG. 3, the diagnostic model 130 can be configured to receive image data 111 received from the image sensor 110 every unit time as input data and output diagnostic result data 131 including one or more of the presence or absence of an abnormality in the object to be inspected, the location where the abnormality occurs, and the type of abnormality.

[0064] Here, the diagnostic model 130 may compare predefined normal pattern data with the received image data 111 and output diagnosis result data based on the comparison result. Specifically, the diagnostic model generating device 200 may generate normal pattern data based on image data of an inspection target in a normal state. Here, the diagnostic model 130 may be pre-trained to compare the defined normal pattern data with input image data and output diagnosis result data. For example, the diagnostic model 130 may compare image data sequentially input from an image sensor with a bolt fastening pattern image in a normal state derived based on an image of a normally fastened bolt, and output diagnosis result data 131 including the presence or absence of a bolt fastening defect and a defect position based on the comparison result.

[0065] FIG. 4 is a reference diagram for explaining the operation of the diagnostic model according to the embodiment of the present invention.

[0066] The diagnostic model 130 may calculate an error score based on a comparison result between the normal pattern data and the image data. Here, the error score may be defined as data that quantifies the difference between the normal pattern data and the input image data.

[0067] If the calculated error score is equal to or greater than a predefined threshold, the diagnostic model 130 can determine that an abnormal condition has occurred in the inspected object.

[0068] 4, the diagnostic model 130 may sequentially compare image data inputted for each time unit with normal pattern data to calculate an error score, and may determine that an abnormal condition has occurred if the error score is equal to or greater than a predefined threshold value. For example, as shown in FIG 4, if a threshold value that is a criterion for determining whether or not an abnormality exists is predefined as 1.18, it may be determined that an abnormal condition has occurred in the object to be inspected when the 8700th frame of image data is generated.

[0069] FIG. 5 is an example of a screen of a monitoring terminal for explaining a GUI according to an embodiment of the present invention.

[0070] The monitoring terminal 140 can display the image data received from the image sensor 110 and the diagnosis result data received from the abnormality diagnosis device 120 through a predefined GUI.

[0071] The monitoring terminal 140 can output diagnostic result data by item through a diagnostic information output window 510 and can output an image of an object to be inspected through an image output window 520, as shown in FIG.

[0072] The diagnostic information output window 510 can be defined to output information regarding whether an abnormality has occurred, the location where the abnormality has occurred (e.g., an identifier of the object being inspected or the image sensor), the time when the abnormality has occurred, and the type of abnormality, according to a predefined layout.

[0073] The image output window 520 can be defined to output an image of the object to be inspected. Here, the monitoring terminal 140 can visualize and display (for example, as a square box) the abnormality occurrence area (A) of the object to be inspected so as to be distinguished from other areas.

[0074] According to an embodiment of the present invention, it is possible to support an operator to intuitively grasp the abnormal state, abnormality location, abnormality type, etc. of an object to be inspected through the diagnosis information output window 510 and the image output window 520, and to quickly take corrective measures against the abnormal state.

[0075] 6 to 9 are reference diagrams for explaining an abnormality diagnosis method according to an embodiment of the present invention. Hereinafter, various embodiments that can further improve the diagnostic accuracy of the abnormality diagnosis device will be described with reference to Figs. 6 to 9.

[0076] At least a portion of the image data received from the image sensor can be predefined as a region of interest, and the diagnostic model can be configured to compare the region of interest with predefined normal pattern data to output diagnostic result data.

[0077] During the operation of the process equipment, the angle of the image sensor may be slightly changed due to vibration of the equipment or an operator's operation error. In this case, the area photographed by the image sensor is changed, and the area of ​​interest on the image data is shifted. As a result, the diagnostic model may erroneously diagnose that an abnormal condition has occurred even though an abnormal condition has not occurred. The abnormality diagnosis device according to the embodiment of the present invention can correct the position of the area of ​​interest by a predefined process to prevent erroneous diagnosis due to the angle shift of the image sensor.

[0078] Specifically, the abnormality diagnosis device may compare image data received from the image sensor with pre-stored standard image data. Here, the standard image data may correspond to image data of an inspection target in a normal state. The standard image data may be pre-stored in a storage device, and the abnormality diagnosis device may sequentially compare the received image data with the pre-stored standard image data.

[0079] The anomaly diagnostic device may be configured to compare feature points between standard image data and the received image data, or to compare the location of objects identified using a predefined object identification algorithm.

[0080] When the position difference value according to the comparison result is equal to or greater than a predefined threshold value, the abnormality diagnosis device may correct the position of the region of interest in the image data based on the comparison result. Here, the abnormality diagnosis device may correct the position of the region of interest on the image data to a correct position based on the position difference of the feature points or the position difference of the identified object.

[0081] Thereafter, the abnormality diagnosis apparatus can diagnose the presence or absence of an abnormality in the inspection target based on the position of the corrected region of interest. That is, the abnormality diagnosis apparatus can input the position-corrected region of interest to a diagnosis model and output diagnosis result data.

[0082] For example, referring to Fig. 6, the abnormality diagnosis apparatus may compare image data 610 received from an image sensor with pre-stored standard image data 620. Here, the abnormality diagnosis apparatus may identify a bolt fastening region by comparing feature points between the standard image data 620 and the image data 610 or by using a pre-defined object identification algorithm, and calculate a position difference value for the bolt fastening position. Then, the abnormality diagnosis apparatus may change the position of a region of interest (ROI) on the image data 610 using the calculated position difference value, as shown in Fig. 6. Then, the abnormality diagnosis apparatus may input the position-corrected ROI to a diagnosis model and output diagnosis result data.

[0083] During the operation of the process equipment, the characteristics of the image data may change due to changes in the surrounding environment of the inspection target. For example, when the lighting around the inspection target is turned off or the lighting brightness changes, the characteristic values ​​of the image data, such as the hue, brightness, and saturation, may change. As a result, the diagnostic model may erroneously diagnose that an abnormal condition has occurred even though an abnormal condition has not occurred. The abnormality diagnosis device according to the embodiment of the present invention may correct the image data according to a predefined process to prevent erroneous diagnosis due to changes in the surrounding environment of the inspection target.

[0084] Specifically, the abnormality diagnosis device may compare image data received from the image sensor with pre-stored standard image data. Here, the standard image data may correspond to image data of an inspection target in a normal state. The standard image data may be pre-stored in a storage device, and the abnormality diagnosis device may sequentially compare the received image data with the pre-stored standard image data.

[0085] The abnormality diagnosis device may be configured to compare characteristic values ​​(eg, one or more of hue, brightness, and saturation) of the standard image data and the received image data.

[0086] If the difference between the standard image data and the received image data is equal to or greater than a predefined threshold, the abnormality diagnosis device may derive a correction function for the image data based on the comparison result, where the correction function may be embodied as a data conversion code that converts the received image data into the standard image data.

[0087] Thereafter, the abnormality diagnosis device can diagnose whether or not an abnormality exists in the inspection target based on the image data corrected by the correction function. That is, the abnormality diagnosis device can input the corrected image data to a diagnosis model and output diagnosis result data.

[0088] For example, referring to Fig. 7, when image data (B or C) is received from an image sensor, the abnormality diagnosis device may compare characteristic values ​​of the received image data (B or C) with standard image data (A) pre-stored in a storage device. If the difference in characteristic values ​​is equal to or greater than a pre-defined threshold, the abnormality diagnosis device may derive a correction function capable of converting the received image data (B or C) into the standard image data (A). Thereafter, the abnormality diagnosis device may correct the image data (B or C) based on the derived correction function, and then input the corrected image data to a diagnosis model to output diagnosis result data.

[0089] During the operation of the process equipment, when the process equipment is switched to a stopped state, other objects may be included in the image data generated by the image sensor in addition to the object to be inspected. For example, when the process equipment is stopped, as shown in FIG. 8, a part of the body of the worker may be captured together with the object to be inspected. This may cause the diagnostic model to erroneously diagnose that an abnormal condition has occurred even though an abnormal condition has not occurred. In order to prevent erroneous diagnosis due to switching of the operating state of the object to be inspected, the abnormality diagnosis device according to the embodiment of the present invention can correct the diagnosis result or invalidate the diagnosis result according to a predefined process.

[0090] Specifically, the abnormality diagnostic device can check the operating state of the object to be inspected.

[0091] In an embodiment, the abnormality diagnosis device can check the operating state of the inspection object in cooperation with the inspection object or a control device of the inspection object.

[0092] In another embodiment, the abnormality diagnosis device may check the operating state of the inspection object using a predefined operating state detection algorithm. For example, the diagnosis model may be configured to be pre-trained to learn an operating video of the inspection object and determine whether the inspection object operates at a normal operating speed. Here, the abnormality diagnosis device may input image data received from an image sensor to the pre-trained diagnosis model to determine whether the inspection object operates at a speed lower than the normal speed, thereby checking whether the inspection object has been switched to a stopped state.

[0093] When it is confirmed that the object to be inspected has been switched to a stopped state, the abnormality diagnosis device can either not input image data received after the point at which the object to be inspected has been switched to the stopped state into the diagnostic model, or invalidate the diagnosis result data based on the diagnostic model.

[0094] Thereafter, when it is confirmed that the inspection object has been switched from a stopped state to an operating state, the abnormality diagnosis device can diagnose whether or not there is an abnormality in the inspection object based on the received image data.

[0095] When raw materials are not supplied to the process equipment due to a defect judgment in a previous process step during operation of the process equipment, the shape of the raw materials may not be included in the image data generated by the image sensor. As a result, the diagnostic model may erroneously diagnose an abnormal condition when an abnormal condition does not occur. Such a misdiagnosis case occurs very rarely during the process.

[0096] In order to prevent erroneous diagnosis, which occurs infrequently, the abnormality diagnosis device according to an embodiment of the present invention can correct or invalidate the diagnosis result by a predefined exception handling process.

[0097] Specifically, the diagnostic model may be configured by being pre-learned in a supervised learning manner so as to classify image data that is a target of exception processing into an exception processing type. Here, when the image data is classified into the exception processing type by the diagnostic model, even if the calculated error score is equal to or greater than a predefined threshold, the anomaly diagnosis device may determine that the inspection target is in a normal state or invalidate the diagnostic result data.

[0098] When diagnostic data using the diagnostic model according to the present invention is accumulated, the diagnostic model may be re-learned and updated to improve diagnostic accuracy. Here, the diagnostic model may be updated by re-learning image data labeled based on the diagnosis result.

[0099] First, the abnormality diagnosis device can collect a plurality of image data that are determined to be in an abnormal state by the diagnosis model.

[0100] Thereafter, the abnormality diagnosis device may extract feature data for each of the collected image data, and cluster the image data into a plurality of clusters based on the extracted feature data. For example, referring to FIG. 9, a two-dimensional map may be generated based on a plurality of features extracted from the collected image data, and the image data may be clustered into a plurality of clusters (C1 to C5) according to the positions of the features on the two-dimensional map. Here, each cluster may be labeled based on defect type information input by an operator. For example, C1 may be labeled as [misdiagnosis], C2 as [position defect], C3 as [exception handling], C4 as [bolt tightening defect], and C5 as [wear defect].

[0101] The anomaly diagnosis device can re-learn the diagnostic model using the image data labeled for each cluster. Here, the anomaly diagnosis device can transmit the labeled image data to the diagnostic model generation device so that the diagnostic model generation device re-learns the diagnostic model. When the re-learning is completed, the anomaly diagnosis device can receive the re-learned diagnostic model from the diagnostic model generation device and update the pre-stored diagnostic model to the received diagnostic model.

[0102] FIG. 10 is a flow diagram of an abnormality diagnosis method according to an embodiment of the present invention.

[0103] The image sensor 110 can generate image data 111 for the inspection target 10 for each unit time and transmit the image data 111 to the abnormality diagnosis device 120 (S1010).

[0104] The abnormality diagnosis device 120 can sequentially input the image data 111 received from the image sensor 110 to the pre-trained diagnosis model 130 (S1020).

[0105] The abnormality diagnosis device 120 can diagnose in real time, using the diagnosis model 130, one or more of the presence or absence of an abnormality in the inspection target, the location of the abnormality, and the type of the abnormality (S1030).

[0106] If it is determined that an abnormality has occurred in the inspection target 10, the abnormality diagnosis device 120 may output diagnosis result data including one or more of the presence or absence of an abnormality in the inspection target, the location of the abnormality, and the type of the abnormality (S1040). Here, the abnormality diagnosis device 120 may transmit the diagnosis result data to the monitoring terminal 140.

[0107] The monitoring terminal 140 can synchronize the diagnosis result data transmitted from the abnormality diagnosis device 120 with the image data received from the image sensor 110 and output the synchronized data through a display device (S1050).

[0108] FIG. 11 is a block diagram of an abnormality diagnosis device according to an embodiment of the present invention.

[0109] The abnormality diagnosis device 1100 according to an embodiment of the present invention may include at least one processor 1110, a memory 1120 for storing at least one instruction executed through the processor, and a transceiver device 1130 connected to a network for communication.

[0110] The at least one instruction may include an instruction to receive image data of an inspection object from an image sensor; and an instruction to diagnose the presence or absence of an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnostic model.

[0111] The command to receive the image data may include a command to receive image data of the inspection object at a preset unit time.

[0112] The diagnostic model may be pre-trained to receive image data of the object to be inspected and output diagnostic result data including one or more of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality.

[0113] The command to diagnose the presence or absence of an abnormality in the inspection target may include a command to output diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.

[0114] Here, the command to diagnose the presence or absence of an abnormality in the inspection object may include a command to calculate an error score based on a comparison result between the normal pattern data and the image data; and a command to determine that an abnormal state has occurred in the inspection object if the calculated error score is equal to or greater than a predefined threshold value.

[0115] The command to diagnose the presence or absence of an abnormality in the inspection object may include a command to compare the received image data with pre-stored standard image data; a command to correct a position of a region of interest defined as at least a portion of the received image data based on the comparison result; and a command to diagnose the presence or absence of an abnormality in the inspection object based on the position of the corrected region of interest.

[0116] The command to diagnose the presence or absence of an abnormality in the object to be inspected may include a command to compare the received image data with pre-stored standard image data; a command to derive a correction function for converting the received image data to the standard image data based on the comparison result; a command to correct the received image data using the derived correction function; and a command to input the corrected image data to the diagnostic model.

[0117] The command to diagnose whether or not there is an abnormality in the object to be inspected may include a command to check the operating state of the object to be inspected; and, if the object to be inspected is in a stopped state, a command to not input the received image data to the diagnostic model or to invalidate the diagnostic result data output by the diagnostic model.

[0118] The command to diagnose whether the inspection object is abnormal may include a command to determine that the inspection object is in a normal state if the calculated error score is equal to or greater than a predefined threshold but is classified as a predefined exception handling type.

[0119] The at least one instruction may further include an instruction for collecting a plurality of image data determined to be in an abnormal state as a result of the diagnosis; an instruction for extracting feature data for each of the image data; and an instruction for clustering the image data into a plurality of clusters based on the extracted feature data.

[0120] Here, the at least one instruction may further include an instruction to label each of the clusters based on defect type information input by a user; and an instruction to re-train the diagnostic model using the labeled clusters.

[0121] The abnormality diagnosis device 1100 may further include an input interface device 1140, an output interface device 1150, a storage device 1160, etc. The components included in the abnormality diagnosis device 1100 may be connected to each other via a bus 1170 and communicate with each other. The storage device 1160 may store a pre-trained diagnostic model.

[0122] Here, the processor 1110 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which the method according to the embodiment of the present invention is performed. The memory (or storage device) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory may be composed of at least one of a read only memory (ROM) and a random access memory (RAM).

[0123] The operation of the method according to the embodiment of the present invention can be embodied as a computer readable program or code in a computer readable recording medium. The computer readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. In addition, the computer readable recording medium can be distributed to computer systems connected via a network, and the computer readable program or code can be stored and executed in a distributed manner.

[0124] Some aspects of the invention have been described in the context of an apparatus, but it may also be presented in terms of a corresponding method, where a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method may be presented in terms of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0125] Although the present invention has been described with reference to preferred embodiments thereof, those skilled in the art will understand that various modifications and variations of the present invention can be made without departing from the spirit and scope of the present invention as set forth in the following claims. [Explanation of symbols]

[0126] 10: Inspection object 100: Factory monitoring system 110: Image sensor 111: Image data 120: Abnormality diagnosis device 130: Diagnostic model 131: Diagnostic result data 140: Monitoring terminal 200: Diagnostic model generator 510: Diagnostic information output window 520: Image Output Window

Claims

1. An abnormality diagnosis device located in a factory monitoring system, at least one processor; a memory for storing at least one instruction for execution by said at least one processor; The at least one instruction: instructions for receiving image data for the inspection object from the image sensor; and and an abnormality diagnosis device, the abnormality diagnosis device including an instruction for diagnosing the presence or absence of an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model.

2. The instructions for receiving the image data include:

2. The abnormality diagnosis device according to claim 1, further comprising a command for receiving image data of the inspection object at predetermined unit times.

3. The diagnostic model comprises:

2. The abnormality diagnosis device of claim 1, which is pre-trained to receive image data of the object to be inspected and output diagnosis result data including one or more of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality.

4. The command to diagnose the presence or absence of an abnormality in the inspection object is 2. The abnormality diagnosis device according to claim 1, further comprising instructions for outputting diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.

5. The command to diagnose the presence or absence of an abnormality in the inspection object is instructions for calculating an error score based on a comparison between the normal pattern data and the image data; and The abnormality diagnosis device according to claim 4 , further comprising an instruction for determining that an abnormal state has occurred in the inspection object if the calculated error score is equal to or greater than a predefined threshold value.

6. The command to diagnose the presence or absence of an abnormality in the inspection object is instructions for comparing the received image data with pre-stored standard image data; instructions for correcting a position of a region of interest defined as at least a portion of the received image data based on the comparison; and The abnormality diagnosis device according to claim 1 , further comprising a command for diagnosing the presence or absence of an abnormality in the inspection target object based on the position of the corrected region of interest.

7. The command to diagnose the presence or absence of an abnormality in the inspection object is instructions for comparing the received image data with pre-stored standard image data; instructions for deriving a correction function for converting the received image data to the standard image data based on the comparison; instructions for correcting the received image data using the derived correction function; and The abnormality diagnosis apparatus of claim 1 , further comprising instructions for inputting the corrected image data into the diagnostic model.

8. The command to diagnose the presence or absence of an abnormality in the inspection object is An instruction to verify the operational status of the inspection object; and The abnormality diagnosis device according to claim 1 , further comprising an instruction to not input the received image data to the diagnostic model or to invalidate the diagnosis result data output by the diagnostic model when the object to be inspected is in a stationary state.

9. The command to diagnose the presence or absence of an abnormality in the inspection object is The abnormality diagnosis device of claim 5, further comprising an instruction to determine that the inspection object is in a normal state if the calculated error score is equal to or greater than a predefined threshold but is classified as a predefined exception handling type.

10. instructions for acquiring a plurality of image data items that indicate an abnormal condition as a result of the diagnosis; instructions for extracting feature data for each of said image data; and The abnormality diagnosis apparatus of claim 1 , further comprising instructions for clustering the image data into a plurality of clusters based on extracted feature data.

11. instructions for labeling each of the clusters based on defect type information input by a user; and The abnormality diagnosis device according to claim 10 , further comprising instructions for retraining the diagnostic model using the labeled clusters.

12. 1. A method for diagnosing an abnormality by an abnormality diagnosis device located in a factory monitoring system, comprising: receiving image data for the inspection object from the image sensor; and The method includes a step of diagnosing the presence or absence of an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model.

13. The step of receiving image data includes: The method of claim 12, further comprising the step of receiving image data of the object to be inspected at predetermined unit times.

14. The diagnostic model comprises: The abnormality diagnosis method of claim 12, further comprising: learning in advance to receive image data of the object to be inspected and output diagnosis result data including one or more of the presence or absence of an abnormality in the object to be inspected, the location of the abnormality, and the type of the abnormality.

15. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: The method of claim 12, further comprising the step of outputting diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.

16. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: Calculating an error score based on a comparison result between the normal pattern data and the image data; and The abnormality diagnosis method according to claim 15, further comprising the step of determining that an abnormal state has occurred in the inspection object when the calculated error score is equal to or greater than a predefined threshold value.

17. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: comparing the received image data with pre-stored standard image data; correcting a position of a region of interest defined as at least a portion of the received image data based on the comparison result; and The abnormality diagnosis method according to claim 12 , further comprising the step of diagnosing the presence or absence of an abnormality in the inspection object based on the corrected position of the region of interest.

18. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: comparing the received image data with pre-stored standard image data; deriving a correction function based on the comparison result to convert the received image data to the standard image data; correcting the received image data using the derived correction function; and The method of claim 12 further comprising the step of inputting the corrected image data into the diagnostic model.

19. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: confirming the operating state of the test object; and The abnormality diagnosis method according to claim 12, further comprising a step of not inputting the received image data to the diagnostic model or invalidating the diagnosis result data output by the diagnostic model when the object to be inspected is in a stationary state.

20. The step of diagnosing the presence or absence of an abnormality in the inspection object includes: The method for diagnosing an anomaly according to claim 16, further comprising: determining that the object to be inspected is in a normal state when the calculated error score is equal to or greater than a predefined threshold value but is classified as a predefined exception handling type.

21. collecting a plurality of image data items determined to be in an abnormal state as a result of the diagnosis; extracting feature data for each of the image data; and The method of claim 12 , further comprising the step of clustering the image data into a plurality of clusters based on extracted feature data.

22. labeling each of the clusters based on defect type information input by a user; and The method for diagnosing anomalies according to claim 21 , further comprising the step of retraining the diagnostic model using the labeled clusters.

23. an image sensor for generating image data for at least a portion of an area of ​​an object to be inspected; and and an abnormality diagnosis device that receives the image data from the image sensor and diagnoses whether or not there is an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model.

Images