Abnormality diagnosis method based on artificial intelligence model, abnormality diagnosis device, and factory monitoring system using same
By using artificial intelligence-based diagnostic models in the factory monitoring system for image data analysis, the problem of insufficient diagnostic accuracy in process management in the existing technology is solved, real-time abnormality detection and accurate diagnosis of battery production lines is realized, and the product is improved Quality control.
Patent Information
- Application Number
- CN202380033147.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-13
- Filing Date
- 2023-08-30
- Publication Date
- 2024-11-19
AI Technical Summary
The existing factory monitoring vision system has limited diagnostic accuracy in real-time process management, making it difficult to achieve real-time abnormality detection on the battery production line.
Using an artificial intelligence-based diagnostic model, the image data is received through the image sensor and compared with the pre-trained diagnostic model, the error score is calculated, if the threshold is exceeded, the abnormality is determined, the location of the region of interest and the image data are further corrected, and the diagnosis is carried out, and the Improve diagnostic accuracy by clustering and retraining the model.
It significantly improves the diagnostic accuracy of the inspection objects, realizes real-time monitoring and abnormal detection, and improves the product quality control capabilities of the production line.
Smart Images

Figure CN118984979A_ABST
Abstract
Description
Technical Field
[0001] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0109750 filed in the Korean Intellectual Property Office on August 31, 2022, and Korean Patent Application No. 10-2023-0090949 filed in the Korean Intellectual Property Office on July 13, 2023, the entire contents of which are incorporated herein by reference.
[0002] The present invention relates to an abnormality diagnosis method, an abnormality diagnosis device and a factory monitoring system, and more particularly, to an abnormality diagnosis method, an abnormality diagnosis device and a factory monitoring system using the device for diagnosing an inspection object using a diagnosis model based on artificial intelligence. Background Art
[0003] Secondary batteries that can be recharged and reused are used as energy sources for small devices such as smartphones, tablet PCs, and vacuum cleaners, and medium and large energy sources for energy storage systems (ESS) for personal mobile devices, automobiles, and smart grids.
[0004] Depending on the requirements of a system, the secondary battery may be used in the form of an assembly such as a battery module in which a plurality of battery cells are connected in series or in parallel or a battery pack in which the battery modules are connected in series or in parallel.
[0005] The production of battery cells or battery assemblies requires various processes. Here, automated equipment that performs each process is installed on the production line, and multiple automated equipment are organically connected to make mass production of battery cells or battery assemblies possible.
[0006] To ensure that product quality meets the required standards, monitoring is implemented at major processing points along the production line. Typically, a video-based monitoring system (FMVS; Factory Monitoring Vision System) using camera equipment installed at major processing points on the production line is used. In the case of FMVS, real-time process management is impossible because an operator must visually check an image output through a display and determine whether there is an abnormality in a specific device or processing point, and there are limitations in diagnostic accuracy.
[0007] As a technology for solving the above-mentioned problems, there is a need for a suitable plant monitoring solution that can realize real-time process management with significantly improved diagnostic accuracy. Summary of the invention
[0008] Technical issues
[0009] To eliminate one or more problems of the related art, an embodiment of the present disclosure provides an abnormality diagnosis device based on an artificial intelligence model.
[0010] To eliminate one or more problems of the related art, embodiments of the present disclosure also provide an abnormality diagnosis method performed by an abnormality diagnosis device.
[0011] To eliminate one or more problems of the related art, embodiments of the present disclosure also provide a plant monitoring system including an abnormality diagnosis device.
[0012] Technical Solution
[0013] To achieve the purpose of the present disclosure, an abnormality diagnosis device located in a plant monitoring system may include: at least one processor; and a memory configured to store at least one instruction executed by the at least one processor.
[0014] Here, the at least one instruction may include: an instruction to receive image data related to the inspection object from an image sensor; and an instruction to diagnose an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnostic model.
[0015] The instruction to receive the image data may include an instruction to receive the image data of the inspection object per a preset unit time.
[0016] The diagnosis model may be pre-trained to output diagnosis result data by receiving image data of the inspection object, the diagnosis result data including one or more of whether the inspection object is abnormal, a location where the abnormality occurs, and a type of the abnormality.
[0017] The instruction to diagnose the abnormality in the inspection object may include an instruction to output diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.
[0018] The instructions for diagnosing an abnormality in the inspection object may include: instructions for calculating an error score based on a comparison result between the normal pattern data and the image data; and instructions for determining that an abnormal state has occurred in the inspection object when the calculated error score is greater than or equal to a predefined threshold.
[0019] Instructions for diagnosing an abnormality in the inspection object may include: instructions for comparing received image data with pre-stored standard image data; instructions for correcting the position of a region of interest defined as at least a portion of the received image data based on the comparison result; and instructions for diagnosing whether an abnormality exists in the inspection object based on the corrected position of the region of interest.
[0020] Instructions for diagnosing abnormalities in the inspection object may include: instructions for comparing received image data with pre-stored standard image data; instructions for deriving a correction function for converting the received image data into the standard image data based on the comparison result; instructions for correcting the received image data using the derived correction function; and instructions for inputting the corrected image data into the diagnostic model.
[0021] The instructions for diagnosing an abnormality in the inspection object may include: instructions for checking the operating status of the inspection object; and instructions for not inputting the received image data into the diagnostic model or invalidating the diagnostic result data output by the diagnostic model when the inspection object is in a stationary state.
[0022] The instructions for diagnosing an abnormality in the inspection object may include instructions for determining that the inspection object is in a normal state even if a calculated error score is equal to or greater than a predefined threshold value, if the image data is classified as a predefined exception processing type.
[0023] The at least one instruction may also include: an instruction to collect multiple image data determined to be in an abnormal state as a result of the diagnosis; an instruction to extract feature data of each of the image data; and an instruction to cluster the image data into multiple clusters based on the extracted feature data.
[0024] The at least one instruction may further include: an instruction for labeling each of the clusters based on defect type information input by a user; and an instruction for retraining the diagnosis model using the labeled clusters.
[0025] According to another embodiment of the present disclosure, an abnormality diagnosis method performed by an abnormality diagnosis device located within a factory monitoring system may include the following steps: receiving image data related to an inspection object from an image sensor; and using the received image data and a pre-trained artificial intelligence-based diagnostic model to diagnose abnormalities in the inspection object.
[0026] Receiving the image data related to the inspection object may include receiving the image data of the inspection object every preset unit time.
[0027] The diagnosis model may be pre-trained to output diagnosis result data by receiving image data of the examination object, the diagnosis result data including one or more of whether the examination object is abnormal, a location where the abnormality occurs, and a type of the abnormality.
[0028] Diagnosing the abnormality in the examination object may include outputting diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.
[0029] Diagnosing an abnormality in the inspection object may include: calculating an error score based on a comparison result between the normal pattern data and the image data; and determining that an abnormal state has occurred in the inspection object when the calculated error score is greater than or equal to a predefined threshold.
[0030] Diagnosing an abnormality in the inspection object may include: comparing received image data with pre-stored standard image data; correcting a position of a region of interest based on the comparison result, the region of interest being defined as at least a portion of the received image data; and diagnosing whether an abnormality exists in the inspection object based on the corrected position of the region of interest.
[0031] Diagnosing an abnormality in the inspection object may include: comparing received image data with pre-stored standard image data; deriving a correction function for converting the received image data into the standard image data based on the comparison result; correcting the received image data using the derived correction function; and inputting the corrected image data into the diagnostic model.
[0032] Diagnosing an abnormality in the inspection object may include: identifying an operating state of the inspection object; and when the inspection object is in a stationary state, not inputting received image data into the diagnosis model or invalidating diagnosis result data output by the diagnosis model.
[0033] Diagnosing the abnormality in the inspection object may include determining that the inspection object is in a normal state even if a calculated error score is equal to or greater than a predefined threshold value in a case where the image data is classified as a predefined exception processing type.
[0034] The method may further include the steps of: collecting a plurality of image data determined to be in an abnormal state as a result of the diagnosis; extracting feature data of each of the image data; and clustering the image data into a plurality of clusters based on the extracted feature data.
[0035] The method may further include the steps of: performing labeling on each of the clusters based on defect type information input by a user; and retraining the diagnosis model using the labeled clusters.
[0036] According to another embodiment of the present disclosure, a factory monitoring system may include: an image sensor, which is configured to generate image data related to at least a portion of an inspection object; and an abnormality diagnosis device, which is configured to receive the image data from the image sensor and use the received image data and a pre-trained artificial intelligence-based diagnostic model to diagnose abnormalities in the inspection object.
[0037] Beneficial Effects
[0038] According to the embodiments of the present disclosure, an inspection object located in a factory can be monitored in real time, so that the diagnostic accuracy of the inspection object is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a block diagram of a plant monitoring system according to an embodiment of the present invention.
[0040] Figure 2 is a block diagram of a plant monitoring system according to another embodiment of the present invention.
[0041] Figure 3 is a reference diagram for explaining a diagnosis model according to an embodiment of the present invention.
[0042] Figure 4 2 is a reference diagram for explaining the operation of the diagnosis model according to the embodiment of the present invention.
[0043] Figure 5 FIG. 1 is an example of a screen of a monitoring terminal for explaining a GUI according to an embodiment of the present invention.
[0044] Figures 6 to 9 2 is a reference diagram for explaining an abnormality diagnosis method according to an embodiment of the present invention.
[0045] Fig.10 is an operation flow chart of an abnormality diagnosis method according to an embodiment of the present invention.
[0046] Fig.11 is a block diagram of an abnormality diagnostic apparatus according to an embodiment of the present invention.
[0047] 10: Check Object
[0048] 100: Factory Monitoring System
[0049] 110: Image sensor
[0050] 111: Image data
[0051] 120: Abnormal diagnosis equipment
[0052] 130: Diagnostic Model
[0053] 131: Diagnosis result data
[0054] 140: Monitoring terminal
[0055] 200: Diagnostic model generation device
[0056] 510: Diagnostic information output window
[0057] 520: Image output window DETAILED DESCRIPTION
[0058] The present invention can be modified in various forms and has various embodiments, and its specific embodiments are shown in the accompanying drawings by way of example and will be described in detail below. However, it should be understood that it is not intended to limit the present invention to specific embodiments, but on the contrary, the present invention will cover all modifications, equivalents and alternative forms that fall within the spirit and technical scope of the present invention. Throughout the description of the drawings, the same reference numerals refer to the same elements.
[0059] It should be understood that, although various elements may be described herein using terms such as "first", "second", A, B, etc., these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any one of a combination of a plurality of associated listed items or a plurality of associated listed items.
[0060] It should be understood that when an element is referred to as being "coupled" or "connected" to another element, it can be directly coupled or connected to the other element, or there can be intermediate elements. Conversely, when an element is referred to as being "directly coupled" or "directly connected" to another element, there are no intermediate elements.
[0061] The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that the terms "include", "comprise", "include", "include" and / or "have" when used herein indicate the presence of the features, wholes, steps, operations, constituent elements, parts and / or combinations thereof, and do not exclude the presence or addition of one or more other features, wholes, steps, operations, constituent elements, parts and / or combinations thereof.
[0062] Unless otherwise specified, all terms (including technical and scientific terms) used in this article have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless so explicitly defined in this article.
[0063] Hereinafter, various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the factory monitoring system according to the present invention will be described by taking a factory that can be installed in a factory that produces or assembles batteries and performs diagnosis on batteries or battery processing equipment as an example, but the scope of the present invention is not limited to these types of inspection objects.
[0064] Figure 1 is a block diagram of a plant monitoring system according to an embodiment of the present invention.
[0065] Reference Figure 1 , the plant monitoring system 100 according to the present invention may be configured to include one or more image sensors 110 and an abnormality diagnosis device 120 .
[0066] The image sensor 110 may be a device that generates image data for one or more inspection objects 10 located in a factory. For example, the image sensor 110 may correspond to a camera fixed to a specific location.
[0067] The image sensor 110 may be connected to the abnormality diagnosis apparatus 120 through a network and transmit the generated image data to the abnormality diagnosis apparatus 120 .
[0068] The image sensor 110 may generate image data of a specific area of the inspection object 10. For example, the image sensor 110 may be installed at a specific location in an assembly facility and configured to photograph a fastening area of a specific bolt. For another example, the image sensor 110 may be installed in a facility near an electrolyte injection device and configured to photograph the interior of the electrolyte injection device.
[0069] The image sensor 110 may generate image data of the inspection object 10 every preset unit time and transmit the image data to the abnormality diagnosis apparatus 120. For example, the image sensor 110 may generate 60 image data per second (60fps) and transmit the image data to the abnormality diagnosis apparatus 120.
[0070] The plant monitoring system 100 may be configured to include a plurality of image sensors 110 . Here, each image sensor may be arranged at a predefined position, and may be configured to generate image data of a corresponding inspection object and transmit the generated image data to the abnormality diagnosis device 120 .
[0071] The image sensor 110 may be configured to have a preset field of view (FOV) according to the installation location, the type of inspection object, the purpose of diagnosis, etc. For example, an image sensor installed for diagnosing a plurality of bolt tightening areas may be configured as a sensor with a wide field of view of 160 degrees. For another example, an image sensor installed for diagnosing mold wear may be configured as a sensor with a narrow field of view of 50 degrees.
[0072] The abnormality diagnosis apparatus 120 may be a device that receives image data from the image sensor 110 and diagnoses abnormalities in the inspection object based on the image data. Here, the abnormality diagnosis apparatus 120 may diagnose whether abnormalities exist in the inspection object using the received image data and a predefined diagnosis model.
[0073] The abnormality diagnosis device 120 may use the received image data and the diagnosis model to generate diagnosis result data including one or more of the presence or absence of an abnormality, the location of the abnormality, and the type of the abnormality in the inspection object 10. For example, the abnormality diagnosis device 120 may be configured to detect whether an abnormality regarding position change, the occurrence of a gap, poor fastening, or the occurrence of wear has occurred, and specify the inspection object where the abnormal condition has occurred or detect a specific location where the abnormal condition has occurred.
[0074] Figure 2 is a block diagram of a plant monitoring system according to another embodiment of the present invention.
[0075] Reference Figure 2 , the plant monitoring system 100 may be configured to include one or more image sensors 110, an abnormality diagnosis device 120, a diagnosis model 130, and a monitoring terminal 140. In other words, in addition to Figure 1 In addition to the configuration of the plant monitoring system shown in , the plant monitoring system 100 may also be configured to include a diagnosis model 130 and a monitoring terminal 140 .
[0076] The image sensor 110 may generate image data of one or more inspection objects 10 located in a factory and transmit the generated image data to the abnormality diagnosis apparatus 120 .
[0077] The abnormality diagnosis apparatus 120 may receive image data from the image sensor 110 , and diagnose abnormalities in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model 130 .
[0078] The diagnostic model 130 may be configured to receive image data of the inspection object and output diagnostic result data. Here, the diagnostic model 130 may correspond to a pre-trained artificial intelligence-based learning model, so that the diagnostic model receives image data related to the inspection object as input data, and outputs diagnostic result data as output data, the diagnostic result data including one or more of the presence or absence of an abnormality in the inspection object, the location where the abnormality occurs, and the type of the abnormality.
[0079] In an embodiment, the diagnosis model 130 may be generated by the diagnosis model generation device 200. Here, the diagnosis model generation device 200 may pre-train the diagnosis model according to the present invention using a plurality of pre-collected learning data, and provide the trained diagnosis model to the plant monitoring system 100.
[0080] The monitoring terminal 140 may be a device that outputs one or more of an image of the inspection object and a diagnosis result. Here, the monitoring terminal 140 may be configured to receive image data from the image sensor 110, receive diagnosis result data from the abnormality diagnosis device 120, and display the image data and the diagnosis result through a predetermined graphical user interface (GUI).
[0081] The monitoring terminal 140 may output the image data received per unit time in real time through the display device, and when an abnormal condition occurs in a specific inspection object, output a diagnosis result of the inspection object through the display device.
[0082] In other words, the monitoring terminal 140 may be a computing device that outputs images and abnormality information of the inspection object in real time, thereby allowing the operator to intuitively determine the abnormal state, location and type of the inspection object, thereby supporting the operator to quickly implement subsequent measures.
[0083] Figure 3 is a reference diagram for explaining a diagnosis model according to an embodiment of the present invention.
[0084] Reference Figure 3 The diagnostic model 130 can be configured to receive image data 111 as input data, which is received from the image sensor 110 per unit time, and output diagnostic result data 131, which includes one or more of whether there is an abnormality in the inspection object, the location where the abnormality occurs, and the type of abnormality.
[0085] Here, the diagnostic model 130 may compare the predefined normal mode data with the received image data 111, and output the diagnostic result data based on the comparison result. Specifically, the diagnostic model generating device 200 may generate the normal mode data based on the image data of the inspection object in a normal state. Here, the diagnostic model 130 may be pre-trained to compare the defined normal mode data with the input image data and output the diagnostic result data. For example, the diagnostic model 130 may compare the image data sequentially input from the image sensor with the normal bolt tightening pattern image derived based on the normal bolt connection image, and output the diagnostic result data 131 including whether the bolt tightening has defects and the location of the defects based on the comparison result.
[0086] Figure 4 2 is a reference diagram for explaining the operation of the diagnosis model according to the embodiment of the present invention.
[0087] The diagnosis model 130 may calculate an error score based on the comparison result between the normal mode data and the image data. Here, the error score may be defined as data that quantifies the difference between the normal mode data and the input image data.
[0088] If the calculated error score is greater than or equal to a predetermined threshold, the diagnosis model 130 may determine that an abnormal state has occurred in the examination object.
[0089] Reference Figure 4 , the diagnosis model 130 can calculate an error score by sequentially comparing the image data input per unit time with the normal mode data, and when the error score is equal to or greater than a predetermined threshold, it is determined that an abnormal state has occurred. Figure 4 As shown in , if the threshold value as a criterion for abnormality is predefined as 1.18, it can be determined that an abnormal state has occurred in the inspection object at the time when the 8700th frame of image data is generated.
[0090] Figure 5 FIG. 1 is an example of a screen of a monitoring terminal for explaining a GUI according to an embodiment of the present invention.
[0091] The monitoring terminal 140 may display the image data received from the image sensor 110 and the diagnosis result data received from the abnormality diagnosis apparatus 120 through a predetermined Graphic User Interface (GUI).
[0092] like Figure 5 As shown in , the monitoring terminal 140 may output the diagnosis result data of each item through the diagnosis information output window 510 , and output the image of the inspection object through the image output window 520 .
[0093] The diagnosis information output window 510 may be defined to output information regarding whether an abnormality occurs, a location of the abnormality (eg, an identifier of an inspection object or an image sensor), a time when the abnormality occurs, and an abnormality type according to a predefined layout.
[0094] The image output window 520 may be defined to output an image of the inspection object. Here, the monitoring terminal 140 may visualize and display an abnormality occurrence region (A) (eg, indicated by a box) of the inspection object to distinguish it from other regions.
[0095] Therefore, the embodiments of the present invention can support the operator to intuitively determine the abnormal state, abnormal location, and abnormal type of the inspection object through the diagnosis information output window 510 and the image output window 520, and quickly implement subsequent measures.
[0096] Figures 6 to 9 is a reference diagram for explaining an abnormality diagnosis method according to an embodiment of the present invention. Figures 6 to 9 , various embodiments will be described that can further improve the diagnostic accuracy of an abnormality diagnostic device.
[0097] At least a portion of the image data received from the image sensor may be predefined as a region of interest. Here, the diagnosis model may be configured to output diagnosis result data by comparing the region of interest with predefined normal pattern data.
[0098] During the operation of the processing device, the angle of the image sensor may change slightly due to vibration of the device or an operation error of the operator. Here, the area captured by the image sensor changes, and the region of interest in the image data becomes misplaced. Therefore, even if no abnormal state occurs, the diagnostic model may misdiagnose that an abnormal state occurs. The abnormality diagnosis device according to an embodiment of the present invention can correct the position of the region of interest according to a predefined process to prevent misdiagnosis caused by the angle deformation of the image sensor.
[0099] Specifically, the abnormality diagnosis device may compare pre-stored standard image data with image data received from the image sensor. Here, the standard image data may correspond to image data of the inspection object in a normal state. The standard image data may be pre-stored in a storage device, and the abnormality diagnosis device may sequentially compare the received image data with the previously stored standard image data.
[0100] The abnormality diagnostic apparatus may be configured to compare feature points of the standard image data and the received image data, or to compare positions of objects recognized using a predefined object recognition algorithm.
[0101] If the position difference value according to the comparison result is greater than or equal to a predefined threshold, the abnormality diagnosis device can correct the position of the region of interest on the image data based on the comparison result. Here, the abnormality diagnosis device can correct the position of the region of interest in the image data to a correct position based on the position difference of the feature points or the position difference of the identified objects.
[0102] Thereafter, the abnormality diagnosis device can diagnose whether there is an abnormality in the inspection object based on the corrected position of the region of interest. In other words, the abnormality diagnosis device can input the region of interest after position correction into the diagnosis model and output diagnosis result data.
[0103] For example, refer to Figure 6 , the abnormality diagnosis device may compare the image data 610 received from the image sensor with the previously stored standard image data 620. Here, the abnormality diagnosis device may calculate the position difference of the bolt tightening area by comparing the feature points of the standard image data 620 and the image data 610 or identifying the bolt tightening area using a predefined object recognition algorithm. Thereafter, the abnormality diagnosis device may use the calculated position difference to change the position of the region of interest (ROI) on the image data 610, such as Figure 6 Thereafter, the abnormality diagnosis device may input the position-corrected region of interest into the diagnosis model and output diagnosis result data.
[0104] During the operation of the processing device, the characteristics of the image data may change due to changes in the surrounding environment of the inspection object. For example, when the lighting around the inspection object is turned off or the lighting brightness changes, the characteristic values of the image data such as color, brightness and saturation may change. Therefore, even if no abnormal state occurs, the diagnostic model may misdiagnose that an abnormal state has occurred. The abnormality diagnosis device according to an embodiment of the present invention can correct the image data according to a predefined process to prevent misdiagnosis caused by changes in the surrounding environment of the inspection object.
[0105] Specifically, the abnormality diagnosis device may compare pre-stored standard image data with image data received from the image sensor. Here, the standard image data may correspond to image data of the inspection object in a normal state. The standard image data may be pre-stored in a storage device, and the abnormality diagnosis device may sequentially compare the received image data with the previously stored standard image data.
[0106] The abnormality diagnostic apparatus may be configured to compare feature values (eg, one or more of color, brightness, and saturation) of the received image data and standard image data.
[0107] If the difference between the standard image data and the received image data is greater than or equal to a predefined threshold, the abnormality diagnosis device may derive a correction function for the image data based on the comparison result. Here, the correction function may be implemented as a data conversion code that converts the received image data into the standard image data.
[0108] Thereafter, the abnormality diagnosis device can diagnose whether there is an abnormality in the inspection object based on the image data corrected according to the correction function. In other words, the abnormality diagnosis device can input the corrected image data into the diagnosis model and output the diagnosis result data.
[0109] For example, refer to Figure 7 , when image data (B or C) is received from the image sensor, the abnormality diagnosis device may compare the received image data (B or C) with the feature value of the standard image data (A) pre-stored in the storage device. If the difference in the feature value is greater than or equal to the predefined threshold, the abnormality diagnosis device may derive a correction function that can convert the received image data (B or C) into the standard image data (A). Thereafter, the abnormality diagnosis device may correct the image data (B or C) based on the derived correction function, and then input the corrected image data into the diagnosis model to output the diagnosis result data.
[0110] If the processing device is switched to a stopped state during operation of the processing device, the image data generated by the image sensor may include objects other than the inspection object. For example, when the processing device is stopped (stationary), a part of the worker's body may be photographed together with the inspection object, such as Figure 8 As shown in . Therefore, even if no abnormal state occurs, the diagnostic model may misdiagnose that an abnormal state occurs. The abnormality diagnosis device according to an embodiment of the present invention can correct the diagnosis result or invalidate the diagnosis result according to a predefined process, thereby preventing misdiagnosis caused by a change in the operating state of the inspection object.
[0111] Specifically, the abnormality diagnostic device can identify the operating state of the inspection object.
[0112] In an embodiment, the abnormality diagnostic apparatus may check an operation state of the inspection object in conjunction with the inspection object or a control device of the inspection object.
[0113] In another embodiment, the abnormality diagnosis device may use a predefined operation state detection algorithm to check the operation state of the inspection object. For example, the diagnosis model may be configured to be pre-trained so that the diagnosis model determines whether the inspection object is operating at a normal operation speed by learning the operation image of the inspection object. Here, the abnormality diagnosis device may input the image data received from the image sensor into the pre-trained diagnosis model to determine whether the inspection object is operating at a normal speed or a lower speed, thereby confirming whether the inspection object is turned into a stopped state.
[0114] If it is confirmed that the inspection object has been switched to the stopped state, the abnormality diagnostic apparatus does not input image data received after the point of transition to the stopped state into the diagnostic model, or may invalidate the diagnostic result data of the diagnostic model.
[0115] Thereafter, when it is confirmed that the inspection object has changed from the stopped state to the operating state, the abnormality diagnostic device may diagnose whether or not an abnormality exists in the inspection object based on the received image data.
[0116] During the operation of the processing equipment, if the raw material is not supplied to the processing equipment due to defect determination in the previous processing step, the shape of the raw material is not included in the image data generated by the image sensor. Therefore, the diagnosis model may misdiagnose that an abnormal state has occurred even if no abnormal state has occurred. This misdiagnosis situation occurs at a very low frequency during the processing.
[0117] The abnormality diagnosis apparatus according to an embodiment of the present invention can correct or invalidate a diagnosis result according to a predefined exception handling procedure, thereby preventing misdiagnosis occurring at a low frequency.
[0118] Specifically, the diagnostic model can be configured to be pre-trained using a supervised learning method to classify image data subject to exception processing as an exception processing type. Here, when classified as an exception processing type by the diagnostic model, even if the calculated error score is equal to or exceeds a predefined threshold, the abnormality diagnostic device can determine that the inspection object is in a normal state, or invalidate the diagnostic result data.
[0119] When diagnostic data using the diagnostic model according to the present invention is accumulated, the diagnostic model can be retrained and updated to improve diagnostic accuracy. Here, the diagnostic model can be updated by relearning the labeled image data based on the diagnostic result.
[0120] First, the abnormality diagnosis device may collect a plurality of image data that are determined to be in an abnormal state by the diagnosis model.
[0121] Then, the abnormality diagnosis device can extract feature data for each of the collected image data, and cluster the image data into a plurality of clusters based on the extracted feature data. Fig. 9 , a two-dimensional map can be created based on multiple features extracted from the collected image data, and the image data can be clustered into multiple clusters (C1 to C5) according to the location of the features on the two-dimensional map. Here, each cluster can be labeled based on the defect type information input by the operator. For example, C1 can be labeled as [misdiagnosis], C2 can be labeled as [position defect], C3 can be labeled as [exceptional processing], C4 can be labeled as [bolt defect], and C5 can be labeled as [wear defect].
[0122] The abnormality diagnosis device may retrain the diagnosis model using the image data labeled for each cluster. Here, the abnormality diagnosis device may send the labeled image data to the diagnosis model generation device to allow the diagnosis model generation device to retrain the diagnosis model. When the retraining is completed, the abnormality diagnosis device may receive the retrained diagnosis model from the diagnosis model generation device and update the previously stored diagnosis model with the received diagnosis model.
[0123] Fig.10 is an operation flow chart of an abnormality diagnosis method according to an embodiment of the present invention.
[0124] The image sensor 110 may generate image data 111 of the inspection object 10 per unit time and transmit the image data to the abnormality diagnosis apparatus 120 ( S1010 ).
[0125] The abnormality diagnosis apparatus 120 may sequentially input the image data 111 received from the image sensor 110 into the pre-trained diagnosis model 130 ( S1020 ).
[0126] The abnormality diagnosis apparatus 120 may diagnose one or more of whether an abnormality exists in the inspection object, a location where the abnormality occurs, and a type of the abnormality in real time using the diagnosis model 130 ( S1030 ).
[0127] When determining that an abnormality occurs in the inspection object 10, the abnormality diagnosis device 120 may output diagnosis result data including one or more of whether an abnormality occurs in the inspection object, the location of the abnormality, and the type of the abnormality (S1040). Here, the abnormality diagnosis device 120 may transmit the diagnosis result data to the monitoring terminal 140.
[0128] The monitoring terminal 140 may synchronize the diagnosis result data transmitted from the abnormality diagnosis apparatus 120 with the image data received from the image sensor 110 and output them through a display device ( S1050 ).
[0129] Fig.11 is a block diagram of an abnormality diagnostic apparatus according to an embodiment of the present invention.
[0130] The abnormality diagnosis apparatus 1100 according to an embodiment of the present invention may include at least one processor 1110 , a memory 1120 storing at least one instruction executed by the processor, and a transceiver 1130 connected to a network and performing communication.
[0131] The at least one instruction may include an instruction to receive image data related to the inspection object from an image sensor; and an instruction to diagnose an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnostic model.
[0132] The instruction to receive the image data may include an instruction to receive the image data of the inspection object per preset unit time.
[0133] The diagnosis model may be pre-trained by receiving image data of the examination object to output diagnosis result data including one or more of whether the examination object is abnormal, a location where the abnormality occurs, and a type of the abnormality.
[0134] The instruction to diagnose the abnormality in the examination object may include an instruction to output diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.
[0135] Instructions for diagnosing an abnormality in the inspection object may include: instructions for calculating an error score based on a comparison result between normal pattern data and image data; and instructions for determining that an abnormal state has occurred in the inspection object when the calculated error score is greater than or equal to a predefined threshold.
[0136] Instructions for diagnosing an abnormality in an inspection object may include: instructions for comparing received image data with pre-stored standard image data; instructions for correcting the position of a region of interest defined as at least a portion of the received image data based on the comparison result; and instructions for diagnosing whether an abnormality exists in the inspection object based on the corrected position of the region of interest.
[0137] Instructions for diagnosing abnormalities in an examination object may include: instructions for comparing received image data with pre-stored standard image data; instructions for deriving a correction function for converting the received image data into standard image data based on the comparison result; instructions for correcting the received image data using the derived correction function; and instructions for inputting the corrected image data into a diagnostic model.
[0138] The instructions for diagnosing an abnormality in the inspection object may include: instructions for checking the operating state of the inspection object; and instructions for not inputting the received image data into the diagnosis model or invalidating the diagnosis result data output by the diagnosis model when the inspection object is in a stationary state.
[0139] The instructions for diagnosing abnormality in the examination object may include instructions for determining that the examination object is in a normal state even if the calculated error score is equal to or greater than a predefined threshold value in a case where the image data is classified as a predefined exception processing type.
[0140] The at least one instruction may also include: an instruction to collect a plurality of image data determined to be in an abnormal state as a result of diagnosis; an instruction to extract feature data of each image data; and an instruction to cluster the image data into a plurality of clusters based on the extracted feature data.
[0141] The at least one instruction may further include: an instruction for performing labeling on each cluster based on defect type information input by a user; and an instruction for retraining the diagnosis model using the labeled clusters.
[0142] The abnormality diagnosis apparatus 1100 may further include an input interface device 1140 , an output interface device 1150 , a storage device 1160 , etc. The various components included in the abnormality diagnosis apparatus 1100 may be connected to each other through a bus 1170 and communicate with each other.
[0143] Here, the processor 1110 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor that executes a method according to an embodiment of the present invention. The memory (or storage device) may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory may include at least one of a read-only memory (ROM) and a random access memory (RAM).
[0144] The operation of the method according to an embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. In addition, the computer-readable recording medium can be distributed in a network-connected computer system to store and execute a computer-readable program or code in a distributed manner.
[0145] Although some aspects of the present invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, wherein a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step may also represent a feature of a corresponding block or item or a corresponding apparatus. Some or all of the method steps may be performed by (or using) hardware devices such as (for example) a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.
[0146] Hereinabove, the present invention has been described with reference to exemplary embodiments thereof, but those skilled in the art will appreciate that the present invention may be variously modified and altered without departing from the spirit and scope of the invention described in the appended claims.
Claims
1. An abnormality diagnosis device located in a factory monitoring system, the abnormality diagnosis device comprising: at least one processor; as well as a memory configured to store at least one instruction executed by the at least one processor, The at least one instruction includes: instructions for receiving image data related to an inspection object from an image sensor; and Instructions for diagnosing an abnormality in the examination object using the received image data and a pre-trained artificial intelligence-based diagnostic model.
2. The abnormality diagnosis device according to claim 1, wherein: The instruction to receive the image data includes an instruction to receive the image data of the inspection object per a preset unit time.
3. The abnormality diagnosis device according to claim 1, wherein: The diagnosis model is pre-trained to output diagnosis result data by receiving image data of the inspection object, the diagnosis result data including one or more of whether the inspection object is abnormal, a location where the abnormality occurs, and a type of the abnormality.
4. The abnormality diagnosis device according to claim 1, wherein: The instruction to diagnose an abnormality in the inspection object includes an instruction to output diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.
5. The abnormality diagnosis device according to claim 4, wherein: The instructions for diagnosing an abnormality in the inspection object include: instructions for calculating an error score based on a comparison between the normal mode data and the image data; and Instructions for determining that an abnormal state has occurred in the inspection object if the calculated error score is greater than or equal to a predefined threshold.
6. The abnormality diagnosis device according to claim 1, wherein: The instructions for diagnosing an abnormality in the inspection object include: An instruction to compare the received image data with pre-stored standard image data; instructions to correct a position of a region of interest defined as at least a portion of the received image data based on the comparison; and Instructions for diagnosing whether an abnormality exists in the inspection object based on the corrected position of the region of interest.
7. The abnormality diagnosis device according to claim 1, wherein: The instructions for diagnosing an abnormality in the inspection object include: An instruction to compare the received image data with pre-stored standard image data; instructions for deriving a correction function for converting the received image data into the standard image data based on the comparison result; instructions to correct the received image data using the derived correction function; and Instructions for inputting the corrected image data into the diagnostic model.
8. The abnormality diagnosis device according to claim 1, wherein: The instructions for diagnosing an abnormality in the inspection object include: an instruction to check the operating status of the inspection object; and Instructions for not inputting received image data into the diagnosis model or for invalidating diagnosis result data output by the diagnosis model when the examination object is in a stationary state.
9. The abnormality diagnosis device according to claim 5, wherein: The instructions for diagnosing an abnormality in the inspection object include: Instructions for determining that the inspection object is in a normal state even if the calculated error score is equal to or greater than a predefined threshold value, if the image data is classified into a predefined exception processing type.
10. The abnormality diagnosis device according to claim 1, wherein: The at least one instruction further comprises: an instruction to collect a plurality of image data determined to be in an abnormal state as a result of the diagnosis; An instruction for extracting feature data of each of the image data; and Instructions for clustering the image data into a plurality of clusters based on the extracted feature data.
11. The abnormality diagnosis device according to claim 10, wherein: The at least one instruction further comprises: executing a marking instruction for each of the clusters based on defect type information input by a user; and Instructions for retraining the diagnostic model using the labeled clusters.
12. An abnormality diagnosis method performed by an abnormality diagnosis device located in a factory monitoring system, the abnormality diagnosis method comprising the following steps: receiving image data related to the inspection object from an image sensor; as well as Abnormalities in the examination object are diagnosed using the received image data and a pre-trained artificial intelligence-based diagnostic model.
13. The abnormality diagnosis method according to claim 12, wherein: The step of receiving image data related to the inspection object includes receiving the image data of the inspection object every preset unit time.
14. The abnormality diagnosis method according to claim 12, wherein: The diagnosis model is pre-trained to output diagnosis result data by receiving image data of the inspection object, the diagnosis result data including one or more of whether the inspection object is abnormal, a location where the abnormality occurs, and a type of the abnormality.
15. The abnormality diagnosis method according to claim 12, wherein: The step of diagnosing an abnormality in the inspection object includes outputting diagnosis result data based on a comparison result between predefined normal pattern data and the received image data.
16. The abnormality diagnosis method according to claim 15, wherein: The step of diagnosing an abnormality in the inspection object comprises: calculating an error score based on a comparison result between the normal mode data and the image data; and In a case where the calculated error score is greater than or equal to a predefined threshold, it is determined that an abnormal state has occurred in the inspection object.
17. The abnormality diagnosis method according to claim 12, wherein: The step of diagnosing an abnormality in the inspection object comprises: comparing the received image data with pre-stored standard image data; correcting a position of a region of interest defined as at least a portion of the received image data based on the comparison result; and Whether or not there is an abnormality in the examination object is diagnosed based on the corrected position of the region of interest.
18. The abnormality diagnosis method according to claim 12, wherein: The step of diagnosing an abnormality in the inspection object comprises: comparing the received image data with pre-stored standard image data; deriving a correction function for converting the received image data into the standard image data based on the comparison result; correcting the received image data using the derived correction function; and The corrected image data is input into the diagnostic model.
19. The abnormality diagnosis method according to claim 12, wherein: The step of diagnosing an abnormality in the inspection object comprises: identifying an operating state of the inspection object; and In a case where the inspection object is in a stationary state, the received image data is not input into the diagnosis model or the diagnosis result data output by the diagnosis model is invalidated.
20. The abnormality diagnosis method according to claim 16, wherein: The step of diagnosing an abnormality in the inspection object comprises: In a case where the image data is classified into a predefined exception processing type, it is determined that the inspection object is in a normal state even if the calculated error score is equal to or greater than a predefined threshold value.
21. The abnormality diagnosis method according to claim 12, further comprising the following steps: collecting a plurality of image data determined to be in an abnormal state as a result of the diagnosis; Extracting feature data of each of the image data; as well as Based on the extracted feature data, the image data is clustered into a plurality of clusters.
22. The abnormality diagnosis method according to claim 21, further comprising the following steps: performing labeling on each of the clusters based on defect type information input by a user; and The labeled clusters are used to retrain the diagnostic model.
23. A factory monitoring system, comprising: an image sensor configured to generate image data relating to at least a portion of an inspection object; as well as An abnormality diagnosis device is configured to receive the image data from the image sensor and diagnose an abnormality in the inspection object using the received image data and a pre-trained artificial intelligence-based diagnosis model.
Citation Information
Cited By
Computer-readable recording medium having stored therein fraud detection program, information processing apparatus, and information processing system
US20260037949A1
