System and method for automated real-time process defect detection using machine learning models

Abstract

To provide an artificial intelligence-based process defect detection method and system that can determine whether work has been performed according to work instructions.SOLUTION: A system may include: a photographing module that collects image data by capturing a process that progresses on an object; a machine learning model that generates work data that is a result of recognizing and reading the object based on the image data; and a detection module that receives instruction data recorded regarding a process for an object optimized for product production, compares the work data with the instruction data, and generates defect information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an automated system and method for real-time process fault detection using a machine learning model. [Background technology]

[0002] The process of manufacturing a single product, which consists of several unit processes, processes materials mechanically, physically, and chemically to change their structure, properties, and external shape to produce finished or intermediate products. As these processes become more complex, the work instructions proposed by engineers during the design process include a significant number of items, which frequently leads to errors and defects that are not linked to actual on-site inspections and manufacturing equipment.

[0003] This has made it even more difficult to reflect the know-how of actual workers, rather than engineers, in work instructions. For example, it may be discovered that a process sequence different from that specified in the work instructions is more efficient based on the know-how of an experienced worker, but in many cases it is not made into a manual due to the practical difficulties in terms of cost and responsibility to immediately apply this to work instructions.

[0004] Therefore, there have been various attempts to improve process efficiency and increase corporate productivity (Patent Document 1, Patent Document 2, etc.), but these only propose process sequences that are theoretically suited using methods such as regression models, and there are limitations to how these can be immediately applied to the workplace.

[0005] Even if such work instructions were created, the only way to determine whether the optimized process that minimized defects was carried out in accordance with the work instructions at the work site was to sample and inspect the work after all the work was completed, or to have workers manually collect internal data from inspection equipment after interrupting the work and then identify the cause of the defect based on their own knowledge.In the end, even if a defect occurred during the process, the cause could only be analyzed after the product was completed, making it difficult to analyze the cause and making it impossible to prevent mass production of defective products.

[0006] This created a need for a system that could derive optimized work instructions in real time without interrupting the work site and ensure that they were followed. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent No. 10-1441230 [Patent Document 2] Republic of Korea Patent Publication No. 10-2003-0068676 Summary of the Invention [Problem to be solved by the invention]

[0008] An object of the present invention is to provide an artificial intelligence-based process defect detection method and system that can determine whether work has been performed in accordance with work instructions.

[0009] However, the technical problems that the present embodiment aims to achieve are not limited to the above-mentioned technical problems, and other technical problems may exist. [Means for solving the problem]

[0010] An artificial intelligence-based process defect detection system according to one embodiment of the present invention may include an imaging module that captures images of processes performed on an object and collects image data; a machine learning model that generates operation data as a result of recognizing and reading the object from the image data; and a detection module that receives instruction data recorded regarding processes for an object optimized for product production, compares the operation data with the instruction data to detect whether the object is normal or defective, and generates defect information if the object is defective.

[0011] The process to be performed consists of unit processes having a series of orders, and the machine learning model recognizes that the object changes as each unit process progresses and generates work data including order information that reads the order in which the processes progress, and the detection module can generate fault information indicating that the order is faulty if the order information of the work data and the instruction data differ.

[0012] The order information of the instruction data may be the order of progress of the unit process that is performed in the shortest time among a plurality of unit processes having different orders.

[0013] The machine learning model recognizes the external shape of the object, which is deformed as the process progresses, and generates work data including status information that reads the work status of the process. The detection module detects the status information of the work data and the instruction data as normal if it determines that they are similar using a preset error standard, and detects the status as poor if they are different, and generates defect information.

[0014] The status information may relate to the degree of assembly and the assembly direction of the object.

[0015] The machine learning model detects the work area where the process of the object is performed and generates work data including location information, and the detection module can search the work data for location information corresponding to the generated defect information.

[0016] The detection module can use the location information to transmit the defect information to an output module located in the work area where the process is performed.

[0017] The machine learning model recognizes that the object changes as the process progresses and generates work data including time information that is the result of reading the time required for the process, and the detection module can generate defect information indicating a time defect if the time information in the work data and the instruction data differ.

[0018] An artificial intelligence-based process defect detection method according to one embodiment of the present invention may include the steps of inputting instruction data, which records an optimization process for a product produced by processing an object, into a detection module; inputting image data generated by a photography module photographing the progress of the object into a machine learning model; generating work data of the progress of the process as a result of the machine learning model recognizing and reading the object from the image data and transmitting the work data to the detection module; and detecting whether the work data is normal or defective by comparing the work data with the instruction data.

[0019] The process to be performed comprises a series of unit processes having a sequence, and further includes a work module in which a work area where the process of the object is performed is located, and the machine learning model detects the work area of ​​the work module and generates work data including position information of the work module, recognizes that the object is changed as each unit process progresses and generates work data including order information that reads the order of the processes, recognizes the outer shape of the object that is deformed as the process progresses and generates work data including status information that reads the work status of the process, and the detection module can compare the work data with the instruction data and generate defect information including the work status, progress order, and work area.

[0020] The machine learning model measures the similarity with the image data based on normal image data that has already been collected for the object, generates work data including status information that reads the work status of the unit process on the image data using a preset similarity standard, and if there is no normal image data that has already been collected for the object, includes a search module that searches for an object associated with the object, measures the similarity with the image data using normal image data that has already been collected for an object associated with the object searched for by the search module, and the detection module detects the work data as normal if it determines that the status information of the work data and the instruction data are similar using a preset error standard, or detects the status as poor if they are different and generates defect information. [Effects of the Invention]

[0021] The present invention has the following advantages.

[0022] First, by automatically monitoring and evaluating the progress of processes, the optimal process sequence can be determined based on the worker's on-site experience, even between unit processes that are not dependent on each other.

[0023] Secondly, it is possible to determine whether the work is being carried out in the optimal order while the process is progressing naturally without the need for a separate simulation.

[0024] Third, since defects are detected by collecting data for each unit process, which is a part of the overall process, it is easy to identify the unit process in which a defect occurred.

[0025] However, the effects of the present invention are not limited to those described above, and effects not mentioned can also be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the accompanying drawings. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a relationship diagram of an artificial intelligence-based process optimization system according to one embodiment of the present invention. [Figure 2] 1 is a flowchart of an artificial intelligence-based process optimization method according to one embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating a process for implementing an artificial intelligence-based process optimization system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] An AI-based process optimization management system according to one embodiment of the present invention provides a process optimization system that can determine an optimal process scenario from among a plurality of process scenarios with different unit process sequences.

[0028] Specifically, in one embodiment of the present invention, an AI-based process optimization management system determines the optimal execution from among multiple executions of each unit process performed in a different order when the overall process for manufacturing a product consists of one or more unit processes that are performed in a sequential order and the unit processes are executed in a different order.

[0029] Data is recorded so that it can be determined whether the ongoing process is being carried out successfully based on the recorded data during the process of deriving the optimal implementation and even after the optimal implementation has been derived, and by reading the data using a processor, it is possible to detect the degree of deviation from the optimal implementation.

[0030] The reading module 100 is capable of collecting and reading image data relating to processes performed on objects provided for the manufacture of a product.

[0031] Here, the object can mean materials and parts used to make a product through a manufacturing process.

[0032] That is, the reading module 100 can read image data of the process currently in progress in accordance with existing instruction data and generate work data.

[0033] The reading module 100 may include a photography module 110 and a machine learning model 120 .

[0034] The imaging module 110 can capture images of the progress of an object and generate image data. The imaging module 110 can include a plurality of cameras installed on the process line, and can transmit the captured image data of the corresponding cameras to the machine learning model 120.

[0035] The camera used by the imaging module 110 can sense the progress of an object through various radio waves such as infrared rays, visible light, X-rays, gamma rays, electromagnetic waves, ultrasound, etc. The image data can be a two-dimensional image, a three-dimensional image, a pre-processed ROI image, a cropped image, etc.

[0036] The machine learning model 120 can recognize and read objects in image data and generate work data for the progress process.

[0037] The machine learning model 120 can recognize that the object changes as each unit process is performed, read the order in which the processes proceed, and generate work data including order information regarding the order in which the processes proceed.

[0038] The machine learning model 120 can recognize the object of a unit process through image data generated by photographing the unit process while the unit process is being performed in series, thereby identifying the type of unit process. If the object changes completely, the unit process is also identified as different, so it can be interpreted that the next unit process was performed after the previous unit process was completed. The machine learning model 120 can grasp the order in which all unit processes are performed using the above method, identify the type of process execution, and generate work data including information on the order in which the corresponding processes are performed.

[0039] Similarly, the machine learning model 120 can recognize the contour of an object that is deformed as the process progresses, read the working state of the process, and generate working data including status information regarding such working state.

[0040] That is, when an object is identified, the machine learning model 120 can read the working status of the unit process based on the degree of deformation of the object's contour recognized through image data before and after the progress of the corresponding unit process. The contour can include not only the shape but also the color, surface, interior, etc.

[0041] For example, if the process is an assembly operation in which a part is the object, the result of the first unit process performed on the first part, which is the object, can be read from the degree to which the outer shape of the part is deformed as a result of the assembly operation. After the machine learning model 120 recognizes the object through image data, it can numerically measure the operation state, such as the assembled direction or assembly strength of the first part, and generate operation data from the recorded state information.

[0042] Alternatively, the machine learning model 120 could generate work data using status information that records a numerical value measuring the degree of similarity between image data about the object and previously collected image data of normal assembly.

[0043] In addition, if there is no normal image data already collected for the object, the machine learning model 120 can use a search module to search for objects associated with the object, and measure the similarity with the image data using normal image data already collected for the object associated with the object searched for by the search module.

[0044] The machine learning model 120 can then detect the work area where the process of the object is performed and generate work data including position information.

[0045] Here, the work area may refer to a location where a specific unit process of a type of work line is carried out, or may refer to a part of an area within a facility where a process is carried out.

[0046] Of course, a GPS module that may be included in the machine learning model 120 can acquire the location and generate operation data that includes location information.

[0047] As a result, defect information that may have been previously generated may be matched with location information and stored as work data, making it possible to search for defect information related to defects that occurred in the area of ​​the corresponding work from the location information.

[0048] In addition, the machine learning model 120 can read the required time for a process by recognizing that the object is changed while the process is being performed, and can measure the difference between the required time information and the time information in the instruction data to determine the difference between the required time for optimal process execution. Thereafter, the detection module 200 can evaluate the difference as the worker's proficiency, excluding environmental factors such as the temperature and humidity of the work area.

[0049] Meanwhile, the reading module 100 can collect sensing data corresponding to quality factors or external factors using the sensing module and read the process of the object based on a preset criterion.

[0050] The sensing module can collect external sensing data through various common sensors such as a microphone, proximity sensor, ultrasonic sensor, gyro sensor, vibration sensor, temperature and humidity sensor, pressure sensor, impact sensor, and gas sensor.

[0051] The quality factors may be the pressure applied to the object, the moving speed of the object, vibration, temperature, humidity, the specific gravity, shrinkage, strength of the object, weather environment, lighting environment, equipment life, equipment information, worker proficiency, material properties, number of cycles, etc.

[0052] The reading module 100 collects sensing data for general process optimization and can use regression analysis or machine learning to determine the relationship between quality variables that are affected by the collected sensing data among the set quality variables.

[0053] When a functional relationship exists between the quality factor and the quality variable, an optimization algorithm can be used to determine the quality factor as an input variable such that the output variable, which is the quality variable, has a desired value, and the input variable value for the optimal implementation can be reflected in the instruction data.

[0054] The detection module 200 compares the operation data transmitted from the reading module 100 with the instruction data to determine whether the process is normal or defective, and if defective, generates defective information.

[0055] Here, the defect information may be classified into types such as defects in the progress order, defects in the work state, and defects in the required time.

[0056] The detection module 200 can compare the work data with the instruction data and compare one or more of the types of defects to generate defect information such as sequence defect, condition defect if the work condition is poor, or time defect if the required time is poor.

[0057] Preferably, for the working conditions, there may be a preset error range or similarity criteria for the working to distinguish between normal and bad cases.

[0058] The detection module 200 compares the status information in the work data with the numerical value of the status information in the instruction data, and if it determines that the value is within a preset level of error range, it detects the value as normal, and if it determines that the value is outside the error range, it generates defective information indicating that the status is poor.

[0059] The detection module 200 can analyze the sensing data for the unit process where the defect occurred, compare it with the quality factor based on the optimal implementation in the instruction data, find the type of quality factor that caused the defect, record it, and store it together with the defect information.

[0060] The output module 300 can output the defect information generated by the detection module 200 .

[0061] When defect information is generated during a process, the output module 300 may receive position information retrieved from the operation data in real time. That is, a signal for defect information may be output to the operation data in the operation area on the position information of the corresponding unit process where the defect occurred.

[0062] Here, the signal may be a visual, auditory, tactile signal, or the like, but is not particularly limited thereto.

[0063] For example, the output module 300 may be installed in a work area identified from the location information of a unit process in which defect information was generated, and may include a display that outputs a signal to visually notify the worker that a defect has occurred, an audio module that issues an audible alarm, and a vibration module that is attached to the worker and outputs vibrations.

[0064] The output module 300 may be provided with an input module as needed, and information regarding the occurrence of atypical defects may be input by an operator.

[0065] In addition, the output module 300 may include a worker terminal that is preset or can be detected by the system as being working in the vicinity of a work area identified from the position information of the unit process in which defect information was generated during the process.

[0066] The output module 300 can output quality factors that are estimated to be the cause of the defect along with the defect information.

[0067] For example, in the case of an assembly process, if the second unit process is performed to weaken the assembly joint strength of parts and the reading module 100 generates work data including status information, which is a numerical value measuring the assembly joint strength, the detection module 200 can compare the work data with the instruction data to detect a defective work status and generate defect information related to the defective work status. The output module 300 can output defect information in the area of ​​the work where the defect occurred, and, if necessary, can also output quality factors estimated to be the cause of the defect.

[0068] On the other hand, the reading module 100 can generate the instruction data.

[0069] As an example, a process for multiple products can be divided into multiple unit processes, and the work data obtained while performing the unit processes multiple times, preferably in different orders, can be analyzed, and the work data of the execution that requires the least amount of time can be determined to be the optimized process order for the product.

[0070] The reading module 100 can transmit such instruction data for optimal application for each product to be stored in a database.

[0071] The method by which the reading module 100 generates the instruction data will now be described in detail.

[0072] The reading module 100 may further include a determining module 130 .

[0073] The determination module 130 can generate instruction data based on the execution data for the optimal execution determined by the determination module from among a plurality of executions performed by changing the order of the unit processes.

[0074] While the unit processes are performed in series, each unit process is evaluated according to the evaluation criteria, and the execution data can be collected by cumulatively evaluating the unit processes in order.

[0075] Here, the above-mentioned work data and execution data may be information generated in the same category, but in the method of generating instruction data, it is referred to as execution data to distinguish it from the data generated by the reading module 100.

[0076] For example, a unit process can refer to one of a multi-step component assembly process to make a product.

[0077] Here, the evaluation criteria are factors that affect the yield and quality of the process, such as the required time and the defect rate, and can be set as needed.

[0078] The reading module 100 can identify the type of unit process through information collected from the unit processes while the unit processes are being performed in series, grasp the order of progress of the unit processes, and identify the type of process execution for each process execution.

[0079] Here, process execution refers to a scenario in which the order of unit processes is changed. Even if the unit processes are the same, different orders result in different process scenarios. In order to determine the optimal process execution from process executions in which the unit processes proceed in different orders, the present invention collects information from the target, identifies the unit processes, understands the order, and maps it to execution data for each process execution.

[0080] For example, while a unit process is being carried out, the required time can be evaluated from the time when the unit process is identified by judging the object to the time when the unit process is completed. At the same time or at a different time, the completion status of the unit process can be grasped and the presence or absence of defects can be evaluated.

[0081] Then, when the next unit process is performed, the required time and the presence or absence of defects can be evaluated in the same way. At the same time or at a different time, the required time and the presence or absence of defects from the start of the entire process to the present can be cumulatively evaluated to collect execution data.

[0082] Here, the completion status of a unit process can be evaluated as completed when the target object is changed, and can also be determined based on data regarding the degree of fastening, fastening direction, separation, etc. of assembled parts.

[0083] More specifically, there may be a list or a list of all processes for a product manufactured using the object, and the reading module 100 inputs this and evaluates it each time a unit process of one object is completed. When all unit processes in the list of all processes have been evaluated, the reading module 100 determines that the entire process or one operation has been completed, and inputs the collected operation data to the determination module 130, thereby ending collection.

[0084] In addition, the search module included in the reading module can search a database in which various types of overall process lists are collected for a list including the object first recognized by the machine learning model and input it into the machine learning model 120.

[0085] Here, the machine learning model 120 can be used to judge the target based on information collected from the unit process, and identify whether the unit process is complete, whether there is a defect, or the type of unit process.

[0086] At this time, the machine learning model 120 continuously learns and improves images of objects using one or more object detection algorithms from the installed application, such as vision fitting, edge, color, and location, and can identify the type of unit process, recognize the assembly or missing state of the object, and determine whether the corresponding unit process is defective or complete.

[0087] Specifically, the machine learning model 120 learns and improves images that represent the shape, color, etc. of objects using deep learning or machine learning, which are algorithmic artificial intelligence technologies that classify or learn the characteristics of the corresponding image data by themselves, and can classify and detect objects and identify the type of unit process.

[0088] The machine learning model 120 can process data using a processor and can preprocess image data of an object to generate training data when detecting the object.

[0089] Here, the learning data may be labeled with information on the assembly state, defect state, and unit process type for the target object along with the preprocessed image data.

[0090] Information about the object to be labeled may be collected through feedback from the worker. That is, if many defective cases of the object are not found initially, the part information of the object may be input using an input interface to label the corresponding image data, or whether assembly is complete or defective may be input to label the object.

[0091] The labeled data can be used to train a classifier model and applied to the machine learning model 120.

[0092] In addition, image data collected through a camera while a unit process is being performed can be preprocessed and then a classifier model can be used to detect whether an object is defective, whether assembly is complete, etc.

[0093] Meanwhile, in one embodiment of the present invention, normal image data collected when a unit process on an object is normally performed without any defects may be collected in advance.

[0094] The machine learning model 120 measures the similarity between the normal image data and newly collected image data based on the normal image data, reads the image data as normal image data using a preset similarity standard, and collects new normal images as trial data.

[0095] If necessary, features can be extracted from the collected normal image data, and the machine learning model 120 can learn only the normal image data.

[0096] At this time, if a product is improved or newly released, some or all of the parts and processes used in the product may be different from those in existing products, and therefore, image data may not already be collected for the object.

[0097] The reading module 100 can collect normal image data for the improved new product process by having the machine learning model 120 measure the similarity between image data of an object of the improved product and normal image data collected for a previous version of the product, in order to immediately determine the improved new product process without collecting image data.

[0098] In addition to the above, the machine learning model 120 may utilize general deep learning vision or machine vision (MV) and may include hardware, software, or interfaces used in industrial factory automation processes for surface defect inspection of wafers, display products, PCB defect inspection, LED chip packages, and other products.

[0099] The process of collecting trial data by the reading module 100 can be repeated until all unit processes have been completed.

[0100] The determination module 130 determines the optimal process implementation from among a plurality of process implementations based on the collected implementation data.

[0101] Specifically, when the reading module 100 collects the implementation data for the process implementation identified by the reading module 100 and the implementation data for each of the multiple process implementations, the judgment module 130 can use this to determine whether the identified process implementation is the optimal process implementation.

[0102] According to one embodiment of the present invention, any one of the optimal process executions may be the case where the sum of the total unit process durations is the smallest.

[0103] That is, using the machine learning model 120 of the reading module 100, execution data measured against the evaluation criteria, which are the required time and defect rate of each unit process, can be collected each time each unit process is performed, and if the judgment module 130 determines based on the execution data that the total required time of the unit processes for any process execution is the minimum time, it can judge that the corresponding process execution is the optimal process execution.

[0104] In addition, the machine learning model 120 can be used to check the work status of each process and calculate the defect rate, which can be reflected in the evaluation.

[0105] Meanwhile, a big data database for process execution can be constructed by labeling and storing the execution data collected from the reading module 100 with each process execution. The big data database can be constructed by managing execution data including the required time for process execution, defect rate, etc.

[0106] In the big data database, information on the worker in charge may be labeled with the execution data while the process is being performed and stored.

[0107] The determination module 130 can then generate instruction data based on execution data for the optimal process execution. Specifically, an existing work instruction format can be obtained from an MES (Manufacturing Execution System) system, and the optimal process execution can be merged into the format to generate instruction data in the form of a new work instruction. The new work instruction can be output via the output module 300, such as displaying the work table where the corresponding process is performed.

[0108] If the data collected reflects information about the workers involved in the process execution, it will be possible to reduce errors in the time required for the process execution and the defect rate due to differences in the worker's skill level, and increase the reproducibility of work instructions that reflect optimal process execution.

[0109] In other words, when information regarding the worker's proficiency level is reflected in the execution data, the judgment module 130 can distinguish between execution data of processes performed by skilled workers and execution data of processes performed by unskilled workers and determine the optimal process execution for each level of proficiency.

[0110] Preferably, by managing the execution data differently for each individual worker, it would be possible to propose the optimum process execution for each worker.

[0111] FIG. 3 illustrates a process for implementing an artificial intelligence-based process optimization management system according to an embodiment of the present invention.

[0112] An embodiment of an artificial intelligence-based process optimization system will now be described with reference to FIG.

[0113] FIG. 3 shows the process in which image data collected by the imaging module 110 during the process of assembling part B to part A is read and execution data is generated and collected by the machine learning model 120 of the reading module 100.

[0114] As shown in FIG. 3, in the above-described artificial intelligence-based process optimization management system, the machine learning model 120 first detects the target part A and identifies the unit process type A, and the reading module 100 can search for the entire list of product parts (A, B, C, D) that include the target of the identified unit process A.

[0115] Then, it is confirmed whether the unit process A identified through the machine learning model 120 has been completed.

[0116] The reading module 100 can collect execution data of the unit process A by evaluating the time required to complete the specified unit process A and the presence or absence of defects in real time.

[0117] When the next unit process B is performed, it can be detected and recognized that the object has been changed through the machine learning model 120. That is, the machine learning model 120 can determine whether the object has been changed by grasping through the image that a new part B has been added to the object and that the shape, color, etc. have been changed.

[0118] Similarly, the reading module 100 can collect execution data by evaluating in real time whether or not the unit process B specified by the changed object is completed, the required time, and whether or not there is a defect.

[0119] Furthermore, the reading module 100 can record and store the fact that the unit process A is performed first and then the unit process B is performed second.

[0120] When the unit process B is completed, the reading module 100 can accumulate and evaluate the required time from the beginning to the present, the presence or absence of defects, etc. in real time, and collect the execution data.

[0121] The above process can be carried out by a series of unit processes until all the parts of the product (A, B, C, D) are assembled.

[0122] Here, the execution of multiple processes may be a unit process permutation. If the entire parts list is A, B, C, D, there will be 4! types of process executions, such as ABCD, ABDC, ACBD, ACDB, ADBC, ADCB, BACD, BADC, ..., DCBA.

[0123] Each process execution may be evaluated through the above process and execution data may be collected. When a certain execution is performed repeatedly, the required time or the presence or absence of defects for the corresponding process execution may be accumulated and stored.

[0124] Of course, not all process implementations need to be evaluated; some types of processes may not be evaluated due to inability to proceed or operator preference.

[0125] Once sufficient implementation data has been accumulated by the machine learning model 120, the determination module 130 can determine the optimal process implementation from among the process implementations evaluated based on the collected implementation data.

[0126] For example, as shown in Table 1 below, if the ABCD process is performed 30 times with an average required time of 1 minute 20 seconds, an average error of 10 seconds, and a defect rate of 5%, and the BCAD process is performed 25 times with an average required time of 1 minute 40 seconds, an average error of 30 seconds, and a defect rate of 7%, and if the preset condition prioritizes the shortest required time, the judgment module 130 can determine that ABCD, which has the shortest required time, is the optimal process execution.

[0127] [Table 1]

[0128] The AI-based process optimization system according to an embodiment of the present invention automatically monitors and evaluates the progress of processes, and can determine the optimal process order based on the field experience of workers even between unit processes that are not dependent on each other. It can also determine the optimal process execution while the process progresses naturally without a separate simulation.

[0129] FIG. 2 shows a flowchart of an artificial intelligence-based process optimization management method according to one embodiment of the present invention.

[0130] An AI-based process optimization management method according to one embodiment of the present invention may include the steps of: a reading module receiving instruction data recorded regarding an optimal execution determined from a plurality of executions performed by changing the order of unit processes; the reading module generating image data regarding a process to be performed on an object and generating work data resulting from reading the image data so as to correspond to the instruction data; a detection module receiving the work data and comparing it with the instruction data to generate defect information regarding the process; and an output module receiving and outputting the defect information.

[0131] Throughout this specification, the terms machine learning model, deep learning based model, computational model, neural network, network function, deep neural network, and neural network may be used interchangeably.

[0132] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. A deep neural network can be used to understand the latent structures of data. That is, it can understand the latent structures of photos, text, videos, audio, and music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.). Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), etc. The above descriptions of deep neural networks are merely examples, and the present disclosure is not limited thereto.

[0133] In one embodiment of the present disclosure, the network function may also include an autoencoder. An autoencoder may be a type of artificial neural network that outputs output data similar to input data. An autoencoder may include at least one hidden layer, and an odd number of hidden layers may be arranged between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then symmetrically expanded from the bottleneck layer to the output layer (symmetric to the input layer). An autoencoder may also perform nonlinear dimensionality reduction. The number of input and output layers may correspond to the dimensionality after preprocessing of the input data. The number of hidden layer nodes included in the encoder in the autoencoder structure may have a structure in which the number of nodes decreases as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (e.g., more than half of the number of nodes in the input layer).

[0134] A neural network can be trained by at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training a neural network can be a process of applying knowledge to the neural network to make it perform a particular operation.

[0135] Neural networks can be trained to minimize output errors. Training a neural network involves repeatedly inputting training data into the neural network, calculating the error between the neural network's output and the target for the training data, and backpropagating the neural network's error from the output layer to the input layer of the neural network in a direction that reduces the error, thereby updating the weights of each node of the neural network. In supervised learning, each piece of training data is labeled with a correct answer (i.e., labeled training data). In unsupervised learning, each piece of training data may not be labeled with a correct answer. For example, in supervised learning for data classification, the training data may be data in which each piece of training data is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the neural network's output (category) with the label of the training data. As another example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weights of each node in each layer of the neural network can be updated through backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network learning to increase efficiency by enabling the neural network to quickly achieve a certain level of performance, and a lower learning rate can be used in the later stages of learning to increase accuracy.

[0136] In neural network training, training data may generally be a subset of actual data (i.e., data to be processed using the trained neural network). Therefore, there may be a learning cycle in which errors in the training data decrease but errors increase for actual data. Overfitting is a phenomenon in which excessive learning on training data increases errors for actual data. For example, a neural network that has learned cats by showing yellow cats may be unable to recognize cats that are not yellow as cats. Overfitting can cause an increase in errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. To prevent overfitting, methods such as increasing the amount of training data, regularization, dropout (which deactivates some nodes in the network during the training process), and batch normalization layers can be used.

[0137] Although the present disclosure has been described generally as being implemented by computing devices, those skilled in the art will understand that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that may be executed on one or more computers and / or in a combination of hardware and software.

[0138] Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Those skilled in the art will also appreciate that the disclosed methods can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can operate in conjunction with one or more associated devices.

[0139] The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0140] A computer typically includes a variety of computer-readable media. Any medium accessible by a computer can be computer-readable, including volatile and nonvolatile media, transitory and non-transitory media, and removable and non-removable media. By way of example and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information accessible by a computer.

[0141] Various embodiments presented herein may be embodied as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0142] It should be understood that the specific order or hierarchy of steps in the processes presented is an example of a sample approach. Based on design priorities, the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

[0143] The description of the presented embodiments is provided to enable any person skilled in the art to which the present disclosure pertains to make or practice the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. [Explanation of symbols]

[0144] 100 Read Module 110 Photography Module 120 Machine Learning Models 130 Judgment Module 200 Detection Module 300 Output Module

Claims

1. An artificial intelligence-based process defect detection system comprising: a reading module including a photographing module, a machine learning model, and a judgment module; a detection module; and an output module, an imaging module for capturing images of the process performed on the object and collecting image data; a machine learning model that generates task data that is a result of recognizing and reading the object in the image data; and a detection module that receives instruction data recorded for a process for an object optimized for product production, compares the operation data with the instruction data to detect whether the operation is normal or defective, and generates defect information if the operation is defective; The instruction data is The overall process for manufacturing a product is comprised of one or more unit processes that are performed in a series of steps, and the unit processes are performed in a different order. During each step, a reading module evaluates each unit process according to an evaluation standard, and accumulates and evaluates the unit processes in order, generating execution data that is collected and transmitted to a determination module. The determination module determines the optimal execution from among multiple executions performed by changing the order of the unit processes based on the execution data of the reading module. The optimal execution is determined by the determination module to be an execution in which the total required time or defect rate of the unit process recorded in the execution data is minimum, In addition, the process performed on the object is made up of unit processes with a series of orders, Further, the apparatus includes a work module in which a work area where a process for the object is performed is located, The machine learning model detecting a working area of ​​the working module and generating work data including position information of the working module; Recognizing that the object changes as each unit process progresses, work data is generated that includes sequence information that reads the sequence of the processes; generating work data including status information obtained by recognizing the outer shape of the object that is deformed as the process progresses and reading the work status of the process; The detection module includes: comparing the work data with the instruction data to generate defect information including the work status, progress order, and work area; Furthermore, the machine learning model measuring a similarity with the image data based on normal image data already collected for the object, and generating work data including status information obtained by reading the work status of the unit process on the image data using a preset similarity standard; a search module for searching for an object associated with the object if there is no previously collected normal image data for the object; Measure the similarity between the image data and the object searched by the search module using normal image data that has already been collected for the object associated with the object searched by the search module; The detection module includes: If the status information of the work data and the instruction data is judged to be similar using a preset error standard, it is detected as normal, and if they are different, it is detected as a status failure and failure information is generated. An artificial intelligence-based process defect detection system.

2. inputting image data generated by the photographing module photographing the progress of the object into a machine learning model; generating operation data of a progress process as a result of the machine learning model recognizing and reading the object in the image data, and transmitting the operation data to a detection module; and The detection module includes a step of comparing instruction data in which an optimization process for a product to be produced by a process on an object is recorded with the work data to detect whether the product is normal or defective, The instruction data is The overall process for manufacturing a product is comprised of one or more unit processes that are performed in a series of steps, and the unit processes are performed in a different order. During each step, a reading module evaluates each unit process according to an evaluation standard, and accumulates and evaluates the unit processes in order, generating execution data that is collected and transmitted to a determination module. The determination module determines the optimal execution from among multiple executions performed by changing the order of the unit processes based on the execution data of the reading module. The optimal execution is determined by the determination module to be an execution in which the total required time or defect rate of the unit process recorded in the execution data is minimum, In addition, the process performed on the object is made up of unit processes with a series of orders, Further, the apparatus includes a work module in which a work area where a process for the object is performed is located, The machine learning model detecting a working area of ​​the working module and generating work data including position information of the working module; Recognizing that the object changes as each unit process progresses, work data is generated that includes sequence information that reads the sequence of the processes; generating work data including status information obtained by recognizing the outer shape of the object that is deformed as the process progresses and reading the work status of the process; The detection module includes: comparing the work data with the instruction data to generate defect information including the work status, progress order, and work area; Furthermore, the machine learning model measuring a similarity with the image data based on normal image data already collected for the object, and generating work data including status information obtained by reading the work status of the unit process on the image data using a preset similarity standard; a search module for searching for an object associated with the object if there is no previously collected normal image data for the object; Measure the similarity between the image data and the object searched by the search module using normal image data that has already been collected for the object associated with the object searched by the search module; The detection module includes: If the status information of the work data and the instruction data is judged to be similar using a preset error standard, it is detected as normal, and if they are different, it is detected as a status failure and failure information is generated. An artificial intelligence-based process defect detection method.

Images