Method and device for detecting welding defect and performing rework on basis of artificial intelligence model

Abstract

A method for detecting a bonding process defect and performing rework on the basis of artificial intelligence may comprise the steps of: collecting measurement / image data; generating defect detection data from the measurement / image data by using a trained artificial intelligence model; determining whether to perform rework, on the basis of the measurement / image data and the defect detection data; and updating the artificial intelligence model on the basis of the measurement / image data and the defect detection data.

Description

Method and device for detecting welding defects and performing rework based on an artificial intelligence model

[0001] The present invention relates to a method and device for detecting defects in a joining process between parts made of metal materials.

[0002]

[0003] Joining processes for metal materials can broadly be categorized into brazing, which involves welding that melts metals to join them; riveting, which involves physically and mechanically connecting and joining; and adhesive-based joining. Because each joining process involves a different bonding process, it is necessary to consider each individual process individually to detect defects. Currently, defects in metal joining processes can only be detected based on the finished product after the bonding process is complete. This method relies on the experience and skill of the inspector, limiting its effectiveness.

[0004] Considering the above, the following describes a method for improving defect detection accuracy by performing defect inspection in real time and performing a defect detection process based on an artificial intelligence (AI) algorithm.

[0005]

[0006] The present invention can provide a method for detecting defects that are difficult to identify with the naked eye based on a specificity of a bonding process.

[0007] The present invention can provide a method for detecting defects occurring at a joint portion of a jointing process and performing an appropriate rework process based on the defects.

[0008] The present invention can provide a method for reducing defect rates and process delay time based on a bonding process performed in real time.

[0009] The present invention can provide a method for detecting defects in a bonding process based on a bonding process method, bonding process components, bonding process location information, and detailed location information for each component.

[0010]

[0011] According to one embodiment, a method for detecting and performing re-construction of a bonding process defect based on artificial intelligence may include a step of collecting measurement / image data, a step of generating defect detection data from the measurement / image data using a learned artificial intelligence model, a step of determining whether to perform re-construction based on the measurement / image data and the defect detection data, and a step of updating the artificial intelligence model based on the measurement / image data and the defect detection data.

[0012] In addition, according to one embodiment, a device for detecting defects in a bonding process and performing re-construction based on artificial intelligence may include a defect inspection unit for acquiring measurement / image data, a defect detection unit for generating defect detection data based on the measurement / image data, a re-construction process judgment unit for determining whether a re-construction process is necessary based on the defect detection data, a re-construction process system for instructing re-construction based on whether a re-construction process is necessary, an artificial intelligence learning unit for updating a learning model based on the defect detection data, and a defect performance verification unit for generating weight data based on the updated learning model and transmitting the weight data to the defect detection unit.

[0013] Additionally, according to one embodiment, the measurement / image data and defect detection data may be generated based on at least one of a bonding process method, a bonding process component, a bonding process location, and a location by subcomponent.

[0014] Additionally, according to one embodiment, measurement / image data and defect detection data may be acquired for each bonding process at the first location where the bonding process of the first component is performed.

[0015] Additionally, according to one embodiment, measurement / image data and defect detection data may be collected at a first location where the bonding process of the first component is performed once all bonding processes are completed.

[0016] Additionally, according to one embodiment, the measurement / image data and defect detection data may be collected when all bonding processes are completed at a first location where the bonding process of the first part is performed and the first part is moved to a second location where a defect inspection is performed.

[0017] Additionally, according to one embodiment, measurement / image data and defect detection data may be generated based on information collected by a vision sensor at the bonding site where the bonding process is in progress.

[0018] In addition, according to one embodiment, the measurement / image data is acquired based on a vision sensor, and the defect detection data is determined to be defective if it corresponds to a preset defect type based on the measurement / image data, thereby generating the defect detection data, wherein the defect type may include at least one of a free hole cover defect type, a position defect type, and a head melting type.

[0019] Additionally, according to one embodiment, the measurement / image data and defect detection data may be generated based on information collected based on at least one of a lidar sensor and a laser sensor that measures depth at a bonding site where a bonding process is in progress.

[0020] In addition, according to one embodiment, the measurement / image data is acquired based on at least one of a lidar sensor and a laser sensor, and the defect detection data is determined to be defective if it corresponds to a preset defect type based on the measurement / image data, thereby generating the defect detection data, wherein the defect type may include at least one of a non-sticking defect type and an ear-fastening defect type.

[0021]

[0022] According to the present disclosure, there is an effect of providing a method for detecting defects that are difficult to identify with the naked eye based on a singularity of a bonding process.

[0023] According to the present disclosure, there is provided a method for detecting defects occurring at a joint portion of a jointing process and performing an appropriate rework process based thereon.

[0024] According to the present disclosure, there is provided an effect of providing a method for reducing defect rates and process delay time based on a bonding process performed in real time.

[0025] According to the present disclosure, there is an effect of providing a method for detecting defects in a bonding process based on bonding process method, bonding process components, and bonding process position information.

[0026] The present disclosure is not limited to the effects described above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0027]

[0028] FIG. 1 illustrates a system according to one embodiment of the present disclosure.

[0029] FIG. 2 illustrates the structure of a device according to one embodiment of the present disclosure.

[0030] Figure 3 illustrates an example of a perceptron that constitutes an artificial intelligence model applicable to the present disclosure.

[0031] Figure 4 illustrates an example of an artificial neural network that constitutes an artificial intelligence model applicable to the present disclosure.

[0032] FIG. 5 is a diagram illustrating an artificial intelligence-based defect detection and re-construction procedure according to one embodiment of the present disclosure.

[0033] FIG. 6 illustrates an example of a welding defect that can be identified through defect detection according to one embodiment of the present disclosure.

[0034] FIG. 7 illustrates an example of a bead appearance defect according to one embodiment of the present disclosure.

[0035] FIG. 8 illustrates an example of an artificial intelligence system capable of recognizing a specific object based on image data according to one embodiment of the present disclosure.

[0036] FIG. 9 illustrates an example of a system for detecting welding defects and performing rework according to one embodiment of the present disclosure.

[0037] FIG. 10 illustrates an example of a welding defect detection and rework performance procedure according to one embodiment of the present disclosure.

[0038] Figure 11 is a drawing showing a defect type applicable to the present disclosure.

[0039] Fig. 12 is a diagram showing a defect detection system applicable to the present disclosure.

[0040] Figure 13 is a flowchart showing a defect detection method applicable to the present disclosure.

[0041]

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0043] In describing embodiments of the present disclosure, detailed descriptions of known configurations or functions will be omitted if they are deemed to obscure the gist of the present disclosure. Furthermore, portions unrelated to the description of the present disclosure in the drawings have been omitted, and similar portions have been designated with similar reference numerals.

[0044] The present disclosure provides a method, device, and system for detecting defects in a metal plate (or metal material) joining process and performing rework based on an artificial intelligence model. Specifically, the present disclosure can provide a method, device, and system for detecting defects in welding, bonding, riveting, and other joining processes used for joining metal parts in vessels, automobiles, and other such applications.

[0045] The joining process for metal plates (or metal materials) may include, but is not limited to, brazing technology, which includes welding that melts metal to join it, riveting technology that physically / mechanically connects / joins it, and joining technology using adhesives.

[0046] Brazing is a metal joining technique that involves melting a filler metal without melting the base metal at temperatures above 450 degrees Celsius. Compared to welding, brazing achieves joining at relatively low temperatures. Brazing encompasses a variety of methods, including high-frequency, resistance, and furnace brazing, and is not limited to a specific form. The main advantages of brazing include high bond strength, resistance to deformation, and the ability to join a variety of materials. Brazing joins metals without melting the base metal, utilizing the melting point of the filler metal. The filler metal can penetrate and diffuse through wetting and capillary action.

[0047] Riveting is a metal joining method that uses a fastener called a rivet to mechanically connect two materials. Riveting is primarily used in applications requiring structural strength, such as automotive chassis. Compared to welding, which involves brazing, riveting uses mechanical rather than thermal means to achieve a joint. This makes it suitable for materials that cannot be exposed to high temperatures or where thermal distortion caused by welding must be minimized. Riveting can also be used in field applications in locations difficult to access by welding equipment. Riveting is characterized by fast operation speeds, consistent joint quality, and ease of use without requiring specialized skills. Furthermore, it can be reused when the joined structure needs to be moved or disassembled.

[0048] Furthermore, adhesive-based metal joining can be a different approach than mechanical joining. Adhesives may not require drilling or separate holes in the metal surface. Adhesives evenly distribute stress across the bonding surface, allowing for the use of thinner, lighter materials. Adhesives can also bond dissimilar materials and simultaneously seal the joint, preventing fluid penetration. Furthermore, adhesives can prevent corrosion on metal surfaces and maintain a clean, smooth appearance. Furthermore, adhesives can also dampen vibrations, allowing energy to dissipate rather than be transmitted, increasing their utility.

[0049] Brazing, riveting, and adhesive metal joining each involve distinct joining processes. Therefore, measures to improve the reliability of each process may be necessary, taking into account their unique characteristics. For example, reliability of the joining process can be analyzed using artificial intelligence (AI) algorithms, utilizing images or video data captured via cameras and sensors during the process. This analysis can automatically detect defects in the joint area, enabling quick and accurate identification of potential defects during the process.

[0050] Below, a detection method based on 'visual inspection' is described using a neural network and an open data set suitable for learning as a method for detecting defects in the joining process of metal plates (or metal materials).

[0051] Specifically, an AI-based defect detection device (hereinafter, “defect detection device”) for a metal plate joining process can recognize various defects that may occur and then identify the type of defect through a pre-trained AI neural network. In addition, the defect detection device can determine a method for reworking based on the type of defect. Here, the joining process may include an operation that takes into account at least one of a joining process method, a component to be joined, a position of the joining process, and a detailed position of the component. The defect detection device can collect and process data necessary for defect detection and apply it to a defect inspection step. The following describes a method for the defect detection device to determine a rework process method based on the collected and processed data and the defect inspection results.

[0052] FIG. 1 illustrates a system according to one embodiment of the present disclosure.

[0053] Referring to FIG. 1, the system includes a service server (110), a data server (120), and at least one client device (130).

[0054] The service server (110) provides a service based on an artificial intelligence model. That is, the service server (110) performs learning and prediction operations using the artificial intelligence model. The service server (110) can communicate with a data server (120) or at least one client device (130) via a network. For example, the service server (110) can receive learning data for training an artificial intelligence model from the data server (120) and perform training. The service server (110) can receive data necessary for learning and prediction operations from at least one client device (130). In addition, the service server (110) can transmit information about a prediction result to at least one client device (130).

[0055] The data server (120) provides training data for the training of the artificial intelligence model stored in the service server (110). The data server (120) may provide publicly accessible data or data requiring permission. The training data may be preprocessed by the data server (120) or the service server (120). As another example, the data server (120) may be omitted. In this case, the service server (110) may use an externally trained artificial intelligence model, or the training data may be provided offline to the service server (110).

[0056] At least one client device (130) transmits and receives data related to an artificial intelligence model operated by the service server (110) to and from the service server (110). The at least one client device (130) is a device used by a user, and may transmit information input by the user to the service server (110) and store or provide (e.g., display) information received from the service server (110) to the user. In some cases, a prediction operation may be performed based on data transmitted from one client, and information related to the result of the prediction may be provided to another client. The at least one client device (130) may be various types of computing devices, such as a desktop computer, a laptop computer, a smartphone, a tablet, or a wearable device.

[0057] Although not illustrated in FIG. 1, the system may further include a management device for managing the service server (110). The management device is a device used by the entity managing the service, and monitors the status of the service server (110) or controls the settings of the service server (110). The management device may be connected to the service server (110) via a network or directly connected via a cable connection. Under the control of the management device, the service server (110) may set parameters for operation.

[0058] As described with reference to FIG. 1, a service server (110), a data server (120), at least one client device (130), a management device, and the like can be connected and interact with each other through a network. Here, the network may include at least one of a wired network and a wireless network, and may be formed by one or a combination of two or more of a cellular network, a local area network, and a wide area network. For example, the network may be implemented based on at least one of a local area network (LAN), a wireless LAN (WLAN), Bluetooth, long term evolution (LTE), LTE-A (LTE-advanced), and 5G (5th generation).

[0059] In addition, the service server (110), data server (120), at least one client device (130), management device, etc. described with reference to FIG. 1 are implemented within the client device (130) and can interact within the one client device (130).

[0060] FIG. 2 illustrates the structure of a device according to one embodiment of the present disclosure. The structure illustrated in FIG. 2 may be understood as the structure of the service server (110), the data server (120), and at least one client device (130) of FIG. 1.

[0061] Referring to FIG. 2, the device includes a communication unit (210), a storage unit (220), and a control unit (230). The communication unit (210) performs a function for connecting to a network and communicating with other devices. The communication unit (210) may support at least one of wired communication and wireless communication. For communication, the communication unit (210) may include at least one of an RF (radio frequency) processing circuit and a digital data processing circuit. In some cases, the communication unit (210) may be understood as a component including a terminal for connecting a cable. Since the communication unit (210) is a component for transmitting and receiving data and signals, it may be referred to as a 'transceiver'.

[0062] The storage unit (220) stores data, programs, microcodes, instruction sets, applications, etc. required for the operation of the device. The storage unit (220) may be implemented as a temporary or non-transitory storage medium. In addition, the storage unit (220) may be implemented in a fixed or detachable form in the device. For example, the storage unit (220) may be implemented as at least one of NAND flash memory such as a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD card, and a magnetic computer memory device such as a hard disk drive (HDD).

[0063] The control unit (230) controls the overall operation of the device. To this end, the control unit (230) may include at least one processor, at least one microprocessor, etc. The control unit (230) may execute a program stored in the storage unit (220) and connect to a network via the communication unit (210). In particular, the control unit (230) may perform algorithms according to various embodiments described below and control the device to operate according to the embodiments described below.

[0064] Based on the structure described with reference to Figures 1 and 2, services based on artificial intelligence algorithms according to various embodiments of the present disclosure can be provided. Here, an artificial intelligence model comprised of an artificial neural network can be used to implement the artificial intelligence algorithm. The concept of the perceptron, a component of the artificial neural network, and the artificial neural network itself are as follows.

[0065] A perceptron is a model of a biological nerve cell, and has a structure that takes multiple signals as inputs and outputs a single signal. Fig. 3 illustrates an example of a perceptron that constitutes an artificial intelligence model applicable to the present disclosure. Referring to Fig. 3, the perceptron multiplies each of input values ​​(e.g., _, _, _쪋, _) by weights (302-1 to 302-n) (e.g., _, _, _쪋, _), and then adds the weighted input values ​​using a transfer function (304). During the adding process, a bias value (e.g., bk) may be added. The perceptron generates an output value (e.g., oj) by applying an activation function (406) to the net input value (e.g., netj), which is the output of the transfer function (304). In some cases, the activation function (406) is a threshold (e.g. ) can operate based on. The activation function can be defined in various ways. Although the present disclosure is not limited thereto, for example, a step function, sigmoid, Relu, Tanh, etc. can be used as the activation function.

[0066] An artificial neural network can be designed by listing perceptrons like those in Fig. 3 and forming layers. Fig. 4 illustrates an example of an artificial neural network that constitutes an artificial intelligence model applicable to the present disclosure. In Fig. 4, each node represented by a circle can be understood as a perceptron of Fig. 3. Referring to Fig. 4, the artificial neural network includes an input layer (402), multiple hidden layers (404a, 404b), and an output layer (406).

[0067] When performing prediction, when input data is provided to each node of the input layer (402), the input data is forward propagated to the output layer (406) through weight application, transformation function calculation, and activation function calculation by the perceptrons forming the input layer (402) and hidden layers (404a, 404b). Conversely, when performing training, errors are calculated through backward propagation from the output layer (406) toward the input layer (402), and the weight values ​​defined for each perceptron can be updated according to the calculated errors.

[0068] FIG. 5 is a diagram illustrating a defect detection and rework procedure for a metal plate bonding process based on artificial intelligence according to one embodiment of the present disclosure. As an example, FIG. 5 illustrates an operating method of a device with computing capabilities (e.g., the service server (110) of FIG. 1 ), but the present disclosure may not be limited thereto.

[0069] Referring to FIG. 5, in step S501, the defect detection device can collect measurement / image data based on at least one of a bonding process method, a bonding process component, a bonding process location, and a detailed component location. The defect detection device can receive information on at least one of a bonding process method, a bonding process component, a bonding process location, and a detailed component location from a data server (e.g., the data server (120) of FIG. 1) or a client device (e.g., the client device (130) of FIG. 1), and acquire measurement / image data based thereon.

[0070] In step S503, the defect detection device may generate defect detection data using at least one of the learned artificial intelligence model and preset information. The defect detection data may be data related to defects occurring during the bonding process based on the defect type. The defect detection data may include information regarding whether a defect was detected, the defect type, and the location of the defect. As another example, if the defect detection device determines that there is no defect based on the measurement / image data, it may generate defect detection data indicating the absence of a defect.

[0071] The AI ​​model can be a pre-trained model and updated based on measurement / image data. The training data can be label data that includes at least one of the following: data indicating the type and location of a bonding process defect, or data indicating the absence of a defect, for the measurement / image data. If the performance of the retrained AI model exceeds a threshold, the defect detection data can be updated by generating new defect detection data using the retrained AI model.

[0072] In step S505, the defect detection device can determine whether rework is necessary based on the defect detection data. Furthermore, the defect detection device can determine whether rework is necessary based on rework determination data that includes information necessary for determining rework. For example, the rework determination data may include information on the type of rework required for each defect type, the location of the bonding process defect, the voltage value at the time of the bonding process defect, and the current value at the time of the bonding process defect.

[0073] In step S507, if it is determined that re-construction is required in step S505, the defect detection device may generate re-construction process data in step S509 and perform re-construction based on the re-construction process data. For example, the re-construction process data may include information on whether re-construction is required, the type of re-construction, and whether re-construction is possible on the current system. If it is determined that re-construction is not required in step S505, the device may not perform re-construction.

[0074] FIG. 6 illustrates an example of a welding defect that can be identified through a welding defect inspection according to one embodiment of the present disclosure.

[0075] Referring to Figure 6, we can see various types of welding defects. Undercut is a defect that occurs when the base metal melts along the weld edge, making the surface concave and not filling it with the weld metal. Welding is a method of joining structural members by melting the joint part with high heat, etc. and compressing it. When an undercut occurs, the welding surface becomes uneven, stress is concentrated in a specific area, reducing the strength of the weld part, and there is a problem that slag may be mixed into the undercut area. Stress is a resistance force that occurs in a material in response to the size of a load (e.g., an external force) such as compression, tension, bending, or torsion applied to the material.

[0076] Porosity (PIT) is a hole-like defect formed by gas trapped during the solidification of the weld metal. Porosity is typically caused by foreign matter such as rust, oil, or paint in the weld joint, or by moisture in the electrode coating, flux, or shielding gas. Porosity can also be caused by extremely high welding currents, arc lengths, and arc voltages.

[0077] A crack is a thread-like defect that occurs after the weld metal has cooled. Depending on the location and direction of occurrence, cracks can be classified as follows:

[0078] - Toe crack: Crack that occurs at the boundary between the bead surface and the parent material.

[0079] - Transverse crack: A crack that occurs parallel to the weld line.

[0080] - Longitudinal crack: Crack that occurs in a direction perpendicular to the weld line

[0081] - Crack under bead: Crack that occurs in the heat affected zone near the melting line of the parent material.

[0082] Subbead cracking commonly occurs when welding high-carbon or low-alloy steels. Furthermore, subbead cracking can be caused by hardening and transformation stress in the heat-affected zone due to welding heat, diffusible hydrogen in the weld metal, or excessive hydrogen in the arc atmosphere. In addition to the types of cracking mentioned above, there are various other types of cracking, including root cracking.

[0083] Another example of a welding defect is a bead appearance defect with an irregular shape in bead width and band shape.

[0084] Figure 7 illustrates an example of a bead appearance defect according to one embodiment of the present disclosure. Referring to Figure 7, a narrow bead, which is a region where the bead width is narrowed, can be identified. In addition to the welding defects described above, overlaps, insufficient penetration, slag inclusions, and the like can also be detected through defect inspection according to one embodiment of the present disclosure.

[0085] Below, a defect detection method according to one embodiment of the present disclosure is described.

[0086] For example, active thermography is a defect detection method. Active thermography is a method for identifying defect locations by observing the temperature difference between a defective location and a healthy area using a thermal imaging camera. When artificial heat is applied to a specimen or structure at a set period of time and cycle, a difference occurs between the temperature of a defect, such as a crack or delamination, and the temperature of a healthy area. Based on this, active thermography is a method for detecting defects by utilizing the fact that thermal waves incident on the surface of the specimen or structure to be inspected propagate through the structure and are reflected at the location of the defect.

[0087] An artificial intelligence model that can be used according to one embodiment of the present disclosure is described in detail.

[0088] Existing DNNs (Deep Neural Networks) are structurally incapable of considering the characteristics (features) of image data (e.g., correlations between data according to their positions, features such as the appearance of a specific object composed of multiple data, etc.). Therefore, research has been conducted to train artificial intelligence models by considering the characteristics of image data. According to the present disclosure, a method can be applied to extract the characteristics of image data using a convolution operation, and to classify the image data according to predefined classification criteria for the corresponding image data through a network (e.g., FCNN (Full-Connected Neural Network)) based on the extracted characteristics. Here, each component of the filter applied to the convolution operation can correspond to a weight (weight: w) of general deep learning.

[0089] According to one embodiment, the deep learning-based vision artificial intelligence algorithm applied to the present disclosure may include one of various algorithms based on a convolutional neural network (CNN), such as Fast RCNN (recurrent convolutional neural network), YOLO (you only look once) series, and mask RCNN (recurrent convolutional neural network). In addition, the deep learning-based vision artificial intelligence algorithm applied to the present disclosure may include one of various algorithms based on VIT (Vision transformer), which is a vision artificial intelligence algorithm that applies a transformer structure.

[0090] FIG. 8 illustrates an example of an artificial intelligence system capable of recognizing a specific object based on image data according to one embodiment of the present disclosure.

[0091] In another embodiment, the present disclosure may be applied to a reward-based reinforcement learning artificial intelligence (AI) system. Reinforcement learning-based AI is AI that learns to achieve better outcomes (e.g., rewards) by interacting with a given environment. To achieve this, a prediction function must be defined that defines the level of reward that an environment (state) and action will provide in the future. Reinforcement learning-based AI can determine the optimal behavioral pattern based on factors such as preference based on the timing of reward provision and the maximum reward that can be obtained when all actions are completed in the environment. The optimal behavioral pattern can be determined by selecting a specific action that yields the highest reward among the behavioral patterns performed so far. Furthermore, rather than continuing the established optimal behavioral pattern, reinforcement learning-based AI can explore and learn new behavioral patterns that yield higher reward values ​​by selecting different behavioral patterns.

[0092] As an example, below, the action is defined as recognizing a bonding process defect by recognizing a bonding process defect in the measurement / image data.

[0093] FIG. 9 is a diagram illustrating a system for detecting defects in a bonding process and performing rework according to one embodiment of the present disclosure. Referring to FIG. 9, the system (900) may include a defect inspection unit (910), a defect detection unit (920), an artificial intelligence reinforcement learning unit (930), a defect performance verification unit (940), a rework process judgment unit (950), and a rework process system (960). For example, the system (910) may operate based on a defect detection device or may include multiple devices to perform respective operations, and may not be limited to a specific form.

[0094] The defect inspection unit (910) can set and inspect one of various joining methods for the joining of similar and different metal materials such as steel, aluminum, and other metals. The defect inspection unit (910) can obtain data for inspecting whether there is a defect in the joining process based on the joining process method, joining process components, joining process location, and location information for each detailed component. The defect inspection unit (910) can collect data through a vision sensor that uses visible light, a thermal imaging camera, and other sensors. As another example, the defect inspection unit (910) can collect data using a film or sensor that can observe the laser or radiation after irradiating a laser or radiation through a laser sensor or a radiation irradiator. Alternatively, the defect inspection unit (910) can collect data through other sensors or devices.

[0095] The system (900) can inspect for defects based on the bonding process method, bonding process components, bonding process location, and detailed component-specific location information. Specifically, the defect inspection unit (910) can obtain at least one of measurement data and image data based on the bonding process method, bonding process components, bonding process location, and detailed component-specific location information. The defect inspection unit (910) can obtain measurement data by performing measurements based on the bonding process method, bonding process components, bonding process location, and detailed component-specific location information, or can obtain image data by taking an image of a portion where the bonding process has been performed.

[0096] More specifically, the system (900) can generate bonding process performance information in the product manufacturing process. Here, the bonding process performance information may be list information on processes in which the bonding process is performed within the product manufacturing process, and each bonding process may be assigned an identifier to distinguish it. In addition, bonding process method, bonding process components, bonding process locations, and location information for each detailed component may be determined for each bonding process. In other words, information on how the bonding process is performed for each bonding process may be provided to the system (900). In addition, the bonding process performance information may include information on the order in which each bonding process is performed. The system (900) may move the defect inspection unit (910) to each bonding process location based on the order information to acquire measurement / image data, but is not limited thereto. As another example, the system (900) may include a plurality of defect inspection units (910). Each of the plurality of defect inspection units (910) may acquire measurement / image data for a corresponding bonding process. Each of the plurality of defect inspection units (910) can acquire measurement / image data in different forms depending on the corresponding bonding process. As a specific example, the defect inspection unit (910) corresponding to the bonding process using the brazing technology can acquire thermal measurement information and image information based on a thermal imaging camera. On the other hand, the defect inspection unit (910) corresponding to the bonding process using the riveting technology can acquire image information captured based on the coordinates of the position of the metal plate, the position of the fixing pin, and other components based on a proximity camera, but may not be limited thereto.

[0097] The defect inspection unit (910) can obtain at least one of measurement data and image data based on the bonding process method, bonding process components, bonding process locations, and detailed component-by-part location information for each bonding process, and for convenience of explanation, it is referred to as measurement / image data below.

[0098] The defect detection unit (920) of the system (900) can recognize the type of the corresponding component based on the measurement / image data provided by the defect inspection unit (910). The defect detection unit (920) can recognize a specific symbol displayed on the corresponding component using information included in the measurement / image data. Specifically, the system (900) can store specific symbol information based on at least one of the bonding process method, bonding process component, bonding process location, and detailed component-by-component location information.

[0099] As a specific example, specific symbol information may include values ​​corresponding to the bonding process method. The specific symbol information may include values ​​corresponding to the bonding process method corresponding to each of brazing, bonding, riveting, and other bonding processes. In other words, a bonding process method indication value may be included in the specific symbol information.

[0100] In addition, specific symbol information may include values ​​corresponding to parts and materials applied to the bonding process. That is, part and material indication values ​​may be included in a specific bonding process. In addition, specific symbol information may include values ​​corresponding to a bonding process position. Here, the bonding process position may be information about a position where the bonding process is performed based on a relative position value determined for the entire product or during the bonding process. That is, a bonding process position indication value may be included in the specific symbol information. In addition, specific symbol information may include location information for each detailed part. The location information for each detailed part may be information about the detailed positions where each detailed part is bonded based on the parts and materials for which the bonding process is performed within the bonding process position. That is, specific symbol information may include location information for each detailed part.

[0101] Specific symbol information is information related to the bonding process and may be stored in the system (900) as preset information according to the bonding process or product manufacturing process and provided to the defect detection unit (920). The defect detection unit (920) may recognize specific symbols from measurement / image information acquired from the defect inspection unit (910) based on the specific symbol information and may detect whether or not a product is defective based on the specific symbol.

[0102] As another example, the defect detection unit (920) may recognize at least one of the bonding process method, bonding process components, bonding process locations, and detailed component-specific location information from the measurement / image data, and transmit a request to a server where a bonding process database is constructed based on the recognized information. The server may provide the information requested by the defect detection unit (920) based on the stored database information, and whether a defect exists may be detected based on the provided information. The server may be physically located within the defect detection unit (920). As another example, the server may be located outside the defect detection unit (920) and may communicate with the defect detection unit (920) to obtain the requested information. As a specific example, the server may be a cloud, an externally constructed server, or a physically separate location within the interior, but is not limited to a specific form.

[0103] The system (900) can always set the same parts and bonding process through pre-defined specifications for the bonding process line and provide the corresponding information to the defect detection unit (920) in advance. Based on the pre-defined specifications information for the bonding process line, the system (900) can control the machinery or devices required for the bonding process to move the corresponding parts to the location where the bonding process is required and perform the bonding process. In addition, the system (900) can also move the defect inspection unit (910) to the location of the corresponding parts and acquire measurement / image data for the bonding process.

[0104] As described above, the system (910) can perform a defect inspection based on Methods 1 to 3 of Table 1 below. Specifically, the system (900) can perform a defect inspection at a location where a bonding process is required within the component and at the same time as the bonding process is completed. Here, the system (900) can move a robot arm or other device performing the bonding process and the defect inspection unit (910) to a location where the bonding process is required. Through this, the system (900) can perform a defect inspection in real time at a location where a bonding process is required within the component and at the same time as the bonding process is completed, and the defect detection unit (920) can check for defects in real time. (Method 1)

[0105] As another example, the system (900) can perform a defect inspection immediately after all bonding processes for the corresponding component are completed. Specifically, the system (900) can move the defect inspection unit (910) to acquire measurement / image data for the corresponding component after all bonding processes for the corresponding component are completed, and the defect detection unit (920) can check whether there are any defects in the bonding processes completed for the corresponding component (Method 2).

[0106] As another example, the system (900) can move the component to a location where a defect inspection process can be performed once all bonding processes for the component are completed, and perform a defect inspection. That is, the defect inspection unit (910) can be located at a location where a defect inspection process can be performed, and the component for which all bonding processes have been completed can be moved to that location. At that location, the defect inspection unit (910) can obtain measurement / image data for the component, and the defect detection unit (920) can check whether there are any defects in the bonding process completed on the component. (Method 3)

[0107] Here, the system (900) can perform a defect inspection through at least one of the following methods 1 to 3. For example, the system (900) can perform a defect inspection through all of methods 1, 2, and 3 to improve the reliability of defect detection. As another example, the system (910) can perform a defect inspection by applying different methods to each component. The system (910) can perform a defect inspection based on method 1 for components with a high defect rate or requiring real-time defect inspection. On the other hand, the system can perform a defect inspection based on method 2 for components whose defects can only be determined after the entire bonding process of the component is completed. In addition, the system (910) can apply a defect inspection method in different forms in consideration of the component bonding process or the product manufacturing process, and is not limited to a specific form.

[0108]

[0109] The system (900) can move the defect inspection unit (910) to the bonding portion where the bonding process is performed through at least one of the bonding process method, bonding process components, bonding process location, and detailed component-by-component location information for defect inspection. For example, the defect inspection unit (910) can be configured to be movable within the system by a robot arm or other mechanical device as described above. In addition, the defect inspection unit (910) performs defect inspection based on a vision sensor including a visible light camera and other sensors or other sensors. That is, the defect inspection unit (910) can obtain measurement / image data. The defect detection unit (920) can perform defect detection at the corresponding bonding portion based on the measurement / image data collected from the defect inspection unit (910), and can determine whether the bonding process is normal. For example, the defect detection unit (920) can compare the measurement / image data based on preset information to determine whether it is a specific type of defect. Specifically, the preset information may include individual location information for each sub-component within the bonding process based on the specific symbol information described above. Furthermore, the preset information may include information generated based on measurement data obtained when the bonding process for the corresponding component is performed normally. The defect detection unit (920) may determine whether a sub-component is defective based on whether the sub-component does not exist at the corresponding location or whether the measured data differs based on the preset information.

[0110] As a specific example, FIG. 10 is a diagram showing a defect type to which the present disclosure can be applied. Referring to FIG. 10, the system (900) can recognize that a bonding process is performed at a specific location, and based on this, move the defect inspection unit (910) to the location to acquire measurement / image data. As an example, FIG. 10 may be a case where the bonding process for the corresponding component is performed at a specific bonding process location. Specifically, a process in which a first metal (1010), a second metal (1020), and a third metal (1030) are bonded as one through a fixing pin (1040) for each detailed component can be performed, and through the bonding process, the fixing pin (1040) and the first metal (1010), the second metal (1020), and the third metal (1030) can be bonded adjacently without a free hole space, thereby producing a completed component. Here, the positions at which the fixing pin (1040) is coupled with the first metal (1010), the second metal (1020), and the third metal (1030) in the bonding process position can be preset in the system (900), and the fixing pin (1040) and the first metal (1010), the second metal (1020), and the third metal (1030) can be coupled at corresponding positions. In addition, the fixing pin (1040) can be configured with a head portion (1041) and a body portion (1042) in a screw shape, and the body portion (1042) can be coupled adjacent to the first metal (1010), the second metal (1020), and the third metal (1030) so that the respective metals are coupled. In addition, the head portion (1041) can be coupled so as to be positioned above the first metal (1010) at the top. That is, the system (900) can be set with preset information for the above-described joint form, and the defective type can be determined based on the information.

[0111] For example, FIG. 10(a) may be a defective type in which a free hole exists. Specifically, the first metal (1010), the second metal (1020), and the third metal (1030) must be bonded adjacently without a space separated from the fixing pin (1040), but a space may be created between the fixing pin (1040) and each metal during the bonding process, and the defective inspection unit (910) may obtain measurement / image data related to the relevant portion and transmit it to the defective detection unit (920). In the defective detection unit (920), metal information that must exist at each position or coordinate centered on the position of the fixing pin (1040) may be preset, and if there is no metal information at the position or coordinate corresponding to the measurement / image data obtained from the defective inspection unit (910), it may be determined as a defective type in which a free hole exists.

[0112] Also, referring to FIG. 10(b), the fixing pin (1040) should be joined to the first metal (1010), the second metal (1020), and the third metal (1030) at a specific position, but may be joined at a position that is slightly off from the specific position. The defect detection unit (920) can check whether there is a gap between positions in the measurement / image data based on the preset position information of the fixing pin (1040), and thereby a defect type of the positional defect type can be detected. For example, the defect detection unit (920) can determine a defect if the gap distance between the position where the current bonding process is performed and the position where the bonding process is performed based on the preset information is greater than a threshold value. The threshold value may be a value preset in advance. In addition, referring to FIG. 10(c), the fixed pin (1040) may be composed of a head portion (1041) and a body portion (1042), and the defect inspection unit (910) may obtain measurement / image data information as information on the form in which the fixed pin (1040) is joined to the first metal (1010), the second metal (1020), and the third metal (1030). The defect detection unit (920) may check whether the head portion (1041) is located at a preset position from the information obtained from the defect inspection unit (910), and if the head portion (1041) is not detected, a defect type of a head melting type may be detected.

[0113] Fig. 11 is a diagram showing a defect type applicable to the present disclosure. Referring to Fig. 11, the defect inspection unit (910) may be equipped with a lidar sensor, a laser sensor, and other sensors capable of measuring the depth of the fixed plate (1140), and may acquire measurement / image data based thereon. For example, a process may be performed in which a first metal (1110), a second metal (1120), and a third metal (1130) are joined as one through a fixed pin (1140) at a specific location, and by the joining process, the fixed pin (1140) and the first metal (1110), the second metal (1120), and the third metal (1130) may be joined adjacently without a space separating them. Here, the defect detection unit (920) can detect the defective type of the under-sealing type of FIG. 11(a) or the defective type of the double-headed fastening type of FIG. 11(b) based on the measurement / image data acquired from the defect inspection unit (910). For example, the defective type of the under-sealing can be determined as a defect by setting the value of the difference in the length of the head portion (1141) itself compared to the height of the head portion (1141) according to the corresponding bonding process as a threshold value, and if it is greater than the threshold value, it can be determined as a defect. That is, in FIG. 11(a), the height information of the head portion (1141) is acquired from the surface of the first metal (1110) located at the top, and the length information of the head portion (1141) is compared to determine whether it is greater than the threshold value to determine whether it is defective. As another example, FIG. 11(b) can be a defective type for double-headed fastening. Referring to Fig. 11(b), a threshold value can be set as the difference value between the highest and lowest positions among the heights of the head portion (1141) in the two-way connection, and a determination can be made on the basis of this whether it is defective or not. To this end, the amount of change in the height of the head portion (1141) can be calculated. Specifically, the amount of change in the height of the head portion (1141) can be derived by calculating the difference value between the highest and lowest positions by combining the width information of the head portion (1141) based on the slope value of the height of the head portion (1141).That is, the change in the height of the head portion (1141) can be calculated by reflecting the width information of the head portion (1141) in the slope value, and the corresponding value can be compared with a threshold value to determine whether it is defective. That is, information can be derived based on the measurement / image data acquired through the defect inspection unit (910), and whether it is defective can be determined by comparing it with preset information.

[0114] Here, the defect detection unit (920) first determines whether there is a defect in Fig. 10, and if the corresponding bonding process is normal, it can determine whether there is a defect in Fig. 11. As another example, the defect detection unit (920) can perform defect detection in Fig. 10 and Fig. 11 in parallel, and if either type of defect is detected, it can detect it as a defect, and if no type of defect is detected, it can determine it as normal. Thereafter, the defect detection unit (920) can generate defect data. The defect detection unit (920) can transmit the defect detection data to the artificial intelligence reinforcement learning unit (930). The artificial intelligence reinforcement learning unit (930) can perform learning using the defect detection data. The defect performance verification unit (940) can verify the performance of the learned artificial intelligence model. If the performance of the learned artificial intelligence model is higher than a threshold, the defect performance verification unit (940) can provide weight data to the defect detection unit (920). The defect detection unit (920) that receives the weighted data can update the system by applying the artificial intelligence model to which the weighted data is applied. The defect detection unit (920) can transmit the defect detection data and the measurement data to the rework process judgment unit (950). The rework process judgment unit (950) can determine the rework process decision using the defect detection data and the measurement data. If the rework process judgment unit (950) determines that the rework process system (960) performs at least one of the supportable rework processes based on the defect detection data and the measurement data, the defect detection data, the measurement / image data, and the judgment result (e.g., the type of rework, whether to perform rework, etc.) can be provided to the rework process system (960). The rework process system (960) can perform the rework process (e.g., removal of welded parts, rewelding, reinforcement of welded parts, etc.) based on the received information.

[0115] FIG. 12 is a diagram illustrating a defect detection system applicable to the present disclosure. Referring to FIG. 12, the defect inspection unit (910) may include at least one of a control unit (911), a moving unit (912), a storage unit (913), and an inspection unit (914). The control unit (911) of the defect inspection unit (910) may be a configuration that controls each of the components. In addition, the moving unit (912) may be a configuration that controls movement to a position where a defect inspection is performed based on a bonding process position, the storage unit (913) may store preset information, and the inspection unit (914) may be a configuration that performs a defect inspection based on the stored information to acquire measurement / image data. Here, the defect inspection-related data may be acquired based on at least one of the bonding process method (1201), the bonding process component (1202), the bonding process position (1203), and the detailed position (1204) of each component as described above. For convenience of explanation, in the following, the process method (1201), bonding process component (1202), bonding process location (1203), and detailed location for each component (1204) are referred to as bonding process-related data, but may not be limited thereto.

[0116] Specifically, the defect inspection unit (910) can obtain measurement / image data based on at least one of the bonding process-related data transmitted from the system. Here, the measurement / image data may be data obtained based on any one of Methods 1 to 3 based on Table 1. Here, the measurement / image data may be data labeled based on at least one of the bonding process-related data. The defect detection unit (920) can generate defect detection data based on the measurement / image data, and the defect detection data may also be data labeled based on at least one of the bonding process-related data. For example, the defect detection data may be data in which the defect type information of FIGS. 10 and 11 is generated and distinguished based on at least one of the bonding process-related data, thereby recognizing characteristic information of the bonding process in which a defect is detected. Thereafter, the defect detection data may be transmitted to the artificial intelligence learning unit (930), and the artificial intelligence learning unit (930) may be updated based on the defect detection data and transmit the defect sample data to the defect performance verification unit (940). The defective performance verification unit (940) can generate weighted data based on defective sample data and transmit this back to the defective detection unit (920) to increase the reliability of defective inspection. Here, each data can be generated based on data related to the bonding process. As a specific example, the defective sample data and the weighted data can be updated by utilizing data having the same bonding process method (1201) and bonding process components (1202) but different bonding process locations (1203). In this case, the defects may be of the same type, and the defective sample data and the weighted data can reflect the corresponding data together to increase the reliability of the bonding process.

[0117] As another example, the defect detection data may be transmitted to the rework process judgment unit (950) so that the rework process system (960) may instruct whether to perform rework. Here, the defect detection data and the rework instruction data for determining whether to perform rework may be generated based on data related to the bonding process. Specifically, the defect detection data and the rework instruction data may be generated based on the bonding process-related data according to the above-described methods 1 to 3. Here, the rework instruction data may generate instruction information for causing the component bonding process applied in real time to be performed again based on method 1. As another example, the rework suggestion data may instruct rework after all bonding processes of the corresponding component are completed based on method 2, so that all bonding processes for the corresponding component may be newly started. Here, the above-described instruction information may be data generated based on the bonding process-related data, and may reduce unnecessary rework and increase efficiency based on the above-described operation.

[0118] Figure 13 is a flowchart showing a defect detection method applicable to the present disclosure.

[0119] Referring to FIG. 13, the defect detection device can collect measurement / image data. (S1310) For example, the defect inspection unit of the defect detection device can collect measurement / image data. Thereafter, the defect detection device can generate defect detection data from the measurement / image data using the learned artificial intelligence model. (S1320) Here, the measurement / image data and the defect detection data can be generated based on at least one of the bonding process method, bonding process components, bonding process locations, and locations by detailed components, as described above.

[0120] For example, the measurement / image data and the defect detection data may be acquired for each bonding process at the first location where the bonding process of the first component is performed. That is, the data may be acquired for each bonding process of the component based on Method 1 of Table 1 described above. As another example, the measurement / image data and the defect detection data may be collected when all bonding processes are completed at the first location where the bonding process of the first component is performed. That is, the data may be acquired after all bonding processes of the component are completed based on Method 2 of Table 2 described above. As another example, the measurement / image data and the defect detection data may be collected when all bonding processes are completed at the first location where the bonding process of the first component is performed and the first component is moved to the second location where a defect inspection is performed. That is, the data may be collected when all bonding processes for the component are completed and the component is moved to a specific location based on Method 3 of Table 1 described above, but is not limited thereto.

[0121] As another example, measurement / image data and defect detection data may be generated based on information collected by a vision sensor at a bonding site where a bonding process is performed. A defect inspection unit of a defect detection device may be equipped with a vision sensor including a visible light camera, and may generate an image acquired based thereon. Here, if the defect detection data generated from the measurement / image data corresponds to a preset defect type, it may be determined as a defect, and if it does not correspond to a defect type, it may be determined as normal. For example, if a defect is detected, defect detection data including the defect type and defect-related information may be generated. On the other hand, if no defect is detected, defect detection data indicating that a normal bonding process is performed may be generated. The defect type may include at least one of a prehole cover defect type, a position defect type, and a head melting type, which may be as shown in FIG. 10.

[0122] In addition, as an example, the measurement / image data and the defect detection data may be generated based on information collected based on at least one of a lidar sensor and a laser sensor that measure the depth at the bonding portion where the bonding process is performed. The defect inspection unit of the defect detection device may be equipped with a lidar sensor and a laser sensor that measure the depth of the bonding portion, and may generate the measurement / image data acquired based thereon. Here, if the defect detection data generated from the measurement / image data corresponds to a preset defect type, it may be determined as a defect, and if it does not correspond to the defect type, it may be determined as normal. For example, if a defect is detected, defect detection data including the defect type and defect-related information may be generated. On the other hand, if no defect is detected, defect detection data indicating that a normal bonding process is performed may be generated. The defect type may include at least one of an under-insertion defect type and an inner-hole fastening defect type, which may be as shown in FIG. 11.

[0123] After that, the defect detection device can determine whether to re-construct based on the measurement / image data and the defect detection data. (S1330) In addition, the defect detection device can update the artificial intelligence model based on the measurement / image data and the defect detection data, as described above. (1340)

[0124] Additionally, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, the embodiments may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, etc.

[0125] The scope of the present disclosure includes software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium having such software or instructions stored thereon and executable on the device or computer.

[0126] The various embodiments of the present disclosure are not intended to list all possible combinations but rather to illustrate representative aspects of the present disclosure, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0127]

[0128] The above may also apply to other systems.

Claims

1. A method for detecting and re-working defects in a bonding process based on artificial intelligence, Step of collecting measurement / image data; A step of generating defect detection data from the above measurement / image data using a learned artificial intelligence model; A step of determining whether to re-construct based on the above measurement / image data and the above defect detection data; and A method comprising the step of updating the artificial intelligence model based on the measurement / image data and the defect detection data.

2. In paragraph 1, A method wherein the above measurement / image data and the above defect detection data are generated based on at least one of a bonding process method, a bonding process component, a bonding process location, and a location by subcomponent.

3. In paragraph 2, A method wherein the above measurement / image data and the above defect detection data are acquired for each bonding process at the first location where the bonding process of the first part is performed.

4. In paragraph 2, A method wherein the above measurement / image data and the above defect detection data are collected when all bonding processes are completed at a first location where the bonding process of the first part is performed.

5. In paragraph 2, A method wherein the above measurement / image data and the above defect detection data are collected when all bonding processes are completed at a first location where a bonding process of the first part is performed and the first part is moved to a second location where a defect inspection is performed.

6. In paragraph 2, A method in which the above measurement / image data and the above defect detection data are generated based on information collected by a vision sensor at a bonding site where a bonding process is in progress.

7. In paragraph 6, The above measurement / image data is acquired based on the vision sensor, The above defect detection data is determined to be defective if it corresponds to a predetermined defect type based on the measurement / image data, and the above defect detection data is generated. A method wherein the above-mentioned defective type includes at least one of a free hole cover defective type, a position defective type, and a head melting type.

8. In paragraph 2, A method wherein the above measurement / image data and the above defect detection data are generated based on information collected based on at least one of a lidar sensor and a laser sensor that measures depth at a bonding site where a bonding process is performed.

9. In paragraph 8, The above measurement / image data is acquired based on at least one of the lidar sensor and the laser sensor, The above defect detection data is determined to be defective if it corresponds to a predetermined defect type based on the measurement / image data, and the above defect detection data is generated. A method wherein the above-mentioned defective type includes at least one of a poor fit defective type and a poor connection defective type.

10. A program stored on a medium for executing a method according to any one of claims 1 to 9 when operated by a processor.

11. In a device that detects defects in the bonding process and performs re-construction based on artificial intelligence, Defect inspection unit that acquires measurement / image data; A defect detection unit that generates defect detection data based on the above measurement / image data; A rework process judgment unit that judges whether a rework process is necessary based on the above defect detection data; A re-construction process system that instructs re-construction based on whether the above re-construction is necessary; An artificial intelligence learning unit that updates a learning model based on the above defect detection data; and A device including a defect performance verification unit that generates weight data based on the updated learning model and transmits the generated weight data to the defect detection unit.

Images