Automatic vision inspection system for automotive processing line

Abstract

An inspection system includes a processing line configured to support an object, at least one camera adjacent to the processing line, the at least one camera configured to capture at least one image of the object and create a multi-channel image set of the images, and a control module communicating with the at least one camera. The control module is configured to receive the multi-channel image set of the images, detect boundaries associated with objects in the multi-channel image set, and detect defects on the boundaries using a machine learning network based on the multi-channel image set, a first training dataset including an annotated multi-channel image set, and a second training dataset including a multi-channel image set with manually created defects. Other example inspection systems and methods are also disclosed.

Description

[0001] introduction

[0002] The information provided in this section is intended to provide a general overview of the context of this disclosure. To the extent described in this section, the work of the currently named inventors, and descriptions that may not conform to the prior art at the time of submission, are neither explicitly nor implicitly acknowledged as prior art that is detrimental to this disclosure. Technical Field

[0003] This invention relates to an automated visual inspection system for vehicle manufacturing lines. Background Technology

[0004] Vehicle components are typically manufactured along production lines. Some of these components, such as crankshafts and camshafts, are high-precision parts, requiring defects no larger than a few micrometers in size. For example, the crankshaft is a crucial component of a vehicle's engine, responsible for converting the internal combustion power of gasoline in the cylinders into rotational motion that powers the wheels. In traditional production lines, crankshafts are machined incrementally, with tolerances decreasing progressively, from the initial raw workpiece to the finished part. While this process is extremely precise and robust, in rare cases, defects can occur in the crankshaft. Defects as small as a few micrometers can have a catastrophic impact on the engine's future performance. Traditionally, the task of human operators has been to inspect the crankshaft for such defects. Summary of the Invention

[0005] An inspection system for automatically detecting defects in an object includes a processing line configured to support the object, at least one camera adjacent to the processing line, at least one camera configured to capture at least one image of the object and create a multi-channel image set of the images, each channel of the multi-channel image set providing a different view of the object, and a control module communicating with the at least one camera. The control module is configured to receive the multi-channel image set of the images, detect boundaries associated with the object in the multi-channel image set, and detect defects on the boundaries using a machine learning network based on the multi-channel image set, a first training dataset including an annotated multi-channel image set, and a second training dataset including a multi-channel image set with manually created defects.

[0006] Among other features, the control module is configured to generate a control signal in response to the detection of a defect on an object, and based on the control signal, control the actuator to move the defective object to a defined position.

[0007] Among other features, the detected defect on the object is one of multiple recurring defects associated with multiple objects, and the control module is configured to generate a notification signal based on the multiple recurring defects to check the area of ​​the processing line used to create the multiple objects.

[0008] Among other features, the camera is configured to capture at least one image of the object as it rotates.

[0009] Among the other features, the object is a vehicle crankshaft or camshaft.

[0010] Among other features, the multi-channel image set includes at least a normal channel, a specular reflection channel, a diffuse channel, a glossiness channel, and a shape channel, and the control module is configured to detect the boundary in the diffuse channel associated with the vehicle crankshaft or camshaft based on the change in image brightness in the diffuse channel.

[0011] Among other features, the vehicle crankshaft or camshaft includes at least one oil hole, and the control module is configured to detect at least one edge of the oil hole based on a combination of diffuse and gloss channels, and to detect defects associated with the edge of the oil hole.

[0012] Among other features, the machine learning network is a convolutional neural network, and the control module is configured to create a stack of overlapping patches from a multi-channel image set and use the convolutional neural network to detect defects on the boundary based on the stack of these overlapping patches.

[0013] Among other features, the second training dataset includes at least one multi-channel image set of intentionally defective objects.

[0014] Among other features, the second training dataset includes at least one multi-channel image set created based on a transformation of a previous multi-channel image set of the defective object.

[0015] Among other features, the machine learning network is continuously trained on both annotated and unannotated training data.

[0016] An inspection method for automatically detecting defects in an object includes creating a multi-channel image set from images of an object captured by a camera, each channel in the multi-channel image set providing a different view of the object, detecting boundaries associated with the object in the multi-channel images, and using a machine learning network to detect defects on the boundaries based on the multi-channel image set, a first training dataset including an annotated multi-channel image set, and a second training dataset including a multi-channel image set with manually created defects.

[0017] Among other features, the inspection method also includes generating a control signal in response to detecting a defect on an object, and controlling an actuator to move the defective object to a defined position based on the control signal.

[0018] Among other features, the detected defect on the object is one of multiple recurring defects associated with multiple objects, and the inspection method also includes generating a notification signal based on the multiple recurring defects to inspect the area of ​​the processing line used to create the multiple objects.

[0019] Among other features, the object is a vehicle crankshaft or camshaft, and creating the multi-channel image set from images of the object captured by a camera includes creating the multi-channel image set from images of the vehicle crankshaft or camshaft captured by a camera while the vehicle crankshaft or camshaft is rotating.

[0020] Among other features, the multi-channel image set includes at least a normal channel, a specular reflection channel, a diffuse channel, a glossiness channel, and a shape channel. Detecting boundaries associated with objects includes detecting boundaries associated with vehicle crankshafts or camshafts in the diffuse channel based on changes in image brightness in the diffuse channel.

[0021] Among other features, the machine learning network is a convolutional neural network, and the inspection methods also include creating a stack of overlapping patches from a multi-channel image set. Detecting defects on the boundary involves using a convolutional neural network based on the stack of overlapping patches to detect defects on the boundary.

[0022] Among other features, the machine learning network is continuously trained on both annotated and unannotated training data.

[0023] Among other features, the second training dataset includes at least one multi-channel image set of intentionally defective objects and at least one multi-channel image set created based on transformations of existing multi-channel image sets of defective objects.

[0024] An inspection system for automatically detecting defects in vehicle crankshafts includes a machining line configured to support the vehicle crankshaft, at least one camera adjacent to the machining line, at least one camera configured to capture at least one image of the vehicle crankshaft as it rotates and create a multi-channel image set of images, each channel of the multi-channel image set providing a different view of the vehicle crankshaft, and a control module communicating with the at least one camera. The control module is configured to receive the multi-channel image set of images, detect boundaries associated with the vehicle crankshaft in the multi-channel image set, and detect defects on the boundaries using a machine learning network based on the multi-channel image set, a first training dataset including an annotated multi-channel image set, and a second training dataset including at least one multi-channel image set of a deliberately defective vehicle crankshaft and at least one multi-channel image set created based on a transformation of a previous multi-channel image set of the defective vehicle crankshaft.

[0025] Other applicable areas of this disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0026] This disclosure becomes more fully understood through detailed description and accompanying drawings, in which:

[0027] Figure 1This is a block diagram of an inspection system for automatically detecting defects in objects according to the present disclosure;

[0028] Figure 2 A multi-channel image set depicting the journal surface of a vehicle crankshaft according to the present disclosure includes a diffuse channel, a gloss channel, a shape channel, a normal channel, and a specular reflection channel;

[0029] Figure 3 A set of images depicting the detection and analysis of oil holes in a vehicle crankshaft according to this disclosure;

[0030] Figure 4 Depicting the relationship with this disclosure Figure 2 The stacking of overlapping blocks corresponding to a portion of the diffuse channel, gloss channel, shape channel, normal channel, and specular channel, as well as two supplementary position blocks;

[0031] Figure 5 This is a block diagram of a system for creating supplementary artificial datasets according to this disclosure;

[0032] Figure 6 Based on a set of frequency diagrams of crankshafts analyzed according to this disclosure, a defect that recurs at the same location on the crankshaft is shown.

[0033] Figure 7 It is based on the present disclosure for implementation Figure 1 A block diagram of a system for continuous learning of machine learning networks;

[0034] Figure 8-9 This is a flowchart of an example inspection method for automatically detecting defects in an object according to this disclosure; and

[0035] Figure 10-11 It is to draw according to this disclosure and can be used Figure 1 The graph shows the performance accuracy of the machine learning network in the inspection system during multi-round continuous learning.

[0036] In the accompanying drawings, reference numerals may be used repeatedly to identify similar and / or identical elements. Detailed Implementation

[0037] High-precision vehicle components are typically manufactured along a production line. These components (such as crankshafts and camshafts) are usually machined incrementally, with tolerances decreasing progressively to create finished parts from initial raw workpieces, and defects no larger than a few micrometers are required. For example, defects as small as a few micrometers can have a catastrophic impact on the future utility of a vehicle's engine. Therefore, identifying such components with defects is crucial before incorporating them into a vehicle engine.

[0038] Traditionally, the task of human operators is to inspect parts for defects. This inspection process performed by human operators is tedious and an imperfect visual inspection task. For example, for a vehicle crankshaft, the operator must inspect the crankshaft and, within a short time, visually inspect approximately 200 square centimeters of metal surface, rejecting defective parts. The human operator can then repeat this process about 250 times per shift, meaning the total inspection time for each crankshaft is about 30 seconds. This traditional inspection process is prone to errors associated with missed defects because, due to fatigue, insufficient time, or inability to see the defects, human operators may not be able to spot them with the naked eye. Such undetected defects can lead to catastrophic events caused by including defective parts in a working engine.

[0039] The automated visual inspection system and method according to this disclosure utilize artificial intelligence to perform end-to-end visual inspection of objects (such as crankshafts, camshafts, etc.) on a manufacturing line. The inspection system and method specifically focus on automatically detecting and characterizing surface defects on the object, such as scratches, dents, stains, etc., and checking for the presence of features related to object characteristics (such as oil holes). In various embodiments, and as further explained herein, the inspection system and method create a multi-channel image set from images of the object being inspected captured by a camera, and then utilize a machine learning network to detect defects on the object boundaries based on the multi-channel image set, a received annotated multi-channel image set, and a multi-channel image set with manually created defects.

[0040] By leveraging artificial intelligence for end-to-end visual inspection, the inspection systems and methods presented in this paper achieve inspection metrics superior to human levels, particularly excelling in detecting and characterizing surface defects such as scratches, dents, and stains—some of which are undetectable by human operators. In doing so, the systems and methods enhance accuracy and reliability, contributing to improved product quality. Furthermore, unlike manual inspections relying on human operators, the automated solutions presented in this paper can operate continuously, potentially enabling 24 / 7 operation of the production line, thereby increasing productivity. Therefore, the advantages of adopting these systems and methods extend to cost reduction by minimizing and potentially eliminating human labor and mitigating the risks associated with missed defects, potentially preventing warranty claims, product recalls, and catastrophic events caused by incorporating defective parts into the working engine.

[0041] Furthermore, while the embodiments described herein pertain to the inspection of vehicle crankshafts, it should be understood that the inspection systems and methods described herein can be used for other vehicle objects (e.g., camshafts) and / or other non-vehicle industries. For example, these systems and methods demonstrate adaptability, indicating their application in a variety of manufacturing scenarios, further emphasizing their importance in advancing quality control and production processes. In some examples, the object may have multiple planar portions as well as curved and / or planar (e.g., flat) surfaces. In this case, and as further explained herein, if the object has curved surfaces, multi-channel image sets can be used with the model, and if the object has flat surfaces, conventional rectangular images can be used with the model.

[0042] Now for reference Figure 1 A block diagram of an example inspection system 100 for automatically detecting defects in objects is given. Figure 1 As shown, the inspection system 100 typically includes a processing line 102, at least one camera 106 adjacent to the processing line 102, and a control module 108 communicating with the camera 106 for automatically detecting defects in the object.

[0043] exist Figure 1 In one example, the inspection system 100 automatically detects defects associated with the vehicle crankshaft 104. In such an example, the vehicle crankshaft 104 typically includes various journals 112 and oil holes 114 drilled into some or all of the journals 112. In various embodiments, the journals 112 may include main journals and connecting rod journals. When the crankshaft 104 is installed, the main journals can be clamped into the engine block. The connecting rod journals can be attached to the ends of the connecting rods that extend upwards to the pistons in the engine. Although... Figure 1 The inspection system 100 is shown as inspecting a vehicle crankshaft, but it should be understood that the inspection system 100 can inspect other suitable objects, such as vehicle camshafts, objects with multiple planar portions and curved surfaces (e.g., vehicles or non-vehicle objects), etc.

[0044] As shown, the processing line 102 typically supports the vehicle crankshaft 104. In such an example, the processing line 102 may include a movable conveyor belt. In this example, the vehicle crankshaft 104 may rest on the conveyor belt (e.g., a support extending from the conveyor belt) as the conveyor belt moves in and out of the view frame of the camera 106, as shown by dashed line 118. In other examples, the camera 106 may be movable along the length of the vehicle crankshaft 104 (as shown by dashed line 120).

[0045] Figure 1The camera 106 can be any suitable type of camera capable of creating a multi-channel image set of the captured images. For example, the camera 106 can be a line scan camera to capture high-resolution images of the crankshaft 104, particularly the surface of one of the journals 112. In such an example, the line scan camera is a high-speed camera that captures a single line of pixels at a time while also capturing data across multiple different spectral bands (or channels) within the spectrum, thus allowing for multispectral imaging analysis of the crankshaft 104 as it passes the camera 106. While the following discussion may focus on detecting defects on the surface of one of the journals 112, it should be understood that the inspection system 100 can detect defects on the surface of any one of the journals 112 of the crankshaft 104.

[0046] exist Figure 1 In the example, camera 106 captures at least one image of crankshaft 104 (e.g., the surface of one of journals 112). In various embodiments, camera 106 can capture images of crankshaft 104 as it rotates, as shown by dashed line 116. For example, vehicle crankshaft 104 can be rested on a support of machining line 102, causing crankshaft 104 to rotate as machining line 102 moves or as camera 106 moves. This allows camera 106, which can capture a single line of pixels at a time, to obtain a panoramic image of crankshaft 104.

[0047] Camera 106 then creates a multi-channel image set of the captured images. In such an example, each channel of the multi-channel image set provides a different view of crankshaft 104, and more specifically, a different view of the surface of one of the journals 112. These multi-channels can be specialized for extracting different types of defects. In various embodiments, the multi-channel image set may include five-channel images (or other suitable number of channel images) obtained after post-processing the raw images from camera 106. Each channel represents the unwrapped log (e.g., a panoramic view) in each different channel filter. In various embodiments, the channels may include at least, for example, a normal channel, a specular channel, a diffuse channel, a gloss channel, and a shape channel. To efficiently integrate this information, the channels can be stacked to obtain a 5-channel image containing comprehensive details about the journal surface.

[0048] For example, Figure 2 Depicting Figure 1 A multi-channel image set 200 is provided for the surface of one of the journals 112 in the crankshaft 104. As shown, the multi-channel image set 200 includes a diffuse channel 202, a gloss channel 204, a shape channel 206, a normal channel 208, and a specular reflection channel 210, each channel stacked together to form a 5-channel image. Each different channel 202, 204, 206, 208, 210 provides a different view of the surface of one of the journals 112 with the oil hole 114. Figure 2 In the example, the diffuse channel 202 can represent the primary color of the journal surface, thus determining its overall appearance under illumination. Furthermore, the glossiness channel 204 can be used to control the distribution of specular highlights, indicating how smooth or rough the journal surface is, and the specular reflection channel 210 can be used to control the gloss level of different portions of the journal surface. Additionally, the shape channel 206 can be used to control the grayscale image affecting the shape of the journal surface to allow for more pronounced journal surface details, and the normal channel 208 can be used to provide information about the surface normal at each pixel.

[0049] Return to reference Figure 1 The control module 108 includes a machine learning network 110, such as a convolutional neural network (CNN) classifier, which can be trained based on raw data collection (e.g., images from camera 106) and its annotations, as well as artificially augmented data, as explained further below. For example, and as... Figure 1 As shown, the control module 108 receives an annotated image set 122 and an image set 124 with artificially created defects for training the machine learning network 110 and detecting defects on the crankshaft.

[0050] In various embodiments, it can be based on one or more multi-channel image sets (such as...) Figure 2 A multi-channel image set 200 is used to create an annotated image set 122. For example, control module 108 (or another control module) can receive the multi-channel image set 200 and then process it to verify basic standards for acceptable journal surfaces. This includes ensuring that the boundary of journal 112 does not contain undercut and that the oil hole 114 within journal 112 is correctly drilled.

[0051] To establish the basic facts for boundary detection, reflective paint can be used to mark the boundaries of the test logs. Standard computer vision techniques using the CV2 Python library can be used to threshold the multi-channel image set of 200 and extract boundary information. The adaptive threshold used responds to changes in image brightness, adapting to various factory lighting settings. This boundary detection process can be applied to one or more channels, for example... Figure 2 The diffuse channel 202 is shown. In such an example, control module 108 (or another control module) can detect changes in image brightness based on variations in the diffuse channel 202. Figure 2 The boundaries 220, 230 associated with the vehicle crankshaft 104 within the diffuse channel 202.

[0052] The quality of the oil hole can be assessed, with a focus on detecting sharp edges that may be detrimental to the crankshaft 104. In some examples, control module 108 (or another control module) can implement an algorithm (e.g., a "sea level rise" method) that analyzes the gradual increase in surface area of ​​the oil hole 114 over time to detect the inner and outer edges of the oil hole 114. This can provide indicators for judging the quality of the oil hole drilling.

[0053] In various embodiments, Figure 2 The diffuse channel 202 and gloss channel 204 of the multi-channel image set 200 can be used for porosity analysis. For example, Figure 3 Depicting and Figure 1 An example of the detection and analysis of one of the oil holes 114 in the crankshaft 104, associated with an image set 300. Figure 3 The image includes Figure 2 Images of the gloss channel 204 on the journal surface of the sample with oil hole 114, and images 304 and 306. Image 304 is created by control module 108 (or another control module) to represent the image of gloss channel 204 combined with diffuse channel 202, wherein the combined image is then placed below a threshold so that only a specific percentage of pixels are highlighted. Then, as shown in image 306, control module 108 (or another control module) detects the contours (e.g., edges) of the inner and outer holes by using contours, taking the convex hull, and then fitting ellipses 308 and 310.

[0054] Then, control module 108 (or another control module) can create a stack of overlapping patches from the multi-channel image set 200. For example, after the boundary and porosity detection described above, control module 108 can segment the multi-channel image set 200 into overlapping patches. This creates a stack with three-dimensional dimensions, such as 750 x 740 x 5 pixels. Considering the circular nature of the sample journal 112, this can facilitate downstream processes and ensure that the journal surface is within the boundary (e.g., Figure 2 A comprehensive representation within the boundaries 220, 230). Two additional channels can be introduced to capture the position of each small block relative to the detected oil hole 114. This positional information can be crucial for capturing defects that may periodically occur at specific locations during machining operations.

[0055] As an example, Figure 4 Depicting respectively corresponding to Figure 2The control module 108 is a stack 400 of overlapping blocks (or pixels) 402, 404, 406, 408, and 410 representing a portion of the diffuse channel 202, gloss channel 204, shape channel 206, normal channel 208, and specular reflection channel 210. To incorporate the positional information of each block 402, 404, 406, 408, and 410, the control module 108 can supplement the 5-channel blocks 402, 406, 408, and 410 with two additional images (or blocks) 414 and 416 of equal size to the blocks 402, 404, 406, 408, and 410. In such an example, two additional images 414 and 416 can be created to include the "x" and "y" positions of each block 402, 404, 406, 408, and 410. In this example, the outer fitting ellipse of the oil hole 114 (e.g., ...) Figure 3 The center of the ellipse (308) can be used as the origin, and the positions of the smaller blocks 402, 404, 406, 408, and 410 can be normalized to +1 and -1 in both directions. Then, each of the smaller blocks 402, 404, 406, 408, and 410 can be supplemented with two images 414 and 416, one for the x-position value of each block and one for the y-value. This results in a 7-channel stack.

[0056] In various embodiments, Figure 2 The multichannel image set 200 and other multichannel image sets can be analyzed and annotated by a human operator to create an annotated image set 122. In such an example, the human operator could be an expert familiar with the crankshaft 104 and the machining line 102. In some examples, annotations representing OK (e.g., non-catastrophic defects) and non-OK or NOK (e.g., catastrophic defects) can be color-coded and overlaid on one of the channel images of the multichannel image set (e.g., multichannel image set 200).

[0057] For example, non-catastrophic defects (OK) can be identified by a green annotation (e.g., stains), while catastrophic defects can be identified by a red annotation (e.g., dents and scratches). In other examples, color-coded annotations can be based on the type of defect, the category of which can be inferred from the annotation line color. In such examples, each type of defect may correspond to a different defined color. By way of example only, defect types may include unrefined (NCU) machining marks, scratches, grinding (unpolished), rust, rough bands (unpolished surfaces), pits (dents, shot peening), pores, reflections, stains, spots, water droplets, etc. Furthermore, in some examples, color-coded annotations may include a symbol for a perfect part corresponding to another defined color.

[0058] In various embodiments, the multi-channel image set can be analyzed simultaneously with the generated patches. In this way, the control module 108 can assign a label to each patch based on whether it contains a defect. In such an example, if the bounding boxes of the annotations overlap, with a coverage of at least 25%, indicating a potential defect in a particular patch, then that particular patch may be considered defective (e.g., NOK). This annotated data serves as the basis for training the machine learning network 110, ensuring accurate identification of defects during subsequent inspections.

[0059] As mentioned above, Figure 1 The control module 108 also receives an image set 124 with artificially created defects for training the machine learning network 110 and detecting defects on the crankshaft. This artificially created data complements the annotated image set 122, addressing the aforementioned challenges of defect rarity and imbalance between defective (NOK) and non-defective (OK) patches. For example, conventional machine learning methods for anomaly detection typically require extensive training on large datasets to effectively capture the true distribution of defects. However, this approach faces significant challenges in the machining line environment of precision parts such as vehicle crankshafts. An inherent drawback of conventional machine learning for anomaly detection is its need for well-represented sampling of the true distribution of defects, which is particularly problematic when dealing with rare defects in the machining line. In these scenarios, the defects themselves are insufficiently representative of the training dataset, leading to a lack of robustness in the detection model. Furthermore, the distribution of defect types is uneven, with some defects, such as dents on metal surfaces, being rarer than others like scratches. This unevenness further complicates the training process and reduces the effectiveness of conventional machine learning techniques. Therefore, the combination of the image set 124 with artificially created defects and the annotated image set 122 provides a robust training set for the machine learning network 110 with a uniform distribution of defect types.

[0060] Supplementary artificial datasets can be created using various methods to complement the defective (NOK) data and ensure a balanced dataset for effective machine learning model training. For example, one approach is to create an artificially generated training dataset (e.g., image set 124) that includes one or more multi-channel images of one or more intentionally defective vehicle crankshafts (or another suitable object being analyzed). In such an example, the defects are manually induced by human operators who intentionally damage normal crankshafts using tools such as hammers, drills, and punches, causing catastrophic defects on otherwise acceptable journal surfaces. These defective crankshafts are then incorporated into machining line 102 and imaged by camera 106, providing a realistic representation of the defects in the dataset. Notably, these defects exhibit high density and regular spacing, offering unique insights into the impact of the induced defects on the journal surfaces.

[0061] Furthermore, and / or alternatively, the artificially created training dataset (e.g., image set 124) may include one or more multichannel image sets created based on transformations of previous multichannel image sets of a defective vehicle crankshaft (or another suitable object being analyzed). In such an example, new artificial defects can be created by transforming existing defective (NOK) patches, thereby creating new instances in the training dataset. For example, if the extracted patch still contains the bounding box of the defect, a large existing defective (NOK) patch can be transformed by randomly rotating and / or flipping it and then extracting a smaller portion of the large patch (e.g., a centered 750x750 pixel patch) to create a new artificially defective (NOK) patch. In various embodiments, transformations may be selectively applied, focusing on defect types that are underrepresented in the defective (NOK) dataset, such as dents, scratches, and stains. In this example, the goal is to enhance the diversity and comprehensiveness of the training set by increasing the sparsity of the defective (NOK) dataset.

[0062] Figure 5 A system 500 for creating supplementary artificial datasets is illustrated. As shown, system 500 includes a data augmentation module 508 that receives an annotated image set 122 with existing defective (NOK) patches and then transforms the existing defective (NOA) patches into a new multi-channel image set 526 with new defective (NOK) patches. Then, as described above, the new multi-channel image set 526 with new defective (NOK) patches, along with the annotated image set 122 with existing defective (NOK) patches and the multi-channel image set 224 with defective (NOK) patches created from intentionally defective vehicle crankshafts, is passed to a control module 108 for training a machine learning network 110 (e.g., a CNN classifier).

[0063] Continue to refer to Figure 1 The machine learning network 110 can be a CNN classifier. In this case, a training dataset enriched with various augmentation techniques serves as the basis for training the neural network to predict whether a defect (e.g., label 0) or the absence of a defect (e.g., label 1) exists in a given patch. In some examples, the CNN classifier can be a custom ResNet-50 neural network adapted to the 7-channel images constituting the patch dataset. In such examples, the last layer of the neural network includes two neurons that contribute to binary classification (label 0 or label 1), which differs from the default 1000-neuron layer designed for ImageNet image classification. In various embodiments, the training process of the machine learning network 110 can employ the Adam optimizer and extend to more than 100 iterations.

[0064] In various embodiments, given the inherent asymmetry in our dataset—where the number of non-catastrophic defect (OK) patches exceeds the number of defective (NOK) patches due to the rarity of defects—a data balancing strategy can be employed. For example, defective (NOK) patches can be manually weighted to balance their importance during training, ensuring the CNN classifier encounters the same number of non-catastrophic defect (OK) and defective (NOA) instances in each iteration. Furthermore, within the defective (NOK) dataset, each patch can be weighted according to the type of defect it contains, ensuring that each defect type has the same representation throughout training. This approach enhances the CNN classifier's ability to identify and characterize defects before evaluating them on the test set.

[0065] In various embodiments, the inspection system 100 can implement multiple combined machine learning models for prediction. In other words, ensembles can be used to further improve performance. For example, multiple CNN classifiers initialized with different random seeds can be trained on the same dataset. In such an example, a democratic voting scheme can be used to combine the predictions of these different classifiers, mitigating the impact of outlier predictions and avoiding traits that might arise from training on limited data. An odd number of classifiers is preferred for ensembles to prevent ties and ensure a robust and reliable defect prediction system.

[0066] Once the machine learning network 110 is sufficiently trained, the inspection system 100 can implement the machine learning network 110 to detect anomalies associated with the vehicle crankshaft (e.g., vehicle crankshaft 104) or another suitable object on which the machine learning network 110 is trained. For example, as Figure 1 As shown and described above, when the crankshaft rotates, camera 106 captures images of the vehicle crankshaft (e.g., crankshaft 104 or another crankshaft) and creates a multi-channel image set. Control module 108 then receives the multi-channel image set, detects boundaries associated with the crankshaft within the multi-channel image set, and then uses a trained machine learning network 110 to detect defects on the boundaries based on the multi-channel image set and a training dataset (e.g., annotated image set 122 and image set 124 with artificially created defects). In some examples, control module 108 may create a stack of overlapping patches from the multi-channel image set created by camera 106 as described above, and then detect defects on the boundaries based on the stack of overlapping patches. Furthermore, control module 108 may detect at least one edge of an oil hole (e.g., one of oil holes 114) on the crankshaft based on a combination of diffuse and gloss channels from the multi-channel image set created by camera 106 as described above, and then detect defects associated with the oil hole edge.

[0067] In various embodiments, Figure 1The inspection system 100 can implement control functions based on the output of the machine learning network 110. For example, and as... Figure 1 As shown, the inspection system 100 may also include an actuator 126 that communicates with the control module 108. In such an example, the control module 108 may generate a control signal in response to the machine learning network 110 detecting a defect on the vehicle crankshaft. The control module 108 may then transmit the control signal to the actuator 126 to control the actuator 126 to move the defective vehicle crankshaft to a defined location, such as the defect box 128. In other examples, the control module 108 may generate another control signal in response to the machine learning network 110 not detecting a defect on the vehicle crankshaft. In such an example, the control module 108 may transmit this control signal to the actuator 126 to control the actuator 126 to move the defective vehicle crankshaft to another defined location, such as the non-defect box 130.

[0068] In further examples, control module 108 may generate notification signals to examine areas of machining line 102 used to create multiple objects. For example, a defect on a vehicle crankshaft detected by machine learning network 110 may be one of multiple recurring defects associated with the analyzed crankshaft. In such an example, control module 108 (or another control module) may generate a frequency map (e.g., a heatmap) for a set of analyzed crankshafts, showing a specific defect that reappears in the same place (e.g., at the bottom of a particular journal). If this occurs, control module 108 may generate notification signals based on multiple recurring defects to examine areas of machining line 102 in an attempt to resolve potential defects causing problems along machining line 102.

[0069] As an example, Figure 6 A frequency map 600 of a set of analyzed crankshafts is shown, illustrating that scratch-type defects occurred at the same location on the bottom of specific journals of a large number of crankshafts. In this example, the frequency map 600 includes position coordinates 602 for locating the identified area of ​​the crankshaft and a frequency range 604 for indicating the number of similar defects detected at the same location.

[0070] Continue to refer to Figure 1 By continuously training the machine learning network 110 based on both annotated and unannotated training data, the machine learning network 100 can be improved over time. In such an example, the inspection system 100 can use feedback between one or more human experts (e.g., operators) who annotate the dataset and the model output (e.g., the unannotated dataset) to progressively improve the machine learning network 110 with each iteration of training and testing.

[0071] For example, Figure 7 Describing for Figure 1An example system 700 demonstrates the continuous learning of a machine learning network 110. In this example, the machine learning network 110 is formed by multiple CNN classifiers. As shown, once the classifiers are trained based on initial annotated training data 706, a trained ensemble defect classifier 702 is created, ready to directly predict small patches derived from unannotated training data 704 on the factory floor. This is achieved through... Figure 7 The predictions made from the unannotated training data 708 are shown. Then, as shown in the new annotated training data 710, the predictions from the unannotated training data 708 are evaluated and annotated (if needed) and used as pseudo-annotated training data. Then, once the classifier is continuously or incrementally trained based on the new annotated training data 710, a retrained ensemble defect classifier 712 is created, ready to directly predict small patches derived from the next batch of unannotated training data 714 from the factory floor. This is achieved by... Figure 7 The unannotated training data 716 is shown for prediction.

[0072] In this example, subsequent evaluation of predictions on the unannotated training data 708 proved to be much faster than evaluating the logs entirely from scratch. This rapid evaluation not only saved time but also allowed human experts to validate the predictions and create new annotations as needed. These annotations became a valuable supplement to the training set used in subsequent training epochs.

[0073] When these annotations are specifically tailored to the initially made predictions (e.g., predictions for unannotated training data 708), the additional information strategically addresses the regions in the CNN classifier that require the greatest improvement, thereby enhancing overall performance. The retrained CNN classifier learns from its previous errors, avoids repetition, and gains the ability to make accurate predictions on new, unannotated logs, thus initiating a continuous learning cycle. This iterative process can be repeated until the desired level of capability is reached, ensuring continuous improvement and optimization of System 100.

[0074] Figure 8-9 It shows Figure 1 The inspection system may employ example inspection methods 800 and 900 for automatically detecting defects in objects such as vehicle crankshafts. Although example inspection methods 800 and 900 pertain to the inclusion of a control module 108 and a machine learning network 110... Figure 1 The inspection system 100 is described, but any of the inspection methods 800 and 900 can be adopted by another suitable system and / or module.

[0075] like Figure 8As shown, inspection method 800 begins at 802, where control module 108 receives an annotated multi-channel image set. In various embodiments, the multi-channel image set can be created based on images captured from camera 106 (e.g., Figure 2 The multichannel image set 200 is then analyzed and annotated by a human operator to create an annotated image set. Then, inspection method 800 proceeds to 804, where control module 108 receives the multichannel image set with artificially created defects. As described above, in such an example, the multichannel image set with artificially created defects may include a multichannel imaging set of a deliberately defective vehicle crankshaft and a multichannel image set created by transforming an existing multichannel image set of a defective vehicle crankshaft. In various embodiments, the annotated multichannel image set of 802 and the multichannel image set with artificially created defects of 804 can be used to train machine learning network 110. Then, inspection method 800 proceeds to 806.

[0076] At 806, a new multi-channel image set is created from the captured images of the vehicle crankshaft. For example, and as described above, camera 106 can capture images of the vehicle crankshaft as it rotates, and then create a multi-channel image set based on the captured images. In other examples, control module 108 can create a multi-channel image set based on the captured images from camera 106. Then, inspection method 800 proceeds to 808, where control module 108 detects a boundary associated with the vehicle crankshaft based on the multi-channel image set. For example, and as described above, control module 108 can use the diffuse channel of the multi-channel image set to detect the boundary. In such an example, control module 108 can detect the boundary based on changes in image brightness in the diffuse channel. Then, inspection method 800 proceeds to 810.

[0077] At 810, control module 108 utilizes the trained machine learning network 110, based on the multi-channel image set at 806, the annotated image set at 802, and the image set with artificially created defects at 804, to detect the presence of defects on the vehicle crankshaft boundary. Then, if a defect is detected (yes at 812), inspection method 800 proceeds to 814, where control module 108 generates a control signal to control actuator 126 to move the defective vehicle crankshaft to a defined location, such as defect box 128. If no defect is detected (no at 812), inspection method 800 proceeds to 816, where control module 108 generates a control signal to control actuator 126 to move the vehicle crankshaft to another defined location, such as defect box 130. Inspection method 800 can then proceed as follows: Figure 8 The process can end as shown, or, if necessary, return to another appropriate step.

[0078] Figure 9 The inspection method 900 is similar to Figure 8 The inspection method is 800, but includes alternative steps. For example, such as... Figure 9 As shown, inspection method 900 is in Figure 8 Starting at point 802, and proceeding to the point mentioned above relative to... Figure 8 The aforementioned 804, 806, 808, 810, and 812.

[0079] If no defect is detected (no at position 812), then inspection method 900 can proceed as follows: Figure 9 The process can end as shown, or, if necessary, return to another suitable step. However, if a defect is detected (yes at 812), inspection method 900 continues to 914, where control module 108 generates a frequency map (e.g., a heat map) for a set of analyzed crankshafts, indicating a defect that repeats or recurs in the same location within that set of crankshafts. This set of analyzed crankshafts may include new analyzed crankshafts with the defect detected at 810. Inspection method 900 then proceeds to 916, where control module 108 generates a notification signal based on multiple recurring defects to inspect areas of machining line 102 in an attempt to resolve potential defects causing problems along machining line 102. Inspection method 900 can then proceed as follows: Figure 9 The process can end as shown, or, if necessary, return to another appropriate step.

[0080] By leveraging artificial intelligence for end-to-end visual inspection, the inspection system and method presented in this paper minimize and potentially eliminate the need for manual inspection, thereby reducing costs and improving quality, consistency, and defect detection rates. Furthermore, compared to traditional classifier systems, the inspection system and method improve the accuracy of distinguishing between acceptable (OK) and unacceptable (NOK) parts (e.g., vehicle crankshafts). For example, the inspection system achieves an accuracy of 94.3% for unacceptable (NOK) parts, compared to 85.1% for traditional classifier systems. Moreover, the inspection system achieves an accuracy of 98.1% for acceptable (OK) parts, compared to 97.3% for traditional classifier systems.

[0081] Furthermore, continuous learning of machine learning networks (such as CNN classifiers) plays a crucial role in enhancing the accuracy of the systems and methods examined in this paper. For example, Figure 10-11 Describing as above relative to Figure 7 The aforementioned graphs 1000 and 1100 are plotted on the x-axis to represent the performance-accuracy (y-axis) curves during multi-round continuous learning. Specifically, Figure 10 The curve 1000 shows the performance accuracy of the unacceptable (NOK) part, and Figure 11 The curve 1100 shows the performance accuracy of the acceptable (OK) part.

[0082] exist Figure 10In Figure 1000 (NOK part), there is line 1002 representing a CNN classifier used with one of the inspection systems described herein, and line 1012 representing a traditional classifier system. In this example, points 1004, 1006, 1008, and 1010 on line 1002 represent the initially trained CNN classifier, the first round of continuous learning, the second round of continuous learning, and the third round of continuous learning, respectively. Similarly, points 1014, 1016, 1018, and 1020 on line 1012 represent the initially trained traditional classifier system, the first round of continuous learning, the second round of continuous learning, and the third round of continuous learning, respectively. As shown, compared to the traditional classifier system, the CNN classifier demonstrates improved performance accuracy in each round of continuous learning, while the traditional classifier system remains largely stable and significantly lower in performance accuracy than the CNN classifier.

[0083] exist Figure 11 In Figure 1100 (Acceptable (OK) parts), line 1102 represents a CNN classifier used with one of the inspection systems described herein, and line 1112 represents a traditional classifier system. In this example, points 1104, 1106, 1108, and 1110 of line 1102 represent the initially trained CNN classifier, the first round of continuous learning, the second round of continuous learning, and the third round of continuous learning, respectively. Similarly, points 1114, 1116, 1118, and 1120 of line 1112 represent the initially trained traditional classifier system, the first round of continuous learning, the second round of continuous learning, and the third round of continuous learning, respectively. As shown, the performance accuracy of both the CNN classifier and the traditional classifier system decreases during the first few rounds of continuous learning but increases back to the initial level during the third round of continuous learning (at points 1110 and 1120). This is likely due to an over-representation of the number of unacceptable (NOK) patches used in the initial rounds of continuous learning compared to the acceptable (OK) patches. For the third round of continuous learning (at points 1110 and 1120), the acceptable (OK) and unacceptable (NOK) chunks are more balanced, as in the initial training data decomposition (at points 1104 and 1114).

[0084] The foregoing description is illustrative in nature and is by no means intended to limit this disclosure, its application, or use. The broad teachings of this disclosure can be implemented in various forms. Therefore, while this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent from a study of the accompanying drawings, description, and appended claims. It should be understood that one or more steps in the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, although each embodiment is described above as having certain features, any one or more of these features described with respect to any embodiment of this disclosure can be implemented and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and arrangements between one or more embodiments remain within the scope of this disclosure.

[0085] Spatial and functional relationships between elements (e.g., between modules) are described using various terms, including “connection,” “joint,” “coupled,” “adjacent,” “next to,” “top,” “above,” “below,” and “arrangement.” Unless explicitly described as “direct,” when describing the relationship between first and second elements in the foregoing disclosure, such relationship can be a direct relationship where no other intervening elements exist between the first and second elements, but it can also be an indirect relationship where one or more intervening elements (spatially or functionally) exist between the first and second elements. As used herein, the phrase “at least one of A, B, and C” should be interpreted as using the non-exclusive logic OR to represent the logic (A OR B OR C), and should not be understood as “at least one of A, at least one of B, and at least one of C.”

[0086] In a diagram, the direction of the arrows (as indicated by the arrow) typically shows the flow of information (such as data or instructions) that the diagram is interested in. For example, when elements A and B exchange various kinds of information, but the information transmitted from element A to element B is relevant to the diagram, the arrow can point from element A to element B. This one-way arrow does not mean that no other information is transmitted from element B to element A. Furthermore, for information sent from element A to element B, element B can send a request for information or an acknowledgment of receipt of information to element A.

[0087] In this application, the terms "module" or "controller" are used in accordance with the following definitions, and may be replaced by the term "circuit". The term "module" may refer to, be a subset of, or include: application-specific integrated circuits (ASICs); digital, analog, or mixed-signal analog / digital discrete circuits; digital, analog, or mixed-signal analog / digital integrated circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processor circuits (shared, dedicated, or grouped) that execute code; memory circuits (shared, dedicated, or grouped) that store code executed by the processor circuits; other suitable hardware components that provide the aforementioned functionality; or a combination of some or all of the above.

[0088] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that connect to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module disclosed herein can be distributed among multiple modules connected via the interface circuits. For example, multiple modules can allow for load balancing. In another example, a server (also known as a remote or cloud) module may perform some functions on behalf of a client module.

[0089] As described above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" includes a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" includes a processor circuit that, in conjunction with additional processor circuitry, executes some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on a discrete die, multiple processor lines on a single die, multiple cores on a single processor line, multiple threads on a single processor circuit, or combinations thereof. The term "shared memory circuit" includes a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" includes a memory circuit that, in conjunction with additional memory, stores some or all of the code from one or more modules.

[0090] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not include transient electrical or electromagnetic signals propagating through a medium (e.g., on a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transient. Non-limiting examples of non-transient tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0091] The apparatus and methods described in this application can be partially or fully implemented by a special-purpose computer created by configuring a general-purpose computer to execute one or more specific functions embodied in a computer program. The aforementioned functional blocks, flowchart components, and other elements serve as software specifications that can be converted into computer programs through the routine work of skilled technicians or programmers.

[0092] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. A computer program may also include or depend on stored data. A computer program may include a basic input / output system (BIOS) for interacting with dedicated computer hardware, device drivers for interacting with specific devices of the dedicated computer, one or more operating systems, user applications, background services, background applications, etc.

[0093] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by the compiler from the source code; (iv) source code executed by the interpreter; (v) source code compiled and executed by a just-in-time compiler, etc. As an example only, source code can be from C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, etc. Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language 5th Edition), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK and The syntax of languages ​​such as [list of languages].

Claims

1. An inspection system for automatically detecting defects in objects, the inspection system comprising: A processing line configured to support the object; At least one camera adjacent to the processing line is configured to capture at least one image of the object and create a multi-channel image set of the image, each channel of the multi-channel image set providing a different view of the object; and A control module that communicates with the at least one camera, the control module being configured to: Receive the multi-channel image set of the image; Detecting boundaries associated with objects in the multi-channel image set; and Defects on the boundary are detected using a machine learning network based on the multi-channel image set, a first training dataset including an annotated multi-channel image set, and a second training dataset including a multi-channel image set with artificially created defects.

2. The inspection system according to claim 1, wherein the control module is configured to: A control signal is generated in response to the detection of a defect on the object; and Based on the control signal, the actuator is controlled to move the defective object to a defined position.

3. The inspection system according to claim 1, wherein: The defect detected on the object is one of multiple recurring defects associated with multiple objects; as well as The control module is configured to generate a notification signal based on the plurality of recurring defects to check the area of ​​the processing line used to create the plurality of objects.

4. The inspection system of claim 1, wherein the camera is configured to capture at least one image of the object as the object rotates.

5. The inspection system of claim 4, wherein the object is a vehicle crankshaft or camshaft.

6. The inspection system according to claim 5, wherein: The multi-channel image set includes at least a normal channel, a specular reflection channel, a diffuse channel, a glossiness channel, and a shape channel; and The control module is configured to detect the boundary in the diffuse channel associated with the vehicle crankshaft or camshaft based on changes in image brightness in the diffuse channel.

7. The inspection system according to claim 6, wherein: The vehicle crankshaft or camshaft includes at least one oil hole; and The control module is configured to detect at least one edge of the oil hole based on a combination of the diffuse channel and the gloss channel, and to detect defects associated with the edge of the oil hole.

8. The inspection system according to claim 4, wherein: Machine learning networks are convolutional neural networks; and The control module is configured to create a stack of overlapping patches from the multi-channel image set and use the convolutional neural network to detect defects on the boundary based on the stack of overlapping patches.

9. The inspection system of claim 4, wherein the second training dataset comprises at least one multi-channel image set of intentionally defective objects.

10. The inspection system of claim 4, wherein the second training dataset comprises at least one multi-channel image set created based on a transformation of a previous multi-channel image set of the defective object.

Images