Method and device for testing a component

The automatic inspection method employs a machine learning-based feature encoder and classification unit to classify components, adapting to feedback and process changes, enhancing the reliability and precision of component evaluation beyond manual and camera-based limitations.

WO2024245678A1PCT designated stage expired Publication Date: 2024-12-05BAYERISCHE MOTOREN WERKE AG
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/062039
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-02
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Manual final inspection of components is costly and prone to errors, and camera-based inspections are sensitive to changes in the manufacturing process, necessitating a robust and reliable automatic inspection method.

Method used

A device and method using a feature encoder and classification unit, trained with machine learning, to determine a feature tensor from sensor data, such as camera images, and adapt based on feedback to classify components as OK or not OK, incorporating a scattering tensor for error analysis and adjustment of reference feature tensors.

Benefits of technology

This approach enables a reliable and robust automatic inspection, reducing errors and adapting to manufacturing process changes, thereby improving the quality and precision of component classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024062039_05122024_PF_FP_ABST
    Figure EP2024062039_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a device (101) for testing a first component (110) from a set of components (110) identical in design. The device (101) is designed, on the basis of sensor data (103) in relation to the first component (110) and using a feature encoder (201), to determine a first feature tensor (205) for the first component (110); and, on the basis of the first feature tensor (205) and by means of a classification unit (220), to determine a classification result (104) indicating whether the first component (110) is in order, iO, or exhibits a deviation. The device (101) is also designed, when the classification result (104) indicates that the first component (110) exhibits a deviation, to acquire feedback information (105) as to whether the first component (110) has been classified by a following test entity as iO or as not in order, niO, and to adjust the classification unit (220) according to the feedback information (105).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method and device for checking a component

[0002] The invention relates to a method and a corresponding device designed to automatically check a component, in particular within the framework of quality control.

[0003] During the production of (identical) components, a final quality control check is typically carried out. This involves manually inspecting the individual components by a final inspector. As part of the inspection, the final inspector can check the surface of the individual components to identify manufacturing defects (such as scratches, holes, dents, etc.). A component can be classified by the final inspector as "OK" or "not OK". A not OK component can be reworked to eliminate detected manufacturing defects, or the not OK component can be rejected.

[0004] Manual final inspection is typically associated with relatively high costs and is relatively error-prone. On the other hand, the quality of camera-based inspection of components is often compromised by changes in the manufacturing process (such as a change of machine, a change of tool, a process change, etc.).

[0005] This document deals with the technical task of achieving a particularly robust and reliable automatic final inspection of components.

[0006] The problem is solved by each of the independent claims. Advantageous embodiments are described, among other things, in the dependent claims. It should be noted that additional features of a patent claim dependent on an independent patent claim can form a separate invention, independent of the combination of all features of the independent patent claim, without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim, which invention can be made the subject of an independent claim, a divisional application, or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

[0007] According to one aspect, a device for checking or inspecting a first component from a set (in particular from a sequence) of identical components is described. The device can be configured to check the surface of the first component, e.g., to detect one or more impairments (such as a dent, a crack, a scratch, etc.) on the surface of the first component. The individual components can be, for example, (metal) housings. The individual components can each be components of motor vehicles.

[0008] The device is configured to determine a first feature tensor for the first component based on sensor data relating to the first component and using a feature encoder. The sensor data may have been acquired by one or more sensors, e.g., cameras. The sensor data may include one or more camera images of the first component (from one or more different perspectives).

[0009] The feature encoder can have been trained in advance using a machine learning algorithm (e.g., together with a corresponding feature decoder). The first feature tensor can comprise feature values ​​for a specific number of features (e.g., between 10 and 100 features). Thus, the first feature tensor can represent a compact representation of the quality, in particular the surface quality, of the first component. The device is configured to determine, based on the first feature tensor and a (preliminarily determined, in particular trained) classification unit, a classification result that indicates whether the first component is OK or exhibits a deviation (and thus cannot be clearly classified as OK or not OK).

[0010] The classification unit can comprise a set of reference feature tensors, where the set of reference feature tensors can describe the components that are to be classified as OK by the classification unit. The set of reference feature tensors can, for example, comprise all feature tensors for OK components.

[0011] The classification unit can be configured to determine the classification result for a component based on the set of reference feature tensors. For example, the classification unit can be configured to classify the first component as OK if the first feature tensor is part of the set of reference feature tensors. On the other hand, a deviation can be detected for the first component if the first feature tensor is not part of the set of reference feature tensors.

[0012] The device may further be configured, if the classification result indicates that the first component has a deviation, to determine feedback information as to whether the first component was classified as OK or as NOK by a subsequent (manual and / or automatic) inspection instance (e.g. by a final inspector and / or by a camera-based inspection).

[0013] Furthermore, the device can be configured to adapt the classification unit depending on the feedback information. In particular, the set of reference feature tensors can be adapted depending on the feedback information. Preferably, the determined first feature tensor can be included (or not included) in the set of reference feature tensors depending on the feedback information. By adapting the classification unit based on feedback information, a particularly reliable and robust inspection of (identical) components can be achieved.

[0014] The device can be configured to determine a feature tensor for each individual component from the set of identical components based on the sensor data relating to the respective component and using the feature encoder. It can then determine the number L of components for which the first feature tensor was determined as the feature tensor. The number L can be adjusted sequentially as the individual components are checked.

[0015] The classification unit can be adapted in a particularly robust and precise manner depending on the determined number L of components.

[0016] The device can be configured, in particular, to determine feedback information for all L components for which the first feature tensor was determined as a feature tensor, as to whether the respective component was classified as OK or as NOK by the verification instance. The classification unit can be adapted in a particularly robust and precise manner depending on the feedback information for all L components. In this case, the classification unit can be adapted; in particular, the first feature tensor can be included in the set of reference feature tensors of the classification unit if (in particular only if)

[0017] • the determined number L of components is equal to or greater than a threshold value (e.g. between 50 and 150, about 100); and

[0018] • the feedback information for all L components indicates that the components have been classified as OK by the review body.

[0019] This allows the classification unit to be adapted in a particularly robust and precise manner.

[0020] The device can be configured to determine reconstructed sensor data using the feature decoder (which, for example, was trained in advance using a machine learning algorithm, together with the feature encoder) and based on the first feature tensor. Furthermore, a sensor data error can be determined based on the sensor data relating to the first component and the reconstructed sensor data, in particular as the difference between the sensor data and the reconstructed sensor data. The classification result for the first component can then be determined in a particularly reliable and robust manner based on the sensor data error.

[0021] In particular, a scatter tensor can be determined using a scatter encoder (e.g., one that has been trained in advance using a machine learning algorithm, possibly together with a scatter decoder) and the sensor data error. This scatter tensor provides information regarding the mean and / or standard deviation of the sensor data error. The classification result for the first component can then be determined in a particularly reliable and robust manner using the classification unit based on the scatter tensor.

[0022] According to a further aspect, a method for inspecting a first component from a set of identical components is described. The component can be, for example, a housing, in particular a crankcase of an engine.

[0023] The method comprises determining, based on sensor data relating to the first component and using a feature encoder, a first feature tensor for the first component. The sensor data can, in particular, comprise one or more camera images of the surface of the first component from one or more different perspectives. The first feature tensor can have a number of tensor entries that is a factor of 10 or more, or 100 or more, smaller than the number of measured values ​​(in particular pixels) of the sensor data.

[0024] Furthermore, the method comprises determining, based on the first feature tensor and using a classification unit, a classification result that indicates whether the first component is OK or has a deviation (and therefore it is not clearly recognized whether the first component is OK or NOK). If the classification result indicates that the first component has a deviation, the method further comprises determining feedback information regarding whether the first component was classified as OK or not OK by a subsequent inspection instance, and adapting the classification unit depending on the feedback information.

[0025] According to another aspect, a software (SW) program is described. The SW program can be configured to be executed on a processor and thereby to carry out the method described in this document.

[0026] According to a further aspect, a storage medium is described. The storage medium can comprise a software program configured to be executed on a processor and thereby to carry out the method described in this document.

[0027] It should be noted that the methods, devices, and systems described in this document can be used both alone and in combination with other methods, devices, and systems described in this document. Furthermore, any aspects of the methods, devices, and systems described in this document can be combined in a variety of ways. In particular, the features of the claims can be combined in a variety of ways. Furthermore, features listed in parentheses are to be understood as optional features.

[0028] The invention will be described in more detail below using exemplary embodiments.

[0029] Figure 1 shows an exemplary system for checking components;

[0030] Figure 2a shows an exemplary autoencoder for determining a feature tensor, in particular a feature vector or a feature matrix, for a component;

[0031] Figure 2b shows an exemplary autoencoder for determining scatter information, in particular a scatter tensor; Figure 2c shows an exemplary classification unit for detecting a manufacturing deviation; and

[0032] Figure 3 is a flowchart of an exemplary method for detecting a manufacturing deviation.

[0033] As stated at the beginning, this document deals with the reliable and robust, automatic inspection of a component within a manufacturing process. In this context, Fig. 1 shows an exemplary system 100 for inspecting components 110. The individual components 110 can, for example, be conveyed past a measuring unit 102 on a conveyor belt. The measuring unit 102 can be configured to acquire sensor data 103 relating to a component 110. The sensor data 103 can relate to the surface of the component 110. The sensor data 103 can include, for example,

[0034] • Camera images from one or more cameras; and / or

[0035] • a point cloud of at least one lidar and / or radar sensor.

[0036] The sensor data 103 relating to a component 110, in particular relating to the surface of a component 110, can be evaluated by an (evaluation) device 101. The device 101 can be configured to determine, based on the sensor data 103, whether the component 110 is OK or whether the component 110 exhibits a deviation. A classification result 104 can thus be provided as to whether the component 110 is OK or exhibits a deviation. If it is detected that the component 110 exhibits a deviation, the component 110 can be subjected to further (possibly manual or automatic) inspection. During the further inspection, a decision can be made as to whether the component 110 is OK after all, or whether the component 110 is not OK and must be reworked or discarded.

[0037] The device 101 can be configured to determine, in particular to receive, feedback information 105 relating to a component 110 for which a deviation was detected. The feedback information 105 can be provided for each individual component 110 (by the entity that performed the further check) for which a deviation was detected and / or which was not classified as OK. The feedback information 105 can be provided explicitly or implicitly. For example, an explicit feedback for a component 110 can indicate that the component 110 was not OK. On the other hand, a lack of feedback for a component 110 can implicitly indicate that the component 110 was OK.

[0038] The feedback information 105 can be used to adapt the classification of components 110 in order to increase the quality of the classification and thus the quality of the component inspection.

[0039] The device 101 can be configured to use a feature encoder 201 to determine a feature tensor 205 for a component 110 based on the sensor data 103. The feature tensor 205 can specify a feature value for a plurality of different features. The feature tensor 205 can be represented, for example, in the form of a vector or in the form of a matrix, wherein the individual vector or matrix entries are each associated with a feature and specify the feature value for a specific feature. A feature matrix can, for example, have N x M matrix entries, for example, with N and M each equal to or greater than 10.

[0040] The feature encoder 201 may have been trained in advance (using a machine learning algorithm). The feature encoder 201 may have been trained together with a corresponding feature decoder 202, wherein the feature decoder 202 is configured to determine corresponding reconstructed sensor data 203 based on the feature tensor 205, which was determined based on the sensor data 103 with respect to the component 110. The feature encoder 201 and the feature decoder 202 may have been trained (based on training data) with the goal of reducing, in particular minimizing, the error, in particular the difference, between the sensor data 103 and the reconstructed sensor data 203.

[0041] In one example, the sensor data 103 comprises, in particular, a camera image, and the reconstructed sensor data 203 comprises, in particular, a reconstructed camera image. The error between the sensor data 103 and the reconstructed sensor data 203 comprises, in particular, in this case, an error image in which the individual pixels correspond to the differences between the corresponding pixels from the camera image and the reconstructed camera image.

[0042] By combining training of an encoder 201 and a decoder 202 (i.e., a so-called autoencoder), it is possible to represent the sensor data 103 for a component 110 in a compact and representative manner using a corresponding feature tensor 205. The feature tensor 205, in particular the feature matrix, has a number of features that is smaller than the number of measurement points, such as pixels, of the sensor data 103 (e.g., smaller by a factor of 10 or more).

[0043] The feature encoder 201 can, for example, be configured to determine a feature tensor 205 from a total set of different possible feature tensors 205 based on the sensor data 103. The total set of possible feature tensors 205 can, for example, comprise 1000 or fewer different possible feature tensors 205.

[0044] The feature tensor 205 can be used to classify whether the component 110 is OK or has a deviation. In particular, the device 101 can comprise a classification unit 220 (see Fig. 2c) that is configured to determine the classification result 104 for a component 110 based on the feature tensor 205 for the component 110. The classification unit 220 can, for example, comprise a set 222 of reference feature tensors that describe the components 110 that are to be classified as OK. The set 222 of reference feature tensors can, for example, comprise all feature tensors 205 that are to be classified as OK. A component 110 with a feature tensor 205 that is part of the set 222 of reference feature tensors can thus be classified as OK. On the other hand, a component 110 with a feature tensor 205 that is not part of the set 222 of reference feature tensors may be classified as a component 110 with a deviation.The device 101 may be configured to update the set 222 of reference feature tensors based on the feedback information 105 in order to increase the quality of the classification results 104 over time.

[0045] As explained above, changes in the manufacturing process of the individual components 110, such as changing a component (e.g., the milling head) of a machine, can lead to changes in the (optical) properties of the surface of the manufactured components 110 (such as the shape and / or depth of milling grooves). Such changes in the manufacturing process can lead to significant changes in the feature tensors 205 of the components 110, even though the components 110 have no manufacturing defects and are OK.

[0046] The device 101 can be configured to analyze the sensor data error 213 between the sensor data 103 and the reconstructed sensor data 203 for a component 110 in order to determine whether the component 110 is OK or actually exhibits a deviation. For this purpose, a further encoder 211 can be used (see Fig. 2b), which can be referred to as a scatter encoder 211. The scatter encoder 211 can be configured to determine statistical scatter data, in particular a scatter tensor, 215 based on the error 213, wherein the statistical scatter data or the scatter tensor 215 can comprise, for example, the mean value, in particular a mean value image or a mean value matrix, and / or the standard deviation, in particular a standard deviation image or a standard deviation matrix.

[0047] The scatter encoder 211 may have been trained in conjunction with a scatter decoder 212 (using a machine learning algorithm), wherein the scatter decoder 212 is configured to determine a reconstructed error 214 based on the statistical scatter data 215. The training may have been carried out with the aim of reducing, in particular minimizing, the difference between the reconstructed error 214 and the error 213. The classification unit 220 may be configured to take the statistical scatter data or the scatter tensor 215 into account when determining the classification result 104. Thus, variations in the components 110 resulting from changes in the manufacturing process can be reliably taken into account in order to be able to determine in a particularly reliable and robust manner whether a component 110 is OK or exhibits a deviation.

[0048] The classification unit 220 may include a storage unit in which the one or more feature tensors 205 of the components 110 for which a deviation was detected are recorded. Furthermore, a counter may be stored for each individual feature tensor 205, indicating the number of components 110 that have exhibited the respective feature tensor 205.

[0049] Thus, a list 221 of different feature tensors 205 can be created for the components 110 for which a deviation was detected (and which were thus not classified as OK), wherein the number of components 110 that exhibited the respective feature tensor 205 is specified for each feature tensor 205. The list 221 of feature tensors 205 for components 110 with a deviation can be considered a "short-term memory" in which short-term changes to the components 110 are detected and stored.

[0050] Furthermore, for the individual components 110 for which a deviation was detected, it can be checked on the basis of the feedback information 105 whether the individual components 110 were classified as OK or as NOK by a subsequent checking instance.

[0051] A feature tensor 205 from the list 221 that has occurred with a relatively high frequency and / or with a relatively high number without the corresponding components 110 being classified as not OK by the subsequent verification instance can be included in the set 222 of reference feature tensors. For example, for a feature tensor 205 from the list 221 of feature tensors 205 for components 110 with a deviation, it can be recognized that the feature tensor 205 has already occurred L times (e.g., L between 10 and 150) without the corresponding component 110 being classified as not OK by the subsequent verification instance. This feature tensor 205 can be included in the set 222 of reference feature tensors because it can be assumed that this feature tensor 205 belongs to an OK component 110. In this way, the reliability of the detection of OK components 110 can be further increased.

[0052] The set 222 of reference feature tensors can be regarded as a “long-term memory” in which the feature tensors 205 for iO components 110 are stored.

[0053] Fig. 3 shows a flowchart of a (possibly computer-implemented) method 300 for inspecting a first component 110 from a set, in particular from a sequence, of identical components 110 (which are provided sequentially, for example, by a production line). The method 300 can be directed to inspecting the surface of the first component 100.

[0054] The method 300 includes determining 301, based on sensor data 103 relating to the first component 110 and using a feature encoder 201, a first feature tensor 205 for the first component 110. The feature tensor 205 may include feature values ​​for a specific number of features (e.g., for 10 to 100 features). The feature tensor 205 may be a feature matrix.

[0055] The method 300 further includes determining 302, based on the first feature tensor 205 and using a classification unit 220, a classification result 104 that indicates whether the first component 110 is OK, OK, or has a deviation. The classification unit 220 can be configured to determine the classification result 104 using a (preliminarily determined) set 222 of reference feature tensors, e.g., using a support vector machine (SVM).

[0056] Furthermore, if the classification result 104 indicates that the first component 110 has a deviation, the method 300 includes determining 303 feedback information 105 as to whether the first component 110 was classified as OK or as not OK (NOK) by a subsequent (manual and / or automatic) inspection instance. The method 300 further includes adapting 304 the classification unit 220 depending on the feedback information 105.

[0057] The measures described in this document can efficiently increase the reliability and robustness of an automatic component inspection 110.

[0058] The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and figures are intended only to illustrate the principle of the proposed methods, devices, and systems by way of example.

Claims

Claims 1) Device (101) for checking a first component (110) from a set of identical components (110); wherein the device (101) is configured - to determine a first feature tensor (205) for the first component (110) on the basis of sensor data (103) relating to the first component (110) and using a feature encoder (201); - on the basis of the first feature tensor (205) and using a classification unit (220), to determine a classification result (104) which indicates whether the first component (110) is OK, OK, or has a deviation; - if the classification result (104) indicates that the first component (110) has a deviation, to determine feedback information (105) as to whether the first component (110) was classified as OK or as not OK, not OK, by a subsequent verification instance; and - adapt the classification unit (220) depending on the feedback information (105). 2) Device (101) according to claim 1, wherein - the classification unit (220) comprises a set (222) of reference feature tensors; - the classification unit (220) is configured to determine the classification result (104) for a component (110) on the basis of the set (222) of reference feature tensors; and - the device (101) is configured to adapt the set (222) of reference feature tensors as a function of the feedback information (105), in particular to include the determined first feature tensor (205) in the set (222) of reference feature tensors as a function of the feedback information (105). 3) Device (101) according to claim 2, wherein the set (222) of reference feature tensors describes components (110) that are to be classified as OK by the classification unit (220). 4) Device (101) according to one of the preceding claims, wherein the device (101) is arranged - to determine a feature tensor (205) for each individual component (110) from the set of identical components (110), on the basis of sensor data (103) relating to the respective component (110) and using the feature encoder (201); - to determine a number L of components (110) for which the first feature tensor (205) was determined as the feature tensor (205); and - to adapt the classification unit (220) depending on the determined number L of components (110). 5) Device (101) according to claim 4, wherein the device (101) is arranged - for all L components (110) for which the first feature tensor (205) was determined as the feature tensor (205), to determine feedback information (105) as to whether the respective component (110) was classified as OK or as NOK by the verification authority; and - to adapt the classification unit (220) depending on the feedback information (105) for all L components (110). 6) Device (101) according to claim 5, wherein the device (101) is configured to adapt the classification unit (220), in particular to include the first feature tensor (205) in a set (222) of reference feature tensors of the classification unit (220), if - the determined number L of components (110) is equal to or greater than a threshold value; and - the feedback information (105) for all L components (110) indicates that the components (110) have been classified as OK by the verification authority became. 7) Device (101) according to one of the preceding claims, wherein the device (101) is arranged - to determine reconstructed sensor data (203) using a feature decoder (202) and on the basis of the first feature tensor (205); - to determine a sensor data error (213) on the basis of the sensor data (103) relating to the first component (110) and on the basis of the reconstructed sensor data (203), in particular as a difference between the sensor data (203) and the reconstructed sensor data (203); and - to determine the classification result (104) for the first component (110) also on the basis of the sensor data error (213). 8) Device (101) according to claim 7, wherein the device (101) is arranged - using a scattering encoder (211) and on the basis of the sensor data error (213), to determine a scattering tensor (215) which in particular provides information relating to a mean value and / or a standard deviation of the sensor data error (213); and - to determine the classification result (104) for the first component (110) using the classification unit (220) on the basis of the scatter tensor (215). 9) Device (101) according to one of the preceding claims, wherein the feature encoder (201) was trained using a machine learning algorithm. 10) Method (300) for checking a first component (110) from a set of identical components (110); the method (300) comprising - determining (301), on the basis of sensor data (103) relating to the first component (110) and using a feature encoder (201), a first feature tensor (205) for the first component (110); - determining (302), on the basis of the first feature tensor (205) and using a classification unit (220), a classification result (104) which indicates whether the first component (110) is OK, OK, or has a deviation; - if the classification result (104) indicates that the first component (110) has a deviation, determining (303) feedback information (105) as to whether the first component (110) was classified as OK or as not OK, nO, by a subsequent inspection authority; and - adapting (304) the classification unit (220) depending on the feedback information (105).

Citation Information

Patent Citations

  • Method and system for defect inspection based on deep learning

    US20220189002A1

  • Automated monitoring using image analysis

    US20230092247A1