Method and tool and production facility controller for optical quality control of intermediate or end products
The method employs a digital twin and generative adversarial networks to adapt real product images into a synthetic space for efficient quality control, addressing the high setup costs of conventional methods by minimizing retraining needs.
Patent Information
- Application Number
- CN202410845038.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-27
- Filing Date
- 2024-06-27
- Publication Date
- 2024-12-27
AI Technical Summary
Existing optical quality control methods for production facilities, particularly in autonomous automated systems, require significant engineering efforts to adapt to new products and changing conditions, leading to high setup costs.
A computer-implemented method using a digital twin and generative adversarial networks for domain adaptation, transforming real product images into a synthetic image space for comparison with simulated images, reducing the need for extensive retraining when product changes occur.
Enables adaptive and efficient optical quality control with reduced engineering expenditures by leveraging domain transformation, allowing seamless integration of new products and maintaining high reliability.
Smart Images
Figure CN119211706A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a computer-implemented method for optical quality control of intermediate or final products of a production facility as claimed in the preamble of patent claim 1, a computer-implemented tool for optical quality control of intermediate or final products of a production facility as claimed in the preamble of patent claim 6, and a production facility controller as claimed in the preamble of patent claim 11. Background Art
[0002] The production processes of a production facility, regardless of the area or scope in which they are carried out in the respective production facility, are never free of defects and usually have to be confirmed by optical quality control. This applies in particular to automated production processes. During optical quality control, this involves recording an image of the product or a part of the product and using this image as the basis for calculating whether all features meet the requirements. In this case, setting up such quality control is associated with engineering expenditures, which is a particular hindrance in the case of autonomous automated production facilities in the industrial sector, which should be adapted to new products with minimal expenditure.
[0003] A typical example of such a production process is the assembly of products in the industrial sector (industrial scope) with the aid of a robotic system or an automation system, in which universal automatic mobile machines are used to perform handling, maintenance and / or manufacturing tasks to achieve the assembly, and in which, for example, after the connection process, it is necessary to check whether the component to be installed has been inserted in the correct position and has been fully latched in the appropriate position.
[0004] For this purpose, it is known to provide commissioning engineers of production facilities, in particular of robotic systems, with tools for defining quality rules as simply as possible. For example, certain optical features (such as circles, lines, patterns) in an optically captured image must be at certain locations or have certain dimensions. The commissioning engineer then checks these rules using some sample images and adjusts the parameters until the desired reliability is achieved. Summary of the invention
[0005] The object on which the invention is based is to specify a computer-implemented method and tools for optical quality control of intermediate or final products of a production facility and a production facility controller, with the help of which the expenditure for quality control for establishing quality control under constantly changing conditions and relationships at the production facility can be reduced in order to achieve a corresponding adaptive matching use of the production facility.
[0006] Starting from the computer-implemented method defined in the preamble of patent claim 1 , this object is achieved by the features specified in the characterizing part of patent claim 1 .
[0007] Furthermore, starting from the computer-implemented tool defined in the preamble of patent claim 6, this object is achieved by the features specified in the characterizing part of patent claim 6.
[0008] Furthermore, starting from the production facility controller defined in the preamble of patent claim 11 , this object is achieved by the features specified in the characterizing part of patent claim 11 .
[0009] The basic idea of the invention as claimed in independent patent claims 1, 6 and 11 is to control the optical quality of intermediate or final products of a production facility, wherein a product image of the production facility (Pa) captured by an image capture device for given intrinsic and extrinsic parameters and digital twin data of a digital twin of the production facility (PA) are used, wherein the digital twin is synchronized with the production facility when the production facility is in operation, (i) a synthetic simulated image based on the digital twin data is rendered, wherein the rendered synthetic simulated image is based on the same intrinsic and extrinsic parameters as during the capture of the product image, (ii) the product image is converted from a real domain to an artificial domain by means of trained domain adaptation, and in the process, a synthetic product image is generated from the product image by means of the domain conversion parameters obtained by training, (iii) the synthetic product image is compared with the synthetic simulated image by means of a comparison operator, and (iv) a comparison result is output that qualitatively evaluates the product.
[0010] Compared to conventional rule-based approaches, the present invention pursues a fundamentally different approach that offers better scalability possibilities.
[0011] In this case, for quality control, a key aspect of the present invention is the use of a domain transformation from the real image domain to the artificial image domain. This results in simpler image comparisons for identifying image differences.
[0012] An advantageous development of the invention as claimed in claims 2 and 7 consists in that the domain adaptation is implemented as a generative adversarial network <gan>”, in which, for generating data, two competing artificial neural networks, called generator and discriminator, are used, wherein the generator generates artificial data, the discriminator checks the artificial data on the basis of real data captured, for example, with the aid of images, and wherein the two networks are logically and mathematically combined with each other in such a way that the artificial data generated by the generator appear increasingly genuine and, eventually, the discriminator can no longer distinguish the genuine data from the real data.
[0013] Furthermore, advantageously, as claimed in claims 3 and 8, the trained domain adaptation with domain conversion parameters is implemented in a two-stage training with the following steps "S1" and "S2":
[0014] "S1": generating a data set based on a plurality ("n") of image pairs formed by captured product images and associated synthetic simulated images for uniformly given intrinsic and extrinsic parameters in each case;
[0015] "S2": With the help of the generated dataset, such as "Generative Adversarial Network <gan>” The learning method trains the conversion of product image related data to simulation image related data.
[0016] Furthermore, the present invention is advantageously characterized by the fact that:
[0017] - As claimed in claims 4 and 9, said comparison operator is configured such that said comparison is performed pixel by pixel.
[0018] - As claimed in claims 5 and 10, the production facility is a robotic system or an automated system with universal automatic mobile machines for performing handling, maintenance and / or manufacturing tasks.
[0019] The following should outline potential basic scenarios for the present invention.
[0020] The starting point for the scenario outlined here is the prerequisite that a detailed digital twin of the production facility exists in operation. The digital twin includes:
[0021] The geometry and / or texture of the objects, robots and components used in the production facility
[0022] The expected positions and orientations of objects, moments and components of the production facility when in operation (i.e. at a given point in time)
[0023] This prerequisite is usually met for autonomously acting production facilities or systems; however, it can also be met or adapted for conventional production facilities or systems.
[0024] Taking this as a starting point or on the basis of this, according to the method of the invention, a non-forged image of a part or product to be checked (e.g. an intermediate product or a final product) of a production facility is recorded and this image is transformed into a simulated image space by means of domain adaptation. Furthermore, the transformed image is compared with a synthetically rendered image that has been generated by means of the digital twin. Now, the "seemingly synthetic" image, which now exists in the same image space, can be checked for differences by means of comparison operators. In this case, the output of the comparison result can be, for example, a quality result of "good" or "not good".
[0025] This quality result can then be automatically taken into account in the further course of the process; for example, defectively produced products (such as intermediate products or end products) can be automatically sorted out, or a user of the production facility can be called for assistance.
[0026] The method of the present invention has two main stages:
[0027] 1. Training for domain adaptation: This step forms the basis for converting images from the real domain to the artificial domain. This is part of domain adaptation. In this case, one possible exemplary training method for domain adaptation is the already mentioned "Generative Adversarial Networks" <gan>"technology.
[0028] The training can be characterized as follows:
[0029] a. In a first step, data sets may be recorded from the non-forged image and the associated synthetic image, each data set comprising, for optical capture, the same viewing angle and the same parameters (extrinsic and intrinsic parameters) of the image capture device (eg, camera).
[0030] - Data recording and training here can take place at the initial stage of commissioning of a production facility and, depending on the variability of the products to be produced (intermediate or final products), only need to be implemented once for the production facility. In other words, if the product changes, there is no need to restart the training.
[0031] - In this case, data recording is performed concomitantly during the testing or start-up phase. In this case, non-artifacted images are recorded and stored when quality control is required, and camera parameters (intrinsic and extrinsic parameters of the image capture device) are likewise stored. In addition, as part of the data recording for training, associated synthetic images are rendered from the same viewpoint and with the same parameters by means of the digital twin and stored in the dataset.
[0032] b. Finally, a training step ensues, in which the transformation of real data to synthetic data is learned. For example, the already mentioned "Generative Adversarial Networks <gan>”Technology applies here as well.
[0033] 2. Production facility operations:
[0034] a. Position the image capture device so that the product or component, intermediate product or final product is visible. The appropriate position can be calculated by heuristics (heuristic algorithms) while the production facility is operating. This is particularly applicable to autonomous production facilities.
[0035] b. Record product images
[0036] c. With the help of, for example, "generative adversarial networks <gan>The domain adaptation of the technology transforms the genuine product image to the image domain of the synthetic (simulated) image
[0037] d. Rendering of the composite image recorded in the digital twin from the same position and with the same parameters (intrinsic and extrinsic parameters of the image capture device).
[0038] e. Comparing the images with the aid of comparison operators. In an advantageous embodiment of the invention, neural networks can be used here, but conventional pixel-based methods are also suitable. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Refer to the single attached figure Figure 1 (which comprises two parts A and B as a whole), other advantages of the present invention will become apparent from the following description of an exemplary embodiment of the present invention.
[0040] The figure shows a production facility PA with a device for optical quality control of intermediate or final products of the production facility PA, which can be used in various technical fields (technical ranges) for product production, for example in factories, production lines and large-scale installations with the aid of suitable methods and technologies. DETAILED DESCRIPTION
[0041] In a first embodiment variant of the invention for optical quality control of intermediate or final products according to option "A", the production facility PA comprises a production facility controller PAS, a database DB, an image capture device BEE configured, for example, in the form and shape of a camera, and an output unit AEH. While the database DB and the image capture device BEE are connected to the production facility controller PAS for access thereof, the output unit AEH is connected to the production facility controller PAS for access thereof like the database DB and the image capture device BEE in the first embodiment according to option "I", or is included in the production facility controller PAS in the second embodiment according to option "II".
[0042] As an alternative to the first embodiment variant of the present invention according to option "A", in the second embodiment variant of the present invention for optical quality control of intermediate or final products according to option "B", the production facility PA, the production facility controller PAS, the database DB and the image capture device BEE are not combined "under the same coverage (the coverage of the production facility PA)", but they all act as independent units, wherein the production facility controller PAS is connected to the production facility PA, the database DB and the image capture device BEE for access, and the output unit AEH is included in the production facility PA for access by the production facility controller PAS in the first embodiment according to option "I", or is included in the production facility controller PAS in the second embodiment according to option "II".
[0043] Between these two "extreme" embodiment variants of the invention, further variants are conceivable (not explicitly shown in the drawings) in which only the database DB, the image capture device BEE or the production facility controller PAS are contained in the production facility PA, or in which the production facility PA contains in each case two of the mentioned units.
[0044] In the two illustrated embodiment variants according to options "A" and "B", the production facility controller PAS is preferably a control unit STE for a robotic system or an automation system having a universal automatic mobile machine for performing handling, maintenance and / or manufacturing tasks.
[0045] With regard to the illustrated embodiment variant according to option “B”, the production facility controller PAS can alternatively also be a conventional personal computer or controller.
[0046] The explanations given below with respect to the description of the figures apply to the two illustrated embodiment variants of the invention for optical quality control of intermediate or final products according to options “A” and “B”.
[0047] For this purpose
[0048] -The image capture device BEE captures product images PB of the production facility PA (which may relate to intermediate products or final products) for given intrinsic and extrinsic parameters of the image capture device BEE, and captures product images PB1, ..., PBn for uniformly given intrinsic and extrinsic parameters of the image capture device BEE for the purpose of generating a data set based on a plurality ("n") of image pairs.
[0049] - The database DB stores digital twin data DZD of the digital twin DZ of the production facility PA, wherein the digital twin DZ is synchronized with the production facility PA when the production facility PA is in operation.
[0050] - Use of a computer implemented tool CIW, which is preferably a computer program product CPP configured as an APP and loadable into a production facility controller PAS for optical quality control of intermediate or final products of the production facility PA.
[0051] The computer-implemented tool CIW comprises a non-volatile readable memory SP in which processor-readable control program instructions of a program module PGM for optical quality control are stored, and a processor PZ connected to the memory SP, which executes the control program instructions of the program module PGM for optical quality control of intermediate or final products of the production facility PA.
[0052] For this purpose, the computer-implemented tool CIW uses
[0053] - product images PB of the production facility PA captured by the image capture device BEE for given intrinsic and extrinsic parameters, and product images PB1, ..., PBn captured by the image capture device BEE for given intrinsic and extrinsic parameters for the purpose of generating a dataset based on a plurality ("n") of image pairs,
[0054] - Digital twin data DZD of a digital twin DZ of the production facility PA stored in a database DB, wherein the digital twin DZ is synchronized with the production facility PA when the production facility PA is in operation.
[0055] During the loading of the computer-implemented tool CIW into the production facility controller PAS, these data are requested as input data by access from the processor PZ and are then collected or supplied.
[0056] The program module PGM of the computer-implemented tool CIW is structured such that, and the processor PZ of the computer-implemented tool CIW executing the control program instructions of the program module PGM for optical quality control is configured such that, for optical quality control, the following steps are performed:
[0057] - Rendering rdn Synthetic simulation image SB based on digital twin data DZD syn Here, the rendered synthetic simulated image SB syn Based on the same intrinsic and extrinsic parameters as during product image capture.
[0058] -Convert product images PB from the real domain trf to the artificial domain with the help of the trained trn domain adaptation DA.
[0059] --Domain adaptation is preferably implemented as a generative adversarial network based on <gan>”, in which, for generating data, two competing artificial neural networks, called generator and discriminator, are used, wherein the generator generates artificial data, the discriminator checks the artificial data on the basis of real data captured, for example, with the aid of images, and wherein the two networks are logically and mathematically combined with each other in such a way that the artificial data generated by the generator appear increasingly genuine and, eventually, the discriminator can no longer distinguish the genuine data from the real data.
[0060] --Domain adaptation DA includes domain transformation parameters DTP obtained by training trn. The trained trn domain adaptation DA with domain transformation parameters DTP is preferably implemented with a two-stage training trn with the following steps "S1" and "S2":
[0061] "S1": for uniformly given intrinsic and extrinsic parameters based on captured product images PB1, ..., PBn and associated synthetic simulated image SB1 syn , …, SBn syn A plurality of ("n") image pairs are formed to generate a data set;
[0062] "S2": With the help of the generated dataset, such as "Generative Adversarial Network <gan>” The learning method trains the conversion of product image related data to simulation image related data.
[0063] --After training, a synthetic product image PB is generated from the product image PB according to the domain transformation parameters DTP using the trained domain adaptation DA syn Therefore, in the artificial image space, a synthetic simulated image SB appears. syn and synthetic product images PB syn The images were paired for comparison purposes.
[0064] - With the help of comparison operator VO, the composite product image PB syn With synthetic simulation image SB syn The comparison vgl is performed. The comparison operator VO is, for example, preferably configured such that the comparison is performed pixel by pixel.
[0065] - Output asg A comparison result VGE qualitatively evaluating the product is output asg. This comparison result VGE is output asg, for example, preferably via an output unit AEH of the production facility PA or a production facility controller PAS of the production facility PA.< / gan> < / gan> < / gan> < / gan> < / gan> < / gan> < / gan>
Claims
1. A computer-implemented method for optical quality control of an intermediate or final product of a production facility, wherein: Regarding the quality control - using product images (PB) of the production facility (PA) captured by an image capture device (BEE) for given intrinsic and extrinsic parameters, - using digital twin data (DZD) of a digital twin (DZ) of said production facility (PA), wherein said digital twin (DZ) is synchronized with said production facility (PA) when said production facility (PA) is in operation, It is characterized in that a) rendering (rdn) a synthetic simulation image (SBsyn) based on the digital twin data (DZD), wherein the rendered synthetic simulation image (SB syn ) based on the same intrinsic and extrinsic parameters as during product image capture, b) converting (trf) the product image (PB) from a real domain to an artificial domain by means of a trained (trn) domain adaptation (DA), the trained (trn) domain adaptation (DA) comprising domain transformation parameters (DTP) obtained by training (trn), and generating a synthetic product image (PB) from the product image (PB) according to the domain transformation parameters (DTP) by means of the trained (trn) domain adaptation (DA); syn ), whereby the synthetic simulated image (SB syn ) and the synthetic product image (PB syn ) are used for comparison purposes. c) comparing the synthetic product image (PB) with the aid of a comparison operator (VO) syn ) and the synthetic simulated image (SB syn ) for comparison (vgl), d) outputting (asg) a comparison result (VGE) which qualitatively evaluates the product, in particular via an output unit (AEH) of the production facility (PA) or a production facility controller (PAS) of the production facility (PA).
2. The computer-implemented method of claim 1, wherein: The domain adaptation (DA) is implemented as a generative adversarial network based on <gan> ”, in which, for generating data, two competing artificial neural networks, called generator and discriminator, are used, wherein the generator generates artificial data, the discriminator checks the artificial data on the basis of real data captured, for example, with the aid of images, and wherein the two networks are logically and mathematically combined with each other in such a way that the artificial data generated by the generator appear increasingly genuine and, eventually, the discriminator can no longer distinguish the genuine data from the real data.< / gan> 3. The computer-implemented method according to claim 1 or 2, characterized in that The trained (trn) domain adaptation (DA) with the domain transformation parameters (DTP) is implemented with a two-stage training (trn) with the following steps “S1” and “S2”: "S1": for uniformly given intrinsic and extrinsic parameters based on captured product images (PB1, ..., PBn) and associated synthetic simulated images (SB1 syn , …, SBn syn ) formed by multiple image pairs, i.e., "n" image pairs, to generate a data set; "S2": With the help of the generated dataset, such as "Generative Adversarial Network <gan> ” The learning method trains the conversion of product image related data to simulation image related data.< / gan> 4. The computer-implemented method according to claim 1 or 2, characterized in that: The comparison operator (VO) is configured such that the comparison is performed pixel by pixel.
5. The computer-implemented method according to any one of claims 1 to 4, characterized in that The Production Facilities (PA) are robotic or automated systems with general-purpose autonomous mobile machines for performing handling, maintenance and / or manufacturing tasks.
6. A computer-implemented tool (CIW), in particular a computer program product (CPP) configured as an APP, for optical quality control of intermediate or final products of a production facility, wherein: Regarding the quality control - using product images (PB) of the production facility (PA) captured by an image capture device (BEE) for given intrinsic and extrinsic parameters, - using digital twin data (DZD) of a digital twin (DZ) of said production facility (PA), wherein said digital twin (DZ) is synchronized with said production facility (PA) when said production facility (PA) is in operation, It is characterized in that A non-volatile readable memory (SP) in which processor-readable control program instructions of a program module (PGM) for optical quality control are stored, and a processor (PZ) connected to said memory (SP), said processor executing said control program instructions of said program module (PGM) for optical quality control of intermediate products or final products of a production facility, wherein said program module (PGM) is constructed such that, and said processor (PZ) executing said control program instructions of said program module (PGM) for optical quality control is configured such that a) rendering (rdn) a synthetic simulated image (SB) based on the digital twin data (DZD) syn ), where the rendered synthetic simulated image (SB syn ) based on the same intrinsic and extrinsic parameters as during product image capture, b) converting (trf) the product image (PB) from a real domain to an artificial domain by means of a trained (trn) domain adaptation (DA), the trained (trn) domain adaptation (DA) comprising domain transformation parameters (DTP) obtained by training (trn), and generating a synthetic product image (PB) from the product image (PB) according to the domain transformation parameters (DTP) by means of the trained (trn) domain adaptation (DA); syn ), whereby the synthetic simulated image (SB syn ) and the synthetic product image (PB syn ) for comparison purposes, c) comparing the synthetic product image (PB) with the aid of a comparison operator (VO) syn ) and the synthetic simulated image (SB syn ) for comparison (vgl), d) outputting (asg) a comparison result (VGE) which qualitatively evaluates the product, in particular via an output unit (AEH) of the production facility (PA) or a production facility controller (PAS) of the production facility (PA).
7. The computer-implemented tool (CIW) according to claim 6, characterized in that The processor (PZ) and the program module (PGM) for optical quality control are configured such that The domain adaptation is implemented as a <gan> ”, in which, for generating data, two competing artificial neural networks, called generator and discriminator, are used, wherein the generator generates artificial data, the discriminator checks the artificial data on the basis of real data captured, for example, with the aid of images, and wherein the two networks are logically and mathematically combined with each other in such a way that the artificial data generated by the generator appear increasingly genuine and, eventually, the discriminator can no longer distinguish the genuine data from the real data.< / gan> 8. The computer-implemented tool (CIW) according to claim 6 or 7, characterized in that The processor (PZ) and the program module (PGM) for optical quality control are configured such that The trained (trn) domain adaptation (DA) with the domain transformation parameters (DTP) is implemented with a two-stage training (trn) with the following steps “S1” and “S2”: "S1": for uniformly given intrinsic and extrinsic parameters based on captured product images (PB1, ..., PBn) and associated synthetic simulated images (SB1 syn , …, SBn syn ) formed by multiple image pairs, i.e., "n" image pairs, to generate a data set; "S2": With the help of the generated dataset, such as "Generative Adversarial Network <gan> ” The learning method trains the conversion of product image related data to simulation image related data.< / gan> 9. Computer-implemented tool (CIW) according to any one of claims 6 to 8, characterized in that The processor (PZ) and the program module (PGM) for optical quality control and the comparison operator (VO) are configured such that the comparison is performed pixel by pixel.
10. Computer-implemented tool (CIW) according to any one of claims 6 to 9, characterized in that The Production Facilities (PA) are robotic or automated systems with general-purpose autonomous mobile machines for performing handling, maintenance and / or manufacturing tasks.
11. A production facility controller (PAS) for optical quality control of an intermediate product or a final product of a production facility (PA), wherein - an image capture device (BEE) capturing images (PB) of products of said production facility (PA) for given intrinsic and extrinsic parameters, being part of said production facility (PA) and therefore connected to said production facility controller (PAS), or being assigned to said production facility (PA) and therefore connected to said production facility controller (PAS), a database (DB) storing digital twin data (DZD) of a digital twin (DZ) of the production facility (PA) is assigned to the production facility (PA) and is therefore connected to the production facility controller (PAS), wherein the digital twin (DZ) is synchronized with the production facility (PA) when the production facility (PA) is in operation Features A computer-implemented tool (CIW) according to any one of claims 6 to 10, which can be loaded into the production facility controller (PAS) in order to implement the method according to any one of claims 1 to 5.
12. The production facility controller (PAS) according to claim 11, characterized in that Control unit (STE) for a robotic system or automation system having a universal autonomous mobile machine for carrying out handling, maintenance and / or manufacturing tasks.