Computer-implemented method and tool for optical quality control of intermediate or final products of a production facility and production facility control device

Abstract

To provide a computer-implemented method and tool for reducing the cost of quality control in production installations.SOLUTION: Optical quality control of intermediate or end products in a production installation uses a product image of the production installation captured by an image capture device for given intrinsic and extrinsic parameters, and digital twin data of a digital twin of the production installation, where the digital twin is synchronized with the production installation at the time of operation of the production installation, so as to: render a synthetic simulation image based on the digital twin data, where the synthetic simulation image is rendered on the basis of the same intrinsic and extrinsic parameters as during product image capture; transfer the product image from a real domain into an artificial domain by means of a trained domain adaptation, where a synthetic product image is generated from the product image on the basis of domain transfer parameters obtained by the training; compare the synthetic product image with the synthetic simulation image by means of a comparison operator; and qualitatively assess the product.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method for optical quality control of intermediate or final products of a production facility according to the generic concept of claim 1, a computer-implemented tool for optical quality control of intermediate or final products of a production facility according to the generic concept of claim 6, and a production facility control device according to the generic concept of claim 11. [Background technology]

[0002] Regardless of the field or domain in which the production process of a production facility is carried out, the production process is never free from defects and often needs to be ensured by optical quality control. This is particularly true for automated production processes. Here, optical quality control involves acquiring an image of a product or a part of a product and calculating, based on this image, whether all characteristics meet the requirements. Furthermore, setting up quality control is associated with engineering costs that must be adjusted to new products at minimal cost, which is particularly disadvantageous in autonomous, automated production facilities in the industrial field.

[0003] A common example of such a production process is the assembly of products in the industrial domain using robotic or automation systems, where versatile mobile machines are used to perform processing, service and / or manufacturing tasks for the assembly, and after a joining process, for example, it is necessary to check whether the parts to be installed are in the correct position and fully engaged.

[0004] For this purpose, it is known to provide tools to the commissioning engineers of production facilities, in particular robotic systems, to define quality rules as simply as possible. For example, certain optical features in optically acquired images, such as circles, lines, patterns, etc., must be in a specific position or have specific dimensions. The commissioning engineer then checks these rules using several example images and adjusts the parameters until the desired reliability is achieved. Summary of the Invention [Problem to be solved by the invention]

[0005] The problem underlying the present invention is to identify a computer-implemented method and tool, as well as a production facility control device, for optical quality control of intermediate or final products of a production facility, which can reduce the cost of setting up quality control in the face of constantly changing conditions and relationships in the production facility, so that quality control is performed by adaptively adjusting the use of the production facility accordingly. [Means for solving the problem]

[0006] The problem is solved by the computer-implemented method defined in the generic part of claim 1 by the features set forth in the characterizing part of claim 1.

[0007] The problem is also solved by the features of claim 6 based on the computer-implemented tool defined in the generic concept of claim 6.

[0008] Furthermore, this problem is solved by the features of claim 11 based on the production equipment control device defined in the generic concept of claim 11.

[0009] The idea underlying the inventions described in independent claims 1, 6 and 11 is to use product images of a production facility acquired by an image acquisition device for given internal and external parameters for optical quality control of intermediate or final products of the production facility, and to use digital twin data of the digital twin of the production facility (PA), the digital twin being synchronized with the production facility when the production facility is running, (i) to render a synthetic simulation image based on the digital twin data, the rendered synthetic simulation image being based on the same internal and external parameters as when the product image was acquired, (ii) to transfer the product image from a real domain to an artificial domain using trained domain adaptation, in which a synthetic product image is generated from the product image based on the domain transfer parameters acquired by training, (iii) to compare the synthetic product image with the synthetic simulation image using a comparison operator, and (iv) to output a comparison result for qualitatively evaluating the product.

[0010] Compared to traditional rule-based approaches, the present invention pursues a fundamentally different approach that offers better scaling potential.

[0011] Here, one essential aspect of the present invention is to use domain transformation from real image domain to artificial image domain for quality control, which achieves easier image comparison to detect image differences.

[0012] One advantageous development of the invention as defined in claims 2 and 7 is that Domain Adaptation (DA) is implemented as a "machine learning" model according to the principles of "Generative Adversarial Networks (GAN)", and for data generation two competing artificial neural networks are used, called the generative network and the discriminative network, the generative network generating artificial data which the discriminative network checks against real data, e.g. obtained from images, the two networks being logically and mathematically linked to each other in such a way that the artificial data generated by the generative network looks more and more realistic until, eventually, the discriminative network can no longer distinguish realistic data from real data.

[0013] Also advantageously, according to claims 3 and 8, the trained domain adaptation with domain transfer parameters comprises the following steps "S1" and "S2": Step “S1”: generating a dataset based on a number “n” of image pairs consisting of acquired product images and associated synthetic simulation images, each for uniformly given internal and external parameters; Step “S2” is carried out in a two-stage training step, which includes training the transfer of product image-related data to simulation image-related data using a learning method, for example, a “generative adversarial network (GAN),” based on the generated dataset.

[0014] The present invention also advantageously comprises: According to claims 4 and 9, the comparison operator is configured such that the comparison is performed pixel by pixel, According to claims 5 and 10, the production facility is characterized in that it is a robotic or automation system with universally usable mobile machines for carrying out processing, service and / or manufacturing tasks.

[0015] In the following, the principle scenario underlying the present invention is explained.

[0016] The starting point for the scenario described here is the prerequisite that the details of the digital twin of the production facility are available at runtime. This digital twin: The geometry and / or texture of the objects, robots and components of the production equipment used Runtime, i.e., having the expected positions and poses of objects, robots and components of production equipment at a given time.

[0017] This prerequisite is generally met for autonomously operating production facilities or systems, but can also be met or retrofitted for conventional production facilities or systems.

[0018] According to this or based on this, the inventive approach involves taking a realistic image of the part or product to be checked of the production equipment, e.g., an intermediate or final product, and transferring this image to a simulated image space using domain adaptation. The transferred image is then compared with a synthetically rendered image, generated using a digital twin. Now, a "synthetic-looking" image available in the same image space can be checked for differences using a comparison operator. Here, the output of the comparison result can be, for example, a quality result of "OK" or not "OK." In that case, the quality results can be automatically taken into account in further processes, for example, production defective products (e.g., intermediate products or final products) can be automatically sorted out or support can be requested from the user of the production facility.

[0019] The approach of the present invention has two main steps: 1. Training for domain adaptation: In this step, we create the basis for transferring images from a real domain to an artificial domain. This is a part of domain adaptation. One possible exemplary method for training for domain adaptation is the above-mentioned "generative adversarial network (GAN)" technique.

[0020] Training can be characterized as follows: a. In a first step, for optical acquisition, a data set is acquired consisting of a real image and its associated synthetic image, each of the same perspective and the same parameters (internal and external) of the image acquisition device, e.g., camera.

[0021] In this case, data acquisition and training can be carried out at the early stage of the production equipment commissioning and only needs to be carried out once for the production equipment according to the variety of products (intermediate or final products) to be produced, i.e. no new training is required when changing products.

[0022] Data acquisition is then carried out jointly during the testing or commissioning phase, where realistic images are acquired and saved where quality control is required, as are the camera parameters (internal and external parameters of the image acquisition device). Furthermore, as part of the data acquisition for training, the digital twin is used to render related synthetic images from the same perspective with the same parameters and saved in a dataset.

[0023] b. Finally, a training step is carried out in which the transfer of real data to synthetic data is learned. For this purpose, for example, the above-mentioned "generative adversarial network (GAN)" technique can be used.

[0024] 2. Production equipment operation: a. Position the image capture device in a suitable position so that the product or part, intermediate product or final product is visible. The suitable position can be calculated by heuristics at the runtime of the production facility. This applies especially to autonomous production facilities. b.Get product images. c. Transforming realistic product images into the image domain of synthetic (simulated) images using domain adaptation, e.g., generative adversarial networks (GAN) techniques. d. Rendering a composite image acquired with identical parameters (internal and external parameters of the image acquisition device) based on the same position in the digital twin. e. Compare the images using a comparison operator, where in one advantageous embodiment of the present invention a neural network can be used, although traditional pixel-based approaches are also suitable.

[0025] Further advantages of the invention will become apparent from the following description of an example embodiment of the invention based on the single drawing. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 shows a production facility PA that uses optical quality control of intermediate or final products of the production facility, which can be used in various technical fields (technical domains) for product production using appropriate methods and techniques, for example in factories, production lines, large-scale equipment, etc. [Figure 2] FIG. 2 shows a production facility PA that uses optical quality control of intermediate or final products of the production facility, which can be used in various technical fields (technical domains) for product production using appropriate methods and techniques, for example in factories, production lines, large-scale equipment, etc. DETAILED DESCRIPTION OF THE INVENTION

[0027] In a first variant of the invention for the optical quality control of intermediate or final products according to option "A", the production facility PA comprises a production facility control device PAS, a database DB, an image acquisition device BEE, e.g. configured in the form and shape of a camera, and an output device AEH. The database DB and the image acquisition device BEE are connected to the production facility control device PAS for accessing it, while the output device AEH, like the database DB and the image acquisition device BEE, is either connected to the production facility control device PAS for accessing it, in the first embodiment according to option "I", or is included in the production facility control device PAS, in the second embodiment according to option "II".

[0028] As an alternative to the first variant of the invention according to option "A", in a second variant of the invention for the optical quality control of intermediate or final products according to option "B", the production equipment PA, the production equipment control device PAS, the database DB and the image acquisition device BEE are not "integrated" in the production equipment PA, but rather they all function as separate devices, and the production equipment control device PAS is connected for access to the production equipment PA, the database DB and the image acquisition device BEE, while the output device AEH is either included in the production equipment PA for access by the production equipment control device PAS in the first embodiment according to option "I", or is included in the production equipment control device PAS in the second embodiment according to option "II".

[0029] Other variations between these two "extreme" variations of the invention are also possible (not explicitly shown), in which either only the database DB, the image acquisition device BEE or the production equipment control device PAS are included in the production equipment PA, or two of the above devices are each included in the production equipment PA.

[0030] In the two illustrated variants according to options "A" and "B", the production equipment control device PAS is preferably a control device STE for a robotic or automation system having universally usable mobile machines for performing processing, service and / or manufacturing tasks.

[0031] Regarding the variant shown according to option "B", the production facility control device PAS can alternatively be a general personal computer or control device.

[0032] The following explanations regarding the description of the figures apply to both the illustrated variants of the invention for the optical quality control of intermediate or final products according to options "A" and "B".

[0033] To this end, The image acquisition device BEE acquires both product images PB of the production equipment PA, which may relate to an intermediate product or a final product, with respect to given internal and external parameters of the image acquisition device BEE, and product images PB1,...,PBn, with respect to uniformly given internal and external parameters of the image acquisition device BEE, to generate a dataset based on a plurality of "n" image pairs. The database DB stores the digital twin data DZD of the digital twin DZ of the production equipment PA, and when the production equipment PA is run, the digital twin DZ is synchronized with the production equipment PA. Preferably, a computer program product CPP configured as an app, a computer-implemented tool CIW loadable into a production facility control device PAS, is used for optical quality control of intermediate or final products of the production facility PA.

[0034] The computer-implemented tool CIW includes a non-volatile readable memory SP in which processor-readable control program instructions of a program module PGM for optical quality control are stored, and a processor PZ connected to the memory SP for executing the control program instructions of the program module PGM for optical quality control of intermediate products or final products of the production facility PA. Includes:

[0035] For this purpose, the computer-implemented tool CIW · A product image PB of a production equipment PA acquired by an image acquisition device BEE with respect to given internal and external parameters, and product images PB1,...,PBn acquired by the image acquisition device BEE with respect to given internal and external parameters to generate a dataset based on a plurality of "n" image pairs; -Digital twin data DZD of the digital twin DZ of the production equipment PA stored in the database DB, Using this, the digital twin DZ is synchronized with the production equipment PA when the production equipment PA is executed.

[0036] These data are requested through access as input data by the processor PZ when loading the computer-implemented tool CIW into the production facility control device PAS, and are either subsequently collected or provided.

[0037] a program module PGM of the computer-implemented tool CIW and a processor PZ of the computer-implemented tool C for executing control program instructions of the program module PGM for optical quality control; ·Synthetic simulation image SB based on digital twin data DZD syn a step of rendering rdn the rendered composite simulation image SB syn is based on the same internal and external parameters as in the product image acquisition step; transferring product images PB from a real domain to an artificial domain using the trained domain adaptation DA, Domain adaptation is preferably implemented, for example, according to the principles of "generative adversarial networks (GAN)" as a "machine learning" model, in which two competing artificial neural networks, called a generative network and a discriminative network, are used for data generation, the generative network generates artificial data, which the discriminative network checks based on real data, e.g., obtained from images, and the two networks are logically and mathematically linked to each other so that the artificial data generated by the generative network looks increasingly realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the real data; The domain adaptation DA has a domain transfer parameter DTP obtained by training trn, and the trained domain adaptation DA having the domain transfer parameter DTP preferably performs the following steps “S1” and “S2”, namely: "S1": For uniformly given internal and external parameters, the acquired product images PB1,...,PBn and their associated synthetic simulation images SB1 syn ,...,SBn syn generating a dataset based on a number "n" of image pairs, "S2": A step performed in a two-stage training trn, which includes a step of training the transfer of product image-related data to simulation image-related data by a learning method, for example, a "generative adversarial network (GAN)", based on the generated dataset; Trained using After training, the trained domain adaptive DA converts the product image PB into the synthesized product image PB according to the domain transfer parameter DTP. syn , which generates a synthetic simulation image SB for comparison purposes. syn and composite product image PB syn An image pair consisting of Steps and ·Synthetic product image PB syn A synthetic simulation image SB syn and comparing vgl using a comparison operator VO, which is preferably configured such that the comparison is performed pixel by pixel; a step of outputting a comparison result VGE for qualitatively evaluating the product, the comparison result VGE being preferably output by the production facility PA or an output device AEH of the production facility PA's production facility control device PAS, for example; is configured to run.

Claims

1. 1. A computer-implemented method for optical quality control of intermediate or final products of a production facility, comprising: Product images (PB) of a production facility (PA) are used, acquired by an image acquisition device (BEE) for given internal and external parameters, Digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is used, and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running.

1. A computer-implemented method comprising: a) A synthetic simulation image (SB) is generated based on the digital twin data (DZD). ｓｙｎ ) and rendering (rdn) the rendered synthetic simulation image (SB ｓｙｎ ) is based on the same intrinsic and extrinsic parameters as in product image acquisition; b) transferring (trf) the product image (PB) from a real domain to an artificial domain using a trained (trn) domain adaptation (DA), the domain adaptation (DA) having a domain transfer parameter (DTP) obtained by the training (trn), and converting the product image (PB) into a synthetic product image (PB) by the domain adaptation (DA) according to the domain transfer parameter (DTP); ｓｙｎ ) and thereby generate the composite simulation image (SB ｓｙｎ ) and the composite product image (PB ｓｙｎ ) and an image pair consisting of c) Using a comparison operator (VO), the composite product image (PB ｓｙｎ ) to the composite simulation image (SB ｓｙｎ ) and comparing (vgl) d) outputting (asg) a comparison result (VGE) for qualitatively evaluating the product, in particular by an output device (AEH) of said production facility (PA) or by a production facility control device (PAS) of said production facility (PA); 1. A computer-implemented method comprising:

2. The domain adaptation (DA) is implemented as a "machine learning" model according to the principles of "generative adversarial networks (GAN)." For data generation, two competing artificial neural networks, called a generative network and a discriminative network, are used. The generative network generates artificial data, which the discriminative network checks based on real data, e.g., obtained from images. the two networks are logically and mathematically linked together such that the artificial data generated by the generative network appears increasingly realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the genuine data.

10. The computer-implemented method of claim 1.

3. The trained (trn) domain adaptation (DA) with the domain transfer parameters (DTP) is "S1": For uniformly given internal and external parameters, the acquired product images (PB1, ..., PBn) and their associated synthetic simulation images (SB1 syn , . . . , SBn syn generating a dataset based on a number "n" of image pairs consisting of: "S2": Training the transfer of product image-related data to simulation image-related data by a learning method, such as a "generative adversarial network (GAN)", based on the generated dataset; 2. The computer-implemented method of claim 1, wherein the method is performed in a two-stage training (trn) comprising:

4. 2. The computer-implemented method of claim 1, wherein the comparison operator (VO) is configured such that the comparison is performed pixel by pixel.

5. 2. The computer-implemented method of claim 1, wherein the production equipment (PA) is a robotic or automation system having universally usable mobile machines for performing processing, service and / or manufacturing tasks.

6. A computer-implemented tool (CIW) for the optical quality control of intermediate or final products of a production facility, in particular a computer program product (CPP) configured as an app, which for said quality control comprises: Product images (PB) of a production facility (PA) are used, acquired by an image acquisition device (BEE) for given internal and external parameters, Digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is used, and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running. In computer implemented tools (CIW), a non-volatile readable memory (SP) in which processor-readable control program instructions of a program module (PGM) for optical quality control are stored; and a processor (PZ) connected to said memory (SP) for executing said control program instructions of said program module (PGM) for optical quality control of said intermediate or final products of a production facility; When loaded by a computer system including a) A synthetic simulation image (SB) based on the digital twin data (DZD) ｓｙｎ ) is rendered (rdn), and the rendered composite simulation image (SB ｓｙｎ ) is based on the same internal and external parameters as used to capture the product image, b) the product image (PB) is transferred (trf) from a real domain to an artificial domain using a trained (trn) domain adaptation (DA), the domain adaptation (DA) having a domain transfer parameter (DTP) obtained by the training (trn), and the domain adaptation (DA) converts the product image (PB) into a synthetic product image (PB) according to the domain transfer parameter (DTP). ｓｙｎ ) and thereby generate the composite simulation image (SB ｓｙｎ ) and the composite product image (PB ｓｙｎ ) and an image pair consisting of c) The composite product image (PB ｓｙｎ ) is compared with the synthesized simulation image (SB ｓｙｎ ) and compared (vgl) d) the comparison result (VGE) for qualitatively evaluating the product is output (asg) in particular by an output device (AEH) of the production facility (PA) or a production facility control device (PAS) of the production facility (PA), expressing a computer system in which said program module (PGM) is configured and in which said processor (PZ) is configured to execute said control program instructions of said program module (PGM) for optical quality control, A computer implemented tool (CIW) comprising:

7. The processor (PZ) and the program module (PGM) for optical quality control are configured such that the domain adaptation is implemented according to the principles of "generative adversarial networks (GAN)" as a "machine learning" model, and two competing artificial neural networks, called a generative network and a discriminative network, are used for data generation, the generative network generating artificial data and the discriminative network checking the artificial data based on real data, e.g., obtained from images, the two networks are logically and mathematically linked together such that the artificial data generated by the generative network appears increasingly realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the genuine data. A computer implemented tool (CIW) according to claim 6, characterized in that it

8. The processor (PZ) and the program module (PGM) for optical quality control train (trn) the domain adaptation (DA) with the domain transfer parameter (DTP) by the following steps "S1" and "S2": Step "S1": For uniformly given internal and external parameters, the acquired product images (PB1, ..., PBn) and their associated synthetic simulation images (SB1 syn , . . . , SBn syn generating a dataset based on a number "n" of image pairs consisting of: Step "S2": training the transfer of product image-related data to simulation image-related data by a learning method, for example, a "generative adversarial network (GAN)", based on the generated dataset; 7. A computer implemented tool (CIW) according to claim 6, characterized in that it is adapted to be performed in a two-stage training (trn) comprising:

9. 7. A computer-implemented tool (CIW) according to claim 6, characterized in that the processor (PZ) and the program module (PGM) for optical quality control and the comparison operator (VO) are configured so that the comparison is performed pixel by pixel.

10. 7. A computer implemented tool (CIW) according to claim 6, characterized in that the production equipment (PA) is a robotic or automation system having universally usable mobile machines for carrying out processing, service and / or manufacturing tasks.

11. A production installation control system (PAS) for optical quality control of intermediate or final products of a production installation (PA), comprising: an image acquisition device (BEE) for acquiring product images (PB) of said production equipment (PA) for given internal and external parameters is either a component of said production equipment (PA) and thus connected to said production equipment control device (PAS) or is assigned to said production equipment (PA) and thus connected to said production equipment control device (PAS); A database (DB) storing digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is assigned to the production equipment (PA) and is thus connected to the production equipment control device (PAS), and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running; In the production equipment control system (PAS), 7. A production installation control system (PAS) characterized by a computer implemented tool (CIW) according to claim 6, which is loadable into said production installation control system (PAS) for carrying out the method according to claim 1.

12. 12. A production equipment control device (PAS) according to claim 11, characterized in that it is a control device (STE) for a robotic or automation system with universally usable mobile machines for carrying out processing, service and / or manufacturing tasks.

Images