Computer-implemented method and tool for optical quality control of intermediate or final products of production plants as well as production plant control

Description

[0001] The invention relates to a computer-implemented method for optical quality control of intermediate or end products of production plants according to the preamble of claim 1, a computer-implemented tool for optical quality control of intermediate or end products of production plants according to the preamble of claim 6 and a production plant control system according to the preamble of claim 11.

[0002] Production processes in manufacturing plants, regardless of the field or domain, are never error-free and often require optical quality control. This is especially true for automated production processes. Optical quality control involves capturing an image of the product or a part of it and using this image to calculate whether all features meet the requirements. Setting up quality control involves engineering effort, which is particularly problematic for autonomous, automated production plants in industrial environments that are designed to adapt to new products with minimal effort.

[0003] A typical example of such a production process is the assembly of a product in an industrial environment (industrial domain) using a robot system or automation system, where a universally applicable motion machine is used to perform handling, service and / or manufacturing tasks for assembly, and where, for example after a joining process, it must be checked whether a component to be assembled has been inserted in the correct place and is fully locked in place.

[0004] For this purpose, it is common practice to provide the commissioning engineer of the production plant, and especially the robot system, with tools to define quality rules as simply as possible. For example, certain optical features, such as circles, lines, or patterns, must be present in specific positions or have specific dimensions in the optically captured image. The commissioning engineer then verifies these rules using a few sample images and adjusts parameters until the desired reliability is achieved.

[0005] A document CN 15239621 A discloses a visual intelligent inspection method for substations, based on the matching of virtual and real scenes, and falls within the field of substation inspection.

[0006] Document WO 2022269470 A1 concerns a modular device and a method for inspecting industrial products.

[0007] According to document US 2021311455 A1, a computer system, procedure, and operational verification program are provided to make it easier and more accurate to verify the operation of a machine tool.

[0008] The object of the invention is to provide a computer-implemented method and tool for optical quality control of intermediate or final products of production plants, as well as a production plant control system, which reduces the effort required to set up quality control under constantly changing conditions and circumstances at the production plants for a correspondingly adaptive use of the production plants.

[0009] This problem is solved by the features specified in the characterizing portion of claim 1, starting from the computer-implemented method defined in the preamble of claim 1.

[0010] Furthermore, this problem is solved starting from the computer-implemented tool defined in the preamble of claim 6 by the features specified in the characterizing portion of claim 6.

[0011] Furthermore, this task is solved starting from the production plant control defined in the preamble of claim 11 by the features specified in the characterizing portion of claim 11.

[0012] The idea underlying the invention according to independent claims 1, 6 and 11 is that for optical quality control of intermediate or finished products of a production plant, in which a product image of the production plant acquired by an image acquisition device for given intrinsic and extrinsic parameters is used and digital twin data from a digital twin of the production plant (PA) is used, wherein the digital twin is synchronized with the production plant at runtime, (i) a synthetic simulation image based on the digital twin data is rendered, wherein the rendered synthetic simulation image is based on the same intrinsic and extrinsic parameters as in the product image acquisition,(ii) the product image from a real domain is transferred to an artificial domain using a trained domain adaptation, and a synthetic product image is generated from the product image using domain transfer parameters obtained through training, (iii) the synthetic product image is compared with the synthetic simulation image using a comparison operator, and (iv) a comparison result that qualitatively evaluates the product is output.

[0013] In comparison to classical rule-based methods, the invention pursues a fundamentally different approach that offers better scaling possibilities.

[0014] A key aspect of the invention for quality control is the use of domain transformation from a real image domain to an artificial one. This results in simpler image comparison for detecting image differences.

[0015] An advantageous further development of the invention according to claims 2 and 7 consists in the domain adaptation being implemented as a "machine learning" model based on the principle of a "generative adversarial network". <gan>"is carried out in which two competing artificial neural networks, called generator and discriminator, are used for data generation, the generator producing artificial data, which the discriminator checking against authentic data, e.g. captured from images, and in which the two networks are logically and mathematically linked in such a way that the artificial data produced by the generator appears increasingly real and in the end the discriminator is no longer able to distinguish the real data from the authentic data.

[0016] Furthermore, it is advantageous if, according to claims 3 and 8, the trained domain adaptation with the domain transfer parameters is carried out in a two-stage training with the following steps "S1" and "S2". "S1": Generating a dataset based on a large number "n" of image pairs, each formed for uniformly given intrinsic and extrinsic parameters from a captured product image and a corresponding synthetic simulation image; "S2": Training the transfer of product image-related data to simulation image-related data using the generated dataset through learning methods such as a Generative Adversarial Network. <gan>".

[0017] Furthermore, the invention is advantageously characterized by the fact that According to claims 4 and 9, the comparison operator is configured such that the comparison is performed pixel by pixel. According to claims 5 and 10, the production plant is a robot system or automation system with a universally applicable motion machine for performing handling, service, and / or manufacturing tasks.

[0018] The basic scenario underlying the invention will be outlined below.

[0019] The starting point for the scenario outlined is the prerequisite that a detailed digital twin of the production plant exists at runtime. This digital twin includes: Geometries and / or textures of the objects, robots and components used in the production plant. Expected positions and orientations of the objects, robots and components of the production plant at runtime, i.e., at the given time.

[0020] This prerequisite is usually met for autonomously acting production plants or systems, but it can also be met or retrofitted for classic production plants or systems.

[0021] Based on this, or on this foundation, according to the inventive approach, a real image of a part or product to be inspected, e.g., an intermediate or final product, from the production plant is captured and this image is transferred into a simulation image space using domain adaptation. Furthermore, the transferred image is compared with a synthetically rendered image generated using the digital twin. The two images, now present in the same image space and appearing synthetic, can then be examined for differences using a comparison operator. The output of a comparison result can, for example, be a quality result of "OK" or "not OK".

[0022] The quality result can then be automatically taken into account in the further process; for example, defective products (e.g., intermediate or final products) can be automatically sorted out, or a user of the production plant can be called in for support.

[0023] The invention approach has two main stages: 1. Domain adaptation training: This step lays the foundation for transferring images from a real domain to an artificial domain. This is part of domain adaptation. One possible, exemplary method for domain adaptation training is the previously mentioned Generative Adversarial Network (GAN). <gan>"-Technology. The training can be characterized as follows: a. In a first step, a dataset is acquired consisting of real images and corresponding synthetic images. These images are captured from the same perspective and with the same parameters (extrinsic and intrinsic) of an image acquisition device, such as a camera. Data acquisition and training can be performed during the initial commissioning phase of the production plant and, depending on the variability of the products being manufactured (intermediate or final products), only need to be carried out once for the entire plant. This means that no retraining would be necessary when changing products. Data acquisition runs concurrently during a test or ramp-up phase. Real images are captured and stored at points requiring quality control, and the camera parameters (intrinsic and extrinsic parameters of the image acquisition device) are also recorded.As part of the data acquisition for training, the corresponding synthetic image is also rendered from the same perspective and with the same parameters using the digital twin and stored in the data set. b. Finally, the training step follows, in which the transfer of real-world data to synthetic data is learned. For this, the previously mentioned "Generative Adversarial Network" is used, for example. <gan>"-Technology in question." 2. Production plant operation: a. Position the image capture device appropriately so that the product or component, intermediate or final product, is visible. Suitable positions can be calculated using heuristics during production plant operation. This is particularly relevant for autonomous production plants. b. Taking a product image c. Transformation of the real product image into the image domain of a synthetic (simulated) image using domain adaptation, e.g., the "Generative Adversarial Network". <gan>"-Technology d. Rendering a synthetic image that is captured in the digital twin from the same position with identical parameters (intrinsic and extrinsic parameters of the image acquisition device). e. Comparison of the images using a comparison operator. In an advantageous embodiment of the invention, a neural network can be used here, but classical pixel-based methods are also suitable.

[0024] Further advantages of the invention will become apparent from the following description of an embodiment of the invention using a single FIGURE.

[0025] The figure shows a production plant PA with optical quality control for intermediate or final products of the production plant, which can be used in various technical fields (in technical domains) for product manufacturing using appropriate methods and techniques, e.g. in factories, production lines and large equipment.

[0026] In a first embodiment of the invention for optical quality control of intermediate or final products according to an option "A" The production plant PA includes a production plant control system PAS, a database DB, an image acquisition device BEE (e.g., in the form of a camera), and an output unit AEH. While the database DB and the image acquisition device BEE are connected to the production plant control system PAS for access, the output unit AEH is, in a first embodiment according to option "I", such as the database DB and the image acquisition device BEE, connected to the production plant control PAS for access, or in a second embodiment according to option "II" included in the production plant control system PAS.

[0027] As an alternative to the first embodiment of the invention according to the option "A" are in a second embodiment of the invention for optical quality control of intermediate or final products according to an option "B" The production plant PA, the production plant control PAS, the database DB and the image acquisition device BEE are not united "under one roof" within the production plant PA, but rather they all function as separate units, with the production plant control PAS being connected to the production plant PA, the database DB and the image acquisition device BEE for access, while the output unit AEH is either in the first embodiment according to the option "I" in the production plant PA for access to the production plant control system PAS, or in the second embodiment according to the option "II" which is included in the production plant control system PAS.

[0028] Between these two "extreme" embodiments of the invention, other variants are also conceivable (not explicitly shown in the FIGURE), where either only the database DB, the image acquisition device BEE or the production plant control PAS is included in the production plant PA, or the production plant PA contains two of the aforementioned units.

[0029] In both depicted design variants according to the options "A" and "B" The production plant control system (PAS) is preferably a control unit (STE) for a robot system or automation system with a universally applicable motion controller for performing handling, service and / or manufacturing tasks.

[0030] Regarding the depicted design variant according to the option "B" Alternatively, it is also possible that the production plant control system (PAS) is an ordinary personal computer or controller.

[0031] The following descriptions of the FIGURE apply to both illustrated embodiments of the invention for optical quality control of intermediate or final products according to the options. "A" and "B".

[0032] For this purpose The image acquisition device BEE captures, for given intrinsic and extrinsic parameters of the image acquisition device BEE, both a product image PB of the production plant PA, which can relate to an intermediate or final product, and, for uniformly given intrinsic and extrinsic parameters of the image acquisition device BEE, generates a data set based on a plurality "n" of image pairs product images PB1, ..., PBn. The database DB stores digital twin data DZD of a digital twin DZ of the production plant PA, whereby the digital twin DZ is synchronized with the production plant PA at runtime. A computer-implemented tool CIW is used, which is preferably a computer program product CPP designed as an APP and can be loaded into the production plant control system PAS for optical quality control of intermediate or final products of the production plant PA.

[0033] The computer-implemented tool CIW contains a non-volatile, readable memory SP in which processor-readable control program instructions of a program module PGM for optical quality control are stored, and a processor PZ connected to the memory SP, which executes the control program instructions of the program module PGM for optical quality control of intermediate or final products of the production plant PA.

[0034] For this purpose, the computer-implemented tool CIW is used. the product image PB of the production plant PA captured by the image acquisition device BEE for given intrinsic and extrinsic parameters, and the product images PB1, ..., PBn captured by the image acquisition device BEE for given intrinsic and extrinsic parameters to generate the data set on the basis of the multitude "n" of image pairs, the digital twin data DZD of the digital twin DZ of the production plant PA stored in the database DB, wherein the digital twin DZ is synchronized with the production plant PA at runtime.

[0035] When loading the computer-implemented tool CIW into the production plant control system PAS, this data is requested as input data by the processor PZ and then either drawn in or fed in.

[0036] The program module PGM of the computer-implemented tool CIW is designed and the processor PZ of the computer-implemented tool CIW, which executes the control program commands of the program module PGM for optical quality control, is designed such that the following steps for optical quality control are carried out: Render a synthetic simulation image SB syn based on the Digital Twin Data (DZD). The rendered synthetic simulation image SB syn uses the same intrinsic and extrinsic parameters as the product image acquisition. Transfer the product image PB from a real domain to an artificial domain using a trained domain adaptation DA. The domain adaptation is preferably implemented as a machine learning model based on the principle of a Generative Adversarial Network. <gan>"This process is carried out in which two competing artificial neural networks, designated as generator and discriminator, are used for data generation. The generator produces artificial data, which the discriminator checks against authentic data, e.g., data captured from images. The two networks are logically and mathematically linked in such a way that the artificial data generated by the generator appears increasingly realistic, until ultimately the discriminator is no longer able to distinguish the real data from the authentic data. -- The domain adaptation (DA) contains domain transfer parameters (DTP) obtained through the training process (trn)."The trained trn domain adaptation DA with the domain transfer parameter DTP is preferably performed in a two-stage training trn with the following steps "S1" and "S2": "S1": Generating a dataset based on a large number "n" of image pairs, which are formed for uniformly given intrinsic and extrinsic parameters from the acquired product images PB1, ..., PBn and the corresponding synthetic simulation image SB1 syn , ..., SBn syn; "S2": Training the transfer of product image-related data to simulation image-related data using the generated dataset by learning methods such as "Generative Adversarial Network". <gan> "After training, the trained domain adaptation DA generates a synthetic product image PB syn from the product image PB according to the domain transfer parameters DTP. This creates an image pair in an artificial image space, formed from the synthetic simulation image SB syn and the synthetic product image PB syn, for comparison purposes. The synthetic product image PB syn is compared with the synthetic simulation image SB syn using a comparison operator VO. The comparison operator VO is preferably configured such that the comparison is performed pixel by pixel. A comparison result VGE, which qualitatively evaluates the product, is output asg. This comparison result VGE is preferably output asg by the output unit AEH of the production plant PA or the production plant control PAS of the production plant PA."< / gan> < / gan> < / gan> < / gan> < / gan> < / gan> < / gan>

Claims

1. Computer-implemented method for optical quality control of intermediate or end products of production installations, in which for the quality control - a product image (PB) of a production installation (PA) captured by an image capture device (BEE) for given intrinsic and extrinsic parameters is used, - digital twin data (DZD) of a digital twin (DZ) of the production installation (PA) are used, wherein the digital twin (DZ) is synchronized with the production installation (PA) at the time of operation thereof, characterized by a) rendering (rdn) a synthetic simulation image (SBsyn) based on the digital twin data (DZD), wherein the rendered synthetic simulation image (SBsyn) is based on the same intrinsic and extrinsic parameters as during product image capture, b) transferring (trf) the product image (PB) from a real domain into an artificial domain by means of a trained (trn) domain adaptation (DA) which contains domain transfer parameters (DTP) obtained by the training (trn) and with which a synthetic product image (PBsyn) is generated from the product image (PB) in accordance with the domain transfer parameters (DTP), whereby an image pair formed from the synthetic simulation image (SBsyn) and the synthetic product image (PBsyn) arises in an artificial image space of a simulation for comparison purposes, c) comparing (vgl) the synthetic product image (PBsyn) with the synthetic simulation image (SBsyn) by means of a comparison operator (VO), d) outputting (asg) a comparison result (VGE) which qualitatively assesses the product, in particular by way of an output unit (AEH) of the production installation (PA) or a production installation controller (PAS) of the production installation (PA).

2. Computer-implemented method according to Claim 1, characterized in that the domain adaptation (DA) is implemented as a "machine learning" model according to the principle of a "generative adversarial network <GAN>", in which data are generated by the use of two competing artificial neural networks referred to as generator and discriminator, of which the generator generates artificial data which the discriminator checks on the basis of authentic data, e.g. captured with the aid of images, and wherein the two networks are logically and mathematically combined with one another in such a way that the artificial data generated by the generator seem more and more genuine and at the end the discriminator is no longer able to differentiate the genuine data from the authentic data.

3. Computer-implemented method according to Claim 1 or 2, characterized in that the trained (trn) domain adaptation (DA) with the domain transfer parameters (DTP) is implemented in a two-stage training (trn) with the following steps "S1" and "S2" "S1": generating a data set on the basis of a multiplicity "n" of image pairs which are formed from captured product images (PB1, ..., PBn) and associated synthetic simulation images (SB1syn, ..., SBnsyn) for uniformly given intrinsic and extrinsic parameters; "S2": training the transfer of product-image-related data to simulation-image-related data with the aid of the generated data set by way of learning methods such as e.g. "generative adversarial network <GAN>".

4. Computer-implemented method according to Claim 1 or 2, characterized in that the comparison operator (VO) is configured in such a way that the comparison is implemented pixel by pixel.

5. Computer-implemented method according to any of Claims 1 to 4, characterized in that the production installation (PA) is a robot system or automation system with a universally usable automatic movement machine for executing handling, service and / or manufacturing tasks.

6. Computer-implemented tool (CIW), in particular a computer program product (CPP) configured as an APP, for optical quality control of intermediate or end products of production installations, in which for the quality control - a product image (PB) of a production installation (PA) captured by an image capture device (BEE) for given intrinsic and extrinsic parameters is used, - digital twin data (DZD) of a digital twin (DZ) of the production installation (PA) are used, wherein the digital twin (DZ) is synchronized with the production installation (PA) at the time of operation thereof, characterized by a nonvolatile, readable memory (SP), in which processor-readable control program instructions of a program module (PGM) for optical quality control are stored, and a processor (PZ) connected to the memory (SP), said processor executing the control program instructions of the program module (PGM) for optical quality control of the intermediate or end products of production installations, wherein the program module (PGM) is constituted in such a way, and the processor (PZ) that executes the control program instructions of the program module (PGM) for optical quality control is configured in such a way, that a) a synthetic simulation image (SBsyn) based on the digital twin data (DZD) is rendered (rdn), wherein the rendered synthetic simulation image (SBsyn) is based on the same intrinsic and extrinsic parameters as during product image capture, b) the product image (PB) is transferred (trf) from a real domain into an artificial domain by means of a trained (trn) domain adaptation (DA) which contains domain transfer parameters (DTP) obtained by the training (trn) and with which a synthetic product image (PBsyn) is generated from the product image (PB) in accordance with the domain transfer parameters (DTP), whereby an image pair formed from the synthetic simulation image (SBsyn) and the synthetic product image (PBsyn) arises in an artificial image space of a simulation for comparison purposes, c) the synthetic product image (PBsyn) is compared (vgl) with the synthetic simulation image (SBsyn) by means of a comparison operator (VO), d) a comparison result (VGE) which qualitatively assesses the product is output (asg), in particular by way of an output unit (AEH) of the production installation (PA) or a production installation controller (PAS) of the production installation (PA).

7. Computer-implemented tool (CIW) according to Claim 6, characterized in that the processor (PZ) and the program module (PGM) for optical quality control are configured in such a way that the domain adaptation is implemented as a "machine learning" model according to the principle of a "generative adversarial network <GAN>", in which data are generated by the use of two competing artificial neural networks referred to as generator and discriminator, of which the generator generates artificial data which the discriminator checks on the basis of authentic data, e.g. captured with the aid of images, and wherein the two networks are logically and mathematically combined with one another in such a way that the artificial data generated by the generator seem more and more genuine and at the end the discriminator is no longer able to differentiate the genuine data from the authentic data.

8. Computer-implemented tool (CIW) according to Claim 6 or 7, characterized in that the processor (PZ) and the program module (PGM) for optical quality control are configured in such a way that the trained (trn) domain adaptation (DA) with the domain transfer parameters (DTP) is implemented in a two-stage training (trn) with the following steps "S1" and "S2" "S1": generating a data set on the basis of a multiplicity "n" of image pairs which are formed from captured product images (PB1, ..., PBn) and associated synthetic simulation images (SB1syn, ..., SBnsyn) for uniformly given intrinsic and extrinsic parameters; "S2": training the transfer of product-image-related data to simulation-image-related data with the aid of the generated data set by way of learning methods such as e.g. "generative adversarial network <GAN>".

9. Computer-implemented tool (CIW) according to any of Claims 6 to 8, characterized in that the processor (PZ) and the program module (PGM) for optical quality control and also the comparison operator (VO) are configured in such a way that the comparison is implemented pixel by pixel.

10. Computer-implemented tool (CIW) according to any of Claims 6 to 9, characterized in that the production installation (PA) is a robot system or automation system with a universally usable automatic movement machine for executing handling, service and / or manufacturing tasks.

11. Production installation controller (PAS) for optical quality control of intermediate or end products of a production installation (PA), wherein - an image capture device (BEE) which captures a product image (PB) of the production installation (PA) for given intrinsic and extrinsic parameters either is part of the production installation (PA) and as such is connected to the production installation controller (PAS) or is assigned to the production installation (PA) and as such is connected to the production installation controller (PAS), - a database (DB) which stores digital twin data (DZD) of a digital twin (DZ) of the production installation (PA) is assigned to the production installation (PA) and as such is connected to the production installation controller (PAS), wherein the digital twin (DZ) is synchronized with the production installation (PA) at the time of operation thereof characterized by a computer-implemented tool (CIW) according to any of Claims 6 to 10, which is loadable into the production installation controller (PAS) in order to implement the method according to any of Claims 1 to 5.

12. Production installation controller (PAS) according to Claim 11, characterized by a control unit (STE) for a robot system or automation system with a universally usable automatic movement machine for executing handling, service and / or manufacturing tasks.