Computer-implemented method and device for verifying correctness of assembly

Abstract

The invention relates to a computer-implemented method which is designed to verify correctness of assembly. The method comprises determining an actual 3D model of the assembly based on image data of the assembly, and comparing the 3D model so determined with a target 3D model of the assembly for the purpose of verifying correctness of assembly.

Description

[0001] Computer-implemented method and device for checking the correctness of an assembly

[0002] The present disclosure relates to a computer-implemented method configured to check the correctness of an assembly. Furthermore, a data processing device configured to at least partially execute the method is provided. Furthermore, a computer program is provided, which comprises instructions that, when executed by a computer, cause the computer to at least partially execute the method. Furthermore, a computer-readable medium is provided, which comprises instructions that, when executed by a computer, cause the computer to at least partially execute the method.

[0003] The discussion of the prior art in the description is in no way to be interpreted as an admission that this prior art is generally known or forms part of the general technical knowledge in the technical field of the present disclosure.

[0004] During an assembly process, especially a complex assembly, as is often the case in mechanical and plant engineering, errors can occur during the assembly of the components that make up the assembly. Depending on the application and the device, such an error can lead to production downtime and, among other things, significant rework costs for the assembly manufacturer after delivery of the assembly to a customer.

[0005] Traditionally, assemblies are therefore manually evaluated by one or more human quality inspectors (so-called end-of-line testing) to assess the quality of the assembly by determining whether the correct components have been assembled, whether the assembled components are complete (i.e., whether all components have been assembled), and whether the alignment of all components involved, e.g., relative to each other, is correct. Such a manual process is time-consuming and can be error-prone.

[0006] Therefore, automated solutions for checking the correctness of an assembly are described in the state of the art.

[0007] In this context, reference is made to US 9,187,188 B2 and US 10,242,438 B2.

[0008] US 9,187,188 B2 relates to a method for inspecting an assembly of components in an aircraft structure. The method comprises capturing a visual representation of at least a portion of the structure comprising a plurality of components, storing an electronic file of the visual representation on a computer-readable medium, and accessing a three-dimensional model of the structure, wherein the three-dimensional model contains information about a correct or desired position of each of the plurality of components within the structure.The method further comprises comparing the captured visual representation with the three-dimensional design using a computer by graphically overlaying an image related to the visual representation with a second image related to the three-dimensional design to determine whether each of the plurality of components encompassed by the visual representation is in a correct position in the structure, as determined by a position of each corresponding component encompassed in the three-dimensional design. The method ultimately comprises generating feedback indicating a result of the comparison.

[0009] US 10,242,438 B2 describes a method for determining whether the assembly of an assembly was successful or not, and, as part of this method, a method for determining the position and orientation of components of the assembly. The method for determining whether the assembly of an assembly was successful or not comprises the three steps described below. In a first step, a grayscale image and a range image (i.e. an image with depth information) of the assembly are recorded. In a second step, the position and orientation of the components of the assembly are determined based on the two recorded images and a 3D model of the assembly. In a third step, it is determined whether the assembly of the assembly was successful or not based on the determined position and orientation.In the second step, in order to determine the position and orientation of the components of the assembly, edges extracted from the acquired grayscale image as well as depth points contained in the range image are iteratively matched to the 3D model of the assembly as best as possible, so that based on this, a deviation of the shape of the assembly from the 3D model is determined, so that by comparing the determined deviation with a limit value in the subsequent third step of the process, it can be determined whether the assembly of an assembly was successful or not.

[0010] A disadvantage of the procedure according to US 9,187,188 B2 as well as US 10,242,438 B2 is that, according to the teachings of both, the assembly can only be viewed from one perspective. This means that this described procedure reaches its limitations for complex assemblies, as is often the case in mechanical and plant engineering, in that the method for such complex assemblies would have to be carried out for a multitude of different perspectives, which in turn is very time- and computationally intensive. Since both methods use perspective projections on a 2D image plane that do not fully depict 3D reality, inspection poses that do not allow for quick or intuitive use or implementation typically have to be predefined so that the number of perspectives required for the inspection can be kept to a minimum.

[0011] DE102020 134680 A1 relates to a method for quality inspection of an object in a real environment using a camera, an optical display device, and a processing device. The method comprises the following steps: defining a test geometry and a reference geometry within a computer-aided data model, defining a test pose in which the camera is to be positioned by a user as a target position for a quality inspection of the object to be inspected, and visualizing the test pose on the optical display device. In a second phase, at least one image of the real environment is captured by the camera, the pose of which is located in an area encompassing the test pose, and the test geometry and reference geometry are tracked in the image.Furthermore, a pose of the tracked test geometry in relation to the reference geometry is determined, as well as at least one parameter based on how the pose of the stretched test geometry relates to a target pose of the test geometry defined in the data model. Furthermore, a quality indicator is determined based on the at least one parameter and output to the user via a human-machine interface.

[0012] The teaching of DE10 2020 134 680 A1, like US 9,187,188 B2 and US 10,242,438 B2, therefore requires predefined observation perspectives or inspection poses depending on the specific component to be tested, which limits the field of application of the method or makes its use inflexible.

[0013] Against the background of this prior art, an object of the present disclosure can be seen in specifying a device and / or a method which are each suitable for enriching the prior art.

[0014] A specific embodiment of the disclosure can solve the problem of providing a method for automated verification of the correctness of a complex assembly, as is the case, for example, in mechanical and plant engineering. This problem is solved by the features of the independent claim. The subordinate claims and dependent claims, each in combination, contain optional developments of the disclosure.

[0015] The problem is then solved by a computer-implemented method designed to check the correctness of an assembly. The method comprises determining an actual 3D model of the assembly based on image data of the assembly, and comparing the determined actual 3D model with a target 3D model of the assembly to check the correctness of the assembly.

[0016] A computer-implemented method can be understood as a method in which one step, several steps or all steps of the method are or are carried out at least partially by a data processing device or a computer.

[0017] Therefore, an automated solution for inspecting an assembly is proposed. It is conceivable that the assembly in question could also be scanned automatically and / or manually, e.g., using a mobile device, and that feedback on the quality of the assembly could be provided automatically, e.g., via a display of the mobile device.

[0018] The image data initially represents 2D or 2.5D information regarding the assembly, which is then converted into 3D information. It is conceivable that the image data may include depth information (e.g., as a depth map) and / or images (e.g., RGB images). Both can be captured or recorded from one or more perspectives. Depth images (even with inaccurate depth resolution) can resolve ambiguous scenes and / or depth scaling ambiguities when using a single camera. Synchronous recording of depth images with image data can therefore complement perspective estimation from a (mobile) camera, thus achieving more robust and faster convergence. It is conceivable that one or more mobile and compact sensors, such as an RGB-D depth camera (RGB-D can be understood as a colored point cloud), a solid-state lidar, or similar, are used for this purpose.In other words, image data acquisition can be performed in various modalities, e.g., using a monochrome and / or color camera, a depth sensor, a lidar / ToF sensor, and / or other or additional 3D sensors that, for example, utilize a patterned Z-stripe projection and / or include a laser scanner. The use of 2D / RGB information has, inter alia, the advantage that sensitivity to 3D sensor noise is comparatively low, so that even relatively dark, small, and / or shiny components can be reliably captured. It should also be noted that dark areas and small components can be easily captured or accessed with a mobile camera. For shiny objects, using different perspectives can be advantageous.

[0019] An assembly is an object that exists in the real world. An assembly is a subassembly (shortened to "group" according to DIN 199), which is a self-contained object consisting of two or more parts or subassemblies of a lower order, which can generally be disassembled. A single part, however, can be distinguished from an assembly in that it cannot be disassembled non-destructively (see DIN 199 Technical Product Documentation). In other words, an assembly comprises several individual parts, which may be grouped into subassemblies.

[0020] The term “component” used below therefore refers both to an individual part and to sub-assemblies comprising several individual parts, each of which is part of the assembly.

[0021] The term "correctness" can be understood broadly in relation to assembly in light of the above definition of assembly and can refer, inter alia, to the completeness of the assembly with regard to the individual components that make up the assembly. Additionally or alternatively, the correctness of the installed components can be checked with regard to whether the correct component is installed at all and / or whether the installed component is installed correctly, i.e. whether the installed component is installed in the right or correct place or position. By checking whether the correct component is installed, it can also be ensured that no components have been mixed up. Since similar components with slightly different dimensions are typically available for other series or processes in assembly, excluding mix-ups in this way is advantageous.

[0022] Correctness is defined or specified in this case via the target state. A prefixed target, such as in the target 3D model (hereinafter also used for position and orientation), therefore indicates the desired state that is to be represented or achieved by the actual state. Correctness testing can be affirmed if the actual state and target state match, and denied if the aforementioned deviations occur. Consequently, the actual state describes the actual physical state of the assembly in the real world, e.g., achieved through assembly.

[0023] The comparison step can therefore also be understood as a target-actual comparison, in which the correctness of the assembly is checked.

[0024] In contrast to the prior art, the disclosed method offers several advantages. Among other things, the method allows for three-dimensional (3D) testing of the assembly for correctness and is therefore also suitable for testing complex assemblies where conventional methods reach their limits.

[0025] In detail: A model in this case can be understood to be a computer model which - as a so-called "digital twin" - represents a simplified image of reality or of the assembly. The model used according to the disclosure is at least three-dimensional, i.e. it essentially reproduces the external dimensions of the assembly in spatial terms. The assembly can therefore be viewed from different perspectives or angles of view and thus also inspected from different angles. The conventional methods described at the beginning, on the other hand, use a perspective 2D model (i.e. an image) in the case of US 9,187,188 B2 and a 2.5D model (i.e. an image paired with depth information) of the assembly in the case of US 10,242,438 B2, which means that only that part of the assembly which is in the field of view (FoV) of the camera used can be checked for correctness.If the assembly is to be inspected using conventional methods from two different directions or perspectives, e.g., a front and a back of the assembly, the respective conventional method must be carried out completely twice, once with the camera looking at the front and once with the camera looking at the back, with the 2D or 2.5D model being calculated. Deviating from this, it is sufficient, according to the disclosure, to generate the 3D model of the assembly only once in order to be able to carry out a comprehensive inspection of the assembly from different perspectives by comparing the actual 3D model with the target 3D model. The step of comparing with the disclosed method also only needs to be carried out once and not separately for each perspective, as is conventional.

[0026] The use of the 3D model offers, inter alia, compared to the use of the 2D or 2.5D model, the technical effect that in order to check the correctness of a complex assembly part or a complex assembly from several perspectives, only a single target-actual comparison step can be carried out using a single calculated (virtual or digital) model of the assembly to be checked.

[0027] Based on the prior art, the person skilled in the art may therefore be faced with the objective technical task of modifying methods known from the prior art - as described, for example, in US 9,187,188 B2 or US 10,242,438 B2 - such that a complex assembly can be inspected from several perspectives with only a single target-actual comparison step using a single (virtual or digital) model of the assembly to be inspected that is to be calculated.

[0028] As explained in detail above, this is achieved according to the disclosure at least through the use of the 3D model. Such a procedure or the solution according to the disclosure is neither known from the prior art nor is it suggested to the person skilled in the art.

[0029] The above description can be summarized as follows, which is not limiting for the disclosure and refers to a specific embodiment of the teaching according to the disclosure: First, image data of the assembly can be acquired, e.g., in the form of a video stream or one or more individual images. Subsequently, individual parts (of the visible shell) of the assembly can be detected and identified in 2D. This can include a prediction of a 2D object center and a pose (e.g., using a rotation matrix) as well as a 2D segmentation mask in the video image or in the individual image recording. In particular, if a video stream is available, the predicted object position can be optimized based on multiple (perspective) predictions.The most similar CAD model can be selected from a database based on a similarity in an appearance embedding space, and the position of the identified component in relation to other components in the vicinity of the identified component can be detected. This can be repeated for all detected components. The assembly can then be virtually created as a 3D model with all detected components and their poses relative to one another. Finally, the virtual 3D model can be compared with the digital 3D model of the assembly, allowing missing, misaligned, and incorrect components of the assembly to be identified. Possible further developments of the above method are explained in detail below. These further developments, either individually or in combination, at least reinforce the advantages of the method described above.

[0030] The method may include identifying a plurality of components forming the assembly in the image data, and determining an actual position and actual orientation of the identified components relative to each other and / or with respect to a predetermined camera perspective from which the image data was acquired, based on the image data.

[0031] During identification, the presence of an object or component, e.g., a single component and / or an assembly comprising multiple components, can be detected in the image data, and it can be determined what type, or more specifically, which component from a multitude of previously known or predetermined components, it is. Expressed more specifically, the presence can be understood, for example, insofar as the respective component to be identified is visible in a visualization of the image data and can therefore be or should be recognized using an algorithm, e.g., an object recognition algorithm.

[0032] An actual position can be understood as the position, e.g., of a geometric center and / or a center of mass, of a component in space in the real world. An actual orientation can be understood as the orientation of this component in space in the real world. The above description applies analogously to the target position and target orientation, which are contained as information in the target 3D model and can be extracted from it directly or at least indirectly using the method.

[0033] A camera perspective can be understood as a camera's viewing angle of the assembly. The method is not limited to varying the camera perspective; rather, the camera's field of view can be varied in size (i.e., the assembly can be viewed partially and / or in its entirety) and / or the camera's distance from the assembly can be varied (e.g., virtually via a zoom and / or physically by actually decreasing or increasing the camera's distance from the assembly).

[0034] Identifying the components and determining their actual position and orientation has, inter alia, the technical effect of enabling a correctness check for each component comprising the assembly. This means that (as explained in more detail below) the correctness of the assembly can be checked component by component, so that—which is particularly advantageous for complex assemblies—it is not only possible to indicate that the target 3D model as a whole does not match the actual 3D model, but also to identify which components are faulty. This enables targeted rework and / or targeted manual inspection of the assembly (e.g., as part of a guided user interaction).

[0035] It is conceivable that component identification follows a multi-stage approach. This may mean that, in a first step, components that are larger than a predefined threshold are identified. Subsequently, in a second step, objects that are smaller than the predefined threshold can be identified. For this purpose, image data captured at a higher zoom level or with a closer camera distance from the assembly than the image data used in the first step can be used in the second step. The first and second steps can run sequentially or at least partially simultaneously.

[0036] The identification of the multiple components and / or the determination of the position and orientation of the identified components can be performed using an optionally single, artificial intelligence-based model (e.g., comprising one or more artificial neural networks). The model can be trained to identify individual components from a pool of components from the image data of the assembly, optionally simultaneously, and / or to determine their actual position and orientation.

[0037] In more concrete terms, the model can be used, for example, to perform 2D instance segmentation of components available in a pool of possible components. The pool of possible components defines the so-called embedding space. Components can be recognized and segmented using the trained model, which is made possible by prior learning from 2D projections (views) of 3D CAD models of the components. Methods such as multi-object recognition, including the recognition of object poses (i.e., actual position and actual orientation), can be used. The model can be trained in such a way that it can identify the respective components and determine their actual position and actual orientation even when the components are partially occluded in the image data, based on the assembly state of the component during assembly and / or a respective camera perspective.

[0038] It is conceivable that for object pose estimation (or to determine the actual orientation), previously trained keypoints are estimated in views that, for example, describe the object's 3D bounding box. These estimated keypoints can represent an intermediate result that ultimately determines the object's perspective or pose.

[0039] It is conceivable that the model is further designed or trained in such a way that it can determine the uncertainty or ambiguity of a result of identifying and determining the actual position and orientation of the individual components, i.e., for example, the probability that the component and its actual position and orientation were correctly identified. In other words, in addition to identification and / or pose, an ML network can also learn how certain or probable a result is. This can correspond to a reliability value between 0 and 1. For example, a component can be recognized with a high probability from a certain perspective, but less reliably from a different perspective or with partial shading or partial occlusion. This uncertainty can be taken into account when generating the actual 3D model.Specifically, the degree of reliability can be taken into account when object identifications and / or poses are combined from different directions or perspectives. It is conceivable that the result with the highest reliability is used. However, it is also conceivable that a suitable fusion of the various (partial) information on object identifications and / or poses is performed. Furthermore, it is conceivable that the expected object identifications and poses (from the target 3D model) are used, for example, to nevertheless include results with an insufficient confidence value or to confirm their correctness, and optionally to display them with a different format or a predetermined color in the feedback for the user.

[0040] Additionally, the model can be trained to identify objects within or adjacent to components that do not match the expected or corresponding objects in the CAD pool. This allows for further verification of the actual 3D model for correctness, as it can detect incorrectly over-assembled components.

[0041] The determination of the actual orientation and position described above can, in one possible concrete embodiment, be understood as a prediction of the 3D properties of each identified component, which—as already described above—can be performed together with the identification of the component. In addition to a rotation matrix that specifies the orientation of the component in 3D space, a 2D projection of a 3D center of the component onto an image plane of the image data and a shape code vector of the component can be estimated or determined to determine the position of the component. This shape code vector represents an embedding according to a 3D model (e.g., 3D CAD model) that corresponds to the identified component.In other words, by determining the position in the 2D image from different perspectives, the 3D position can be determined. (Pixel-by-pixel) segmentation and object identification using the silhouette allow orientation determination. The shape code vector can correspond to the identification in the embedding space and thus specify the component, which is initially independent of its pose. An estimate of the position of the individual component in space relative to other components or relative to a known camera perspective can also be made.

[0042] An artificial intelligence-based model can be understood as a model generated by machine learning, which in this case is designed to identify individual components from a pool of components based on assembly image data and / or to determine their actual position and orientation. Machine learning can be understood as an "artificial" generation of knowledge from experience, in which an artificial system learns from examples and can generalize them after the learning phase. This means that the examples are not simply memorized, but rather patterns and regularities are recognized in the learning data. This allows the trained model to evaluate even unknown data (so-called learning transfer).

[0043] It is conceivable that the identification and determination of the actual position and orientation could occur simultaneously, i.e., using a single model. This can be achieved through so-called end-to-end learning (E2E), in which all intermediate steps necessary for the result are integrated into a unified model.

[0044] A component pool can be understood as a database containing an identifier (e.g., M12 screw) and an associated 3D model for a specific number of predetermined components. The model can only be specifically trained to recognize the components contained in the component pool in the image data. For this purpose, (synthetic) image data of the respective components can be used to train the model.

[0045] The model can, in particular, be a so-called deep learning model. These generally require a large amount of training data for training, i.e., a large number of 2D CAD pairs, to obtain a robust model capable of generalization. However, realistic 2D CAD pairs with embedded / assembled 3D components can be generated automatically using synthetic rendered images and their augmentations, which leads to an increase in available training data and makes the large amount of training data easier to achieve.

[0046] It should be noted that the disclosure also relates to a method for training the artificial intelligence-based model described herein.

[0047] The use of the artificial intelligence-based model described above offers, inter alia, the technical effect that the method can be used across assemblies, i.e., for assemblies of different types, as long as these assemblies contain or consist of components from the pool of components. This offers flexibility in the use of the method, making it suitable for small series production as well, since a model does not have to be retrained for each new assembly or assembly that differs from previous assemblies.

[0048] The method may include obtaining a respective 3D model for each of the identified components, and determining the actual 3D model of the assembly by assembling the obtained 3D models based on their determined actual positions and actual orientations.

[0049] Assembly can be understood as a digital reconstruction of the assembly, resulting in the actual 3D model. This offers - inter alia - the technical effect that the resulting actual 3D model is compatible with conventionally constructed 3D models, which can subsequently be used as target 3D models. In other words, 3D models are regularly constructed using CAD (computer-aided design) programs. Such 3D models, like the actual 3D model, are composed of individually constructed models of individual components. The compatibility of the actual 3D model with the target 3D model, which originates, for example, from a development phase of the assembly, avoids additional steps of adapting the actual 3D model, and thus the target-actual comparison can be performed with comparatively low computing power.

[0050] Obtaining the 3D model may include loading the respective associated 3D model from the pool of components, which includes a respective associated 3D model for each component of the pool.

[0051] This offers the advantage that the pool of 3D models can be managed and adapted independently, in particular independently of any artificial intelligence-based model that may be used.

[0052] The target 3D model can be composed of 3D models of individual components forming the assembly. Comparing the determined actual 3D model with the target 3D model of the assembly to verify the correctness of the assembly can involve a component-by-component comparison of the 3D models forming the target 3D model with the 3D models forming the actual 3D model.

[0053] Reference is made here to the advantages described above with regard to determining the actual 3D model.

[0054] The component-by-component comparison of the 3D models forming the target 3D model with the 3D models forming the actual 3D model can include a comparison of the respective 3D models of the target 3D model and the actual 3D model itself in order to identify 3D models of individual components that are missing in the actual 3D model compared to the target 3D model and / or 3D models of individual components that are present in the actual 3D model but not in the target 3D model. Additionally or alternatively, the component-by-component comparison of the 3D models forming the target 3D model with the 3D models forming the actual 3D model can include a comparison of a target position and target orientation with the actual position and actual orientation of the respective 3D models of the target 3D model and the actual 3D model in order to identify 3D models of individual components that are arranged differently in the actual 3D model than the target 3D model.

[0055] Specifically, this can mean comparing the actual 3D model, i.e., the virtual 3D replica of the assembly, with the target 3D model, i.e., the digital 3D model of the assembly (e.g., a 3D CAD model of the assembly), including a component-related assessment of correctness. This component-related assessment of correctness can include identifying missing components, identifying misaligned components, and / or identifying incorrect components (based on mismatched features of the 3D models).

[0056] Reference is made here to the advantages described above with reference to the component-by-component comparison of the actual 3D model. Furthermore, this special design offers, inter alia, the technical effect of not only detecting errors regarding the position and orientation of existing components during assembly, but also enabling a completeness check.

[0057] The actual 3D model can be determined taking into account the 3D models contained in the target 3D model itself and / or the target position and target orientation of the 3D models contained in the target 3D model.

[0058] In other words, the assembly of the actual 3D model from the individual loaded 3D models of the identified components can be simplified by taking into account not only the actual orientations and positions known from the image data, but also the desired target positions and orientations of the individual 3D models of the identified components known from the target 3D model. The target 3D model can serve as an initialization; all parts of the target 3D model that were not identified in the actual 3D model can be hidden.

[0059] Determining the actual 3D model may include identifying at least one component of the assembly contained in the image data but incorrectly unidentified based on an initially determined actual 3D model. The method may further include determining an actual position and actual orientation of the at least one component contained in the image data but incorrectly unidentified relative to the remaining identified components and / or with respect to the predetermined camera perspective with which the image data was acquired, based on the image data.The method may include obtaining an associated 3D model of the at least one component contained in the image data but erroneously unidentified, and adapting the initially determined actual 3D model by adding the 3D model of the at least one component contained in the image data but erroneously unidentified based on their determined actual position and actual orientation to obtain the actual 3D model.

[0060] Additionally or alternatively, determining the actual 3D model can include performing a plausibility check of the determined actual positions and the actual orientations of the identified components based on an initially determined actual 3D model. The method can include adapting the initially determined actual 3D model by adapting the positions and orientations of the 3D models contained in the initial actual 3D model based on a result of the plausibility check. In other words, the digital 3D reconstruction described above can initially lead to an unsatisfactory result, the so-called initial actual 3D model, since individual components in the image data may not have been identified at all or their actual position and orientation may have been incorrectly determined initially. In a post-processing step, this can be done, for example, with a plausibility check based on the initial actual 3D model, e.g.due to gaps in the initial actual 3D model and / or overlapping individual 3D models of the actual 3D model. A more detailed analysis of the image data, among other things, can then be performed and / or the initial actual 3D model can be optimized, e.g., based on the target 3D model, in order to obtain the (final) actual 3D model to be used for the target-actual comparison.

[0061] It is also conceivable that appearance-based considerations, such as methods using Neural Radiance Fields (NeRF), could be used to resolve ambiguities in the identification of individual components, e.g., due to glare or unfavorable lighting and / or challenging surface finishes of the assembly.

[0062] Furthermore, it is conceivable that a (local) 3D reconstruction of the assembly could be used if ambiguities in the 2D CAD matching cannot be resolved, i.e., if components contained in the image data cannot be clearly or with sufficient certainty assigned to one of the 3D models in the pool. This can, in particular, include a 3D reconstruction of certain shapes / objects that are otherwise unresolvable. A 3D comparison with the target 3D model can then be performed in various data formats, e.g., by displaying the component as a point cloud, with a mesh, and / or as a voxel grid.

[0063] The method can comprise capturing image data of the assembly, advantageously from multiple perspectives. In this regard, reference is first made to the above explanations regarding the image data and its capture. Furthermore, capturing image data from multiple or different perspectives has the advantage over capturing image data from a single perspective, i.e., compared to, for example, a single image, that an ambiguity regarding the size of components along an image axis resulting from a projection from 3D to 2D can be resolved. It is conceivable that the method, additionally or alternatively, comprises determining a respective perspective (e.g., using SLAM (Simultaneous Localization and Mapping) and / or odometry (e.g., motion data integrated from IMU acceleration data)) from which the image data was recorded or captured.

[0064] The method may comprise outputting information comprising a result of comparing the determined actual 3D model with the target 3D model, optionally audibly and / or visually, optionally by controlling a display device.

[0065] Specifically, this can mean that feedback is output to a user. For this purpose, an interface, e.g. in the form of a display, can be provided, which can be controlled according to the disclosure. Optionally, feedback of evaluation values ​​from the 3D comparison described above can be output, e.g. by means of a visualization for the human user, e.g. comprising a recommendation for rework and / or additional human testing. However, the interface is not limited to such a human machine interface (HMI), but can additionally or alternatively be a data interface, via which a result of the target-actual comparison can be provided to a data processing device that is connected to the interface, e.g. wired and / or wirelessly.

[0066] The possibility of interacting with the user is not just limited to the presentation of the result of the process, but the user can be supported by appropriate information output, for example as part of guided user interaction when capturing the image data, e.g. by displaying advantageous camera positions, etc. It is conceivable that a hierarchical approach will be implemented in this way. This can expand on the approach described above in that it can create a better view of hidden components of the assembly and / or options for zooming in on small or not easily identifiable components of the assembly. For this purpose, the uncertainty metrics described above from the output of the artificial intelligence-based model can be used to interact with the user about which parts should be scanned again with modified or missing information after an initial acquisition of image data.adjusted image acquisition settings (e.g., changed perspective or zoom or distance of the camera from the assembly) are to be captured during a second or subsequent image acquisition. In other words, since the comparison of the 3D data depends significantly on the actual 3D model, which may contain different confidence values / reliability scores, it is conceivable that the actual 3D model could be prepared as a color-coded visualization (e.g., green for all components with a confidence value greater than 90%, yellow for all components with confidence values ​​up to 65%, and red for all other components not detected or detected with deviations). A results list with percentages would also be conceivable.

[0067] Furthermore, it should be noted that although the method described above is aimed at the use of a 3D model, it does not exclude the additional use of contour-based testing methods known from the prior art, such as those described in US 9,187,188 B2 and US 10,242,438 B2.

[0068] Furthermore, the disclosure relates to a data processing device which is designed to at least partially carry out the method described above.

[0069] The data processing device may be a computer. The data processing device may be a distributed system, or several individual, physically separate data processing devices may be connected together, e.g., via the Internet, and execute the method jointly.

[0070] The data processing device may be part of a data processing system which, in addition to the data processing device, has one or more sensors for capturing the image data and / or one or more interfaces, e.g. for user interaction.

[0071] It is conceivable that an embedding space is kept centrally on a server, while only object shape vectors are exchanged, which allow identification of the components.

[0072] Furthermore, the disclosure relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to at least partially carry out the method described above.

[0073] The computer program may be software, e.g., comprising firmware and / or application software. Firmware can be understood as software that is (permanently) embedded in electronic devices, such as, in this case, a sensor for image data acquisition, and performs basic functions there. The firmware can occupy an intermediate position between the hardware of the device (i.e., the physical components of the device) and any existing application software (the possibly replaceable programs of the device). The application software, which carries out the disclosed evaluation of the acquired image data, can be executable on a computer connected to the sensor, e.g., a personal computer, a tablet, a smartphone, etc.

[0074] Furthermore, the disclosure relates to a computer-readable medium, e.g., a computer-readable storage medium and / or a data signal, comprising instructions that, when executed by a computer, cause the computer to at least partially execute the method described above. In particular, a computer-readable medium comprising the computer program described above can be provided.

[0075] The computer-readable medium can be any digital data storage device, such as a USB stick, a hard drive, a flash memory, a CD-ROM, an SD card or an SSD card.

[0076] The computer program does not necessarily have to be stored on such a computer-readable storage medium in order to be made available to the computer, but can also be obtained as a data signal via the Internet or otherwise externally.

[0077] An optional embodiment is described below with reference to Figures 1 and 2.

[0078] Fig. 1 schematically shows a data processing system designed to carry out a computer-implemented method according to the disclosure, which in turn is designed to check the correctness of an assembly, and

[0079] Fig. 2 shows a schematic flow diagram of this process.

[0080] Figure 1 schematically shows a side view of an assembly 1 comprising, in this case, a plurality of individual components 2-5. Furthermore, Figure 1 schematically shows a sensor 6 configured to capture image data of the assembly 1 (FoV shown with a dashed line), a computer 7 connected to the sensor 6, and a display 8 connected to the computer 7. The latter units 6-8 are part of a data processing system configured to carry out the method described in detail below. This method is a computer-implemented method configured to check the correctness of the assembly 1. A flowchart of the method is schematically shown in Figure 2.

[0081] As can be seen from Figure 2, the process can be roughly divided into four steps S1 -S4.

[0082] In a first step S1 of the method, image data of the assembly 1 (optionally from several perspectives, here in Figure 1 for simplicity only one perspective is shown) are captured by means of the sensor 6, e.g. a camera, and output to the computer 7.

[0083] In a second step S2 of the method, an actual 3D model of the assembly is determined based on the image data of the assembly 1 acquired in the first step S1 using a computer program stored in a memory 72 of the computer 7, which is executed by a processor 71 of the computer 7. The second step S2 is explained in further detail below.

[0084] In a first sub-step S21 of the second step S2, the components 2-5 that form the assembly are identified in the image data recorded by the sensor 6, and an actual position and actual orientation of the identified components 2-5 are determined based on the image data. The actual position and actual orientation of the identified components 2-5 can be determined, for example, relative to one another and / or with respect to a predetermined camera perspective with which the image data was acquired, in particular an image axis Z.

[0085] The identification of the multiple components 2-5 and the determination of the actual position and orientation of the identified components 2-5 are performed in the first sub-step S21 using a single model based on artificial intelligence. The model is trained and thus configured to simultaneously identify individual components 2-5 of the assembly 1 from the image data of the assembly 1 that correspond to components from a pool of components, and to determine their actual position and orientation.

[0086] In a second sub-step S22 of the second step S2, a respective associated 3D model is obtained for each of the identified components 2 - 5 by loading the respective associated 3D model from the pool of components which is stored in the memory of the computer 7 and which comprises a respective associated 3D model for each component of the pool.

[0087] In a third sub-step S23 of the second step S2, an initial or first actual 3D model of the assembly 1 is determined by assembling the related 3D models based on their determined actual positions and actual orientations, as well as taking into account 3D models contained in a target 3D model of the assembly and / or the target position and target orientation of the 3D models contained in the target 3D model (which in turn can refer to the 3D models corresponding to the individual components 3-5 relative to one another or (absolutely) to the camera perspective). The target 3D model is composed of 3D models of individual components 2-5 forming the assembly and includes information about the respective target position and target orientation of these 3D models.

[0088] More precisely, an actual position and actual orientation of the respectively loaded 3D model in the actual 3D model (which in turn can refer to the 3D models corresponding to the individual components 3-5 relative to one another or (absolutely) to the camera perspective) can be set equal to the determined actual position and actual orientation of the respective, determined individual components 2-5. The target 3D model can be taken into account, for example, in that actual positions and actual orientations of 3D models in the first actual 3D model that deviate slightly or within a predetermined tolerance range are adjusted based on the target positions and target orientations of the individual 3D models contained in the target 3D model, in order to generate a coherent first actual 3D model. This generated initial or first actual 3D model can then be (further) optimized as described below.

[0089] For this purpose, in a fourth sub-step S24 of the second step S2, based on the initially determined actual 3D model, it is checked whether a component 4 of the assembly 1 contained in the image data but incorrectly not identified is present. If so, it is identified (see first sub-step S21 of the second step S2). Here, it is assumed, for example, that component 4 was not identified or was not recognized with sufficient reliability. To identify the unidentified component 4 of the assembly 1, the initial actual 3D model can, for example, be subjected to a plausibility check.The first actual 3D model can, for example, be compared with a target 3D model, as described in detail below with reference to a third step S3 of the method, and if it is determined that component 4 is not contained in the actual 3D model but is contained in the target 3D model, the image data can be examined again specifically for component 4. If this verification is sufficiently successful (i.e., has a sufficiently high confidence value), this component 4 can be treated in the same way as the previously clearly identified components. This is only one example of a plausibility check, and other methods are also conceivable.

[0090] In a fifth sub-step S25 of the second step S2, an actual position and actual orientation of the erroneously unidentified component 4 relative to the remaining identified components 2, 3, 5 and / or with respect to the predetermined camera perspective with which the image data were acquired are determined based on the image data (see first sub-step S21 of the second step S2).

[0091] In a sixth sub-step S26, an associated 3D model of the incorrectly unidentified component 4 is obtained (see second sub-step S22 of the second step S2). In a seventh sub-step S27, the initially determined actual 3D model is adapted by adding the loaded 3D model of the incorrectly unidentified component 4 based on its determined actual position and actual orientation (see third sub-step S23 of the second step S2) to obtain a second actual 3D model.

[0092] The inspection of the initial actual 3D model was described above for the sake of completeness. Additionally or alternatively, a check can also be performed to determine whether the identified components 2–5 of assembly 1 are correctly arranged in the generated actual 3D model. In this case, this is done based on the second, adjusted actual 3D model. However, it would also be conceivable to perform the eighth and ninth sub-steps S28 and S29 of the second step S2, described below, based on the first or initially generated actual 3D model.

[0093] In the eighth sub-step S28 of the second step S2, a plausibility check is performed on the determined actual positions and the actual orientations of the identified components 2-5 based on an initially determined actual 3D model. More specifically, the second actual 3D model can be compared with the target 3D model, for example, as described in detail below with reference to the third step S3 of the method. If it is determined that a 3D model of one of the components 2-5, e.g., the 3D model of component 4, is arranged differently from the 3D model of component 4 in the target 3D model, the image data can be examined again specifically for this component 4. This is only one example of the plausibility check, and other methods are also conceivable.An adjusted actual position and actual orientation of component 4, and thus an adjusted actual position and actual orientation of the 3D model corresponding to component 4 in the second actual 3D model, can then be determined in the manner described above (see third sub-step S23 of the second step S2). In the ninth sub-step S29 of the second step S2, the second determined actual 3D model is adjusted by adjusting the actual positions and actual orientations of the 3D models contained in the second actual 3D model based on a result of the eighth sub-step S28. In the present example, this can mean that the actual position and actual orientation of the 3D model corresponding to component 4 in the second actual 3D model are updated based on the adjusted actual position and actual orientation of said second 3D model determined in the eighth sub-step S28 of the second step S2. This results in a final orthird actual 3D model, which is hereinafter referred to as the actual 3D model.

[0094] In the third step S3 of the method, the actual 3D model determined in the second step S2 is compared with the target 3D model of the assembly 1 stored in the memory 72 to check the correctness of the assembly 1, again using the computer program stored in the memory 72 and executed by the processor 71, component by component, i.e., 3D model by 3D model of the two 3D models of the assembly 1. In other words, comparing the determined actual 3D model with the target 3D model of the assembly 1 to check the correctness of the assembly 1 comprises a component-by-component comparison of the 3D models forming the target 3D model with the 3D models forming the actual 3D model. For this purpose, the target 3D model is, as described above, composed of 3D models of individual components 2 - 5 forming the assembly and includes the information regarding the target orientation and the target position of the individual 3D models.The third step, S3, is explained in more detail below. Optionally, some of the results from the optimization steps S24 - S29 described above can be used.

[0095] In a first sub-step S31 of the third step S3, ie the component-wise comparison, a comparison of the respective 3D models of the target 3D model and the actual 3D model itself is carried out in order to identify 3D models of individual components 2 - 5 that are missing in the actual 3D model compared to the target 3D model and / or 3D models of individual components 2 - 5 that are present in the actual 3D model but not in the target 3D model.

[0096] In a second sub-step S32 of the third step S3, ie the component-wise comparison, the target position and target orientation are compared with the actual position and actual orientation of the respective 3D models of the target 3D model and the actual 3D model in order to identify 3D models of individual components 2 - 5 that are arranged differently in the actual 3D model than the target 3D model.

[0097] In a fourth step S4 of the method, information comprising a result of comparing the determined actual 3D model with the target 3D model, ie the result of the third step S3, is output visually in this case by controlling the display device 8 (comprising generating a corresponding control signal by means of the computer program stored in the memory 72 and executed by the processor 72).

[0098] The steps S1 - S4 and sub-steps S21 - S29, S31, S32 described above represent one possible sequence of the method, although deviating sequences are also conceivable. The individual steps S1 - S4 and sub-steps S21 - S29, S31, S32 do not necessarily have to be implemented as separate steps S1 - S4 and sub-steps S21 - S29, S31, S32 using software. Rather, several steps S1 - S4 and / or sub-steps S21 - S29, S31, S32 can be combined. For example, the sub-steps S24 - S26 of the second step S2 can be combined into a single (sub-)step. List of reference symbols

[0099] 1 Assembly

[0100] 2 - 5 Components of the assembly 6 Sensor, e.g. camera

[0101] 7 computers

[0102] 71 processor

[0103] 72 storage

[0104] 8 Display device, e.g. display

[0105] S1 - S4 process steps

Claims

Patent claims 1. A computer-implemented method designed to check the correctness of an assembly (1), characterized in that the method comprises: - determining (S2) an actual 3D model of the assembly (1) based on image data of the assembly (1), and - comparing (S3) the determined actual 3D model with a target 3D model of the assembly (1 ) to check the correctness of the assembly (1 ).

2. Computer-implemented method according to claim 1, characterized in that the method comprises capturing (S1) the image data of the assembly (1), preferably from several perspectives.

3. Computer-implemented method according to claim 1 or 2, characterized in that the method comprises: - identifying (S21) a plurality of components (2 - 5) forming the assembly (1) in the image data, and - Determining (S21) an actual position and actual orientation of the identified components (2 - 5) relative to one another and / or with respect to a predetermined camera perspective with which the image data were acquired, based on the image data.

4. Computer-implemented method according to claim 3, characterized in that the identification (S21) of the plurality of components (2 - 5) and / or the determination of the actual position and actual orientation of the identified components (2 - 5) is carried out by means of an optionally single model based on artificial intelligence, which is trained to identify, optionally simultaneously, individual components (2 - 5) from a pool of components from the image data of the assembly (1). to identify and / or determine their actual position and orientation.

5. Computer-implemented method according to claim 3 or 4, characterized in that the method comprises: - Obtaining (S22) a corresponding 3D model for each of the identified components (2 - 5), and - Determining (S23) the actual 3D model of the assembly (1) by assembling the related 3D models based on their determined actual positions and actual orientations.

6. Computer-implemented method according to claim 5, as far as dependent on claim 3, characterized in that the obtaining (S21) of the 3D models comprises loading the respectively associated 3D model from the pool of components, which comprises a respectively associated 3D model for each component of the pool.

7. Computer-implemented method according to claim 5 or 6, characterized in that: - the target 3D model is composed of 3D models of individual components (2 - 5) forming the assembly, and - comparing (S3) the determined actual 3D model with the target 3D model of the assembly (1) to check the correctness of the assembly (1) comprises a component-by-component comparison of the 3D models forming the sol I 3D model with the 3D models forming the actual 3D model.

8. Computer-implemented method according to claim 7, characterized in that the component-wise comparison of the 3D models forming the target 3D model with the 3D models forming the actual 3D model comprises: - comparing (S31) the respective 3D models of the target 3D model and the actual 3D model itself, in order to identify 3D models of individual components (2 - 5) that are missing in the actual 3D model compared to the target 3D model and / or 3D models of individual components (2 - 5) that are present in the actual 3D model but not in the target 3D model, and / or - Comparing (S32) a target position and target orientation with an actual position and actual orientation of the respective 3D models of the target 3D model and the actual 3D model in order to identify 3D models of individual components (2 - 5) that are arranged differently in the actual 3D model than the target 3D model.

9. Computer-implemented method according to claim 7 or 8, characterized in that the determination (S2) of the actual 3D model takes place taking into account the 3D models contained in the target 3D model and / or the target position and target orientation of the 3D models contained in the target 3D model.

10. Computer-implemented method according to one of claims 5 to 9, as far as dependent on claim 2, characterized in that the determination (S2) of the actual 3D model comprises: - Identifying (S24) at least one component (4) of the assembly (1) contained in the image data but incorrectly not identified based on an initially determined actual 3D model, - Determining (S25) an actual position and actual orientation of the at least one component (4) contained in the image data but erroneously not identified relative to the remaining identified components (2, 3, 5) and / or with respect to the predetermined camera perspective with which the image data was acquired, based on the image data, - obtaining (S26) an associated 3D model of the at least one component (4) contained in the image data but incorrectly not identified, and - adapting (S27) the initially determined actual 3D model by adding the 3D model of the at least one component (4) contained in the image data but erroneously not identified based on their determined actual position and actual orientation in order to obtain the actual 3D model.

11. Computer-implemented method according to one of claims 5 to 10, as far as dependent on claim 2, characterized in that the determination (S2) of the actual 3D model comprises: - Carrying out (S28) a plausibility check of the determined actual positions and the actual orientations of the identified components (2 - 5) based on an initially determined actual 3D model, - Adjusting (S29) the initially determined actual 3D model by adjusting actual positions and actual orientations of the 3D models contained in the initial actual 3D model based on a result of the plausibility check.

12. Computer-implemented method according to one of claims 1 to 11, characterized in that the method comprises outputting (S4) information comprising a result of comparing the determined actual 3D model with the sol I 3D model, optionally audibly and / or visually, optionally by controlling a display device (8).

13. Device for data processing (7), characterized in that the device (7) is designed to carry out the method according to one of claims 1 to 12.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to one of claims 1 to 12.

15. A computer-readable medium comprising instructions which, when executed by the Commands by a computer cause it to carry out the method according to one of claims 1 to 12.

Images