Method and device for detecting missing joining elements in 3D vehicle geometry data of a digital twin for vehicle construction

Abstract

The invention relates to a method and a device (100) for detecting missing connection elements (40') in the connection element counterparts (22, 32) of a digital twin (10) of the physical structure of a vehicle under investigation. One or more 2D images of a part of the vehicle (20) (S200) are generated from one or more spatial directions of the digital twin (10) by rendering digital 3D geometry data of the vehicle. The resulting 2D image is fed into a trained detection model (130), which is trained to recognize the unoccupied connection element counterparts (22').The method also allows the detection (S500) of the absence of a connecting element (40') in a 3D position of the digital twin (10) for use in vehicle construction based on the recognition (S400) of a missing connecting element (40') in the input 2D image by the trained detection model (130), the vehicle's 3D digital geometry data, the spatial direction and the recognized position in the 2D image.

Description

[0001] The invention relates to a method and a device for detecting missing connecting elements by means of image recognition and image processing in 3D vehicle geometry data of digital twins for the construction of vehicles.

[0002] US Patent 2015 / 0356209 A1 describes a geometric modeling system in which a user can employ CAD software to implement geometric modules for multiple components. A connector module and a pairing module are described for identifying a first and second component and adding connections to the components by identifying matching pairs.

[0003] US Patent 2009 / 0198464 A1 discloses a method for inspecting assemblies in which a digital image of an assembled product is obtained and one or more objects are extracted from the digital image of the assembled product, each of which is recognized as a component based on its extracted image and a library of standard components. One or more features of each recognized component are identified and compared to features of a standard component to detect an assembly defect if the identified features do not correspond to the relevant standard features.

[0004] In the publication LI HAO ET AL: "A detection and configuration method for welding completeness in the automotive body-in-white panel based on digital twin", SCIENTIFIC REPORTS, Vol. 12, No. 1, May 13, 2022 (2022-05-13), XP093277553, US ISSN: 2045-2322, DOI: 10.1038 / s41598-022-11440-0, found on the Internet: URL:https: / / www.nature.com / articles / s41598-022-11440-0> * Page 1 - Page 10; Figures 1, 6; Table 1*, a method for detecting missing fasteners of a real panel based on real images as image input is carried out.

[0005] Vehicles are physically constructed. Their planning and design now almost exclusively utilize digital 3D vehicle geometry data. A vehicle consists of many individual components with connecting elements, each designed by separate trades and then digitally / virtually assembled in subsequent steps. The result is a digital twin of the planned physical structure. Based on this digital twin, the physical structure is created, manufactured, and assembled.

[0006] If errors occur in the digital twin, they will only become apparent during the physical assembly process. However, making changes at this stage is expensive and time-consuming, as parts may have been incorrectly designed or simply not procured / manufactured and are therefore unavailable during assembly. They may also be present in the wrong quantity (too few, too many). It is therefore crucial to have a digital twin that is as accurate and error-free as possible to ensure a correct and trouble-free physical assembly. Since its creation is distributed across multiple processes, there are countless points at which errors can occur, such as duplicate, incorrect, or forgotten vehicle parts.

[0007] In existing solutions, the digital geometry data of the digital twin is manually checked by experts virtually "flying" through the virtual vehicle and identifying and resolving problems based on their experience, or contacting and consulting individual trades as needed. Otherwise, errors such as missing fasteners are only detected and corrected later, i.e., during the physical assembly of the machine. This is correspondingly time-consuming and expensive, and prevents a smooth process. Furthermore, these existing solutions are subjective and heavily reliant on experience and perception. This leads to significant variations in the quality of the checks due to a lack of experience or concentration. Additionally, checks for missing fasteners can only be performed on a sample basis. Therefore, problems are inevitably overlooked.

[0008] However, it is often impossible to intervene in the process, as individual trades have their own working methods and requirements. Furthermore, the same problems arise across all trades, such as lack of experience and lack of concentration. Inaccurate coordination between trades can also lead to problems, for example, the designer for vehicle part A believes that designer B is planning connecting elements between the vehicle parts, and designer B relies on designer A.

[0009] The invention is based on the objective of developing a method for detecting missing connecting elements in 3D vehicle geometry data of a digital twin, which has high accuracy and high efficiency in order to accelerate vehicle assembly. Furthermore, an objective of the invention is to efficiently correct this digital twin so that vehicle assembly can be carried out without unnecessary delays.

[0010] The subject matter of the invention is defined by the attached claims.

[0011] In one embodiment of the invention, a method according to claim 1 is provided.

[0012] Further preferred embodiments of the invention result from the other features mentioned in the dependent claims.

[0013] The invention has the advantage that, using the trained detection model, missing connecting elements can be automatically and with high accuracy detected in a 3D digital twin via image recognition. During rendering, the 3D vehicle geometry data is converted into 2D images, allowing the trained detection model to advantageously perform optical recognition based on these 2D images. The trained detection model thus operates solely optically with 2D images, and the detections are projected back, thereby also determining the position and orientation within the 3D twin. Furthermore, the present trained detection model does not exhibit a reality gap, as training and application take place in the synthetic domain. The information gained from the detection of missing connecting elements will lead to a reduction in design time during vehicle construction.

[0014] A digital twin is a digital 3D representation or model of a vehicle, consisting of several vehicle components. A digital twin used for vehicle assembly should contain all structural parts required for construction. A fastener is typically a screw, but it can also be a plug, clip, or similar component. A counterpart to a fastener is, for example, a hole or the mating point of another fastener. The counterpart to a fastener can also be called a connection point. Position in 3D space can encompass location and orientation. The 3D geometry data can be stored, for example, in a *.jt file format. The trained detection model represents a model of artificial intelligence (AI).

[0015] In a preferred embodiment of the invention, the method comprises generating, for each vehicle part, a 2D image from a spatial direction of the digital twin by rendering the digital 3D vehicle geometry data, wherein the 2D image includes connecting elements and connecting element counterparts. Furthermore, the method comprises detecting the absence of a connecting element at a 3D position of the digital twin corresponding to the physical assembly of the vehicle when the trained detection model recognizes an unoccupied connecting element counterpart and its position in the 2D image. The method further comprises determining the 3D position of the detected unoccupied connecting element counterpart by backprojection using the digital 3D vehicle geometry data, the spatial direction, and the recognized position in the 2D image.Furthermore, the process can include the digital merging of the various vehicle components of the digital twin using digital 3D vehicle geometry data. This approach has the advantage that the detection of the corresponding fastener in 2D directly (and simultaneously) corresponds to the detection of a missing fastener. This is because fasteners containing fasteners are not (and cannot be) detected. Thus, only those fasteners indicating the absence of a fastener are detected. This increases the performance and reduces the error rate of the detection.

[0016] In a preferred embodiment of the invention, a 2D image of each vehicle part in the digital twin is generated from a specific spatial direction by rendering the digital 3D vehicle geometry data after removing connecting elements. The method further comprises, for each vehicle part, using the trained detection model for identifying unoccupied connecting element counterparts based on 2D images, at least one unoccupied connecting element counterpart and its position in the generated 2D image. The method further comprises determining the 3D position of the at least one identified unoccupied connecting element counterpart by backprojection using the digital 3D vehicle geometry data, the identified position in the 2D image, and the spatial direction.The process further includes the digital merging of the various vehicle components of the digital twin using the digital 3D vehicle geometry data. It also includes detecting the absence of a fastener at its counterpart by comparing it to a digital test specimen at the counterpart location. In this case, all counterpart locations are identified by the trained detection model through the removal of the fasteners during the rendering process. This necessitates a subsequent comparison to test whether the counterpart location is actually occupied, which can be performed by comparing it to a digital test specimen in the digital twin or the 3D vehicle geometry data. If no comparison is detected, the missing fastener has been identified.

[0017] In a preferred embodiment of the invention, the digital joining of different vehicle parts can be achieved based on the 3D position of the at least one detected unoccupied connecting element counterpart by pairing or forming pairs of unoccupied connecting element counterparts of different vehicle parts. This can also be done by fitting a distance-representing or spatially descriptive structure with all central points. Each central point exists only once per vehicle part. By forming pairs using the 3D positions, it is possible to infer which vehicle parts belong together. In preferred embodiments, similarity in geometric information, such as matching hole sizes, can also be used. KD-Tree can be employed. In this case, closely spaced central points are assigned the same ID, and vehicle parts with the same ID represent a connection.

[0018] In a further preferred embodiment of the invention, the method comprises generating digital bounding boxes, for example, of connecting elements of the vehicle parts of the digital twin from the digital 3D vehicle geometry data. The method further comprises detecting the absence of a connecting element at its counterpart by comparing a cross-sectional view of the digital bounding box with the identified counterpart. This allows for rapid cross-sectional inspection. The spatial position of all connecting counterparts is intersected with the digital bounding boxes of the connecting elements in 3D. The cross-sectional geometry can be determined. A cross-sectional view indicates the presence of a connecting element, while a cross-sectional view indicates the absence of a connecting element. This method is a fast and reliable test that generates reliable detections. The digital bounding boxes of the connecting elements, e.g.,The bounding boxes can be derived from the position of existing and known connecting elements, e.g. from lists such as a standard parts list and using the 3D geometry data.

[0019] In a further preferred embodiment, the method comprises generating a multitude of 2D images from different spatial directions for each vehicle part from the digital 3D vehicle geometry data. The method further comprises inputting the multitude of 2D images into the trained detection model to identify unoccupied fastener counterparts based on the input 2D images. The method also includes identifying, for each of the 2D images of a vehicle part, a fastener counterpart and its position within the 2D image using the trained detection model. Finally, the method includes determining 3D positions for each of the identified fastener counterparts for each 2D image per vehicle part by backprojection based on the digital 3D vehicle geometry data and the identified 2D position.Furthermore, the method includes clustering the determined 3D positions and identifying the 3D position of a detected counterpart as the central point of each cluster. This method increases the accuracy of determining the 3D position of a detected counterpart, thereby improving the accuracy of the entire detection process. Sampled points can be backprojected using various methods (e.g., K-medoids) to remove outliers, and additional features can be acquired (e.g., via plane fitting). The final points can be clustered and optionally filtered again (e.g., via DBSCAN (Density-Based Spatial Clustering of Applications with Noise)), and previously acquired features (e.g., weights) can also be used.For example, confidence values ​​from the estimation results of the trained detection model can be used to determine weights. The confidence value represents a measure of the certainty of the location of the detected connecting element counterpart as part of the detection result. These confidence values ​​can then be used as additional features or as weights for the final determination of the location during clustering. The result is highly accurate location descriptions of potential connection points in 3D for each machine part. The central point of a cluster can, for example, be weighted Euclideanically (i.e., by distance), e.g., with a confidence level of [value missing in original text].

[0020] In a further preferred embodiment, the method also includes filtering 3D positions based on confidence values ​​from the estimate of the trained detection model, as described above, and / or distance information of the predicted 3D positions of connecting element counterparts per solid angle, in order to remove outliers. Prior filtering and removal of outliers can statistically improve clustering and thus increase the accuracy of determining the 3D position. For example, DBSCAN can be used as a filter.

[0021] In a further preferred embodiment, the method comprises generating a 3D point cloud from the digital 3D vehicle geometry data of the digital twin. Furthermore, the method includes determining the 3D position of the at least detected counterpart of the connecting element by backprojection based on the 3D point cloud, the spatial direction, and the detected position in the 2D image. The 3D point cloud allows for particularly efficient projection from 2D information at the image plane to 3D information by utilizing the previously generated 3D point clouds for high performance. Using the 3D point cloud and the detected position in the 2D image, a rapid backprojection to the 3D position is thus possible. This is achieved, in particular, by sampling the 3D point cloud, e.g., via complete dimensions of the bounding rectangles of the detected counterpart of the connecting element in the 2D image.

[0022] The process further includes correcting the digital twin based on the detection of a missing connector by adding the missing connector at its 3D position in the digital twin and / or displaying information about the missing connectors. The result is, for example, a message indicating the location of missing connectors, which can then be used to correct the digital twin. This information can also be embedded in a graph description (e.g., a vehicle graph). The graph can be designed so that the edges represent the connection points or connectors between the vehicle components (nodes). This enables partial automation through manual correction of the identified missing connector counterparts.Furthermore, full automation is also conceivable by determining a connecting element according to its type and placing and correcting it in the digital 3D vehicle data.

[0023] In a further preferred embodiment of the invention, the method comprises determining the orientation of detected opposing connection element counterparts of assembled different vehicle parts as a vector between the identified unoccupied connection element counterparts. Furthermore, the method comprises adding the missing connection element to the digital twin based on the orientation of the connection element counterparts. The orientation can be derived from the pairing process during the digital merging of opposing unknown connection element counterparts. This enables full automation by placing and correcting a connection element, e.g., one of a specific type, within the digital machine data. Alternatively, an intermediate user check can be performed.

[0024] In a further preferred embodiment of the invention, the method comprises issuing a control command to a robot or actuator to control the robot or actuator based on the corrected digital twin or the twin containing information about the missing connecting elements at the detected, unoccupied connecting element counterparts for assembling the vehicle. For example, the control command to attach the missing connecting element can be based on the detections described above.

[0025] In a further preferred embodiment of the invention, the method comprises determining the type of missing fastener. The method includes acquiring geometric information at a detected unoccupied fastener counterpart for which a missing fastener has been detected. The method further comprises recognizing, using a trained classifier, the type of missing fastener based on the geometric information of the fastener counterparts. Furthermore, additional information can also be used, such as the directional information of the fastener (vector of the orientation of the fastener, such as a screw), the plane fitting, the extent of the cluster, or the detected size of the 2D detection.Furthermore, the process includes displaying the type of missing connector based on its detection on a screen, or automatically correcting the digital twin by adding the missing connector based on its type. In this variant, full automation is possible using the classifier. This accelerates the (image) processing of the 3D digital twin with the automatic correction step. The trained classifier represents a model of artificial intelligence (AI).

[0026] In a further preferred embodiment of the invention, the method comprises identifying a set of vehicle parts of the digital twin (excluding fasteners) whose digital envelopes contain the identified fastener counterpart or are located within a predetermined maximum distance. In a further step, a subset of one or more vehicle parts that do not have a previously identified fastener counterpart is determined from the multitude of identified vehicle parts (or the vehicle parts that do have fastener counterparts to be tested are removed). In a further step, a digital test object with a predetermined geometry is placed at the identified fastener counterpart to be tested.In a further step, the system detects that the identified fastener counterpart is not actually a fastener counterpart when the digital test piece cuts a vehicle part from the identified subset of vehicle parts. Based on the cut, a distinction can be made between contact points that only appear to be fastener counterparts and actual connection interfaces. For example, a contact point might contain two holes and a cable passing through them. This means that there are some locations in vehicles that appear to be fastener counterparts based on their features, but are not actually fastener counterparts, although they always have a certain geometry. Thus, false-positive fastener counterparts that are not actually fastener counterparts can be excluded.This increases the sensitivity of the detection of missing fasteners through subsequent geometric verification of the detected fastener counterpart.

[0027] In a further preferred embodiment of the invention, the method comprises dividing a vehicle part to be examined into several sub-segments from the digital 3D vehicle geometry data. The method further comprises generating, for each sub-segment, a 2D image from one spatial direction per sub-segment of the vehicle part of the digital twin by rendering the digital 3D vehicle geometry data. The method includes inputting the 2D image into the trained detection model to recognize unoccupied connecting element counterparts and their position in the 2D image based on the input 2D images.The process further includes detecting the absence of a connecting element at a 3D position of the digital twin relative to the physical assembly of the vehicle. This is achieved by recognizing an unoccupied connecting element counterpart in the input 2D image of the sub-segment using the trained detection model, the digital 3D vehicle geometry data, the spatial direction, and the detected position in the 2D image. This allows for the creation of high-resolution models of even larger vehicle parts, as the individual sub-segments can be decomposed to achieve a higher resolution than the entire part.

[0028] In a further preferred embodiment of the invention, the method includes displaying the detected missing fasteners on a screen. The method further includes capturing user feedback, including information about confirmation or non-confirmation via a feedback button in a user interface. The method also includes adapting the trained detection model for recognizing fastener counterparts in response to the user feedback. Thus, retraining based on user-verified data is possible. This allows the trained detection model to be improved in the application. This feature provides a multi-level improvement (MLOps). For example, weights can be adjusted.

[0029] In a further preferred embodiment of the invention, the method comprises generating a graph-based data structure for the physical assembly of the vehicle, wherein the graph-based data structure includes vehicle parts of the digital twin as nodes or vertices and the connecting element counterparts or connecting elements recognized by the trained detection model as edges, i.e., connections lying between the nodes. Furthermore, the position of nodes and edges in the graph, or in the data structure, can correspond to the determined spatial 3D position. With this data structure, assembly sequences for the physical assembly of the vehicle can be easily derived and carried out. The graph-based data structure can be used for algorithmic, computational purposes, etc. In one embodiment, this graph-based data structure can also be displayed visually on a screen.

[0030] In a further preferred embodiment, a device for detecting missing fasteners at counterparts of digital twins of a vehicle's physical structure is described. The device comprises at least one processor and an operationally connected memory containing executable instructions, wherein the at least one processor is configured to execute the steps of the method by carrying out the instructions as described above. The instructions can be used by one or more programs, software, and / or applications.

[0031] In a further embodiment of the invention, a computer program or computer program product is described, comprising instructions which, when the program is executed by a computer, cause it to perform the above process steps.

[0032] In a further preferred embodiment, a computer-readable (storage) medium is described, comprising instructions which, when executed by a computer, cause it to carry out the procedure described above.

[0033] The electronic or electrical devices and / or other relevant devices or components according to the embodiments of the present invention described herein can be implemented using any suitable hardware, firmware (e.g., an application-specific integrated circuit), software, or a combination thereof. For example, the various components of these devices can be located on an integrated circuit (IC) or on separate IC chips. Furthermore, the various components of these devices can be a process or thread running on one or more processors in one or more computer devices, executing computer program instructions, and / or interacting with other system components to perform the various functions described herein. The computer program instructions are stored in a memory located in a computer device using standard memory, such as...The computer program instructions can be implemented in main memory (RAM). They can also be stored on other non-transferable, computer-readable media, such as a CD-ROM, a flash drive, or similar devices.

[0034] Unless otherwise stated in individual cases, the various embodiments of the invention mentioned in this application can be advantageously combined with one another.

[0035] The invention is explained below using exemplary embodiments with reference to the accompanying drawings. These show: Figure 1: A method for detecting missing fasteners at fastener counterparts of a digital twin to be examined, relating to the physical structure of a vehicle according to an embodiment of the invention; Figure 2: A device for detecting missing fasteners at fastener counterparts of a digital twin to be examined, relating to the physical structure of a vehicle according to an embodiment of the invention; Figure 3: A visual representation of a digital twin; Figure 4: A method for generating the trained detection model according to an embodiment of the invention; Figure 5: A method for detecting missing fasteners at fastener counterparts of a digital twin to be examined, relating to the physical structure of a vehicle according to embodiments of the invention;Figure 6 shows a method for detecting missing fasteners at fastener counterparts of a digital twin of the physical structure of a vehicle under investigation, according to further embodiments of the invention; Figure 7 shows a graphical representation of the method for generating 2D images according to embodiments of the invention; Figure 8 shows a graphical representation of the method for detecting missing fasteners according to embodiments of the invention; Figure 9 shows a visual output of detected missing fasteners in a vehicle, taking all components into account; Figure 10 shows a user interface according to an embodiment of the invention; Figure 11 shows a representation of a data structure in the form of a graph (tree structure) of vehicle components according to an embodiment of the invention; Figure 12 shows a method for detecting incorrectly detected fastener counterparts according to an embodiment of the invention.Figure 13 shows a method for detecting missing fasteners at fastener counterparts of vehicle components for the physical assembly of a vehicle according to further embodiments of the invention; Figure 14 shows a casing of a vehicle part according to an embodiment of the invention; and Figure 15 shows an exemplary section between a digital test specimen and a vehicle part according to an embodiment of the invention.

[0036] Figure 1 shows a procedure and Figure 2 A corresponding device 100 for detecting missing connecting elements at connecting element counterparts 22, 32 of a digital twin 10 under investigation. The digital twin 10 serves for the physical construction of a vehicle. The method is carried out by the device 100 as described in the Figure 2The device 100 comprises at least one processor 120 and an operationally connected memory 110. Instructions are stored in the memory 10, which are executed by the at least one processor 120 to carry out the steps of the method according to the embodiments of the invention.

[0037] The procedure includes providing S100, S100' of digital 3D vehicle geometry data of the digital twin 10 of a vehicle to be investigated. A purely exemplary digital twin 10 of a vehicle is in the Figure 3 shown. The digital twin 10 comprises a multitude of vehicle parts 20, 30 with respective connecting element counterparts 22, 32 to other vehicle parts 30. This is shown in the Figure 2schematically represented. The 3D vehicle geometry data comprises the 3D position and shape of the vehicle parts 20, 30 of the digital twin 10 and / or individual points thereof, thus representing the digital twin 10. The digital 3D vehicle geometry data can be stored or retrieved from the device 10.

[0038] The method further comprises generating S200, S200', for a vehicle part 20, a 2D image from a (digital) spatial direction 60 for the vehicle part 20 of the digital twin 10 by rendering the digital 3D vehicle geometry data. According to the Figures 5 and 6 This can be achieved either by removing all existing connecting elements ( Figure 5 ) or be retained. For each vehicle part 20, a large number of 2D images from different (digital) spatial directions 60 can also be generated to improve situational awareness, as described below.

[0039] The procedure further includes inputting the 2D image S300 into a trained detection model 130 for detecting unoccupied connecting element counterparts 22' based on the input 2D images. The trained detection model 130 can be generated according to the procedure described above. Figure 4 The trained detection model 130 can detect unoccupied connecting element counterparts 22' and record their position. For example, a detection can be output by a rectangular box in the 2D image.

[0040] The method further comprises detecting S500 the absence of a connecting element 40' at a 3D position of the digital twin 10 for the assembly of the vehicle, based on the recognition S400 of a missing connecting element 40' in the input 2D image by the trained detection model 130. Furthermore, the digital 3D vehicle geometry data, as well as the spatial direction and the recognized position in the 2D image, are used for this purpose. The detection S500 can be a separate step from the recognition, as in Figure 5 describe, or directly from the trained detection model 130.

[0041] The invention has the advantage that, using the trained detection model, missing connecting elements in a 3D digital twin can be automatically and with high accuracy detected from the original 3D data via image recognition. Rendering the data into 2D images transforms it for the detection step, allowing the trained detection model 130 to generate the detection based on the 2D images and project the detections back, thus determining the position and orientation in the 3D twin 10. The detection can be displayed on a display 150. In some embodiments, automatic or semi-automatic correction can also be performed, as described in more detail below. The digital twin 10, corrected or with the information about the missing connecting elements 40', is used for the physical assembly of a vehicle.For example, the device 100 can also correct a control command for assembling the vehicle to an actuator / robot 200 based on the digital twin 10, or transmit information about the detected missing connecting elements 40'. The in . Figure 2 The described device 100 is further explained using the following performance variants.

[0042] Figure 4 Figure 1 shows a method for generating the trained detection model 130 according to an embodiment of the invention. Figure 4 This shows a training process according to which the detection model 130 is trained to detect and locate unoccupied connecting element counterparts 22, 32 before being used in the application phase as described above. Figure 1 or the Figures 5 and 6 a digital twin 10 is used to examine unoccupied connecting element counterparts 40'.

[0043] In a first step, S1200 provides digital 3D vehicle geometry data of a synthetic digital twin 10 of a vehicle as training data. This digital twin 10 comprises a multitude of vehicle parts 20 with corresponding connecting elements 22, 32 to other vehicle parts 30 within the 3D vehicle geometry data. The training dataset can thus be obtained from various purely synthetic or virtual data sources. For example, databases S1100' from previous vehicle projects can be used to compile the training dataset. In other examples, geometry design programs such as Catia or Connect can be used as a data source to generate the digital 3D vehicle geometry data of a synthetic digital twin 10. The training data can be provided in a *.jt format, but this is not the sole limitation of the invention.

[0044] In a further step, a 2D image from a spatial direction 60 is generated for each vehicle part 20 by rendering the digital 3D vehicle geometry training data. The rendering can be performed, for example, by CAD programs or a game engine, although the invention is not limited to this.

[0045] Preferably, in step S1400, a large number of 2D images from different spatial directions (60 per vehicle part, 20, 30) are generated from the digital 3D vehicle geometry training data during the rendering process. This is indicated by the parallel blocks in step S1400. This allows for the acquisition of more training statistics.

[0046] These rendered 2D images are then passed as input to the detection model 130 to be trained, and the detection model 130 is trained (S1500). Labels (S1420) are used as references for loss during training. These labels can be detected and localized unoccupied connection element counterparts 22, 32 in the 2D images (S1410), which were determined and localized using classical or conventional image processing algorithms. The labels for training can thus be derived directly from the 2D images and / or improved with minimal post-processing, e.g., through classical image processing / computer vision. For example, methods such as template matching or 2D images with and without connection elements can be used, where the detection result is clearly defined and is to be reproduced by the training detector module 130. Manually adjusted labels (S1420) can be added to cover, for example, special cases.

[0047] In a preferred embodiment, the rendering process for generating the 2D images from the 3D vehicle geometry data includes removing connecting elements 40 in the connecting element counterparts 22, 32. This provides more detectable, unoccupied connecting element counterparts for training in each 2D image. However, the invention is not limited to this, and the connecting elements 40 can also be retained.

[0048] As a result, if a defined accuracy criterion for the training process S1500 is achieved, a trained detection model 130 is generated, which is generalized for the detection of unoccupied connecting element counterparts 22', 32' based on input 2D images of vehicle parts 20, 30.

[0049] The trained detection model 130 exhibits no reality gap. Typically, a so-called reality gap occurs when using synthetic data. This corresponds to a performance deficit when the trained detection model 130 is applied to real data. Since the subsequent application also takes place in the synthetic domain, i.e., for virtual twins 10, no reality gap arises. Thus, the trained detection model 130 has high performance.

[0050] The result is a trained detection model 130, which detects unoccupied connecting element counterparts 22', 32', e.g., screw holes or bracket positions. These can be displayed, for example, as a 2D bounding rectangle in the 2D image.

[0051] Figure 5further shows a method for detecting missing connecting elements 40' at connecting element counterparts 22, 32 of a digital twin 10 to the physical structure of a vehicle under investigation according to further embodiments of the invention. The method is described below together with the Figures 7 to 13 described.

[0052] In a first step, as described above, S100 is provided with digital 3D vehicle geometry data of the digital twin 10 to be investigated. The digital twin 10 of a vehicle comprises a multitude of vehicle parts 20, 30 with respective connecting element counterparts 22, 32 to other vehicle parts 30. This digital 3D vehicle geometry data, compared to the training according to Figure 4, correspond to the vehicle data of the digital twin 10 to be investigated. The 3D vehicle geometry data can be extracted from various sources. For example, the 3D vehicle geometry data may have been obtained from databases S110' by exporting the geometries or from geometry design data, although the invention is not limited to this. The result is a digital twin 10 to be investigated with digital 3D vehicle geometry data of the vehicle parts 20, 30. In the Figure 7 An exemplary logo component is shown as an example of a vehicle part 20, comprising a logo body 20 and a logo holder 30. The connecting element counterparts 22, 32 are holes in this case, and the connecting elements 40 are screws, although the invention is not limited thereto.

[0053] In the next step, a 2D image of the digital twin (S200) is generated from one spatial direction (60) for each vehicle part (20, 30). This can be done in two ways, as described below. Figure 5 and in the Figure 6 is described.

[0054] In this present embodiment according to Figure 5 The 2D image of the vehicle part 20, 30 is generated by rendering the digital 3D vehicle geometry data while removing connecting elements 40. This means that the 2D image has connecting element counterparts 22, 32 but no connecting elements. In the Figure 7This is shown using the example of the logo component as a vehicle part 20, which comprises a logo body 20 and a logo holder 30. The connecting element counterparts 22, 32 are holes in this case, and the connecting elements 40 are screws. This data has been extracted, for example, into a *.jt file and from Connect / Evie. However, the invention is not limited to this example; see also the training for the detection of unoccupied connecting element counterparts 22', 32'.

[0055] To illustrate the rendering process, in Figure 7A virtual camera 50 is shown as an example of a suitably chosen spatial direction 60, from which a 2D image is generated from the 3D vehicle geometry data of the logo body 20 and a logo holder 30. In this version, the 2D image is rendered such that no connecting elements are included. The connecting element counterparts 22', 32' in the rendered 2D image are therefore unoccupied. An alternative to this is described in the method according to Figure 6 described, in which connecting elements are retained. This design is schematically shown in the Figure 7 This is illustrated by the fact that only the connecting element counterpart 22', 32' marked with a dashed circle is unoccupied, and all others in this example would be occupied.

[0056] In the following step, the 2D image S300 is entered into the trained detection model 130 to detect unoccupied connecting element counterparts 22' based on the entered 2D images.

[0057] In the following step, S400, for each vehicle part 20, 30, the trained detection model 130 is used to detect unoccupied connecting element counterparts 22', 32' and their position based on 2D images, specifically at least one unoccupied connecting element counterpart 22' in the 2D image. This means that the trained detection model 130 automatically detects the unoccupied connecting element counterparts 22', 32' in the 2D image. In the present case, in step S400 in Figure 8 , all three unoccupied connecting element counterparts 22', 32' are each detected.

[0058] Preferably, in step S200, to better determine the 3D position, a large number of 2D images from different spatial directions 60 for each vehicle part 20, 30 are generated from the digital 3D vehicle geometry data. In this case, the large number of 2D images are input S300 into the trained detection model 130 to detect unoccupied connection element counterparts 22', 32' based on the large number of input 2D images. Then, for each of the 2D images of a vehicle part, an unoccupied connection element counterpart 22', 32' is detected using the trained detection model 130. The large number of 2D images for different spatial directions 60 is represented by the three parallel blocks to S400 in Figure 5 The recognition result is, for example, a bounding rectangle in each 2D image.

[0059] In a further step of the procedure, the 3D position S410 of the at least one detected connecting element counterpart 22', 32' is determined by backprojection into 3D space or into the 3D vehicle geometry using the (original) digital 3D vehicle geometry data, the spatial direction 60 and the detection position in the 2D image (envelope rectangle).

[0060] In a preferred embodiment, this step can be performed by generating a 3D point cloud S460 from the digital 3D vehicle geometry data of the digital twin 10 and the detection position in the 2D image (enveloping rectangle). This can be done by sampling the 3D point cloud over the dimensions of the detection bounding rectangles. Outliers can be excluded using K-medoids with distance criteria. In this process, further geometric dimensions such as plane normals, areas, or maximum dimensions can be determined. Then, the determination of a 3D position S410, S510' of the minimum detected counterpart of the connecting element can be achieved by backprojection based on the 3D point cloud, the spatial direction, and / or the detection position in the 2D image (enveloping rectangle), i.e., by sampling the 3D point cloud.

[0061] With a large number of 2D images per vehicle part, the determination of 3D position positions S410 for each of the detected connecting element counterparts for each 2D image per vehicle part is carried out by backprojection based on the digital 3D vehicle geometry data and the detection position (enveloping rectangle). These determined 3D position positions are, for example, in the Figure 8 This is shown in step S410. It can be seen that the 3D positions are not identical, but rather exhibit a variation. This is because the trained detection model cannot estimate the recognition of the connecting element counterparts and their positions equally well for every solid angle of 60°, as the 2D image contains different information for each solid angle of 60°.

[0062] Subsequently, for a plurality of 3D position positions, a clustering S420 of the specific 3D position positions for a vehicle part 20, 30 takes place, as in the Figure 8displayed. Furthermore, a 3D position S430 of a detected connecting element counterpart is determined as the central point C from each cluster. This is in the Figure 8 This is shown in step S420. The result is a very accurate position description of an unoccupied connecting element counterpart 22', 32' in 3D through the clustering process, compared to the case where only a solid angle of 60 is used.

[0063] Furthermore, filtering of 3D position data from unoccupied connection element counterparts can be performed before and / or after clustering, for example, based on weights derived from label confidence and / or distance information, in order to remove outliers (outliners). This further improves the accuracy of determining the 3D position.

[0064] These steps are now repeated S440 for all or predetermined vehicle components 20, 30 of the digital twin 10. Thus, the positions of unoccupied connecting element counterparts 22', 32' are determined sequentially for the various vehicle components, as described above. The result is a very precise 3D positional description of potential connecting element counterparts for the multitude of vehicle components 20, 30.

[0065] In the next step, the digital merging of the various vehicle parts 20, 30 of the digital twin 10 takes place using the digital 3D vehicle geometry data. The digital merging of different vehicle parts can be carried out, in particular, based on the 3D position of the at least one detected unoccupied connection element counterpart by pairing or forming pairs of unoccupied connection element counterparts from different vehicle parts and merging them opposite each other. Appropriate distance criteria can be used for this purpose. Such a step is performed for the logo object example in step S450 in the Figure 8 shown where the connecting element counterparts 22', 32' are located. A digital merging of all vehicle parts 20, 30 S450 is purely exemplary in the Figure 9 shown.

[0066] In a further step, the absence of a connecting element 40' at the connecting element counterpart 22', 32' is detected by S500 based on a cross-sectional comparison with a digital test object (e.g., digital envelope 70) at the connecting element counterpart 22', 32'. An example of a digital envelope 70, here for a vehicle part 20, is found, for instance, in the Figure 14 Shown for illustrative purposes only.

[0067] Since the connecting elements were removed during the rendering of the 2D images in this embodiment, all connecting element counterparts 22', 32' are generally detected automatically. Therefore, it must now be checked whether geometry was present at the detected connecting element counterparts 22', 32' in the 3D vehicle geometry data of the twin 10. That is, to determine whether these are occupied by a connecting element 40 such as a screw or similar, a cross-sectional inspection can be performed according to the invention.

[0068] This can preferably be achieved by generating S520 of digital envelopes 70, e.g., bounding boxes, and connecting elements 40 of the vehicle parts 20, 30 of the digital twin 10 from the digital 3D vehicle geometry data. This can be done from the original digital twin 10 or the digital 3D vehicle geometry data before generating S200 of the 2D images as shown in Figure 5 take place.

[0069] This is followed by the detection (S500) of the absence of a connecting element 40' at the connecting element counterpart 22', 32' based on a cross-sectional comparison of the digital envelope 70 at the detected connecting element counterpart. Here, it can be checked whether the connecting element counterpart lies within the envelope 70, e.g., by a min-max test. In detail, the geometry of the connecting element (screw, plug, clip, etc.) of the digital twin 10 under investigation is used to generate envelopes 70, such as bounding boxes (an enclosing cube for quick "cross-sectional check"). This yields both the spatial location of all existing connecting elements 40 (Yn) and the spatial location of all detected connecting element counterparts 20, 30 (Xn) from the 3D geomatry data. By cross-sectional comparison between A) and B) (e.g.,If the counterpart of the connecting element X1 lies within the bounding box Y1, it can be determined which detected counterparts 22, 32 are already occupied by connecting elements 40 and which are not. This results in a set of points that are either occupied or unoccupied. This is, for example, in the... Figure 8 The logo object is shown, in which the missing connecting element 40' was detected. For the unoccupied connecting element counterparts 40', this means that at this point in space there is a gap between two vehicle components, or that no connecting element is present at this connecting element counterpart 22, 32. Thus, the absence of a connecting element 40' can be detected very reliably by this cross-sectional comparison. A detection is, for example, in Figure 8 shown in the S500 or several also in the Figure 9 .

[0070] In a further step of the process, the digital twin 10 can be corrected S700 based on the detection S500, S500' of a missing connecting element 40 and by adding the missing connecting element at the determined 3D position in the digital twin 10. This can be done automatically or semi-automatically, for example, by the user selecting the appropriate type of connecting element (receiving selection input). Furthermore, either together with the correction or as an alternative, information about the missing connecting element 40' can be displayed on a display 150 of the device 100. The result is, for example, a message indicating the position where connecting elements 40' are missing, or this information can be used to correct the digital twin 10.However, partial automation can also be achieved by manually correcting the identified defects, while full automation is also conceivable by placing and correcting the exact connecting element in the 3D vehicle geometries at the identified connecting element counterparts 22, 32.

[0071] In a particular embodiment, the method includes adding the missing connecting element 40' to the digital twin 10 based on the orientation of the connecting element counterpart 22', 32' and / or geometric information about the connecting element counterpart 22', 32'. Determining the orientation of detected opposing connecting element counterparts 22', 32' of opposing vehicle parts can be described as a vector (director) between the identified unoccupied connecting element counterparts. Knowledge of the orientation can enable the addition of the missing connecting element 40' to the digital twin based on the orientation of the connecting element counterparts.

[0072] To improve automation, the process can include determining the type of S600 for a missing 40' connector. This can be achieved through the following steps. First, geometric information about a detected unoccupied connector counterpart 22, 32' can be determined. Next, a trained classifier 140 identifies the type of missing 40' connector based on the geometric information of the unoccupied connector counterpart 22', 32'. Such a trained classifier 140 can then automatically determine suitable connectors. These connectors and their corresponding connector counterparts can be retrieved from a database.

[0073] In this case, the correction S700 of the digital twin 10 can be based on the detection of the type of missing connecting element 40', so that the missing connecting element 40' is added to the specified 3D position of the connecting element counterpart 22, 32 in the digital twin 10 and / or information about the missing connecting element 40' is displayed on a display 150 (automatically).

[0074] In the Figure 6 A further method for processing a digital twin 10 of the physical structure of a vehicle, which is to be examined for missing connecting elements at connecting element counterparts, is described according to further embodiments of the invention. In the following, only the differences to the embodiment of the Figure 5 described.

[0075] In this embodiment, the generation S200' of a 2D image from one spatial direction per vehicle part of the digital twin 10 is carried out by rendering the digital 3D vehicle geometry data, wherein the 2D image includes connecting elements 40 and connecting element counterparts 22, 32. Compared to the method in the Figure 5 The connecting elements are therefore not rendered away. Consequently, in the rendered 2D images, connecting element counterparts are populated with connecting elements, provided the connecting elements actually exist in the digital twin 10. In the Figure 7 The connecting elements 40 are represented by the dashed circles, which are used in the procedure according to Figure 5 are not part of the rendered 2D images.

[0076] The method further includes detecting S500 the absence of a connecting element 40' at a 3D position of the digital twin 10 of the physical structure of the vehicle when the trained detection model 130 detects an unoccupied connecting element counterpart 22' in the rendered 2D image S400'. The detection of missing connecting elements is thus performed directly by the trained detection model 130. Occupied connecting element counterparts are not detected by the detection model. Therefore, in the Figure 8 In the illustrated step S500', S400', only the unoccupied connecting element counterparts 22', 32' in the vehicle parts were already detected, which is indicated by the square marking in Figure 8 This is indicated. This results in immediate detection of the unoccupied connection element counterparts (S500'). Detection and recognition thus coincide, creating a particularly efficient detection process.

[0077] Furthermore, the 3D position of the detected unoccupied connecting element counterpart 22' is then determined by backprojection using the digital 3D vehicle geometry data, the spatial direction, and the detected position in the 2D image. A merging process, as shown in S450', can also be performed using the methods described above. This is done analogously to steps S410' to S440' in Figure 5 .

[0078] In this embodiment, the cut comparison is no longer required, since all fasteners 40, e.g., screws, plugs, clips, etc., are already included in the images during the rendering of the vehicle part. This means that the detection S500' of a missing fastener 40 occurs directly through recognition S400' by the detection model 130. Therefore, fasteners such as screws are already included in the rendered images for the logo object. If all fasteners 40 are present, the trained detection model 130 will not detect any holes or unoccupied fastener counterparts 22'. However, if fasteners 40' are missing, the unoccupied fastener counterparts 22' are detected directly by the trained detection model 130 without requiring a cut comparison.The discovered unoccupied connecting element counterparts 22' at the 3D position positions can be detected in the images and the information can be forwarded to the user, as described above. The detection is faster compared to the method according to [reference missing]. Figure 5 It is pre-selected and arises at the output of the trained detection model 130.

[0079] In a special version, only the connecting elements that are close to, e.g., in the bounding box or within a threshold distance of, the vehicle component are rendered, so that the 2D images do not contain all connecting elements of the vehicle.

[0080] The further steps, which in detail lead to the Figure 5 The methods described, with the exception of detection by comparison, can be applied analogously to the Figure 6 to be transferred. Therefore, a redundant description is omitted below, with reference to the description of the Figure 5, which of these can be extracted.

[0081] In the Figure 10 An embodiment of displaying the detection result on a user interface 160 of the display 150 is described. In this described embodiment, the user can intervene before correction. Any detected missing connecting elements 40' are displayed on the user interface 160. This allows, for example, consultation with the relevant trades. The user interface 160 can display a list 162 of the missing connecting elements 40' and / or graphically represent which connecting elements 40' are missing. For example, the user can select individual connecting elements 40' from the list 162 and display them on a graphic display area 164 with a corresponding vehicle part 22, 32. Here, the detection result from Figure 10selected and displayed. In a further embodiment, during the flight or search of the digital twin 10, the detection results can be displayed simultaneously on an image display surface 164 during the flight or search of the digital twin 10 in a partially automated process.

[0082] The user interface 160 can also have at least one feedback button 165, 165'. User feedback can be captured via this button, including information about whether the missing connecting element 40' is confirmed or not confirmed. This feedback button 165, 165' of the user interface 160 allows the user to confirm or deny the detected missing connecting elements. That is, the user can confirm or deny the detected missing connecting elements by pressing the feedback button 165, 165'. This human-machine interaction and information can be used to adapt the trained detection module 130.

[0083] Therefore, the procedure includes adapting or retraining the trained detection model 130 for the detection of unoccupied connection element counterparts based on the collected user feedback. This information can be used to adapt or successively improve the trained detection models 130 (and / or the trained classifier 140), i.e., to retrain them. In particular, the weights of the trained detection model 130 for the detection of connection element counterparts can be adjusted based on the collected user feedback if, for example, a missing connection element is not confirmed by the user. If the user confirms the absence, the digital twin 10 can be corrected and the missing connection element 40' can be added, which corresponds to partial automation. The user can then perform a plausibility check.

[0084] In the Figure 11A graph-based data structure 115 of the digital twin 10 for the physical structure of the vehicle according to an embodiment of the invention is shown. This graph-based data structure 115 (tree structure) is stored in memory 110. This graph-based data structure 115 can be used for computational purposes. For example, a visual output can also be displayed on the display 150 of the graph-based data structure 115, although the invention is not limited to this. The graph-based data structure 115 comprises vehicle parts 20, 30 as nodes or vertices and the connecting element counterparts 22, 32 or connecting elements as edges, i.e., connections between the nodes, recognized by the trained detection model. Because the method determines the connecting element counterparts 22, 32, the relationships between the vehicle parts 20, 30 can thus be automatically determined.Therefore, a graph-based data structure can easily be generated in which vehicle parts 20, 30 are not structured by a parent-child relationship, but rather by their spatial position and their connection to each other via the connecting elements 40. With this representation, assembly / disassembly sequences for the physical structure of the vehicle can easily be derived and implemented. This generation can be achieved particularly with the embodiment shown. Figure 5 They work together, since all connecting element counterparts 22, 32 are determined there.

[0085] In the Figure 12 A method for correcting incorrectly detected connecting element matings according to an embodiment of the invention is described. In the method described above according to Figure 5 or Figure 6It is possible for the system to detect seemingly unoccupied connection element counterparts, which are actually contact points or not (true) connection element counterparts at all. For example, a contact point might contain two holes and a cable passing through them. This means that there are some locations in vehicles that appear to be connection element counterparts 22 and 32 based on features such as these, but are not actually connection element counterparts, such as holes in two vehicle parts through which a cable is routed. However, these contact points are always occupied by geometry (e.g., a cable).

[0086] The procedure according to this Figure 12 The method further includes the detection of incorrectly identified connecting element counterparts according to one embodiment of the invention. The method can be combined with the various embodiments, in particular according to... Figure 5 and 6 .

[0087] The procedure involves identifying a set of vehicle parts from the digital twin 10 (excluding fasteners) whose digital envelopes 70 contain the identified fastener counterpart or have a predefined maximum distance from it. This results in a set of vehicle parts that reduces the complexity for the subsequent cutting test, as not all vehicle parts need to be considered for the test.

[0088] Furthermore, the procedure includes identifying S810, from the multitude of identified vehicle parts, a subset of those parts that do not have a corresponding fastener. This involves removing those vehicle parts that were already involved in identifying the fastener, i.e., those parts where a fastener to be tested was detected. The number of vehicle parts is thus further reduced.

[0089] Furthermore, the procedure includes placing S820 a digital test specimen 80 (e.g., cylinder or sphere) or a test geometry with a predetermined geometry at the detected counterpart of the connecting element. Such a digital test specimen 80 is shown purely as an example in the Figure 15 illustrated. Furthermore, detection S830 (cut test) then takes place to show that the identified fastener counterpart is not a fastener counterpart when the digital test piece 80 cuts a vehicle part from the identified subset of vehicle parts.

[0090] For example, in the S830 cut test, the 3D point cloud can be used to minimize the performance required for the cut test. This means that it is checked whether the test geometry of the digital test object intersects and / or almost intersects the geometry of one of the vehicle parts used at the counterpart of the connecting element (e.g., the sphere test geometry lies within a very thick cable).

[0091] If this is the case and the cross-sectional analysis is positive, the counterpart of the connecting element is already occupied, e.g., by a cable. Therefore, no connecting element (e.g., screw) is missing; rather, there is a contact point that is not a counterpart of a connecting element. If this is not the case and the cross-sectional analysis is negative, there is no geometry at the counterpart of the connecting element, and the absence of connecting element 40' can be confirmed. The sensitivity of the detections can thus be increased.

[0092] If no connecting elements are present in the 2D images during rendering, see embodiment according to Figure 5Many holes or unoccupied connection element matings are found, which may, however, be occupied by a hose / cable. To exclude such apparent connection element matings, a digital test object 80, e.g., a sphere, can be placed at this location in 3D space, as described above. For example, the digital test object 80 can correspond approximately to the diameter of the hole opening. This check is performed when all geometries are "activated" so that hoses / cables occupy all holes. The test geometry or test object then checks whether it intersects with another object's geometry (cable / hose). If so, it can no longer be a screw hole. Furthermore, it can be additionally differentiated or verified, for example, by using a part designation of the intersecting object with which the test geometry intersects, whether it is a hose or cable, etc.

[0093] The procedure described above is further explained below using an example in which two vehicle parts are made of sheet metal with a common hole serving as the counterpart to be tested for fasteners. A hose or pipe runs through this hole, and therefore it is not a fastener. Consequently, this hole should not be recognized as a counterpart to fasteners such as screws. All three parts are rendered to obtain 2D images. This can be done with or without fasteners, but since none exist in this example, none are present during rendering. Thus, 2D images of both sheet metal parts and the hose are obtained. The trained detection model then recognizes the counterpart to the fastener, or the hole, in each sheet metal part, but not in the pipe, as there is none there. The detection process is already described in detail above.At this point, a missing screw could therefore be falsely detected.

[0094] As described above in general terms, the procedure according to steps S800 to S840 is now applied to the example. According to step S800, the digital envelopes of all three vehicle parts, including the hose, are generated and checked to see if the determined counterpart of the connecting element lies within each of the three envelopes and / or within a specified maximum distance.

[0095] According to step S810, those vehicle parts that do not have mating connection points are now identified. The two metal sheets are removed from the identified set of vehicle parts, as these were used to form the mating connection point. Thus, in this example, only the hose remains.

[0096] According to step S820, a digital test object 80 (e.g., a ball the size of the determined hole) is now inserted, see for example in the Figure 15 , to the position of the connecting element counterpart of the digital test specimen 70 (e.g., a sphere the size of the determined hole). Then, in step S830, the digital test specimen 80 is cut with the subset of vehicle parts from step S810 (in this specific case, with the hose). For example, the triangular mesh can be removed from the hose and the triangular mesh from the digital test specimen 80. It is then checked whether the triangles intersect. Such a cut is, for example, in the Figure 15 Shown for illustrative purposes only.

[0097] If a cut is detected during the S830 detection step, the hole or the corresponding fastener is occupied by a vehicle part / hose that is not a fastener. It can therefore be determined with certainty that this hole is not one in which one or more fasteners are missing. Extending the procedure and for further clarification, there could be another hole or fastener in the sheet metal or vehicle parts that should contain a screw or fastener, but this screw or fastener is missing. In this case, the S830 cut test will not detect any cut geometry. Therefore, it can be determined with certainty that the hole is unoccupied and that no other vehicle part belongs there. This indicates a missing fastener 40' or a missing screw.

[0098] In the Figure 13 A method for detecting missing fasteners at fastener counterparts of vehicle components for the physical assembly of a vehicle according to further embodiments of the invention is shown. This method can be combined with the embodiments according to the Figures 5 and 6 can be combined.

[0099] The detection of unoccupied connection points relies on image information. However, resolution plays a crucial role in this detection. Vehicle components can be very large (e.g., long support elements on trucks). This means that the resolution for detecting missing 22' and 32' connection points may not be high enough to identify potential connection points in the 2D image (e.g., screw holes are rendered at <1 pixel because the ratio of image resolution to the size of the vehicle component is too small).

[0100] To address this, a vehicle part under investigation can be divided into several sub-segments from the digital 3D vehicle geometry data (e.g., an equidistant grid, analogous to voxelization). Each sub-segment can correspond to a size that matches the maximum image resolution (ratio of pixels to actual millimeters). Subsequently, a 2D image is generated for each sub-segment from at least one spatial direction by rendering the digital 3D vehicle geometry data. Furthermore, the 2D image is fed into the trained detection model to identify unoccupied connection element counterparts based on the input 2D images.Furthermore, the step of detecting S950 the absence of a connecting element at a 3D position of the digital twin 10 for the assembly of the vehicle is based on the recognition S940 of a missing connecting element 40' in the input 2D image of the sub-segment by the trained detection model 130, the digital 3D vehicle geometry data, the spatial direction and the recognized position in the 2D image. The detection is thus carried out individually for the sub-segments.

[0101] This allows even missing connecting elements in oversized vehicle parts to be detected. The sub-segments can then be digitally merged into the vehicle component in an analogous manner, as already described in the step of backprojection in 3D space or in clusters and / or filters. This simply creates another layer on which information must be combined. In this way, the detection can identify even the smallest defects in larger objects. This can also be done during classification in step S600.

[0102] In summary, a method for detecting missing connecting elements 40' at connecting element counterparts 22, 32 of a digital twin 10 of the physical structure of a vehicle under investigation is carried out. The method can very accurately detect, report, and correct missing connecting elements 40' in 3D geometry data using optical recognition in 2D images for connecting element counterparts 22, 32 in 2D of a digital twin according to the above descriptions, with possible automatic placement. Reference symbol list

[0103] S100, S100'Providing digital 3D vehicle geometry data S200, S200'Generating a 2D image S300, S300'Inputting the 2D image into the detection model S400, S400'Detecting a missing connecting element S410, S410'Determining the 3D position S420, S420'Clustering / Filtering S430, S430'Determining the 3D position of the cluster's center point S450, S450'Digital merging S460, S460'Generating a 3D point cloud S500, S500'Detect the absence of a connecting element S600, S600'Output the detection S610, S610'Adjust the trained detection model S700, S700'Correct the digital twin S800 Identify a set of vehicle parts whose digital envelopes contain the mating element location or are within a maximum distance. S810 Identify a subset of vehicle parts. S820 Place a digital test object at the detected connection point. S830 Detect that the detected mating element location is a contact point and not a mating element location. S900 Disassemble a vehicle part S910 Generate a 2D image for each part segment S920 Input the 2D image into the trained detection model S930 Detect an unoccupied connecting element counterpart S940 Detect the absence of a connecting element 10 digital twin (of a vehicle) 20 vehicle part 22 connecting element counterpart 22' unoccupied connecting element counterpart 30 vehicle part 32 connecting element counterpart 32' unoccupied connecting element counterpart 40 connecting element 40'missing connecting element 50 virtual camera 60 spatial direction 70 digital enclosure 80 digital test object 100 Device 110 Memory 115 Graph-based data structure 120 Processor 130 Detection model 140 Classifier 150 Display 160 User interface 162 List 164 Image display area 200 Actuators / Robots

Claims

1. Computer-implemented method for detecting missing connecting elements (40') at connecting element counterparts (22', 32') of a digital twin (10) under investigation with respect to the physical construction of a vehicle, comprising the following steps: providing (S100; S100') digital 3D vehicle geometry data of the digital twin (10) under investigation of a vehicle, comprising a number of vehicle parts (20) with respective connecting element counterparts (22) to other vehicle parts (30); generating (S200, S200'), for a vehicle part (20), a rendered 2D image from one spatial direction (60) for each vehicle part (20) of the digital twin (10) under investigation by rendering the digital 3D vehicle geometry data; recognizing (S400, S400') unoccupied connecting element counterparts (22', 32') and their position in the rendered 2D image using a trained detection model (130) which is trained in a training phase to recognize unoccupied connecting element counterparts (22', 32') based on 2D images; detecting (S500, S500') the absence of a connecting element (40') at a 3D position of the digital twin (10) under investigation with respect to the physical construction of the vehicle on the basis of the recognition (S400, S400') of an unoccupied connecting element counterpart (22', 32') and its position in the generated 2D image by the trained detection model (130), the digital 3D vehicle geometry data, the spatial direction (60) and the recognized position in the rendered 2D image; and correcting the digital twin (10) on the basis of the detection of a missing connecting element (40') by adding the missing connecting element at the 3D position in the digital twin (10), and / or issuing information regarding a missing connecting element (40') on a display (150).

2. Method according to claim 1, further comprising: generating (S200'), for a vehicle part, a rendered 2D image from one spatial direction for each vehicle part of the digital twin (10) under investigation by rendering the digital 3D vehicle geometry data, wherein the generated 2D image comprises connecting elements (40) and connecting element counterparts (22, 32); detecting (S500) the absence of a connecting element (40') at a 3D position of the digital twin (10) with respect to the physical construction of the vehicle when the trained detection model (130) recognizes an unoccupied connecting element counterpart (22') and its position in the rendered 2D image; determining the 3D position (S410') of the detected unoccupied connecting element counterpart (22') by back projection using the digital 3D vehicle geometry data, the spatial direction and the recognized position in the rendered 2D image (60).

3. Method according to claim 1, further comprising: generating (S200), for a vehicle part, a rendered 2D image from one spatial direction for each vehicle part of the digital twin (10) under investigation by rendering the digital 3D vehicle geometry data, with connecting elements being removed; recognizing (S400), for each vehicle part, using the trained detection model (130), at least one unoccupied connecting element counterpart (22') and its position in the generated 2D image; determining the 3D position (S410) of the at least one recognized unoccupied connecting element counterpart by back projection using the digital 3D vehicle geometry data, the spatial direction and the recognized position in the rendered 2D image; digitally merging (S450) the different vehicle parts (20, 30) of the digital twin (10) using the digital 3D vehicle geometry data; and detecting (S500) the absence of a connecting element (40') at the connecting element counterpart (22', 32') on the basis of a cross-sectional comparison with a digital test piece (70) at the recognized connecting element counterpart (22', 32').

4. Method according to claim 3, further comprising: generating (S520) digital envelopes (70) of connecting elements (40) of the vehicle parts (20, 30) of the digital twin (10) from the digital 3D vehicle geometry data; detecting (S500) the absence of a connecting element (40') at the connecting element counterpart (22', 32') on the basis of a cross-sectional comparison of the digital envelope at the recognized connecting element counterpart (22', 32').

5. Method according to any of claims 1 to 4, further comprising: generating (S200, S200') a number of rendered 2D images from different spatial directions for each vehicle part (20, 30) from the digital 3D vehicle geometry data of the digital twin under investigation (10); recognizing (S400, S400'), for each of the rendered 2D images of a vehicle part, an unoccupied connecting element counterpart (22', 32') and its position in the 2D image using the trained detection model (130); determining 3D positions (S410, S410') for each of the recognized connecting element counterparts (22', 32') per vehicle part by back projection on the basis of the digital 3D vehicle geometry data, the spatial direction and the recognized position in the rendered 2D image; clustering (S420, S420') the determined 3D positions; and ascertaining a 3D position (S430, S430') of a recognized connecting element counterpart (22', 32') as the central point (C) from each cluster.

6. Method according to claim 5, further comprising: filtering 3D positions of unoccupied connecting element counterparts (22', 32') before and / or after clustering on the basis of weightings from confidence of the label and / or distance information of the 3D positions of connecting element counterparts predicted per spatial direction (60).

7. Method according to any of claims 1 to 6, further comprising: generating a 3D point cloud (S460, S460') from the digital 3D vehicle geometry data of the digital twin under investigation (10); determining a 3D position (S410, S410') of the at least recognized connecting element counterpart by back projection on the basis of the 3D point cloud, the spatial direction and the recognized position.

8. Method according to any of claims 1 to 7, further comprising: determining the orientation of recognized opposing connecting element counterparts (22, 32) of different merged vehicle parts (20, 30) as a vector between the ascertained unoccupied connecting element counterparts (22', 32'); and adding the missing connecting element (40') to the digital twin (10) on the basis of the orientation of the connecting element counterparts (22', 32').

9. Method according to any of claims 1 to 8, further comprising: receiving geometric information from a recognized unoccupied connecting element counterpart (22', 32'); recognizing, using a trained classifier (140), a type of missing connecting element (40') on the basis of the geometric information of the unoccupied connecting element counterpart (22', 32'); and correcting (S700) the digital twin (10) on the basis of the recognition of the type of missing connecting element by adding the missing connecting element (22', 32') at the 3D position in the digital twin (10) and / or issuing information regarding the missing connecting element (40') on a display (150).

10. Method according to any of claims 1 to 9, further comprising: ascertaining (S800) a set of vehicle parts of the digital twin (10) without connecting elements of which the digital envelopes (70) contain the recognized connecting element counterpart and / or are within a maximum distance from the recognized connecting element counterpart; ascertaining (S810), from the number of ascertained vehicle parts, a subset of vehicle parts which do not have any previously recognized connecting element counterparts; positioning (S820) a digital test piece (80) with predetermined geometry at the recognized connecting element counterpart; and detecting (S830) that the recognized connecting element counterpart is not a connecting element counterpart when the digital test piece (80) cuts a vehicle part from the identified subset of vehicle parts.

11. Method according to any of claims 1 to 10, further comprising: splitting (S900) a vehicle part under investigation into a plurality of sub-segments from the digital 3D vehicle geometry data; generating (S910) a rendered 2D image from one spatial direction (60) for each sub-segment of the vehicle part of the digital twin (10) under investigation by rendering the digital 3D vehicle geometry data; recognizing (S930) unoccupied connecting element counterparts (22', 32') and their position in the rendered 2D image using the trained detection model (130); detecting (S940) the absence of a connecting element at a 3D position of the digital twin (10) with respect to the physical construction of the vehicle on the basis of the recognition (S930) of an unoccupied connecting element counterpart (22', 32') in the rendered 2D image of the sub-segment by the trained detection model (130), the digital 3D vehicle geometry data, the spatial direction and the recognized position in the 2D image.

12. Method according to any of claims 1 to 11, further comprising: issuing recognized missing connecting elements (40') and / or unoccupied connecting element counterparts on a user interface (160); capturing user feedback, including information regarding confirming or not confirming the absence of the connecting element (40') in the digital twin (10) via at least one feedback button (165, 165') of the user interface (160); adapting (S610, S610') the trained detection model (130) on the basis of the captured user feedback.

13. Method according to any of claims 1 to 12, further comprising: issuing a control command to an actuator (200) or a robot to construct the vehicle on the basis of the corrected digital twin (10) or using the information regarding the recognized missing connecting elements (40).

14. Method according to any of claims 1 to 13, further comprising: generating a graph-based data structure (115) with respect to the physical construction of the vehicle, wherein the generated graph-based data structure (115) comprises vehicle parts (20) as nodes and the connecting element counterparts (22, 32) recognized by the trained detection model as edges of the nodes.

15. Device (100) for detecting missing connecting elements (40') at connecting element counterparts (22, 32) of a digital twin (10) under investigation with respect to the physical construction of a vehicle, comprising: a display (150), at least one processor (120), which is configured to perform the steps of the method according to any of claims 1 to 14.

Images