Method, device and system for detecting a shape deviation

Abstract

The invention relates to detecting a shape deviation in three dimensions, 3D, of an inspection object (11) of a 3D target geometry (33), in which back projections on the 3D target geometry (33) are determined by at least one processing circuit (34), wherein, for a plurality of 2D images of the inspection object (11), one back projection each is determined on at least one part of the 3D target geometry (33). The determined back projections for the plurality of 2D images are compared by the at least one processing circuit (34) in order to detect the shape deviation.

Description

[0001] Method, device and system for detecting a shape deviation

[0002] TECHNICAL FIELD

[0003] The invention relates to methods, devices, and systems for detecting a shape deviation using electromagnetic radiation. The invention particularly relates to methods, devices, and systems for image-based detection of a deviation between a three-dimensional geometry of an inspection object and its three-dimensional target geometry.

[0004] BACKGROUND

[0005] Detection of a shape deviation between an actual three-dimensional (3D) geometry of an inspection object and its 3D target geometry has various application areas, for example in quality control.

[0006] The detection of shape deviation using electromagnetic radiation, in particular using optical techniques, is attractive for several reasons, such as the achievable speed and its non-contact operation.

[0007] A deviation between the desired and actual geometry can be determined, for example, by determining the actual geometry. Methods using structured illumination allow the determination of the actual geometry, but require well-defined illumination patterns and knowledge of these illumination patterns. The techniques disclosed in EP 2 321 614 B1 can measure a surface with high accuracy, but this requires a light source that is not typically manually manageable.

[0008] DE 10 2020 134 680 A1 discloses a method for quality inspection in which images can be captured using a mobile capture device and subsequently evaluated. However, the application of this technology for quality inspection can entail a relatively large amount of configuration effort for the quality inspection to be performed, such as the definition of a reference point and the definition of the target geometry. In order to obtain detailed information at the component level during this inspection, the target geometry must be in a format such that individual components can be individually removed and / or addressed. However, in some application areas, this requirement is not met. The inspection result can be subject to undesirably strong influences, for example, due to the texture of the inspected object.There is therefore still a need in technology for improved methods, devices and systems for detecting shape deviation.

[0009] SUMMARY

[0010] It is an object of the present invention to provide improved methods, devices, and systems for detecting a shape deviation. In particular, it is an object to specify such methods, devices, and systems that can be operated with minimal setup effort and are applicable to different object geometries. Optionally, it is desirable to provide methods, devices, and systems in which the required data acquisition on the inspection object can be carried out in a simple manner.

[0011] According to the invention, methods, devices, and systems are provided as defined in the independent claims. The dependent claims define preferred and advantageous embodiments.

[0012] According to one aspect, the invention relates to a method for detecting a shape deviation in three dimensions (3D) of an inspection object from a 3D target geometry of the inspection object. The method comprises: receiving two-dimensional, 2D, images of the inspection object by at least one processing circuit; determining a plurality of backprojections onto the 3D target geometry by the at least one processing circuit, wherein for a plurality of 2D images of the inspection object, a backprojection onto at least a portion of the 3D target geometry is determined; and performing a comparison of the backprojections determined for the plurality of 2D images by the at least one processing circuit to detect the shape deviation.

[0013] The method achieves various technical effects and advantages. The method is based on the use of multiple 2D images in combination with the 3D target geometry. This enables the detection of inconsistencies that indicate a deviation from the 3D target geometry. It is not necessary to reconstruct the actual geometry of an inspection object in 3D in order to detect deviations from the 3D target geometry. Instead, the backprojections of at least two, and advantageously more than two, 2D images onto the 3D target geometry are used to detect whether and where a shape deviation exists between the actual geometry and the 3D target geometry of the inspection object. A technical advantage of the method is that it works on the basis of 2D images that can be captured with a conventional camera, for example a camera integrated into a mobile device.A further technical advantage is that structured lighting is not required. Data acquisition of the 2D images can therefore be performed even by untrained personnel. Compared to conventional techniques, the method offers good robustness against varying lighting conditions and / or different surface textures. Only the outer 3D shell of the inspection object is required as the 3D target geometry. It is not necessary for individual components to be separately addressable within the 3D target geometry. A digitized 3D object (i.e., a shape model without texture) can also be used as the 3D target geometry. This opens up new areas of application.

[0014] Compared to conventional techniques, the method also offers a good ratio of the information content obtained to the time required for data acquisition and evaluation of the 2D images. In particular, it enables a time-efficient comparison between the 3D target geometry and the actual geometry captured in the 2D images. The low setup effort also contributes to time efficiency. Complex setup steps for data acquisition, which are required in numerous conventional methods, are unnecessary with the method according to the invention.

[0015] The comparison can be performed for each of several different facets of the 3D target geometry.

[0016] This allows for the detection of potential deviations for each of the multiple different facets of the 3D target geometry. The comparison can be performed for facets represented in at least two of the received 2D images.

[0017] Based on the comparison, the at least one processing circuit can determine in a spatially resolved manner in which areas an outer shell of the inspection object deviates from an outer shell of the 3D target geometry.

[0018] This means that only the outer shell of the target geometry of the inspection object is required as the 3D target geometry. The 3D target geometry can, in particular, be a 3D target geometry of only the outer shell of the inspection object. This enables efficient processing. Furthermore, the method can be applied to inspection objects for which, for example, only information about the outer shell is available.

[0019] The method may further comprise providing an output based on the detected shape deviation.

[0020] This allows the result of the form deviation detection to be utilized. The result of the form deviation detection can be used in various ways, for example, to output to human operators, to issue a warning when a form deviation is detected, or to directly influence an industrial

[0021] Process, for example a manufacturing or maintenance process.

[0022] The output can include a spatially resolved visualization of the detected form deviation. This makes it easy to see where the form deviation was detected in relation to the 3D target geometry or the actual geometry. The localization of the

[0023] This simplifies the identification of shape deviations and possible elimination of the cause.

[0024] The determination of the plurality of backprojections can be based on poses of an image capture device during capture of the 2D images. The method can further comprise determining the poses by the at least one processing circuit.

[0025] This method enables shape deviation inspection with particularly simple acquisition of 2D images. The poses can be determined automatically by the at least one processing circuit, for example, image-based. It is not required, but possible, for the 2D images to be acquired at specific positions and with specific directions of the image acquisition device relative to the inspection object.

[0026] Determining a pose for a 2D image can be done using the 2D image, the 3D target geometry and camera parameters of an image acquisition device used to capture the 2D image.

[0027] This enables the pose (i.e., the three translational coordinates and the three rotational coordinates of the image capture device relative to the inspection object) to be determined in an image-based manner. The camera parameters can be determined once for any image capture device subsequently used for form deviation inspection and stored non-volatilely.

[0028] Alternatively or additionally, the pose for a 2D image can be determined using sensor data associated with the 2D image.

[0029] This makes it possible to use sensor data to determine the pose. The sensor data can include sensor data acquired by sensors integrated into the image acquisition device (e.g., a global navigation satellite system (GNSS) or an acceleration sensor). Alternatively or additionally, the sensor data can include sensor data acquired and provided by a sensor system separate from the image acquisition (e.g., a tracking system). This allows for even more precise pose determination, which is subsequently used to determine the back projections. Potential ambiguities can be resolved using the sensor data.

[0030] The method may further comprise: reading 3D target geometry data from a storage system and generating a scaled model of an outer shell of the 3D target geometry based on the read 3D target geometry data by the at least one processing circuit, wherein the backprojections are determined based on the scaled model of the outer shell. This allows suitable scaling to be performed before determining the backprojections. For example, linear scaling can be performed such that details are no longer included in the scaled model if they are no longer resolvable in the 2D images anyway. The scaling can thus be performed depending on a resolution in the 2D images and the read 3D target geometry data.

[0031] The method may further comprise: selecting the plurality of 2D images from the received 2D images by the at least one processing circuit.

[0032] This allows the 2D images to be backprojected to be automatically selected. The multiple 2D images to be backprojected can, for example, be automatically selected from a sequence of 2D images captured in a sequence (e.g., a video sequence) by an image capture device.

[0033] The selection of the multiple 2D images can be done in such a way that each facet of the 3D target geometry for which a deviation is to be detectable is visible in multiple 2D images that are back-projected

[0034] This ensures that possible inconsistencies between the backprojections onto the corresponding facet, resulting from a deviation between the actual geometry and the 3D target geometry, can be detected for each relevant facet.

[0035] The selection of the multiple 2D images can be based on the 3D target geometry and poses of the image capture device upon acquisition of the received 2D images. The poses can be determined automatically, as explained above, for example, image-based and / or using sensor data.

[0036] This allows the processing circuit to easily make an automatic selection from the received 2D images based on objective criteria.

[0037] The method may further comprise: receiving additional 2D images by the at least one processing circuit and comparing further backprojections determined from the additional 2D images onto the 3D target geometry to improve the detection of the shape deviation.

[0038] This allows the detection of the shape deviation to continue when the additional 2D images become available. For example, this can be the case if image acquisition for inspection continues while the already available 2D images are evaluated in parallel with the continued image acquisition.

[0039] The method may further comprise: generating control data or control signals by the at least one processing circuit for controlling or guiding the acquisition of the 2D images. This ensures that at least two 2D images are available for backprojection for each facet for which a deviation is to be detected. On-site work on the inspection object is simplified by controlling or guiding the image acquisition.

[0040] The at least one processing circuit can use the 3D target geometry to generate the control data or control signals.

[0041] This automatically ensures that at least two 2D images are available for backprojection for each facet where a deviation is to be detected. On-site work on the inspection object is further simplified.

[0042] The method may further comprise capturing the 2D images using a camera and providing the captured 2D images to the at least one processing circuit. The method may further comprise inducing a relative movement between the camera and the inspection object in order to acquire images from several different perspectives for evaluation.

[0043] This allows the images required for shape deviation detection to be obtained from different perspectives.

[0044] The camera can be incorporated into a movable image capture device. The movable image capture device can be moved relative to the inspection object during the method to acquire 2D images from several different perspectives for evaluation.

[0045] This makes it possible to acquire images from different perspectives by moving the image capture device with the camera relative to the inspection object.

[0046] The camera can be installed in a handheld image capture device. The handheld device can be moved relative to the inspection object during the process to acquire 2D images from several different perspectives for analysis.

[0047] This allows the 2D images to be captured in a simple manner, for example by having the handheld device moved relative to the inspection object by a human user or by using controllable actuators.

[0048] The method may include transmitting the 2D images from the camera to the at least one processing circuit. The transmission may be wired or wireless, for example, via a communications interface.

[0049] This makes it possible to carry out the evaluation of the 2D images to detect the shape deviation using a computing device different from the handheld device.

[0050] The method may include preprocessing or pre-evaluation of images captured by the camera prior to transmission. This allows preprocessing or pre-evaluation to be performed, for example, in a handheld device or other mobile device. The preprocessed data can then be transmitted to the computing device as intermediate processing results.

[0051] This makes it possible to carry out the evaluation of the 2D images to detect the shape deviation partly in the handheld device and partly in the computing device.

[0052] The handset may have a wireless communication interface. The handset may be configured, for example, as a communications terminal, such as a smartphone.

[0053] This makes it possible to use the camera now commonly installed in communications devices (e.g., the rear-facing camera) to capture 2D images. This ensures easy handling during image capture.

[0054] Alternatively or in addition to causing a movement of the image capture device relative to the inspection object, the method may include causing a movement of the inspection object. For this purpose, one or more actuators may be controlled to move the inspection object translationally and / or rotationally relative to the image capture device.

[0055] This allows the images required for shape deviation detection to be obtained from different perspectives.

[0056] Inducing the movement of the inspection object can have control functions that depend on the 3D target geometry.

[0057] This allows the inspection object to be moved in a targeted manner so that images can be captured from multiple perspectives for each facet for which a shape deviation is to be detected.

[0058] The multiple 2D images for which the backprojections are determined comprise at least two 2D images depicting each facet of an outer shell of the 3D target geometry to be examined for shape deviations.

[0059] This ensures that a shape deviation, if present, can be detected from the rear projections.

[0060] The at least two 2D images depicting a facet differ in their pose, advantageously in both their translational and rotational coordinates relative to a coordinate system of the inspection object.

[0061] This ensures that a shape deviation, if present, can be detected from the rear projections.

[0062] The acquisition of the 2D images can be carried out in such a way that an illumination of the inspection object relative to the inspection object remains spatially unchanged, while a relative pose between a camera used for image acquisition and the inspection object changes between acquisition of different 2D images.

[0063] This essentially cancels out shadow effects when comparing rear projections.

[0064] The 2D images can be acquired using one or more illumination sources arranged in a fixed position relative to the inspection object to illuminate the inspection object.

[0065] This essentially cancels out shadow effects when comparing rear projections when the camera is moved relative to the inspection object to capture the 2D images.

[0066] The one or more illumination sources may be arranged to produce illumination other than structured illumination, for example volumetric illumination.

[0067] This allows shape deviation detection to be carried out with little setup effort.

[0068] The comparison may include determining a difference image of two backprojections on at least one facet of the 3D target geometry.

[0069] This makes it possible to determine pixel by pixel for the at least one facet that is displayed in at least two of the 2D images whether and optionally how much the back projections differ from each other.

[0070] Comparing the back projections can take into account that determining the back projections may involve interpolating image pixels during the back projection to enable comparability of the back projections. The perspective and perspective resolution of a single facet are generally different in the 2D images. This can be accounted for by interpolation.

[0071] This makes comparison easier when the perspective and perspective resolution of a single facet are different in the 2D images.

[0072] The comparison may include determining a scatter of pixel values ​​from two or more than two backprojections on at least one facet of the target 3D geometry.

[0073] This makes it possible to determine pixel by pixel for the at least one facet that is displayed in at least two of the 2D images whether and optionally how much the back projections differ from each other.

[0074] The method may include preprocessing the 2D images and / or the backprojections by the at least one processing circuit before performing the comparison. This can reduce potential interference effects that could negatively impact shape deviation detection.

[0075] The preprocessing may include one, several or all of the following preprocessing operations: reduction of specularities and / or reflections;

[0076] Compensation of brightness fluctuations and / or shadows;

[0077] Selection and weighting of 2D images depending on image quality (e.g., depending on sharpness, contrast, and / or other quality attributes) and / or pose quality; sharpening and / or motion compensation.

[0078] This can reduce possible interference effects that could negatively influence the shape deviation detection.

[0079] The detected shape deviation may be a shape deviation caused by a faulty manufacturing process of the inspection object, by a deformation of the inspection object after manufacturing, or by a foreign body remaining on the inspection object.

[0080] This enables the process to automatically detect various shape deviations based on images.

[0081] According to a further aspect of the invention, a machine-readable instruction code is provided which has machine-readable instructions which, when executed by at least one processing circuit, cause the at least one processing circuit to carry out the method according to one aspect or embodiment.

[0082] The effects and advantages achieved by the machine-readable instruction code correspond to the effects and advantages explained with reference to the method.

[0083] According to a further aspect of the invention, a non-volatile storage medium having machine-readable instruction code stored thereon is provided, which has machine-readable instructions which, when executed by at least one processing circuit, cause the at least one processing circuit to carry out the method according to one aspect or embodiment.

[0084] The effects and advantages achieved by the non-volatile storage medium correspond to the effects and advantages explained with reference to the method.

[0085] According to a further aspect of the invention, a device or system for detecting a shape deviation of an inspection object in 3D from a 3D target geometry is provided. The device or system comprises at least one processing circuit configured to determine a backprojection onto at least a portion of the 3D target geometry for a plurality of 2D images of the inspection object and to compare the backprojections determined for the plurality of 2D images to detect the shape deviation. The effects and advantages achieved by the device or system correspond to the effects and advantages explained with reference to the method.

[0086] The device or system may further comprise a camera for capturing the plurality of 2D images.

[0087] The camera can be incorporated into a movable image capture device. The movable image capture device can be movable relative to the inspection object in order to acquire 2D images from several different perspectives for evaluation.

[0088] This makes it possible to acquire images from different perspectives by moving the image capture device with the camera relative to the inspection object.

[0089] The camera can be incorporated into a handheld image capture device. The handheld device can be movable relative to the inspection object to acquire 2D images from several different perspectives for analysis.

[0090] This allows the 2D images to be captured in a simple manner, for example by having the handheld device moved relative to the inspection object by a human user or by using controllable actuators.

[0091] The device or system can be configured to transmit the 2D images from the camera to the at least one processing circuit. The transmission can be wired or wireless, for example, via a communications interface.

[0092] This makes it possible to perform the analysis of the 2D images for the detection of shape deviations using a computer separate from the handheld device. At least part of the analysis can also be performed in the handheld device, with the remaining analysis steps being performed in the computer device.

[0093] The handset may have a wireless communication interface. The handset may be configured, for example, as a communications terminal, such as a smartphone.

[0094] This makes it possible to use the camera now commonly installed in communications devices (e.g., the rear-facing camera) to capture 2D images. This ensures easy handling during image capture.

[0095] Alternatively or in addition to causing a movement of the image capture device relative to the inspection object, the device or system can be configured to cause a movement of the inspection object relative to the camera. For this purpose, the device or system can comprise one or more actuators that can be controlled to move the inspection object translationally and / or rotationally relative to the image capture device.

[0096] This allows the images required for shape deviation detection to be acquired from different perspectives. The device or system can include a control system for inducing the movement of the

[0097] inspection object, which generates control signals depending on the 3D target geometry in order to

[0098] to move the inspection object.

[0099] This allows the inspection object to be moved in a targeted manner so that images can be captured from multiple perspectives for each facet for which a shape deviation is to be detected.

[0100] The device or system may comprise one or more illumination sources arranged in a fixed position relative to the inspection object for illuminating the inspection object.

[0101] This essentially cancels out shadow effects when comparing rear projections when the camera is moved relative to the inspection object to capture the 2D images.

[0102] The one or more illumination sources may be arranged to produce illumination other than structured illumination, for example volumetric illumination.

[0103] This allows shape deviation detection to be carried out with little setup effort.

[0104] The device or system can be configured to carry out the method according to one aspect or embodiment. The processing steps and / or control steps can be executed by the at least one processing circuit. Accordingly, the at least one processing circuit can be configured to carry out the steps explained in connection with optional features of the method.

[0105] The device or system may be configured to detect a shape deviation caused by a faulty manufacturing process of the inspection object, by a deformation of the inspection object after manufacturing, or by a foreign body remaining on the inspection object.

[0106] This enables the device or system to automatically detect various shape deviations based on images.

[0107] According to a further aspect of the invention, a use of the method or device or system according to one aspect or embodiment is provided for detecting a faulty manufacturing process of the inspection object. Depending on the detection of the shape deviation, the manufacturing process can be temporarily stopped and / or modified to correct the shape deviation.

[0108] According to a further aspect of the invention, a use of the method or device or system according to one aspect or embodiment is provided for detecting a deformation of the inspection object after production. The inspection object can be sorted out or modified depending on the detection of the shape deviation.

[0109] According to a further aspect of the invention, a use of the method or device or system according to one aspect or embodiment is provided for detecting a foreign body remaining on the inspection object. The foreign body can be removed depending on the detection of the shape deviation.

[0110] The methods, devices and systems enable the determination of both positive and negative form deviations.

[0111] The methods, devices, systems, and system components according to embodiments of the invention achieve various effects. In particular, detection of a shape deviation is enabled while keeping setup costs to a minimum. Processing by the processing circuitry can be performed efficiently (namely, by determining and comparing backprojections of acquired 2D images). Thus, a result of the shape deviation detection can be provided quickly, for example, in the form of a visualization via a human-machine interface of an image acquisition device with which the 2D images are acquired.

[0112] BRIEF DESCRIPTION OF THE CHARACTERS

[0113] Embodiments of the invention are described with reference to the figures. In the figures, similar or identical reference numerals denote elements with similar or identical design and / or function.

[0114] Figure 1 is a schematic representation of a system according to an embodiment.

[0115] Figure 2 shows the system with a first relative pose between the camera and the inspection object.

[0116] Figure 3 shows the system with a second relative pose between the camera and the inspection object.

[0117] Figure 4 is a flowchart of a method according to an embodiment.

[0118] Figure 5 is a flowchart of a procedure that can be used in the method according to an embodiment.

[0119] Figure 6 shows an exemplary actual geometry of an inspection object to explain embodiments.

[0120] Figure 7 shows an example 3D target geometry for the inspection object of Figure 6.

[0121] Figure 8 shows a 2D image of the inspection object to explain embodiments.

[0122] Figure 9 shows another 2D image of the inspection object to explain

[0123] Examples of implementation.

[0124] Figure 10 shows a backprojection of the 2D image of Figure 8. Figure 11 shows a backprojection of the 2D image of Figure 9.

[0125] Figure 12 shows a comparison result of the backprojections of Figure 10 and Figure 11.

[0126] Figure 13 shows a visualization of a result of the shape deviation detection.

[0127] Figure 14 shows another visualization of a result of the shape deviation detection.

[0128] Figure 15 is a flowchart of a procedure that can be used in the method according to an embodiment.

[0129] Figure 16 is a block diagram representation of a processing circuit of a system or apparatus according to one embodiment.

[0130] Figure 17 is a schematic representation of a system according to an embodiment.

[0131] Figure 18 is a block diagram representation of an image pickup device of a system according to one embodiment.

[0132] Figure 19 is a block diagram representation of an image pickup device of an apparatus according to an embodiment.

[0133] Figure 20 is a flowchart of a method according to an embodiment.

[0134] Figure 21 is a flowchart of a method according to an embodiment.

[0135] Figure 22 is a flowchart of a procedure usable in the method of Figure 21.

[0136] Figure 23 is a schematic representation of a system according to an embodiment.

[0137] Figure 24 is a schematic representation of a system according to an embodiment.

[0138] Figure 25 is a schematic representation of a system according to an embodiment.

[0139] Figure 26 is a schematic representation of a system according to an embodiment.

[0140] Figure 27 is a flowchart of a method according to an embodiment.

[0141] Figure 28 is a schematic representation of a system according to an embodiment.

[0142] Figure 29 is a schematic representation of a 3D target geometry.

[0143] Figure 30 is a flowchart of a method according to an embodiment.

[0144] DETAILED DESCRIPTION OF EMBODIMENTS

[0145] Embodiments of the invention are described with reference to the figures. In the figures, similar or identical reference numerals designate elements with similar or identical design and / or function.

[0146] The features of the embodiments can be combined with each other unless this is expressly excluded in the following description.

[0147] Methods, devices, and systems according to exemplary embodiments enable the image-based detection of shape deviations between the actual geometry of an inspection object and a 3D target geometry. Backprojections of multiple 2D images onto one or more surfaces of the 3D target geometry are determined. The multiple 2D images are captured from different perspectives, i.e., for different poses of a camera relative to the inspection object. From deviations between the backprojections, it can be recognized that the actual geometry deviates from the 3D target geometry, i.e., that a shape deviation exists.

[0148] As explained in more detail below, the methods, devices, and systems according to the invention can detect shape deviations with minimal setup effort. Conventional cameras can be used for 2D image acquisition. The 3D target geometry can be provided in various forms, for example, as a CAD model, a surface mesh, or in another form that defines at least an outer shell of the inspection object.

[0149] The term "3D target geometry," as used here, is to be broadly understood to mean that the 3D target geometry specifies a desired (i.e., nominal) shape of an outer shell of the inspection object. It is possible, but not necessarily required, that the 3D target geometry also contains further information, for example, about components of the inspection object that are not visually perceptible from the outside and / or about materials or surface textures.

[0150] The 3D target geometry can have multiple faces or facets. For any shape, even complex ones, the geometry of the outer shell can be approximated by facets, for example, in the form of a triangular lattice or other faceting. The faces or facets can each be planar.

[0151] The term "facet" refers to a section of 3D geometry. The facet can be flat, i.e., planar.

[0152] Figure 1 shows a system 10 according to one embodiment. The system 10 is configured for image-based detection of a shape deviation of an inspection object 11. The system 10 has a camera 20 for capturing 2D images of the inspection object 11 from several different perspectives (i.e., in several different relative translational and / or rotational coordinates of the camera 20 relative to the inspection object 11).

[0153] The system 10 has a processing circuit 34. The processing circuit 34 is configured to determine backprojections of multiple 2D images onto one or more surfaces of a 3D target geometry and to compare them with each other. This allows the processing circuit 34 to detect deviations between the backprojections. Such deviations between multiple backprojections, each covering the same surface or facet of the 3D target geometry, indicate a shape deviation of the actual geometry of the inspection object 11 from the 3D target geometry.

[0154] The processing circuit 34 is configured to retrieve data defining the 3D target geometry 33 from a storage system 32. To determine the backprojections of 2D images onto a surface of the 3D target geometry, the processing circuit 34 uses pose information that defines the pose (generally three translational coordinates and one rotational coordinate) of the camera 20 relative to the inspection object 11. To determine the backprojections, the processing circuit 34 also uses camera parameters, in particular a camera matrix of the camera 20 or other data (in particular distortion parameters) that define the image point of the camera 20 into which a world point is imaged. To determine the backprojections, the processing circuit 34 can also take distortion parameters into account, in particular.

[0155] Backprojection can be performed computationally efficiently using the aforementioned data. Techniques for mathematically projecting a 2D image onto a surface(s) of a 3D geometry (here: the 3D target geometry) are familiar to those skilled in the art. The intrinsic camera matrix defines the direction along which a pixel value was recorded relative to the image sensor. The pose, in combination with the 3D target geometry, defines the distance from the image sensor to the world point corresponding to the pixel.

[0156] The processing circuit 34 is configured to perform shape deviation detection 35. Through the shape deviation detection 35, the processing circuit 34 determines whether a shape deviation exists and, optionally, where this shape deviation exists. The shape deviation detection 35 comprises the backprojection 36 of at least two 2D images onto the same surface of the 3D target geometry 33 and a comparison 37 of the backprojections. The comparison 37 can be performed in different ways, for example by determining a difference image of the backprojections (where the difference is determined for the surface(s) of the inspection object represented in both backprojections) or by determining a scatter, variance, or other deviation measure of multiple backprojections.

[0157] A result of the shape deviation detection 35 is provided by the processing circuit 34. For this purpose, for example, a human-machine interface (HMI) and / or a data interface can be controlled depending on a result of the shape deviation detection 35.

[0158] The processing circuit 34 can be provided in a processing device 30, which is configured as a separate structural unit from the camera 20. The processing device 30 can have one or more interfaces 31 for receiving the 2D images wirelessly or wired from the camera 20.

[0159] In a further embodiment, the components of the processing device 30 or at least the processing circuit 34 can be integrated together with the camera 20 in a common housing, for example, an image recording device or another handheld device. Figures 2 and 3 show a system 1 according to an exemplary embodiment, which comprises the camera 20 and the processing device 30. The processing device 30 is configured to receive, via at least one data communication connection (for example, via a data communication connection of a communication network 15), a first 2D image 41 that the camera 20 captures when it is in a first pose relative to the inspection object 11. The first pose can comprise three translational coordinates of a characteristic point of the camera (for example, the center of an image sensor) in a coordinate system 12 that is fixedly connected to the inspection object.The first pose may have three rotational coordinates (e.g., three Euler angles) that define the orientation of the camera 20 relative to the first coordinate system 20.

[0160] Figure 3 illustrates that the processing device 30 is configured to receive, via the at least one data communication connection, a second 2D image 42 that the camera 20 captures when it is in a second pose relative to the inspection object 11 that differs from the first pose. The second pose can differ from the first pose by a translational displacement 13 and a rotational movement 14 of the camera 20 relative to the first pose.

[0161] The first 2D image 41 and the second 2D image 42 both show at least one same surface of the inspection object. The processing device 30 is configured to determine a first backprojection of the first 2D image 41 onto the 3D target geometry of the inspection object 11. Information about the first pose is used for this purpose. The processing device 30 is configured to determine a second backprojection of the second 2D image 42 onto the 3D target geometry of the inspection object 11. Information about the second pose is used for this purpose. The processing device 30 is configured to compare the first backprojection and the second backprojection in order to determine whether a shape deviation is present on a surface or facet of the inspection object 11 shown in both the first 2D image 41 and the second 2D image 42.

[0162] Figure 4 is a flowchart of a method 45. The method 45 may be executed automatically by the processing device 30 or by the processing circuit 34.

[0163] Image data is received at 46. The image data comprises 2D images of the inspection object 11. The multiple 2D images can be captured with a camera 20 installed in a handheld device. The multiple 2D images comprise images showing a same surface or facet of the inspection object 11. The multiple 2D images comprise images showing the inspection object from different poses relative to the inspection object.

[0164] At 47, backprojections of multiple 2D images of the received 2D images are determined. The backprojections can be determined depending on the pose of the camera 20 relative to the inspection object upon acquisition of the corresponding 2D image, an intrinsic camera matrix of the camera 20, and the 3D target geometry. At 48, the backprojections are compared. In this process, deviations between 2D images backprojected onto the same surface of the 3D target geometry are determined. Determining the deviations can involve determining a difference image, determining a variance or scatter pixel by pixel, or another pixel-resolved determination of deviations between backprojections.

[0165] A comparison result can be provided, indicating whether and where shape deviations exist. This can involve controlling an HMI or a data interface. A manufacturing process can be controlled and modified based on the comparison result. Depending on the comparison result, information can be provided, alternatively or additionally, regarding where the shape deviation exists on the inspection object. Depending on the comparison result, information can be provided, alternatively or additionally, regarding where a foreign body remains on the inspection object.

[0166] Figure 5 is a flowchart of a procedure 50 that may be executed to implement step 47 of method 45. Procedure 50 may be executed automatically by processing device 30 or processing circuit 34.

[0167] At 51, one or more surfaces or facets of the inspection object that are depicted in at least two of the received 2D images are identified. The identification of these surfaces or facets can be performed depending on the desired 3D geometry and the relative poses between the camera 20 and the inspection object present during image acquisition.

[0168] At 52, optional preprocessing of the image data can be performed. This optional preprocessing can include reducing potential interference effects, for example, through suitable filtering. For example, light reflections can be reduced during preprocessing. Preprocessing can include one, several, or all of the following preprocessing steps:

[0169] Reduction of glare and / or reflections;

[0170] Compensation of brightness fluctuations and / or shadows;

[0171] Selection and weighting of 2D images depending on image quality (e.g. depending on sharpness, contrast and / or other quality characteristics) and / or pose quality;

[0172] Resharpening and / or compensation of movement.

[0173] At 54, the backprojections are determined depending on the poses, the 3D target geometry and the intrinsic camera matrix in order to backproject the 2D images onto surfaces or facets of the 3D target geometry.

[0174] The functioning of the processing circuit 34 and of methods according to exemplary embodiments is illustrated and explained in more detail with reference to FIGS. 6 to 14 using exemplary geometries and 2D images. Figure 6 shows an exemplary actual geometry 60 of an inspection object. The inspection object has a plurality of side surfaces 61, 62, 63. Projections 64 can be formed on a first surface 61. A projection 65 can be formed on a further surface 62. A projection 67 can be formed on yet a further surface 63. The yet further surface 63 can furthermore have, for example, a region 66 that is contrasting in terms of color or texture.

[0175] Figure 7 shows, by way of example, the 3D target geometry 70 of the inspection object. The 3D target geometry 70 defines the side surfaces 71, 72, 73. The 3D target geometry 70 defines projections 74 formed on a first surface 71 of the 3D target geometry. The 3D target geometry 70 defines a projection 75 formed on the further surface 72. In this respect, in the exemplary configurations shown, the actual geometry for the surfaces 61, 62 corresponds to the 3D target geometry of the corresponding surfaces 71, 72. However, after the 3D target geometry 70, the further surface 73 has no projection. The projection 67 thus represents a deviation between the actual geometry and the 3D target geometry.

[0176] Figure 8 shows, by way of example, a first 2D image 41 of the inspection object, on which the surfaces 61, 62, and 63 are depicted. Figure 9 shows, by way of example, another 2D image 42 of the inspection object, on which the surfaces 62 and 63 and an additional surface, which is perspectively hidden in Figure 6, are depicted. The 2D images 41 and 42 differ with regard to the relative pose, here in particular both with regard to the translational position and the rotational orientation, between the camera 20 and the inspection object.

[0177] Figure 10 shows a first backprojection 81. The first backprojection 81 is obtained by projecting the first 2D image 41 onto the 3D target geometry 70. For simplicity, only the backprojection onto the surfaces 72 and 73 is shown, which are visible in both the first 2D image 41 and the second 2D image 42. The shape deviation 67 of the actual geometry is backprojected into a region 87 in the first backprojection 81. The region 87 lies on a surface or facet 84 that is assigned to the surface 73 of the 3D target geometry 70. Similarly, by backprojecting onto the surface 72 of the 3D target geometry 70, a projection 83 of the first 2D image 41 onto this surface or facet is obtained.

[0178] Figure 11 shows a second backprojection 82. The second backprojection 82 is obtained by projecting the second 2D image 42 onto the 3D target geometry 70. For simplicity, only the backprojection onto the surfaces 72 and 73 is shown, which are visible in both the second 2D image 42 and the second 2D image 42. The shape deviation 67 of the actual geometry is backprojected into a region 88 in the second backprojection 82. The region 88 lies on a surface or facet 86 that is assigned to the surface 73 of the 3D target geometry 70. Similarly, by backprojecting onto the surface 72 of the 3D target geometry 70, a projection 85 of the second 2D image 42 onto this surface or facet is obtained. As can already be seen from Figure 10 and Figure 11, the projection surfaces 83, 85 are consistent with each other.However, there are inconsistencies on the projection surfaces 84, 86, which are caused by the projection of the projection 67 visible in the 2D images 41, 42 onto the surface 73 of the 3D.

[0179] Target geometry 70 can be caused.

[0180] To quantify and localize the deviation in the shape of the inspection object from the 3D target geometry, the backprojections can be quantitatively compared. For example, a difference image, a pixel-by-pixel variance, or a pixel-by-pixel scatter of the backprojections can be determined.

[0181] Figure 12 shows, by way of example, a difference image 90 obtained by pixel-by-pixel subtraction of the backprojections 81 and 82 from Figures 10 and 11. The comparison of the backprojections takes into account that determining the backprojections 81, 82 may require interpolation of image pixels during the backprojection for their comparability. The perspective and perspective resolution of an individual facet are generally different in the 2D images 41, 42. This can be accounted for by interpolation. This facilitates a comparison if the perspective and perspective resolution of an individual facet are different in the 2D images 41, 42.

[0182] The differences between the backprojections 81 and 82 caused by the shape deviation 67 result in the difference image 90 having pixel values ​​other than zero (i.e., differences between the backprojections other than zero) in an area assigned to the first surface 63. A corresponding area 91, in which an absolute value of the difference image is other than zero, indicates that the shape deviation 67 is present on the surface 63 of the actual geometry or the surface 73 of the 3D target geometry.

[0183] The result of determining whether a form deviation exists and locating the form deviation can be used in various ways. For example, a human-machine interface can be controlled in such a way that, with reference to the inspection object, it is displayed whether and, if so, where a form deviation exists between the actual geometry and the 3D target geometry.

[0184] Figures 13 and 14 show exemplary visualizations 95, 96 of the determined shape deviation. Figure 13 shows a visualization in which the presence and location of the shape deviation are represented as an overlay 97 on one of the 2D images 41. Figure 14 shows a visualization 96 in which the presence and location of the shape deviation are represented as an overlay 97 on a reproduction of the 3D target geometry 70.

[0185] Embodiments of the invention thus utilize a camera 20, e.g., the camera built into a smartphone or other communications device, such as a tablet. This camera 20 captures images, which may be part of a video sequence, from different directions with a view of the inspection object. The inspection object is recognized in the images, and the pose (translation and orientation) between the camera and the object is determined.

[0186] For form deviation detection, the processing circuit 34 backprojects the digital 2D image recordings of the actual state as a texture onto the 3D target geometry during digital processing. The overlay from different perspectives is always perfectly consistent when the actual geometry matches the 3D target geometry. However, the digital backprojections do not match if a 3D form deviation is present. The form deviation can result from a missing component, an additionally installed component, the attachment of a component with an incorrect shape or incorrect component alignment (translation or rotation), and / or one or more foreign objects (e.g., tools) on the inspection object. The evaluation is performed on various individual facets of the 3D target geometry, which are depicted in at least two 2D images.The evaluation can be carried out for each of the individual facets of the 3D target geometry, which are depicted in at least two 2D images.

[0187] Since only the back-projected perspective overlay is processed as information from the real world, the result is largely independent of the appearance of the inspection object. In particular, the disclosed methods, devices, and systems are relatively robust against variations in the environment (brighter or darker environment) or illumination of the inspection object. The color or texture of the inspection object also plays a subordinate role, because as long as the back-projection is consistent, the color or texture information is eliminated when evaluating the differences between multiple back-projections.

[0188] The reference model (3D target geometry) is solely the visibly perceptible outer 3D shell of the inspection object. Individual components of a structure do not need to be separately addressable or capable of being virtually disassembled. This allows a digitized 3D object (a shape model without texture) to be used as a reference, opening up new applications (for example, in reengineering or computer-aided manufacturing). The application to shape models without texture is particularly important in practical terms.

[0189] The inspection result of this 3D form deviation is available as a 2D difference image for all surfaces or facets of the 3D target geometry. False color-coded (e.g., green - no deviation, red - significant deviation of the perspective back projection), the view from different perspectives allows for a quick assessment of the areas where deviations exist.

[0190] A further advantage of the methods, devices, and systems is that the evaluation of already available 2D images can be performed in parallel with (further) data acquisition. Additional 2D images (e.g., from a continuation of an image sequence or another image sequence) can be added to the evaluation to improve or supplement the result with additional perspectives (rear view, detailed images of individual areas).

[0191] A further advantage of the methods, devices, and systems is that the setup effort is minimal. Only the 3D shell of the inspection object must be available as a reference model (3D target geometry). Additional data used, such as calibration parameters of the camera 20 (camera matrix and / or distortion parameters), can be determined once for the camera 20 used and stored non-volatilely.

[0192] The poses of the camera 20 relative to the inspection object (for example, relative to the coordinate system 12), which are also used to determine the backprojections, can be determined based on the image. Alternatively or additionally, sensor data can be used to determine the poses (in particular, all six degrees of freedom of the pose). From the camera's calibration parameters in combination with the poses, the 2D images can be backprojected as a texture onto the individual facets or surfaces of the 3D target geometry. In this process, a region of the 2D image is rolled over the corresponding facet or surface, with the assignment of pixels of the 2D image to pixels of the backprojection depending on the camera's calibration parameters (to determine the direction of the world point with respect to the camera) as well as the pose and the 3D target geometry (to determine the position of the world point along the beam direction).

[0193] Figure 15 is a flowchart of a method 55 that may be automatically executed by the processing device 30 or the processing circuit 34. Processing steps that were explained with reference to Figure 4 will not be explained again.

[0194] The method 55 includes a determination 53 of the poses assigned to the 2D images. The determination of the backprojections at 54 is performed based on the poses thus determined.

[0195] Poses can be determined image-based. This involves registering the 2D images with respect to the 3D target geometry. The camera's calibration parameters can be computationally taken into account to determine which 2D image is assigned to which pose.

[0196] Sensor data can be used in pose determination. For example, a position sensor (e.g., a GNSS sensor, e.g., a GPS or Galileo sensor), an acceleration or direction sensor, and / or other sensors incorporated into a handheld device including camera 20 can be used to determine poses or pose changes between 2D images.

[0197] Figure 16 shows a block diagram of the processing circuit 34. For shape deviation detection 35, the processing circuit 34 can perform pose determination to determine the relative poses between the camera and the inspection object for the 2D images. The pose determination can be based on the 2D images 105 in combination with 3D target geometry data 106. Imaging parameters 107, in particular calibration parameters of the camera 20, can also be used in the pose determination 101. Optionally, sensor data 108 acquired by sensors of the handheld device housing the camera 20 and / or a different positioning system can be used for the pose determination 101.

[0198] The back projections 102 are calculated depending on the 2D images 105, the 3D target geometry 106 and the imaging parameters 107 as well as the poses determined for the 2D images.

[0199] A determination 103 of differences between backprojections onto the same individual facets or surfaces of the 3D target geometry can be performed by calculating the difference or otherwise determining the scatter or variance between the backprojections. This involves determining, with spatial resolution, which individual surfaces or facets of the 3D target geometry exhibit inconsistencies between the backprojections.

[0200] An interface controller 104 is configured to generate and control one or more control signals 109 depending on the determined differences between rear projections. The one or more control signals 109 can be used to control a human-machine interface. Alternatively or additionally, the one or more control signals 109 can initiate a control function with which, for example, a manufacturing process is influenced in response to a detected shape deviation.

[0201] The human-machine interface, via which a detected shape deviation can be visualized, can be integrated into the processing device 30 or provided remotely therefrom.

[0202] Figure 17 shows an embodiment of system 10 in which the camera 20 is provided in a communications terminal embodied as a handheld device 21. The handheld device 21 also has the human-machine interface 24. A result of the shape deviation detection is transmitted from the processing device 30 via a data communication connection to the handheld device 21 and output there via the human-machine interface 24.

[0203] Figure 18 shows a block diagram of an image capture device embodied as a handheld device 21. The image capture device comprises the camera 20 with imaging optics 22 and a camera chip 23. The image capture device 21 comprises one or more data interfaces 26 for providing captured 2D images to the processing device 30. The image capture device comprises the human-machine interface 24 for outputting a result of the shape deviation detection.

[0204] The image capture device can optionally comprise one or more sensors 25 that acquire data useful for determining the pose of the image capture device. The one or more sensors 25 can comprise, for example, position sensors and / or direction sensors and / or motion sensors. The data acquired by the one or more sensors 25 can be provided to the processing device 30 via the data interface(s) 26, for example, as metadata of the 2D images or in another manner that enables the sensor data to be associated with the 2D images.

[0205] The image acquisition device has a control circuit 110. The control circuit 110 can have one or more integrated circuits, for example, processors, application-specific special-purpose circuits, controllers, field-programmable gate arrays, and / or other integrated semiconductor circuits. The control circuit 110 can execute a camera controller 111 for controlling image acquisition by the camera 20. The control circuit 110 can have optional preprocessing 112 of the acquired 2D images before they are output via the data interface 26. The control circuit 110 can have an interface controller 113 for controlling the data interface 26 to provide the 2D images. Alternatively or additionally, the interface controller 113 can be configured to control the human-machine interface 24 based on a result of the shape deviation detection by the processing device 30 received via the data interface 26.

[0206] Figure 19 shows a modification in which the image recording device is configured such that the control circuit 110 performs the shape deviation detection 35. The result of the shape deviation detection 35 can be output locally at the human-machine interface 24. The result of the shape deviation detection can be output via the at least one data interface 26, for example, for storage and / or for use in correcting the shape deviation.

[0207] Figure 20 is a flowchart of a method 120 that can be executed automatically by the processing device 30 or the processing circuit 34. In the method 120, the result of the shape deviation detection is used to control a human-machine interface. The human-machine interface can be provided remotely from the processing device 30 and integrated into a handheld device that also includes the camera 20.

[0208] At 121, the processing circuitry determines multiple backprojections of 2D images onto surfaces or facets of a 3D target geometry. This unrolls the 2D images onto the surfaces or facets as a texture. At 121, the backprojections are compared to identify inconsistencies caused by an existing shape deviation.

[0209] At 122, a human-machine interface is controlled to visualize the shape deviation relative to the inspection object. The visualization can, for example, be performed as an overlay on one of the acquired 2D images or as an overlay on a representation of the 3D target geometry. Even though the output and use of the detection result by visualizing the shape deviation was explained as an example, the result can also be used in other ways to provide feedback on the existing shape deviation and / or to further utilize the determined information about the existing shape deviation.

[0210] The methods, devices, and systems can utilize (fused) motion sensor technology 25 to facilitate and / or stabilize pose determination. For this purpose, explicit fusion with acceleration sensor data and / or depth image data can be performed, for example. Alternatively or additionally, indirect use can be achieved through devices that support AR ("augmented reality"), for example, communication devices such as a smartphone or tablet.

[0211] In the methods, devices and systems, a handheld device with a display can be used as a human-machine interface 24 for visualizing the shape deviation, user guidance or for displaying the areas that have already been imaged with 2D images.

[0212] In the methods, devices, and systems, the 2D images can be captured using filters (e.g., color and / or polarization filters) to suppress interference. This can suppress gloss, transparency, or similar interference. Alternatively or additionally, the image can be captured in a spectrally shifted range, for example, in the IR or UV spectral range.

[0213] In the methods, devices, and systems, it is advantageous if the digital 3D model of the visible shell, i.e., the 3D target geometry, has a suitable scale to the real object. If this is not the case, scaling can be performed to facilitate processing. If the 3D target geometry is not scaled correctly or with sufficient accuracy, it can be sufficiently scaled or adjusted using a scaling method. For this purpose, fused motion sensors can be used, for example.

[0214] The data defining the 3D target geometry can contain one or more meshes to define the outer visible shell. It is not required, but possible, for individual components in the 3D target geometry to be separately addressable.

[0215] The methods, devices, and systems can preprocess the data defining the 3D target geometry. This preprocessing can be performed in such a way that the backprojections can be computationally performed as efficiently as possible. For example, a conversion to a representation as a mesh (often referred to as a "mesh" in technology), as multiple meshes of appropriate size, and appropriate indexing of the surfaces, facets, or meshes can be performed.

[0216] The methods, devices, and systems can be configured to perform intelligent image acquisition of the 2D images and / or image data reduction. This can include one, several, or all of the following measures: removing or repairing poor image data (e.g., compensated by image information from neighboring 2D images of a sequence); selectively triggering an image acquisition when a minimum rotational and / or translational movement relative to the last acquired image has occurred. For the last measure, the movement can be determined using the additional sensor 25 or the additional sensors 25.

[0217] The methods, devices and systems may be configured to perform processing of acquired 2D images and to visualize or otherwise utilize intermediate shape deviation detection results while additional 2D images are still being acquired.

[0218] The methods, devices and systems can be configured to visualize an intermediate result and / or a final result of the shape deviation detection from different perspectives.

[0219] The methods, devices and systems can be configured so that the visualization is performed as an augmentation from the current perspective of the user or the handheld device in spatial relationship to the inspection object.

[0220] The methods, devices and systems can be configured to receive and process additional 2D images or image sequences of 2D images to improve the result or to add additional views (e.g. back side)

[0221] The methods, devices, and systems can be configured to guide or control image acquisition. This can be particularly advantageous for complex geometries. Desired observation poses are calculated by the processing circuit 34 or the processing device 30. These can be used to guide a user to adopt these or similar camera perspectives. Alternatively or additionally, actuators can be controlled to position the camera and the inspection object in a desired relative pose to each other.

[0222] Figure 21 is a flowchart of a method 125. The method 125 may be performed automatically by the system 10.

[0223] At 126, the image acquisition of the 2D images is guided or controlled under the control of the processing circuit 34. For this purpose, desired poses can be determined from which the inspection object is to be captured with the camera 20. Guided image acquisition can include generating and issuing instructions to a user of the handheld device 21. The instructions can depend on the current pose of the handheld device 21.

[0224] At 127, the acquired 2D images are rolled over as a texture onto surfaces or facets of the 3D target geometry to determine the backprojections. The backprojections are compared surface-by-surface or facet-by-facet to determine whether and, if so, where a shape deviation exists. At 128, an output is generated based on the result of the comparison of the backprojections. The output can be a visualization of the detected shape deviation and / or

[0225] Have control signals to perform a control function.

[0226] Figure 22 is a flowchart of a procedure 130. The procedure 130 may be automatically executed by the system 10 to implement step 126 of the method 125.

[0227] At 131, a human-machine interface can optionally be controlled in such a way that an input is enabled that specifies surfaces that are to be checked for the presence of form deviations.

[0228] At 132, the processing circuit 34 can determine poses in which an image should be captured based on the specified surfaces and the 3D target geometry. This can be implemented such that each of the specified surfaces is mapped onto at least two 2D images.

[0229] At 133, the processing circuit 34 can execute a control function that depends on the determined poses. The control function can guide a user of the handheld device 21 to capture the image. Alternatively or additionally, the control function can control or regulate one or more actuators to achieve the desired relative poses between the camera 20 and the inspection object.

[0230] Figures 23 and 24 show exemplary systems configured for shape deviation detection according to the techniques disclosed in detail herein. The systems 10 of Figures 23 and 24 each include one or more actuators, with processing circuitry 34 configured to control the actuator(s) to achieve desired relative poses between the camera 20 and the inspection object 11.

[0231] Figure 23 shows a system 10 comprising a robot 140. The robot 140 may comprise a movable robot base 141 and / or one or more additional actuators 142 for positioning the camera 20. The robot 140 may comprise a multi-axis robot arm, which allows the camera 20 to be positioned in different orientations relative to the inspection object 11.

[0232] The processing of the captured 2D images by the processing device 30 is carried out as already explained in detail above.

[0233] Figure 24 shows a system 10 having a controllable flying object 150. Under the control of the processing device 30, the controllable flying object 150 can position a camera 20 mounted thereon into desired relative poses to the inspection object 11. The flying object 150 can have actuators that allow the camera 20 to be displaced relative to the flying object 150 and positioned in different orientations relative to the inspection object 11. TI

[0234] As already explained, the methods, devices, and systems according to exemplary embodiments require relatively little setup effort. In particular, structured lighting is not required.

[0235] It is advantageous if one or more illumination sources for illuminating the inspection object 11 maintain their position relative to the inspection object 11 during the acquisition of the 2D images. Such a configuration at least partially compensates for shadow effects when the rear projections are compared with each other.

[0236] Figure 25 is a schematic representation of the system 10 according to an embodiment that includes one or more illumination sources 16 in addition to the camera 20 and the processing circuit 34 provided in a processing device 30. When the camera 20 moves relative to the inspection object 11, the one or more illumination sources 16 remain stationary in the coordinate system 12 of the inspection object 11.

[0237] While embodiments have been described with reference to Figure 2, Figure 3, Figure 23 and Figure 24 in which the camera 20 is moved and the inspection object 11 can remain stationary, the system 10 can alternatively or additionally be configured such that the inspection object 11 is moved translationally and / or rotationally.

[0238] Figure 26 shows a system 10 according to an embodiment, which, in addition to the camera 20 and the processing device 30 with the processing circuit 34, has one or more actuators 145, 146. The actuator or actuators 145, 146 are configured to cause a translational movement 148 and / or a rotational movement 147 of the inspection object 11. The one or more actuators 145, 146 can be controlled by the processing device 30. The control can be carried out such that each relevant surface or facet of the inspection object 11 is imaged in at least two and optionally more than two 2D images.

[0239] In all disclosed embodiments, the method, device, or system can be configured to provide feedback to the user on which areas were captured with no or insufficient coverage of 2D images. The processing device 30 can be configured to generate suggestions or instructions and output them via the handset 21 to instruct the user on where and how the handset should be moved or positioned.

[0240] Figure 27 is a flowchart of a method 160 that may be automatically executed by the processing device 30 or the processing circuit 34. With the method 160, intermediate results determined from a group of 2D images may be output while additional 2D images are acquired and processed in parallel.

[0241] At steps 161 and 162, as already explained, backprojections of 2D images onto the 3D target geometry are determined and compared. An output is generated based on the comparison result of the backprojections. Additional 2D images can be acquired in parallel to these steps 161 and 162.

[0242] At 163, a check is performed to determine whether additional 2D images are available. If additional 2D images are available, additional backprojections can be determined for them. Alternatively or additionally, a check can be performed to determine whether additional image acquisitions are required or recommended. The user can be guided to acquire additional 2D images via an output via a human-machine interface. Processing can be updated by considering additional 2D images for previously verified surfaces or facets of the 3D target geometry and / or by comparing backprojections for additional surfaces or facets for which no comparison result was previously available.

[0243] As already mentioned, the data defining the 3D target geometry can exist in a variety of different forms. It can be advantageous to define the 3D target geometry in such a way that backprojections can be performed efficiently while simultaneously representing a variety of different geometries. For the area- or facet-wise determination of backprojections, it can be particularly advantageous to define the 3D target geometry using a mesh. If the 3D target geometry originally exists in a different format, it can be converted into one or more meshes.

[0244] Figure 28 shows an application of the disclosed devices and methods. The devices and methods can be used to detect a shape deviation resulting from a foreign body 17 (e.g., a tool 17) on the inspection object 11. Thus, the disclosed devices and methods can be configured to detect tools that remain on the inspection object 11, for example, after maintenance work.

[0245] Figure 29 shows an example of such a representation, in which a curved surface section 18 of the inspection object is approximated by a mesh 19. The described techniques of backprojection and backprojection comparison can be implemented such that the 2D images are each rolled onto the different (typically plane) facets of the mesh and then compared facet-by-face.

[0246] Figure 30 is a flowchart of a method 170. In the method, at step 171, a plurality of backprojections are determined and compared with each other, as already described.

[0247] The method 170 includes a step 172 of using the comparison results. Using the comparison results may include identifying a shape deviation, which may result from a missing component, an additionally installed component, the attachment of a component with an incorrect shape or incorrect component alignment (translation or rotation), and / or foreign bodies (e.g., tools) on the inspection object. The evaluation is performed on one or more individual facets of the 3D target geometry, which are depicted in at least two 2D images. Step 172 may include removing a foreign body 17 remaining on the inspection object 11 depending on the detection of the shape deviation.

[0248] All disclosed methods, devices and systems may employ optional additional or alternative features.

[0249] The methods, devices and systems may employ a camera 20 for image capture that includes a stereo camera or depth imaging camera.

[0250] The methods, devices and systems may employ a camera 20 for image acquisition having a shifted spectral range (for example, in the IR or UV range).

[0251] Camera 20 can be configured for monochrome or multispectral image acquisition. The comparison of the back projections can be performed separately for each of the image channels. A deviation of the back projections in one of the image channels may indicate a shape deviation.

[0252] The methods, devices, and systems may utilize a camera 20 for image capture that is integrated into a communications terminal. The communications terminal may be a handheld device, such as a smartphone, a tablet, or a laptop.

[0253] The methods, devices, and systems can use a camera 20 for image acquisition, which can be permanently installed or moved automatically. Designs in which the camera is attached to a measuring arm are also possible.

[0254] The methods, devices, and systems can be configured such that, alternatively or in addition to a movement of the camera 20, a movement of the inspection object 11 can be induced. For this purpose, the system 10 can, for example, comprise a turntable on which the inspection object 11 is positioned.

[0255] The methods, devices, and systems can be configured so that, in addition to the 3D target geometry, further information about the inspection object is stored and used in shape deviation detection. This information can include, for example: o Texture information (e.g., regarding color, gloss, transparency, absorption level) o Resolution into individual components o Information with additional construction stages or integration phases of the inspection object.

[0256] The methods, devices and systems may be arranged to

[0257] User guidance or feedback is provided in addition to or as an alternative to a visualization on the handset 21 in haptic form (e.g. vibration), acoustic form (e.g. different tones) or on a separate optical output device.

[0258] The methods, devices, and systems can be configured to reduce processing to partial surfaces of the 3D reference model in order to accelerate the process. For example, the backprojections can be selective for facets or surfaces of the 3D target geometry if their geometric dimensions reach or exceed a minimum size.

[0259] The methods, devices and systems can be configured such that when actuators are used to change the relative pose between camera 20 and inspection object 11, the movement is automatically calculated and executed before and during the process.

[0260] The methods, devices, and systems can be used not only to detect positive shape deviations (i.e., in cases where the surface of the inspection object is offset toward the camera 20 compared to the 3D target geometry), but also to detect negative shape deviations (i.e., in cases where the surface of the inspection object is offset away from the camera 20 compared to the 3D target geometry). The methods, devices, and systems can thus be configured to detect negative shape deviations.

[0261] A visualization of shape deviations can indicate the presence of a shape deviation based on location. For example, a heat map can be generated and displayed to indicate where a shape deviation exists. Different color coding can be used.

[0262] The methods, devices, and systems can be configured to preprocess the 2D images. The preprocessing can be performed in the handheld device 21 and / or in the processing device 30. The processing steps can be based on individual 2D images or on the entire set of 2D images to utilize different perspectives.

[0263] Possible processing steps may include:

[0264] Reduction of specularity and reflections: For this purpose, an additive model can be used, for example, to decompose the image into a specular or glossy component and a diffuse component. A trained machine learning model, such as a trained neural network, can be used to separate the diffuse component from the glossy component. The trained machine learning model can have an input for receiving the 2D image. The trained machine learning model can provide the diffuse component or the diffuse component and the glossy image as output.

[0265] Compensating for brightness fluctuations and shadows: Classic analytical image processing methods can be used for this. Examples include histogram-based techniques. It is also possible to use a trained machine learning model, as explained in the previous bullet point.

[0266] Selection and weighting of images based on image quality (sharpness, contrast, etc.) and / or pose quality: Pose quality can be determined by an assessment based on the pose calculation, which reflects the uncertainty of the determined pose. Image quality can be assessed based on the image data (e.g., by detecting reflections or poor histograms).

[0267] Sharpening and motion compensation: This allows motion blur to be reduced based on the entire image.

[0268] Implementation details of such techniques are familiar to those skilled in the art.

[0269] The 2D images can, in particular, comprise 2D images of an image sequence, for example, a video sequence. Advantageously, the 2D images can be captured while the camera 20 continuously changes its pose relative to the inspection object. This makes it possible, in particular, to capture the 2D images as images of a video sequence. Suitable 2D images from the image sequence can be selected based on image quality, the quality of pose determination, and the required coverage of the facets or surfaces of the inspection object.

[0270] In one embodiment, the device or system is thus used as follows: i. The camera 20 is provided in a handheld device 21. The handheld device 21 is put into an operating state for capturing the 2D images. ii. The 2D images are captured during a relative movement between the handheld device 21 and the inspection object 11. The 2D images are provided to the processing circuit 34. iii. The processing circuit 34 can select several of the 2D images for further processing. The processing circuit 34 determines the poses associated with the plurality of 2D images. The selection can be made, as described above, based on quality criteria relating to the 2D images, the pose determination, and / or the desired coverage of the inspection object. iv. The plurality of 2D images are computationally back-projected by the processing circuit 34 onto the facets of the 3D target geometry (i.e., rolled as a texture onto the facets) depicted in them.v. For each facet for which at least two backprojections are available, a comparison of the backprojections can be carried out by the processing circuit 34. This can involve quantifying the differences, for example by pixel-by-pixel subtraction (in the facet onto which the backprojection was made), or pixel-by-pixel determination of a scatter or variance (in each case in the facet onto which the backprojection was made). vi. An output dependent on the comparison result is generated. This can involve controlling the human-machine interface 24 to visualize the shape deviation. vii. Feedback can be generated and output to the user, providing information about the coverage already achieved with 2D images and / or guiding or controlling further image acquisition. The feedback can also include the quality of the spatially resolved detection of the shape deviation already achieved. viii.Continued image acquisition and processing of the additional 2D images to improve the evaluation results can be enabled while a first group of 2D images has already been processed.

[0271] As mentioned, the processing circuit 34 and the camera 20 can be implemented in different, movable structural units or combined in a single device. The processing circuit 34 can comprise one or more integrated circuits to perform the required processing steps. The one or more integrated circuits can, for example, comprise any one or any combination of the following circuits or circuit components: an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a processor, a controller, one or more quantum gates, a quantum information processing circuit, and other integrated circuits.

[0272] While exemplary embodiments have been described with reference to the figures, modifications can be implemented in further exemplary embodiments. Exemplary specific modifications and developments have already been explained above. Further modifications are possible. While, for example, the acquisition of the 2D images and optional preprocessing prior to transmission of the 2D images is possible using a handheld device, other mobile, particularly portable, devices can also be used.

[0273] While embodiments have been described that may be used in industrial manufacturing or quality control systems, the disclosed techniques may also be used in other application areas.

[0274] The present disclosure also encompasses embodiments with any combination of features mentioned or shown for different embodiments. It also encompasses individual features in the figures, even if they are shown there in connection with other features and / or are not mentioned above or below. Furthermore, the alternative embodiments described in the figures and the description and individual alternative features thereof may be excluded from the subject matter of the invention or from the disclosed subject matter.

[0275] The terms "comprise" and "have" and derivatives thereof indicate a non-exhaustive relationship and do not exclude the presence of other elements or steps. The indefinite article "a" or "an" and derivatives thereof do not exclude the presence of a plurality of the corresponding elements. The functions of several features listed in the claims may be fulfilled by one unit or step.

[0276] A machine-readable instruction code that can be executed by a programmable circuit to perform methods according to embodiments may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware. The instruction code may also be distributed in another form, such as a modulated data signal sequence.

[0277] Embodiments of the invention provide improved techniques for image-based 3D shape inspection, which in particular offer improvements in terms of setup effort and speed in providing the inspection results.

Claims

CLAIMS 1. A method for detecting a shape deviation in three dimensions, 3D, of an inspection object (11; 60) from a 3D target geometry (33; 70), the method comprising: Receiving two-dimensional, 2D, images (41, 42) of the inspection object (11; 60) by at least one processing circuit (34), Determining a plurality of backprojections (81, 82) onto the 3D target geometry (33; 70) by the at least one processing circuit (34), wherein for a plurality of 2D images (41, 42) of the inspection object (11; 60) a backprojection onto at least a part of the 3D target geometry (33; 70) is determined, and Carrying out a comparison of the backprojections (81, 82) determined for the plurality of 2D images (41, 42) by the at least one processing circuit (34) to detect the shape deviation.

2. The method of claim 1, wherein the comparison is performed for each of a plurality of different facets of the 3D target geometry (33; 70).

3. The method according to claim 1 or claim 2, wherein the at least one processing circuit (34) determines, based on the comparison, in a spatially resolved manner, in which regions an outer shell of the inspection object (11; 60) deviates from an outer shell of the 3D target geometry (33; 70).

4. The method of any preceding claim, further comprising: providing an output based on the detected shape deviation.

5. The method of claim 4, wherein the output comprises a spatially resolved visualization of the detected shape deviation.

6. The method according to any one of the preceding claims, wherein the determination of the plurality of backprojections (81, 82) is carried out based on poses of an image capture device when capturing the 2D images (41, 42), the method further comprising determining the poses by the at least one processing circuit (34).

7. The method according to claim 6, wherein determining a pose for a 2D image is carried out using the 2D image, the 3D target geometry (33; 70) and camera parameters of an image capture device used to capture the 2D image, and / or sensor data associated with the 2D image.

8. The method according to any one of the preceding claims, further comprising: reading out 3D target geometry data (33) from a storage system (32), and generating a scaled model of an outer shell of the 3D target geometry (33; 70) based on the read out 3D target geometry data (33) by the at least one processing circuit (34), wherein the backprojections (81, 82) are determined based on the scaled model of the outer shell.

9. The method according to any one of the preceding claims, further comprising: selecting the plurality of 2D images (41, 42) from the received 2D images (41, 42) by the at least one processing circuit (34).

10. The method according to any one of the preceding claims, further comprising: receiving additional 2D images (41, 42) by the at least one processing circuit (34) and comparing further backprojections (81, 82) determined from the additional 2D images (41, 42) onto the 3D target geometry (33; 70) to improve the detection of the shape deviation.

11. The method according to any one of the preceding claims, further comprising: generating control data or control signals by the at least one processing circuit (34) for controlling or guiding an acquisition of the 2D images (41, 42), wherein the at least one processing circuit (34) uses the 3D target geometry (33; 70) to generate the control data or control signals.

12. Method according to one of the preceding claims, wherein the detected shape deviation is caused by a faulty manufacturing process of the inspection object (11; 60), by a deformation of the inspection object (11; 60) after manufacturing or by a foreign body remaining on the inspection object.

13. Machine-readable instruction code comprising machine-readable instructions which, when executed by at least one processing circuit (34), cause the at least one processing circuit (34) to carry out the method according to one of the preceding claims.

14. A device (21) or system (10) for detecting a shape deviation in three dimensions, 3D, of an inspection object (11; 60) from a 3D target geometry (33; 70), comprising: at least one processing circuit (34) configured to determine a backprojection (81, 82) onto at least a portion of the 3D target geometry (33; 70) for a plurality of two-dimensional, 2D images (41, 42) of the inspection object (11; 60), and to compare the backprojections (81, 82) determined for the plurality of 2D images (41, 42) in order to detect the shape deviation.

15. A device or system according to claim 14, further comprising: an image capture device (20, 21) for capturing the plurality of 2D images (41, 42).

Images