Method, computer program and apparatus for directional camera

Abstract

The invention relates to a method for orienting a predetermined number of real cameras (18) for measuring real three-dimensional objects and to a corresponding device. The device comprises cameras and a pattern in a predetermined coordinate system, in which the pattern is projected onto the surface of the real object by means of at least one projector (15) for measurement, and said pattern is at least partially recorded with each of a predetermined number of real cameras. Furthermore, there is a three-dimensional virtual model of the ideal object (12 ') corresponding to the real object. In order to reduce the workload required to orient the plurality of cameras, indication and / or control information for orienting each camera is determined by a plurality of target markers (23) on the ideal object surface in the three-dimensional virtual model and provided at a predetermined interface.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for orienting a predetermined number of real cameras to measure a real three-dimensional object in a three-dimensional coordinate system using the cameras and a pattern. Background Art

[0002] Stereoscopic methods are generally used for non-contact measurement, in particular for non-contact measurement of three-dimensional objects in a three-dimensional coordinate system. In the stereoscopic method, a pattern is projected onto the surface of an object by a projector, and the pattern projected onto the surface of the object is recorded using at least two cameras. For example, this method is known from document DE 10 2018 108 874 A1. The known method uses a camera that has been calibrated internally and externally. In order to achieve a more robust and less interfering measurement, a two-dimensional code and a time code are generated during the pattern projection by using a (completely) two-dimensional coded pattern and recording the pattern projected onto the surface of the object using a camera. In addition, a time-coded pattern with different two-dimensional codes is projected multiple times in succession, and multiple records of the pattern projected onto the surface are generated in sequence using a camera, each of which is triggered simultaneously.

[0003] The known method is very suitable for objects which are measured in large numbers or in a similar form under constant conditions. If a large number of different objects are measured at the measuring point, the above method is relatively complicated due to calibration.

[0004] If a large number of cameras are used at a measuring point to measure different objects, the arrangement and orientation of the cameras must also be changed and adapted to the respective measuring task. For a large number of cameras, setting up and calibrating the measuring point and especially the orientation of a large number of cameras is very time-consuming.

[0005] From document DE 10 2012 109 351 A1, an evaluation unit for processing images and / or image sequences of a measuring area detectable on a collision vehicle is known. The measuring camera can be oriented using an orientation camera permanently mounted on the measuring camera. Alternatively or in addition, at least three reference marks arranged in the measuring area can be recorded. Document DE 10 2009 035 840 A1 discloses a method for positioning and / or adjusting a high-speed camera for a motor vehicle collision test, wherein the position, angular position and / or focal length of the high-speed camera are calculated by photogrammetric orientation data from a database or a CAD system and are reproducibly positioned and / or adjusted by a photogrammetric device. Finally, document US2017 / 0026636 A1 describes a method for spatially accurately projecting a mark on an object, wherein a two-dimensional or three-dimensional model of the object is used. Summary of the invention

[0006] The object of the present invention is therefore to propose a method or to create a device which allows easy and time-saving orientation of a camera for precise measurement of objects of any shape and size.

[0007] This object is achieved by a method having the features of claim 1 , a computer program having the features of claim 7 , an apparatus having the features of claim 8 and a system having the features of claim 15 .

[0008] In particular, the object is achieved by a method for orienting a predetermined number of real cameras for measuring a real three-dimensional object in a predetermined coordinate system using the cameras and a pattern, wherein for said measuring said pattern is projected onto a surface of the real object by means of at least one projector and said pattern is at least partially recorded with each of said predetermined number of real cameras, wherein there is a three-dimensional virtual model of an ideal object corresponding to the real object, the method comprising the following steps:

[0009] a) directing at least one projector onto the surface of the real object so that the position of the pattern on the surface of the real object corresponds to the predetermined position of the pattern on the virtual model;

[0010] b) defining or providing a plurality of target marks on a surface of an ideal object in the three-dimensional virtual model, the surface of the ideal object corresponding to the surface of the real object to be measured;

[0011] c) automatically determining reference information for each camera, wherein the reference information comprises identification information containing information of at least one target marker among a plurality of target markers captured by the corresponding camera when the corresponding camera and the projector of the pattern are arranged and oriented in a predetermined manner, and the reference information further comprises position information associated with the corresponding camera and the target marker, the position information containing position information of the at least one target marker appearing in a captured image of a field of view of the corresponding camera when the corresponding camera and the at least one projector of the pattern are arranged and oriented in a predetermined manner, wherein the reference information for each camera is determined based on a three-dimensional virtual model of an ideal object and a corresponding virtual representation of the camera and the pattern; and

[0012] d) recording an image of a surface of a real object using a real camera, automatically calculating instruction and / or control information, and providing the instruction and / or control information to the corresponding camera at a predetermined interface, the instruction and / or control information being used to find a specified target marker in the image of the corresponding real camera, and orienting the corresponding real camera using the real object based on reference information determined for the corresponding camera,

[0013] Wherein, step d) is performed for each camera among the predetermined number of cameras.

[0014] As described above, the above method is used to set up measuring points for measuring real three-dimensional objects (e.g. chassis, housing parts, battery modules, windshields, etc.), in particular for orienting cameras at the measuring points. If such measuring points are used for very different objects with different shapes and sizes and a large number of cameras, this setting work using the above orientation method will be simplified and can be completed in a shorter time. The automatic generation of reference information for orientation and indication and / or control information, which will be described in more detail below, enables efficient display and automatic assistance or implementation of orientation, for example on a screen / monitor, thereby also automatically reducing the time required for setting up the measuring points and the time required for the entire measurement.

[0015] In particular, in the steps of automatically determining the reference information and automatically calculating the indication and / or control information to find the target mark of each real camera and align the real camera, the above method can be implemented as a computer-implemented method, that is, a method executed using a computing unit (computer). The determination of multiple target marks on the surface of the ideal object can also be performed automatically or semi-automatically, that is, partially automatically and partially manually, or completely manually. In the case of manual determination, the computer-implemented method can be configured to interactively request (manually) determine multiple target marks on the surface of the ideal object, for example, by inputting the target mark in a corresponding predetermined interface and storing the input target mark in a storage unit. When the target mark is semi-automatically or automatically determined based on the three-dimensional virtual model of the object, the target mark can be selected so that (when the camera is oriented) a predetermined number of target marks are arranged in the field of view of each camera, and these target marks are distributed over the entire image of the corresponding camera.

[0016] For example, after the cameras have been oriented and any necessary internal and / or external calibration of the cameras has been performed, the above-described stereoscopic method can be used to measure a three-dimensional object with a predetermined number of cameras that capture the surface facing the respective cameras. One or more projectors can be used to generate the pattern required for this, which is projected onto the surface of the object.

[0017] For example, the specified coordinate system is a Cartesian coordinate system; in one embodiment, the coordinate system used may be a coordinate system of the object. The coordinate system is used as a global coordinate system, in which a corresponding transformation matrix is ​​provided to transform the camera coordinates and / or the pattern coordinates into a predetermined coordinate system.

[0018] A three-dimensional virtual model is a virtual model of an ideal object corresponding to the real object to be measured, for example a CAD model of the ideal object. Usually, a large number of real objects are to be measured, all of which correspond essentially to the ideal object and each of which has greater or lesser deviations from the ideal object, for example due to production and / or material reasons. The model of the ideal object is stored in a storage unit, for example connected to a computing unit, and can be retrieved there.

[0019] According to the present invention, a three-dimensional virtual model of an ideal object is known, and a plurality of target markers on the surface of the ideal object are identified in the model. These target markers are virtual markers, and for example constitute special points such as holes, edges, protrusions, etc. on the surface of the ideal object, or are arranged in the area of ​​these points, which are easily identifiable, for example, by contrasting with the surrounding environment in the camera image through their shapes. For example, each target marker is a small part of the surface of the three-dimensional model, for example, including several points / pixels of the model. The surface portion of the object in the virtual model may include a pattern that exists in the portion on the surface of the virtual model and is projected onto the surface. As an alternative to two-dimensional coding, if a time-varying pattern is projected onto one or more pixels of the virtual model, the target marker may contain a unique time code in one or more pixels. It is very advantageous to orient the target marker to the three-dimensional virtual model of the ideal object and the projected pattern so that it can be oriented faster. The target marker is set on the surface of the ideal object with a predetermined density or distance, wherein in one embodiment, the target marker is set individually and separately for each virtual model / object.

[0020] Using a three-dimensional virtual model of an ideal object and corresponding virtual representations of cameras and patterns, benchmark information is determined for each camera in a predetermined number of cameras, wherein the benchmark information includes identification information and position information. The benchmark information is calculated based on the three-dimensional virtual model of the ideal object and the target marker defined thereon. In addition, the position of the corresponding camera in a specified coordinate system (i.e., its position in a three-dimensional coordinate system) and the direction of the corresponding optical axis orientation are defined for each camera, so these positions are already known before calculating the benchmark information. This known data is called the representation of the corresponding camera. In addition, the position of at least one projector that projects a pattern and the direction of its projected pattern (i.e., its orientation direction) are defined and known. This data is called the representation of the pattern. The pattern can be generated with a single projector, two projectors, or multiple projectors, wherein, if multiple projectors are used, the corresponding representation of each projector is of course known. In this method, the benchmark information is calculated entirely from these known data by modeling / virtualization, wherein, as described in more detail below, the determined camera benchmark information is used as a benchmark for the orientation of the corresponding real camera. In one embodiment, the benchmark information can be calculated using a known photogrammetry method (e.g., close-range photogrammetry).

[0021] For each camera, the identification information forming part of the reference information respectively comprises information about at least one target marker of a plurality of target markers captured by the respective camera, i.e. information about at least one target marker appearing in the image produced by the respective camera. For example, the target markers may be numbered and the number of target markers seen by the respective camera may be determined. For example, the target markers may be 1 to 4 target markers distributed over the entire image of the respective camera.

[0022] The position information constituting another component of the reference information is data containing the following information: when the corresponding camera and the projector of the pattern are arranged and oriented in a specified manner, the position of the at least one target mark appears in the virtually calculated (virtual captured) image of the field of view of the corresponding (virtual) camera. The position information may include, for example, three-dimensional or two-dimensional position / location coordinates in a coordinate system associated with the corresponding camera, or three-dimensional or two-dimensional position / location coordinates in a coordinate system of a two-dimensional image (virtually) captured by the corresponding camera.

[0023] In the method, the orientation of the real camera is performed after the reference information is determined. To this end, the camera is first arranged at a specified position in a specified coordinate system. In addition, the (multiple) projectors of the pattern are arranged in advance at a specified position in the specified coordinate system and oriented. Now, each real camera records an image of the surface of the real object recorded by the corresponding camera corresponding to the three-dimensional virtual model, and compares it with the reference information (i.e., the identification information and (multiple) position information associated with the corresponding camera) in the analysis. Wherein, when the image is taken, the pattern generated by the (multiple) projectors is projected onto the surface of the real object. As a result of the comparison, the instructions and / or control information for finding the defined target mark in the image of the corresponding real camera and for orienting the corresponding real camera are calculated. These instructions and / or control information are calculated separately for each real camera, and then provided at a predetermined interface (i.e., the interface of the computing unit). When the instructions and / or control information are transmitted to the automatic adjustment device or displayed to the user (e.g., displayed on a screen), the user and / or the corresponding automatic adjustment device can orient each real camera based on the instructions and / or control information.

[0024] In one embodiment, the calculation of the indication and / or control information may be performed multiple times for each real camera, so that the orientation is performed in multiple steps, which improves the accuracy of the orientation. Therefore, the images captured by the corresponding cameras are compared with the reference information multiple times. For example, after the indication and / or control information for orientation is calculated and provided for the first time, all cameras can be roughly oriented, and then after the indication and / or control information is calculated and provided for the second time, fine adjustments can be made.

[0025] In an embodiment, the camera or a portion of a predetermined number of cameras is at least substantially internally calibrated. This may improve the accuracy of the orientation of the camera and (if applicable) the projector(s).

[0026] In one embodiment, at least one target mark on the surface of the real object can be searched for based on the identification information in the image of the corresponding camera in order to determine the indication and / or control information for orientation, and if necessary, a real pattern projected onto the surface of the real object can be used. Then, based on the (multiple) target marks found in the image of the corresponding camera, the indication and / or control information for finding the at least one target mark is calculated, i.e., whether the camera must be rotated left / right / upward / downward or not moved, so that the (multiple) target marks expected / specified based on the identification information in the image of the corresponding camera appear in the image of the camera after the corresponding orientation. Therefore, the indication and / or control information calculated based on the identification information can include direction and / or length information, which describes the direction and / or amount in which the corresponding camera is to be oriented (for example, to be rotated).

[0027] In one embodiment, the instruction and / or control information for orienting the corresponding real camera is determined by comparing the identification information with at least one target mark visible in the image of the corresponding real camera, and / or by comparing the determined position information of at least one target mark of the corresponding camera with the position of at least one target mark in the image of the corresponding real camera. For example, the position information may contain information that the target mark is arranged on the optical axis of the camera. When the real image of the camera is compared with the position information, the distance between the collision point of the optical axis and the target mark in the image recorded by the real camera and the direction in which the target mark is located at the collision point of the optical axis are determined. Therefore, the instruction and / or control information is calculated to move the camera so that after orientation, the target mark is located at the collision point of the optical axis or as close to the collision point of the optical axis as possible in the image of the real camera. Therefore, the instruction and / or control information calculated based on the position information may include direction and / or angle (size) information, which describes the orientation direction of the corresponding camera and / or the angle (size) to be oriented (for example, to be rotated).

[0028] In one embodiment, a quality evaluation (Gütemaβ) of the deviation of the current orientation of the corresponding real camera from the reference information determined for the corresponding camera is determined and provided as indication and / or control information at a predetermined interface. In one embodiment, a quality evaluation with a smaller absolute value or actual amount indicates that the alignment is already very good, while a quality evaluation with a larger absolute value or actual amount indicates that the orientation of the corresponding camera (or a predetermined camera group among multiple cameras) is still insufficient. For example, the quality evaluation can be calculated as the sum of the square of the distance (or: the absolute value of the distance) of each target in the real camera image from the determined position information of the corresponding target over all targets of the corresponding camera.

[0029] In one embodiment, the pattern projected onto the surface is a completely two-dimensionally coded pattern, wherein a completely two-dimensionally coded pattern is understood as a pattern that is encoded at least in the entire projection area of ​​interest, which is a portion of the pattern that is displayed on an object (or more precisely, the surface of an object) by projection. In principle, the pattern that implements two-dimensional coding is projected in the case of two-dimensional coding, which means that the pattern point can be clearly identified in the entire pattern or at least in a specific pattern environment surrounding the pattern point. It has been proven to be particularly advantageous here that the resolution of the two-dimensional pseudo-random pattern is selected so that the camera can just reliably distinguish a single pattern point, that is, the resolution of the camera pixel is higher than that of the single pattern point. In other words, the pixel of the camera on which the pattern is imaged is smaller than the pattern point of the pattern imaged in the camera pixel. Therefore, a pattern point is described by several pixels of a camera or a camera image, and these pixels are preferably adjacent in two dimensions. How high the resolution of the pattern point must be so that the image processing system can reliably locate (i.e., distinguish) a single pattern point depends on the respective optical conditions and can be determined and set by those skilled in the art by theoretical considerations and / or experiments when setting up the system. Depending on the application, a pattern point can usually be imaged with only one camera pixel. However, it is often meaningful that the pattern points consist of a plurality of camera pixels arranged in two dimensions.

[0030] In one embodiment, the pattern can be a completely two-dimensional or time-coded pattern as described above or any other two-dimensional pattern. To project such a pattern, a digital micromirror device (DMD) or other digital light processing projector (DLP projector) or a simple (conventional) slide projector can be used, which projects a pattern (physically) present on a transparent carrier (slide), wherein different brightnesses are generated by appropriately controlling the projection lamp and / or by filters of different densities (e.g., gray filters). These filters are superimposed on the transparent carrier of the pattern (pattern carrier) during the projection. Such a slide projector can also be used for spatial displacement of a two-dimensional pattern. In this case, it is only necessary to (minimally) spatially displace and / or tilt the slide carrier of the originally fixed slide projector (including its spatially fixed projection optics), on which the slide carrying the pattern is arranged. Then, according to the projection optics, the movement is displayed on the surface of the object. The advantage of such a slide projector that can be used to perform the method described herein is that they are inexpensive, rugged and technically easy to operate. Therefore, a slide projector is particularly suitable for industrial environments. When using a digital projector, any desired pattern that can be displayed on a display can be projected. In addition to two-dimensional (structured) patterns, the projector can also project a brightness distribution (which itself has no structure) onto the object. This can be achieved relatively easily using a digital projector. Alternative gray value gradients (or in broader terms: brightness gradients) can also be used.

[0031] Programs and methods using artificial intelligence (computer vision), fuzzy logic and / or the like may be used to determine baseline information for each camera and / or calculate instruction and / or control information.

[0032] In one embodiment, the interface is connected to a display device such as a screen or mixed reality glasses, and / or a speaker, and / or a control device for a plurality of motors, wherein the indication and / or control information is processed by the display device and / or the speaker and / or the control device, and / or is output on the display device and / or the speaker, and / or is transmitted to the motor by the control device, so that the real camera is automatically oriented according to the determined reference information of the camera. The indication and / or control information can be displayed and / or output as an arrow and / or a corresponding information text, for example. Alternatively or in addition, the image of the real camera and the image of the camera embodying the reference information can be displayed on the screen, for example, superimposed on top of each other with different brightness. The embodiment of the reference information can, for example, display at least one target mark of the corresponding camera in the image, so that the mark (for example, a circle and / or a crosshair) is exactly arranged at the position where the target mark should appear according to the position information. Therefore, the user can very easily determine whether the image of the real camera matches the camera image representing the reference information.

[0033] In one embodiment, if at least two target marks are provided for the orientation of one camera, the target marks can be connected to each other and, if necessary, a connecting line can be included as another target mark in the determination of the reference information and / or in the calculation and provision of the indication and / or control information. Thus, the target mark can have any two-dimensional shape, such as a point shape or a line shape, wherein target marks with different shapes can be combined for one camera. For example, when using indication and / or control information on a screen, a linear target mark can facilitate the orientation of the user.

[0034] In one embodiment, at least one auxiliary camera can be additionally provided to perform the method, which is used to orient at least one projector to the real object in step a). Typically, during the orientation of the projector, an operator or at least one camera (e.g., the at least one auxiliary camera mentioned above) observes the scene, in particular the position of the pattern relative to the real object. The position of the pattern is then compared with the position of the same pattern relative to the virtual model (manually by the operator or automatically by a computing unit), and it is determined whether the position of the pattern relative to the real object corresponds to the predetermined position of the pattern on the virtual model. Based on the result of the comparison, the orientation of at least one projector is changed. This means that the projector is oriented differently so as to adjust the position of the pattern relative to the real object so that it corresponds to the predetermined position of the pattern relative to the virtual model. Here, the pattern can be located by markings and visible points (e.g., edges, holes and similar structures) on the surface of the real object. As an alternative to at least one auxiliary camera, at least one camera can be used to orient at least one projector, which is then used to measure the real three-dimensional object and is oriented for this purpose (after orienting at least one projector and before measuring).

[0035] Since measurement methods such as stereoscopy require at least two images of an image point, according to one embodiment, it is proposed to use at least three cameras, preferably four cameras, for measuring real objects, which, after camera orientation and calibration, record pattern portions or object surface portions from different viewing angles. The cameras are thus also oriented toward the same or similar portions of the object surface. This results in redundant image recording during measurement, thereby increasing the robustness of the measurement process.

[0036] By using at least three cameras, preferably four cameras and possibly even more cameras for a pattern or part of the object surface, the problem existing in stereo measurement practice, namely the problem that the system reacts very sensitively to defects in the calibration, can be solved. In order to be able to carry out stereo measurements, the relevant cameras must be calibrated internally and externally (as accurately as possible). Here, a rough internal calibration of the camera can be carried out before orienting the camera, and a fine internal calibration can be carried out after the camera is oriented according to the above method. Alternatively, the camera can be calibrated internally and externally according to known photogrammetric methods. In particular, target marks defined automatically, semi-automatically or manually (interactively) can be used as new correspondences in the photogrammetric beam block adjustment calculation. Since many control points are also included in the adjustment calculation in addition to the photogrammetric marks and scales, this optimizes the calibration of the system on the surface of the real object to be measured.

[0037] Intrinsic calibration consists in determining all the parameters describing the camera itself, regardless of its position, such as focal length, optical distortion, principal point offset, etc. Many different methods are known in the literature on how to perform such a calibration.

[0038] In extrinsic calibration, the position of the camera is determined relative to a specified reference coordinate system. In this case, the reference coordinate system can be the coordinate system of the object. Again, several methods are described in the literature.

[0039] According to one embodiment of the proposed device, at least ten cameras are provided and are arranged so as to record a portion of the surface of an object or a pattern from at least four different angles, wherein (as described above and for the proposed system and the proposed method) all cameras are calibrated externally and internally.

[0040] According to an embodiment of the device, the device comprises a computing unit (a computer with a corresponding processor) configured to perform the above method or its steps (see also the above description of the method implemented by the simulation computer). In particular, the computing unit is arranged to perform the following steps: determining reference information and calculating instructions and / or control information, and providing this information at a predetermined interface. As mentioned above, the step of determining the target mark can also be performed by the computing unit. Please refer to the description of the above method. A person skilled in the art will combine the device features together or proportionally with the functions into the proposed system according to the present invention as needed.

[0041] The computing unit (also referred to as a controller, control device or control unit) may include a processing unit and a memory (also referred to as a storage unit), in which computer executable instructions for performing the methods described herein are stored. The processing unit or other unit may include any suitable device configured to perform a series of steps to implement the method, so that when the instructions are executed by the computing device or other programmable device, the functions / actions / steps specified in the methods described herein can be performed. The processing unit or other unit may include, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processor (DSP), a central processing unit (CPU), an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, other appropriately programmed or programmable logic circuits, or any combination thereof. The memory may be any suitable known or other machine-readable storage medium. The memory (data carrier) may be a non-volatile computer-readable storage medium, such as an electronic system, a magnetic system, an optical system, an electromagnetic system, an infrared or semiconductor system, or any suitable combination of the above media. The memory may include any suitable combination of any type of computer memory located inside or outside the device or computing unit, such as random access memory (RAM), read-only memory (ROM), compact disk read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), ferroelectric RAM (FRAM), etc. The memory may include any storage device (e.g., device) suitable for retrievably storing a computer program executed by the processing unit. The methods described herein may be implemented in a high-level procedural or object-oriented language or a scripting language or a combination thereof to communicate with the control device or processing unit or to support the operation of the control device or processing unit. Alternatively, the programs described herein may be implemented in assembly language or machine language. The language may be a compiled language or an interpreted language. The program code for implementing the methods described herein may be stored on a storage medium or in a device, such as a ROM, a disk, an optical disk, a flash drive, or any other suitable storage medium. The program code may be read by a general or special-purpose programmable computing device to configure and operate the computer when the computer reads the storage medium or device to perform the methods described herein. Computer executable instructions (computer programs) can take a variety of forms, including program modules that are executed by one or more computers or other devices. Program modules typically include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functionality of program modules can be combined or distributed in different embodiments as desired.

[0042] The above object is therefore achieved by a computer program with a program code which is stored on a machine-readable data carrier for executing the method steps according to the above method when the computer program is read and executed by a computing unit (computer).

[0043] The above-mentioned object is also achieved in particular by a device for orienting a predetermined number of real cameras to measure a real three-dimensional object in a predetermined coordinate system using the cameras and a pattern, wherein the pattern can be projected onto the surface of the real object by at least one projector and the pattern can be recorded at least partially by each real camera, wherein there is a three-dimensional virtual model of an ideal object corresponding to the real object, wherein,

[0044] A) at least one projector can be directed onto the surface of the real object so that the position of the pattern corresponds to a predetermined position of said pattern on the virtual model,

[0045] The device comprises a computing unit, which is configured to:

[0046] B) defining or providing a plurality of target markings on a surface of an ideal object corresponding to the surface of the real object to be measured in the three-dimensional virtual model,

[0047] C) automatically determining reference information for each camera, wherein the reference information includes identification information containing information of at least one target marker among a plurality of target markers captured by the corresponding camera when the corresponding camera and the projector of the pattern are arranged and oriented in a predetermined manner, and the reference information also includes position information associated with the corresponding camera and the target marker, the position information containing position information of the at least one target marker appearing in a captured image of a field of view of the corresponding camera when the corresponding camera and the at least one projector of the pattern are arranged and oriented in a predetermined manner, wherein the reference information for each camera is determined based on a three-dimensional virtual model of an ideal object and a corresponding virtual representation of the camera and the pattern, and

[0048] D) recording an image of a surface of a real object using a real camera, automatically determining instruction and / or control information, and automatically providing the instruction and / or control information at a predetermined interface, the instruction and / or control information for finding a specified target marker in the image of the corresponding real camera, and orienting the corresponding real camera using the real object based on reference information determined for the corresponding camera,

[0049] wherein the computing unit is configured to perform step D) for each camera of a predetermined number of cameras,

[0050] In one embodiment, the data of the three-dimensional virtual model of the ideal object is CAD data, and / or the pattern is a two-dimensional coded pattern.

[0051] In one embodiment, the computing unit is configured to use a real pattern projected onto the surface of a real object, using each real camera and the real object, to search for at least one target mark on the surface of the real object based on identification information in the image recorded by the corresponding camera, and then calculate instructions and / or control information for finding the at least one target mark.

[0052] In one embodiment, the computing unit is configured to determine instructions and / or control information for directing the corresponding real camera by comparing identification information with at least one target marker visible in the image of the corresponding real camera and / or by comparing the determined position information of at least one target marker of the corresponding camera with the position of the at least one target marker in the image of the corresponding real camera.

[0053] In an embodiment, the calculation unit is configured to determine a quality assessment of the deviation of the current orientation of the respective real camera from the reference information determined for the respective camera and to provide it as indication and / or control information at a predetermined interface.

[0054] In an embodiment of the device, the device comprises at least one auxiliary camera, which, as described above, can be used for the orientation of at least one projector.

[0055] In one embodiment, the interface of the device is configured to be connectable to a display device (e.g., a screen) and / or a speaker and / or a control device for multiple motors, wherein instruction and / or control information is transmitted to the display device and / or the speaker and / or the control device.

[0056] The above object is also achieved by a system, which includes the above device and a display device (e.g., a screen) and / or a speaker and / or a control device for multiple motors, wherein an interface is connected to the display device and / or the speaker and / or the control device, and the indication and / or control information provided at the interface can be transmitted to the display device, the speaker and / or the control device, wherein the display device and / or the speaker and / or the control device processes the indication and / or control information and / or outputs the indication on the display device and / or the speaker, and / or the control information is transmitted by the control device to the motor, so that the real camera is automatically oriented according to the determined reference information of the camera. For example, each camera may include a servo motor / actuator, which uses a target marker to orient the camera accordingly according to the control information transmitted to it.

[0057] Further advantages, features and possible applications of the invention will be explained below with reference to preferred embodiments and the accompanying drawings.All features described and / or shown constitute the subject matter of the invention, even independently of their grouping in the claims and references. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 An example of a stereo measurement is schematically shown in a side perspective view.

[0059] Figure 2 The system according to the invention is schematically shown in a side perspective view.

[0060] Figure 3 The front view schematically shows the Figure 2 A CAD model of the ideal object for the system.

[0061] Figure 4 Schematically shows Figure 3 Sectional view DETAILED DESCRIPTION

[0062] Figure 1 An example of a system 1 for measuring a three-dimensional object 2 using stereopsis is shown schematically. Figure 1 In the figure, for the sake of clarity, the three-dimensional object is shown as three rectangular parallelepiped stones stacked on top of each other; however, the present invention clearly relates to any three-dimensional object 2 including also such objects.

[0063] The system 1 comprises a projector 5 for projecting a (area and / or time) coded pattern 7 onto the surface of an object 2. In the system 1 according to a particularly preferred embodiment, four cameras 8.1, 8.2, 8.3 and 8.4 are provided to record the pattern 7 (referred to as a scene) projected onto the surface of the object. The projector 5 can be a conventional slide projector or a digital projector which can generate the pattern 7 to be projected on a transparent display based on a digital template.

[0064] Furthermore, a computing unit 9 connected to the projector 5 and the cameras 8.1, 8.2, 8.3 and 8.4 is provided, which is configured to measure the object 2 by stereoscopy using the pattern 7, which can be a completely area-coded pattern (e.g. a pseudo-random pattern). Such a measurement is described, for example, in document DE 10 2018 108 874 A1.

[0065] Figure 2An embodiment of a measuring unit for stereo measurement of a vehicle body part 12 is shown, which uses as an example fifty cameras 18 and fifteen projectors 15, which are arranged on a support or frame 21 surrounding the measuring unit. Both the cameras 18 and the projectors 15 are attached to the frame 21 at predetermined positions and roughly oriented. In order to orient the projectors 15 and the cameras 18 before measuring the vehicle body 12, a CAD model of the vehicle body part 12 is available in a storage unit of a computing unit 19 connected to the cameras 18 and the projectors 15. The orientation of the projectors 15 is performed before the orientation of the cameras 18 by orienting the pattern imaged by the projectors on the surface of the vehicle body part 12. The orientation of the pattern defined on the CAD model is taken as a comparison and the orientation of the projectors 15 is changed until the pattern extends on the surface of the vehicle body part 12 in a manner that matches the (virtual) position of the pattern on the CAD model.

[0066] In addition, the CAD model of the vehicle body part 12 is used to automatically, semi-automatically or interactively determine a plurality of target markers 23 as described above, so that each camera can perceive at least one target marker distributed on the camera image, preferably two or three target markers. Each target marker has a unique identifier, such as a unique number. These target markers 23 can be provided and stored in a storage unit, or input by a user, or automatically or semi-automatically determined by the method. Based on the CAD model, the defined or provided target markers 23 and the known positions of the projector 15 and the camera 18 on the frame 21, the calculation unit 19 uses a known photogrammetry algorithm to calculate the reference information of each camera, which contains information about which target markers 23 of the plurality of target markers are perceived in the image of the camera 18 in the ideal orientation as identification information. For example, the identification information includes the number of target markers 23 that the corresponding camera fully perceives under the ideal orientation. In addition, position information (e.g., pixel coordinates of the corresponding target marker) is calculated for each camera 18 and each target marker photographed by the camera 18, which position information represents the position where the corresponding target marker will appear in the image of the camera 18 under the ideal orientation. For example, the position information can be calculated in the coordinate system of the corresponding camera.If there are a plurality of target marks 23 in the camera image, they can also be connected by a line and this line can be used as the next line.

[0067] Once the reference information has been determined for each of the predetermined number of cameras (ie, all fifty cameras 18 ), it may be stored in a storage unit of the computing unit 19 .

[0068] Figure 3 Shows in more detail Figure 4Target marking of a CAD model 12' of a vehicle body part 12 is shown. The target marking is formed by an approximately oval opening 23' of the CAD model 12', which is more conspicuous in the image than the surroundings due to the dark contrast. Two obliquely extending lines indicate the area of ​​the CAD model 12' in which the opening 23' is located. Alternatively and additionally, other openings, edges, protrusions or recesses of the CAD model 12' can also form the target marking.

[0069] The orientation of the cameras 18 is carried out for each camera relative to a real object 12 in the form of a real motor vehicle body part 12, which is arranged at a specified position in a measuring unit with a frame 21. For each camera 18, an image of the body part 12 is recorded and transmitted to a computing unit 19, where it is compared with the determined reference information. For this purpose, a search is made for a defined target mark in the image recorded by the respective camera 18, for example using a cross-correlation algorithm, and it is determined whether the target mark determined based on the identification information is already visible in the image of the respective camera 18. The computing unit 19 then determines, based on the identification information, an instruction and / or control information indicating in which direction and / or by how much angle (absolute value) the camera must be rotated so that the respective camera can capture the determined target mark. Subsequently, the image recorded by the respective camera 18 is compared with the position information, and a deviation of the position of at least one target mark displayed in the camera image from the position information is determined. Therefore, the calculation unit 19 also determines instruction and / or control information which indicates in which direction and / or along which angle (absolute value) the camera must be oriented so that the position of the at least one target marking corresponds to the position information.

[0070] Then, the corresponding instruction and / or control information is transmitted to the interface, which is, for example, a display screen / monitor 20 (see Figure 2 ) and / or an interface of the control device. The interface is provided on the computing unit. Then, for example, by arranging the target mark 23 in the image of the camera 18 corresponding to the reference information, the reference information is displayed on the screen 20 connected to the interface. Then, the image of the corresponding camera 18 recorded by the camera 18 can also be superimposed and displayed on the screen 20, so that the user can recognize the difference between the target arrangement of the target mark 23 generated by the reference information and the actual arrangement of the target mark in the image of the corresponding camera 18. In addition, as Figure 2 As shown, an arrow 25 may be displayed which, for example, indicates the orientation of the camera 18 (via the direction of the arrow) and indicates the angle (absolute value) of the camera 18 (via the length of the arrow).

[0071] Before and after each orientation step, a quality rating is determined from the sum of squared distances of target markers in the real camera image and based on the reference information (see the above example), which represents how good the orientation has been achieved. For example, in the case of automatic orientation of the camera 18, the orientation can be cancelled if the quality rating is below a predefined threshold.

[0072] As an alternative to the orientation by a user (e.g. a technician), the interface can be connected to a control device which in turn is connected to the camera. In this embodiment, each camera has a servo motor which rotates the camera to achieve the desired orientation of the camera based on control information transmitted by the control device. The control device calculates the control information for the camera based on reference information for the respective camera and the image of the real object taken by the camera. This can be done, for example, using an iterative method. The control device can also calculate a quality evaluation of each camera orientation in the above-described manner and output it to a respective display device connected to the control device or transmit it back to the computing unit for further processing.

[0073] After the cameras are oriented, they can still be calibrated internally and / or externally as described above. Subsequently, the real object used for orientation and / or other real objects corresponding to the corresponding CAD model can be measured. If a new object with a different shape is to be measured, the above process may need to be repeated before measuring the new object, which includes mounting the camera, mounting and orienting the projector, and calibrating the camera using the new CAD model and other target markers.

[0074] The method described makes preparation for measuring three-dimensional objects using a large number of cameras quick and simple.

Claims

1. A method for orienting a predetermined number of real cameras (18) to measure a real three-dimensional object in a predetermined coordinate system using said cameras and a pattern, wherein: In order to carry out the measurement, the pattern is projected onto the surface of a real object by means of at least one projector (15) and the pattern is at least partially recorded by means of each of the predetermined number of real cameras, wherein there is a three-dimensional virtual model of an ideal object (12') corresponding to the real object, the method comprising the following steps: a) orienting the at least one projector onto the surface of the real object such that the position of the pattern corresponds to a predetermined position of the pattern on the virtual model; b) defining or providing a plurality of target markers (23) on the surface of the ideal object in the three-dimensional virtual model; c) automatically determining reference information for each of said cameras, wherein said reference information comprises identification information containing information of at least one target marker among a plurality of target markers captured by a corresponding camera when the corresponding camera and said projector of said pattern are arranged and oriented in a predetermined manner, and said reference information further comprises position information associated with the corresponding camera and said target marker, said position information containing position information of said at least one target marker appearing in a captured image of a field of view of the corresponding camera when the corresponding camera and said at least one projector of said pattern are arranged and oriented in a predetermined manner, wherein said reference information for each of said cameras is determined based on said three-dimensional virtual model of said ideal object and said camera and said corresponding virtual representation of said pattern; and d) recording an image of the surface of the real object using the real camera, automatically calculating instruction and / or control information, and providing the instruction and / or control information to the respective camera at a predetermined interface, the instruction and / or control information being used to find a specified target mark in the image of the respective real camera and orienting the respective real camera with the aid of the real object based on the reference information determined for the respective camera, Wherein, step d) is performed for each camera in the predetermined number of cameras.

2. The method according to claim 1, characterized in that Utilizing the real pattern projected onto the surface of the real object, using each of the real cameras and the real object, searching for the at least one target mark on the surface of the real object based on the identification information in the image of the corresponding camera, and then calculating the indication and / or control information for finding the at least one target mark.

3. The method according to any one of the preceding claims, characterized in that Instructions and / or control information for orienting the corresponding real camera are determined by comparing the identification information with the at least one target mark visible in the image of the corresponding real camera, and / or by comparing the determined position information of the at least one target mark of the corresponding camera with the position of the at least one target mark in the image of the corresponding real camera.

4. The method according to any one of the preceding claims, characterized in that A quality assessment of a deviation of a current orientation of the respective real camera from the reference information determined for the respective camera is determined and the quality assessment is provided as indication and / or control information at the predetermined interface.

5. The method according to any one of the preceding claims, characterized in that The data of the three-dimensional virtual model of the ideal object is CAD data, and / or the pattern is a two-dimensional coded pattern.

6. The method according to any one of the preceding claims, characterized in that The interface is connected to a display device (20) and / or a speaker and / or a control device for multiple motors, wherein the display device (20) is, for example, a screen, wherein the indication and / or control information is processed by the display device and / or the speaker and / or the control device, and / or is output on the display device and / or the speaker, and / or is transmitted to the motor by the control device, so that the real camera is automatically oriented according to the determined camera reference information. 7 . A computer program having a program code, which is stored on a machine-readable data carrier in order to carry out the method steps according to claim 1 when the computer program is executed on a computing unit.

8. A device for orienting a predetermined number of real cameras (18) to measure a real three-dimensional object (12) in a predetermined coordinate system using said cameras and a pattern, wherein: For the measurement, the pattern is projected onto the surface of the real object by at least one projector and is at least partially recorded by each of the real cameras, wherein there is a three-dimensional virtual model of an ideal object (12') corresponding to the real object, wherein A) the at least one projector can be directed onto the surface of the real object such that the position of the pattern corresponds to a predetermined position of the pattern on the virtual model, The device comprises a computing unit (19) configured to: B) defining or providing a plurality of target markings (23) on the surface of the ideal object in the three-dimensional virtual model, C) automatically determining reference information for each of the cameras, wherein the reference information comprises identification information, the identification information comprising information of at least one target marker among a plurality of target markers captured by the corresponding camera when the corresponding camera and the projector of the pattern are arranged and oriented in a predetermined manner, and the reference information further comprises position information associated with the corresponding camera and the target marker, the position information comprising position information of the at least one target marker appearing in a captured image of the field of view of the corresponding camera when the corresponding camera and the at least one projector of the pattern are arranged and oriented in a predetermined manner, wherein the reference information for each of the cameras is determined based on the three-dimensional virtual model of the ideal object and the corresponding virtual representation of the camera and the pattern, and D) recording an image of the surface of the real object using the real camera, automatically calculating instruction and / or control information, and automatically providing the instruction and / or control information at a predetermined interface, the instruction and / or control information being used to find a specified target marker in the image of the respective real camera and orienting the respective real camera with the aid of the real object based on the reference information determined for the respective camera, The computing unit is configured to perform step D) for each camera in the predetermined number of cameras.

9. The device according to claim 8, characterized in that The computing unit is configured to utilize the real pattern projected onto the surface of the real object, through each of the real cameras and the real object, to search for the at least one target mark on the surface of the real object according to the identification information in the image of the corresponding camera, and then calculate the indication and / or control information for finding the at least one target mark.

10. The device according to any one of claims 8 to 9, characterized in that The computing unit is configured to determine instructions and / or control information for directing the corresponding real camera by comparing the identification information with the at least one target marker visible in the image of the corresponding real camera, and / or by comparing the determined position information of the at least one target marker of the corresponding camera with the position of the at least one target marker in the image of the corresponding real camera.

11. The device according to any one of claims 8 to 10, characterized in that The calculation unit is configured to determine a quality assessment of a deviation of a current orientation of the respective real camera from the reference information determined for the respective camera and to provide the quality assessment as indication and / or control information at the predetermined interface.

12. The device according to any one of claims 8 to 11, characterized in that The data of the three-dimensional virtual model of the ideal object is CAD data, and / or the pattern is a two-dimensional coded pattern.

13. The device according to any one of claims 8 to 12, characterized in that The device also includes at least one auxiliary camera.

14. The device according to any one of claims 8 to 13, characterized in that The interface can be connected to a display device, such as a screen, and / or a loudspeaker and / or a control device for a plurality of motors, wherein the indication and / or control information is transmitted to the display device and / or the loudspeaker and / or the control device.

15. A system comprising a device according to any one of claims 8 to 13 and a display device (20), such as a screen, and / or a loudspeaker and / or a control device for a plurality of motors, wherein: The interface is connected to the display device and / or the speaker and / or the control device, and the indication and / or control information provided at the interface can be transmitted to the display device, the speaker and / or the control device, wherein the display device and / or the speaker and / or the control device processes the indication and / or control information, and / or outputs indications on the display device and / or the speaker, and / or the control information is transmitted by the control device to the motor, so that the real camera is automatically oriented according to the determined camera reference information.

Images