Method for determining at least one degree of freedom of camera, computer program, machine-readable storage medium and electronic control unit or automation device

Abstract

The invention relates to a method for determining at least one degree of freedom of a camera (1) relative to a code arrangement (2) from a camera image of the camera (1), the code arrangement (2) having a dot grid with a plurality of elementary symbols (20), the coarse positions of the elementary symbols (20) being encoded in the code arrangement (2), wherein a starting field (28) having basic symbols (20) is determined, the starting field (28) having at least three basic symbols (20), two independent main directions along the lattice of points are estimated in the camera image by means of the basic symbols (20) of the starting field (28), the basic symbols (20) of the starting field forming effective basic symbols, and the basic symbols (20) of the starting field forming effective basic symbols. Wherein in a search step, starting from at least one active basic symbol (20), a further basic symbol (20) is searched in at least one of the main directions, and wherein in the event of a successful search, the further basic symbol (20) is marked as an active basic symbol (20), the search step being carried out a plurality of times, wherein a coarse position of at least one of the basic symbols (20) in the code arrangement and / or a coarse position of a coordinate system of the point grid is determined as two degrees of freedom of the camera relative to the code arrangement (2) on the basis of the effective basic symbols (20) or a subset thereof.

Description

Technical Field

[0001] This invention relates to a method for determining at least one degree of freedom of a camera relative to a code arrangement in a camera image, as described in claim 1. The invention also relates to a computer program, a machine-readable storage medium having a computer program, and an electronic control unit or an automation device having an electronic control unit. Background Technology

[0002] In modern manufacturing and other applications, it is necessary to detect the position of a moving object relative to its surrounding environment. For this purpose, a wide range of sensor systems exist, such as displacement sensing elements and readout coded positions. All existing sensor systems offer specific advantages for their respective applications, which may be based on accuracy, simplicity, low-cost integration, robustness, or other characteristics.

[0003] In the applicant's publication DE 10 2016 216 221 A1, which constitutes the closest prior art, a method is proposed for locating and / or determining the position of an object in space and / or on a surface using a two-dimensional code arrangement. This two-dimensional code arrangement has basic symbols arranged on the surface, and the basic symbols on the surface constitute a two-dimensional periodic grid. Based on this code arrangement, the position of a camera relative to the code arrangement can be determined.

[0004] The applicant’s publication DE 10 2016 216 196 A1 discloses a sensor device in which the sensor device uses the method for positioning and / or determining position of the aforementioned publication. Summary of the Invention

[0005] The invention proposes a method for determining at least one degree of freedom of a camera relative to a code arrangement from camera images, having the features of claim 1; an electronic control unit or automated device having the features of the independent claim; a computer program; and a machine-readable storage medium. Preferred or advantageous embodiments of the invention are derived from the dependent claims, the following description, and the accompanying drawings.

[0006] The method according to the invention is used to determine at least one degree of freedom of a camera relative to a code arrangement from camera images of a camera.

[0007] This method is based on the method of publication DE 10 2016 216 221 A1 and / or the application of publication DE 10 2016 216196 A1, the disclosure of which is incorporated herein by reference.

[0008] Define the camera: A camera, in particular, includes an optical sensor unit (also called an imaging sensor element, camera chip, or image sensing element) and an imaging optical system for recording camera images. Specifically, the camera is configured as a color camera or a monochrome camera. The optical sensor unit is configured to provide individual camera images and / or sequences of camera images, such as video, in a coded sequence. The camera has an image sensing element as the optical sensor unit. The camera may include a lens, which is preferably configured as a wide-angle lens. In particular, the camera implements vanishing perspective and / or central perspective. Here, especially when viewed at an angle, parallel edges are not represented as image parallels, but rather optically converge at an imaginary point, the so-called vanishing point. In particular, cameras including lenses implement desired perspective distortion relevant to the application.

[0009] Define optical axis 5: Ideally, an imaging optical system is a rotationally symmetric optical system, where the axis of symmetry is the optical axis. The imaging optical system is characterized in that a light beam along the optical axis does not undergo deflection as it passes through the optical system.

[0010] Define the image center 7: The intersection of the optical axis 5 and the optical sensor unit 12 is called the image center 7. This intersection may, but is not required to, coincide with the geometric center of the optical sensor unit 12.

[0011] Define the camera image: The camera image of the camera is specifically constructed as a matrix, which has image points, such as 8-bit grayscale points or color points, at matrix points.

[0012] Define the code arrangement: The code arrangement can be imaged by a camera, where the absolute position of a machine or machine part can be measured in one to six degrees of freedom, for example, based on the camera image. In particular, the code arrangement is configured for detection by a camera via non-contact reading, where multidimensional real-world position determination can be performed by non-contact detection by the camera. This two-dimensional code arrangement is preferably used in production and / or inspection equipment where workpieces and / or inspection devices and / or working devices must be positioned.

[0013] Define basic symbols: The two-dimensional code arrangement includes basic symbols arranged on a surface, forming a two-dimensional periodic grid, i.e., a dot grid. The basic symbols are preferably geometric shapes, such as circles, squares, triangles, or lines. Particularly preferred are the basic symbols constructed as circles. Basic symbols are also referred to as dots. The basic symbols preferably represent numbers in a number system. In particular, the two-dimensional code arrangement includes at least two different basic symbols. In one possible design of the invention, the two-dimensional periodic grid includes empty spaces as block symbols in grid positions and / or mesh positions not occupied by basic symbols.

[0014] Basic symbols are arranged on a surface, which can be curved or non-curved. For example, the surface is the floor of a production and / or inspection equipment. The basic symbols are arranged in a two-dimensional grid in the surface, wherein the centroids of the basic symbols preferably constitute grid points in a point grid.

[0015] Define the grid of points for code arrangement: Grid points are also referred to below as grid locations and / or mesh locations. In particular, two-dimensional periodic point grids constitute a two-dimensional mesh.

[0016] Define a block (Parzelle): The surface is preferably divided into similar, regularly arranged blocks, wherein the blocks have, for example, a basic shape of a square, a basic shape of a rectangle, a basic shape of a triangle, or a basic shape of a hexagon. Blocks of the same size and / or the same shape are particularly considered similar blocks. Preferably, each block includes n basic symbols. A block specifically includes an integer number of basic symbols, and particularly an even number. Preferably, a block includes more than ten basic symbols, particularly more than twenty, and particularly more than forty. Furthermore, the number of basic symbols in a block is preferably less than one hundred. Blocks composed of basic symbols can be displayed optically in the code arrangement, such as by boundaries, or can be displayed non-optically, and constitute only conceptual and / or logical units.

[0017] Define the block area: A block has at least one first and one second block region. Block region X includes the first block region, and block region Y includes the second block region. In particular, each block region occupies one continuous face or multiple distributed non-contiguous subfaces within the block.

[0018] Each block region includes multiple basic symbols. In particular, the X block region and the Y block region include the same multiple basic symbols. In particular, at least two block regions are arranged in the block such that the at least two block regions have p-fold rotational symmetry about the center of the block as a rotation point, where p-fold rotational symmetry is, for example, double, triple, or quadruple rotational symmetry.

[0019] Define block symbols: Each block has at least one block symbol, which represents a fixed reference point within each block. The block symbols enable the reading and / or decoding of basic symbols in a prescribed order. Specifically, the block symbols are arranged regularly and / or periodically within a two-dimensional periodic grid of basic symbols. Block symbols may be located on or beside grid points. In particular, the block symbols are arranged at the same locations within the block, such as at the center of the block. For example, block symbols are graphic elements different from the basic symbols, such as triangles, hexagons, or lines. Alternatively and / or supplementarily, block symbols are represented by omitting one or more basic symbols from the block. In one possible design, the block symbols constitute the symmetry points of the block's p-fold rotational symmetry. Specifically, the reading direction and / or decoding order are specified, i.e., the order in which basic symbols within the X block region and / or the Y block region must be read. The reading direction and / or decoding order preferably correspond to a preset of which basic symbols can be read and / or decoded sequentially.

[0020] Define the encoding: In the X block region, the X coordinate values ​​are encoded using basic symbols, and in the Y block region, the Y coordinate values ​​are encoded using basic symbols. Specifically, the X and Y coordinate values ​​are the coordinates of the basic symbols within the blocks, which indicate the coordinates in a Cartesian coordinate system of the plane defined by the basic symbols. Alternatively / supplementarily, the X and Y coordinate values ​​can also be used to indicate the position within that plane in other coordinate systems, such as oblique coordinates, cylindrical coordinates, or spherical coordinates.

[0021] Preferred implementation of defining a point grid: In a particularly preferred design of the present invention, the two-dimensional periodic grid is a rectangular grid, where the blocks are also rectangular. In particular, the rectangular grid and the rectangular blocks are square grids and / or square blocks. Preferably, the distances of the basic symbols along the long axis and the short axis of the rectangular grid are of the same size. For example, for a square grid with square blocks, the number of basic symbols in the X direction and the Y direction of the planar two-dimensional periodic grid is the same. In particular, the X block area and the Y block area are each composed of two spatially separated rectangular sub-faces within a block, where the sub-faces have a longitudinal extension. The longitudinal extension of the sub-face of the X block area is preferably perpendicular to the longitudinal extension of the sub-face of the Y block area. Preferably, the face occupied by the X block area can be transformed into the face occupied by the Y block area by a 90° rotation.

[0022] In a particularly preferred design of the present invention, the periodic grid has a grid length and a grid width. The grid length extends along the X direction of the Cartesian coordinate system, and the grid width extends along the Y direction. The coordinate system thus constituted assigns a uniquely determined position vector (X, Y) to each point of the coding surface.

[0023] A continuous sequence of g basic symbols in particular constitutes a total sequence. This total sequence is entered into the X block area, in particular into a plurality of blocks adjacent in the X direction. In particular, according to the order of the basic symbols in the total sequence, preferably along the rising X direction of the block, the basic symbols are entered into the X block area in each block in a previously specified reading and / or decoding order. The quantity g is large enough such that the X block areas of all blocks adjacent in the X direction can be completely filled. In particular, g is greater than fifty, in particular greater than one thousand, and particularly greater than one million. The content of the X block areas of the blocks adjacent in the Y direction is the same. Particularly preferably, a segment of t consecutive basic symbols in the total sequence constitutes a subsequence. In particular, each segment of t consecutive basic symbols of the total sequence constitutes a subsequence, where the consecutive basic symbols follow in the decoding and / or reading order. In particular, the total sequence is constructed such that each subsequence of t consecutive basic symbols is included only once in the total sequence in the forward reading manner, and each subsequence read backward is not included in the total sequence in the forward reading manner. Preferably, the subsequence includes at least five consecutive basic symbols, in particular at least twenty consecutive basic symbols, and particularly at least thirty basic symbols. In addition, the subsequence preferably includes less than fifty basic symbols, and particularly less than thirty basic symbols. In particular, t < g applies.

[0024] In a particularly preferred embodiment of the invention, basic symbols are constructed to encode numbers with a base b. The base b is preferably the base of a positional numeral system. Preferably, the base b = 2, i.e., the binary base, where the binary digits include 0 and 1. Alternatively, the base b = 10, constituting the base of a decimal system, where the decimal numbers include the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. Alternatively, the base b = 16, i.e., the hexadecimal base, where the hexadecimal numbers include the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F. In particular, the base b can be arbitrarily chosen, where the base numbers include elements 0, 1, ..., b-1. Alternatively and / or additionally, the base b is the base of an additive number system, such as the Roman numeral system.

[0025] In a particularly preferred embodiment of the invention, the base number b = 2 is chosen. Specifically, the chosen base number b = 2 constitutes a binary system. This binary system and / or the number system having a base number b = 2 comprises two digits, specifically the digits 0 and 1. Preferably, two distinct basic symbols encode the digits 0 and 1 of the binary system. The two distinct basic symbols are formed by two circles: a first circle with radius R1 and a second circle with radius R2. Specifically, radius R1 is chosen to be smaller than radius R2. Specifically, the ratio of radius R2 to radius R1 is greater than the square root of 2. For example, radius R2 is chosen such that radius R2 is less than half the grid spacing of the code grid, such that two adjacent circles with R2 in a periodic grid are not tangent. This design is based on the consideration of achieving particularly high information density on the one hand, and reliable readability using standard image processing methods, such as segmentation by means of blob analysis on the other.

[0026] In one possible design of the invention, the overall sequence is constructed such that the sum of the digits of the subsequences is less than half of the maximum possible sum of the digits of the subsequences. In particular, the sum of the digits of each arbitrary subsequence selected from the overall sequence is less than half of the maximum possible sum of the digits of the subsequences. For example, for a number system with a base b, the sum of the digits of each subsequence with length t is less than t·(b–1) / 2.

[0027] In a particularly preferred embodiment of the invention, the two-dimensional code arrangement includes a negated overall sequence. This negated overall sequence preferably represents the negation of the overall sequence. This negated overall sequence is preferably entered into the Y-block regions of multiple adjacent blocks in the Y-direction. Specifically, based on the order of the basic symbols in the negated overall sequence, the basic symbols are preferably entered into the Y-block regions within each block in the ascending Y-direction according to a previously predetermined reading and / or decoding order. The number g is large enough that the Y-block regions of all adjacent blocks in the Y-direction can be completely filled. In particular, the contents of the Y-block regions of adjacent blocks in the X-direction are identical.

[0028] In one possible design of the present invention, the basic symbol at the k-th position of the inverted total sequence is the encoded digit m(k), where the digit m(k) and the digit n(k) encoded by the basic symbol at the k-th position in the total sequence satisfy the relationship: m(k) = b – n(k) – 1. For example, for a binary choice with a base b = 2, the inverted total sequence and the total sequence constitute the relationship: m(k) = 1 – n(k).

[0029] Define control points (Aufpunkt): This segment particularly preferably includes control points. These control points are, for example, the intersection of the camera's optical axis and the plane containing the code arrangement. The method is preferably configured to determine the camera's position relative to the working area based on images recorded by the sensor unit, such as the X (rx) and Y (ry) coordinates of the control points in a coordinate system defined by the working area and / or the basic symbols and / or the code arrangement.

[0030] First, within the scope of this method, only a coarse position is determined as a two-degree-of-freedom coded X-coordinate and coded Y-coordinate. This coarse position yields a load-bearing and usable location for coarse camera positioning.

[0031] By determining the X and Y coordinates as a rough position within the code arrangement, the relative position of the camera with respect to the code arrangement can be inferred. If, for example, the rough position is determined near the control point, the camera's position can be inferred from knowledge of the orientation of its optical axis. A near-precise position of the camera is obtained when the control point is at the same location as the code arrangement and the angle of the optical axis relative to the code arrangement is known. The simplest case is when the optical axis is perpendicular to the orientation of the code arrangement.

[0032] Define degrees of freedom: In a particularly preferred embodiment of the invention, the method is configured to determine the position and / or pose of a working module relative to a working area in up to six degrees of freedom. The degrees of freedom are, in particular: rx: Position coordinates in the coordinate system of the code arrangement (as a coarse position and / or fine position).

[0033] ry: Position coordinates in the coordinate system of the code arrangement (as a coarse position and / or fine position).

[0034] In particular, rx and ry involve control points, especially the penetration points through which the optical axis passes through the code arrangement.

[0035] rz: The distance of the reference point along the optical axis of the camera.

[0036] phiz: The Z-axis rotation angle of the camera around the optical axis.

[0037] phix: Especially regarding the first principal direction, the first pitch angle of the optical axis relative to the code arrangement.

[0038] phiy: Especially regarding the second principal direction, the second pitch angle of the optical axis relative to the code arrangement.

[0039] Define X and Y coordinates (rx, ry): For example, the six degrees of freedom of the camera's position and / or pose relative to the work area include coordinates X and Y with respect to a Cartesian coordinate system defined by basic symbols, specifically the coordinate system of the code arrangement in the code arrangement plane (X... C ,Y C The coordinates X and Y in ().

[0040] Define the distance (rz) between the control point and the optical center of the camera: In addition, the six degrees of freedom include the distance Z or r of the camera along the optical axis control point. z The distance, for example, represents the distance between a control point and, for example, the optical center of a camera lens.

[0041] Define the rotation angle: Furthermore, three independent rotation angles of the camera are determined in the camera coordinate system. Two of the rotation angles are specifically defined as the intermediate angles (phix, phiy) between the optical axis and the code arrangement. The third rotation angle (phiz) represents the rotation of the camera around the optical axis.

[0042] Define the coordinate system for the code arrangement: Coordinate system for code arrangement (X) C , Y C It lies in the plane of code arrangement.

[0043] Define the camera's image sensing elements and / or the coordinate system in the camera image: The coordinate system of the image sensing element (X I , YI It is located in the plane of the image sensing element or in the camera image.

[0044] Define the camera coordinate system: The camera's coordinate system (X, Y, Z) has its origin at the optical center of the camera's lens, where the negative Z direction coincides with the camera's optical axis.

[0045] Summary: The coordinate system is illustrated here in terms of origin and orientation, but an equivalent coordinate system may also be used.

[0046] Define method: In some variations of the method, the X and Y coordinate values ​​are characterized by being decoded and / or determined based on camera images recorded by the camera. In a possible improvement to the method, the orientation of a segment about the three coordinate axes is determined based on a code arrangement segment recorded in the image. Specifically, the orientation of the segment recorded in the image about the X and Y axes of a Cartesian coordinate system defined by fundamental symbols is determined. Alternatively and / or additionally, the coordinates of the control point's position in the code arrangement are determined by encoding the code arrangement segment contained in the image. In a particularly preferred embodiment of the invention, the pose of the working module relative to the working area is determined in up to six degrees of freedom. Specifically, the pose of the camera in the working area is determined in three Cartesian coordinates X, Y, and Z, where X and Y are coordinates in a coordinate system defined by fundamental symbols, and Z is the distance of the working module from the working area. Furthermore, in this method, for example, the three Euler angles of the camera are determined.

[0047] Alternatively and / or supplementarily, two-dimensional code arrangements are used in fields outside of automation technologies, such as for monitoring motion processes in nature and the environment, biology and medicine, architecture, entertainment electronics and / or sensing technologies.

[0048] The general advantages of this invention, according to the variants, are: The core objective of this invention is to provide an efficient, reliable, and inexpensive method for locating and identifying one or more objects in a workspace in up to six degrees of freedom (6D locating = simultaneous position detection in six degrees of freedom of motion).

[0049] This method processes image data provided by an imaging measurement system, such as a camera. For example, a smartphone can also be used for image recording and for determining location using the method according to the invention, wherein the method is implemented as an algorithm in an application carried out by an embedded computer in the smartphone or in the cloud.

[0050] To determine the position of an object in up to six degrees of freedom, the camera detects the arrangement of two-dimensional codes associated with the object to be measured.

[0051] The method according to the invention provides the following characteristics according to a variant: - Complete 6D positional information for the Gundam: translation (X, Y, Z) and rotation (φ_X, φ_Y, φ_Z) - Absolute position information (non-incremental), i.e., eliminating the reference travel as in the case of incremental sensors. -Very large measurement range: - "Almost infinite" in X and Y, limited only by the code arrangement. Example: Explicitly coded faces using the code and 2.5mm dot grid described in [B]: With a block size of 5×5 dots: 2m×2m With a block size of 7×7 dots: 8km×8km With a block size of 9×9 dots: 550,000km × 550,000km On Z: A single dimension with boundaries. The grid spacing can be adapted to the application through camera resolution, lens, and code arrangement.

[0052] In φ_X and φ_Y: Current + / - 60° If the object has code on all four sides, then it is infinite (e.g., ...). Figure 5 (as in a and b) In φ_Z, it is infinite (0°-360°). - High measurement rates, such as 100Hz–10,000Hz - Short latency (the time from image recording to output of 6D position measurements), for example, 10ms–100µs.

[0053] -High measurement accuracy across all six degrees of freedom: With a code grid spacing of 2.5 mm, the current repeatability is 1 µm and 0.01° (in the case of 3σ (sigma)). - Maximum read security even in the event of localized interference or fluctuations in image brightness. - Not only the sensing system but also the code arrangement can be implemented in a cost-effective manner. - Simultaneously with the measurement, additional data can be read. For example, the ID code used to identify the object can be read, and the ID code is integrated into the code arrangement.

[0054] - By changing the lens, image sensor resolution, grid spacing code arrangement, readout field size, computing performance, etc., it is possible to expand in terms of accuracy, measurement range in Z, redundancy and measurement rate.

[0055] - Simply integrate it into existing systems, such as as a smartphone app.

[0056] This allows for the simple implementation of many automation technologies that have been unable to be implemented or implemented economically to date, such as: - Precisely locate axes or rotors in axis systems or robotic systems to enable real-time pose adjustment in up to six dimensions.

[0057] -Building absolute positioning systems and robots that can operate without reference distances. -Simplify the positioning system because multiple one-dimensional measurement sensors can be replaced by a single multi-dimensional measurement sensor.

[0058] - Visual servoing, for example, to enable a robot gripper to track a moving target.

[0059] - Monitor and track multiple objects in space through cyclic position measurement - Record and analyze the movement process - Vibration analysis of the machine in 6 dimensions - Determine the relative pose of two objects. For example, a camera in a handheld device detects two objects in an image, each equipped with a point code. The proposed method allows for the positioning of two objects relative to a common camera coordinate system. The relative pose of the two objects can be determined through vector difference formation. Here, the camera position is computationally eliminated, meaning the method is largely independent of the camera position.

[0060] - Determine the relative poses of more than two objects, each equipped with a camera and / or a code arrangement.

[0061] - For example, a positioning network can be constructed in a manufacturing workshop to position multiple stationary or moving objects relative to each other or relative to the workshop coordinate system. For instance, multiple autonomous vehicles, each equipped with a camera, can perform positioning by detecting a code arrangement mounted on the workshop ceiling.

[0062] The code arrangement has a dot grid, which is particularly planar, having a plurality of basic symbols, wherein the center of the basic symbols is preferably located at a grid point, and wherein the approximate position of the basic symbols is encoded in the code arrangement.

[0063] Within the scope of this method, a starting field is determined. This starting field has at least three fundamental symbols, preferably adjacent to each other, arranged at an angle to each other. Connecting lines between the fundamental symbols of the starting field are defined in the camera image along two independent principal directions of the point grid. In the case where the starting field has exactly three fundamental symbols, these symbols are arranged, for example, at an angle to each other. The fundamental symbols define the coordinate system of the planar point grid, where one of the at least three fundamental symbols constitutes the origin of the coordinate system. In a preferred embodiment of the invention, the starting field has nine fundamental symbols, arranged in a square and / or rectangular manner. In particular, the starting field has a side length with three fundamental symbols. The fundamental symbols of the starting field are valid fundamental symbols. Valid fundamental symbols are understood, in particular, to be selectively defined as valid within the scope of this method, or to be classified as valid fundamental symbols based on image features and / or set as valid fundamental symbols by identifying fundamental symbols and successfully processing them with usable processing results. Image structures not classified as valid fundamental symbols are interpreted as image interference and excluded from further processing.

[0064] In this method, during the search step, starting from at least one valid basic symbol, other, especially valid, basic symbols are searched along the main direction.

[0065] In the event of a successful search, especially when other basic symbols are found, mark the other basic symbols as valid basic symbols.

[0066] The search step is performed multiple times. If new elementary symbols are found as valid elementary symbols during the search step, the search step can optionally begin with either the elementary symbols of the initial field or the newly found valid elementary symbols. In this way, starting from the initial field with valid elementary symbols, other valid elementary symbols are added to the point grid step by step. The search step is repeated until a sufficient number of valid elementary symbols have been found, especially to the point where decoding is possible.

[0067] Based on the valid basic symbols or a subset thereof, in particular by decoding the valid basic symbols, the approximate position of at least one basic symbol in the code arrangement is determined as two degrees of freedom of the camera relative to the code arrangement.

[0068] In particular, valid basic symbols or subsets thereof are decoded, wherein at least one degree of freedom of the camera relative to the code arrangement is determined from the decoded coarse position and optionally supplemented from the coarse orientation of the coordinate system of the point grid in the camera image.

[0069] Here, the present invention considers finding the basic symbols in the code arrangement and / or in the camera image, for example, by using an area function of image processing. However, finding the basic symbols by means of blob analysis or pattern-based search (pattern matching) requires frequent access to the pixels of the camera image, which results in costly image processing.

[0070] By starting from the search field and searching only along the principal directions, prior knowledge of the point grid's structure allows the search to proceed along the principal directions, not face-oriented, but line-oriented. The search field defines the two principal directions of the point grid in the camera image and thus defines the point grid's coordinate system. In practice, it is sufficient to start from a valid fundamental symbol and search along a line in the principal directions for another fundamental symbol. Clearly, simplifying the face search to a line search significantly reduces the number of pixel accesses and / or the overhead of image processing. Therefore, the method according to the invention enables its implementation very efficiently.

[0071] In a preferred embodiment of the invention, the corresponding midpoint position is determined relative to the basic symbol, particularly the centroid position of the basic symbol in the camera image. This allows for the determination of multiple intersection points in the point grid based on the valid basic symbols.

[0072] The procedure stipulates that the search steps should begin from the midpoint of the corresponding valid basic symbol, particularly along one of the main directions, in a pre-defined search direction. Finding the basic symbol initially only allows for a rough determination of its position within the point grid and / or camera image, while the precise position is determined by detecting the midpoint, thus specifying the point grid. Advantageously, especially the line-oriented search steps, are performed from the center of the basic symbol, as this prevents the unintentional omission of adjacent basic symbols.

[0073] In a preferred embodiment of the invention, two connection vectors are determined along a point grid in two independent principal directions in the camera image based on the initial field and / or other valid elementary symbols. By determining the connection vectors, the positions of the nearest and / or adjacent elementary symbols being searched can be estimated by extrapolation within a range of linear combinations of known valid elementary symbols.

[0074] Particularly preferred is determining the midpoint position of the basic symbol by shifting the initial point of the basic symbol along the main direction to the midpoint position through multiple intermediate steps, until the midpoint position is found to be centrally located within the basic symbol along the main direction. By shifting the initial position to the midpoint position, ensuring that the initial position is always centrally located along the main direction, it is only necessary to search the boundaries of the basic symbol along the main direction for each shift step, and then shift the initial position to the midpoint between the boundaries.

[0075] In the example of the circular basic symbol, starting from an initial position within the basic symbol, the boundaries of the basic symbol are searched by performing a linear search along the first principal direction of the camera image, followed by a search along the negative first principal direction. Averaging the boundaries results in a more accurate position estimate. This step is then repeated meaningfully for the second principal direction, and subsequently, again for the first principal direction.

[0076] In this way, the calculation is performed efficiently along the main direction by evaluating the shift from the initial position to the midpoint position through a series of lines.

[0077] In subsequent steps, the area of ​​the basic symbols in the camera image is determined, where this area constitutes the encoded data for the code arrangement. In particular, this area is used to classify the basic symbols.

[0078] This data, for example, could be 0 or 1 in a binary system, where the basic symbol is, for example, the area of ​​a circle of different sizes. Given that the corresponding midpoint position is known and the extension of the basic symbol in the principal direction due to the shift step is also known, the area of ​​the basic symbol can be inferred from these values ​​in a simple way. Therefore, it is computationally efficient to first determine the midpoint position in the point grid and then determine the area of ​​the basic symbol.

[0079] The area of ​​basic symbols in a camera image is strongly affected by perspective distortion. For example, the area decreases as the distance from the camera to the basic symbol increases and as the camera's tilt angle increases (elliptic distortion). These effects can lead to misclassification, for example, when similar basic symbols with significantly different distances from the camera appear in the camera image. Therefore, it is preferable to use the normalized area of ​​the basic symbols for classification.

[0080] To calculate the normalized area, a reference area is determined, which suffers approximately the same perspective distortion as the basic symbol. A parallelogram is used as the reference area, which is formed by two vectors extending from the grid position of the basic symbol toward adjacent grid positions along the principal axis. The reference area is derived as the value of the cross product of the two vectors.

[0081] The normalized area of ​​the basic symbol is in particular the quotient of the area calculated as the basic symbol and the reference area.

[0082] In a preferred embodiment of the invention, a readout field is defined in the camera image and / or in a planar point grid, and a corresponding two-dimensional basic symbol matrix, particularly a dot matrix, is constructed. Each point in the readout field is assigned an entry in the basic symbol matrix, and the position, area, and / or data of the basic symbols are entered into the basic symbol matrix. From the data in the basic symbol matrix, the approximate position of the point grid's coordinate system and, optionally, its orientation, can be decoded, and / or the approximate position of at least one basic symbol in the code arrangement can be determined as two degrees of freedom.

[0083] In a preferred embodiment of the invention, at least one starting position is defined in the preparatory step for creating the starting field. The starting position is arbitrarily defined. Preferably, the starting position is arranged near or adjacent to a control point. If the starting position is randomly within a basic symbol, this starting position is considered the initial position acting on the starting basic symbol. If the starting position is outside a basic symbol, adjacent basic symbols are searched along a search ray starting from the starting position as the starting basic symbol. Therefore, even in this preparatory step, computationally intensive face-oriented image processing is not performed; instead, basic symbols are searched only along the search ray starting from the starting position. The boundaries of the basic symbols can be detected, for example, by the contrast variation along the search direction in the camera image. Here, it is possible to use at least or exactly 8 or 16 search rays in this method, which start at the starting position and are arranged in a 360° distribution with regular angular steps. A minimum radius can be defined for the search rays, from which other basic symbols are searched. Optionally, a maximum search radius can be specified, wherein if the maximum search radius is reached and no adjacent basic symbols are found, the starting position is discarded and an alternative starting position is selected.

[0084] In subsequent steps, starting from the position of the initial basic symbol, adjacent basic symbols are searched along other search rays as first auxiliary initial basic symbols. The search for the first auxiliary initial basic symbols can be performed using the same search ray distribution as previously described.

[0085] In subsequent steps, a second auxiliary starting basic symbol is searched for from the initial basic symbol and the first auxiliary starting basic symbol in a direction angular to the connection between the initial basic symbol and the first auxiliary starting basic symbol (and / or along the first principal direction). However, fewer search rays can be used here, particularly those distributed over a smaller angular range, to accelerate the search. Here, the prior knowledge that the first three basic symbols should be arranged at an angle to each other is fully utilized, allowing for prior knowledge of the direction in which the second auxiliary starting basic symbol is searched. For example, only one, two, or three to six search rays are still used, distributed around the principal search direction oriented perpendicular to the connection between the initial basic symbol and the first auxiliary starting basic symbol.

[0086] For completeness, starting from the initial basic symbol, the search is conducted by linearly combining the connections between the initial basic symbol and the first auxiliary initial basic symbol, as well as the connections between the initial basic symbol and the second auxiliary initial basic symbol, to find all directly adjacent basic symbols in the principal axis or diagonal direction of the point grid, especially those directly adjacent to the initial basic symbol. In particular, the connection vector and / or principal direction can be derived from the first three basic symbols found. These expand the two-dimensional coordinate system of the point grid. To improve the method, the midpoint position can be determined to define the connection vector as precisely as possible. Thus, an initial field with 3×3 valid basic symbols exists.

[0087] In the search step, neighboring elementary symbols are searched starting from the initial field and / or other valid elementary symbols identified in earlier search steps. The method preferably searches for elementary symbols at grid points that have at least two valid neighbors in the main direction or diagonal direction. All neighbors of the grid point provide independent location estimates through extrapolation, and these location estimates are averaged to result in a more accurate location estimate of the grid point.

[0088] The condition of searching only grid points and / or using at least two valid basic symbols as neighbors further facilitates planar extension of the detected region and avoids linear extension in the form of spears, dendrites, or spikes. Planar extension is more robust and tolerant of disturbances in the image compared to linear extension.

[0089] The basic symbols define a first principal direction along the point grid and a second principal direction independent of it. Depending on the viewing angle, the two independent principal directions are imaged perpendicularly or obliquely to each other in the camera image.

[0090] A fundamental symbol matrix, particularly two-dimensional, is determined from the camera image. An entry in the fundamental symbol matrix is ​​assigned to each grid point of the code arrangement, and the midpoint position of the fundamental symbol in the camera image is also recorded in the fundamental symbol matrix. Therefore, the entries in the fundamental symbol matrix refer to the positions of the fundamental symbols in the assigned grid points, that is, particularly in the coordinate system of the image sensing element and / or the coordinate system of the camera image.

[0091] In camera images, point grids, especially those with perspective distortion, cause the rows and columns to be arranged at an angle to each other, even though the rows and columns of the point grid are constructed as straight lines.

[0092] In addition, optical distortion may still occur, which can optionally be compensated for by correction.

[0093] Preferably, a first line function is determined from the data of the basic symbol matrix using a first function variable to define the first line in the coordinate system of the camera image. When the first line is imaginarily transferred to the coordinate system of the code arrangement, the first line remains parallel to the first principal direction of the point grid, independent of the first function variable. By changing the first function variable, the first line is shifted parallel to the second principal direction in the coordinate system of the code arrangement. Therefore, by changing the first function variable, for example, the line can be placed on a row of the point grid, that is, not only in the coordinate system of the code arrangement but also in the coordinate system of the camera image.

[0094] In particular, this can be implemented as follows: Initial situation: a) The midpoints of basic symbols form a two-dimensional point grid in the code plane, the two-dimensional point grid having regularly arranged rows and columns of straight lines. b) The camera images the code plane onto the image sensor with perspective distortion. In the camera image, rows typically appear as fan-shaped lines that share a common vanishing point. The same applies to columns.

[0095] c) Due to lens distortion, rows and columns appear as curved lines in the camera image.

[0096] Method steps in camera coordinate system: a) By correcting, the distortion of the lens is mathematically eliminated, and curved lines are transformed into straight lines.

[0097] (b) Straight lines are fitted to rows using linear interpolation and numbered sequentially using integer indices as function arguments. These straight lines constitute the first bundle of straight lines. Columns are treated in terms of meaning, and these columns constitute the second bundle of straight lines.

[0098] c) Each line is described by its line angle and intercept value. The line angles of the bundle form a numerical sequence, which is approximated by quadratic interpolation. Integer line indices are used as the function's independent variables. The polynomial has three interpolation parameters. The intercept value is treated meaningfully, from which three other interpolation parameters are derived. Therefore, each bundle of lines is fully and compactly described by six interpolation parameters.

[0099] d) By inserting rational function values ​​into the interpolation function, it is also possible to calculate a straight line between two rows or columns.

[0100] Alternatively or supplementarily, the second line function is determined in the coordinate system of the camera image based on the fundamental symbol matrix, using the second function's independent variable. When the second line is transferred to the coordinate system of the code arrangement, it remains parallel to the second principal direction of the point grid, independent of the second function's independent variable. By changing the second function's independent variable, the second line is shifted parallel to the first principal direction in the coordinate system of the code arrangement. Therefore, by changing the second function's independent variable, the line can, for example, be placed on a row of the point grid, that is, not only in the coordinate system of the code arrangement but also in the coordinate system of the camera image.

[0101] It should be noted that the terms row and column are used only for naming purposes and do not imply a specific orientation of the point grid.

[0102] At least one other degree of freedom of the camera is determined based on at least one linear function. Optionally, at least one other degree of freedom of the camera may be determined based on two linear functions.

[0103] A further consideration here is that the amount of data should be reduced along the path from the camera image to at least one degree of freedom, so that at least one degree of freedom can be computed more quickly and / or more efficiently. For example, a camera image with an image size of 200×200 pixels still has 40,000 values, while the basic symbol matrix has been simplified to position and correspondingly, in the case of a camera image segment with a side length of, for example, a point grid with 15 basic symbols, there are still only 225 values ​​for position. By deriving at least one linear function, the amount of data is reduced to the parameters of the linear function.

[0104] In this case, it has been shown that for each linear function, the six parameters, optionally plus one data point, are sufficient to provide enough information about the basic symbol matrix or the main information content of the camera image on the path leading to at least one degree of freedom, thus reducing the amount of data from 225 entries to 12 or 14 entries in this example. This significantly reduces the computation of at least one degree of freedom in terms of cost.

[0105] Another advantage of this implementation is that the rows and columns of the basic symbol matrix in the code arrangement are arranged parallel to each other and regularly spaced apart, allowing for a type of averaging of the basic symbol matrix by deriving a linear function in the camera image, where the linear function describes the average information of the basic symbol matrix. Information compression is performed by using the linear function while simultaneously improving the information content.

[0106] Thus, the method according to the invention allows for the computationally efficient implementation of a method for determining at least one degree of freedom of the camera relative to the code arrangement from camera images. In terms of application technology, this determination can be performed, for example, on a microcontroller capable of determining at least one degree of freedom at least 100 times per second. In this way, it is possible, for example, to utilize this method to perform real-time applications in manufacturing.

[0107] In a preferred embodiment, the first linear function is determined based on at least two rows, preferably more than two rows, and especially on all rows of the basic symbol matrix. Alternatively or supplementarily, the second linear function is determined based on at least two columns, preferably more than two columns, and especially on all columns of the basic symbol matrix. This embodiment emphasizes that the linear function carries averaging and / or compression information about multiple rows or columns.

[0108] In a preferred embodiment of the invention, a first fitted straight line is formed for each row along a first main direction. The first straight line function is constructed based on a plurality of first fitted straight lines. By constructing the first fitted straight lines from rows, the first fitted straight lines can be adapted to the direction of the rows, such that the first fitted straight lines constitute the average and / or compressed information of the base rows. The first straight line function is constructed based on a plurality of first fitted straight lines, wherein here a double averaging or compression is performed, such that the first straight line function is constructed through a double averaging of the original information.

[0109] Alternatively or supplementarily, a second fitted line is constructed for each column along the second principal direction. The second line function is constructed based on multiple second fitted lines. By constructing the second fitted lines from the columns, the second fitted lines can be adapted to the direction of the columns, such that the second fitted lines constitute the average and / or compressed information of the base columns. The second line function is constructed based on multiple second fitted lines, wherein here it is performed by double averaging or compression, such that the second line function is constructed by double averaging of the original information.

[0110] In a preferred implementation, the first function's independent variable is constructed as a first integer value for a row, and / or the second function's independent variable is constructed as a second integer value for a column. Therefore, for the first integer value in the case of the first linear function, the first line corresponds to the first fitted line. Similarly, when the second integer value is the second integer value in the case of the second linear function's independent variable, the second line corresponds to one of the columns of the basic symbol matrix. However, the first and / or second lines are not exactly the first or second fitted lines, because the linear functions have undergone quadratic averaging / compression, making them corrected first or second fitted lines.

[0111] In a preferred embodiment, the fitted line is described by a straight angle as an intersection angle and an intersection point with a coordinate system or at least one axis of the coordinate system of the image sensing element and / or camera image. For the fitted line, the straight angle and the intersection point with at least one or exactly one coordinate axis of the coordinate system are sufficient to definitively determine the fitted line in the coordinate system.

[0112] With this implementation, the position of the fitted line in the point grid of its respective row or column is simplified to two values.

[0113] In a preferred embodiment, the linear function is constructed by combining a line angle function relating to the line angle of the linear function and an axis intersection function relating to the axis intersection points of the linear function. Therefore, the linear function is also determined by a line angle and at least one axis intersection point.

[0114] Preferably, the line-angle function is constructed as a quadratic polynomial and / or the axial intersection function is constructed as a quadratic polynomial, wherein the polynomial has the corresponding line-angle function as its independent variable. This allows the line-angle and / or axial intersection points to be determined based on the independent function variable. By choosing a quadratic polynomial, approximation can be performed particularly easily, further improving computational efficiency.

[0115] In a preferred embodiment, the intersection points of the coordinate axes of the coordinate system are selected based on the straight-line angle between the linear function and the coordinate system, where each intersection point results in a smaller intermediate angle perpendicular to the corresponding coordinate axis. Furthermore, data encoding the selected coordinate axes is assigned to the linear function. The consideration here is that only the intersection point of a single coordinate axis is needed to describe the line, and intersection points with two coordinate axes are not required. To achieve the greatest possible efficiency, the intersection point whose assigned intersection angle is more perpendicular to the intersecting coordinate axis is selected.

[0116] Preferably, a reference point is placed within the camera image. This reference point can be arbitrarily positioned. In a preferred design as described later, the reference point is configured as the intersection of the optical axis with the image sensing element and / or with the camera image. Therefore, the reference point is predetermined by the camera in the camera's coordinate system and / or within the camera image.

[0117] The first axis intersection function is preferably formed by a first straight line in the coordinate system of the camera image. The first straight line is oriented parallel to a first principal direction of the point grid in the coordinate system of the code arrangement. The axis intersection function has a first function argument, wherein the first straight line is shifted parallel to a second principal direction in the coordinate system of the code arrangement by changing the first function argument. The first axis intersection function defines a first axis intersection point along a first axis of the coordinate system of the camera image, where the first axis extends through a reference point, according to the first function argument. Because the first straight line is shifted in the coordinate system of the code arrangement by changing the first function argument, thereby shifting the first straight line in the coordinate system of the camera and / or camera image (in a perspective distortion manner), the axis intersection point migrates along the first axis.

[0118] Furthermore, the second axis intersection function is preferably formed by a second straight line in the coordinate system of the camera image. This second straight line is oriented parallel to the second principal direction of the point grid in the coordinate system of the code arrangement. The axis intersection function has a second function argument, wherein changing the second function argument causes the second straight line to shift parallel to the second principal direction in the coordinate system of the code arrangement. Based on the second function argument, the second axis intersection function defines the second axis intersection along the second axis of the coordinate system of the camera image, where the second axis extends through a reference point. Because changing the second function argument causes the second straight line to shift in the coordinate system of the code arrangement, thereby (in a perspective distortion manner) shifting the second straight line in the coordinate system of the camera and / or camera image, the axis intersection migrates along the second axis.

[0119] Based on the axis intersection function, determine the independent variables of the first and second functions such that the reference point forms the first and second axis intersection points. Figuratively speaking, continuously change the independent variable of the first function until the first straight line passes through the reference point in the coordinate system of the camera and / or the camera image, and / or the first axis intersection point lies on the reference point. Similarly, continuously change the independent variable of the second function until the second straight line passes through the reference point in the coordinate system of the camera and / or the camera image, and / or the second axis intersection point lies on the reference point.

[0120] Subsequently, based on the first and second function variables and the coarse position, the fine position of the reference point in the coordinate system of the code arrangement is determined as the other degrees of freedom. Conceptually, the coarse position of the basic symbol in the coordinate system of the code arrangement is first determined, and then the displacement of the basic symbol from the reference point is determined based on the function variables.

[0121] This method offers the advantage of determining the approximate location of basic symbols by decoding the code arrangement. Subsequently, the displacement up to the reference point is calculated based on the axis intersection function, where the computation can be performed efficiently with a small number of calculations. This allows the method to be implemented in real-time applications even on digital data processing devices with low computational performance, particularly microcontrollers.

[0122] In a preferred embodiment of the invention, the first function's independent variable is constructed as a row value in the point grid, and / or the second function's independent variable is constructed as a column value. Figuratively speaking, its approximate position is the position of a known basic symbol shifted to a reference point by the grid spacing of the point grid in whole or sub-step increments.

[0123] As already discussed, the reference point is particularly preferably constructed as the intersection of the camera's optical axis and the image sensing element. Therefore, the reference point is defined as its structural position in the coordinate system of the camera and / or the camera image. However, it is not mandatory that the reference point and / or the intersection be precisely centered in the camera image and / or on the camera's image sensing element. Instead, the position of the reference point can be defined by calibration.

[0124] In one possible design of the invention, the axial intersection function is constructed, in particular, as previously described, as a linear function for describing a straight line. In this case, the axial intersection function comprises a complete mathematical description of the straight line based on the corresponding function arguments.

[0125] Alternatively, a line function, particularly as previously described, can be used, which is a combination of a line angle function (which is the angle between the line and one of the axes of the coordinate system of the camera image, relating to the function's independent variable) and an axis intersection function. Thus, the line is entirely described by the axis intersection and by the line angle. This division has the advantage that, to determine a fine position, only the axis intersection function must be determined and / or evaluated, i.e., without information about the line angle. This design further improves the efficiency of the method.

[0126] Then, a fine position can be determined based on the function's independent variables and the known grid spacing of the point grid. Figuratively speaking, the fine position is determined such that, for example, starting from a position of a basic symbol with a known coarse position, one must travel a fraction of the grid spacing in a first principal direction and a fraction of the grid spacing in a second principal direction to reach a reference point in the coordinate system of the code arrangement. This representation is particularly computationally efficient.

[0127] Preferably, a first and / or second function variable of the linear function are determined such that the first and / or second lines intersect a reference point. For example, the first and second lines are shifted by changing the function variable, so that the first and second lines intersect the reference point.

[0128] Based on the first and / or second straight lines intersecting the reference point, the rotation angle Phiz of the camera around its optical axis can be derived as the camera's degree of freedom relative to the code arrangement. Since the straight line function is defined in the camera image, this derivation is possible. Therefore, it is possible to determine the rotation angle Phiz as a straight line angle in the coordinate system of the camera and / or the camera image. Theoretical considerations have shown that a straight line angle in the coordinate system of the camera and / or the camera image corresponds to the rotation angle Phiz of the camera around its steering axis, meaning that a coordinate system transformation from the coordinate system of the camera and / or the camera image to the coordinate system of the code arrangement is unnecessary. Therefore, based on at least one straight line function, it is possible to determine the rotation angle Phiz in a simple way. Thus, this improved scheme presents a way to determine the rotation angle Phiz with extremely high accuracy and without significant computational cost. It should be emphasized here that this determination is particularly simple if the intersection of the camera's optical axis through the camera image or through the image sensing element is chosen as the reference point. In this particular layout, the rotation angle Phiz can be derived particularly simply. Therefore, this method allows for the simple and highly accurate determination of the rotation angle Phiz from camera images, which is then available as a degree of freedom of the camera relative to the code arrangement.

[0129] In a preferred embodiment of the invention, a fundamental symbol matrix is ​​determined from the camera image, wherein the positions of the fundamental symbols in the camera image are recorded in the fundamental symbol matrix. Therefore, the positions in the fundamental symbol matrix are referenced to the positions of the fundamental symbols in the point grid, that is, particularly in the coordinate system of the image sensing element and / or in the coordinate system of the camera image. Preferably, a first linear function and / or a second linear function are determined based on the fundamental symbol matrix.

[0130] Particularly preferably, a linear function is used, which is a combination of a linear angle function (which is the angle between the line and one of the axes of the coordinate system of the camera image, relating to the function's independent variable) and an axis intersection function. The axis intersection function defines the axis intersection point along an axis of the coordinate system of the camera image, based on the first function's independent variable, where the axis extends through a reference point. Because changing the first function's independent variable shifts the line in the coordinate system of the code arrangement, thereby shifting the line in the coordinate system of the camera and / or the camera image (in a perspective distortion manner), the axis intersection point migrates along that axis. One combination of a linear function or linear angle function and an axis intersection function is assigned to the first principal direction, while another combination of linear functions or linear angle functions is assigned to the second principal direction. The advantage of this division is that the function's independent variable can be determined by the axis intersection function, and the rotation angle Phiz can subsequently be read from the linear angle function based on the determined function's independent variable.

[0131] In a preferred embodiment of the invention, a first value of the rotation angle Phiz is determined from a first linear function, and a second value of the rotation angle Phiz is determined from a second linear function. The rotation angle Phiz is then determined as the average of the two values. By independently determining the rotation angle Phiz from the two linear functions, two independent values ​​are obtained, which can then be averaged to determine the rotation angle Phiz, thus improving measurement accuracy. Alternatively, a reasonableness check can be performed, and if one of the two values ​​is unreasonable or invalid, that value can be discarded.

[0132] A reference point, or the reference point itself, is positioned within the camera image. This reference point is configured as the intersection of the optical axis with the image sensing element and / or with the camera image. Therefore, the reference point is pre-defined in the camera's coordinate system and / or within the camera image generated by the camera.

[0133] In a preferred improvement, the following values ​​are determined as feature values ​​from the camera image at the reference point: Determine the rotation angle phiz (also known as the Z-rotation angle) of the camera about its optical axis. The rotation angle phiz is particularly preferably determined by a line function and / or an intersection function and / or an axis intersection function.

[0134] Furthermore, at least one local grid spacing, or said local grid spacing, of the point grid in the camera image is determined. The local grid spacing describes the distance between two adjacent straight lines of the point grid in the camera image. The local grid spacing is particularly preferably determined by a line function and / or an intersection function and / or an axis intersection function. The point grid in the code arrangement is constructed regularly / or with a regular grid spacing. By utilizing the camera recording code arrangement that results in the camera image, the grid spacing is imaged in a distorted manner, causing the grid spacing to vary in the camera image. The local grid spacing at a reference point is understood as the value of the grid spacing at the reference point.

[0135] Furthermore, at least one first and second local angular divergence of the point grid in the camera image is determined. In principle, straight lines are arranged parallel to each other in the point grid of the code arrangement. However, by imaging the code arrangement onto the camera image, the point grid is distorted, causing the straight lines to take non-zero angular differences between two adjacent straight lines. The local angular divergence of the point grid in the camera image is understood as the angular difference between two adjacent straight lines of the point grid at a reference point. Here, it is the first local angular divergence with respect to a first principal direction and the second local angular divergence with respect to a second principal direction. The first and second local angular divergences can be determined, in particular, by a line function and / or by a line angle function.

[0136] From a physical perspective, the aforementioned eigenvalues ​​fully describe the three degrees of freedom of the camera relative to the code arrangement: the distance rz between the code arrangement and the camera at the reference point and / or along the optical axis, and the two pitch angles phix and phy, which describe the midpoint between the optical axis and the plane of the code arrangement in the first principal direction and the second principal direction. Therefore, the aforementioned eigenvalues ​​are sufficient to determine the aforementioned three degrees of freedom.

[0137] A further consideration here is that the aforementioned eigenvalues ​​can be determined from camera images in a simple manner. Besides determining the eigenvalues ​​in the camera images "manually," it is also possible to derive these eigenvalues ​​using digital image processing methods. Knowing these eigenvalues, the aforementioned three degrees of freedom can be inferred.

[0138] By selecting the aforementioned eigenvalues, a novel method for determining the degrees of freedom is proposed, characterized by using only a small number of eigenvalues ​​to determine the degrees of freedom. This makes it possible to design the method in a computationally efficient and highly accurate manner.

[0139] In a preferred embodiment of the invention, the aforementioned feature values ​​are inserted into an imaging model that describes optical imaging with codes arranged on an image sensor, taking into account the position and orientation of the camera relative to the code arrangement.

[0140] This model describes the physical and thus analytical interrelationship between the aforementioned eigenvalues ​​as input values ​​and the aforementioned three degrees of freedom as output values. Therefore, this interrelationship leads to the determination of the three degrees of freedom.

[0141] In one possible design of the invention, three equations are determined for the three degrees of freedom, forming a system of equations. It should be emphasized that the three equations represent a possible representation of the analytical relationship between the eigenvalues ​​and the three degrees of freedom; other mathematical or analytical representations are possible. However, the physical relationships can be represented particularly simply and compactly by three equations and / or a system of equations.

[0142] In particular, the equation used to determine the first pitch angle phix is ​​a function of the first local angular divergence dalphac0, the distance rz between the code arrangement and the camera, and the second pitch angle phiy.

[0143] In particular, the first pitch angle is determined by the following equation: ,in : The grid distance of the point grid, in meters. In particular, the equation used to determine the second pitch angle phiy is a function of the second local angular divergence dalphac1, the distance rz between the code arrangement and the camera, and the first pitch angle phix.

[0144] In particular, the second pitch angle is determined by the following equation: In particular, the equation used to determine the distance rz between the code arrangement and the camera is the following function: in b: Camera image distance, in meters (m). g C0 g C1 The local grid spacing of the point grid in the camera image, in meters (m). The columns involve two independent variables, and the rows involve two alternatives based on the definition and / or agreement of the rotation angle Phiz.

[0145] Alternatively or additionally, the distance between the code arrangement and the camera is determined as the average of variables and / or functions using the following equation: In principle, this system of equations can be solved analytically. In a preferred embodiment of the invention, the system of equations comprising three equations is solved iteratively. Here, initial values ​​are first given, and then the degrees of freedom are determined iteratively in the optimization routine.

[0146] In particular, this method can specify that a camera image is first recorded, then at least one degree of freedom of the camera relative to the code arrangement is determined, and then, for example, the actuators of an automated device are manipulated. For example, at least one degree of freedom can be output on an optical output device, such as a display. This at least one degree of freedom can be used to perform position control and / or adjustment of the actuators of the automated device in such a way that the degree of freedom is used as an actual value. For example, a robot with a sensor unit of an automated device can determine its absolute pose relative to the code arrangement and output the absolute pose as actual information, or move to another pre-given position, wherein the robot continues to face the code arrangement in its actual position.

[0147] Another subject of the invention relates to a control unit and / or an automated device having a control unit, wherein the control unit is configured to perform the method as previously described. Optionally, the control unit includes a camera and / or is connected to the camera in a data technology.

[0148] Another subject of the invention relates to a computer program configured to perform the aforementioned methods when executed on a digital data processing device and / or on a control unit.

[0149] Another subject of the present invention relates to a machine-readable storage medium having a computer program. Attached Figure Description

[0150] Other features, advantages, and effects of the invention will become apparent from the following description of preferred embodiments and the accompanying drawings. The drawings are as follows: Figure 1 A flowchart of the overall method is shown using an embodiment of the method according to the present invention; Figure 2 A schematic diagram of the optical model shown in 3D; Figure 3 A schematic diagram of the optical model is shown in 2D. Figure 4 A schematic diagram of the optical model is shown in 2D. Figure 5 , 6 Figure 7 shows a schematic diagram of the coordinate system; Figure 8 A schematic diagram of the coordinate system used for the imaging model is shown; Figure 9Show Figure 8 An imaging model with a tilted code plane; Figure 10 A diagram illustrating the arrangement of codes; Figure 11 This shows another diagram of the code arrangement; Figure 12 A dot grid showing a code arrangement with two exemplary drawn read fields; Figure 13 This shows the different angular positions of the reading field; Figure 14 An example of a read field is shown; Figure 15 This illustrates an exemplary structure of blocks in a code arrangement; Figure 16 Other exemplary structures of blocks in code arrangement are shown; Figure 17 The flowchart for the validity test is shown; Figure 18 Show details of the validity test; Figure 19 An illustrative view is shown of a camera image with the identified starting base and readout field; Figure 20 An example of a basic symbol matrix of entered data with faces is shown; Figure 21 A flowchart is shown for determining the initial basis; Figure 22 This shows multiple predefined starting positions in the camera's image field; Figure 23 A diagram illustrating the method used to determine the initial basis; Figure 24 A diagram illustrating the method used to determine the initial basis; Figure 25 This shows a flowchart of the function Center_Pos; Figure 26 A diagram illustrating the function Center_Pos is shown. Figure 27 a and b show the diagrams of the function Nearest_dot; Figure 28 A diagram illustrating the process for detecting all dots / basic symbols in the reading field; Figure 29 The diagram illustrates the process for detecting all dots / basic symbols in the reading field. Figure 30 A diagram illustrating the detection of basic symbols in a distorted raster; Figure 31 A diagram illustrating the correction is provided. Figure 32 A flowchart is shown for classifying basic symbols; Figure 33 A flowchart for determining a rough location is shown; Figure 34 a and b show the basic symbol matrix containing the entered data and the reading trajectory; Figure 35 A diagram showing the decoding at a rough location; Figure 36 A graphical representation of the fitted line used for the linear function is shown; Figure 37 A diagram showing the angle between lines; Figure 38 ac shows a graphical representation of the fitted line and the linear function; Figure 39 A diagram showing the determination of the fine position and rotation angle phiz (Z rotation angle); Figure 40 The flowchart shown illustrates the process of determining camera distance and pitch angle. Detailed Implementation

[0151] As an algorithm for accurately determining the absolute 6D position of camera 1 relative to the flat code arrangement 2 recorded by camera 1, a disclosed method is implemented, wherein the algorithm obtains camera image 3 as input information.

[0152] Code arrangement 2 serves as both an analog and digital scale in two dimensions, X and Y. The code arrangement consists of symbols arranged in a regular grid. Preferably, the grid is a binary code with two symbols arranged in a square grid: small dots (Dots) represent the digit 0, and large dots represent 1. The dot type contains digital information, while the dot center contains analog information.

[0153] Camera 1 detects segments of code arrangement 2 and transmits camera image 3 to a computer or any data processing device. The algorithm selects the smallest readout field (e.g., 7×7 dots) with symbolic dimensions in camera image 3 and calculates the position of camera 1 relative to code arrangement 2 in six dimensions from it. For this purpose, the method uses not only digitally encoded position information but also precisely measured midpoint positions of all dots in the measurement or readout field. These dots form a grid, which is distorted due to camera perspective and lens distortion.

[0154] The method for performing location detection includes the following steps (see...) Figure 1 ): Step 100: Read the camera image 3 into the computer's working memory.

[0155] Step 200: Optionally: Perform a first check on image quality based on feature data such as brightness and contrast.

[0156] Step 300: Perform a coarse XY position assessment and, optionally, a coarse z angle assessment. Step 310: Search for a starting field (launch pad) with 3×3 dots in the reading field. Step 320: Detect dots in the readout field of the camera image. Step 330: Accurately measure the midpoint position and area of ​​the dot. Step 340: Optionally: Perform mathematical correction on lens distortion by correcting the position of the dots. As a result, the curved lines of the dot grid in camera image 3 are transformed into straight lines.

[0157] Step 350: Classify the dots based on area and assign them to binary numbers 0 and 1. Step 360: Read the digital codes on the X and Y axes to determine the X, Y, and optional φ. Z The absolute coarse position in the direction is used as the z-coarse angle for evaluation.

[0158] Step 400: Distortion Assessment Step 410: Fit the fitted straight line (beam) to each row and each column of the dot grid of the readout field. Step 420: Fit the first bundle of lines (bunch) to the code grid by interpolating all lines in the rows, and fit the second bundle to the code grid by interpolating all lines in the columns. Each bundle of lines is fully described by six interpolation parameters (bundle data). The interpolated line between the two measured lines can also be calculated using the interpolation function.

[0159] The 6D camera position is calculated from eigenvalues ​​obtained from measured dot positions and describing the distorted grid in the camera image. The equation used for position calculation is derived from the inverse optics imaging model of camera 1 and the laws of geometric optics: Step 500: Fine-grained XY position assessment: The position in the X and Y directions is calculated from the coarse XY position and the pose of the grid relative to the optical axis (image center).

[0160] Step 600: Evaluate the Z-angle from the angle of the (interpolated) line at the midpoint of the image. Step 700: Z-position evaluation and XY angle evaluation Step 710: Derive the interpolated feature values ​​from the bundle data, which describe the perspective-distorted grid at points in the image: - The grid spacing of the straight lines in the two straight beams - Angular divergence of the straight lines in two straight line bundles The camera position in the Z direction is primarily determined by the grid spacing of the line at the midpoint of the image. Camera angle and The angular divergence of the two straight line bundles at the midpoint of the image is mainly determined.

[0161] Iterative algorithms numerically solve a system of equations, using these equations to satisfy parameters Z, and Related.

[0162] Step 800: Output 6D location information and optional validity information, which is derived from the results of numerous diagnostic functions in the process.

[0163] This algorithm concentrates camera image data into a small number of data points relevant to location determination with minimal computational cost. Accuracy is improved through interpolation, and it performs robustly even with camera image interference. The following process positions are particularly important (numerical values ​​are provided as examples): Process Location I: Input data: Camera image (e.g., 200×200 pixels). 40,000 values Process Position II (Step 300): Convert camera image data into dot data 2,250 values Process Position III (Steps 400 / 500): Convert dot data into beam data (beam-Daten) 90 values Process location IV (steps 400 / 500): Convert beam data to bundle data (bunch-Daten) 12 values Process position V (steps 500 / 600 / 700): Convert bundle data to 6D position. 6 values Process location II-V has the following special characteristics.

[0164] Process Location II: a) The method quickly identifies the dots in camera image 3 by mathematically modeling the perspective-distorted code grid in code arrangement 2. This method is based on the already measured positions of the dots.

[0165] Advantages: Quickly identifies dots, eliminating time-consuming searches within the pixel grid. Robust against interference in the camera image: if a dot is not found at the expected location, it is classified as "invalid." Invalid dots are not processed further, but generally do not hinder further processes.

[0166] b) A stack-based algorithm for fast and robust searching of dots in a code grid of code arrangement 2.

[0167] Advantages: Fault-tolerant because the search method does not stop at erroneous points, but instead completely surrounds them. Accurate (high gain at the found points for each readout field) and robust (by prioritizing the detection of points with many valid neighbors). c) Function Center_Pos: A fast method for determining the position and area of ​​dots in a camera image with subpixel precision. Advantages: Fast (accesses a small number of pixels), accurate (through subpixelation), robust (uses dynamic contrast thresholding instead of grayscale thresholding) d) Correct the position of the dot, not the entire camera image.

[0168] Advantages: Fast (corrects only 225 positions, instead of all 40,000 pixels in the camera image) e) Locally normalize the area of ​​the dots to the area of ​​the dot grid cells. Use the normalized dot area to classify the dots.

[0169] Advantages: Robust reading of digital codes (dot classification tolerates perspective distortion and changes in distance between the camera and the dots). f) Perform redundant reading, error identification, and partial error correction of digital codes. Advantages: Low error rate even with interference in the camera image, and correct readings even with invalid dots.

[0170] Process locations III and IV: g) The positions of the dots extracted from camera image 3 are simplified and accurately represented as bundle parameters (only 12 values).

[0171] Advantages: Fast (due to small data volume), accurate (through averaging in two stages and best-fit interpolation: from the dot to the fitted line and from the fitted line to the bundle of lines), fault-tolerant and robust (by excluding invalid dots from further processing).

[0172] Process location V: h) is a mathematical method for transforming 12 beam parameters into 6D positions.

[0173] As an advantage, some or all of the following improvements are achieved: • Full 6D position measurement, absolute position information • Detect all dimensions simultaneously using a single measurement process (in the camera image).

[0174] • High position measurement rate and low latency. This method requires only a small number of computational steps from a computer. Utilizing an embedded computer, typical measurement rates of approximately 100Hz–10000Hz are achieved, enabling the use of sensors, for example, in closed pose adjustment loops.

[0175] This method also achieves high readout speed by minimizing the number of accesses to camera image points (pixels). It avoids time-intensive planar camera image manipulation. Furthermore, the amount of data used in each processing step is significantly reduced.

[0176] • High read security due to redundant read point codes. Position detection is possible even in the presence of interference in the camera's image field, such as partial occlusion, ineffective points, or under adverse lighting conditions.

[0177] The validity of the processing steps is checked to identify erroneous states and prevent the output of unreliable position values.

[0178] • High accuracy in position measurement across all degrees of freedom. This is achieved through various measures, such as: º For each camera image 3, average over numerous dots, for example, more than 100 or 1000. The dots are categorized based on their validity, and invalid dots are excluded from further processing. Determining the precise position of sub-pixels at the edge of the dot. Using a contrast threshold instead of a grayscale threshold results in high robustness to local variations in image brightness. The circular dot ensures virtually no midpoint error when tilting or rotating the code plane. •exist Unrestricted measurement range • Virtually unlimited measurement range in X and Y. For example, a measurement range of 550,000 km × 550,000 km can be achieved using a code with 9 × 9 dots and a 2.5 mm dot grid.

[0179] • Large angle measurement range under high precision conditions , Approximately + / - 50°. Extended to 360° through spatially distributed code arrangement on the object.

[0180] • An inverse imaging model based on a geometric optics system is used to calculate the camera's position from camera image data.

[0181] • This method can be used without active lighting, allowing processing of images captured in ambient light, such as those recorded by a camera using a smartphone.

[0182] • Minimal error rate. This method determines the validity of location measurements by using different diagnostic methods to examine the errors and reasonableness of each processing step. Validity is output along with the location measurements. The goal here is to output only valid measurements for further processing.

[0183] Other advantages are: • Specifically, a single measurement sensing element configured as camera 1 detects the position of one or more objects in six dimensions. Compared to systems with many distributed sensors for detecting individual degrees of freedom, the overall system installation cost and complexity are reduced, resulting in cost advantages.

[0184] • By detecting all six degrees of freedom, system components are saved. For example, most rotary angle encoders require a pivot bearing for the measuring axis so that lateral position deviations of the code disk do not contribute to measurement errors. Using the proposed method, mechanical guidance can be eliminated because the full 6D pose information of the code disk is simultaneously detected as code arrangement. Lateral position deviations of the sensor do not contribute to errors in angle measurement. This reduces the cost and mechanical complexity of the position measurement system.

[0185] • A positioning system can be implemented where dividing the position sensor system along multiple axes is structurally impossible. For example, a suspended planar robot can be implemented that can be positioned in six dimensions without any structural connection between the robot and the stator located below it.

[0186] • By simultaneously detecting multiple dimensions, measurement errors that may occur due to different measurement time points in the case of distributed sensing systems are reduced.

[0187] • The cost of a sensor is largely independent of the number of degrees of freedom detected. Therefore, the sensor can also be advantageously used in applications requiring fewer than six degrees of freedom. Additional information provided enables additional functionalities such as system self-diagnosis or permanent observation of the system's operating status (“Condition Monitoring”).

[0188] • This method is scalable, for example, by changing the grid spacing of the code arrangement and adapting the imaging optics to the code arrangement. The resolution can vary over several orders of magnitude, from the nanometer range (application example: nanometer positioning system) to the m range (application example: automated landing of aircraft or drones at airports, where airports utilize code arrangement markings).

[0189] The resolution-to-measurement-range ratio can range from several orders of magnitude. For example, combining a path measurement system with 10 nm resolution with a 10 m long code arrangement results in a resolution-to-measurement-range ratio of 1:10. 9 .

[0190] • Sensors utilizing this method offer high flexibility in use because they can be easily installed and configured parametrically for specific applications. This is particularly advantageous under conditions of frequent switching.

[0191] This method can read additional information contained in the location code. For example, in addition to location, it can also read object identification data.

[0192] • During cyclic image recording, the method provides an independent position estimate for each individual image, which is independent of previous information from previous images. Therefore, the measurement rate corresponds to the image repetition rate.

[0193] This method enables positioning of at least one camera 1 relative to at least one object 8 in six degrees of freedom, wherein at least one code arrangement is applied to the surface of each object 8. Camera 1 detects the code arrangement 2 on the object 8 and represents the code arrangement completely or piecewise in a camera image 3. Using the proposed method and other common mathematical / technical methods, the 6D position of the object 8 can be calculated from the camera image 3.

[0194] The code arrangement 2 constitutes a planar digital encoding scale. The code arrangement contains multiple different symbols, preferably circular symbols (dots), which are arranged in a regular, preferably square grid, and enable position determination in 6 degrees of freedom.

[0195] Camera 1 includes at least • Imaging sensor elements, typically camera chips, especially image sensing elements 12.

[0196] • An imaging system that images the scale clearly and with high contrast onto the sensor elements, particularly lens 9. Since central perspective imaging is required to determine all six degrees of freedom, the imaging system specifically includes a wide-angle lens as lens 9.

[0197] • An interface for outputting camera image data and / or camera images 3.

[0198] Optionally, the camera system of camera 1 includes • Illumination system, especially for illuminating code arrangement. This makes camera 1 more independent of the lighting conditions in the surrounding environment. Short measurement times are achieved during flash operation, allowing for the detection of fast-moving objects as well. For example, light-emitting diodes (LEDs) can be used as the light source.

[0199] • Devices used to shield against incoming light, such as apertures or optical filters. Spectral filters can be present in the optical path of an imaging system so that only light from a limited wavelength range reaches the sensor element. Ideally, a monochromatic light-emitting device with the same wavelength characteristics as a color filter is used.

[0200] Additionally, a computer system is required as a digital data processing device, for example, configured as a control unit, for information processing. The computer system receives digitized camera image data (camera image 3) as input information, and at its output, provides the determined 6D position of the identified object and, optionally, an evaluation of the validity of the position measurement.

[0201] This method is implemented as an algorithm in a computer system. The algorithm is either executed based on a request from an external source or executed cyclically, for example, in fixed time slots, to detect the motion trajectory of an object (tracking).

[0202] The computer system can be implemented as an embedded system, parallel computer, GPU, FPGA, ASIC, or cloud system, for example. A smartphone can also be used as the overall system for location detection, where an integrated camera 1 (with active illumination if necessary) is used for image recording. This method can be implemented in a smartphone application and executed in a manner embedded in the smartphone or in the cloud.

[0203] Figure 2 A typical apparatus for determining absolute position in six degrees of freedom is shown. Camera 1 records camera image 3 of a flat code arrangement 2. From the perspective-distorted camera image 3, the 6D position of the camera in the coordinate system of the code arrangement 2 is calculated according to the method of the invention, and said 6D position is used as a position vector. and angle vector Output.

[0204] coordinate and In the rectangular coordinate system of code arrangement 2 (X C , Y CThe coordinates are described in the diagram. The coordinates represent the intersection point 4 of the optical axis 5 of the camera 1 and the plane in which the code arrangement is located. Since the optical axis 5 is perpendicular to the plane of the image sensing element 12, the optical axis can be represented by point 7 in the camera image 3.

[0205] Even if code permutation 2 is only represented at the edge of the image field and not at the position of optical axis 5, the method still determines the intersection point (X, Y).

[0206] distance It lies on optical axis 5 and extends from intersection 4 to the optical center of the lens of camera 1. Since optical axis 5 is not always perpendicular to code arrangement 2, therefore... With (X) C , Y C Together, they typically form a non-orthogonal coordinate system. Using angle vectors, position vectors can be... Transform to a Cartesian coordinate system.

[0207] This method allows for the positioning of multiple simultaneously recorded objects 8 within the camera's image field, even if the objects partially overlap. To read the position, at least one region (e.g., 7×7 dots) must be identifiable in the camera image 3, allowing for the identification of code blocks of that size.

[0208] Figure 3 and 4 The optical path of camera 1 is schematically shown. Figure 3 As can be seen, the camera 1, which has a lens 9 and an image sensing element 12, is looking downwards and aligned with the code plane of the code arrangement 2. The optical axis 5 is drawn as a vertical dashed line, and the focal point of the lens 9 is drawn as points 10 and 11.

[0209] According to the laws of radiation optics, the vector arrow G on the code plane of code arrangement 2 is imaged onto image sensor 12 as vector arrow B. Line of sight 14 intersects optical axis 5 at a point, which is referred to here as the optical center 13 of lens 9.

[0210] exist Figure 4 The complete optical path is shown, with lens 9 simplified to a lens. Radiation optics, with its ray theorem and lens equations, forms the basis of mathematical methods for position determination. In this case, the applicable principles are: as well as in: B Image Size B Image distance G Object size g object distance f focal length Figure 5, 6 7. Coordinate system involved in the diagram: The two-dimensional coordinate system 15 of the image sensing element 12 is defined by the axis (X) of the camera chip. I , Y I The position on the image sensing element 12 is described in units of image points [pixels], where the pixel rows and columns of the camera chip are numbered consecutively. In this example, it is assumed that the image sensing element 12 has 200 × 200 pixels. Integer X and Y position values ​​are derived for each pixel from the numbering. The actual number may also appear as the camera position during evaluation through subpixelation, a method used to interpolate grayscale values ​​between adjacent pixels. To convert the unit of position from [pixels] to [m], the position is multiplied by the distance between adjacent pixels on the camera chip in units of [m / pixel]. Pixel distance This is a characteristic of the image sensing element 12. Each pixel provides an integer grayscale value.

[0211] Coordinate system of code arrangement 2 (X) C , Y C 16 is extended by the main axis of code arrangement 2, which corresponds to a square grid of dots. This coordinate system is then extended with a third axis Z perpendicular to code arrangement 2. C Establish a three-dimensional coordinate system, in which code arrangement 2 is positioned at Z. C =0. The unit is either [dots], i.e., the consecutive numbering of rows and columns of dots, or [m]—multiplied by the dot grid distance in [m / dots]. Then, code arrangement 2 explicitly specifies the coordinate system (X... C , Y C Encode local positions within ).

[0212] The 3D camera coordinate system (X, Y, Z) 17 is fixedly connected to the camera 1. Its origin is located on the optical axis 5 between the image sensing element 12 and the code arrangement 2, at a distance of twice the image distance b from the image sensing element 12. The optical axis 5 forms the Z-axis, and the camera 1 is looking in the -Z direction.

[0213] Figure 8 The coordinate system used for the imaging model corresponds to camera coordinate system 17 (X, Y, Z). Here, b is the image distance of camera 1. The image plane lies at height (0, 0, 2b) and is parallel to the X / Y plane of camera coordinate system 17. As a model concept, another "virtual" image plane 18 can be constructed in the (X, Y) plane of coordinate system 17, where the virtual image is a point-symmetric mirror image of the real image on image sensing element 12 about the midpoint of the image.

[0214] The code plane of code arrangement 2 lies at height (0, 0, z0), where (z0 < 0). With the camera orientation at (0°, 0°, 0°), the code plane is parallel to the X / Y plane of camera coordinate system 17, and the axes of coordinate system 16 of code arrangement 2 (X...)... C , Y C The axis (X) of the coordinate system 15 of the image sensing element 12 is pointed to. I , Y I (in the same direction)

[0215] This method determines the 6D position of the camera. • In translation: ,as well as • During rotation: .

[0216] The tilt and rotation angles phix and phy of camera 1 are centered at point (0, 0, z0). The roll angle phiz is measured around the optical axis 5.

[0217] Figure 9 An imaging model with a tilted code plane and code arrangement 2 is shown. Value It is the distance between the rotation center (0, 0, z0) and the optical center of the lens (0, 0, b), which makes the following applicable: .

[0218] Based on the ray theorem and geometric optics, a point (x0, y0) in the code plane can be mapped to a point (B) in the image plane. X B Y The imaging of a camera can be mathematically described by the imaging equation in camera coordinate system 17: • The optical center 13 of lens 9: • The two-dimensional coordinates of a circle or point in the plane of the code arrangement in coordinate system 16 of code arrangement 2. in Let i be the centroid of the chosen basic symbol, and j be integers. • Spatial coordinates of a circle or point in camera coordinate system 17: The rotation matrix used for vectors in three-dimensional space is: Image points in the virtual image plane within camera coordinate system 17: •from arrive line of sight: • The intersection of the line of sight and the virtual image plane 18: • Virtual images of dots or circles: • For the real image, the negative sign must be omitted: For intersection point 4 with indices (i=0; j=0), obtain the image coordinates in the real image: 2.4.3 Code Arrangement This method requires a planar coded scale in code arrangement 2, from which sub-regions are read using an imaging sensor, such as camera 1, so that the pose of the sensor and / or camera 1 relative to the scale can be determined from image information in up to six spatial directions. Further details regarding code arrangement 2 are derived from the applicant's publication DE 102016 216 221 A1, the contents of which are incorporated herein by reference, particularly regarding the construction, decoding, and variations of the code arrangement.

[0219] The coded scale, as a sensor-readable mark constructed as a code arrangement 2, is applied to a surface that extends substantially in two dimensions, but may also have curvature. For better description, it is assumed below that the coded scale is printed as an optically readable pattern on a flat surface, without limiting the claims to other markings and sensor principles as well as curved surfaces.

[0220] The encoding scale of Code Arrangement 2 is composed of different basic symbols arranged in a regular grid. The basic symbols carry two pieces of information: their shape encodes the digital information, and their centroid marks a specific position on the surface.

[0221] In simple cases, digital information is encoded using a binary number system with a base b = 2. Thus, only two basic symbols 20 are used, such as a small circle and a large circle, which symbolically represent the values ​​"0" and "1". Their centroids (centers) mark the grid points on the surface.

[0222] The centroids of the face of basic symbol 20 form a periodic two-dimensional pattern on the scale plane of code arrangement 2, for example, as shown in... Figure 10 A square grid in which adjacent symbols have the same basic distance in the X and Y directions. .

[0223] By dividing the plane into equally sized regions and filling them with blocks 19, the basic symbols within each block 19 are combined into logical units. Square blocks 19 are preferably used. Figure 10 The diagram shows a dot grid with square blocks 19, each of which can hold 7×7 binary symbols, corresponding to a maximum information content of 49 bits. Figure 10 The lines and squares drawn in the diagram are for illustration purposes only and are not represented in the actual code permutation 2. Figure 11 An exemplary code arrangement 2 without lines is shown, which has the basic symbol 20 in block 19.

[0224] The rule arrangement of basic symbols 20 is superimposed by the rule arrangement of block symbols 21, in such a way that each block 19 is marked with block symbols 21 in the same way. For example, block symbols 21 can be represented by omitting basic symbols 20.

[0225] exist Figure 10 , 11 In this example, block symbol 21 consists of empty spaces in a dot grid located at the center of each block 19. Block symbol 21 allows the orientation of block 19 to be identified so that the basic symbols 20 can be read out in the correct order along the read path. Since block symbol 21 occupies grid space, the information content of block 19 is reduced to 48 bits in this example.

[0226] Reading field 22 is a field on the encoding scale of code arrangement 2, and the field has at least the size of block 19. The reading field is bound to the dot grid, but not to the grid of block 19.

[0227] Figure 12 A point grid is shown with two exemplary drawn readout fields 22, representing a code arrangement 2. The pose of readout field 2 in the coordinate system of the point grid is defined by its midpoint position 23. This pose can be defined by a basic distance. The integer step size changes.

[0228] Furthermore, the reading field 22 has an angular position relative to the coordinate system of the point grid, which can vary in 90° increments in the case of a square grid. Figure 12 The angle position is visualized by marking the corners of the reading field. Figure 13 The possible angular positions are shown in the diagram. Figure 12 The reading field 22 shown in the example is explicitly described by the following description: • Read field 22, left side: position = (4, 11); angle position = 0° • Read field 22, right side: Position = (12, 6); Angle position = 90° The code for code permutation 2 is constructed such that the basic symbol 20 in the reading field 22 contains enough information to digitally encode the position (X, Y) and orientation of the reading field 22 in the coordinate system 16 of code permutation 2. X and Y are based on a fundamental distance. The location is indicated by integer multiples of the approximate position. The basic distance is used for reference. The fine positions represented by fractions and the precise angles represented by fractions of 90° are not encoded digitally; they are determined by precisely positioning the basic symbol 20 in the camera coordinate system 17.

[0229] Figure 14 An exemplary illustration shows a read field 22 with 15×15 positions for basic symbols 20. This read field provides more information than is minimally required for the size of block 19 (7×7 symbols in this case). Redundant information is used for error identification and / or error correction.

[0230] Figure 15 An exemplary diagram illustrates the structure of a block 19 with 7×7 grid points. A block symbol 21 is located at the center of block 19. Two fields 24 characterize an X-block region, representing continuous code for location in the X direction. The X-block region comprises 24 grid points, and correspondingly, represents a code with a length of 24 bits.

[0231] To read the code, basic symbols 20 are read column by column from left to right and from top to bottom within each column. The reading order in block region 24 (read trajectory) is based on the grid point coordinates (X... C ,Y C The values ​​in the table are: (0,6); (0,5); (0,4); (1,6); (1,5); (1,4); (2,6); (2,5); (2,4); (3,6); (3,5); (3,4); (3,2); (3,1); (3,0); (4,2); (4,1); (4,0); (5,2); (5,1); (5,0); (6,2); (6,1); (6,0).

[0232] Two fields 25 characterize the Y block region, where consecutive codes for position are represented in the Y direction. The reading order corresponds to the reading order of the X block region 24, but rotated 90° counterclockwise. Therefore, the reading trajectory extends line by line from bottom to top and from left to right within each line.

[0233] The code from block region 24 (X code) is a subsequence of length t = 24 numbers from a total sequence of g numbers, where g is much larger than t. The quantity g is large enough that the X block region 24 of all adjacent blocks 19 in the X direction can be completely filled. In particular, g is greater than fifty, especially greater than one thousand, and especially greater than one million. The contents of the X block region 24 of adjacent blocks in the Y direction are identical. The total sequence is constructed such that each subsequence of t consecutive basic symbols 20 is included exactly once in the total sequence in a forward-reading manner, while each backward-reading subsequence is not included in the total sequence in a forward-reading manner.

[0234] The code from block region 25 (Y code) is represented in an inverted form, where each number z is replaced by the number (b-1-z). For a binary system with b=2, this corresponds to bit-by-bit inversion. The inverted Y code is a subsequence of length t=24 bits from the total sequence, where the same total sequence as for the X code can be used. The inverted Y code also appears only once in the total sequence and does not appear in the total sequence in backward reads. The contents of block region 25 of adjacent blocks 19 in the X direction are identical.

[0235] The overall sequence is constructed such that the sum of the digits of the subsequences is less than half the maximum possible sum of the digits of the subsequences. In particular, the sum of the digits of each arbitrary subsequence selected from the overall sequence is less than half the maximum possible sum of the digits of the subsequences. For example, for a number system with cardinality b, the sum of the digits of each subsequence of length t is less than q = t·(b–1) / 2.

[0236] The subsequences of the total sequence are encoded with coordinate values, such as the X-coordinate of the starting position of the reading process. Similarly, the subsequences of the inverted total sequence are encoded with Y-coordinates, such as the position from which the subsequence begins to be read. The X and Y codes read in block 19 are assigned to the X and Y coordinates of a reference point in block 19, such as the midpoint of the reading field.

[0237] Each read field 22 contains exactly one block symbol 21, t basic symbols from X block regions 24, and t basic symbols from Y block regions 25. From the positions of the block symbols 21, the pose of the block grid can be derived, and thus the poses of the X and Y block regions 24 and 25 in the read field 22 and their respective read order can be derived. The basic symbols 20 read according to the read order yield a sequence of numbers with t positions, which (after inversion if necessary) is a subsequence of the total sequence.

[0238] In the case of an unknown orientation of reading field 22, it is initially unknown which of the two axes is the X-axis and which is the Y-axis. This is determined by calculating the digit sum of the read codes and comparing it with q: the digit sum of the X codes is less than q, and the digit sum of the Y codes is greater than q. From the direction (forward or backward) of the read codes relative to the overall sequence, the direction of the coordinate axes of reading field 22 relative to the coordinate axes of code arrangement 2 can be explicitly deduced.

[0239] In this way, the pose and orientation of the read field 22 in the coordinate system 16 of the code arrangement 2 can be determined, with the pose determined in integer steps of grid width and the orientation determined in integer steps of 90°.

[0240] exist Figure 16 On the left, a block 19 with 9 rows and 9 columns is shown for its meaning. In addition to block areas X and Y 24, 25 representing the position codes relative to the corresponding axes of coordinate system 16, a block area is provided for additional data 26. This block area comprises (5×5) grid points, with the middle grid point left empty for block symbol 21, leaving 24 grid points, each capable of accommodating basic symbol 20. In this way, 24 bits of additional information can be represented in each block 19, which is not needed for position determination and is read in a prescribed order. Figure 16 On the right is another code arrangement 2 with additional data. Here, four block regions 26, each with 10 grid points, are set up for the additional data, so that a total of 40 bits of additional information are available for each block 19.

[0241] exist Figure 1 The algorithm for location determination is shown as a sequence of data processing steps, which are explained in detail below with reference to embodiments. Mathematical and image processing methods, as well as standard methods for diagnosis and error detection, are not explained in detail.

[0242] This method is particularly optimized for high accuracy and rapid executability based on location determination. The sequence of steps rapidly reduces the amount of data, which supports rapid executability.

[0243] In step 100, at process position I, digital image data is transmitted from camera 1 to the computer as a grayscale matrix. Step 200 is used to roughly verify the validity of the image data based on the feature values. Other steps for image processing may follow, such as preparing the image data or segmenting it into individual code regions. These other steps are not shown in detail here. In the case of a 200×200 camera image 3, the data volume is 40,000 pixels, and therefore 40,000 bytes in terms of grayscale values.

[0244] In this process, the symbols (dots) contained in the image are located and entered into a dot matrix according to their arrangement in the dot code raster, the dot matrix corresponding to the size of the readout field (here: 15×15 dots). Each symbol is determined in terms of its position and area. Position is corrected to correct lens distortion. The area of ​​the symbols is normalized to eliminate the effects of perspective distortion. The symbol type is classified based on the normalized area. Thus, the data volume is reduced to 225 dots with metadata and thus 9,000 bytes. In order to also classify unoccupied raster points or symbols that are incorrectly imaged, a local model of the dot code raster in the camera image is created from the positions of successfully identified symbols. Within the readout field, symbols are searched at all raster points in the model. Due to local occlusion, image errors, or image field limits, not all symbols can always be identified. Thus, the corresponding raster positions are classified as "invalid".

[0245] In process position III, fitted straight lines (beams) are fitted to rows and columns of valid dots in readout field 22. Each fitted straight line is described by the straight-line angle in the image field of camera 1 and its intersection with the X or Y axis of the image field coordinate system. Each readout field 22 provides two bundles of fitted straight lines, corresponding to the two principal axis directions of code arrangement 2. In this example, each bundle of lines includes up to 15 straight lines. Therefore, the position data of 15 × 15 = 225 dots is reduced to the data of 15 + 15 = 30 fitted straight lines.

[0246] In flow position IV, the lines of each bundle are interpolated using two quadratic polynomials: one polynomial describes the line angles, and the second polynomial describes the axial intersections of the lines in the bundle. The polynomial parameters for both bundles are represented by 12 real values, which corresponds to a reduction to one-fifth of the representation relative to straight lines.

[0247] In process position V, an inverse mathematical imaging model is also used to calculate the 6D position of camera 1 from the polynomial parameters of the two line bundles. Additionally, the validity of the position value is estimated by evaluating multiple diagnostic results from various steps in the process flow. In step 800, the 6D camera position and validity are output to the higher-level system.

[0248] Step 200 - Validation / Validity Test of Image Data Based on statistical characteristic data, examine whether there exists an evaluable camera image 3 (see [reference]). Figure 17 (See the flowchart in the example). In this example, image brightness B and contrast C are estimated and tested for compliance with limits. Further evaluation is terminated when the values ​​deviate from the pre-given limits, and the results are classified as invalid.

[0249] To save computation time, only a small number of image points are included in the determination of the feature number, for example, intersection pattern 27 ( Figure 18 The grayscale value at the intersection of the straight line and the circle in the diagram.

[0250] In sub-step 210, a set of pixels is selected, the pixels being determined according to an arbitrary intersection pattern 27. Pixels at the intersections in the intersection pattern 27 are used. Therefore, G i It is the gray level value of the pixel at position i (i=1..n).

[0251] In sub-step 220, the image brightness B and contrast C are estimated: In sub-step 230, the image brightness B and contrast C are evaluated using the following conditions: or If any of the conditions are met, further evaluation is suspended and the result is classified as invalid.

[0252] Detecting dot data / Steps 310 to 350: The objective is to identify, measure, and classify dots as basic symbols 20 within a readout field 22 of a pre-defined size (here: 15×15 grid points). Therefore, a two-dimensional dot matrix 29, or more generally, a basic symbol matrix, of the dotted data is output, where the indices of matrix 29 are assigned to the rows and columns of code permutation 2. For each dot, an eigenvalue is determined.

[0253] The parameter Dot.Typ contains the classification result. If Dot.Typ is positive, the dot is evaluated as valid. • Dot.Typ=2: Big dot; logical "1" • Dot.Typ=1: Small dot; logical "0" •Dot.Typ=0: Missing dot; block symbol 21 If Dot.Typ is negative, the dot cannot be explicitly identified.

[0254] In sub-step 310, the starting base / starting field (Launch Pad) 28 is first searched in the image region of the camera image 3 with the dot code of code arrangement 2 and / or in the read field 22. This can be understood here as a field of type 1 or 2, for example, 3×3 adjacent dots.

[0255] Figure 19An illustrative view is shown of a camera image 3 with an identified starting base 28 and a reading field 22, wherein a dot is searched as a basic symbol 20 starting from the starting base 28.

[0256] In sub-step 320, a starting base 28 is used to construct a local model of the point grid within the environment of the starting base 28. Other dots are searched at adjacent grid locations predicted by the model, accurately determined upon success, and recorded as valid dots in the dot matrix 29. This process is repeated cyclically, registering more and more valid dots around the starting base 28 until the entire 15×15 dot readout field 22 is detected. The search process is robust to local readout errors: if a dot cannot be clearly identified or extends beyond the edge of the image field, the dot is marked as invalid in the dot matrix 29 and excluded from further processing. Figure 20 Showing based on Figure 19 The results of the successful classification of camera image 3 in the dot matrix 29.

[0257] In sub-step 330, the position of the midpoint and the area of ​​the dot are determined.

[0258] Sub-step 340 converts the measured positions of all valid dots in the reading field 22 into corrected coordinates. This step is used to compensate for the distortion error (barrel distortion) of the wide-angle lens. After correction, points that are on a straight line in the code coordinate system 16 are also on a straight line in the coordinate system of the image sensing element 15 in the corrected image. Sub-step 340 is optional, as optical correction can be performed using the corresponding lens 9 instead of computer-aided correction.

[0259] Based on the measured data, the dots are classified in sub-step 350. Then, in sub-step 360, the numerical codes are read in two spatial directions and converted into integer position descriptions and rough directional descriptions in 90° steps using a code table. Thus, the determination of the rough position in the preceding step 300 is completed.

[0260] Regarding sub-step 310 – Determining the starting basis The process used to determine the initial basis 28 is in the flowchart. Figure 21 As shown in the image.

[0261] Sub-step 310.1: First, define the starting position 30, which is used to search for the first point of the starting base 28. Figure 22Multiple predefined starting positions 30 are shown in the image field of camera 1. The search begins at one of these starting positions. If the process fails (e.g., due to interference in the image field), a second search is started at a second starting position, and so on, until sub-step 310.1 can be successfully led to the end. If all starting positions have been used and the search is unsuccessful, sub-step 310.1 is terminated with a negative result.

[0262] Sub-step 310.2: From the starting position Set off and search for the nearest dot. Figure 23 Find the location in the middle. The dot at that point serves as the starting basic symbol.

[0263] Sub-step 310.3: Then from Start by searching for the nearest point, and use... It is marked as the first auxiliary starting basic symbol.

[0264] Sub-step 310.4: Then construct the connection vector. .

[0265] Sub-step 310.5: From Starting from, in connection with the vector Search for the nearest point in orthogonal directions, and use... It is marked as the second auxiliary starting basic symbol.

[0266] Sub-step 310.6: Three points , and Spread out with axis and The oblique coordinate system, where The origin of the coordinate system.

[0267] Sub-step 310.7: If this is a left-handed coordinate system (condition: Then by exchanging axes and Transform it into a right-handed coordinate system, see [link / reference] Figure 24 , on the left side.

[0268] Sub-step 310.8: via vector and Linear combination, estimate with The positions of all eight adjacent dots ( Figure 24 (Second from the left), and uses it as the starting field 28 for searching and determining the dots. Therefore, a starting base 28 of 3×3 dots is available ( Figure 24 (Second from the right, on the right side).

[0269] Sub-step 310.9: If no dot is found in this process, assume it is block symbol 21 (missing dot).

[0270] In sub-step 310.10, then... Shift one dot position in the direction opposite to the missing dot, and with and The re-search continues the process. In the case of the second traversal, it can be assumed that all 3×3 dots are marked, since the next block symbol 21 is far from block 19 (here: 7 dots), and the starting base is only 3×3 dots in size.

[0271] Subsequently, sub-step 310.11 is performed for error identification, wherein in the case of an identified error, a new starting position 30 is used in sub-step 310.12, and the action is repeated from sub-step 310.2.

[0272] As part of the previously described process, there is a challenge in identifying dots in the image field of camera 1 from the estimated starting position and determining their position and area with sub-pixel accuracy. To achieve a short measurement time, this process must be performed very quickly, as a total of 225 dots must be identified in the readout field 22.

[0273] For this purpose, the function Center_Pos, described below, is used, which is optimized for the minimum number of pixel accesses. Figure 25 The flowchart is shown.

[0274] When calling the function Center_Pos, pass the dot. The estimated midpoint position. If If the result is outside the search area, the search is terminated and a negative result is returned. Otherwise, proceed with steps a)–h), see also: Figure 26 a–h: a) Using a two-dimensional gradient method based on grayscale values, from Calculate an optimized starting point closer to the center of the circle. To calculate the gradient, only the starting point is needed. Accessing 4 pixels in the environment.

[0275] b) From the optimized point Starting from this point, the edges of the dots are searched in the +X and -X directions using methods for edge recognition. The distance from the right edge is r0, and the distance from the left edge is r1.

[0276] c) Make relative to the left and right edges on the X-axis Center it, and derive the optimized point from it. .

[0277] d) From point Starting from this point, the edges of the dots are searched in the +Y and -Y directions using methods for edge recognition. The distance from the top edge is r2, and the distance from the bottom edge is r3.

[0278] e) Make relative to the top and bottom edges on the Y-axis Center it, and derive the optimized point from it. .

[0279] f) From point Starting from this point, the edges of the dots are re-searched in the +Y and -Y directions using the methods used for edge recognition. The distance from the right edge is r4, and the distance from the left edge is r5.

[0280] g) On the X-axis relative to the left and right edges... Centered. From this, the optimized point is derived. The point is output with sub-pixel precision as the midpoint position of the circle.

[0281] h) The area of ​​the dot is calculated as the area of ​​the circumscribed rectangle, multiplied by a factor. This is used to convert the area of ​​a square to the area of ​​a circle or ellipse. As part of the previously described process, an additional challenge exists: finding the starting position in camera image 3. The nearest dot. This task can be solved using the function Nearest_dot described below. This function obtains the input data: Starting position r min Minimum search radius r max Maximum search radius ND Boolean value (ND = "neighbor dot"): If the starting position is inside the dot and ND=false, then output the dot at the starting position. If ND=true, then ignore the dot at the starting position and output the nearest neighboring dot.

[0282] The function starts from the beginning. Starting from the point of origin, the image field is scanned along 16 search rays at an angle of 22.5°. Figure 27 a, b). The radius of all search rays 31 is gradually increased, starting from radius r. minStart, until the maximum radius r max .

[0283] If ND = false (false) Figure 27 a) then it searches not only the edges leading to the point but also the edges emanating from the point. If ND = true (true) Figure 27 (b) Then only the edges leading to the circle are searched. Once an edge is found, the search stops. This edge is located at position... At that point, the edge belongs to the nearest circle being searched. From Starting from this point, the function Center_Dot is used to determine and output the midpoint position and area of ​​the found circle.

[0284] Sub-step 320 – Detecting dots in the read field Figure 28 The flowchart illustrates the process for detecting all dots in the reading field 22. For example, a previously determined starting base 28 of 3×3 dots is used as input information. Furthermore, step 330: accurately measuring the midpoint position and area of ​​the dots is integrated into sub-step 320.

[0285] In the first sub-step 320.1, the midpoint position 32 of the readout field 22 is defined. This midpoint position is chosen such that the readout field 22 has the largest possible overlap with the image region in which the code is displayed, enabling the detection of as many valid dots as possible. Typically, this is the case when the midpoint of the readout field 22 is located at the midpoint of the code region. If the code region occupies the entire image plane, as in this example, the midpoint position 32 of the readout field should be placed as close as possible to the image midpoint (see [link to example]). Figure 29 a).

[0286] At the start of this method, only the 3×3 dots of the initial base 28 are known. All other dots in the reading field 22 remain unknown, i.e., their exact locations and areas have not yet been determined. The goal of this method is to progressively determine and classify the unknown dots.

[0287] Sub-step 320.2: For each known point, determine the vector. and ( Figure 30 The vector points to the next nearest point on the principal axes 1 and 2, respectively. Due to perspective distortion, the vector expands the oblique coordinate system. Through the vector... and A linear combination of these parameters can estimate the grid position of unknown neighboring points. Position estimation for unknown points can be improved by extrapolating and averaging the estimates from multiple valid neighbors. The more valid neighbors an unknown point has, the better its position can be estimated. To improve the reliability of the reading process, only unknown points with at least two valid neighbors are checked. For each unknown point, the number of valid neighbors is counted. This count can be between 0 and 8; see [link to relevant documentation]. Figure 29 c): The center dot has 8 neighboring locations (dark-colored).

[0288] At the start of the search, only the initial 3x3 dots are valid. Figure 29 b) shows each unknown point with two or more valid neighbors in enhanced form. Figure 29 d) Schematically showing an unknown point 20a with two valid neighbors and an unknown point 20b with three valid neighbors. All other unknown points (20c) do not yet have valid neighbors.

[0289] The algorithm is based on a stack, in which all unknown dots with at least two valid neighbors are entered into the stack in substep 320.3.

[0290] The stack is processed in a circular manner: • Sub-step 320.3: The topmost entry is removed from the stack as an unknown dot and is processed in the following steps.

[0291] Sub-step 320.4: Estimate the position of the unknown point by averaging the extrapolated positions of all valid neighbors. If the unknown point is outside the expected read field 22, it is not further processed. If the unknown point is within the expected read field 22, it is identified and determined at the estimated position using the function Center_Pos.

[0292] • Sub-step 320.5: If an unknown dot cannot be identified, the unknown dot is classified as defective (sub-step 320.6), and the next unknown dot is removed from the stack.

[0293] • Step 330 / Sub-step 320.7: If a dot has already been identified, its position and area are measured using the function Center_Pos, and the dot is recorded as a new valid dot in dot matrix 29. A vector is also calculated and entered for this dot. and .

[0294] • Sub-step 320.8: For all 8 neighbors of the new valid point, the number of valid neighbors is increased by 1. If the unknown point has two or more valid neighbors, the unknown point is placed on the stack for determination.

[0295] • Sub-step 320.9: Repeat this process until the stack is empty. At this point, all dots in the read field have been processed.

[0296] Sub-step 320.10: If not enough dots are found for determining the position, the reading process is evaluated as invalid.

[0297] Regarding sub-step 340 - Correction Imaging with a wide-angle lens can lead to barrel distortion in the image field. A dot that lies on a straight line in the code plane of code arrangement 2 may appear as a curved line in the camera image 3 in some cases.

[0298] This curvature can be eliminated through mathematical methods, namely correction. For computational time reasons, the correction is only applied to the midpoint position 32 of the circle, and not to all image points in the input image. Figure 31 An uncorrected grid (33) and a corrected grid (34) are shown as examples.

[0299] Distortion center Located at the intersection of optical axis 5 and the camera chip of image sensing element 12. In the fully assembled state of camera 1, this is typically the center of the camera chip. Otherwise, this center is determined through a one-time calibration process of camera 1. The position of camera 3 in the image field is determined using a second-order polynomial. Converted to corrected position : ,in Relative to the position of the distortion center Distance from the center of distortion Distortion only changes the vector The length of the polynomial is determined without changing its direction. The polynomial parameters p0, p1, and p2 are specific to lens 9 and are adapted to the lens during calibration. Alternatively, a distortion-free lens 9 can also be used.

[0300] Regarding sub-step 350 – classifying the dots Small and large dots symbolically represent the values ​​"0" and "1" in the binary number system. Dots are classified based on a threshold of their normalized area. This process is illustrated in the flowchart. Figure 32 As shown in the image.

[0301] Perspective distortion has a strong effect on the area of ​​dots in an image field. For example, dots that are farther away are imaged as smaller, while circular dots on an inclined plane are imaged as ellipses.

[0302] Sub-step 350.1: To compensate for these effects, the area of ​​the dot is normalized relative to the area of ​​its dot-Zelle unit. The dot-Zelle unit can be understood as a parallelogram, with vectors at the dot's location... and That is, the distance vector from the point to the adjacent points on the principal axes 1 and 2 expands the parallelogram.

[0303] Figure 30 Showing vectors and Perspective distortion dots within a partially expanded distorted grid. The area of ​​each cell is... .

[0304] The normalized area of ​​the dots is calculated as follows: Sub-step 350.2: Use statistical methods to calculate the normalized area of ​​all dots (Dot.Area). scaled Determine the threshold.

[0305] Sub-step 350.3: Then classify all the dots: Missing dots: Type 0 If the normalized area is less than the threshold: Type 1; otherwise: Type 2.

[0306] Sub-step 350.3: If an error occurs, the state is set to invalid; if no error occurs, the dot is valid.

[0307] Regarding sub-step 360 – Reading Code After classifying the dots, the dot matrix 29 contains all the information to determine the 6D position.

[0308] Dot matrix 29 contains the following information: The flowchart for this is in Figure 33 As shown in the diagram, the approximate positions of the numbers in the X and Y directions are determined by reading the X and Y bit chains in the dot matrix.

[0309] Sub-step 360.1: Block symbol 21 "empty dot" (type 0) is used as a reference point to identify the pose of the bit chain in dot matrix 29. If multiple entries of type 0 are contained, the entries can be validated against each other for reasonableness, since the "empty dot" is repeated regularly in the block grid (here: 7×7 dots).

[0310] Sub-step 360.2: Starting from block symbol 21, determine the position of the reading trajectory for the code on the two axes of the code plane. Here, what is initially unknown is: which sequence in the sequence is assigned to the X-axis and which sequence is assigned to the Y-axis, and in which direction (forward or backward) the code is read. The code is read along the reading trajectory and stored as Code_0 and Code_1. Figure 34 a) An example shows the dot type entered in the dot matrix. Block symbols (type 0) are shown in gray. Figure 34 b) Additionally, read trajectories for both axes are drawn. The read trajectories for the two axes are rotated 90° relative to each other, with block symbol 21 forming the center of rotation.

[0311] Since the read field (15×15 dots) is significantly larger than the block (7×7), the code can be read redundantly: the copy of Code_0 is in the block (35a) adjacent in the j direction, and the copy of Code_1 is in the block (36a) adjacent in the i direction.

[0312] There is also redundancy in the length of the readable code. A code length of 24 bits in each direction, i.e., the content of a 7×7 block, is sufficient for location determination. However, a larger read field with 15×15 dots provides 51 or 54 bits in each direction. Redundancy information is used to identify and correct erroneously read bits in Code_0 and Code_1.

[0313] In this example, the following code is read: Code_0: Code_1: Sub-step 360.3: Select any segment with t=24 consecutive bits from Code_0 and verify its integrity.

[0314] Sub-step 360.4: Then calculate the digit sum of Code_0. If the digit sum is less than t / 2 = 12, then Code_0 is the X code, and Code_1 is the Y code. If the digit sum is greater than t / 2, then Code_0 is the Y code, and Code_1 is the X code. The Y code is inverted. The X code remains unchanged.

[0315] Sub-step 360.5: Then, using a fault-tolerant string search, search the code table for Code_0 and Code_1. Output the bit positions with the best consistency as the search results; search results with too large a deviation are evaluated as invalid. To transform Code_0 and Code_1 to integer space coordinates (X, Y), the bit positions of the codes are converted to space positions according to the code structure, as exemplarily in... Figure 35 The following is schematically shown: - Select any segment with t=24 consecutive bits and verify its integrity: Search the code table forwards for the segment, and locate the segment at position 37 where the underscore is located: - Each X dot is assigned a bit position within the code table.

[0316] - This bit position corresponds to the initial point in the case of the read order within the block.

[0317] - Each Y-dot is assigned a bit position within the code table.

[0318] 360.6 Validity Test according to Figure 35 This corresponds to the dot position Xc=10=Pos0. The Yc value of Pos1 is determined in a similar manner. For the two codes, the reading directions (Dir_0 and Dir_1) are also determined by searching the code in two directions in the code table: 0 = found forward and upward, 1 = found backward and upward. From the reading directions of Code_0 and Code_1, the coarse orientation of the reading window 22 is determined in steps of 90° according to the following table.

[0319] As a result of substep 360, prepare to read the orientation of window 22 and its integer coarse position (r). X,int ,r Y,int ).

[0320] Step 400 – Determine beam data To determine the precise position, the corrected position data of the valid dots in dot matrix 29 are evaluated. Figure 36 An example is shown showing the dots of the readout field in a camera image after correction.

[0321] Step 410: Fit the fitted line To this end, a fitted straight line (beam) is fitted to the rows and columns of valid points in the reading field 22. The fitted straight line is calculated from the position data of the valid points in the corresponding rows or columns using common mathematical methods for minimizing errors. Invalid points are excluded from this method.

[0322] Each fitted line i of the bundle k=0..1 is described by the following parameters ( Figure 37 ): With X I Intersection of axes With Y I Intersection of axes Angle of line i in the image field The fitted straight lines to the dotted row form the first bundle of lines (bunch0) 36, where the lines intersect with Y... C Parallel to the axis. The straight lines of the dotted line array form the second bundle of lines (bunch1) 37, where the straight lines are parallel to the X-axis. C The axes are parallel. Therefore, the positional data of 15×15=225 points are reduced to 15+15=30 points for fitting the straight line.

[0323] Interpolation also has the effect of averaging the positions of the dots on each line, making the fitted line robust to individual fluctuations at each dot position. This improves the stability and accuracy of the 6D position values.

[0324] exist Figure 36 The figure shows the fitted straight lines for the reading field with 15×15 dots. The straight line from bundle0 is shown as 36, and the straight line from bundle1 is shown as 37.

[0325] Step 420 – Determine bundle data In this step, the three line parameters SX of each fitted line 36 and 37 are fitted using a quadratic polynomial. ki SY ki and α ki Interpolation is performed. The interpolation method only considers valid lines. Invalid lines (too few valid points are available for interpolation of these invalid lines) are excluded from the bundle interpolation. To do this, weighted interpolation based on minimizing the squared error is performed using common mathematical methods. Valid lines are evaluated with a weight of 1.0, while invalid lines are evaluated with a weight of 0.0 and are therefore hidden.

[0326] Scheme interpolation function: Here, i is the consecutive number (line index) of the lines in the corresponding line bundle.

[0327] A general scheme for interpolating linear parameters: ;in or To explicitly describe a straight line mathematically, only one of the intersection points SX of the two axes needs to be specified. ki or SY ki Together with the straight angle α ki That is sufficient. Ideally, the intersection point with the axis that is as perpendicular to the line as possible should be specified: in the case of a horizontal line, the intersection point with the Y-axis is preferably specified; in the case of a steep line, the intersection point with the X-axis is specified. Therefore, the following method is applied: • In bundle 1, select a reference line that is close to the midpoint 32 of the reading field 22.

[0328] • If the angle of the reference line is within the range or ( Figure 38 a) X I In the sector on the axis), it is a straight line. Therefore, for all the straight lines in bundle 0 (36), specify the intersection point SX with the X-axis. 0i And for all lines in bundle 1 (37), specify the intersection point SY with the Y-axis. 1i ( Figure 38 b) Set rot_status to 0 as a flag (Merker).

[0329] • If the angle of the reference line is outside the above range (in Figure 38 a) Y I If the sector on the axis is a steep straight line, then for all straight lines in bundle 0 (36), specify the intersection point SY with the Y-axis. 0i And for all lines in bundle 1 (37), specify the intersection point SX with the X-axis. 1i ( Figure 38 c) Set rot_status to 1 as a flag.

[0330] This step further reduces the data size: a bundle of lines is described by only 6 parameters: for α ki Regarding the three parameters and one of the axis intersection points SX ki or SY ki There are three parameters. In the case of two bundles, there are 12 parameters and the flag is rot_status.

[0331] Therefore, regardless of the number of lines or the size of the measurement field, the two bundles 36 and 37 are always described by 12 parameters and the Boolean variable rot_status. The 6D position is calculated from these parameters in the next step.

[0332] Interpolation also has the effect of averaging over the straight lines of each bundle and thus over all the dots, making the interpolated values ​​robust to individual fluctuations of each straight line or dot. This improves the stability and accuracy of the 6D position values.

[0333] Step 6 – Calculate the position of the 6D camera Calculate the six coordinates of the camera position from the 12 interpolation parameters of the linear bundle (Table 9.1) in the following order: Sub-step 500: Position r X and r Y Sub-step 710: Calculate the eigenvalues ​​of the distorted raster. Sub-step 600: Camera angle φ Z Sub-step 700: Position r Z Sub-step 700: Camera angle φ X and φ Y Sub-step 700: Used for φ X φ X and r X Iterative algorithms for solving systems of equations The calculations involve deriving the eigenvalues ​​of the measured grid at the position of optical axis 5. These eigenvalues ​​are correlated with the position of the 6D camera through a set of equations derived from the imaging model and the laws of ray optics.

[0334] The camera position is determined by solving a system of equations. Additionally, the validity of the position value is estimated by evaluating multiple diagnostic results from various steps in the program flow.

[0335] Regarding substep 500 - calculating the camera position r X and r Y The camera positions on X and Y are calculated as integer coarse positions (r). X,int , r Y,int The sum of the real fractions [0..1] (fine position) and the dot grid distance: Unit: [m] Unit: [m] in : As determined in step 300, the integer position of the read field is expressed in [dots]. Read the real fraction of the field position, in units of [dots]. : Dot grid distance, in [m / dot].

[0336] The reference point used to measure the camera position is the intersection point 4 of the optical axis 5 and the code arrangement 2, which is the intersection point of the optical axis 5 and the image sensing element 12 in the camera image 3. Figure 2 , point (4) or Figure 39 The image of the midpoint 4, i.e., the image center 4. This image center IC (4) is determined by the pose of the lens 9 relative to the camera chip; the image center does not necessarily have to be the same as the center of the camera chip. The image center is determined during the calibration process and is used as a two-dimensional constant vector (IC). X IC Y The fraction (Bruchteile) is stored in the program. X,fract and r Y,fract Calculate the intersection point of the interpolated line and point 4 in the image. For the two ray beams k=0 and k=1 (36, 37), the interpolation equation for the axis intersection point applies: ;in or It depends on rot_status.

[0337] Figure 39 shows the midpoint line for two line bundles k=0 and k=1.

[0338] Applicable to: ■For rot_status=0: ■For rot_status=1: By appropriately selecting the interpolation parameters (i, j), the axial intersection of the two midpoint lines is shifted to the image midpoint. This is satisfied by the following equation: ■For linear bundle 0: ■For linear bundle 1: ■Among them The solution to the equation leads to the index of the straight line. and The score searched: Regarding substep 710 – calculating the eigenvalues ​​of the distorted raster Obtain other feature values ​​representing the distorted raster from the interpolation equation. Fine-grained positioning... and Substitute these values ​​into the interpolation equation to obtain the feature values ​​of the points in the image. These feature values ​​are needed for the calculation in r. Z φ X φ Y and φ Z Determine the location in terms of dimensions.

[0339] a) Grid spacing at the axial intersection of two straight lines: g C0 g C1 .

[0340] By targeting the intersection point S of the axes k (i) The grid spacing is determined by differentiating the interpolation function with respect to the dimensionless line index i at the position of the point in the image: For bunch k : in b) The angle of a line at a point in the image: α C0 , α C1 (See) Figure 37 ), From the interpolation function for line angles: For bunch k : c) Angular divergence of the line bundle at the midpoint of the image: dα C0 dα C1 .

[0341] This can be understood as the angular difference between adjacent lines close to the midpoint of the image. The angular divergence is derived from the derivative of the interpolation function for the line angle with respect to the dimensionless line index i at the midpoint of the image: For bunch k : Regarding step 600 – calculating the camera angle φ Z The straight angle φ is determined from the imaging equation derived in 2.4.2. Z : ;in Distance between the code plane and the optical center By setting image coordinates B X and B Y Taking the derivative with respect to the code grid coordinate x0, we obtain The slope of the straight line in the straight line bundle 1 in the camera image is We consider the slope at the center where y0=0, and obtain .

[0342] is the slope of the straight line at the midpoint of the image. It corresponds to the eigenvalue from 6.2 b). ,Right now .

[0343] This leads to the camera angle φ Z The equation: Regarding substep 700 – calculating the camera position r Z From two straight bundles i The intersection point g of the axes Ci Determine the camera position r from the grid distance. Z Since there are two straight line bundles in each image, the two r values ​​can be determined in principle. Z Position. When rot_status=k, four values ​​r are obtained. Zik Where (i, k=0..1). For Figure 9 The imaging equations are applicable to: ;in Distance between the code plane and the optical center For example, for rot_status=0, from B X The measured grid distance g at the intersection of the axes C0 Determined value .

[0344] In B X Applicable on the axis is B Y =0, derived from the imaging equation By substituting this term into the case of B X In the equation, we obtain Grid distance g C0 Corresponding to in derivative at point ;in Line index, in units of [dots]. From this, we can draw the following conclusions. At x0=0, we obtain And solve : By substituting matrix element Rij We obtained targeting The equation: For rot_status=1, the position r is calculated in the same way. Z01 ,plan: From g C1 The position r is derived from the middle Z1k Used for r Z10 Solution: For r Z11 : The results are summarized in Table 10.1. rot_status (from (derived from the middle) (from (derived from) 0 1 Table: Calculating the camera position value r Zik .

[0345] By introducing angles The equation can be summarized as follows: To output only the value r for each image Z Calculate the average of the two z values: Regarding substep 700 – Calculating the camera angle From two straight bundles k angular divergence at the center of the image The angle φ is derived from the middle X φ Y In step 600, we have already derived the slope m1 of the line bundle bundle 1 as a function of index y0: We calculate the line angle from it. And make the line angle pair Take the derivative of the [dot] to obtain the angular divergence: ■ At the center of the image, y0=0, the angular divergence is ■ and the measured angular divergence Equivalent, resulting in the following for φ Y The equation: In the same way, the angle φ is derived from the slope m0 of the straight line in bundle 0. X Similar to the calculation in 600, we obtain ■We make the angle right Find the derivative and obtain the angular divergence. ■ At the image center x0=0, the angular divergence is ■ and the measured angular divergence Equivalent, resulting in the following for φ X The equation: Regarding substep 700 – iteratively calculate the camera position Z and camera angle. In the function φ X φ Y There is a mutual dependency between Z and Z: Therefore, the equations are solved in the iterative process. With each iteration step, the camera position Z and the camera angle φ... X and φ Y The accuracy is improved. This algorithm is shown in the flowchart. Figure 40 As shown in the image.

[0346] Sub-step 700.1: Calculate the real fraction of the reading field position in units of [dots]. Sub-step 700.2: Calculate position r x and r y Sub-step 700.3: Calculate camera rotation / camera angle φ Z Sub-step 700.4: Initialize camera angles phix=0; phiy=0 Sub-step 700.5: Initialize a counter with, for example, 4 for iteration. Sub-step 700.6: Calculate r Z0k and r Z1k And finally calculate Sub-step 700.7: Calculate camera angles phix and phiy Sub-step 700.8: Counter = 0 Recalculate; otherwise, terminate the iteration. Sub-step 700.9: Error lookup The algorithm converges quickly; after approximately four iterations, the results become sufficiently stable. The final output is as follows: • Camera position • Camera angle • Validity information List of reference numerals in the attached diagram: 1. Camera 2. Code Arrangement 3 Images 4. The intersection of optical axis 5 and code arrangement 2; control point; 5 optical axes 6 empty 7. Intersection of the optical axis and the image sensing element 8. Objects with at least one code permutation 2 9 lenses 10 Lens 9 Focus 11. Focus of lens 9 12 Image sensing elements 13. Optical center of lens 9 14. Line of sight 15. Coordinate system of image sensing element 12 in the plane of image sensing element 12 (X I, Y I, ) 16. Coordinate system of code arrangement 2 in the plane of code arrangement 2 (X) C , Y C ) 17. The coordinate system of camera 1 (X, Y, Z) 18 Virtual Image Plane Block 19 20 Basic Symbols 21 block symbols 22 Reading Field 23 Midpoint position 24 X Block Area 25 Y Block Area 26 Block areas for additional data 27. Intersection Pattern 28. Starting site / starting foundation / starting platform 29 Dot Matrix 30 Starting position 31 Search Rays 32. Read the midpoint position of field 22. 33 Uncorrected dot grid 34 Corrected dot grid 35. Reading trajectory in the first code direction 36. Reading trajectory in the second code direction 37 First Straight Beam / bunch0 38 Second straight beam / bunch1.

Claims

1. A method for determining at least one degree of freedom of a camera (1) relative to a code arrangement (2) from camera images of the camera (1), The code arrangement (2) has a dot grid with multiple basic symbols (20), the approximate positions of which are encoded in the code arrangement (2). A starting field (28) with basic symbols (20) is determined, wherein the starting field (28) has at least three basic symbols (20), wherein two independent principal directions along the point grid are estimated in the camera image by means of the basic symbols (20) of the starting field (28), wherein the basic symbols (20) of the starting field constitute valid basic symbols. In the search step, starting from at least one valid basic symbol (20), other basic symbols (20) are searched along at least one of the main directions, and in the case of a successful search, the other basic symbols (20) are marked as valid basic symbols (20). The search step is executed multiple times. Based on the effective basic symbols (20) or a subset thereof, the approximate position of at least one basic symbol in the code arrangement and / or the approximate position of the coordinate system of the point grid are determined as two degrees of freedom of the camera relative to the code arrangement (2).

2. The method according to claim 1, characterized in that, The basic symbol (20) of the starting field (28) defines two connection vectors in the camera image along two independent main directions of the point grid.

3. The method according to claim 1 or 2, characterized in that, For the basic symbol (20), the corresponding midpoint position of the basic symbol (20) is determined in the camera image, wherein the search step is performed starting from the midpoint position of the corresponding basic symbol (20).

4. The method according to any one of the preceding claims, characterized in that, The area of ​​the basic symbol (20) is determined in the camera image, wherein the area constitutes data in the encoding of the code arrangement (2).

5. The method according to any one of the preceding claims, characterized in that, In the preparatory steps for creating the starting field (28), at least one starting position (30) is specified, and from the starting position (30), adjacent basic symbols (20) are searched along the search ray (31) as starting basic symbols.

6. The method according to claim 5, characterized in that, Starting from the position of the initial basic symbol, search along the search ray (31) for adjacent basic symbols (20) as the first auxiliary initial basic symbol.

7. The method according to claim 6, characterized in that, Starting from the initial basic symbol and the first auxiliary initial basic symbol, search for a second auxiliary initial basic symbol in a direction perpendicular to the connection between the initial basic symbol and the first auxiliary initial basic symbol.

8. The method according to claim 7, characterized in that, Starting from the initial basic symbol, all directly adjacent basic symbols of the initial basic symbol are searched by linear combination of the connections between the initial basic symbol and the first auxiliary initial basic symbol and the second auxiliary initial basic symbol (20).

9. The method according to any one of the preceding claims, characterized in that, Starting from the starting field (28) and / or the effective basic symbol (20), search for adjacent basic symbols (20) that have at least two effective neighbors in the main direction or diagonal direction.

10. The method according to any one of the preceding claims, characterized in that, A readout field (22) is defined in the camera image and / or planar point grid, and a corresponding basic symbol matrix is ​​constructed, wherein each point in the readout field (22) is assigned an entry in the basic symbol matrix, wherein the position of the basic symbol (20) and, in particular, the normalized area or data are entered in the basic symbol matrix, wherein, based on the basic symbol matrix, the coordinate system of the point grid and / or the approximate position of at least one basic symbol in the code arrangement is determined to have two degrees of freedom.

11. The method according to any one of the preceding claims, characterized in that, Based on the basic symbol matrix, for a first straight line, a first straight line function with a first function independent variable is derived in the coordinate system of the camera image, wherein the first straight line is oriented parallel to the first principal direction in the coordinate system of the code arrangement (2), wherein by changing the first function independent variable, the first straight line is shifted parallel to the second principal direction in the coordinate system of the code arrangement (2), and / or for a second straight line, a second straight line function with a second function independent variable is derived in the coordinate system of the camera image, wherein the second straight line is oriented parallel to the second principal direction in the coordinate system of the code arrangement (2), wherein by changing the second function independent variable, the second straight line is shifted parallel to the first principal direction in the coordinate system of the code arrangement (2), wherein at least one other degree of freedom of the camera is determined based on at least one of the straight line functions.

12. The method according to any one of the preceding claims, characterized in that, Reference points are arranged in the camera image, having a first axis intersection function formed by a first straight line in the coordinate system of the camera image, wherein the first straight line is in the coordinate system (X... C Y C The first line is oriented parallel to the first principal direction (2) of the point grid in the coordinate system (X) of the code arrangement, having a first functional independent variable, wherein by changing the first functional independent variable, the first line is oriented in the coordinate system (X) of the code arrangement. C Y C The first axis intersection function is shifted parallel to the second principal direction in the coordinate system (X) of the camera image, according to the first function argument. I Y I The first axis determines the first axis intersection point, wherein the first axis extends through the reference point and has a second axis intersection point function formed by a second straight line in the coordinate system of the camera image, wherein the second straight line is in the coordinate system of the code arrangement (X). C Y C Oriented parallel to the second principal direction in the coordinate system (X, Y, Z) of the code arrangement, the second line has a second functional independent variable, wherein by changing the second functional independent variable, the second line is oriented in the coordinate system (X, Y, Z) of the code arrangement. C Y C The second axis intersection function is shifted parallel to the first principal direction in the coordinate system (X) of the camera image according to the second function's independent variable. I Y I The second axis determines the intersection point of the second axis, wherein the second axis extends through the reference point, wherein the first function independent variable and the second function independent variable are determined based on the axis intersection function, such that the reference point constitutes the intersection point of the first axis and the intersection point of the second axis, wherein the reference point is placed in the coordinate system (X) of the code arrangement (2) in the plane of the code arrangement (2) based on the first function independent variable, the second function independent variable, and the approximate position. C Y C The fine position in ) is determined as at least one other degree of freedom.

13. The method according to any one of claims 10 or 11, characterized in that, Determine the first function independent variable and / or the second function independent variable such that the first line or the second line intersects the reference point, wherein based on the first line and / or the second line, the rotation angle (phiz) of the camera about the optical axis is derived as the degree of freedom of the camera (1) relative to the code arrangement (2).

14. The method according to any one of the preceding claims, characterized in that, A reference point or the reference point is arranged in the camera image, wherein the reference point is configured as the intersection of the optical axis of the camera and the image sensing element and / or the camera image, wherein the following values ​​are determined from the camera image at the reference point as feature values: - The rotation angle of the camera about the optical axis of the camera or the rotation angle (phiz); - The first local grid spacing (gc0) of the point grid in the first main direction in the camera image and / or the second local grid spacing (gc1) of the point grid in the second main direction in the camera image. - The point grid has a first local angular divergence (dalphac0) in the first principal direction in the camera image, wherein the first local angular divergence (dalphac0) describes the angular difference between two adjacent lines in the first principal direction in the point grid; - The point grid in the camera image has a second local angular divergence (dalphac1) in the second principal direction, wherein the second local angular divergence (dalphac1) describes the angular difference between two adjacent lines in the point grid in the second principal direction. Based on the feature values, the following three additional degrees of freedom of the camera relative to the code arrangement are determined: - The distance (rz) between the code arrangement and the camera; - The optical axis is relative to two independent pitch angles (phix; phy) of the code arrangement.

15. An electronic control unit or an automation device having an electronic control unit, wherein the control unit is configured, in a programming and / or circuitry manner, to perform the method according to any one of the preceding claims.

16. A computer program, wherein the computer program is configured to perform the method according to any one of claims 1 to 13 when the computer program is executed on a computer or on a control unit or automation device according to claim 14.

17. A machine-readable storage medium wherein a computer program according to claim 15 is stored on the storage medium.

Images