Method for determining at least one degree of freedom of camera, computer program, machine-readable storage medium, and electronic control unit or automation device

Abstract

The invention relates to a method for determining at least one degree of freedom of a camera (1) relative to a coding array (2) on the basis of a camera image of the camera (1), in which a basic symbol matrix is determined on the basis of the camera image, in which basic symbol matrix the position of the basic symbols in the camera image is entered, and in which at least one degree of freedom of the camera (1) relative to the coding array (2) is determined on the basis of the camera image. On the basis of the basic symbol matrix, a first straight line function of a first straight line having a first function argument in a coordinate system of the camera image is derived, the first straight line being oriented parallel to the first main direction in a coordinate system of the coding array (2), the first straight line being oriented parallel to the first main direction in the coordinate system of the coding array (2) by changing the first function argument. The first straight line is shifted in parallel in the coordinate system of the coding array (2) in the second main direction, and / or a second straight line function of a second straight line having a second function argument in the coordinate system of the camera image is derived, the second straight line is oriented parallel to the second main direction in the coordinate system of the coding array (2), and wherein the second straight line is shifted parallel in the first main direction in the coordinate system of the coding array (2) by changing the second function arguments, and wherein the second function arguments are changed on the basis of at least one of the linear functions. The at least one degree of freedom of the camera (1) is determined.

Description

Technical Field

[0001] This invention relates to a method for determining at least one degree of freedom of a camera relative to an coded array in a camera image, as described in claim 1. The invention also relates to a computer program, a machine-readable storage medium having the computer program, and an electronic control unit or an automation device having the electronic control unit. Background Technology

[0002] In modern manufacturing and other applications, it is necessary to detect the position of moving objects relative to their environment. Various sensing technologies exist for this purpose, such as displacement sensors and readout-coded positions. All existing sensing technologies offer specific advantages for their respective applications, which can be based on accuracy, simplicity, low-cost integration, robustness, or other characteristics.

[0003] In the prior art publication DE 10 2016 216 221 A1, which is closest in composition to the applicant, a method is proposed for locating an object in space and / or on a surface and / or for determining the position of an object in space and / or on a surface, which uses a two-dimensional coded array. The two-dimensional coded array has basic symbols arranged on a surface, forming a two-dimensional periodic grid on that surface. Based on the coded array, the position of a camera relative to the coded array can be determined.

[0004] The applicant's publication DE 10 2016 216 196 A1 discloses a sensor system in which the sensor system uses the methods for positioning and / or determining location from the aforementioned publication. Summary of the Invention

[0005] A method for determining at least one degree of freedom of a camera relative to an coded array based on camera images from a camera, having the features of claim 1, an electronic control unit having the features of the co-independent claims, or an automated device having said electronic control unit, a computer program, and a machine-readable storage medium are proposed. Preferred or advantageous embodiments of the invention are derived from the dependent claims, the following description, and the accompanying drawings.

[0006] The method according to the invention is used to determine at least one degree of freedom of the camera relative to the coded array based on camera images from the camera.

[0007] This method is based on the method of published document DE 10 2016 216 221 A1 and / or the application of published document DE 10 2016 216 196 A1, the contents of which are incorporated herein by reference.

[0008] Definition of a camera: The camera specifically includes an optical sensor unit (also called an imaging sensor element, camera chip, or image sensor) and an imaging optical system for recording camera images. In particular, the camera is designed to be a color camera or a monochrome camera. The optical sensor unit is designed to provide single-frame camera images and / or sequences of camera images, such as in video format, of segments of the encoded array. The camera has an image sensor as the optical sensor unit. The camera may include a lens, which is preferably designed as a wide-angle lens. In particular, the camera implements vanishing perspective and / or central perspective. Here, when viewed at an angle, parallel edges, in particular, are not presented parallel to the image but converge optically to an imaginary point, the so-called vanishing point. In particular, the camera (especially including the lens) implements perspective distortion required in relation to the application.

[0009] Definition of optical axis 5: An imaging optical system is ideally a rotationally symmetric optical system, where the axis of symmetry is the optical axis. The characteristic of this optical axis is that a beam of light traveling along it will not be deflected as it passes through the optical system.

[0010] Definition of image center 7: The intersection of the optical axis 5 and the optical sensor unit 12 is called the image center 7. This intersection may, but does not necessarily have to, coincide with the geometric center of the optical sensor unit 12.

[0011] Definition of a camera image: The camera's image is specifically designed as a matrix, which has image points, such as 8-bit grayscale points or color points, at matrix points.

[0012] Definition of a coding array: Encoded arrays are particularly capable of being imaged by a camera, whereby, based on camera images, the absolute position of a machine or machine part in one to six degrees of freedom can be measured, for example. In particular, the encoded array is designed to be detected by a camera through non-contact reading, wherein multi-dimensional real-world position determination can be performed through non-contact detection by the camera. Preferably, two-dimensional encoded arrays can be used in production and / or testing facilities where the positioning of workpieces and / or testing tools and / or production materials is required.

[0013] Definition of basic symbols: The two-dimensional encoding array comprises basic symbols arranged on a surface to form a two-dimensional periodic grid, or dot grid. These basic symbols are preferably geometric shapes, such as circles, squares, triangles, or lines. Particularly preferred are these basic symbols designed as circles. These basic symbols are also called dots. They preferably represent numbers in a number system. In particular, the two-dimensional encoding array contains at least two distinct basic symbols. In one possible embodiment of the invention, the two-dimensional periodic grid includes unoccupied spaces at grid positions and / or dot grid positions, serving as block symbols.

[0014] Basic symbols are arranged on a surface, which can be curved or non-curved. For example, the surface is the floor of a production and / or testing facility. These basic symbols are arranged in a two-dimensional grid on the surface, wherein, preferably, the centroids of these basic symbols form grid points in the grid.

[0015] Definition of the dot grid of the encoding array: Grid points are also referred to below as grid locations and / or lattice locations. In particular, a two-dimensional periodic point grid forms a two-dimensional lattice.

[0016] Block definition: The surface is preferably divided into similar, regularly arranged blocks, wherein these blocks have a basic shape, such as a square, rectangle, triangle, or hexagon. Similar blocks specifically refer to blocks of the same size and / or shape. Preferably, each block contains n basic symbols. These blocks particularly contain an integer number of basic symbols, and especially an even number. Preferably, these blocks contain more than ten basic symbols, especially more than twenty, and particularly more than forty. Furthermore, the number of basic symbols in a block is preferably less than one hundred. The blocks formed by the basic symbols can be displayed optically in an coded array, such as by borders, or they can be displayed non-optically and constitute only imaginary and / or logical units.

[0017] Definition of a block area: These blocks have at least a first block region and a second block region. Block region X contains the first block region, and block region Y contains the second block region. In particular, each block region occupies a contiguous area or multiple scattered, non-contiguous sub-regions within the block.

[0018] Each block region contains multiple basic symbols. In particular, the X block region and the Y block region contain the same number of basic symbols. In particular, at least two block regions are arranged in the block such that these block regions have p-fold rotational symmetry with the center of the block as the pivot, where p-fold rotational symmetry is, for example, double, triple, or quadruple rotational symmetry.

[0019] Definition of block symbol: These blocks each have at least one block symbol, which represents a fixed reference point within each block. The block symbols are capable of reading and / or decoding these basic symbols in a prescribed order. The block symbols are arranged regularly and / or periodically, particularly within a two-dimensional periodic grid of basic symbols. The block symbols may be located at or near grid points. The block symbols are particularly arranged in the same positions within the block, such as at the center of the block. The block symbols are, for example, graphic elements other than basic symbols, such as triangles, hexagons, or lines. Alternatively and / or supplementarily, the block symbols are presented by omitting one or more basic symbols from the block. In one possible design, the block symbols constitute the p-fold rotationally symmetric points of the block. In particular, the reading direction and / or decoding order are specified, i.e., the order in which the basic symbols within the X block region and / or within the Y block region need to be read. Preferably, the reading direction and / or decoding order conform to a specification regarding which basic symbols should be read and / or decoded sequentially.

[0020] Definition of encoding: In the X block region, the X coordinate values ​​are encoded using basic symbols, and in the Y block region, the Y coordinate values ​​are encoded using basic symbols. Specifically, the X and Y coordinate values ​​are the coordinates of the basic symbols within the blocks, indicated in a Cartesian coordinate system of the surface extended by the basic symbols. Alternatively and / or additionally, the X and Y coordinate values ​​may also indicate the position within the surface as coordinates in other coordinate systems (such as oblique, cylindrical, or spherical coordinate systems).

[0021] Definition of the preferred implementation scheme of point grid: In a particularly preferred embodiment of the invention, the two-dimensional periodic grid is a rectangular grid, wherein the blocks are also rectangular. In particular, the rectangular grid and rectangular blocks are square grids and / or square blocks. Preferably, the distances of these basic symbols along the length and width axes of the rectangular grid are equal. For example, for a square grid with square blocks, the number of basic symbols in the X and Y directions of the planar two-dimensional periodic grid is equal. Specifically, the X block region and the Y block region are each composed of two spatially separated rectangular sub-regions within the block, wherein these sub-regions have a longitudinal extension. The longitudinal extension of the sub-regions of the X block region is preferably perpendicular to the longitudinal extension of the sub-regions of the Y block region. Preferably, the surface occupied by the X block region can be converted into the surface occupied by the Y block region by rotating 90°.

[0022] In a particularly preferred embodiment of the invention, the periodic grid has a grid length and a grid width. The grid length extends along the X direction of the Cartesian coordinate system, and the grid width extends along the Y direction of the Cartesian coordinate system. The resulting coordinate system assigns a unique position vector (X, Y) to each point on the encoded surface.

[0023] In particular, g consecutive basic symbols constitute a total sequence. This total sequence is entered into the X block region, particularly into multiple adjacent blocks along the X direction. The entry of basic symbols is performed, in particular, according to the order of these basic symbols in the total sequence, preferably in ascending order along the X direction of the blocks, and within each block, they are entered into the X block region according to a pre-defined reading and / or decoding order. The number g is large enough that the X block regions of all adjacent blocks along the X direction can be completely filled. In particular, g is greater than fifty, more particularly greater than one thousand, and especially greater than one million. The contents of the X block regions of adjacent blocks along the Y direction are identical. Particularly preferred is that fragments of t consecutive basic symbols in the total sequence form a subsequence. In particular, each fragment of the t consecutive basic symbols in the total sequence forms a subsequence, wherein these consecutive basic symbols are consecutive in decoding and / or reading order. In particular, the overall sequence is designed such that each subsequence of t consecutive basic symbols, read forward, is included only once in the overall sequence, and each subsequence read backward is not included in the overall sequence when read forward. Preferably, the subsequence contains at least five consecutive basic symbols, especially at least twenty consecutive basic symbols, and particularly at least thirty basic symbols. Furthermore, the subsequence preferably contains fewer than fifty basic symbols, and particularly fewer than thirty basic symbols. Especially applicable to: t <g。

[0024] In a particularly preferred embodiment of the invention, the basic symbol is designed to encode numbers into a base b. The base b is preferably a base of a positional number system. Preferably, the base b = 2 is a binary base, where the binary digits include 0 and 1. It is also possible that the base is b = 10, forming a decimal base, where the decimal numbers include the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. Alternatively, the base is b = 16, i.e., a hexadecimal base, where the hexadecimal numbers include the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F. In particular, the base b can be arbitrarily chosen, where the base numbers include the elements 0, 1, ... b-1. Alternatively and / or additionally, the base b is a base of an additive number system, such as the Roman numeral system.

[0025] In a particularly preferred embodiment of the invention, the base number b = 2 is chosen. Specifically, the chosen base number b = 2 constitutes binary. The binary and / or base number system b = 2 contains two digits, specifically the digits 0 and 1. Preferably, two distinct basic symbols encode the binary digits 0 and 1. These two distinct basic symbols are formed by two circles: a first circle with radius R1 and a second circle with radius R2. Specifically, radius R1 is chosen to be smaller than radius R2. Specifically, the ratio of radius R2 to radius R1 is greater than the square root of 2. For example, radius R2 is chosen such that it is less than half the grid size of the encoding grid, such that two adjacent R2 circles in a periodic grid are not tangent. This design is based on the considerations that, on the one hand, a particularly high information density is achieved; on the other hand, reliable readability is achieved using standard image processing methods such as segmentation with the aid of blob analysis.

[0026] In one possible design of the invention, the overall sequence is designed such that the horizontal sum of the subsequences is less than half of the maximum possible horizontal sum of the subsequences. In particular, the horizontal sum of each arbitrary subsequence selected from the overall sequence is less than half of the maximum possible horizontal sum of the subsequences. For example, for a number system with a cardinality b, the horizontal sum of each subsequence with length t is less than t·(b – 1) / 2.

[0027] In a particularly preferred embodiment of the invention, the two-dimensional coded array includes a reversed total sequence. The reversed total sequence preferably represents a reversal of the total sequence. Preferably, the reversed total sequence is entered into the Y-block regions of multiple adjacent blocks along the Y-direction. The entry of basic symbols is performed, particularly according to the order of these basic symbols in the reversed total sequence, preferably in ascending order along the Y-direction of the blocks, and within each block, they are entered into the Y-block regions according to a pre-defined reading and / or decoding order. The number g is large enough that the Y-block regions of all adjacent blocks along the Y-direction can be completely filled. In particular, the contents of the Y-block regions of adjacent blocks along the X-direction are identical.

[0028] In one possible design of the present invention, the basic symbol at the k-th position of the reversed total sequence is the encoded digit m(k), wherein the digit m(k) and the digit n(k) encoded by the basic symbol at the k-th position in the total sequence satisfy the relationship: m(k) = b – n(k) – 1. For example, for choosing a binary number with a base b = 2, the reversed total sequence and the total sequence form the relationship: m(k) = 1 – n(k).

[0029] Definition of the point of incidence: Particularly preferably, the segment includes an incident point. The incident point is, for example, the intersection of the camera's optical axis and the plane containing the encoding array. Preferably, the method is designed to determine the camera's position relative to the working area based on the image recorded by the sensor unit, such as the X coordinates (rx) and Y coordinates (ry) of the incident point in a coordinate system extended through the working area and / or the basic symbols and / or the encoding array.

[0030] First, within the scope of this method, only a coarse position, in the form of coded X and coded Y coordinates, is determined as two degrees of freedom. This coarse position alone provides a reliable and usable location for the coarse positioning of the camera.

[0031] By determining the X and Y coordinates in the coded array as approximate positions, the camera's relative position to the array can be deduced. For example, if this approximate position is determined to be adjacent to the point of incidence, the camera's position can be inferred by knowing the orientation of its optical axis. Since the point of incidence is equivalent to the coded position and the angle of the optical axis relative to the coded array is known, a near-precise position of the camera can be obtained. The simplest case is where the optical axis is oriented perpendicular to the coded array.

[0032] Definition of degrees of freedom: In a particularly preferred embodiment of the invention, the method is designed to determine the position and / or orientation of the working module relative to the working area in up to six degrees of freedom. These degrees of freedom are, in particular: rx: Position coordinates in the coordinate system of the encoded array (as a coarse position and / or a fine position). ry: Position coordinates in the coordinate system of the encoded array (as a coarse position and / or a fine position). rx and ry specifically relate to the point of incidence, particularly the point where the optical axis passes through the coded array. rz: The distance of the reference point relative to the camera along the optical axis. phiz: The Z-axis rotation angle of the camera around the optical axis. phix: The first pitch angle of the optical axis relative to the coding array, especially relative to the first principal direction. phiy: The second pitch angle of the optical axis relative to the coding array, especially relative to the second principal direction.

[0033] Definition of X and Y coordinates (rx, ry): For example, the six degrees of freedom of the camera's position and / or orientation relative to the work area include: the X and Y coordinates relative to the Cartesian coordinate system extended from the basic notation, especially the coordinate system of the coded array in the plane of the coded array (X... C Y C )middle.

[0034] Definition of the distance (rz) between the incident point and the optical center of the camera: In addition, these six degrees of freedom include the distance Z or r of the camera relative to the incident point along the optical axis. z The distance, for example, refers to the distance between the point of incidence and, for example, the optical center of a camera lens.

[0035] Definition of rotation angle: In addition, three independent rotation angles of the camera in the camera coordinate system are determined. Two of these rotation angles are specifically defined as the angles (phix, phiy) between the optical axis and the encoding array. The third rotation angle (phiz) represents the rotation of the camera about the optical axis.

[0036] Definition of the coordinate system of the encoded array: The coordinate system of the encoding array (X) C Y C It lies within the plane of the encoding array.

[0037] Definition of the coordinate system of the camera's image sensor and / or the coordinate system in the camera image: Image sensor coordinate system (X I Y I It is located in the plane of the image sensor or in the camera image.

[0038] Definition of camera coordinate system: The origin of the camera's coordinate system (X, Y, Z) is located at the optical center of the camera lens, where the negative Z direction coincides with the camera's optical axis.

[0039] General instructions: Although the coordinate system is presented here using the origin and orientation as examples, an equivalent coordinate system can also be used.

[0040] Method definition: In some variations, the method is characterized in that the X and Y coordinate values ​​are decoded and / or determined based on camera images recorded by the camera. In a possible further extension of the method, the orientation of a segment of the coded array recorded in the image relative to the three coordinate axes is determined. Specifically, the orientation of the segment recorded in the image relative to the X and Y axes of a Cartesian coordinate system extended from the basic notation is determined. Alternatively and / or additionally, the coordinates of the location of the incident point in the coded array are determined by decoding the segment of the coded array contained in the image. In a particularly preferred embodiment of the invention, the method determines the pose of the working module relative to the working area in up to six degrees of freedom. Here, in particular, the pose of the camera in the working area is determined using three Cartesian coordinates X, Y, and Z, where X and Y are coordinates in a coordinate system extended from the basic notation, and Z is the distance of the working module from the working area. Furthermore, in this method, for example, the three Euler angles of the camera are determined.

[0041] Alternatively and / or supplementarily, two-dimensional coded arrays are used in fields outside of automation technologies, such as for monitoring motion processes in nature and the environment, biology and medicine, construction, consumer electronics and / or sensing technologies.

[0042] The overall advantages of this invention depend on the following variations: The core objective of this invention is to provide a powerful, reliable, and economical method for locating and identifying one or more objects in a workspace with up to six degrees of freedom (6D positioning = simultaneous position detection in six degrees of motion).

[0043] This method processes image data provided by an imaging measurement system (e.g., a camera). For example, a smartphone can also be used to record images and determine location using the method according to the invention, wherein the method is implemented as an algorithm in an application executed by an embedded computer in the smartphone or in the cloud.

[0044] To determine the position of an object in up to six degrees of freedom, the camera detects a two-dimensional coded array associated with the object to be located.

[0045] Depending on the variant, the method according to the invention provides the following characteristics: - Up to complete 6D positional information: translation (X, Y, Z) and rotation (φ_X, φ_Y, φ_Z) - Absolute position information (non-incremental), that is, eliminating the need for a reference oscillation as in incremental sensors. - Extremely large measurement range: - The X and Y directions are "virtually unlimited," limited only by the coded array. Example: Uniquely coded area when using the coding described in [B] and a 2.5mm dot grid: For a 5x5 block size: 2m x 2m For a 7x7 point block size: 8km x 8km For a block size of 9x9 points: 550,000 km x 550,000 km In the Z direction: single-dimensional, with boundaries. It can be adapted to the application by adjusting the camera resolution, lens, and grid size of the encoding array. Regarding φ_X and φ_Y: Current + / - 60° Infinite in the φ_Z direction (0°-360°) - High measurement rates, such as 100Hz – 10000Hz - Low latency (time from image recording to output of 6D position measurements), e.g., 10ms – 100μs. - High measurement accuracy across all six degrees of freedom: For a 2.5mm coding grid size, the current repeatability is 1 μm and 0.01° (at 3 times the standard deviation (3 sigma)). - Maintains maximum read reliability even with localized interference or fluctuations in image brightness. Both sensor systems and encoder arrays can be implemented economically. - Additional data can be read during location. For example, the ID code integrated into the encoding array can be read to identify the object. - By changing the resolution of the lens and image sensor, the grid size of the encoding array, the size of the readout area, computing power, etc., scaling can be achieved in terms of accuracy, measurement range in the Z direction, redundancy, and measurement rate. - It can be easily integrated into existing systems, such as as a smartphone app.

[0046] This allows for the simple implementation of various automation technologies that were previously impossible or uneconomical to implement, such as... - Precisely position axes or sliders in a system of axes or robotic systems to enable real-time posture control in up to six dimensions. - Build absolute positioning systems and robots that operate without reference. - The positioning system is simplified because multiple one-dimensional measurement sensors can be replaced with a single multi-dimensional measurement sensor. - Visual servoing, for example, used to enable a robot gripper to track a moving target. - Monitor and track multiple objects in space through cyclic position measurement. - Record and analyze the motion process - Vibration analysis of the machine in 6 dimensions. - Determine the relative pose of two objects. A camera, such as one in a handheld device, records two objects in an image, each equipped with a point code. The proposed method allows the positioning of these two objects relative to a common camera coordinate system. The relative pose of the two objects relative to each other can be determined by vector subtraction. Here, the camera position is eliminated through computation; that is, the method is largely independent of camera position. - Determine the relative pose of more than two objects, each equipped with a camera and / or an encoding array. - Establish a positioning grid, such as in a production workshop, to position multiple stationary or moving objects relative to each other or relative to the workshop coordinate system. For example, multiple autonomous vehicles, each equipped with a camera, perform self-localization by detecting an array of codes mounted on the workshop ceiling.

[0047] The encoding array has a point grid, particularly a planar grid, which has a plurality of basic symbols, wherein, preferably, the center of these basic symbols is located at a grid point, and wherein the approximate position of these basic symbols is encoded in the encoding array.

[0048] These basic symbols define a first principal direction along the point grid and a second principal direction independent of that first principal direction. Depending on the viewpoint, the two independent principal directions in the camera image are imaged perpendicular to or at an angle to each other.

[0049] A fundamental symbol matrix, specifically a two-dimensional matrix, is determined based on the camera image. Each grid point of the coded array is assigned an entry in the fundamental symbol matrix, and the center position of the fundamental symbol in the camera image is entered into this matrix. Therefore, the entry in the fundamental symbol matrix refers to the position of the fundamental symbol at the assigned grid point, more precisely, in the coordinate system of the image sensor and / or in the coordinate system of the camera image.

[0050] Perspective distortion is particularly prevalent in point grids in camera images, causing them to appear at an angle to each other, even though the rows and columns of the point grid are designed as straight lines.

[0051] In addition, optical distortion may occur, which can optionally be compensated for by correction.

[0052] Based on the data from the basic symbol matrix, a first line function with a first functional variable is determined (derived) in the coordinate system of the camera image. When the first line is imagined to be transferred to the coordinate system of the encoding array, regardless of the first functional variable, the first line is always parallel to the first principal direction of the point grid. By changing the first functional variable, the first line is shifted parallel to the second principal direction in the coordinate system of the encoding array. Therefore, by changing the first functional variable, the line can be placed, for example, on a row of the point grid, more precisely, both in the coordinate system of the encoding array and the coordinate system of the camera image.

[0053] In particular, this can be achieved as follows: Initial situation: a) The center of the basic symbol forms a two-dimensional point grid in the encoding plane, which has regularly arranged rows and columns of straight lines. b) The camera images the coded plane onto the image sensor with perspective distortion. In the camera image, these rows typically appear as fan-shaped lines with a common vanishing point. The same applies to the columns. c) Due to lens distortion, these rows and columns appear as curves in the camera image.

[0054] Method steps in the camera coordinate system: a) By correcting the distortion of the lens mathematically, the curve is converted into a straight line. b) Lines are fitted to rows using linear interpolation, and these lines are numbered sequentially using integer indices as the function arguments. These lines form the first bundle of lines. Columns are processed in the same way, and these lines form the second bundle of lines. c) Each line is described by its line angle and axial intercept values. The line angles of the line bundle form a numerical sequence, which is approximated by quadratic interpolation. Integer line indices are used as the function's independent variables. The polynomial has three interpolation parameters. The axial intercept values ​​are processed in the same way, from which three additional interpolation parameters are derived. Thus, each line bundle is fully and compactly described by six interpolation parameters. d) By substituting rational function values ​​into the interpolation function, a straight line between two rows or columns can also be calculated.

[0055] Alternatively or supplementarily, based on the basic symbol matrix, a second line function with a second function independent variable is determined (derived) in the coordinate system of the camera image for the second line. When the second line is transferred to the coordinate system of the encoding array, regardless of the second function independent variable, the second line is always parallel to the second principal direction of the point grid. By changing the second function independent variable, the second line is shifted parallel to the first principal direction in the coordinate system of the encoding array. Therefore, by changing the second function independent variable, the line can be placed, for example, on a row of the point grid, more precisely, both in the coordinate system of the encoding array and the coordinate system of the camera image.

[0056] It is important to note that the terms “row” and “column” are used only for naming purposes and do not indicate a specific orientation of the point grid.

[0057] At least one degree of freedom of the camera is determined based on at least one of these linear functions. Alternatively, the at least one degree of freedom of the camera can be determined based on both linear functions.

[0058] The present invention considers reducing the amount of data in the process of deriving at least one degree of freedom from a camera image, so that the at least one degree of freedom can be calculated more quickly and / or more efficiently. For example, a camera image with an image size of 200 x 200 pixels still has 40,000 values, while the basic symbol matrix has been reduced to positions, and correspondingly, for a segment of a camera image with a side length of 15 basic symbols for a point grid, the basic symbol matrix still has only 225 position values. By deriving at least one linear function, the amount of data is reduced to the parameters of that linear function.

[0059] In this case, it is evident that for each linear function, the six parameters, optionally supplemented by one data item, are sufficient to preserve the essential information of the basic symbol matrix or camera image while traversing to that at least one degree of freedom, reducing the data volume from 225 entries to 12 or 14 entries in this example. Thus, the computation for that at least one degree of freedom is significantly reduced in terms of workload.

[0060] Another advantage of this implementation scheme is that the rows and columns of the basic symbol matrix in the encoding array are arranged parallel to each other and regularly spaced, which allows for a kind of averaging of the basic symbol matrix by deriving linear functions from the camera image. These linear functions describe the averaging information of the basic symbol matrix. Through these linear functions, information compression is performed while improving the information content.

[0061] Therefore, the method according to the invention allows for computationally efficient determination of at least one degree of freedom of the camera relative to the coded array based on camera images. From an application technology perspective, this determination can be performed, for example, on a microcontroller capable of determining the at least one degree of freedom at least 100 times per second. In this way, the method can be used, for example, in manufacturing to perform real-time applications.

[0062] In a preferred extension, the first linear function is determined (derived) based on at least two rows, preferably more than two rows, and especially all rows of the basic symbol matrix. Alternatively or supplementarily, the second linear function is determined (derived) based on at least two columns, preferably more than two columns, and especially all columns of the basic symbol matrix. This extension emphasizes that the linear function carries mean-based and / or compressed information across multiple rows or columns.

[0063] In a preferred embodiment of the invention, for each row along a first main direction, a first fitted straight line is formed. The first straight line function is formed based on multiple first fitted straight lines. By forming a first fitted straight line from a single row, the first fitted straight line can be adapted to the direction of the row, thus forming the averaged and / or compressed information of the row on which the first fitted straight line is based. The first straight line function is formed based on multiple first fitted straight lines, wherein a second averaging or compression is performed, so that the first straight line function is formed through two averagings of the original information.

[0064] Alternatively or supplementarily, for each column along the second principal direction, a second fitted straight line is formed. The second straight line function is formed based on multiple second fitted straight lines. By forming a second fitted straight line from a column, the second fitted straight line can be adapted to the direction of the column, so that the second fitted straight line has formed the mean and / or compressed information of the column on which it is based. The second straight line function is formed based on multiple second fitted straight lines, wherein a second mean or compression is performed, so that the second straight line function is formed by the two mean operations of the original information.

[0065] In a preferred implementation, the first function's independent variable is designed as a row-based integer first count value, and / or the second function's independent variable is designed as a column-based integer second count value. Therefore, for the integer first count value in the first linear function, the first line corresponds to the first fitted line. Similarly, for the integer second count value as the second function's independent variable in the second linear function, the second line corresponds to a column in the basic symbol matrix. However, since the linear functions have undergone a second mean-compression, the first and / or second lines are not exactly the first or second fitted line, but rather a modified first or second fitted line.

[0066] In a preferred implementation, these fitted lines are described by a line angle intersecting with the coordinate system of the image sensor and / or camera image, or with an axis of that coordinate system, and at least one axis intersection point. For these fitted lines, a line angle intersecting with at least one or exactly one axis of the coordinate system and an axis intersection point are sufficient to uniquely determine these fitted lines within that coordinate system.

[0067] With this implementation, the positions of these fitted lines in the point grid of the relevant rows or columns are reduced to two values.

[0068] In a preferred extension, the linear function is formed by a combination of the line angle function relating to the line angle of the linear function and the axis intersection function relating to the axis intersection point of the linear function. Therefore, the linear function is also determined by the line angle and at least one axis intersection point.

[0069] Preferably, the line-angle function is designed as a quadratic polynomial, and / or the axis intersection function is designed as a quadratic polynomial, wherein these polynomials have the corresponding linear function as their independent variable. This achieves the goal of determining the line-angle and / or axis intersection based on the independent function variable. By choosing quadratic polynomials, approximation can be performed particularly simply, further improving computational efficiency.

[0070] In a preferred embodiment, the intersection point of the coordinate system's coordinate axes is selected based on the line angle between the linear function and the coordinate system, choosing the axis with the smaller angle between the line function and the perpendicular to the corresponding coordinate axis. Furthermore, a data item is assigned to the linear function, encoding the selected coordinate axis. The consideration here is that to describe a line, only intersection with a single coordinate axis is needed, but not with two. To achieve the highest possible efficiency, the intersection point with the assigned angle more perpendicular to the intersecting coordinate axis is chosen.

[0071] In an extended embodiment of the invention, the approximate position of at least one of the basic symbols in the encoding array is determined based on the basic symbols of the basic symbol matrix or a subset thereof. Specifically, a readout region is defined in the camera image and / or a planar point grid, and a corresponding basic symbol matrix is ​​formed, wherein each point in the readout region is assigned an entry in the basic symbol matrix, wherein the positions of these basic symbols, and in particular, normalized areas or data items, are entered in the basic symbol matrix, and based on this basic symbol matrix, the approximate position of the point grid coordinate system and / or the approximate position of at least one basic symbol in the encoding array are determined as two degrees of freedom.

[0072] Preferably, the first straight line forms a first axis intersection function in the coordinate system of the camera image. The first straight line is oriented parallel to a first principal direction of the point grid in the coordinate system of the encoding array. The axis intersection function has a first function independent variable, wherein changing the first function independent variable causes the first straight line to be shifted parallel to a second principal direction in the coordinate system of the encoding array. The first axis intersection function defines the first axis intersection point along a first axis of the coordinate system of the camera image based on the first function independent variable, wherein the first axis passes through a reference point. Because the first straight line is shifted in the coordinate system of the encoding array by changing the first function independent variable, thus—with perspective distortion—the first straight line is shifted in the coordinate system of the camera and / or camera image, causing the axis intersection point to move along the first axis.

[0073] Furthermore, preferably, the second straight line forms a second axis intersection function in the coordinate system of the camera image. The second straight line is oriented parallel to the second principal direction of the point grid in the coordinate system of the encoding array. The axis intersection function has a second function independent variable, wherein changing the second function independent variable causes the second straight line to shift parallel to the second principal direction in the coordinate system of the encoding array. The second axis intersection function defines the second axis intersection point along the second axis of the coordinate system of the camera image, where the second axis passes through a reference point, based on the second function independent variable. Because the second straight line is shifted in the coordinate system of the encoding array by changing the second function independent variable, thus—with perspective distortion—the second straight line shifts in the coordinate system of the camera and / or camera image, causing the axis intersection point to move along the second axis.

[0074] Based on these axis intersection functions, the independent variables of the first and second functions are determined such that the reference point forms the first and second axis intersections. Figuratively speaking, the independent variable of the first function changes until the first line passes through the reference point and / or the first axis intersection lies on the reference point in the coordinate system of the camera and / or the camera image. Similarly, the independent variable of the second function changes until the second line passes through the reference point and / or the second axis intersection lies on the reference point in the coordinate system of the camera and / or the camera image.

[0075] Subsequently, based on the first and second function variables and the coarse position, the fine position of the reference point in the coordinate system of the coding array within the plane of the coding array is determined as another degree of freedom. Logically, the coarse position of the basic symbol in the coordinate system of the coding array is first determined, and then the displacement of the basic symbol from the reference point is determined based on these function variables.

[0076] The advantage of this method is that the approximate position of the basic symbol can be determined by decoding the encoded array. Subsequently, based on these axis intersection functions, the displacement up to the reference point is calculated. This calculation can be performed efficiently using a small number of computational values. Furthermore, this method can be implemented in real-time applications even on digital data processing devices with limited computing power (especially microcontrollers).

[0077] In a preferred embodiment of the invention, the first function's independent variable is designed as the count value of rows in the point grid, and / or the second function's independent variable is designed as the count value of columns in the point grid. Figuratively speaking, the position of a basic symbol with a roughly known location is shifted to a reference point through all or part of the steps, in units of the point grid's grid size.

[0078] As already discussed, the reference point is particularly preferably designed as the intersection of the camera's optical axis and the image sensor. Therefore, the reference point is defined as the structural location in the coordinate system of the camera and / or the camera image. However, the reference point and / or intersection does not necessarily have to be located at the exact center of the camera image and / or the camera's image sensor. Instead, the position of the reference point can be defined through calibration.

[0079] In one possible design of the invention, the axial intersection function is designed as a linear function to describe a straight line, particularly as described above. In this case, the axial intersection function includes a complete mathematical description of the straight line depending on the respective function's independent variable.

[0080] Instead, a linear function is used, specifically as described above, which is formed by a combination of a line angle function (the angle between the line and one of the axes of the coordinate system of the camera image, which is related to the function's independent variable) and an axis intersection function. Therefore, the line is fully described by both the axis intersection and the line angle. The advantage of this division is that, to determine a fine position, only the axis intersection function needs to be determined and / or evaluated; more precisely, no information about the line angle is required. This design further improves the efficiency of the method.

[0081] Then, based on these functional variables and the known grid size of the point grid, the fine position can be determined. Figuratively speaking, the fine position is determined such that, for example, starting from the position of the basic symbol with a known coarse position, it is necessary to move a fraction of the grid size along a first principal direction and another fraction of the grid size along a second principal direction to reach the reference point in the coordinate system of the encoded array. This representation is computationally particularly efficient.

[0082] Preferably, the first and / or second function variables of the linear function are determined such that the first and / or second lines intersect with a reference point. For example, the first and second lines are shifted by changing the function variables so that the first and second lines intersect with the reference point.

[0083] Based on the first and / or second straight lines intersecting the reference point, the rotation angle of the camera about the optical axis is derived as the camera's degree of freedom relative to the coded array. This derivation is possible because these linear functions are defined in the camera image. Therefore, the rotation angle Phiz can be determined as a line angle in the coordinate system of the camera and / or camera image. Theoretical analysis has shown that the line angle in the coordinate system of the camera and / or camera image corresponds to the rotation angle Phiz of the camera about the swing axis; more precisely, it is not necessary to transform the coordinate system from the coordinate system of the camera and / or camera image to the coordinate system of the coded array. Therefore, the rotation angle Phiz can be determined in a simple way based on at least one linear function. Thus, using this extended scheme, a way to determine the rotation angle Phiz with extremely high accuracy and without a large amount of computational effort is demonstrated. It should be emphasized here that this determination is particularly simple if the intersection of the camera's optical axis through the camera image or through the image sensor is chosen as the reference point. In this particular configuration, the rotation angle Phiz can be derived particularly simply. This method allows for the simple and highly accurate determination of the rotation angle Phiz based on camera images, which can then be used as the camera's degree of freedom relative to the coded array.

[0084] In a preferred embodiment of the invention, a basic symbol matrix is ​​determined based on a camera image, wherein the positions of basic symbols in the camera image are entered into the basic symbol matrix. Therefore, the positions in the basic symbol matrix refer to the positions of the basic symbols in a point grid, more precisely, in the coordinate system of the image sensor and / or in the coordinate system of the camera image. Preferably, a first and / or second linear function is determined based on the basic symbol matrix.

[0085] Particularly preferably, a line function is used, which is formed by a combination of a line angle function (which is the angle between the line and one of the axes of the coordinate system of the camera image, relating to the function's independent variable) and an axis intersection function. The axis intersection function defines the axis intersection point along an axis of the coordinate system of the camera image, based on the first function's independent variable, where the axis passes through a reference point. Since the line is shifted in the coordinate system of the encoding array by changing the first function's independent variable, the line is shifted in the coordinate system of the camera and / or camera image—with perspective distortion—causing the axis intersection point to move along that axis. One combination of a line function or line angle function and an axis intersection function is assigned to the first principal direction, and another combination of a line function or line angle function is assigned to the second principal direction. The advantage of this division is that the function's independent variable can be determined by the axis intersection function, and the rotation angle Phiz can then be read from the line angle function based on the determined function's independent variable.

[0086] In a preferred extension of the invention, a first value of the rotation angle Phiz is determined from a first linear function and a second value of the rotation angle Phiz is determined from a second linear function, and then the rotation angle Phiz is determined as the average of these two values. Since the rotation angle Phiz can be determined independently from these two linear functions, two independent values ​​are obtained, which can then be averaged to determine the rotation angle Phiz, thereby improving measurement accuracy. Alternatively, a reasonableness check can be performed, and if one of the two values ​​is unreasonable or invalid, that value can be discarded.

[0087] A reference point, or such reference point, is placed in the camera image. The reference point is designed to be the intersection of the optical axis with the image sensor and / or with the camera image. Therefore, the reference point is specified by the camera in the camera's coordinate system and / or in the camera image.

[0088] In a preferred extension scheme, based on the camera image, the following values ​​are determined as feature values ​​at the reference point: Determine the rotation angle of the camera about its optical axis, or this rotation angle Phiz—also known as the Z-axis rotation angle. Particularly preferably, the rotation angle Phiz can be determined using a line function and / or an intersection function and / or an axis intersection function.

[0089] Furthermore, at least one local grid size, or the local grid size itself, is determined in the point grid within the camera image. The local grid size indicates the distance between two adjacent straight lines of the point grid in the camera image. Particularly preferably, this local grid size can be determined using a line function and / or an intersection function and / or an axis intersection function. The point grid in the coded array is formed regularly and / or with a regular grid size. When the coded array is recorded using a camera (which produces a camera image), the grid size is distorted in the imaging, causing the grid size to vary across the entire camera image. The local grid size at a reference point refers to the value of the grid size at that reference point.

[0090] Furthermore, at least first and second local angular divergences of the point grid in the camera image are determined. In principle, straight lines are arranged parallel to each other in the point grid of the coded array. However, due to the distortion of the point grid when the coded array is imaged onto the camera image, these straight lines exhibit non-zero difference angles between two adjacent lines. The local angular divergence of the point grid in the camera image refers to the difference angle between two adjacent straight lines of the point grid at a reference point. This is the first local angular divergence with respect to a first principal direction and the second local angular divergence with respect to a second principal direction. The first and second local angular divergences can be determined, in particular, by a line function and / or by a line-angle function.

[0091] From a physical perspective, the aforementioned eigenvalues ​​fully describe the three degrees of freedom of the camera relative to the coded array: the distance rz between the coded array and the camera at a reference point and / or along the optical axis, and the two pitch angles phix and phy, which describe the angles between the optical axis and the plane of the coded array in the first principal direction and the second principal direction. Therefore, the aforementioned eigenvalues ​​are sufficient to determine the aforementioned three degrees of freedom.

[0092] A further consideration here is that the aforementioned eigenvalues ​​can be determined from camera images in a simple way. Besides manually determining the eigenvalues ​​in camera images, these eigenvalues ​​can also be derived using digital image processing methods. Knowing these eigenvalues, the aforementioned three degrees of freedom can be inferred.

[0093] Therefore, by selecting the aforementioned eigenvalues, a novel method for determining the degrees of freedom is proposed. This method is characterized by using only a small number of eigenvalues ​​to determine the degrees of freedom. Consequently, this method can be designed to be computationally efficient and highly accurate.

[0094] In a preferred embodiment of the invention, the aforementioned feature values ​​are substituted into an imaging model that describes the optical imaging of the coded array on an image sensor, taking into account the position and orientation of the camera relative to the coded array. This model describes the physical relationship and analytical relationship between the aforementioned feature values ​​as input values ​​and the aforementioned three degrees of freedom as output values. Therefore, this relationship leads to the determination of these three degrees of freedom.

[0095] In one possible design of the invention, three equations are defined for these three degrees of freedom, forming a system of equations. It should be emphasized that these three equations represent a possible expression of the analytical relationship between the eigenvalues ​​and the three degrees of freedom; other mathematical or analytical representations are also possible. However, the physical relationships can be expressed in a particularly simple and compact manner using these three equations and / or this system of equations.

[0096] The equations used to determine the first pitch angle phix are, in particular, functions of the first local angular divergence dalphac0, the distance rz between the coded array and the camera, and the second pitch angle phiy.

[0097] In particular, the first pitch angle is determined by the following equation: ,in : The grid distance of the point grid in meters The equations used to determine the second pitch angle phiy are, in particular, functions of the second local angular divergence dalphac1, the distance rz between the coded array and the camera, and the first pitch angle phix.

[0098] In particular, the second pitch angle is determined by the following equation: The equation used to encode the distance rz between the array and the camera is, in particular, the following function: in b: Image distance of the camera in meters (m) : The local grid size of the point grid in the camera image, in [m].

[0099] The column involves two independent variations, and the row involves two options depending on the definition and / or convention of the rotation angle.

[0100] Alternatively or supplementarily, the distance between the coded array and the camera is determined as the average of a variant and / or function using the following equation: In principle, this system of equations can be solved analytically. In a preferred extension of the invention, the system of equations, comprising these three equations, is solved iteratively. Here, initial values ​​are first specified, and then the degrees of freedom are determined in an iterative process within the optimization procedure.

[0101] Preferably, within the scope of this method, an initial region is determined. The initial region has at least three basic symbols, preferably adjacent to each other, arranged at an angle to each other. The connecting lines between the basic symbols of the initial region define two independent principal directions along the point grid in the camera image. For the case where the initial region has exactly three basic symbols, these basic symbols are arranged at an angle to each other, for example. These basic symbols define the coordinate system of the planar point grid, wherein one of the at least three basic symbols forms the origin of the coordinate system. In a preferred extension of the invention, the initial region has nine basic symbols, arranged in a square and / or rectangular pattern. In particular, the initial region has a side length of three basic symbols. The basic symbols of the initial region are valid basic symbols. Valid basic symbols specifically refer to those basic symbols that are selectively defined as valid basic symbols within the scope of this method, or classified as valid basic symbols based on image features, and / or set as valid basic symbols because basic symbols are found and successfully processed to obtain usable processing results. Image structures not classified as valid basic symbols are interpreted as image interference and excluded from further processing.

[0102] In this method, during the search step, starting from at least one valid elementary symbol, other, especially valid elementary symbols, are searched along these main directions.

[0103] When a search is successful, especially when other basic symbols are found, mark those other basic symbols as valid basic symbols.

[0104] The rule is that this search step is performed multiple times. If new, other basic symbols have been found as valid basic symbols in the search step, the search step is selectively performed starting from the basic symbols of the initial region or from the newly found valid basic symbols. In this way, the point grid is gradually supplemented with other valid basic symbols starting from the initial region with valid basic symbols. This search step is repeated until enough valid basic symbols are found, especially enough to enable decoding.

[0105] Based on these valid basic symbols or a selected subset thereof, and especially by decoding these valid basic symbols, the approximate position of at least one basic symbol in the coded array is determined as two degrees of freedom of the camera relative to the coded array.

[0106] In particular, these valid basic symbols or a selected subset thereof are decoded, wherein at least one degree of freedom of the camera relative to the coded array is determined based on the coarse position after decoding and optionally, supplementarily, based on the coarse orientation of the coordinate system of the point grid in the camera image.

[0107] A further consideration here is that the fundamental symbols can be searched and found, for example, using planar image processing functions in the coded array and / or in the camera image. However, finding the fundamental symbols using methods such as blob analysis or pattern-based search (pattern matching) requires frequent access to the pixels of the camera image, which complicates image processing.

[0108] By starting from a search region (which defines the two principal directions of the point grid and thereby defines the point grid in the camera image, especially the coordinate system of the point grid) and searching only along these principal directions, prior knowledge about the structure of the point grid can be used to enable a linear search along these principal directions, rather than a planar search. In practice, it is sufficient to search for other basic symbols along a line in the principal directions starting from the valid basic symbols. Clearly, by reducing the planar search to a line search, the number of pixel accesses and / or the image processing workload are significantly reduced. Therefore, the method according to the invention can be implemented very efficiently.

[0109] In a preferred extension of the invention, corresponding center positions are determined for these basic symbols, particularly the centroid positions of the basic symbols in the camera image. This allows for the determination of multiple intersections in the point grid based on the valid basic symbols.

[0110] The procedure stipulates that the search should begin from the center position of the corresponding valid basic symbol, particularly a linear search along a specified search direction, especially one of these main directions. Finding the basic symbol initially only allows for a rough determination of its position within the point grid and / or camera image; by detecting this center position, the precise position within the point grid and / or camera image is determined, thereby specifying the point grid. Starting from the center position of the basic symbol, especially a linear search, is advantageous because it prevents the unintentional omission of adjacent basic symbols.

[0111] In a preferred extension of the invention, two connection vectors are determined along two independent principal directions of the point grid in the camera image, based on an initial region and / or other valid basic symbols. By determining these connection vectors, the positions of the nearest and / or adjacent basic symbols being searched can be estimated by extrapolation within a framework of linear combinations of known valid basic symbols.

[0112] Particularly preferably, the center position of the basic symbol is determined by shifting the starting point of the basic symbol along these principal directions through multiple intermediate steps until the center position is found to be located at the center of the basic symbol along these principal directions, which is then taken as the center position. By shifting the starting position to this center position, ensuring that the starting position is always centered in these principal directions, it is only necessary to search the boundaries of the basic symbol in these principal directions for each shifting step, and then shift the starting position to the center between the boundaries.

[0113] In the example of the circular basic symbol, starting from the initial position within the basic symbol, a linear search is performed along the first principal direction of the camera image to find the boundary of the basic symbol, and then along the negative first principal direction. Mean-averaging these boundaries yields a more accurate position estimate. This process is repeated in the same way for the second principal direction, and then again for the first principal direction.

[0114] In this way, the shift from the starting position to the center position is computationally efficient due to a series of linear evaluations along these main directions.

[0115] In subsequent steps, the area of ​​the basic symbol in the camera image is determined, whereby this area constitutes the encoded data item of the encoding array. In particular, this area is used for the classification of the basic symbol.

[0116] This data item can be 0 or 1 in binary, for example, where the basic symbol is, for example, a circular region of varying size. Knowing the corresponding center position and the extent of the basic symbol along the principal direction through a shift step, the area of ​​the basic symbol can be inferred from these values ​​in a simple way. Therefore, in a computationally very efficient manner, the center position in the point grid is first determined, and then the area of ​​the basic symbol is determined.

[0117] The area of ​​a basic symbol in a camera image is significantly affected by perspective distortion. For example, this area decreases as the distance from the camera to the basic symbol increases and as the camera's pitch angle increases (elliptic distortion). These effects can lead to misclassification, for example, when similar basic symbols appear in the camera image at significantly different distances from the camera. Therefore, for the purpose of classifying basic symbols, it is preferable to use the normalized area of ​​the basic symbol.

[0118] To calculate the normalized area, a reference area is determined, which suffers approximately the same perspective distortion as the base symbol. A parallelogram is used as the reference area, which is extended from the base symbol's grid position toward adjacent grid positions along the principal axes by two vectors. The reference area is the absolute value of the cross product of these two vectors.

[0119] The normalized area of ​​the basic symbol is specifically calculated as the quotient of the area of ​​the basic symbol and the reference area.

[0120] In a preferred embodiment of the invention, a readout region is defined in the camera image and / or in a planar point grid, and a corresponding two-dimensional basic symbol matrix, particularly a dot matrix, is formed. Each point in the readout region is assigned an entry in the basic symbol matrix, and the position, area, and / or data items of the basic symbols are entered into the basic symbol matrix. Based on the data in the basic symbol matrix, the coarse position of the coordinate system of the point grid can be decoded, and optionally, the coarse direction of the coordinate system of the point grid can be decoded, and / or the coarse position of at least one basic symbol in the encoding array can be determined as these two degrees of freedom.

[0121] In a preferred extension of the invention, at least one initial position is defined in the preparatory step for creating the initial region. The initial position is arbitrarily defined. Preferably, the initial position is located near or adjacent to the incident point. If the initial position happens to be within a basic symbol, this initial position is assumed to be the starting position of the initial basic symbol. If the initial position is outside a basic symbol, adjacent basic symbols are searched along the search ray starting from this initial position as initial basic symbols. Therefore, even in this preparatory step, no computationally intensive planar image processing is performed; instead, basic symbols are searched only along the search ray starting from the initial position. For example, the boundary of the basic symbol can be detected by the contrast change along the search direction in the camera image. Here, at least or exactly 8 or 16 search rays can be used in this method, starting from the initial position and arranged in a 360° pattern with uniform angular steps. A minimum radius of these search rays can be defined, and the next basic symbol is searched starting from this minimum radius. Optionally, a maximum search radius can be specified, wherein if no adjacent basic symbol is found when the maximum search radius is reached, the initial position is discarded and an alternative initial position is selected.

[0122] In subsequent steps, starting from the position of the initial basic symbol, adjacent basic symbols are searched along other search rays, serving as the first auxiliary initial basic symbols. The search can be performed starting from the first auxiliary initial basic symbol using the same distribution of search rays as described above.

[0123] In subsequent steps, starting from the initial basic symbol and the first auxiliary initial basic symbol, a second auxiliary initial basic symbol is searched in a direction angular to the connection between the initial basic symbol and the first auxiliary initial basic symbol (and / or along the first principal direction). However, compared to before, fewer search rays can be used here, especially those distributed over a smaller angular range, thus speeding up the search. Here, the fact that the first three basic symbols are arranged at an angle to each other is utilized, giving prior knowledge about the direction in which the second auxiliary initial basic symbol is searched. For example, only one, two, or three to six search rays are used, distributed around the principal search direction, which is oriented perpendicular to the connection between the initial basic symbol and the first auxiliary initial basic symbol.

[0124] To accomplish this, starting with the initial basic symbol, and based on the connections between the initial basic symbol and the first auxiliary initial basic symbol, as well as the connections between the initial basic symbol and the second auxiliary initial basic symbol, a linear combination of these connections is used to search for all basic symbols directly adjacent to the initial basic symbol along the principal axis or diagonal direction of the point grid. Specifically, the connection vectors and / or principal directions can be derived from the first three basic symbols found. These connection vectors and / or principal directions extend the two-dimensional coordinate system of the point grid. To improve the method, the center position can be determined to define these connection vectors as precisely as possible. The result is an initial region with 3 x 3 valid basic symbols.

[0125] In this search step, neighboring basic symbols are searched starting from the initial region and / or other valid basic symbols found through previous search steps. The method preferably searches for basic symbols at grid points that have at least two valid neighboring points along the main direction or along the diagonal direction. All neighboring points of the grid point are extrapolated to provide independent location estimates, which are then averaged to derive a more accurate location estimate for the grid point.

[0126] By searching only grid points with at least two valid elementary symbols as neighbors, planar stretching of the detected region is facilitated, while avoiding spear-like, dendritic, or spike-like linear stretching. Compared to linear stretching, planar stretching is more robust and tolerant of disturbances in the image. Based on valid elementary symbols, the elementary symbol matrix, especially for the readout region, is determined.

[0127] Specifically, the method may specify that: first, a camera image is recorded; then, at least one degree of freedom of the camera relative to the coded array is determined; and then, for example, the actuators of the automated device are manipulated. For example, this at least one degree of freedom may be output on an optical output device, such as a display. By using this at least one degree of freedom as an actual (IST) value, it can be used for position control and / or adjustment of the actuators of the automated device. A robot equipped with the sensor unit of the automated device may, for example, determine its absolute pose relative to the coded array and output this absolute pose as actual (IST) information, or it may move to other specified positions, wherein the robot continues to orient itself on the coded array with respect to its actual (IST) position.

[0128] Another aspect of the invention relates to a control unit and / or an automation device having the control unit, wherein the control unit is designed to perform the methods described above. Optionally, the control unit includes a camera and / or is connected to the camera via data technology.

[0129] Another subject of the present invention relates to a computer program designed to perform the above-described method when executed on a digital data processing device and / or on the control unit.

[0130] Another subject of the present invention relates to a machine-readable storage medium having the computer program. Attached Figure Description

[0131] Other features, advantages, and effects of the invention will become apparent from the following description of preferred embodiments and the accompanying drawings. In these drawings: Figure 1 A flowchart of an overall method having an embodiment of the method according to the present invention is shown; Figure 2 A schematic diagram of a 3D optical model is shown; Figure 3 A schematic diagram of a 2D optical model is shown; Figure 4 A schematic diagram of a 2D optical model is shown; Figure 5 , Figure 6 , Figure 7 A schematic diagram of the coordinate system is shown; Figure 8 A schematic diagram of the coordinate system of the imaging model is shown; Figure 9 It shows Figure 8 An imaging model with a tilted coding plane; Figure 10 A diagram of the encoding array is shown; Figure 11 A further illustration of the encoding array is shown; Figure 12 A dot grid of an encoded array with two exemplary drawn read regions is shown; Figure 13 The various angular positions of the reading area are shown; Figure 14 An example of a read area is shown; Figure 15 An exemplary structure of a block in an coded array is shown; Figure 16 Other exemplary structures of blocks in an coded array are shown; Figure 17 A flowchart of the validity check is shown; Figure 18 Details of the validity check are shown; Figure 19 An example is shown of a camera image with the identified initial reference and readout area; Figure 20 An example of a basic symbol matrix for an entered data item with a surface is shown; Figure 21 A flowchart for determining the initial baseline is shown; Figure 22 This shows several predefined initial positions in the camera's image field; Figure 23 A diagram illustrating the method for determining the initial baseline is shown; Figure 24 A diagram illustrating the method for determining the initial baseline is shown; Figure 25 The flowchart of the Center_Pos function is shown; Figure 26 A diagram of the Center_Pos function is shown; Figure 27 a, Figure 27 b shows a diagram of the Nearest_dot function; Figure 28 A flowchart is shown for the process of detecting all points / basic symbols in the reading area; Figure 29 a- Figure 29 Figure d illustrates the process for detecting all points / basic symbols in the read area; Figure 30 A diagram illustrating the detection of basic symbols in a distorted mesh is shown. Figure 31 A diagram illustrating the correction is shown; Figure 32A flowchart illustrating the classification of basic symbols is shown; Figure 33 A flowchart for determining the approximate location is shown; Figure 34 a and 34b show the basic symbol matrix with the entered data items and the reading trajectory; Figure 35 A diagram illustrating the decoding at a rough location is shown; Figure 36 A graphical representation of the fitted line for the linear function is shown; Figure 37 A diagram of the line angle is shown; Figure 38 a- Figure 38 c shows a graphical representation of the fitted line and the linear function; Figure 39 A diagram showing the determination of fine position and rotation angle phiz (Z-axis rotation angle) is provided; Figure 40 A flowchart for determining camera distance and pitch angle is shown. Detailed Implementation

[0132] A method is disclosed as an algorithm for accurately determining the absolute 6D position of camera 1 relative to a planar coded array 2 recorded by camera 1, wherein the algorithm obtains camera image 3 as input information.

[0133] Encoding array 2 serves as both an analog and digital scale in two dimensions, namely X and Y. This encoding array consists of symbols arranged in a regular grid. Preferably, the grid is a binary code with two symbols arranged in a square grid: small dots represent the digit 0 and large dots represent 1. The dot type contains digital information, while the dot center contains analog information.

[0134] Camera 1 detects segments of the coded array 2 and transmits camera image 3 to a computer or any data processing device. The algorithm selects a readout region in camera image 3 with the smallest symbol size (e.g., 7 x 7 points) and calculates the position of camera 1 relative to the coded array 2 in six dimensions from it. For this purpose, the method uses digitally encoded position information and the precisely measured center position of all points in the measurement or readout region. These points form a grid, which is distorted due to camera viewpoint and lens distortion.

[0135] The location detection method includes the following steps (see Figure 1 ): Step 100: Read camera image 3 into the computer's working memory. Step 200: Optionally: Perform an initial image quality check based on feature data such as brightness and contrast.

[0136] Step 300: Coarse position assessment along the XY axes, and optionally, coarse angle assessment along the z-axis. Step 310: Search for an initial region of 3x3 points (launch pad) within the read area. Step 320: Detect points in the readout area in the camera image. Step 330: Accurately measure the center position and area of ​​these points. Step 340: Optionally: Mathematically correct lens distortion by adjusting point positions. This converts the curves of the point grid in camera image 3 into straight lines. Step 350: Classify these points based on their area and assign them to binary numbers 0 and 1. Step 360: Read the digital codes along the X and Y axes, and determine the angles along the X and Y directions, and optionally as a coarse angle evaluation along the z-axis. The absolute approximate position of the direction.

[0137] Step 400: Distortion Assessment Step 410: Fit the fitted line (beam) to each row and column of the point grid in the reading area. Step 420: The fitting function is adapted to the first bundle of lines by interpolating all the lines in the rows of the coded grid, and the second fitting function is adapted by interpolating all the lines in the columns of the coded grid. Each bundle of lines is fully described by 6 interpolation parameters (bunch data). The interpolation line between the two measured lines can also be calculated using the interpolation function.

[0138] The 6D camera position is calculated based on eigenvalues ​​obtained from measured point locations, which describe the distorted mesh in the camera image. The position calculation equation is derived from the inverse optics imaging model of camera 1 and the laws of geometric optics: Step 500: Fine position evaluation of the XY axes: The position along the X and Y directions is calculated based on the approximate position of the XY axis and the orientation of the grid relative to the optical axis (image center).

[0139] Step 600: Evaluate the Z-axis angle based on the angle of the (interpolated) line at the center of the image.

[0140] Step 700: Z-axis position and XY-axis angle evaluation Step 710: Derive interpolation eigenvalues ​​based on the bundle data. These eigenvalues ​​describe the perspective distortion mesh at the center of the image. - Grid size of the straight lines in the two straight line bundles - Angular divergence of the straight lines of two straight line bundles The camera position along the Z direction is primarily determined by the grid size of the line at the center of the image.

[0141] Camera angle and It is mainly determined by the angular divergence of these two straight line bundles at the center of the image.

[0142] Iterative algorithms are used to convert parameters Z, and The related system of equations is solved numerically.

[0143] Step 800: Output 6D location information and optional validity information, which is derived from the results of numerous diagnostic functions in the process.

[0144] This algorithm concentrates camera image data into a small amount of data important for localization with minimal computational effort. It improves accuracy through interpolation and is robust to interference from camera images. The following process demonstrates that location (using numerical values ​​as an example) is particularly important: Process Location I: Input data: Camera image (e.g., 200x200 pixels) 40,000 values Process Location II (Step 300): Convert camera image data into point data 2250 values Process Location III (Steps 400 / 500): Convert point data into beam data 90 values Process location IV (step 400 / 500): Convert beam data to bundle data 12 values Process location V (steps 500 / 600 / 700): Convert the bundle data into 6D locations. Six values.

[0145] Process position II - V has the following special characteristics.

[0146] Process Location II: a) By mathematically modeling the perspective-distorted coded mesh in coded array 2, points in camera image 3 can be quickly located. This method is based on the already measured point locations. → Advantages: Fast point finding, eliminating time-consuming searches within the pixel grid. Robust against disturbances in camera images: If a point is not found at its expected location, it is classified as "invalid." Invalid points are not processed further, but generally do not hinder subsequent processes. b) A stack-based algorithm for fast and robust point search in the coded grid of coded array 2. → Advantages: High fault tolerance because the search process does not interrupt when erroneous points are encountered, but instead surrounds them from all sides. Accurate (high number of points found per read region) and robust (by prioritizing the detection of points with multiple valid neighbors). c) The Center_Pos function: A fast method for accurately determining the position and area of ​​points in a camera image at the sub-pixel level. → Advantages: Fast (fewer pixel accesses), accurate (through subpixelation), robust (using dynamic contrast thresholding instead of grayscale thresholding) d) Correct the point position, rather than correct the entire camera image. → Advantages: Fast (corrects only 225 locations in the camera image, instead of all 40,000 pixels in the camera image) e) Locally normalize the point area to the area of ​​the point grid cell. Normalized point areas are used to classify points. → Advantages: Robust reading of digital encoding (point classification can tolerate perspective distortion and changes in the distance between the camera and the points) f) Redundant reading of digital codes, error identification, and partial error correction → Advantages: Low error rate when there is interference in the camera image; can read correctly even with invalid points.

[0147] Process locations III and IV: g) The point locations extracted from camera image 3 are compressed and precisely represented as bundle parameters (only 12 values). → Advantages: Fast (due to small data volume), accurate (through mean-fit and best-fit interpolation in two stages: point to fitted line and fitted line to line bundle), highly tolerant and robust (by excluding invalid points from further processing).

[0148] Process location V: h) A mathematical method for transforming 12 bundle parameters to a 6D position.

[0149] As an advantage, some or all of the following improvements have been implemented: ● Complete 6D position measurement, absolute position information ● Detect all dimensions simultaneously using a single measurement process (in a single camera image). ● High position measurement rate and low latency. This method requires only a small number of computational steps from the computer. Using an embedded computer, a typical measurement rate of approximately 100 Hz to 10000 Hz can be achieved, allowing the sensor to also be used in closed-loop position control systems, for example. This method also achieves high readout speed by minimizing the number of accesses to camera image points (pixels). Time-consuming planar camera image operations are not performed. Furthermore, the amount of data is significantly reduced with each processing step. ● High read security is achieved through redundant reads using dot-coded methods. Position detection is possible even in the presence of interference in the camera's image field, such as partial occlusion, invalid points, or unfavorable lighting conditions. By checking the validity of these processing steps, error states are identified and unreliable position values ​​are prevented from being output. ● High position measurement accuracy across all degrees of freedom. This is achieved through various measures, such as... ○ Average a large number of points, such as more than 100 or 1000 points, in each camera image 3. ○ Classify points based on their validity and exclude invalid points from further processing. ○ Perform sub-pixel level precise positioning of point edges ○ Using a contrast threshold instead of a grayscale threshold provides high robustness to local variations in image brightness. The ○ point is the origin, therefore, center error is almost non-existent when the encoding plane is tilted or rotated. ● In The measurement range is unlimited. ● The measurement range in X and Y is virtually unlimited. For example, a measurement range of 550,000 km x 550,000 km was achieved when using encoding units with 9 x 9 points and a dot grid of 2.5 mm. ● Large angle measurement range , Approximately + / - 50°, with high precision. Extended to 360° through a spatially distributed coded array on the object. ● Use a geometric optics-based inverse imaging model to calculate the camera's position based on camera image data. ● This method does not require active lighting, making it possible to process images recorded in ambient light (e.g., using a smartphone). ● Extremely low error rate. This method determines the validity of location measurements by using various diagnostic methods to check for errors and reasonableness at each processing step. Validity is output along with the location measurements. The goal here is to output only valid measurements for further processing.

[0150] Other advantages are: ● A single measurement sensor (especially designed as a camera 1) detects the position of one or more objects in six dimensions. This reduces installation effort and overall system complexity compared to a system with multiple distributed sensors for detecting each degree of freedom, resulting in a cost advantage. ● By detecting all six degrees of freedom, system components are saved. For example, rotary angle encoders typically need to measure the rotational position of the shaft so that lateral positional deviations of the encoder disk do not cause measurement errors. Using the proposed method, mechanical guidance can be eliminated because the complete 6D positional information of the encoder disk, as an encoder array, is detected simultaneously. Lateral positional deviations of the sensor do not cause angle measurement errors. This reduces the cost and mechanical complexity of the position measurement system. ● A positioning system can be implemented in which the position sensing devices cannot be structurally divided across multiple axes. For example, a suspended planar robot can be implemented that can be positioned in six dimensions without the need for a structural connection between the robot and the underlying stator. ● By simultaneously detecting multiple dimensions, measurement errors that may occur in distributed sensor systems due to different measurement time points are reduced. ● The cost of the sensor is largely independent of the number of degrees of freedom detected. Therefore, this sensor can also be advantageously used in applications requiring fewer than six degrees of freedom. Additional information provided enables additional functions, such as system self-diagnosis or continuous monitoring of operational status (condition monitoring). ● This method is scalable, for example, by changing the grid size of the coding array and adapting the imaging optics to the coding array. The resolution can vary across multiple orders of magnitude, from nanometers (application example: nanometer positioning systems) to meters (m) (application example: automated landing of aircraft or drones at airports, where airports use coding arrays for marking). ● The ratio of resolution to measurement range can span multiple orders of magnitude. For example, combining a displacement measurement system with a resolution of 10 nm with a coded array of 10 m in length results in a resolution-to-measurement-range ratio of 1:10. 9 . ● Sensors employing this method offer high application flexibility because they are easy to install and can be parameterized for specific applications. This is particularly advantageous under frequently changing operating conditions. ● This method can read additional information contained in the location encoding. For example, in addition to location, object identification data can be read. ● In the case of cyclic image recording, this method provides an independent position estimate for each individual image, which does not depend on prior information from previous images. Therefore, the measurement rate corresponds to the image refresh rate.

[0151] This method enables the localization of at least one camera 1 relative to at least one object 8 in six degrees of freedom, wherein at least one coded array is coated on the surface of each object 8. Camera 1 detects the coded arrays 2 on the object 8 and renders these coded arrays, either fully or partially, in camera image 3. Using the proposed method and other common mathematical / technical methods, the 6D position of the object 8 is calculated based on camera image 3.

[0152] The encoding array 2 forms a planar digital encoding scale. This encoding array contains multiple different symbols, preferably circular symbols (dots), which are arranged in a regular, preferably square grid and are capable of positioning in 6 degrees of freedom.

[0153] Camera 1 includes at least ● Imaging sensor elements, typically camera chips, especially image sensors 12. ● An imaging system that images the scale clearly and with high contrast onto the sensor elements (especially lens 9). This imaging system specifically includes a wide-angle lens as lens 9, because a central perspective image is needed to determine all six degrees of freedom. ● An interface for outputting camera image data and / or camera images 3.

[0154] Optionally, the camera system of camera 1 includes: ● Illumination system, especially for illuminating the coded array. This makes camera 1 more independent of ambient lighting conditions. In flash mode, short measurement times are achieved, allowing even fast-moving objects to be detected. For example, light-emitting diodes (LEDs) are used as the light source. ● Ambient light shielding devices, such as apertures or optical filters. Spectral filters can be present in the optical path of an imaging system so that only light from a limited wavelength range reaches the sensor element. Ideally, a monochromatic light-emitting device with the same wavelength characteristics as a color filter is used.

[0155] Additionally, a computer system, serving as a digital data processing device, is required for information processing, for example, designed as a control unit. This computer system acquires digitized camera image data (camera image 3) as input information; at output, it provides the determined 6D position of the identified object, and optionally, an evaluation of the validity of the position measurement.

[0156] In this computer system, the method is implemented as an algorithm. The algorithm is executed in response to an external request, or it is executed cyclically, for example, in fixed time frames, to detect the motion trajectory of an object (tracking).

[0157] The computer system can be implemented as an embedded system, parallel computer, GPU, FPGA, ASIC, or cloud system, for example. A smartphone can also be used as the overall system for location detection, wherein an integrated camera 1, with active illumination if necessary, is used for image recording. The method can be implemented in a smartphone application and executed either embedded in the smartphone or in the cloud.

[0158] Figure 2 A typical apparatus for determining absolute position in six degrees of freedom is shown. Camera 1 records camera images 3 of a planar coded array 2. Based on the perspective-distorted camera images 3, the 6D position of the camera in the coordinate system of the coded array 2 is calculated using the method according to the invention, and this position is used as a position vector. and angle vector Output.

[0159] In the Cartesian coordinate system (X) of the encoding array 2 C Y C Indicates coordinates and These coordinates represent the intersection point 4 of the optical axis 5 of camera 1 and the plane containing the encoding array. Since the optical axis 5 is perpendicular to the plane of image sensor 12, it can be represented by point 7 in the camera image 3.

[0160] Even if the coded array 2 is only displayed at the edge of the image field and not at the position of the optical axis 5, the method will still determine the intersection point (X, Y).

[0161] distance Located on optical axis 5, and extending from intersection point 4 to the optical center of the lens of camera 1. Since optical axis 5 is not always perpendicular to the coded array 2, With (X) C Y C They typically form a non-orthogonal coordinate system. Using angle vectors, position vectors can be... Transform to a Cartesian coordinate system.

[0162] Using this method, multiple simultaneously recorded objects 8 can be located in the camera's image field, even if these objects partially overlap. In order to be able to read the position, it is necessary to be able to identify at least one region in the camera image 3 that is the size of an coded block (e.g., 7x7 points).

[0163] Figure 3 and Figure 4 The optical path of camera 1 is schematically shown. Figure 3 In the image, camera 1 with lens 9 and image sensor 12 can be seen, with its line of sight pointing downwards toward the encoding plane of encoding array 2. Optical axis 5 is drawn as a vertical dotted line, and the focal points of lens 9 are points 10 and 11.

[0164] According to the laws of ray optics, the vector arrow G on the encoding plane of the encoding array 2 is imaged onto the image sensor 12 as vector arrow B. The line of sight 14 intersects the optical axis 5 at a point, which is referred to here as the optical center 13 of the lens 9.

[0165] Figure 4 The complete optical path is shown, with lens 9 simplified to a lens. Ray optics, including the ray theorem and lens equations, forms the basis of the mathematical methods used for positioning. In this case, the following applies: B / G = b / g and 1 / f = 1 / b + 1 / g in: B Image Size b Image distance G Object size g object distance f is the focal length.

[0166] Figure 5 , Figure 6 , Figure 7 The coordinate systems involved are illustrated in the diagram: Through the camera chip's axis (X) I Y I This extends the image sensor 12 into a two-dimensional coordinate system 15. Positions on the image sensor 12 are indicated by image points [Pixels], where the pixel rows and columns of the camera chip are numbered consecutively. In this example, it is assumed that the image sensor 12 has 200 x 200 pixels. Integer X and Y position values ​​for each pixel are derived from the numbering. Real numbers may also appear as camera positions during evaluation through subpixelation, i.e., a method for interpolating grayscale values ​​between adjacent pixels. To convert the unit of position from [Pixels] to [m], the position is multiplied by the distance between adjacent pixels on the camera chip, in units of [m / Pixel]. Pixel distance This is a characteristic of the image sensor 12. Each pixel provides an integer grayscale value.

[0167] The coordinate system 16 (X) of the coding array 2 is extended by the principal axis of the coding array 2. C Y C These principal axes correspond to a square point grid. The coordinate system is defined by the third axis Z. C Expanding to a three-dimensional coordinate system, the third axis is perpendicular to the encoding array 2, where the encoding array 2 is located at Z. C =0. The unit is [Dots], that is, the consecutive numbering of rows and columns of dots, or [m]—multiplied by the dot grid distance in [m / Dot]. Then, the coded array 2 pairs of coordinate systems (X... C Y C The local positions in the ) are uniquely encoded.

[0168] A three-dimensional camera coordinate system (X, Y, Z) 17 is fixedly connected to the camera 1. The origin of this three-dimensional camera coordinate system is located on the optical axis 5 between the image sensor 12 and the encoding array 2, and the distance from the image sensor 12 is twice the image distance b. The optical axis 5 forms the Z-axis, and the camera 1 faces the Z direction.

[0169] Figure 8 The coordinate system of the imaging model corresponds to the camera coordinate system 17 (X, Y, Z). Here, b is the image distance of camera 1. The image plane lies at height (0, 0, 2b) and is parallel to the X / Y plane of the camera coordinate system 17. As a model preview, another "virtual" image plane 18 can be constructed in the (X, Y) plane of coordinate system 17, where the virtual image is a point-symmetric mirror image of the real image on the image sensor 12 with respect to the image center.

[0170] The encoding plane of encoding array 2 lies at height (0, 0, z0), where (z0 < 0). For the camera orientation (0°, 0°, 0°), the encoding plane is parallel to the X / Y plane of camera coordinate system 17, and the axes of coordinate system 16 of encoding array 2 (X... C ,Y C ) and the coordinate system 15 (X) axis of image sensor 12 I Y I They point in the same direction.

[0171] This method determines the 6D position of the camera: ● Translation: ;as well as ● Rotation: .

[0172] The center of rotation for the pitch and rotation angles phix and phy of camera 1 is located at point (0, 0, z0). The roll angle phiz is measured about the optical axis 5.

[0173] Figure 9 An imaging model with a tilted coding plane and coding array 2 is shown. Value It is the distance between the rotation center (0, 0, z0) and the optical center (0, 0, b) of the lens, which makes it applicable to: .

[0174] Starting from the ray theorem and geometric optics, the point (x0, y0) in the encoding plane is imaged onto the point (B) in the image plane. X B Y Mathematically, this can be described as the imaging equation in camera coordinate system 17: ■ Optical center 13 of lens 9: ■ Two-dimensional coordinates of a point or circle in the plane of the encoding array in coordinate system 16 of encoding array 2. in, It is the centroid of the chosen basic symbol, and i and j are integers. ■ Spatial coordinates of a point or circle in camera coordinate system 17: It has a rotation matrix for vectors in three-dimensional space: Image points in the virtual image plane within camera coordinate system 17: ■ From arrive line of sight: ■ The intersection of the line of sight and the virtual image plane 18: ■ Virtual images of dots or circles: ■ For the real image, the negative sign must be omitted: For intersection point 4 with indices (i=0; j=0), obtain the image coordinates in the real image: .

[0175] 2.4.3 Encoding Array This method requires a planar coded scale in the coded array 2, and uses an imaging sensor, such as camera 1, to read a portion of the scale, so that the pose of the sensor and / or camera 1 relative to the scale in up to six spatial directions can be determined based on the image information. Further details regarding the coded array 2 are derived from the applicant's publication DE 10 2016216 221 A1, the contents of which are incorporated herein by reference, particularly concerning the design of the coded array and aspects of decoding and variations.

[0176] The coded scale is coated onto a surface as a coded array 2, designed as a sensor-readable mark. This surface extends substantially in two dimensions, but may also have curvature. For ease of description, it is assumed below that the coded scale is printed onto a flat surface as an optically readable pattern, without limiting the requirements for other marking principles, sensor principles, and curved surfaces.

[0177] The encoding scale of encoding array 2 is formed by arranging different basic symbols in a regular grid. These basic symbols carry two pieces of information: the shape of these basic symbols encodes digital information, and the centroid of these basic symbols marks a specific position on the surface.

[0178] In its simplest case, the numerical information is encoded in a binary number system with a base b=2. Then, only two basic symbols 20 are used, such as small circles and large circles, which represent the values ​​"0" and "1". Their centroids (centers) mark the grid points on the surface.

[0179] The centroid of basic symbol 20 forms a periodic two-dimensional pattern on the scale plane of encoding array 2, for example, in... Figure 10 In the square grid shown, the reference distance between adjacent symbols in the X and Y directions. same.

[0180] By dividing the plane into equally sized, surface-filled blocks 19, basic symbols within a block 19 are combined into a logical unit. Preferably, square blocks 19 are used. Figure 10 A dotted grid with 19 square blocks is shown, each block capable of holding 7 x 7 binary symbols, corresponding to a maximum information content of 49 bits. Figure 10 The lines and squares drawn in the diagram are for illustration purposes only and are not shown in the actual coded array 2. Figure 11 An exemplary coded array 2 without lines is shown, which has basic symbols 20 in block 19.

[0181] The rule arrangement of basic symbol 20 is superimposed on the rule arrangement of block symbol 21, and each block 19 is marked in the same way as block symbol 21. For example, block symbol 21 can be presented by omitting basic symbol 20.

[0182] exist Figure 10 , Figure 11 In this example, block symbols 21 consist of empty spaces in a dotted grid, with each empty space located at the center of each block 19. Block symbols 21 allow the identification of the location of blocks 19 so that the basic symbols 20 can be read in the correct order along the reading path. Because block symbols 21 occupy grid positions, the information content of block 19 is reduced to 48 bits in this example.

[0183] Reading field 22 is an area on the encoding scale of encoding array 2 that has at least the size of a block 19. This reading field is bound to the dot grid, but not to the grid of block 19.

[0184] Figure 12 A dot grid of an encoding array 2 with two exemplary drawn read regions 22 is shown. The position of read region 2 in the coordinate system of the dot grid is defined by the center position 23 of the read region. This position can be defined by a reference distance. The integer step size is used to change the value.

[0185] Furthermore, the reading area 22 has an angular position relative to the coordinate system of the point grid, which can vary in 90° increments within the square grid. Figure 12 In this example, the angular position is visualized by marking a corner of the reading area. Figure 13 The possible angular positions are shown in the diagram. Figure 12 The reading area 22 shown in the example is uniquely described by the following information: ● Read area 22, left side: Position = (4,11); Angle position = 0° ● Read area 22, right side: Position = (12,6); Angle position = 90°.

[0186] The encoding of the encoding array 2 is constructed such that the basic symbols 20 in the reading region 22 contain sufficient information to digitally encode the position (X, Y) and orientation of the reading region 22 in the coordinate system 16 of the encoding array 2. X and Y are based on a reference distance... Indicates (rough location) using integer multiples of the reference distance. The fine position represented by the decimal part and the precise angle represented by the decimal part of 90° are not encoded in a numerical manner; they are determined by the precise positioning of the basic symbol 20 in the camera coordinate system 17.

[0187] Figure 14 An exemplary illustration shows a read area 22 with 15x15 positions for basic symbol 20. This read area provides more information compared to the minimum required read area 22 for a block 19, which is 7x7 symbols. Redundant information is used for error identification and / or error correction.

[0188] Figure 15 An exemplary diagram illustrates the structure of block 19 with a 7x7 grid. Block symbol 21 is located at the center of block 19. The two regions 24 represent X-block regions, in which continuous encoding of positions in the X direction is shown. The X-block region contains 24 grid points, which correspondingly represent 24-bit encodings.

[0189] To read the encoding, these basic symbols 20 are read column by column from left to right and from top to bottom within each column. The reading order (reading trajectory) in the X block region 24 is based on the grid point coordinates (X... C Y C The values ​​in the table are: (0,6); (0,5); (0,4); (1,6); (1,5); (1,4); (2,6); (2,5); (2,4); (3,6); (3,5); (3,4); (3,2); (3,1); (3,0); (4,2); (4,1); (4,0); (5,2); (5,1); (5,0); (6,2); (6,1); (6,0).

[0190] These two regions 25 represent the Y-block regions, where continuous encoding of positions in the Y direction is shown. The reading order corresponds to the reading order of the X-block region 24, but rotated 90° counterclockwise. Therefore, the reading trajectory proceeds line by line from bottom to top and from left to right within each line.

[0191] The encoding (X-encoding) in block region 24 is a subsequence of length t = 24 digits from a total sequence of g digits, where g is much larger than t. The quantity g is large enough that the X-block region 24 of all adjacent blocks 19 along the X direction can be completely filled. In particular, g is greater than fifty, especially greater than one thousand, and especially greater than one million. The contents of the X-block region 24 of adjacent blocks along the Y direction are identical. The total sequence is designed such that each subsequence of t consecutive basic symbols 20, read forward, is contained exactly once in the total sequence, and each subsequence read backward is not contained in the total sequence when read forward.

[0192] The encoding from block region 25 (Y encoding) is represented in reverse, meaning each digit z is replaced by the digit (b-1-z). For binary with b=2, this corresponds to bitwise reversal. The reversed Y encoding is a subsequence of length t=24 bits in the total sequence, where the same total sequence as the X encoding can be used. The reversed Y encoding also appears only once in the total sequence and does not appear in the total sequence when read in reverse. The contents of block region 25 in adjacent blocks 19 along the X direction are identical.

[0193] The overall sequence is designed such that the horizontal sum of the subsequences is less than half the maximum possible horizontal sum of the subsequences. In particular, the horizontal sum of every arbitrary subsequence selected from the overall sequence is less than half the maximum possible horizontal sum of the subsequences. For example, for a number system with cardinality b, the horizontal sum of every subsequence with length t is less than q = t·(b –1) / 2.

[0194] Subsequences of the overall sequence are encoded with coordinate values, such as the X-coordinate of the initial position in the reading process. Similarly, subsequences of the reversed overall sequence are encoded with Y-coordinates, such as the position from which the subsequence was read. The X and Y codes read in block 19 are assigned to the X and Y coordinates of a reference point in block 19, such as the center of the reading area.

[0195] Each read region 22 contains exactly one block symbol 21, t basic symbols from block region X 24, and t basic symbols from block region Y 25. Based on the position of the block symbol 21, the position of the block grid can be deduced, and thus the positions of block regions X and Y 24, 25 within read region 22 and the associated read order can be derived. The basic symbols 20 read in this order yield a sequence of numbers with t positions, which (after reversal if necessary) is a subsequence of the total sequence.

[0196] When the orientation of the read region 22 is unknown, it is initially unknown which of the two axes is the X-axis and which is the Y-axis. This is determined as follows: the horizontal sum of the read codes is calculated and compared with q: the horizontal sum of the X codes is less than q, and the horizontal sum of the Y codes is greater than q. Based on the orientation of the read codes relative to the total sequence (forward or backward), the orientation of the coordinate axes of the read region 22 relative to the coordinate axes of the code array 2 is uniquely deduced.

[0197] In this way, the position and orientation of the reading region 22 in the coordinate system 16 of the encoding array 2 can be determined, the position being represented by an integer step of the grid width, and the orientation by an integer step of 90°.

[0198] Figure 16The left side similarly shows a block 19 with 9 rows and 9 columns. In addition to block areas X and Y 24, 25 representing the position codes of the corresponding axes of coordinate system 16, a block area 26 is provided for additional data. This block area contains (5x5) grid points, with the middle grid point reserved for block symbol 21, leaving 24 grid points, each capable of accommodating a basic symbol 20. In this way, 24 bits of additional information can be presented in each block 19, which does not require positioning and is read in a prescribed order. Figure 16 The right side shows another encoded array 2 with additional data. Here, four block regions 26 are provided, each with 10 grid points for additional data, so that each block 19 has a total of 40 bits of additional information available.

[0199] exist Figure 1 In this paper, the localization algorithm is presented as a sequence of data processing steps, which are then described in detail with reference to embodiments. Standard mathematical and image processing methods, as well as diagnostic and error identification methods, are not described in detail.

[0200] This method has been optimized, particularly for achieving high positioning accuracy and rapid execution. The use of a sequence of steps significantly reduces the amount of data, enabling fast execution.

[0201] In step 100, at process position I, the digital image data is transmitted from camera 1 to the computer as a grayscale matrix. Step 200 is used to roughly check the validity of the image data based on feature values. Other image processing steps may be performed, such as for preparing the image data or for segmenting it into individual coded regions. These steps are not detailed here. For a 200x200 camera image 3, the data size is 40,000 pixels, and therefore 40,000 bytes in terms of grayscale values.

[0202] At this stage of the process, the symbols (points) contained in the image are located and entered into a dot matrix according to their arrangement in a dot-coded grid, which corresponds to the size of a readout area (here: 15x15 points). Each symbol is measured for its position and area. The position is corrected to compensate for lens distortion. The area of ​​the symbol is normalized to eliminate the effects of perspective distortion. Based on the normalized area, the symbol type is classified. This reduces the data volume to 225 points with metadata and thus to 9000 bytes. To also classify unoccupied grid points or erroneously imaged symbols, a local model of the dot-coded grid in the camera image is created based on the positions of successfully identified symbols. Within the readout area, symbols are searched for on all grid points of this model. Due to local occlusion, image errors, or field boundaries, not all symbols can always be identified. The corresponding grid positions are then classified as "invalid."

[0203] In process position III, fitted lines (beams) are adapted to the rows and columns of valid points in readout area 22. Each fitted line is described by the line angle in the image field of camera 1 and its intersection with the X or Y axis of the image field coordinate system. Each readout area 22 provides two bundles of fitted lines, corresponding to the two principal axis directions of the encoding array 2. In this example, each bundle contains up to 15 lines. Thus, the positional data of 15 x 15 = 225 points is reduced to the data of 15 + 15 = 30 fitted lines.

[0204] At flow position IV, each bundle of lines is interpolated using two quadratic polynomials: one polynomial describes the line angles, and the second polynomial describes the axial intersections of the lines in the bundle. The polynomial parameters for the two bundles of lines are represented by 12 real values, which is reduced to one-fifth of the representation of a single line.

[0205] At process position V, an inverse mathematical imaging model is also used to calculate the 6D position of camera 1 based on the polynomial parameters of the two linear bundles. Additionally, the validity of the position value is estimated by evaluating multiple diagnostic results from various steps in the process flow. The 6D camera position and validity are output to the higher-level system in step 800.

[0206] Step 200 – Image Data Inspection / Validity Test Based on statistical feature data, check whether there are any evaluable camera images 3 (see Figure 17 (Flowchart in the example). In this example, the image brightness is estimated. and contrast It then checks whether the results conform to specified limits. If the specified limits are exceeded, further evaluation is interrupted, and the results are classified as invalid.

[0207] To save computation time, only a small number of image points are involved in determining the feature values, for example, in intersection pattern 27 ( Figure 18 The grayscale value at the intersection of the straight line and the circle in the diagram.

[0208] In substep 210, a set of pixels is selected, which are determined according to an arbitrary intersection pattern 27. The pixels at the intersections of the intersection pattern 27 are used. Thus, G i It is the grayscale value of the pixel at position i, where i = 1..n.

[0209] In sub-step 220, the image brightness is estimated. and contrast : In sub-step 230, the image brightness is evaluated using the following conditions. and contrast : or If any of these conditions are met, further evaluation is interrupted, and the result is classified as invalid.

[0210] Detection point data / steps 310 to 350: The objective is to identify, measure, and classify points, which serve as basic symbols 20, within a reading region 22 of a specified size (here: 15x15 grid points). The result is a two-dimensional dot matrix 29 containing the point data, or a matrix of basic symbols, where the indices of matrix 29 are assigned to the rows and columns of the encoding array 2. Feature values ​​are determined for each point.

[0211] The parameter Dot.Typ contains the classification result. If Dot.Typ is positive, the point is evaluated as valid. ● Dot.Typ = 2: Large dot; logical "1" ● Dot.Typ = 1: Dot; logical "0" ● Dot.Typ = 0: Missing point; block symbol 21 If Dot.Typ is negative, the point cannot be uniquely identified.

[0212] In sub-step 310, an initial reference / initial region (Launch Pad) 28 is first searched in the image region and / or readout region 22 of the camera image 3, which has point encoding with encoding array 2. Here, the initial reference / initial region is understood as a region of, for example, 3x3 adjacent points of type 1 or 2.

[0213] Figure 19 An example is shown of a camera image 3 with an identified initial reference 28 and a readout region 22, in which points as basic symbols 20 are searched—starting from the initial reference 28.

[0214] In substep 320, an initial reference 28 is used to construct a local model of a point grid around it. Other points are searched at adjacent grid locations predicted by this model. If successful, these points are precisely measured and recorded as valid points in the point array 29. This process is repeated iteratively, recording more and more valid points around the initial reference 28 until the 15x15 point reading area 22 is fully detected. The search process is robust to local reading errors: if a point cannot be uniquely identified or exceeds the edge of the image field, it is marked as invalid in the point array 29 and excluded from further processing. Figure 20 It shows the basis Figure 19 The results of successful classification of point array 29 in camera image 3.

[0215] In substep 330, the center position and area of ​​these points are measured.

[0216] Sub-step 340 converts the measured positions of all valid points in the reading area 22 into corrected coordinates. This step is used to compensate for distortion errors (barrel distortion) of the wide-angle lens. After correction, points that lie on a straight line in the coded coordinate system 16 also lie on a straight line in the coordinate system of the image sensor 15 in the corrected image. Alternatively, optical correction can be performed using the corresponding lens 9, making sub-step 340 optional.

[0217] Based on the measured data, the points are classified in sub-step 350. Then, in sub-step 360, digital codes are read along two spatial directions, and these codes are converted into integer position information and coarse direction information in 90° increments using a coding table. This completes the coarse position determination from the previous step 300.

[0218] For sub-step 310—Determine the initial baseline exist Figure 21 The flowchart illustrates the process for determining the initial baseline 28.

[0219] Next step 310.1: First, define the initial position 30 of the first point used to search the initial reference 28. Figure 22Multiple predefined initial positions 30 in the image field of camera 1 are shown. The search begins at one of these initial positions. If the process fails (e.g., due to interference in the image field), a second search begins at the second initial position, and so on, until the next step 310.1 can be successfully completed. If all initial positions have been used and the search is unsuccessful, the next step 310.1 is terminated with a negative result.

[0220] Next step 310.2: From the initial position Depart at 30:00 and search for the nearest point. Figure 23 In the middle, search in the location The point at that location is used as the initial basic symbol.

[0221] Next step 310.3: Then, from Start searching for the nearest point, and utilize... It is represented as the first auxiliary initial basic symbol.

[0222] Next step 310.4: Then, form the connection vector. .

[0223] Next step 310.5: From Starting from, in connection with the vector Search for the nearest point in orthogonal directions, and utilize... It is represented as the second auxiliary initial basic symbol.

[0224] Next step 310.6: These three points , and Extend to have an axis and In the oblique coordinate system, where, The origin of the coordinate system.

[0225] Next step 310.7: If the coordinate system is a left-handed coordinate system (condition: Then by exchanging axes and To convert the left-handed coordinate system to a right-handed coordinate system, see [link / reference]. Figure 24 Left side.

[0226] Next step 310.8: via vector and Linear combinations, estimate all eight with The positions of adjacent points ( Figure 24 (Second from the left) and use it as the initial region 28 for searching and measuring points. As a result, an initial baseline 28 of 3x3 points is available ( Figure 24 (Second from the right, on the right side).

[0227] Next step 310.9: If no point is found in this process, assume it is block symbol 21 (missing point).

[0228] Next, in the next step 310.10, It was shifted one point in the opposite direction to the missing point, and then re-searched. and Let's continue the process. During the second search, we can assume that all 3x3 points have been identified because the next block symbol 21 is far from a block 19 (here: 7 points), and the initial base size is only 3x3 points.

[0229] Subsequently, a next step 310.11 for error identification can be executed, wherein, when an error is identified, a new initial position 30 is used in the next step 310.12, and the process is repeated starting from the next step 310.2.

[0230] As part of the above process, the challenge lies in identifying points in the image field of camera 1 from the estimated initial position and measuring the position and area of ​​those points with sub-pixel accuracy. To reduce measurement time, this process must be performed very quickly, as a total of 225 points must be identified in the readout area 22.

[0231] Therefore, the following Center_Pos function is used to optimize this by specifying the minimum number of pixel accesses. Figure 25 A flowchart is shown.

[0232] When calling the Center_Pos function, the point is passed. The estimated center location. If If the result is outside the search area, the search is interrupted and a negative result is returned. Otherwise, perform steps a)–h), see also: Figure 26 a - h: a) Using the two-dimensional gradient method based on grayscale values, according to To calculate the starting point of optimization This starting point is closer to the center of the point. For gradient calculation, only the starting point is needed. The surrounding 4 pixels are accessed. b) From the optimization point Starting from the point, edge recognition methods are used to search for the edges of the point in the +X and -X directions. The distance to the right edge is The distance to the left edge is . c) The optimization point is obtained by centering the center relative to the left and right edges on the X-axis. . d) From point Starting from the point, edge recognition methods are used to search for the edges of the point in the +Y and -Y directions. The distance to the top edge is The distance to the bottom edge is . e) The optimization point is obtained by centering the points relative to the top and bottom edges on the Y-axis. . f) From point Starting from there, we again use edge recognition methods in the +X and -X directions to search for the edges of the points. The distance to the right edge is The distance to the left edge is . g) Centered relative to the left and right edges on the X-axis. Optimization points are derived from this. The optimized point is output as the center position of the point with sub-pixel accuracy. h) The area of ​​a point is calculated as the area of ​​its circumscribed rectangle, multiplied by a factor used to convert the area of ​​a square to the area of ​​a circle or ellipse. : As part of the above process, another challenge lies in finding the position relative to the initial location in camera image 3. The nearest point. This task can be solved using the Nearest_dot function described below. This function obtains the input data. initial position Minimum search radius Maximum search radius ND Boolean parameter (ND = "neighbor dot(neighbor points)"): If the initial position is within a point and ND = false, then output that point at the initial position. If ND = true, then ignore that point at the initial position and output the nearest neighbor.

[0233] From the starting point Starting from this point, the function scans the image field along 16 search rays 31 at an angular distance of 22.5°. Figure 27 a, Figure 27 b). The radii of all search rays 31 gradually increase, from the radius... Start, until the maximum radius until.

[0234] If ND = false ( Figure 27 a), then it searches not only the edges leading to the point but also the edges emanating from the point. If ND = true ( Figure 27 (b) then only searches for edges leading to the point. Once an edge is found, the search is terminated. This edge is located at position... This location is the closest point searched. From Starting from this point, use the Center_Dot function to determine and output the center position and area of ​​the found point.

[0235] Sub-step 320 – Detect points in the reading area Figure 28 The flowchart illustrates the process for detecting all points in the reading area 22. A previously determined initial reference 28, for example, 3x3 points, is used as input information. Furthermore, step 330 integrates precise measurements of the center position and area of ​​the points into sub-step 320.

[0236] In the first lower-level step 320.1, the center position 32 of the reading region 22 is defined. This center position is selected such that the reading region 22 has the largest possible overlap with the image region in which the encoding is presented, so that as many valid points as possible can be detected. Typically, this is the case if the center of the reading region 22 is located at the center of the encoding region. If the encoding region occupies the entire image region, as in the current example, the center position 32 of the reading region is as close as possible to the image center (see [link to previous example]). Figure 29 a).

[0237] At the outset of this method, only 3x3 points of the initial baseline 28 are known. All other points in the reading region 22 remain unknown; that is, their precise locations and areas have not yet been measured. The goal of this method is to progressively measure and classify the unknown points.

[0238] Next step 320.2: For each known point, determine the vector. and ( Figure 30 These vectors point along principal directions 1 and 2 to their nearest neighbors. These vectors extend into an oblique coordinate system due to perspective distortion. Through these vectors... and A linear combination of these values ​​can be used to estimate the grid positions of unknown neighboring points. By extrapolating and averaging the estimates of multiple valid neighboring points, the estimation of the unknown point's position can be improved.

[0239] The more valid neighbors an unknown point has, the better its location can be estimated. To increase the reliability of the reading process, only unknown points with at least two valid neighbors are checked. For each unknown point, the number of valid neighbors is counted. This number can be between 0 and 8; see [link to relevant documentation]. Figure 29 c): The midpoint has 8 adjacent points (dark).

[0240] At the start of the search, only the initial 3x3 points of the baseline are valid. Figure 29 b) highlights a single unknown point with two or more valid neighboring points. Figure 29 d) This schematically illustrates an unknown point with two valid neighbors 20a and three valid neighbors 20b. All other unknown points (20c) do not yet have valid neighbors.

[0241] The algorithm is based on a stack, in which, in the next step 320.3, all unknown points with at least two valid adjacent points are entered into the stack.

[0242] The stack is processed in a circular manner: ● Next step 320.3: Remove the topmost entry from the stack as an unknown point and process it as follows. ● Next step 320.4: Estimate the location of the unknown point by averaging the extrapolated positions of all valid neighboring points. If the unknown point is outside the expected reading area 22, it is not further processed. If the unknown point is within the expected reading area 22, the Center_Pos function is used to identify and measure the unknown point at the estimated location. ● Next step 320.5: If the unknown point cannot be identified, it is classified as a defect (next step 320.6), and the next unknown point is removed from the stack. ● Step 330 / Next Step 320.7: If a point is identified, use the Center_Pos function to measure its position and area, and record this point as a new valid point in point matrix 29. For this point, also calculate and record the vector. and . ● Next step 320.8: For all 8 neighboring points of the new valid point, increase the number of valid neighboring points by 1. Thus, if an unknown point has two or more valid neighboring points, the unknown point is placed on the stack for measurement. ● Next step 320.9: Repeat this process until the stack is empty. At this point, all points in the read region have been processed. ● Next step 320.10: If not enough points are found for localization, the reading process is evaluated as invalid.

[0243] For sub-step 340—Correction Using a wide-angle lens for imaging may result in barrel distortion in the image field. A line containing points that lie on a straight line within the encoding plane of encoding array 2 may lie on a curve in the camera image 3.

[0244] This curvature can be eliminated mathematically, i.e., through correction. Due to computation time constraints, this correction is only applied to the center position 32 of the point, and not to all image points in the input image. Figure 31 An example is shown of an uncorrected mesh (33) and a corrected mesh (34).

[0245] Distortion Center Located at the intersection of optical axis 5 and the camera chip of image sensor 12, this distortion center is typically the center of the camera chip when camera 1 is perfectly assembled. Otherwise, this center is determined through a one-time calibration process of camera 1. The position within the image field of camera 3 is then determined using a quadratic polynomial. Converted to corrected position : in Relative to the position of the distortion center Distance from the center of the distortion This distortion only changes the vector. The length of the vector is not changed without altering its direction. Polynomial parameters Specific to lens 9, and adapted to this lens during calibration. Alternatively, a distortion-free lens 9 can be used.

[0246] For sub-step 350—classifying points Small and large dots represent the values ​​"0" and "1" in the binary number system. These dots are classified based on a threshold of their normalized area. Figure 32 The process is illustrated in the flowchart.

[0247] Perspective distortion has a significant impact on the area of ​​points in the image field. For example, points that are farther away appear smaller, and circular points on an inclined plane appear as ellipses.

[0248] Next step 350.1: To compensate for these effects, the area of ​​the points is normalized relative to the area of ​​the point elements at those points. A point element should be understood as a vector at the location of the point. and That is, the parallelogram extended by the distance vector from the point to the points adjacent to it along the main axis 1 and 2.

[0249] Figure 30 This illustrates perspective distortion points in a distorted mesh, which is locally composed of vectors. and Extend outwards. The area of ​​this unit is... .

[0250] The normalized area of ​​a point is calculated as follows: .

[0251] Next step 350.2: Using statistical methods, calculate the normalized area of ​​all points. Determine the threshold.

[0252] Next step 350.3: Then, classify all points: ● Missing point: Type 0 ● Normalized area less than the threshold: Type 1; otherwise: Type 2.

[0253] Next step 350.3: If an error occurs, the status is set to invalid; if no error occurs, the status is set to valid.

[0254] For sub-step 360—reading the encoding After classifying these points, dot matrix 29 contains all the information needed to determine the 6D location.

[0255] Dot 29 contains the following information: .

[0256] exist Figure 33 The flowchart for this is shown. The approximate location of the data in the X and Y directions is determined by reading the X and Y bit strings in the dot matrix.

[0257] Next step 360.1: Block symbol 21 "empty point" (type 0) is used as a reference point to identify the position of the bit string in dot matrix 29. If multiple entries of type 0 are contained, these entries can be checked for rationality with each other because the "empty point" will be repeated regularly in the block grid (here: 7x7 dots).

[0258] Next step 360.2: Starting from block symbol 21, determine the position of the read trajectory for the code on the two axes of the encoding plane. Initially, it is unknown which sequence in the sequence is assigned to the X-axis and which sequence is assigned to the Y-axis, or in which direction (forward or backward) the code is read. The code is read along the read trajectory and stored as Code_0 and Code_1. Figure 34a) An example shows the point types entered in the dot matrix. Block symbols (type 0) are marked with a gray background. Figure 34 In b), the read trajectories of the two axes are additionally drawn. The read trajectories of the two axes are rotated 90° relative to each other, with block symbol 21 forming the center of rotation.

[0259] Since the reading area (15x15 points) is much larger than the block (7x7), the code can be read redundantly: a copy of Code_0 is located in the block (35a) adjacent to it along the j direction, and a copy of Code_1 is located in the block (36a) adjacent to it along the i direction.

[0260] Furthermore, there is redundancy in the length of the readable code. A 24-bit code length in each direction, i.e., the content of a 7x7 block, is sufficient for positioning. However, a larger read area with 15x15 points provides 51 or 54 bits in each direction. This redundancy information is used to identify and correct erroneous bits read in Code_0 and Code_1.

[0261] In this example, the following encoding is read: ● Code_0: 111.111.011.111.011.111.111.111.111.111.111.011.011.111.111.111.111.111.111 ● Code_1: 000.000.000.101.000.001.000.000.000.000.100.110.000.000.000.000.000.000.000.

[0262] Next step 360.3: From Code_0, select any segment with t=24 consecutive bits and check its integrity.

[0263] Next step 360.4: Then, calculate the horizontal sum of Code_0. If the horizontal sum is less than t / 2 = 12, then Code_0 is X-encoded and Code_1 is Y-encoded. If the horizontal sum is greater than t / 2, then Code_0 is Y-encoded and Code_1 is X-encoded. The Y-encoded code is reversed. The X-encoded code remains unchanged.

[0264] Next step 360.5: Then, using a fault-tolerant string search, search for Code_0 and Code_1 in the encoding table. The bit position with the highest matching degree is output as the search result; if the deviation is too large, the search result is evaluated as invalid. To transform Code_0 and Code_1 into integer position coordinates (X, Y), the encoded bit positions are converted into spatial positions according to the encoding structure, as exemplarily in... Figure 35 As shown in the diagram: - Select any segment with t=24 consecutive bits and check its integrity: 000100000000001010100000 This segment is searched in the forward direction in the encoding table and found at position 37, according to the underscore: 0101010100000000100000000001000100000 000100000000001010100000 0001000 000001000101000000001 ... - Each X point is assigned a position within the encoding table. - The position corresponds to the starting point in the read order within the block. - Each Y point is assigned a bit position within the encoding table.

[0265] 360.6 Validity Check according to Figure 35 This corresponds to the point position Xc=10=Pos0. Similarly, the Yc value of Pos1 is determined. For these two codes, the reading directions (Dir_0 and Dir_1) are also determined by searching the code in the encoding table in two directions: 0 = found in the forward direction, 1 = found in the reverse direction. Based on the reading directions of Code_0 and Code_1, the approximate direction of the reading window 22 is determined in 90° steps according to the following table.

[0266] The result of substep 360 is: read the approximate integer position of window 22. And direction are available.

[0267] Step 400 – Determine Beam Data To determine the precise location, the corrected position data of the valid points in the dot matrix 29 are evaluated. Figure 36 An example is shown of the points in the readout area of ​​the camera image after correction.

[0268] Step 410: Adjust the fitted line To this end, the fitted lines (beams) are adapted to the rows and columns of valid points in the reading region 22. Using common error minimization mathematical methods, the fitted lines are calculated based on the position data of the valid points in the corresponding rows or columns. Invalid points are excluded from this method.

[0269] Bunch Each fitted straight line Described by the following parameters ( Figure 37 ): With X I Intersection of axes With Y I Intersection of axes The angle of line i in the image field.

[0270] The fitted lines, which are fitted to the points, form the first line bundle. 36, where the line is parallel to Y. C Axis. The straight lines of the point sequence form a second bundle of lines. 37, where the line is parallel to X. C The axis. Thus, the positional data of 15 x 15 = 225 points is reduced to the data of 15 + 15 = 30 fitted lines.

[0271] Interpolation also has the following effect: it averages the point positions on each line, making these fitted lines robust to individual fluctuations at each point position. This improves the stability and accuracy of the 6D position values.

[0272] exist Figure 36 The figure shows the fitted straight line with a reading region of 15x15 points, i.e., as 36 from The straight line and as 37 from A straight line.

[0273] Step 420 – Determine Bunch Data In this step, a quadratic polynomial is used to fit the three linear parameters of lines 36 and 37 in each bundle. , , Interpolation is performed. This interpolation method only considers valid lines. Invalid lines are excluded from the bundle interpolation because too few valid points are available for its interpolation. To do this, a common mathematical method is used to perform weighted interpolation, which is based on minimizing the squared error. Valid lines are evaluated with a weight of 1.0, while invalid lines are evaluated with a weight of 0.0 and are therefore masked.

[0274] Interpolation function methods: Here, i is the consecutive number (line index) of the line in the corresponding line bundle.

[0275] A general method for linear parametric interpolation: in , or .

[0276] To mathematically uniquely describe a straight line, the intersection of the two axes is used. or Only one of the angles in the line. One indication is sufficient. Ideally, the indication should point where the line intersects the axis that is as perpendicular as possible to the line: for a straight line, preferably the intersection with the Y-axis; for a steep line, the intersection with the X-axis. Therefore, the following method should be applied: ● In bundle 1, select a reference line located near the center 32 of reading area 22. ● If the angle of the reference line is within the range ]or ]Inside( Figure 38 a) in X I If the sector on the axis is a straight line, then the line is a straight line. Next, for all lines in bundle 0 (36), specify the intersection point with the X-axis. And for all lines in bundle 1 (37), specify the intersection point with the Y-axis. ( Figure 38 b) Set rot_status as a flag to 0. ● If the angle of the reference line is outside the above range ( Figure 38 a) in Y I If the sector on the axis is a steep line, then the line is a steep line. Next, for all lines in bundle 0 (36), specify the intersection point with the Y-axis. And for all lines in bundle 1 (37), specify the intersection point with the X-axis. ( Figure 38 c) Set rot_status as a flag to 1.

[0277] This step further reduces the amount of data: a bundle of lines is described by only 6 parameters, with three parameters specifically for... And the three parameters are for one of the axis intersection points, i.e. or In the case of two straight lines, there are 12 parameters and the rot_status flag.

[0278] Thus, the two straight lines 36 and 37 are always described by 12 parameters and the Boolean variable rot_status, regardless of the number of lines or the size of the measurement area. Based on these parameters, the 6D position is calculated in the next step.

[0279] The effect of interpolation is also that it averages all lines and therefore all points in each bundle, making the interpolated values ​​robust to individual fluctuations in each line or point. This improves the stability and accuracy of the 6D position values.

[0280] Step 6 – Calculate the 6D camera position Based on the 12 interpolation parameters of the linear bundle (Table 9.1), calculate the 6 coordinates of the camera position in the following order: ● Sub-step 500: Position and ● Sub-step 710: Calculate the eigenvalues ​​of the distorted mesh ● Sub-step 600: Camera angle ● Sub-step 700: Position ● Sub-step 700: Camera angle and ● Sub-step 700: Used for solving , and Iterative algorithm for system of equations The calculations involve deriving eigenvalues ​​that describe the measured grid at position 5 on the optical axis. These eigenvalues ​​are correlated with the 6D camera position via a system of equations derived from the imaging model and the laws of ray optics.

[0281] The camera position is determined by solving this system of equations.

[0282] Additionally, the validity of the location value is estimated by evaluating multiple diagnostic results from various steps in the procedural flow.

[0283] For sub-step 500—calculating camera position and The camera positions on X and Y were calculated as coarse integer positions. The sum of the decimal parts of the real numbers [0..1] (fine positions), multiplied by the point grid distance: Unit: [m] Unit: [m] in The integer positions of the read region in units of [Dots], as determined in step 300. : Read the real decimal part of the region in units of [Dot]. : Point grid distance in [m / Dot].

[0284] The reference point used to measure the camera position is the intersection point 4 of the optical axis 5 and the encoding device 2, which, in the camera image 3, is the intersection point of the optical axis 5 and the image sensor 12. Figure 2 Imaging of point (4) or intersection point 4; Figure 39 Image center (4). This image center IC (4) The image center is determined by the pose of lens 9 relative to the camera chip, and it does not necessarily have to be the same as the center of the camera chip. This image center is determined during calibration and serves as a constant two-dimensional vector. It is stored in the program. The decimal part... and Calculate the intersection point of the interpolation line and the image center 4. For the two ray beams k = 0 and k = 1 (36, 37), apply the interpolation equation for the axis intersection point: ;in or It depends on rot_status.

[0285] In image 39, two straight bundles are shown. and The center line.

[0286] Applicable to: ■ For rot_status = 0: ■ For rot_status = 1: By appropriately selecting interpolation parameters The intersection point of the two center lines is shifted to the center of the image. This is satisfied by the following equation: ■ For linear beam 0: ■ For linear beam 1: ■ Among them The solution to the equation yields the desired decimal part of the line index. and : ■ ■ .

[0287] For step 710—calculating the eigenvalues ​​of the distorted mesh Obtain other eigenvalues ​​characterizing the distorted mesh from the interpolation equation. Fine-grained location... and Substitute these values ​​into the interpolation equation to obtain the feature values ​​at the center of the image. These feature values ​​are crucial for determining the dimension. , , and The position above is necessary.

[0288] a) Grid size at the axial intersection of the lines in two straight line bundles: , . By interpolating the intersection points of the axes Indexing of the dimensionless line at the center of the image Differentiation is performed to determine the mesh size: against in b) Line angle at the center of the image: , (See) Figure 37 ), From the interpolation function of the line angle, we can derive: against c) Angular divergence of the straight line bundle at the center of the image: , . This should be understood as the angular difference between adjacent lines near the image center. This angular difference is calculated using an interpolation function that indexes the dimensionless line at the image center. By taking the derivative, we can obtain: against .

[0289] For sub-step 600—calculating the camera angle The line angle is determined based on the imaging equation derived in 2.4.2. : ;in Distance between the encoding plane and the optical center By image coordinates and For the coded grid coordinates By taking the derivative, we obtain Straight beams in camera images The slope of the straight line is: Considering The slope at the center of the point, and obtain . yes The slope of the straight line at the center of the image. This slope corresponds to the feature value from 6.2 b). ,Right now .

[0290] This is obtained for the camera angle. The equation: .

[0291] For sub-step 700—calculating camera position Based on two straight beams axial intersection The grid distance is used to determine the camera position. Since there are two bundles of lines in each image, two can be determined in principle. Location. For In this case, four values ​​are obtained. ,in .against Figure 9 The imaging equations are applicable to: ;in Distance between the encoding plane and the optical center For example, for rot_status = 0, based on in Measured grid distance at the intersection of axes To determine the value .

[0292] exist Suitable for use on shafts: From the imaging equation, we can derive By substituting this item The equation is obtained. Grid distance Corresponding to in derivative at point ;in Line index in units of [dots]. From this, we can draw the following conclusions. exist ,get And for Solve the following: By substituting matrix elements ,get The equation: Calculate the position of rot_status=1 in the same way. The method is as follows: according to To deduce the position ,against The method is as follows: ;against The results are summarized in Table 10.1.

[0293] Table: Camera position values The calculation.

[0294] By introducing angles The equation can be summarized as follows: To output only one value for each image. Calculate the average of the two z values: .

[0295] For sub-step 700—calculate the camera angle , Based on the two straight beams at the center of the image angular divergence To derive the angle , In step 600, the straight beam is... slope of the straight line Derived as a linear index Functions: From this, calculate the line angle. And the angle of the line The [dots] symbol is used to differentiate in order to obtain the angular divergence: ■ In the center of the image Angular divergence is ■ Angular divergence as measured By combining the equations, we arrive at the following conclusion: The equation: In the same way, according to The slope of the straight line in To obtain the angle Similar to the calculation in 600, to obtain ■ Angle right Perform differentiation and obtain the angular divergence. ■ In the center of the image Angular divergence is ■ Angular divergence as measured By combining the equations, we arrive at the following conclusion: The equation: .

[0296] For sub-step 700—iteratively calculating the camera position Z and camera angle. , In function , There is a mutual dependency between Z and Z: Therefore, these equations are solved using an iterative process. With each iteration, the camera position Z and camera angle... and The accuracy will be improved. Figure 40 The algorithm is shown in the flowchart.

[0297] Next step 700.1: Calculate the real decimal part of the read region position in units of [Dot]. Next step 700.2: Calculate position r x and r y Next step 700.3: Calculate camera rotation / camera angle Next step 700.4: Initialize camera angles phix = 0; phiy = 0 Next step 700.5: Initialize the iteration counter, for example, initialize it to 4. Next step 700.6: Calculate r Z0k and r Z1k And finally calculate r Z =(r Z0k + rZ1k ) / 2 Next step 700.7: Calculate camera angles phix and phiy Next step 700.8: Counter = 0 Recalculate, otherwise interrupt the iteration. Next step 700.9: Error query.

[0298] The algorithm converges quickly, and the results become sufficiently stable after approximately four iterations. The final output is as follows: ● Camera position ● Camera angle Validity information.

[0299] List of reference numerals in the attached diagram: 1 camera 2. Encoding Array 3 Images 4. The intersection of optical axis 5 and encoding array 2; the point of incidence. 5 optical axes 6 empty 7. Intersection of the optical axis and the image sensor 8. An object with at least one encoded array 2 9 lenses 10 Lens 9 Focus 11. Focus of lens 9 12 Image Sensors 13. Optical center of lens 9 14. Line of sight 15. The coordinate system of image sensor 12 in the plane of image sensor 12 (X... I Y I ) 16. The coordinate system (X) of the coding array 2 in the plane of coding array 2. C Y C ) 17. The coordinate system of camera 1 (X, Y, Z) 18 Virtual Image Plane Block 19 20 Basic Symbols 21 block symbols 22 Reading Area 23. Central position 24 X Block Area 25 Y Block Area 26 Block areas for additional data 27. Intersection Pattern 28. Initial Area / Initial Reference Point / Launch Pad 29-dot matrix 30 Initial position 31 Search Rays 32. Read the center position of region 22 33 Uncorrected point grid 34 Corrected point grid 35. Reading trajectory along the first encoding direction 36. Reading trajectory along the second encoding direction 37 First Straight Beam / bunch0 38 Second straight beam / bunch1.

Claims

1. A method for determining at least one degree of freedom of a camera (1) relative to an coded array (2) based on camera images of the camera (1), in, The encoding array (1) has a dot grid with a plurality of basic symbols (20), wherein, in particular, the coarse positions of the basic symbols (20) are encoded in the encoding array (2), wherein the basic symbols (20) define a first principal direction along the dot grid and a second principal direction independent of the first principal direction. The basic symbol matrix is ​​determined based on the camera image, wherein the positions of the basic symbols (20) in the camera image are entered into the basic symbol matrix. Based on the basic symbol matrix, a first line function with a first functional variable is derived in the coordinate system of the camera image, wherein the first line is parallel to the first principal direction in the coordinate system of the encoding array (2), wherein the first line is shifted parallel to the second principal direction in the coordinate system of the encoding array (2) by changing the first functional variable, and / or a second line function with a second functional variable is derived in the coordinate system of the camera image, wherein the second line is parallel to the second principal direction in the coordinate system of the encoding array (2), wherein the second line is shifted parallel to the first principal direction in the coordinate system of the encoding array (2) by changing the second functional variable. The at least one degree of freedom of the camera (1) is determined based on at least one of these linear functions.

2. The method according to claim 1, characterized in that, The first straight line function is formed based on at least two rows of the point grid along the first main direction; and / or the second straight line function is formed based on at least two columns of the point grid along the second main direction.

3. The method according to claim 1 or 2, characterized in that, For rows along the first main direction, first fitted straight lines are formed, wherein the first straight line function is formed based on multiple first fitted straight lines; and / or for columns along the second main direction, second fitted straight lines are formed, wherein the second straight line function is formed based on multiple second fitted straight lines.

4. The method according to claim 3, characterized in that, The first function argument is designed as an integer count of the row, and / or the second function argument is designed as an integer count of the column.

5. The method according to claim 3 or 4, characterized in that, The fitted straight line is described by its line angle with the axes of the camera image coordinate system and at least one intersection point with the axes.

6. The method according to any one of the preceding claims, characterized in that, The linear function is formed by a combination of a line angle function related to the independent variable of the linear function and an axis intersection function related to the axis intersection point of the independent variable of the linear function.

7. The method according to claim 6, characterized in that, The line angle function is designed as a quadratic polynomial, and / or the axis intersection function is designed as a quadratic polynomial.

8. The method according to any one of the preceding claims, characterized in that, Based on the line angle between the line and the coordinate system, select the axis intersection point of the coordinate system with the smaller angle between the perpendicular line to the corresponding coordinate axis, wherein data items are assigned to the line function, and the data items encode the selected coordinate axis.

9. The method according to any one of the preceding claims, characterized in that, Based on the basic symbols (20) of the basic symbol matrix or a subset thereof, determine the approximate position of at least one of the basic symbols in the coding array; and based on the first linear function and the second linear function, determine the fine position of the camera relative to the two degrees of freedom of the coding array.

10. The method according to claim 9, characterized in that, Reference points are arranged in the camera image, the reference points having a first axis intersection function of a first straight line in the coordinate system of the camera image, wherein the first straight line is in the coordinate system (X) of the encoding array. C Y C The first axis intersection function has a first function independent variable, wherein, by changing the first function independent variable, the first straight line is oriented in the coordinate system (X) of the coding array parallel to the first principal direction (2), and the first axis intersection function has a first function independent variable, wherein, by changing the first function independent variable, the first straight line is oriented in the coordinate system (X) of the coding array. C Y C The image is shifted parallel to the second principal direction, wherein the first axis intersection function determines the coordinate system (X) of the camera image based on the first function's independent variable. I Y I The first axis of the first axis intersects the reference point, wherein the first axis passes through the reference point, and the reference point has a second axis intersection function of the second straight line in the coordinate system of the camera image, wherein the second straight line is in the coordinate system (X) of the encoding array. C Y C The second line is parallel to the second principal direction in the coordinate system (X) of the encoding array, and the second axis intersection function has a second function independent variable, wherein, by changing the second function independent variable, the second line is made to be parallel to the second principal direction in the coordinate system (X) of the encoding array. C Y C The second axis intersection function is shifted parallel to the first principal direction along the coordinate system (X) of the camera image based on the second function's independent variable. I Y I The second axis of the second axis intersects the reference point, wherein the second axis passes through the reference point, wherein, based on these axis intersection functions, the first and second function independent variables are determined such that the reference point forms the first and second axis intersections, wherein, based on the first and second function independent variables and the approximate position, the coordinate system (X) of the reference point in the coding array (2) is determined. C Y C The fine position in the plane of the coded array (2) is considered as at least one additional degree of freedom.

11. The method according to any one of the preceding claims, characterized in that, The first function independent variable and / or the second function independent variable are determined such that the first line and / or the second line intersects the reference point, wherein, based on the first line and / or the second line, the rotation angle (phiz) of the camera about the optical axis is derived as the degree of freedom of the camera (1) relative to the coded array (2).

12. The method according to any one of the preceding claims, characterized in that, in, Arrange reference points or the reference points in the camera image. The reference point is designed to be the intersection of the optical axis of the camera and the image sensor and / or the camera image. Based on the camera image, the following values ​​are determined as feature values ​​at the reference point: - The rotation angle (phiz) of the camera about the optical axis of the camera or the rotation angle; - The first local grid size (gc0) of the point grid in the camera image along the first principal direction and / or the second local grid size (gc1) of the point grid in the camera image along the second principal direction; - The first local angular divergence (dalphac0) of the point grid in the camera image along the first principal direction, wherein the first local angular divergence (dalphac0) describes the difference angle between two adjacent straight lines along the first principal direction in the point grid; - The second local angular divergence (dalphac1) of the point grid in the camera image along the second principal direction, wherein the second local angular divergence (dalphac1) describes the difference angle between two adjacent straight lines along the second principal direction in the point grid; Based on the feature values, the camera is determined to have the following three additional degrees of freedom relative to the coded array: - The distance (rz) between the coded array and the camera; - The optical axis has two independent pitch angles (phix; phy) relative to the coding array.

13. The method according to any one of the preceding claims, characterized in that, An initial region (28) with basic symbols (20) is determined, wherein the initial region (28) has at least three basic symbols (20), wherein two independent principal directions along the point grid in the camera image are estimated by means of the basic symbols (20) of the initial region (28), wherein the basic symbols (20) of the initial region form valid basic symbols, wherein in the search step, starting from at least one valid basic symbol (20), other basic symbols (20) are searched along at least one of these principal directions, and wherein, when the search is successful, the other basic symbols (20) are marked as valid basic symbols (20), wherein the search step is performed multiple times, wherein the basic symbol matrix is ​​determined based on the valid basic symbols (20) or a subset thereof.

14. An electronic control unit or an automation device having said electronic control unit, wherein, The control unit is designed, in terms of programming and / or circuitry, to perform the method according to any one of the preceding claims.

15. A computer program, wherein, The computer program is designed to perform the method according to any one of claims 1 to 13 when executed on a computer or on a control unit according to claim 14.

16. A machine-readable storage medium, wherein, The computer program according to claim 15 is stored on the storage medium.

Images