A method for identifying at least one degree of freedom of a camera, a computer program, a machine-readable storage medium, and an electronic control unit or automation device.

Abstract

A method for identifying at least one degree of freedom of camera (1) relative to a code array (2) from a camera image of camera (1), wherein the code array (2) has a point grid having a plurality of basic symbols (20), the coarse positions of the basic symbols (20) are encoded in the code array (2), the basic symbols (20) define a first principal direction and a second principal direction independent of the first principal direction along the point grid, and the camera image is in coordinate system X I ,Y I A first linear function is provided having a first function argument in the coordinate system of the camera image, and the first linear function is oriented parallel to the first principal direction in the coordinate system of the code array, and by changing the first function argument, the first linear function is shifted parallel to the second principal direction in the coordinate system of the code array, and / or a second linear function is provided having a second function argument in the coordinate system of the camera image, and the first linear function is oriented parallel to the first principal direction in the coordinate system of the code array, and / or a second linear function is oriented parallel to the second principal direction in the coordinate system of the camera image, and / or A method is disclosed in which a linear function is provided, the second linear line is oriented parallel to the second principal direction in the coordinate system of the code array, the second linear line is shifted parallel to the first principal direction in the coordinate system of the code array by changing the second function argument, the first function argument and / or the second function argument are determined such that the first or second linear line intersects a reference point, and the rotation angle phiz of the camera (1) around the optical axis is derived as the degrees of freedom of the camera (1) relative to the code array (2) based on the first and / or second linear lines.

Description

[Technical Field]

[0001] Conventional technology The present invention relates to a method for identifying at least one degree of freedom of a camera relative to a code sequence in a camera image of the camera, as described in claim 1. The present invention also relates to a computer program, a machine-readable storage medium equipped with the computer program, and an electronic control unit or an automated device equipped with an electronic control unit. [Background technology]

[0002] In modern manufacturing and other applications, it is necessary to detect the relative position of a moving object to its surroundings. For this purpose, numerous sensors exist, such as displacement sensors, and encoded position sensors. All existing sensors have unique advantages that can be based on their accuracy, simplicity, low-cost integration, robustness, or other characteristics, depending on the application.

[0003] In the present applicant's German Patent Application Publication No. 102016216221, which forms the closest prior art, a method is proposed for positioning and / or locating objects in space and / or on a surface using a two-dimensional code array. The two-dimensional code array has a plurality of basic symbols, which are arranged on a surface, and these basic symbols on the surface form a two-dimensional periodic grid. Based on the code array, the position of a camera relative to the code array can be determined.

[0004] The present applicant's German Patent Application Publication No. 102016216196 discloses a sensor system which utilizes a method for positioning and / or locating a prior publication. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] German Patent Application Publication No. 102016216221 [Patent Document 2] German Patent Application Publication No. 102016216196 [Overview of the Initiative] [Means for solving the problem]

[0006] Disclosure of the invention A method for determining at least one degree of freedom of a camera relative to a code sequence from a camera image of a camera, having the features of claim 1, is proposed, along with an electronic control unit or an automated device equipped with an electronic control unit, a computer program and a machine-readable storage medium, each having the features of an independent claim. Preferred or advantageous embodiments of the present invention will become apparent from the dependent claims, the following description and the accompanying drawings.

[0007] The method according to the present invention is used to identify at least one degree of freedom of a camera relative to a code array from a camera image of the camera.

[0008] This method is based on the method described in German Patent Application Publication No. 102016216221 and / or German Patent Application Publication No. 102016216196, and the disclosures of these specifications are incorporated herein by reference.

[0009] Camera definition: The camera includes an optical sensor unit (also referred to as an imaging sensor element, camera chip, or imaging device) for capturing camera images, and an imaging optical system. In particular, the camera is configured as a color camera or a monochrome camera. The optical sensor unit is configured to provide a single camera image of one segment of a code sequence, and / or to provide a sequence of multiple camera images, for example, in the form of video. The camera has an imaging device as an optical sensor unit. The camera may include an objective lens, which is preferably configured as a wide-angle objective lens. In particular, the camera performs vanishing point perspective and / or one-point perspective. In this case, parallel edges, in particular, are not displayed parallel to the image when viewed obliquely, but are optically focused at a single virtual point, a so-called vanishing point. In particular, cameras that include an objective lens perform the desired perspective distortion resulting from use.

[0010] Definition of optical axis 5: An imaging optical system is ideally a rotationally symmetric optical system, where the axis of symmetry is the optical axis. An imaging optical system is characterized by the fact that light rays do not experience deflection along the optical axis as they pass through the optical system.

[0011] Definition of Image Center 7: The intersection point of the optical axis 5 and the optical sensor unit 12 is referred to as the image center 7. This image center 7 may coincide with the geometric center of the optical sensor unit 12, but it does not necessarily have to coincide with it.

[0012] Definition of a camera image: The camera image from a camera is specifically constructed as a matrix, and this matrix has pixels at its matrix points, such as 8-bit grayscale points or color points.

[0013] Definition of code array: The code sequence can be imaged by a camera, and based on the camera image, it is possible to measure, for example, the absolute position of a machine or machine part with one to six degrees of freedom. In particular, the code sequence is configured to be detected by non-contact reading from a camera, and non-contact detection by the camera makes it possible to determine the actual position in multiple dimensions. Preferably, a two-dimensional code sequence can be used in production equipment and / or inspection equipment where the positioning of workpieces and / or inspection means and / or working means must be performed.

[0014] Definitions of basic symbols: A two-dimensional code array includes a plurality of basic symbols, which are arranged on a surface, and these basic symbols on the surface form a two-dimensional periodic grid, i.e., a dot grid. The basic symbols are preferably geometric figures such as circles, squares, triangles, or lines. Particularly preferably, the basic symbols are configured as circles. Basic symbols are also referred to as dots. The basic symbols preferably represent digits in a number system. In particular, a two-dimensional code array includes at least two different basic symbols. In a possible embodiment of the present invention, the two-dimensional periodic grid includes empty grid points not occupied by basic symbols in the grid space and / or grid space as parcel symbols.

[0015] The basic symbols are arranged on a surface, which may be curved or not. For example, the surface may be the floor of production equipment and / or inspection equipment. The basic symbols are arranged in a two-dimensional grid within the surface, preferably with the area centroid of the basic symbols forming a grid point within the point grid.

[0016] Definition of a point grid in a code array: Grid points are also referred to as grid spaces and / or lattice spaces below. In particular, a two-dimensional periodic grid of points forms a two-dimensional lattice.

[0017] Definition of a parcel: The surface is preferably divided into a plurality of parcels of the same kind arranged regularly, and these parcels have a basic shape, for example, a square, rectangle, triangle, or hexagon. Parcels of the same kind are understood to be parcels of equivalent size and / or similar shape. Preferably, each parcel contains n basic symbols. A parcel contains an integer number of basic symbols, and especially an even number of basic symbols. Preferably, a parcel contains more than 10 basic symbols, especially more than 20 basic symbols, and especially more than 40 basic symbols. Furthermore, the number of basic symbols in a single parcel is preferably less than 100. Parcels formed by basic symbols may be optically displayed within the code array, for example by a border, or they may not be optically displayed and simply form a conceptual and / or logical unit.

[0018] Definition of parcel area: A parcel has at least one first parcel region and a second parcel region. The X parcel region includes the first parcel region, and the Y parcel region includes the second parcel region. In particular, each parcel region occupies one connected surface within the parcel or occupies multiple distributed, unconnected subsurfaces.

[0019] Each parcel region contains multiple basic symbols. In particular, the X parcel region and the Y parcel region contain the same number of basic symbols. Specifically, at least two parcel regions are arranged within the parcel such that they have p rotational symmetries with respect to the center of the parcel as a point of rotation, where p rotational symmetries are, for example, 2, 3, or 4 rotational symmetries.

[0020] Definition of Parcel Symbols: Each parcel has at least one parcel symbol, which represents a fixed reference point within each parcel. The parcel symbols enable reading and / or decoding of the basic symbols in a defined order. The parcel symbols are regularly and / or periodically arranged, in particular, within a two-dimensional periodic grid of the basic symbols. The parcel symbols can be located at or adjacent to grid points. The parcel symbols are each located in the same position within the parcel, for example, at the center of the parcel. The parcel symbols are graphic elements distinct from the basic symbols, such as triangles, hexagons, or lines. Alternatively and / or supplementally, the parcel symbols are represented by omitting one or more basic symbols within the parcel. In one possible embodiment, the parcel symbols form points of symmetry with p rotational symmetry of the parcel. In particular, the reading direction and / or decoding order, i.e., the order in which the basic symbols within the X parcel region and / or Y parcel region must be read, is defined. Preferably, the reading direction and / or decoding order correspond to the setting of which basic symbols should be read and / or decoded in sequence.

[0021] Definition of coding: Within the X-parcel region, the X-coordinate values ​​are encoded by the basic symbols, and within the Y-parcel region, the Y-coordinate values ​​are encoded by the basic symbols. In particular, the X and Y coordinate values ​​are the coordinates of the basic symbols within the parcel, and the coordinates are shown in the Cartesian coordinate system of the surface stretched by the basic symbols. Alternatively and / or supplementarily, the X and Y coordinate values ​​can also indicate the position within the surface as coordinates in other coordinate systems, such as oblique coordinates, cylindrical coordinates, or spherical coordinates.

[0022] Definition of a preferred embodiment of a point grid: In a particularly preferred embodiment of the present invention, the two-dimensional periodic grid is a rectangular grid, and the parcels are also rectangular. In particular, the rectangular grid and rectangular parcels are square grids and / or square parcels. Preferably, the distances between basic symbols are of equal magnitude along the length axis and width axis of the rectangular grid. For example, in the case of a square grid with square parcels, the number of basic symbols in the X and Y directions of the planar two-dimensional periodic grid is the same. In particular, the X parcel region and the Y parcel region each consist of two spatially separated rectangular sub-faces within the parcel, and the sub-faces have longitudinal extensions. The longitudinal extension of the sub-faces of the X parcel region is preferably perpendicular to the longitudinal extension of the sub-faces of the Y parcel region. Preferably, the surface occupied by the X parcel region can be transitioned to the surface occupied by the Y parcel region by a 90° rotation.

[0023] In a particularly preferred embodiment of the present invention, the periodic grid has a grid length and a grid width. The grid length extends in the X direction of the Cartesian coordinate system, and the grid width extends in the Y direction. The coordinate system thus formed associates each point on the coding surface with a uniquely identifiable position vector (X,Y).

[0024] g consecutive basic symbols form, in particular, one overall sequence. The overall sequence is entered in an X parcel area, in particular in a plurality of adjacent parcels in the X direction. The entry of the basic symbols is carried out, in particular, in the interior of each parcel, preferably in ascending order in the X direction of the parcels, in a predefined reading order and / or decoding order within the X parcel area, according to the order of those basic symbols in the overall sequence. The number g is a number large enough to completely fill the X parcel area of all adjacent parcels in the X direction. In particular, g is greater than 50, in particular greater than 1000, in particular greater than one million. The content of the X parcel areas of adjacent parcels in the Y direction is identical. It is particularly preferred that one segment of t consecutive basic symbols forms one partial sequence in the overall sequence. In particular, each segment of t consecutive basic symbols of the overall sequence forms one partial sequence, and the consecutive basic symbols are consecutive in the decoding order and / or reading order. In particular, the overall sequence is configured such that each partial sequence of t consecutive basic symbols is included only once in the overall sequence when read in the forward direction, and each partial sequence read in the backward direction is not included in the overall sequence when read in the forward direction. Preferably, the partial sequence includes at least 5 consecutive basic symbols, in particular at least 20 consecutive basic symbols, in particular at least 30 basic symbols. Furthermore, the partial sequence preferably includes less than 50 basic symbols, in particular less than 30 basic symbols. In particular, t < g holds.

[0025] In a particularly preferred embodiment of the present invention, the basic symbol is configured to encode digits in base b. Base b is preferably the base of a positional numeral system. Preferably, base b = 2, i.e., the base of a binary system, where the binary digits include 0 and 1. Furthermore, base b = 10 can form a base of a decimal system, where the digits include 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9. Alternatively, base b = 16, i.e., the base of a hexadecimal system, where the digits include 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E and F. In particular, base b is arbitrarily selectable, and the digits of the base include the elements 0, 1, ..., b-1. Alternatively and / or supplementarily, base b is the base of an additive number system, such as Roman numerals.

[0026] In a particularly preferred embodiment of the present invention, the base b=2 is selected. In particular, the selected base b=2 forms a binary system. The binary system and / or the number system using base b=2 includes two digits, in particular the digit 0 and the digit 1. Preferably, two different basic symbols encode the digits 0 and 1 of the binary system. The two different basic symbols are formed by two circles, namely, a first circle with radius R1 and a second circle with radius R2. In particular, radius R1 is selected to be smaller than radius R2. In particular, the ratio of radius R2 to radius R1 is greater than the square root of 2. For example, radius R2 is selected to be smaller than half the grid dimension of the code grid, so that two adjacent circles in a periodic grid do not touch R2. This embodiment is based on the consideration that, on the one hand, particularly high information density is achieved, and on the other hand, reliable readability is achieved using standard image processing methods such as segmentation with bloba analysis.

[0027] In possible embodiments of the present invention, the whole sequence is configured such that the sum of the digits of one subsequence is less than half the maximum possible sum of the digits of one subsequence. In particular, the sum of the digits of each arbitrary subsequence taken from the whole sequence is less than half the maximum possible sum of the digits of one subsequence. For example, the sum of the digits of each subsequence of length t for a number system with base b is less than q = t·(b-1) / 2.

[0028] In a particularly preferred embodiment of the present invention, the two-dimensional code sequence includes an inverted overall sequence. The inverted overall sequence preferably represents an inversion of the overall sequence. Preferably, the inverted overall sequence is entered into the Y-parcel regions of multiple adjacent parcels in the Y-direction. The entry of basic symbols is performed within each parcel, preferably in ascending order in the Y-direction of the parcel, depending on the order of those basic symbols in the inverted overall sequence, in a predetermined reading and / or decoding order within the Y-parcel region. The number g is large enough to completely fill the Y-parcel regions of all adjacent parcels in the Y-direction. In particular, the contents of the Y-parcel regions of adjacent parcels in the X-direction are identical.

[0029] In possible embodiments of the present invention, the base symbol at the k-th position of the inverted global sequence is an encoded digit m(k), and this digit m(k), together with the digit n(k) encoded by the base symbol at the k-th position of the global sequence, satisfies the relation: m(k) = bn(k) - 1. For example, the inverted global sequence and the global sequence when a binary system with base b = 2 is chosen form the relation: m(k) = 1 - n(k).

[0030] Definition of reference point: Particularly preferably, the section includes a reference point. The reference point is, for example, the intersection of the optical axis of the camera and the plane on which the code array is located. Preferably, the method is configured to determine the position of the camera relative to the work area, based on an image captured by the sensor unit, such as the X coordinate (rx) and Y coordinate (ry) of the reference point in a coordinate system stretched by the work area and / or basic symbols and / or code array.

[0031] For the time being, within the framework of this method, only the coarse position of the encoded X and Y coordinates is identified as two degrees of freedom. Even with just the coarse position, a reliable and usable position for determining the camera's coarse position can be obtained.

[0032] By determining the X and Y coordinates within the code sequence as a rough position, the relative position of the camera relative to the code sequence can be estimated. For example, if the rough position is determined in the immediate vicinity of the reference point, the camera's position can be estimated through knowledge of the camera's optical axis extension. The camera's quasi-precise position can be obtained if the reference point is equal to the encoded position and the angle of the optical axis relative to the code sequence is known. The simplest example is when the optical axis is oriented perpendicular to the code sequence.

[0033] Definition of degrees of freedom: In a particularly preferred embodiment of the present invention, the method is configured to specify the position and / or orientation of the work module relative to the work area with up to six degrees of freedom. These degrees of freedom are, in particular, as follows: rx: Location coordinates in the code array's coordinate system (as coarse and / or precise positions).

[0034] ry: Location coordinates in the coordinate system of the code array (as coarse and / or precise positions).

[0035] In particular, rx and ry relate to the reference point, especially the point where the optical axis penetrates through the code array.

[0036] rz: The distance from the reference point along the optical axis to the camera.

[0037] phiz: The Z-axis rotation angle of the camera around the optical axis.

[0038] phix: The first pitch angle of the optical axis with respect to the code array, particularly with respect to the first principal direction.

[0039] phiy: The second pitch angle of the optical axis with respect to the code array, particularly with respect to the second principal direction.

[0040] Definitions of X and Y coordinates (rx, ry): For example, the six degrees of freedom of the camera's position and / or orientation relative to the work area are particularly related to the coordinate system (X) of the code array within the plane of the code array. C ,Y C ) includes coordinates X and Y with respect to the Cartesian coordinate system established by the basic symbols.

[0041] Definition of the distance (rz) between the reference point and the optical center of the camera: Furthermore, the six degrees of freedom are determined by the distance Z or r from the camera to the reference point along the optical axis. z This includes the distance between, for example, a reference point and, for example, the optical center of a camera's objective lens.

[0042] Definition of rotation angle: Furthermore, three independent rotation angles of the camera in the camera coordinate system are identified. Two of these rotation angles are defined as intermediate angles between the optical axis and the code array (phix, phiy). The third rotation angle (phiz) describes the rotation of the camera about the optical axis.

[0043] Definition of the coordinate system for code arrays: Coordinate system of code array (X C ,Y C ) is located within the plane of the code array.

[0044] Definition of the coordinate system of the camera's imaging device and / or the coordinate system within the camera image: Coordinate system of the imaging device (X I ,Y I ) is located within the plane of the imaging device or within the camera image.

[0045] Definition of the camera coordinate system: The camera's coordinate system (X,Y,Z) has its origin at the optical center of the camera's objective lens, and the negative Z direction coincides with the camera's optical axis.

[0046] Generalization: In this specification, coordinate systems are exemplified with respect to their origin and orientation, although equivalent coordinate systems can also be used.

[0047] Definition of the method: In several modifications, this method is characterized in that the X and Y coordinate values ​​are decoded and / or identified based on a camera image captured by the camera. In one possible evolution of this method, the orientation of a segment with respect to three coordinate axes is identified based on the segment captured in the image within the code sequence. In particular, the orientation of the segment captured in the image is identified with respect to the X and Y axes of the Cartesian coordinate system stretched by the basic symbols. Alternatively and / or supplementarily, the coordinates of the position of a reference point within the code sequence are identified by decoding the segment contained in the image within the code sequence. In a particularly preferred embodiment of the present invention, in this method, the orientation of the work module with respect to the work area is identified with up to six degrees of freedom. In particular, in this case, the orientation of the camera within the work area is identified in three Cartesian coordinates X, Y, and Z, where X and Y are coordinates of the coordinate system stretched by the basic symbols, and Z is the distance from the work module to the work area. Furthermore, in this method, for example, three Euler angles of the camera are identified.

[0048] Alternatively and / or supplementally, two-dimensional code sequences are used in fields other than automation technology, for example, in nature and the environment, biology and medicine, architecture, entertainment appliances, and / or sensors, to monitor operational flows.

[0049] The general advantages of this invention depend on modifications: The central objective of this invention is to provide a high-performance, highly reliable, and inexpensive method for locating and identifying one or more objects in a workspace with up to six degrees of freedom (6D positioning = simultaneous position detection in six degrees of motion).

[0050] This method processes image data provided by an imaging measurement system, such as a camera. A smartphone, for example, can be used to capture images and to determine the location using the method according to the present invention, and this method is implemented by the smartphone's embedded computer or as an algorithm within an application running in the cloud.

[0051] To determine the object's position with up to six degrees of freedom, the camera detects a two-dimensional code array associated with the object to be located.

[0052] The method according to the present invention provides the following characteristics depending on the modification: • Up to complete 6D position information can be obtained: translation (X, Y, Z) and rotation (φ_X, φ_Y, φ_Z). • Absolute position information is obtained (not incremental), meaning that the reference drive required in the case of incremental sensors is omitted. • The measurement range is very wide: • In X,Y: It is "almost infinite" and limited only by the code sequence. For example, the area that can be uniquely encoded using the code and 2.5 mm dot grid described in [B] is as follows: For a parcel size of 5x5 dots: 2m x 2m For a parcel size of 7x7 dots: 8km x 8km For a parcel size of 9x9 dots: 550,000 km x 550,000 km In Z: It is a single dimension with limitations. It can be adapted to the application through camera resolution, objective lens, and the grid dimensions of the code array.

[0053] In φ_X and φ_Y, the actual values ​​are + / -60°. In φ_Z: Infinite (0° to 360°) • The measurement rate is high, for example, 100Hz to 10000Hz. • The delay time (the time from when an image is captured until the 6D position measurement is output) is short, for example, 10ms to 100μs.

[0054] • High measurement accuracy in all six degrees of freedom: If the code grid dimension is 2.5 mm, the actual repeatability is 1 μm and 0.01° (in the case of 3 sigma). • Maximum readability is achieved even when localized noise occurs in the image or when image brightness fluctuates. • Both the sensor system and the code arrangement can be made inexpensive. • It is possible to read additional data simultaneously with location determination. For example, an ID code embedded in a code array can be read to identify an object.

[0055] By changing the objective lens, image sensor resolution, code array grid dimensions, reading field size, computational performance, etc., accuracy, measurement range in Z, redundancy, and measurement rate can be scaled.

[0056] For example, it can be easily integrated into existing systems as a smartphone application.

[0057] This makes it easy to implement numerous automation technologies that were previously impossible or economically unfeasible, such as the following: • To achieve real-time attitude control in up to six dimensions, precisely position the axes or rotors in the axis system or robot system. • To build an absolute positioning system and robot that eliminates the need for reference driving. • Simplify the positioning system because multiple sensors that measure in one dimension can be replaced by a single sensor that measures in multiple dimensions.

[0058] For example, visual servo processing to make a robot's gripper follow a moving target. • Monitoring and tracking of multiple objects in space through periodic position measurements. • Capture and analysis of the operation flow • Vibration analysis of machines in 6 dimensions.

[0059] • Identifying the relative orientation of two objects. For example, a camera in a handheld device detects two objects, each assigned a point code, within a single image. The proposed method allows for the positioning of the two objects relative to a common camera coordinate system. By forming a vector difference, the relative orientation of the two objects can be determined. The camera position is computationally removed in this case; that is, the method is sufficiently independent of the camera position.

[0060] Identify the relative orientation of three or more objects, each equipped with a camera and / or code array.

[0061] For example, within a manufacturing plant, a location network is constructed to determine the relative positions of multiple stationary or moving objects relative to each other, or relative to the factory coordinate system. For instance, multiple autonomous vehicles, each equipped with a camera, can be located by detecting a sequence of codes mounted on the ceiling of the factory.

[0062] The code array has a point grid containing multiple basic symbols, where the coarse positions of the basic symbols are encoded within the code array.

[0063] The basic symbol defines a first principal direction and a second principal direction independent of the first principal direction, along a point grid. Depending on the observation angle, the two independent principal directions are oriented perpendicularly or at an angle to each other in the camera image.

[0064] A reference point is placed within the camera image. This reference point is formed as the intersection of the optical axis with the imaging device and / or the camera image. Therefore, the reference point is set by the camera within the camera's coordinate system and / or within the camera image.

[0065] The first linear function is determined in the camera image coordinate system using the first function argument for the first line. The first line is always parallel to the first principal direction of the point grid, independently of the first function argument, when the first line is transferred to the code array coordinate system. By changing the first function argument, the first line is shifted parallel to the second principal direction in the code array coordinate system. Therefore, by changing the first function argument, the line can be placed, for example, in a row of the point grid, and in particular, in both the code array coordinate system and the camera image coordinate system.

[0066] The second linear function is determined in the camera image coordinate system using the second function argument for the second line. The second line is always parallel to the second principal direction of the point grid, independently of the second function argument, when the second line is transferred to the code array coordinate system. By changing the second function argument, the second line is shifted parallel to the first principal direction in the code array coordinate system. Therefore, by changing the second function argument, the line can be placed, for example, in a row of the point grid, and in particular, in both the code array coordinate system and the camera image coordinate system.

[0067] Here, the terms "row" and "column" are used primarily for nominal purposes and do not refer to any particular direction of the point grid.

[0068] Within the framework of the present invention, it is proposed that the first and / or second function arguments are determined such that the first or second line intersects a reference point. For example, the first and second lines are shifted by changing the function arguments so that they intersect a reference point.

[0069] The camera rotation angle Phiz, centered on the optical axis, is derived (determined) as the camera's degrees of freedom relative to the code array, based on a first and / or second straight line intersecting a reference point. This derivation is possible because these linear functions are defined in the camera image. Therefore, the rotation angle Phiz can be determined as a linear angle in the coordinate system of the camera and / or the camera image. Theoretical considerations show that the linear angle in the coordinate system of the camera and / or the camera image corresponds to the camera rotation angle Phiz centered on the pivot axis, and in particular, there is no need to perform a coordinate system transformation from the coordinate system of the camera and / or the camera image to the coordinate system of the code array. Thus, the rotation angle Phiz can be easily determined based on at least one linear function. Therefore, the present invention demonstrates how accurately and without high computational cost the rotation angle Phiz can be determined. Here, it is clear that the determination is successfully performed by selecting the intersection point of the camera's optical axis, either by the camera image or the imaging device, as the reference point. In such a special arrangement configuration, the rotation angle Phiz can be determined (derived) particularly easily. This makes it possible to easily and simultaneously determine the rotation angle Phiz from the camera image with high accuracy, and therefore the rotation angle Phiz is used as the camera's degree of freedom relative to the code sequence.

[0070] In a preferred configuration of the present invention, a basic symbol matrix is ​​identified from the camera image, where the positions of the basic symbols in the camera image are entered. Therefore, the positions in the basic symbol matrix indicate the positions in the point grid, particularly the positions of the basic symbols in the coordinate system of the imaging device and / or the coordinate system of the camera image. A first linear function and / or a second linear function are preferably determined based on the basic symbol matrix.

[0071] Particularly preferably, a single linear function is used, formed by a combination of a linear angle function and an axis intersection function, where the linear angle is the angle of intersection between the line and one of the axes of the camera image coordinate system, depending on the function argument. The axis intersection function, depending on the first function argument, defines an axis intersection along an axis extending through a reference point in the camera image coordinate system. Thus, by changing the first function argument in the coordinate system of the code array, the line is shifted, causing the line in the camera and / or camera image coordinate system to shift (with perspective distortion) so that the axis intersection moves along the axis. One combination of linear function or linear angle function and axis intersection function is associated with a first principal direction, and the other combination of linear function or linear angle function is associated with a second principal direction. The advantage of this association is that the function argument is determined via the axis intersection function, and subsequently, the rotation angle Phiz can be read from the linear angle function based on the determined function argument.

[0072] In a preferred evolution of the present invention, a first value relating to the rotation angle Phiz is determined from a first linear function, a second value relating to the rotation angle Phiz is determined from a second linear function, and subsequently, the rotation angle Phiz is determined as the average of these two values. By being able to determine the rotation angle Phiz independently from the two linear functions, two independent values ​​are obtained, and then these two independent values ​​can be averaged for determining the rotation angle Phiz, thereby improving measurement accuracy. Alternatively, a validation check can be performed to discard one of the two values ​​if it is invalid or not valid.

[0073] From the camera image, preferably a two-dimensional basic symbol matrix is ​​identified, where each grid point in the code array is associated with an entry in the basic symbol matrix, and the entry in the basic symbol matrix represents the center position of the basic symbol in the camera image. Therefore, the entries in the basic symbol matrix indicate the position at the associated grid point, in particular the position of the basic symbol in the coordinate system of the imaging device and / or the coordinate system of the camera image.

[0074] The point grid in the camera image exhibits particular perspective distortion. As a result, although the rows and columns of the point grid are formed in a straight line, they are arranged at an angle to each other due to the corresponding perspective distortion.

[0075] Furthermore, optical distortion may occur, which can be compensated for by arbitrary parallelization processing.

[0076] From the data of the basic symbol matrix, a first line function in the camera image coordinate system is determined for a first line using a first function argument. The first line is always parallel to the first principal direction of the point grid, independently of the first function argument, when the first line is conceptually transferred to the coordinate system of the code sequence. By changing the first function argument, the first line is shifted parallel to the second principal direction in the coordinate system of the code sequence. Therefore, by changing the first function argument, the line can be placed, for example, in a row of the point grid, and in particular, in both the coordinate system of the code sequence and the coordinate system of the camera image.

[0077] In particular, this can be done as follows: Starting point status: a) The central point of a basic symbol forms a two-dimensional grid of points in the code plane, consisting of regularly arranged linear rows and columns; b) The camera images the code plane onto the image sensor with perspective distortion. In the camera image, rows generally appear as fan-shaped lines with a single common vanishing point. The same applies to rows; c) Due to distortion of the objective lens, rows and columns appear as curved lines in the camera image.

[0078] Method steps in the camera coordinate system: a) Through parallelization, the distortion of the objective lens is mathematically eliminated, and curved lines are transformed into straight lines; b) Linear interpolation is used to fit the lines to the rows, and these lines are numbered using integer indices as function arguments, following their progression. These lines form a first line bundle. The same procedure is followed for the columns, and these lines form a second line bundle; c) Each line is described by its line angle and intercept value. The line angles of a bundle form a sequence approximated by quadratic interpolation. Integer line indices are used as function arguments. The polynomial has three interpolation parameters. The same procedure is followed for the intercept values, from which three further interpolation parameters are obtained. That is, each bundle of lines is described completely and compactly by six interpolation parameters; d) By substituting reasonable function values ​​into the interpolation function, it is also possible to calculate a straight line located between two rows or two columns.

[0079] Alternatively or supplementally, based on the fundamental symbol matrix, the second line function in the camera image coordinate system is determined for the second line using the second function argument. The second line is always parallel to the second principal direction of the point grid, independently of the second function argument, when transcribed to the code array coordinate system. By changing the second function argument, the second line is shifted parallel to the first principal direction in the code array coordinate system. Therefore, by changing the second function argument, the line can be positioned, for example, in a row of the point grid, and in particular, in both the code array coordinate system and the camera image coordinate system.

[0080] It should be noted that the terms "row" and "column" are used here solely for nominal purposes and do not refer to any particular direction of the point grid.

[0081] At least one degree of freedom of a camera is determined based on at least one of several linear functions. Arbitrarily, at least one degree of freedom of a camera can be determined based on two linear functions.

[0082] In this case, there is a further consideration that the amount of data should be reduced along the way from the camera image to at least one degree of freedom in order to enable faster and / or more efficient computation of at least one degree of freedom. For example, a camera image with a size of 200 x 200 pixels still has 40,000 values, while the basic symbol matrix has already been reduced to position, and accordingly, for example, in the case of a section of the camera image with side lengths of a point grid of 15 basic symbols, there are only 225 values ​​related to position. By deriving at least one linear function, the amount of data is reduced to the parameters of the linear function.

[0083] In this case, it has been found that, in addition to the six parameters, one data point per linear function is sufficient as an optional choice to ensure that the substantial information content of the basic symbol matrix or camera image is sufficient on the way to at least one degree of freedom. This reduces the amount of data in the example with 225 entries to 12 or 14 entries. This significantly reduces the computation of at least one degree of freedom from a cost perspective.

[0084] A further advantage of this embodiment is that the rows and columns of the basic symbol matrix in the code array are arranged parallel and regularly spaced apart from each other, thereby performing a kind of averaging across the basic symbol matrix by deriving a linear function in the camera image, in which case the linear function represents the averaged information of the basic symbol matrix. The linear function achieves both information compression and improvement of information content.

[0085] In this way, the method enables a computationally efficient implementation of a method for identifying at least one degree of freedom of a camera relative to a code array from a camera image. From an applied technical standpoint, this identification can be performed, for example, on a microcontroller capable of determining at least one degree of freedom at least 100 times per second. In this way, the method can be used for real-time applications, for example, in manufacturing.

[0086] In a preferred evolution, the first linear function is identified based on at least two rows of the basic symbol matrix, preferably three or more rows, and in particular all rows. Alternatively or supplementarily, the second linear function is identified based on at least two columns of the basic symbol matrix, preferably three or more columns, and in particular all columns. This evolution emphasizes that the linear function carries averaged and / or compressed information across multiple rows or columns.

[0087] In a preferred embodiment of the present invention, one first compensation line is formed for each row along a first principal direction. The first linear function is formed based on a plurality of first compensation lines. By forming one first compensation line for each row, this first compensation line can be fitted to the extension of that row, thereby the first compensation line already forms the averaged and / or compressed information of the underlying row. The first linear function is formed based on a plurality of first compensation lines, thereby performing a second averaging or compression, and thus the first linear function is formed by a double average of the original information.

[0088] Alternatively or supplementally, for each column along the second principal direction, one second compensation line is formed. The second linear function is formed based on multiple second compensation lines. By forming one second compensation line for each column, this second compensation line can be fitted to the extension of that column, thereby the second compensation line already forms the averaged and / or compressed information of the underlying column. The second linear function is formed based on multiple second compensation lines, thereby performing a second averaging or compression, and therefore the second linear function is formed by the double average of the original information.

[0089] In a preferred implementation, the first function argument is constructed as the first integer count of a row, and / or the second function argument is constructed as the second integer count of a column. Thus, with respect to the first integer count in the first linear function, the first line corresponds to the first compensation line. Similarly, with respect to the second integer count as the second function argument in the second linear function, the second line corresponds to one of the rows of the basic symbol matrix. However, the first line and / or the second line are not exactly the first or second compensation line, but rather the corrected first or second compensation line, since the linear function has undergone a second averaging / compression.

[0090] In a preferred embodiment, the compensation line is described by a linear angle as an intersection angle, and by at least one axis intersection with the coordinate system of the imaging device and / or camera image. With respect to the compensation line, it is sufficient to have at least one or exactly one coordinate axis angle and axis intersection of the coordinate system to uniquely determine the compensation line in the coordinate system.

[0091] In this embodiment, the position of the compensation line within the point grid of the row or column to which it belongs is reduced to two values.

[0092] In a preferred development, the linear function is formed by a combination of a linear angle function, which depends on the function argument of the linear function, and an axis intersection function, which depends on the function argument of the linear function. Thus, the linear function is also determined by the linear angle and at least one axis intersection.

[0093] It is preferable that the linear angle function is constructed as a quadratic polynomial, and / or the axis intersection function is constructed as a quadratic polynomial, with these polynomials having the function arguments of the respective linear functions as function arguments. This makes it possible to determine the linear angle and / or axis intersections depending on the function arguments. By choosing quadratic polynomials, the approximation can be performed particularly easily, and thus computational efficiency is further improved.

[0094] In a preferred embodiment, the axis intersections of the coordinate system's axes are selected such that, depending on the linear angle of the linear function along with the coordinate system, they produce smaller intermediate angles relative to the perpendiculars to each coordinate axis. Furthermore, the linear function is associated with data encoding the selected coordinate axes. In this case, it is considered that only the axis intersection of a single coordinate axis is needed to describe the line, and the axis intersections with both coordinate axes are unnecessary. To achieve the greatest possible descriptive power, axis intersections are selected such that the associated intersection angle is more perpendicular to the intersecting coordinate axes.

[0095] In an advanced form of the present invention, the coarse position of at least one of several basic symbols in a code array is determined based on the basic symbols or a subset thereof of a basic symbol matrix. In particular, a reading field of a camera image and / or a planar point grid is defined, a corresponding basic symbol matrix is ​​formed, and each point in the reading field is associated with an entry in the basic symbol matrix, the position of the basic symbol, in particular a normalized plane or data is entered into the basic symbol matrix, and based on the basic symbol matrix, the coarse position of the coordinate system of the point grid and / or the coarse position of at least one basic symbol in the code array is identified as two degrees of freedom.

[0096] Preferably, a first axis intersection function in the coordinate system of the camera image is formed by a first line. The first line is oriented parallel to the first principal direction of the point grid in the coordinate system of the code array. The axis intersection function has a first function argument, and by changing the first function argument, the first line is shifted parallel to the second principal direction in the coordinate system of the code array. The first axis intersection function, depending on the first function argument, defines a first axis intersection along a first axis in the coordinate system of the camera image, the first axis extending through a reference point. Therefore, by changing the first function argument in the coordinate system of the code array, the first line is shifted (with perspective distortion) so that the axis intersection moves along the first axis.

[0097] More preferably, a second axis intersection function in the coordinate system of the camera image is formed by a second line. The second line is oriented parallel to the second principal direction of the point grid in the coordinate system of the code array. The axis intersection function has a second function argument, and by changing the second function argument, the second line is shifted parallel to the second principal direction in the coordinate system of the code array. The second axis intersection function, depending on the second function argument, defines a second axis intersection along the second axis in the coordinate system of the camera image, the second axis extending through a reference point. Thus, by changing the second function argument in the coordinate system of the code array, the second line is shifted (with perspective distortion) so that the axis intersection moves along the second axis.

[0098] The first and second function arguments are determined based on the axis intersection function such that the reference point forms the first and second axis intersections. Figuratively speaking, the first function argument is modified until the first line extends through the reference point in the camera and / or camera image coordinate system, and / or until the first axis intersection is located at the reference point. Similarly, the second function argument is modified until the second line extends through the reference point in the camera and / or camera image coordinate system, and / or until the second axis intersection is located at the reference point.

[0099] Subsequently, based on the first and second function arguments and the coarse position, the precise position of the reference point in the coordinate system of the code sequence within the plane of the code sequence is identified as an additional degree of freedom. From a conceptual standpoint, first, the coarse position of the basic symbol in the coordinate system of the code sequence is identified, and subsequently, the shift of the basic symbol to the reference point is determined based on the function arguments.

[0100] This method has the advantage of being able to identify the coarse position of basic symbols by decoding the code sequence. Subsequently, the shift to the reference point is calculated based on the axis intersection function, and by utilizing the axis intersection function, calculations can be performed using a small number of calculated values, and thus computationally efficient. This has the advantage that this method can be executed with minimal computing power in digital data processing devices, especially microcontrollers, for real-time applications.

[0101] In a preferred embodiment of the present invention, the first function argument is configured as the count value of a row in the point grid, and / or the second function argument is configured as the count value of a column in the point grid. Figuratively speaking, the position of a basic symbol whose coarse position is known is shifted to a reference point by integer or fractional increments in the grid dimensions of the point grid.

[0102] As previously described, the reference point is preferably configured as the intersection of the camera's optical axis and the imaging device. Therefore, the reference point is defined as the structural position in the coordinate system of the camera and / or the camera image. However, it is not necessarily required that the reference point and / or intersection be located precisely in the center of the camera image and / or on the camera's imaging device. Rather, the position of the reference point can be defined through calibration.

[0103] In possible embodiments of the present invention, the axis intersection function is configured as a linear function for describing a straight line, particularly as described above. In this case, the axis intersection function includes a complete mathematical description of the straight line, depending on each function argument.

[0104] Alternatively, a linear function is used that is formed by a combination of a linear angle function (the angle of intersection between a line and one of the axes of the coordinate system of the camera image, depending particularly on the function arguments described above) and an axis intersection function. Thus, the line is completely described by the axis intersection and the linear angle. The advantage of this assignment is that only the axis intersection function needs to be determined and / or evaluated to pinpoint the precise location, and more precisely, information about the linear angle is not required. This embodiment further improves the efficiency of the method.

[0105] Next, the precise position can be determined based on the function arguments and the known grid dimensions of the point grid. Figuratively speaking, the precise position is determined by starting from the position of a basic symbol whose coarse position is known, and then moving a portion of the grid dimensions in the first principal direction and a portion of the grid dimensions in the second principal direction, for example, to reach a reference point in the coordinate system of the code array. This representation is particularly computationally efficient.

[0106] Preferably, it is proposed that the first and / or second function arguments of the linear function be determined such that the first or second line intersects the reference point. For example, the first and second lines are shifted by changing the function arguments so that the first and second lines intersect the reference point.

[0107] In a preferred development, the following values ​​are identified as characteristic values ​​at the reference point from the camera image: A rotation angle Phiz (also called the Z rotation angle) is determined around the optical axis of the camera. The rotation angle phiz may be determined particularly preferably by a linear function and / or an intersection function and / or an axial intersection function.

[0108] Furthermore, at least one of the point grids in the camera image or the local grid dimension described above is determined. The local grid dimension represents the distance between two adjacent lines of the point grid in the camera image. The local grid dimension may be determined particularly preferably by a linear function and / or an intersection function and / or an axis intersection function. The point grid in the code array consists of regularly and / or regular grid dimensions. By capturing the code array with a camera that yields a camera image, the grid dimensions are imaged with distortion, thereby causing the grid dimensions to change across the camera image. The local grid dimension at a reference point is understood to be the value of the grid dimension at the reference point.

[0109] Furthermore, at least one first local angular divergence and a second local angular divergence of the point grid in the camera image are determined. Basically, within the point grid in the code array, the lines are arranged parallel to each other. However, the point grid is distorted by the imaging of the code array onto the camera image, and therefore, the lines take an angular difference other than zero between any two adjacent lines. The local angular divergence of the point grid in the camera image is understood to be the angular difference between two adjacent lines of the point grid at the reference point. This angular difference is the angular difference between the first local angular divergence with respect to the first principal direction and the second local angular divergence with respect to the second principal direction. The first and second local angular divergences may be determined in particular via a linear function and / or a linear angular function.

[0110] The characteristic values ​​described above completely describe (from a physical standpoint) the three degrees of freedom of the camera relative to the code array: namely, the distance rz between the code array and the camera at the reference point and / or along the optical axis, and the two pitch angles phix and phiy that describe the midpoint angles between the plane of the code array and the optical axis in the first and second principal directions. Therefore, the characteristic values ​​described above are sufficient to determine the three degrees of freedom described above.

[0111] In this case, a further consideration is that the aforementioned characteristic values ​​can be easily determined from the camera image. Besides manually identifying the characteristic values ​​within the camera image, these characteristics can also be derived through digital image processing techniques. Once the characteristic values ​​are understood, the three degrees of freedom mentioned above can be estimated.

[0112] Therefore, by selecting the characteristic values ​​described above, a new method for determining the degrees of freedom is proposed, which is superior in that it uses only a small number of characteristic values ​​to determine the degrees of freedom. This makes it possible to construct the method computationally efficient and highly accurate.

[0113] In a preferred embodiment of the present invention, the above-described characteristic values ​​are substituted into an imaging model that describes the optical imaging of the code sequence onto an image sensor, and the camera's position and orientation relative to the code sequence are taken into consideration.

[0114] This model describes the physical, and therefore analytical, relationship between the characteristic values ​​described above as input values ​​and the three degrees of freedom described above as output values. Thus, this relationship leads to the determination of the three degrees of freedom.

[0115] In possible embodiments of the present invention, three equations are determined for three degrees of freedom, forming a system of equations. It should be emphasized that these three equations represent a possible expression of the analytical relationship between the characteristic values ​​and the three degrees of freedom, although other mathematical or analytical expressions are possible. However, the physical relationship can be expressed particularly simply and compactly through the three equations and / or the system of equations.

[0116] The equation for determining the first pitch angle phix is ​​a function of the first local angular divergence dalphac0, the distance rz between the code array and the camera, and the second pitch angle phiy.

[0117] In particular, the first pitch angle is given by the following equation: tanφ Χ= (-dα C0 · r Z ) / (c grid · cos 2 φ Y ) is determined by, provided that c grid : is the grid distance of the dot grid in units of [m].

[0118] The equation for specifying the second pitch angle phiy is a function of, in particular, the second local angular divergence dalphac1, the distance rz between the code array and the camera, and the first pitch angle phix.

[0119] In particular, the second pitch angle is given by the following equation: tan φ Y = (-dα C1 · r Z ) / (c grid · cos φ X ) is determined by.

[0120] The equation for the distance rz between the code array and the camera is, in particular, the following function:

Table 1

[0121] Depending on the definition and / or specification of the rotation angle Phiz, the columns relate to two independent variables and the rows relate to two alternative variables.

[0122] Alternatively or additionally, the distance between the code array and the camera is given by the following equation as the average value of variables and / or functions: r Z = (r Z0k + r Z1k ) / 2 is determined by.

[0123] Essentially, the system of equations can be solved analytically. In a preferred evolution of the present invention, the system of equations, which includes three equations, is solved recursively. In this case, a starting value is first set, and then the degrees of freedom are determined in the regression step of the optimization routine.

[0124] Preferably, a start field is determined within the framework of this method. The start field preferably has at least three adjacent basic symbols, which are arranged at predetermined angles to each other. The connecting lines between the basic symbols of the start field define two independent principal directions along the point grid in the camera image. If the start field has exactly three basic symbols, these basic symbols are arranged, for example, at predetermined angles to each other. These basic symbols define a coordinate system of a planar point grid, with at least one of the three basic symbols forming the origin of the coordinate system. In a preferred evolution of the present invention, the start field has nine basic symbols arranged in a square and / or rectangle. In particular, the start field has an edge length with three basic symbols. The basic symbols of the start field are valid basic symbols. A valid basic symbol is understood, particularly within the framework of this method, as a basic symbol selectively defined as a valid basic symbol, or as a basic symbol classified as a valid basic symbol based on image features, and / or as a basic symbol set as a valid basic symbol by the available processing results after discovering and successfully processing the basic symbols. Image structures that are not classified as valid basic symbols are interpreted as image noise and excluded from subsequent processing.

[0125] In this method, during the search step, starting from at least one valid basic symbol, further basic symbols, especially valid basic symbols, are searched along the principal direction.

[0126] If the search is successful, and especially if additional basic symbols are discovered, these additional basic symbols are characterized as valid basic symbols.

[0127] It is assumed that the search step will be performed multiple times. If new and further basic symbols are found as valid basic symbols during the search step, the search step will selectively be performed starting from the basic symbols of the starting field, or from the newly discovered valid basic symbols. In this way, the point grid is progressively filled in, starting from the starting field with valid basic symbols, for each additional valid basic symbol. The search step is repeated until a sufficient number of valid basic symbols are found, in particular so that decoding can be performed.

[0128] Based on a valid set of fundamental symbols, or a selected subset thereof, the coarse position of at least one of the fundamental symbols in the code sequence is determined as two degrees of freedom of the camera relative to the code sequence by decoding a particularly valid set of fundamental symbols.

[0129] In particular, a valid basic symbol or a subset thereof is decoded, and from the decoded coarse position and, optionally supplementarily, the coarse orientation of the coordinate system of the point grid in the camera image, at least one degree of freedom of the camera relative to the code sequence is identified.

[0130] In this case, a more advanced consideration is that the basic symbols within the code array and / or within the camera image can be searched and discovered, for example, using planar functions in image processing. However, discovering basic symbols using, for example, bloba analysis or pattern-based search (pattern matching) requires frequent access to the pixels of the camera image, which results in time-consuming image processing.

[0131] By starting from a search field that defines the two principal directions of the point grid, and consequently the coordinate system of the point grid in the camera image, and performing the search exclusively in the principal directions, prior knowledge of the structure of the point grid can be utilized, and therefore the search is performed on lines along the principal directions, rather than on surfaces. From a practical standpoint, it is sufficient to search for further fundamental symbols along lines in the principal directions, starting from a valid fundamental symbol. It is also obvious that reducing from surface search to line search significantly reduces the number of pixel accesses and / or the cost of image processing. Therefore, the method according to the present invention enables a highly efficient implementation of the method.

[0132] In a preferred evolution of the present invention, the center position, particularly the centroid position, of each basic symbol in the camera image is determined. This allows for the identification of multiple intersections within a point grid based on the valid basic symbols.

[0133] The search step is performed starting from the center position of each valid basic symbol, and is configured to perform a line-targeted search in a predetermined search direction, particularly along one of the principal directions. By discovering the basic symbols, it is initially possible to only roughly determine the position of the basic symbols within the point grid and / or camera image, while detecting the center position allows for the precise location within the point grid and / or camera image, thereby concretizing the point grid. It is particularly advantageous to perform the line-targeted search step from the center position of the basic symbols because this prevents adjacent basic symbols from being mistakenly overlooked.

[0134] In a preferred evolution of the present invention, two connection vectors are determined in two independent principal directions along a point grid in the camera image, based on a starting field and / or on further valid fundamental symbols. By determining the connection vectors, the positions of the nearest and / or adjacent fundamental symbols to be searched can be estimated by extrapolation within the framework of a linear combination of the connection vectors, starting from known valid fundamental symbols.

[0135] Particularly preferably, determining the center position of a basic symbol is performed such that the starting point within the basic symbol is shifted along the principal direction toward the center position over multiple intermediate steps until the center position is found at a location where the center position is located in the center of the basic symbol in the principal direction. By shifting the starting position toward the center position so that the starting position is always located in the center in the principal direction, it is only necessary to search for the boundaries of the basic symbol in the principal direction for each shift step, and then subsequently shift the starting position toward the center between the boundaries.

[0136] In the example of a circular base symbol, the boundaries of the base symbol are explored by a linear search in the first principal direction of the camera image, starting from the initial position inside the base symbol, and then the boundaries of the base symbol are explored in the negative first principal direction. Averaging the boundaries results in a more precise position estimation. This step is repeated correspondingly for the second principal direction, and then again for the first principal direction.

[0137] In this way, the shift of the starting position to the center position is computationally efficient by the flow of line evaluation along the principal direction.

[0138] In subsequent steps, the area of ​​each basic symbol in the camera image is identified, and this area forms the encoded data for the code sequence. In particular, the area is used to classify the basic symbols.

[0139] The data may be, for example, 0 or 1 in a binary system, and the basic symbols may be, for example, circular surfaces of different sizes. If the center position of each is known, and the extent of the basic symbol in the principal direction is known by the shift step, the area of ​​the basic symbol can be easily estimated using these values. This allows the center position within the point grid to be determined first, and then the area of ​​the basic symbol, very computationally efficient.

[0140] The area of ​​basic symbols in a camera image is strongly affected by perspective distortion. For example, the area decreases as the distance from the camera to the basic symbol increases, and as the camera's tilt angle increases (elliptic distortion). These effects can lead to misclassification, for example, when identical basic symbols occur in the camera image at very different distances from the camera. Therefore, to classify basic symbols, it is preferable to use the normalized area of ​​the basic symbols.

[0141] To calculate the normalized area, a reference area exposed to approximately the same perspective distortion as the base symbol is identified. The parallelogram stretched by two vectors from the grid position of the base symbol to adjacent grid positions along the principal axis is used as the reference area. The reference area is obtained as the absolute value of the cross product of the two vectors.

[0142] The normalized area of ​​a basic symbol is calculated specifically as the quotient of the area of ​​the basic symbol and the reference area.

[0143] In a preferred embodiment of the present invention, a reading field is defined in a camera image and / or a planar point grid, a two-dimensional basic symbol matrix, particularly a dot matrix, is formed corresponding to the reading field, one entry in the basic symbol matrix is ​​associated with each point in the reading field, and the position, area and / or data of a basic symbol are entered in the basic symbol matrix. From the data in the basic symbol matrix, the coarse position in the coordinate system of the point grid and, optionally, the coarse orientation can be decoded, and / or the coarse position of at least one basic symbol in the code array can be identified as two degrees of freedom.

[0144] In a preferred evolution of the present invention, at least one starting position is set in a preliminary step for creating a starting field. The starting position is set arbitrarily. Preferably, the starting position is located near or adjacent to a reference point. If the starting position is randomly located inside a basic symbol, the starting position is treated as the starting position for the starting basic symbol. If the starting position is located outside a basic symbol, adjacent basic symbols along the search ray starting from the starting position are searched as the starting basic symbol. Therefore, even in this preliminary step, computationally intensive image processing on a surface is not performed; rather, basic symbols are searched only along the search ray starting from the starting position. For example, the boundaries of basic symbols can be detected by contrast changes along the search direction in a camera image. In this case, the method can use at least or exactly 8 or 16 search rays that start at the starting position and are distributed in regular angular increments over 360°. A minimum radius can be defined for the search rays, and further basic symbols are searched from this minimum radius. The maximum search radius can be optionally defined. If the maximum search radius is reached without finding adjacent basic symbols, the starting position is discarded, and a replacement starting position is selected.

[0145] In subsequent steps, adjacent basic symbols are searched for as first auxiliary starting basic symbols along further search rays, starting from the position of the initial basic symbol. The search for the first auxiliary starting basic symbols may be performed using the same distribution of search rays as described above.

[0146] In subsequent steps, the second auxiliary starting symbol is searched for in a direction angled with respect to the connection between the starting symbol and the first auxiliary starting symbol (and / or along the first principal direction), starting from the starting symbol and the first auxiliary starting symbol. However, in this case, fewer search rays than those described above can be used, particularly distributed over a smaller angular range, thereby accelerating the search. In this case, the fact that the first three symbols should be positioned at angles to each other is fully utilized, and therefore, there is prior knowledge about the direction in which the second auxiliary starting symbol will be searched. For example, only one, two, or three to six search rays distributed around the principal search direction perpendicular to the connection between the starting symbol and the first auxiliary starting symbol may be used.

[0147] To ensure completeness, starting from the initial basic symbol, all basic symbols that are directly adjacent to the initial basic symbol in the principal axis direction or diagonal direction of the point grid are searched for by a linear combination of connections, based on the connection between the initial basic symbol and the first auxiliary initial basic symbol, and based on the connection between the initial basic symbol and the second auxiliary initial basic symbol. In particular, connection vectors and / or principal directions can be derived through the first three basic symbols found. These connection vectors and / or principal directions establish the two-dimensional coordinate system of the point grid. For the purpose of improving this method, a center position can be identified to define the connection vectors as precisely as possible. In that case, a starting field with 3 × 3 valid basic symbols is provided as a result.

[0148] In the search step, adjacent basic symbols are searched starting from the starting field and / or from any further valid basic symbols discovered in previous search steps. This method preferably searches for basic symbols at grid points that have at least two valid neighbors in the principal or diagonal direction. All neighbors of a grid point provide independent position estimates by extrapolation, which, when averaged, result in a more precise position estimate of the grid point.

[0149] The condition that only grid points and / or at least two valid fundamental symbols are searched as neighbors further promotes planar spread of the detected region and avoids linear spread in spearhead, dendritic, or apical morphology. Planar spread is more robust than linear spread and has a greater tolerance for noise in the image. Based on the valid fundamental symbols, the fundamental symbol matrix of the read field is identified in particular.

[0150] In particular, in this method, it is assumed that first, a camera image is captured by a camera, then at least one degree of freedom of the camera relative to the code sequence is identified, and then, for example, an actuator of the automation device is driven. For example, at least one degree of freedom may be output to an optical output device such as a display. At least one degree of freedom may be used to control and / or adjust the position of the actuator of the automation device by using this at least one degree of freedom as an actual value. A robot equipped with a sensor unit of the automation device can, for example, identify its absolute orientation relative to the code sequence and output this absolute position as actual information, or it can travel to a predetermined further position, and the robot is further oriented in accordance with the code sequence with respect to its actual position.

[0151] Further application of the present invention relates to a control unit configured to carry out the methods described above, and / or an automated apparatus comprising the control unit. Optionally, the control unit includes and / or is data-technically connected to the camera.

[0152] Further aspects of the present invention relate to computer programs configured to carry out the above-described methods when executed in a digital data processing device and / or control unit.

[0153] A further subject of the present invention is a machine-readable storage medium equipped with a computer program.

[0154] Further features, advantages, and effects of the present invention will become apparent from the following description of preferred embodiments and the accompanying drawings. [Brief explanation of the drawing]

[0155] [Figure 1] This is a flowchart of the entire method, including one embodiment of the method according to the present invention. [Figure 2] This is a schematic diagram of an optical model in 3D. [Figure 3] This is a schematic diagram of an optical model in 2D. [Figure 4] This is a schematic diagram of an optical model in 2D. [Figure 5] This is a schematic diagram of the coordinate system. [Figure 6] This is a schematic diagram of the coordinate system. [Figure 7] This is a schematic diagram of the coordinate system. [Figure 8] This is a schematic diagram of the coordinate system for the imaging model. [Figure 9] This figure shows the imaging model in Figure 8, which has an inclined code plane. [Figure 10] This is a diagram illustrating the code array. [Figure 11] This is a further diagram illustrating the code array. [Figure 12]A diagram showing a dot grid of a code array with two示例性 filled-in read fields. [Figure 13] A diagram showing various angular orientations of the read field. [Figure 14] A diagram showing an example regarding the read field. [Figure 15] A diagram showing an exemplary structure of a parcel within the code array. [Figure 16] A diagram showing a further exemplary structure of a parcel within the code array. [Figure 17] A flowchart of the validity check. [Figure 18] A diagram showing details of the validity check. [Figure 19] A diagram exemplarily showing a camera image with an identified start base and a read field. [Figure 20] A diagram showing an example regarding a basic symbol matrix with entered data of a face. [Figure 21] A flowchart for identifying the start base. [Figure 22] A diagram showing a plurality of predefined start positions within the image field of a camera. [Figure 23] A diagram explaining a method for identifying the start base. [Figure 24] A diagram explaining a method for identifying the start base. [Figure 25] A flowchart of the function Center_Pos. [Figure 26] A diagram explaining the function Center_Pos. [Figure 27a] A diagram explaining the function Nearest_dot. [Figure 27b] A diagram explaining the function Nearest_dot. [Figure 28] A flowchart of the flow for detecting all dots / basic symbols within the read field. [Figure 29a]This is a diagram explaining the flow for detecting all dots / basic symbols within the reading field. [Figure 29b] This is a diagram explaining the flow for detecting all dots / basic symbols within the reading field. [Figure 29c] This is a diagram explaining the flow for detecting all dots / basic symbols within the reading field. [Figure 29d] This is a diagram explaining the flow for detecting all dots / basic symbols within the reading field. [Figure 30] This is a diagram explaining the detection of basic symbols within a distorted grid. [Figure 31] This is a diagram explaining the parallelization process. [Figure 32] This is a flowchart for classifying basic symbols. [Figure 33] This is a flowchart for identifying the rough position. [Figure 34a] This is a diagram showing a basic symbol matrix with entered data and a reading track. [Figure 34b] This is a diagram showing a basic symbol matrix with entered data and a reading track. [Figure 35] This is a diagram explaining the decoding of the rough position. [Figure 36] This is a diagram explaining the compensation line for a linear function. [Figure 37] This is a diagram explaining the straight line angle. [Figure 38a] This is a diagram explaining the compensation line and the linear function. [Figure 38b] This is a diagram explaining the compensation line and the linear function. [Figure 38c] This is a diagram explaining the compensation line and the linear function. [Figure 39] This is a diagram explaining the identification of the fine position and the rotation angle phiz (Z - rotation angle). [Figure 40] This is a flowchart for identifying the camera distance and the pitch angle.

Embodiments for Carrying Out the Invention

[0156] This method is disclosed as executing an algorithm for accurately determining the absolute 6D position of camera 1 based on a flat code sequence 2 captured by camera 1, the algorithm receiving a camera image 3 as input information.

[0157] Code array 2 is used simultaneously as an analog and digital scale in two dimensions, X and Y. Code array 2 consists of multiple symbols arranged in a regular grid. A binary code is preferred, consisting of two symbols arranged in a square grid, namely a small round dot for digital 0 and a larger dot for digital 1. The dot type contains digital information, while the dot center contains analog information.

[0158] Camera 1 detects a segment of code sequence 2 and transmits the camera image 3 to a computer or any data processing device. The algorithm selects a reading field within the camera image 3 with the smallest size code cell (e.g., 7x7 dots) and calculates the position of camera 1 relative to code sequence 2 in 6 dimensions. To this end, the method utilizes both digitally encoded positional information and the precisely measured center positions of all dots within the measurement field or reading field. These dots form a grid, which is distorted by the camera's viewpoint and the distortion of the objective lens.

[0159] The method for position detection includes the following steps (see Figure 1): Step 100: Load camera image 3 into the computer's main memory; Step 200: Optional: Perform an initial inspection of image quality based on characteristic data such as brightness and contrast; Step 300: Evaluate the XY rough position and optionally evaluate the z rough angle; Step 310: Search for the starting field (launch pad) of 3×3 dots within the reading field; Step 320: Detect the dots within the reading field of the camera image; Step 330: Accurately measure the center position and area of the dots; Step 340: Optionally: Mathematically correct the objective lens distortion by paralleling the dot positions. As a result, the curved lines of the dot grid within the camera image 3 are shifted to straight lines; Step 350: Classify the dots based on their area and associate them with the binary digits 0 and 1; Step 360: Read the digital code in the axial directions X and Y, and identify the absolute coarse positions in the X and Y directions and, optionally, the absolute coarse position in the φ Z direction as an evaluation of the z coarse angle.

[0160] Step 400: Evaluate the distortion; Step 410: Fit the compensation lines (beams) to each row and column of the dot grid in the reading field; Step 420: Fit the compensation function to the first straight line bundle by interpolating all the lines in each row of the code grid, and fit the second compensation function by interpolating all the lines in each column of the code grid. Each straight line bundle is completely described by six interpolation parameters (bunch data). From this, it is also possible to calculate the interpolated straight line located between the two measured straight lines using the interpolation function.

[0161] Calculate the 6D camera position from the characteristic values that describe the distorted grid in the camera image obtained from the measured dot positions. The equations for calculating the position are obtained from the optical inverse imaging model of camera 1 and the laws of geometric optics: Step 500: Evaluate the XY fine position: The positions in the X and Y directions are calculated from the XY coarse positions and the orientation of the grid with respect to the orientation of the optical axis (image center).

[0162] Step 600: Evaluate the Z-angle from the angle of the (interpolated) line at the center of the image. Step 700: Evaluate the Z position and XY angles; Step 710: Derive interpolated characteristic values ​​from bunch data describing a grid with perspective distortion at the center of the image: • Grid dimensions of the lines in two bundles of lines • Angular divergence of the lines in two bundles of lines; The camera position in the Z direction is primarily determined by the dimensions of the straight grid at the center of the image; Camera angle φ X and φ Y This is primarily determined by the angular divergence of two line bundles at the center of the image.

[0163] The iterative algorithm generates variables Z and φ. X and φ Y The system of equations connecting these factors is solved numerically.

[0164] Step 800: Outputs 6D position information and validity information as an optional choice obtained from the results of numerous diagnostic functions in the flow.

[0165] The algorithm concentrates camera image data into a small number of data points relevant to localization with minimal computational cost, improving accuracy through interpolation and behaving robustly even when noise is present in the camera image. The following flow positions (the numbers are examples) are particularly important: Flow position I: Input data is a camera image (e.g., 200 x 200 pixels) 40,000 values Flow position II (step 300): Camera image data is converted to dot data. 2250 values Flow position III (step 400 / 500): Dot data changes to beam data 90 values Flow position IV (step 400 / 500): Beam data changes to bunch data. 12 values Flow position V (steps 500 / 600 / 700): Bunch data changes to position 6D. 6 values Flow positions II-V have the following special characteristics:

[0166] Flow position II: a) By mathematically modeling the perspective-distorted code grid within code array 2, dots in camera image 3 are detected at high speed. This method is based on previously measured dot positions. →Advantages: Dots are found quickly, eliminating the time-consuming search within the pixel grid. It is robust to noise in the camera image; that is, a dot is classified as "invalid" if it is not found at the expected location. Invalid dots are not processed further, but generally do not interfere with the subsequent sequence. b) A stack-based algorithm for fast and robust searching of dots in the code grid of code array 2. →Advantages: The search method is fault-tolerant because it completely surrounds these false dots without being interrupted by them. It is accurate (high yield of dots found per read field) and robust (dots with many valid neighbors are preferentially detected). c) Center_Pos function: A fast method for determining the position and area of ​​a dot in a camera image with sub-pixel accuracy. →Advantages: Fast (fewer pixel accesses), accurate (due to sub-pixelation), and robust (uses dynamic contrast thresholding instead of grayscale thresholding). d) Instead of the entire camera image, the dot positions are parallelized. →Advantages: It is fast (it parallelizes only 225 positions instead of all 40,000 pixels in the camera image). e) Locally normalize the dot area relative to the area of ​​the dot grid cells. The normalized dot area is used to classify the dots. →Advantages: Digital codes are read robustly (dot classification is resistant to perspective distortion and changes in distance between the camera and the dots). f) Digital code reading, error detection, and partially error correction are performed redundantly. →Advantages: Low error rate when noise is present in the camera image, and correct readings are performed even if invalid dots are present.

[0167] Flow positions III and IV: g) The dot positions extracted from camera image 3 are compressed as bunch parameters and accurately represented (only 12 values). →Advantages: It is fast (due to the small amount of data), accurate (through two-stage averaging and best-fit interpolation: from dots to compensation lines, and from compensation lines to line bundles), error-tolerant, and robust (by excluding invalid dots from subsequent processing).

[0168] Flow position V: h) A mathematical method for converting 12 bunch parameters to 6D positions.

[0169] As an advantage, some or all of the following improvements will be achieved: • Complete 6D position measurement, absolute position information; • All dimensions are detected simultaneously using a single measurement process (within the camera image); • The position measurement rate is fast and the delay time is short. This method requires only a few calculation steps on the computer. Using an embedded computer, a typical measurement rate of approximately 100Hz to 10000Hz can be achieved, and therefore the sensor can be used, for example, in a closed attitude control circuit. This method achieves high read speeds by minimizing the number of camera pixel accesses. Time-consuming planar camera image manipulation is avoided. Furthermore, the amount of data in each processing step is significantly reduced. • Redundant reading of point codes ensures high reading reliability. Position detection is possible even when noise is present in the camera's image field, such as in cases of localized occlusion, invalid dots, or unfavorable lighting conditions. The validity check of the processing step detects an incorrect state and prevents the output of unreliable position values; • High accuracy in position measurement across all degrees of freedom. This is achieved through a variety of means, including: ○ Averaging across a large number of dots per camera image, for example, more than 100 or 1000 dots; ○ Classify the dots according to their effectiveness and exclude ineffective dots from subsequent processing; ○ Identify the position of the dot edges with sub-pixel precision; ○By using a contrast threshold instead of a gray value threshold, robustness to localized changes in image brightness is increased; ○ Because the dots are round, there is virtually no centering error when the code plane is tilted or rotated; · φ Z The measurement range is not limited; The measurement range in X and Y is virtually unrestricted. For example, using a code with 9x9 dot code cells and a 2.5mm dot grid, a measurement range of 550,000 km x 550,000 km can be achieved. • Angle measurement range φ Χ ,φ Y However, it has a wide range of approximately + / - 50° and high precision. It can be extended up to 360° by a spatially distributed code array on the object; • To calculate the camera's position from camera image data, an inverse imaging model based on geometrical optics is used; This method can be used even without active lighting, and therefore, for example, it can be used to process camera images taken under ambient light using a smartphone; • The error rate is minimized. This method identifies the validity of position measurements by checking each processing step for errors and validity using different diagnostic methods. Validity is output along with the position measurements. In this case, the goal is to output only valid measurements for subsequent processing.

[0170] Further advantages are as follows: Specifically, a single measuring sensor configured as Camera 1 detects the position of one or more objects in six dimensions. Unlike systems with multiple distributed sensors to detect individual degrees of freedom, the overall installation cost and complexity of the system are reduced, resulting in cost advantages; By detecting all six degrees of freedom, the system components are reduced. For example, a rotary angle encoder typically requires a rotary bearing for the measuring axis to prevent measurement errors caused by lateral displacement of the code disk. According to the proposed method, since the complete 6D position information of the code disk is detected simultaneously as a code sequence, mechanical guides can be omitted. Lateral displacement of the sensor will not cause errors in angle measurement. This reduces the cost and mechanical complexity of the position measurement system. • It is possible to realize positioning systems where it is structurally impossible to assign position sensors to multiple axes. For example, it is possible to realize a floating, flat robot that can be positioned in six dimensions without providing a structural connection between the robot and the stator located beneath it; • By detecting multiple dimensions simultaneously, measurement errors that could occur due to differences in measurement timing when multiple sensor systems are distributed are reduced. The cost of a sensor is largely independent of the number of degrees of freedom detected. Therefore, sensors can be advantageously used even in applications requiring fewer than six degrees of freedom. The additional information provided enables additional functions, such as system self-diagnosis or continuous observation of the operating state ("state monitoring"). This method is scalable, for example, by changing the grid dimensions of the code array and adapting the imaging optical system to the code array. The resolution can be varied over many orders of magnitude, for example, from the nanometer range (e.g., nanometer positioning systems) to the meter range (e.g., automated landing of aircraft or drones at airports where the airfield is marked by a code array). The resolution-to-measurement range ratio can involve many orders of magnitude. For example, if a path measurement system with a resolution of 10 nm is combined with a code sequence 10 m long, the resolution-to-measurement range ratio is 1:10. 9 is; Sensors using this method are easy to install and can be configured by parameterizing them for specific applications, thus offering high flexibility in use. This is particularly advantageous when usage conditions change frequently. This method can read additional information contained in the location code. For example, it can read object identification data in addition to the location; When image capture is performed periodically, this method provides one independent position estimate for each individual image, and this position estimate does not depend on preceding information from previous images. Therefore, the measurement rate corresponds to the refresh rate.

[0171] According to this method, the position of at least one camera 1 relative to at least one object 8 can be determined with six degrees of freedom, and at least one code sequence is attached to the surface of each object 8. Camera 1 detects the code sequences 2 on the object 8 and displays these code sequences 2 completely or partially in the camera image 3. The 6D position of the object 8 is calculated from the camera image 3 using the proposed method and further conventional mathematical / technical methods. Code array 2 forms a digitally encoded planar scale. Code array 2 includes multiple different symbols, preferably circular symbols (dots), which are arranged in a regular, preferably square, grid, allowing for positioning in six degrees of freedom.

[0172] Camera 1 includes at least the following: • Imaging sensor element, usually a camera chip, especially an imaging device 12; • An imaging system, particularly the objective lens 9, that forms a clear and high-contrast image of the scale onto the sensor element. The imaging system includes a wide-angle objective lens, especially the objective lens 9, because a central projection is required to identify all six degrees of freedom; • An interface for outputting camera image data and / or camera image 3. Optionally, the camera system of camera 1 is included; • A lighting system specifically for illuminating the code sequence. This allows camera 1 to be more independent of ambient lighting conditions. Shorter measurement times are possible during flash operation, enabling the detection of fast-moving objects. Light-emitting diodes, for example, are used as the light-emitting means; • Means for blocking external light, such as an aperture or optical filter. A spectral filter may be provided in the light path of the imaging system to ensure that only light from a limited wavelength range reaches the sensor element. Ideally, a monochromatic illumination means with wavelength characteristics similar to those of a color filter is used. Additionally, a computing system is required, for example, as a digital data processing device for information processing, configured as a control unit. The computing system receives digitized camera image data of camera image 3 as input information, and on the output side, provides the identified 6D position of the identified object and optionally an evaluation of the validity of the position measurement.

[0173] In the computing system, this method is implemented as an algorithm. The algorithm is executed in response to external requests, or periodically at fixed time intervals, for example, to detect the movement trajectory of an object (tracking).

[0174] The computing system may be configured as, for example, an embedded system, a parallel computer, a GPU, an FPGA, an ASIC, or a cloud system. A smartphone can also be used as the overall system for position detection, and its built-in camera 1 may be used for image capture, sometimes with active lighting. This method may be implemented in a smartphone application and may be implemented embedded in a smartphone or the cloud.

[0175] Figure 2 shows a typical apparatus for determining absolute position with six degrees of freedom. Camera 1 captures a camera image 3 of a flat code array 2. The method according to the present invention calculates the 6D position of the camera in the coordinate system of the code array 2 from the perspective-distorted camera image 3, and generates a position vector (r X ,r Y ,r Z ) and angular vector (φ X ,φ Y ,φ Ζ Output as ).

[0176] coordinate r X and r Y This is the Cartesian coordinate system (X) of code array 2. C ,Y C It is shown in ). Coordinate r X and r YThis describes the intersection point 4 between the optical axis 5 of camera 1 and the plane on which the code array is located. Since the optical axis 5 is perpendicular to the plane of the imaging device 12, the optical axis 5 can be described by point 7 within the camera image 3.

[0177] This method identifies the intersection point (X,Y) even if code array 2 is displayed only at the edge of the image field and not at the location of optical axis 5.

[0178] distance r Z It is located on the optical axis 5 and extends from intersection 4 to the optical center of the objective lens of camera 1. Since the optical axis 5 is not always perpendicular to the code array 2, r Z Generally speaking, (X C ,Y C ) together with a non-orthogonal coordinate system. Using the angle vector, the position vector (r X ,r Y ,r Z ) can be converted to a Cartesian coordinate system.

[0179] According to this method, multiple simultaneously captured objects 8 can be located within the camera's image field, even if these objects 8 partially overlap. In order for the location to be read, at least one region within the size of the code parcel (e.g., 7x7 dots) in the camera image 3 must be recognizable.

[0180] Figures 3 and 4 schematically show the light ray path of camera 1. In Figure 3, camera 1 equipped with an objective lens 9 and an imaging device 12 can be seen, and its line of sight is directed downwards onto the code plane of the code array 2. The optical axis 5 is indicated as a vertical dashed line, and the focal points of the objective lens 9 are indicated as points 10 and 11.

[0181] The vector arrow G on the code plane of code sequence 2 is imaged onto the imaging device 12 as vector arrow B according to the laws of photooptics. The observation ray 14 intersects the optical axis 5 at a point referred to herein as the optical center 13 of the objective lens 9.

[0182] Figure 4 shows the complete ray path, with the objective lens 9 simplified as a lens. Ray optics, with the theorems of parallel lines and ratios of line segments, and the lens equations, forms the basis for a mathematical method for position determination. In this case: B / G = b / g and 1 / f = 1 / b + 1 / g The following holds true, however, B Image size b Image width G Object size g Object width f focal length That is the case.

[0183] Figures 5, 6, and 7 show the coordinate systems involved.

[0184] The two-dimensional coordinate system 15 of the imaging device 12 is the axis (X) of the camera chip. I ,Y I It is stretched by ). The position on the imaging device 12 is indicated by pixels, and the pixel rows and columns of the camera chip are numbered consecutively. In this example, an imaging device 12 with 200 x 200 pixels is assumed. From the numbering, integer X and Y position values ​​can be obtained for each pixel. In the method of interpolating gray values ​​between adjacent pixels by sub-pixelation, real numbers may be generated as camera positions during evaluation. To convert the unit of position from [pixels] to [m], the distance p in units [m / pixel] between adjacent pixels on the camera chip is used. grid However, it is multiplied by the position. Pixel distance p grid This is one of the characteristics of the imaging device 12. Each pixel provides an integer gray value.

[0185] Coordinate system 16(X) of code array 2C ,Y C ) is stretched by the main axis of code array 2, which corresponds to a square grid of dots. A third axis Z is perpendicular to code array 2. C It is extended to a 3D coordinate system centered on Z, and code array 2 is Z C It is positioned at =0. The unit is [dot], i.e., the numbering of consecutive dot rows and dot columns, or (dot grid distance c in units [m / dot]). grid [m] is the result after multiplication. Code array 2 is the coordinate system (X C ,Y C It uniquely encodes the local location in ).

[0186] The 3D camera coordinate system (X,Y,Z) 17 is fixedly linked to camera 1. The origin of the 3D camera coordinate system (X,Y,Z) 17 is located on the optical axis 5 between the imaging device 12 and the code array 2, at a distance of twice the image width b from the imaging device 12. The optical axis 5 forms the Z axis, and camera 1 is oriented in the Z direction.

[0187] The coordinate system for the imaging model in Figure 8 corresponds to the camera coordinate system 17(X,Y,Z). In this case, b is the image width of camera 1. The image plane is located at a height of (0,0,2b) and is parallel to the X / Y plane of the camera coordinate system 17. As a model representation, a further "virtual" image plane 18 can be constructed within the (X,Y) plane of coordinate system 17, and the virtual image is mirrored point-symmetrically with respect to the real image on the imaging device 12, with respect to the image center.

[0188] The code plane of code array 2 is located at height (0,0,z0), where (z0<0). When the camera orientation is (0°,0°,0°), the code plane is located parallel to the X / Y plane of camera coordinate system 17, and the axis (X) of coordinate system 16 of code array 2. C ,Y C ) is the axis (X) of the coordinate system 15 of the imaging device 12. I ,Y I It points in the same direction as ).

[0189] This method, ·Translation:(r X ,r Y ,r Z ) in, and ·Rotation:(φ X ,φ Y ,φ Z ) in Identify the 6D position of the camera.

[0190] The rotation center related to the tilt angle phix and rotation angle phiy of camera 1 is located at the point (0,0,z0). The roll angle phiz is measured around the optical axis 5.

[0191] Figure 9 shows an imaging model with a tilted code plane for code array 2. Value r Z r is the distance between the rotation center (0,0,z0) and the optical center of the objective lens (0,0,b), and therefore r Z The equation = b - z0 holds true.

[0192] Starting from the theorems of parallel lines and the ratio of line segments, and geometrical optics, the point (x0, y0) in the code plane and the point (B) in the image plane X ,B Y The image formation on ) can be mathematically described as an imaging equation in camera coordinate system 17: • Optical center 13 of objective lens 9:

number

number

number

number

number

number

number

number

number

number

number

number

[0193] 2.4.3 Code Array This method requires a planar encoded scale within a code array 2, from which a sub-region is read using an imaging sensor, such as a camera 1, thereby allowing the orientation of the sensor and / or camera 1 relative to the scale to be determined from the image information in up to six spatial directions. Further details regarding the code array 2 are evident in the applicant's German Patent Application Publication No. 102016216221, the contents of which are incorporated herein by reference, particularly with respect to embodiments of the code array and decoding and variables.

[0194] The encoded scale is applied to a surface as a sensor-readable marking formed as code array 2, the surface extending substantially in two dimensions, but which may be curved. For better explanation, it is assumed below that the encoded scale is printed on a flat surface as an optically readable pattern, but the claims are not limited to these marking and sensoring principles, nor to planes.

[0195] The encoded scale of code array 2 is formed by arranging multiple different basic symbols in a regular grid. Each basic symbol carries two pieces of information: its shape encodes digital information, and its area centroid marks a specific position on the surface.

[0196] In a simple example, digital information is encoded in a binary number system with base b=2. In this case, only two basic symbols 20 are used, for example, a small circle and a large circle that symbolically represent the values ​​"0" and "1". The area centroids (centers of the circles) of the two basic symbols 20 mark grid points on the surface.

[0197] The area centroid of the 20 basic symbols is, for example, the basic distance c between adjacent symbols in the X and Y directions, in a square grid as shown in Figure 10. gridBy placing this, a periodic two-dimensional pattern is formed on the scale plane of code array 2.

[0198] By dividing the plane into multiple parcels 19 of equal size that fill the surface, multiple basic symbols within one parcel 19 are combined into a single logical unit. Preferably, square parcels 19 are used. Figure 10 shows a dot grid with square parcels 19, each capable of containing 7 x 7 binary symbols, which corresponds to a maximum information content of 49 bits. The lines and squares drawn in Figure 10 are for illustrative purposes only and are not shown in the actual code array 2. Figure 11 shows an exemplary code array 2 without lines, with basic symbols 20 within the parcels 19.

[0199] Each parcel 19 is similarly marked by a parcel symbol 21, so that the regular arrangement of the basic symbols 20 is superimposed by the regular arrangement of the parcel symbols 21. For example, the parcel symbols 21 can be represented by omitting the basic symbols 20.

[0200] In Figures 10 and 11, the parcel symbol 21 consists of empty grid points located in the center of each parcel 19 within the point grid. The parcel symbol 21 enables recognition of the orientation of the parcel 19, thereby ensuring that the basic symbols 20 are read in the correct order along the reading track. Since the parcel symbol 21 occupies grid space, the information content of a single parcel 19 is reduced to 48 bits in this example.

[0201] The reading field 22 is a field on the encoded scale of the code array 2 that has the size of at least one parcel 19. The reading field 22 is connected to the point grid but not to the grid of the parcel 19.

[0202] Figure 12 shows a point grid of code array 2 with two exemplary filled-in reading fields 22. The orientation of the reading fields 22 in the coordinate system of the point grid is defined by their center position 23. The orientation of the reading fields 22 is defined by the base distance c grid It can be changed in integer increments.

[0203] Furthermore, the reading field 22 has an angular orientation that can change in 90° increments relative to the coordinate system of the point grid in the case of a square grid. In Figure 12, the angular orientation is visualized by markings at the corners of the reading field. Possible angular orientations are shown in Figure 13. The reading field 22 illustrated in Figure 12 is uniquely described by the following information: Left side reading field 22: Position = (4, 11); Angle orientation = 0° Right-hand reading field 22: Position = (12, 6); Angle orientation = 90° The code of code array 2 is constructed such that the basic symbol 20 in the reading field 22 contains a sufficient amount of information to digitally encode the location (X,Y) and direction of the reading field 22 in the coordinate system 16 of code array 2. X and Y are the basic distance c grid The position is shown as an integer multiple of (rough position). Basic distance c grid The precise position in fractions and the exact angle in fractions of 90° are not digitally encoded, and are determined by the precise positioning of the base symbol 20 in the camera coordinate system 17.

[0204] Figure 14 illustrates a reading field 22 with a 15x15 space for basic symbols 20. This reading field 22 provides more information than a reading field 22 of 7x7 symbols, which is the minimum size required for one parcel 19. The redundant information is used for error recognition and / or error correction.

[0205] Figure 15 illustrates the structure of a parcel 19 with a 7x7 grid of points. A parcel symbol 21 is located in the center of parcel 19. Two fields 24 characterize the X-parcel region, which displays the subsequent code relating to the position in the X direction. The X-parcel region contains 24 grid points and, accordingly, represents a code with a length of 24 bits.

[0206] To read the code, the basic symbols 20 are read from left to right, column by column, and from top to bottom within each column. The reading order (reading track) within the X parcel area 24 is determined by the grid point coordinates (X C ,Y C ):(0,6);(0,5);(0,4);(1,6);(1,5);(1,4);(2,6);(2,5);(2,4);(3,6);(3,5);(3,4);(3,2);(3,1);(3,0);(4,2);(4,1);(4,0);(5,2);(5,1);(5,0);(6,2);(6,1);(6,0).

[0207] The two fields 25 characterize the Y parcel region, which displays the subsequent codes relating to the position in the Y direction. The reading order corresponds to the reading order of the X parcel region 24, but rotated 90° counterclockwise. That is, the reading tracks extend from bottom to top row by row, and from left to right within each row.

[0208] The code from the X parcel area 24 (X code) is a subsequence of a whole sequence of g digits, with a length of t = 24 digits, where g is considerably larger than t. The number g is large enough to completely fill the X parcel area 24 of all adjacent parcels 19 in the X direction. In particular, g is greater than 50, especially greater than 1000, and especially greater than 1,000,000. The contents of the X parcel area 24 of adjacent parcels in the Y direction are identical. The whole sequence is structured such that each subsequence of t consecutive basic symbols 20 is included in the whole sequence exactly once when read in the forward direction, and each subsequence read in the backward direction is not included in the whole sequence when read in the forward direction.

[0209] The code (Y code) from Y parcel area 25 is shown inverted, meaning each digit z is replaced by the digit (b-1-z). In the case of a binary system with b=2, this corresponds to bitwise inversion. The inverted Y code is a subsequence of the whole sequence with a length of t=24 bits, and the same whole sequence can be used as for the X code. The inverted Y code also appears only once in the whole sequence and does not appear in the whole sequence when read backward. The contents of the Y parcel area 25 of adjacent parcels 19 in the X direction are identical.

[0210] The whole sequence is constructed such that the sum of the digits of any subsequence is less than half the maximum possible sum of the digits of any subsequence. In particular, the sum of the digits of any arbitrary subsequence taken from the whole sequence is less than half the maximum possible sum of the digits of any subsequence. For example, the sum of the digits of any subsequence of length t for a number system with base b is less than q = t·(b-1) / 2.

[0211] A subsequence of the overall sequence encodes a coordinate value, for example, the X-coordinate of the starting position of the reading process. Similarly, a subsequence of the inverted overall sequence encodes a Y-coordinate, for example, the starting position read in the subsequence. The X and Y codes read within parcel 19 are associated with the X and Y coordinates of a reference point within parcel 19, for example, the center of the reading field.

[0212] Each reading field 22 contains exactly one parcel symbol 21, t basic symbols from the X parcel area 24, and t basic symbols from the Y parcel area 25. From the position of the parcel symbol 21, the orientation of the parcel grid, and consequently the orientations of the X parcel area 24 and Y parcel area 25 within the reading field 22, and the corresponding reading order can be derived. The basic symbols 20 read according to the reading order generate a t-digit sequence that is a subsequence of the whole sequence (possibly after inversion).

[0213] If the orientation of the reading field 22 is unknown, it is initially unclear which of the two axes is the X-axis and which is the Y-axis. This can be determined by calculating the sum of the digits of the read code and comparing it with q. That is, the sum of the digits of the X code is less than q, and the sum of the digits of the Y code is greater than q. From the overall sequence and the direction of the referenced read code (forward or backward), the orientation of the coordinate axes of the reading field 22 relative to the coordinate axes of the code sequence 2 can be uniquely derived.

[0214] In this way, the orientation and direction of the reading field 22 in the coordinate system 16 of the code array 2 can be determined; that is, the orientation can be determined in integer increments of the grid width, and the direction can be determined in integer increments of 90°.

[0215] On the left side of Figure 16, a parcel 19 is shown, with correspondingly 9 rows and 9 columns. In addition to the X-parcel area 24 and Y-parcel area 25 representing the position code for each axis of the coordinate system 16, a parcel area 26 for additional data is provided. The parcel area 26 for additional data contains (5 × 5) grid points, the central grid point of which is left empty for the parcel symbol 21, thus leaving 24 grid points, each capable of accommodating one basic symbol 20. In this way, 24 bits of additional information, which are not necessary for position identification and are read in a specified order, can be represented within each parcel 19. On the right side of Figure 16, a further code array 2 with additional data is shown. Here, four parcel areas 26 are provided, each with 10 grid points for additional data, so a total of 40 bits of additional information are available for each parcel 19.

[0216] The algorithm for localization is shown in Figure 1 as a progression of data processing steps, and this progression will be described in detail below based on an example. Standard methods for mathematics and image processing, as well as for diagnosis and error recognition, are not described in detail.

[0217] This method is optimized to improve localization accuracy and enable high-speed implementation. The data volume is dramatically reduced through these steps, which further facilitates high-speed implementation.

[0218] At flow position I in step 100, digital image data as a matrix of gray values ​​is transmitted from camera 1 to the computer. Step 200 is used to roughly check the validity of the image data based on characteristic values. Further steps for image processing may follow, for example, to prepare the image data or to segment it into individual code regions. These steps are not shown in detail here. The data capacity is 40,000 pixels in the case of a 200x200 camera image 3, and therefore 40,000 bytes in gray values.

[0219] At this flow position, symbols (dots) contained within the image are located and entered into a dot matrix corresponding to the size of the reading field (here, 15x15 dots), according to their respective placement within the dot code grid. Each symbol is measured for its position and area. The position is parallelized to correct for objective lens distortion. The area of ​​the symbols is normalized to remove the effects of perspective distortion. Based on the normalized area, the type of symbol is classified. This reduces the data size to 225 dots with metadata, and consequently to 9000 bytes. To classify unoccupied grid points or incorrectly imaged symbols as such, a local model of the dot code grid in the camera image is created from the positions of successfully identified symbols. Within the reading field, symbols are searched for at all grid points in the model. Due to local occlusion, image errors, or image field boundaries, not all symbols can always be identified. In such cases, their corresponding grid positions are classified as "invalid".

[0220] At flow position III, compensation lines (beams) are fitted to the rows and columns of valid dots in the reading field 22. Each compensation line is described by the angle of the line in the image field of camera 1 and its intersection with the X or Y axis of the image field coordinate system. Each reading field 22 supplies two bundles of compensation lines, corresponding to the two principal axis directions of the code array 2. In this example, each line bundle (bunch) contains a maximum of 15 lines. This reduces the position data of 15 × 15 = 225 dots to 15 + 15 = 30 compensation line data.

[0221] At flow position IV, the lines of each bundle are interpolated by two quadratic polynomials. That is, one polynomial describes the angle of the line, and the second polynomial describes the axis intersections of the lines in the bundle of lines. The polynomial parameters of the two bundles are expressed by 12 real values, which is equivalent to a reduction to one-fifth compared to the linear representation.

[0222] At flow position V, a mathematical inverse imaging model is also used to calculate the 6D position of camera 1 from the polynomial parameters of two linear bundles. Additionally, the validity of the position values ​​is estimated by evaluating numerous diagnostic results from individual steps of the program flow. The 6D camera position and validity are output to a higher-level system in step 800.

[0223] Step 200 - Image Data Inspection / Validity Test Based on statistical characteristic data, it is checked whether an evaluable camera image 3 exists (see flowchart in Figure 17). In this example, image brightness B and contrast C are estimated and checked to see if they comply with predetermined limits. If they deviate from the predetermined limits, further evaluation is stopped and the result is classified as invalid.

[0224] To save calculation time, only a few pixels, for example, only the gray values at the intersections of the straight lines and the circles within the intersection pattern 27, are incorporated into the determination of the characteristic numbers (FIG. 18).

[0225] In the lower step 210, a set of pixels specified according to any intersection pattern 27 is selected. The pixels at the intersections within the intersection pattern 27 are used. Thus, G i is the gray-scale value of the pixel at the position i, i = 1 ··· n.

[0226] In the lower step 220, the image luminance B and the contrast C are estimated.

[0227] B = mean(G i ) C = (max(G i ) - min(G i )) / (max(G i ) + min(G i ) + 1) In the lower step 230, the image luminance B and the contrast C are evaluated according to the following conditions: (B ≤ B min ) or (B ≥ B max ) (C < C min ) are evaluated by.

[0228] If one of these conditions is satisfied, further evaluation is interrupted and the result is classified as not valid.

[0229] Detection of dot data / Steps 310 to 350: The object is to identify, measure, and classify dots as basic symbols 20 within a reading field 22 of a predetermined size (here, 15 × 15 grid points). As a result, a two-dimensional dot matrix 29 is output, or generally a basic symbol matrix with dot data is output. In this case, the indexes of the matrix 29 are associated with the rows and columns of the code array 2. A characteristic value is specified for each dot.

[0230] The variable Dot.Typ contains the classification result. If Dot.Typ is positive, the dot was evaluated as valid: Dot.Typ=2: Large dot; Logical "1"; • Dot.Typ=1: Small dot, logical "0"; Dot.Typ=0: Missing dot; parcel symbol 21; If Dot.Typ is negative, the dot could not be uniquely identified.

[0231] In partial step 310, the start base / start field (launch pad) 28 is first searched for within the image area of ​​the camera image 3 and / or within the read field 22 by the dot code of the code array 2. The start base / start field (launch pad) 28 is understood here to be, for example, a field of 3x3 adjacent dots of type 1 or type 2.

[0232] Figure 19 illustrates a camera image 3 with an identified starting base 28 and a read field 22, in which a dot as a basic symbol 20 is searched for (starting from the starting base 28) within this camera image 3.

[0233] In partial step 320, a local model of the dot grid around the starting base 28 is constructed using the starting base 28. Further dots are searched for in adjacent grid spaces predicted by the model, accurately measured if successful, and entered into the dot matrix 29 as valid dots. This process is repeated periodically, thereby recording more and more valid dots around the starting base 28 until the 15×15 dot reading field 22 is fully detected. This search process is robust to local reading errors; that is, if individual dots cannot be uniquely identified or if they extend beyond the edge of the image field, those dots are characterized as invalid in the dot matrix 29 and excluded from subsequent processing. Figure 20 shows the result of the successfully classified dot matrix 29 based on the camera image 3 in Figure 19.

[0234] In partial step 330, the center position and area of ​​the dot are measured.

[0235] Partial step 340 converts the measured positions of all valid dot locations in the reading field 22 into parallelized coordinates. This step is used to compensate for the distortion aberration (barrel distortion) of the wide-angle objective lens. After parallelization, points that lie on a straight line in the code coordinate system 16 also lie on a straight line in the parallelized image in the coordinate system of the imaging device 12. Optical parallelization can also be performed using the corresponding objective lens 9 instead of computer-assisted parallelization; therefore, partial step 340 is an optional choice.

[0236] In sub-step 350, the dots are classified based on the measured data. Then, in sub-step 360, the digital codes are read in two spatial directions and converted using a code table into integer positional information and coarse directional information in 90° increments. The identification of the coarse position in the upper step 300 is then completed.

[0237] Regarding partial step 310 - Identify the starting base The flow for identifying the starting base 28 is shown in the flowchart in Figure 21.

[0238] Sub-step 310.1: First, a starting position 30 is defined for searching for a first dot in the starting base 28. Figure 22 shows several predefined starting positions 30 in the image field of camera 1. The search begins at one of these starting positions. If the flow fails (for example, due to noise in the image field), a second search is started at the second starting position, and so on, until sub-step 310.1 can be successfully completed. If all starting positions have been used and the search has not been successful, sub-step 310.1 is terminated with a negative result.

[0239] Lower step 310.2: Starting position 30

number

number

[0240] Lower step 310.3: Next,

number

number

[0241] Sub-step 310.4: Then, the connection vector

number

[0242] Lower step 310.5:

number

number

number

[0243] Sub-step 310.6: Three points

number

number

number

number

number

[0244] Lower step 310.7: Left-handed coordinate system (condition:

number

number

number

[0245] Lower step 310.8: Vector

number

number

number

[0246] Substep 310.9: If no dot is found in this flow, it is assumed to be parcel symbol 21 (missing dot).

[0247] Next, in substep 310.10,

number

number

number

[0248] Subsequently, a lower step 310.11 for error recognition can be performed, and if an error is recognized in lower step 310.12, a new starting position 30 is used and the action is repeated starting from lower step 310.2.

[0249] As part of the flow described above, there is a challenge in identifying dots in the image field of camera 1, starting from an estimated starting position, and measuring the position and area of ​​those dots with sub-pixel accuracy. To achieve a short measurement time, this flow must be performed very quickly because a total of 225 dots must be identified within the reading field 22.

[0250] For this purpose, the function Center_Pos, described below, is used and is optimized for the minimum number of pixel accesses. Figure 25 shows a flowchart.

[0251] When the function Center_Pos is called, the estimated center position of the dot is determined.

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0252] starting position r min minimum search radius r max Maximum search radius ND Boolean variable (ND = "neighbor dot"): If the starting position is located inside a dot and ND = false, then the dot at the starting position is output. If ND = true, then the dot at the starting position is ignored, and the nearest neighbor dot is output.

[0253] starting point

number

number

[0254]

number

[0255] Partial step 320 - Detect dots in the reading field The flowchart in Figure 28 shows the flow for detecting all dots within the reading field 22. A pre-specified start base 28, for example, consisting of 3x3 dots, is used as input information. Furthermore, a step 330 for precisely measuring the center position and area of ​​the dots is incorporated into a sub-step 320.

[0256] In the first substep 320.1, the center position 32 of the reading field 22 is defined. This center position 32 is selected so that the reading field 22 has the largest possible coverage with respect to the image area on which the code is displayed, so that as many valid dots as possible can be detected. Typically, this is the case when the center of the reading field 22 is located at the center of the code area. When the code area occupies the entire surface of the image, as in this example, the center point 32 of the reading field is placed as close as possible to the center of the image (see Figure 29a).

[0257] At the start of this method, only the 3x3 dots of the starting base 28 are known. All other dots in the reading field 22 are still unknown; that is, the exact location and area of ​​all other dots have not yet been measured. The purpose of this method is to gradually measure and classify the unknown dots.

[0258] Sub-step 320.2: For each known dot, vector

number

number

number

number

[0259] For each unknown dot, the number of valid neighbors is counted. This number can be assumed to be between 0 and 8 (see Figure 29c). That is, the central dot position has 8 adjacent positions (indicated by dark colors).

[0260] At the start of the search, only the initial 3x3 dots are valid. Individual unknown dots with two or more valid neighbors are highlighted in Figure 29b). Figure 29d) schematically shows an unknown dot with two valid neighbors 20a and three valid neighbors as 20b. All other unknown dots (20c) still have no valid neighbors.

[0261] The algorithm is stack-based, and in lower step 320.3, all unknown dots with at least two valid neighbors are entered into the stack.

[0262] The stack is processed in a loop: • Lower step 320.3: The topmost entry is taken from the stack as an unknown dot and processed in the following steps; • Sub-step 320.4: The location of the unknown dot is estimated by averaging the extrapolated locations of all valid neighbors. If the unknown dot is located outside the expected reading field 22, it is not processed further. If the unknown dot is located inside the expected reading field 22, it is identified and measured with respect to its estimated location using the function Center_Pos; • Lower step 320.5: If an unknown dot cannot be identified, that unknown dot is classified as a defect (lower step 320.6), and the next unknown dot is removed from the stack; Step 330 / Substep 320.7: If a dot is identified, its position and area are measured using the Center_Pos function and entered into the dot matrix 29 as a valid new dot. For this dot, a vector

number

number

[0263] Partial step 340 - Parallelization process Image formation using a wide-angle objective lens can result in barrel distortion of the image field. Lines of dots that lie in a straight line within the code plane of code array 2 appear, depending on the context, on curved lines within camera image 3.

[0264] This curvature can be removed by a mathematical method, namely, parallelization. For computational time reasons, parallelization is applied only to the center position 32 of the dots, and not to all pixels of the input image. Figure 31 illustrates an unparalleled grid (33) and a parallelized grid (34).

[0265] Center of distortion

number

number

number

number

number

number

[0266] Distortion is vector

number

[0267] Part 350 - Classify the dots The small and large dots symbolically represent the binary values ​​"0" and "1". Dot classification is performed based on a threshold for the normalized area of ​​the dot. This flow is shown in the flowchart in Figure 32.

[0268] Perspective distortion has a strong effect on the area of ​​dots within an image field. For example, dots located further away will appear smaller when imaged, and circular dots on an inclined plane will appear elliptical.

[0269] Sub-step 350.1: To compensate for these effects, the area of ​​the dots is normalized based on the area of ​​the dot cells of those dots. A dot cell is a vector at the location of the dot.

number

number

[0270] Figure 30 shows a vector

number

number

Number

[0271] The normalized area of the dot is

Number

[0272] Lower step 350.2: The threshold is determined using a statistical method from the normalized area Dot.Area of all dots scaled .

[0273] Lower step 350.3: Subsequently, all dots are classified as follows: · Missing dot: type 0 · If the normalized area is less than the threshold: type 1; otherwise: type 2.

[0274] Lower step 350.3: If an error has occurred, the status is set to invalid, and if no error has occurred, the dot is valid.

[0275] Regarding partial step 360 - Reading the code After classifying the dots, the dot matrix 29 contains all the information to identify the 6D position.

[0276] The dot matrix 29 contains the following information:

Table 2

[0277] A flowchart illustrating this is shown in Figure 33. The coarse digital positions in the X and Y directions are determined by reading the X-bit chain and Y-bit chain within the dot matrix.

[0278] Sub-step 360.1: Parcel symbol 21 “empty dot” (type 0) is used as a reference point to identify the orientation of the bit chain in the dot matrix 29. If there are multiple entries with type 0, these entries can be validated against each other because “empty dots” are regularly repeated within the parcel grid (in this case, 7x7 dots).

[0279] Sub-step 360.2: Starting from the parcel symbol 21, the orientation of the reading track relative to the code on the two axes of the code plane is determined. At this point, it is initially unknown which sequences of the sequence correspond to the X axis, which sequences correspond to the Y axis, and in which direction (forward or backward) the code is read. The code is read along the reading track and stored as Code_0 and Code_1. Figure 34a) illustrates the dot types entered in the dot matrix. The parcel symbol (type 0) is highlighted in gray. Figure 34b) shows the reading tracks for the two axes additionally filled in. The reading tracks for the two axes are rotated 90° relative to each other, with the parcel symbol 21 forming the center of rotation.

[0280] The reading field (15x15 dots) is significantly larger than the parcel (7x7), allowing for redundant code reading. Specifically, a copy of Code_0 is located in an adjacent parcel (35a) in the j-direction, and a copy of Code_1 is located in an adjacent parcel (36a) in the i-direction.

[0281] Redundancy also exists in the length of the readable code. For localization, a code length of 24 bits in each direction is sufficient, i.e., the contents of a 7x7 parcel are sufficient. However, a larger readable field with 15x15 dots provides 51 or 54 bits in each direction. The redundant information is used to recognize and correct individual misread bits in Code_0 and Code_1.

[0282] In this example, the following code will be read:

number

[0283] Lower step 360.3: From Code_0, any interval with t=24 consecutive bits is selected and checked for completeness.

[0284] Sub-step 360.4: Subsequently, the sum of the digits of Code_0 is calculated. If this sum of the digits is less than t / 2 = 12, then Code_0 is the X code and Code_1 is the Y code. If this sum of the digits is greater than t / 2, then Code_0 is the Y code and Code_1 is the X code. The Y code is inverted. The X code remains unchanged.

[0285] Sub-step 360.5: Subsequently, Code_0 and Code_1 are searched for within the code table using an error-tolerant string search. The bit position with the best match is output as the search result; if the deviation is too large, the search result is evaluated as invalid. To convert Code_0 and Code_1 into integer location coordinates (X,Y), the bit positions of the codes are converted to spatial positions according to the code structure, as schematically shown in Figure 35: An arbitrary interval with t = 24 consecutive bits is selected and checked for completeness:

number

[0286] This segment is searched in the forward direction within the code table and found at bit position 37 according to the underlined portion:

number

[0287] Each X dot is associated with one bit position in the code table.

[0288] • The bit position corresponds to the starting point in the reading sequence within a single parcel.

[0289] Each Y-dot is associated with one bit position in the code table.

[0290] 360.6 Efficacy Test According to Figure 35, this corresponds to dot position Xc=10=Pos0. Similarly, the Yc value for Pos1 is determined. For both codes, the reading direction (Dir_0 and Dir_1) is also determined by searching for these codes in both directions within the code table. That is, 0 means it was found in the forward direction, and 1 means it was found in the backward direction. From the reading directions of Code_0 and Code_1, the rough orientation of the reading window 22 in 90° increments is determined according to the table below.

[0291] [Table 3]

[0292] As a result of partial step 360, the rough position of an integer (r X,int ,r Y,int ) and the orientation of the reading window 22 are provided.

[0293] Step 400 - Identify beam data To determine the precise location, the parallelized position data of the valid dots in the dot matrix 29 is evaluated. Figure 36 illustrates the dots of the read field in the camera image after parallelization.

[0294] Step 410: Adapt the compensation line. For this purpose, a compensation line (beam) is fitted to the rows and columns of valid dots in the reading field 22. The compensation line is calculated from the position data of valid dots in each row or column using standard mathematical methods to minimize errors. Invalid dots are excluded from this method.

[0295] The compensation lines i for each bunch k=0···1 are given by the following parameters: ○SX ki X I Intersection with the axis ○SY ki Y I Intersection with the axis ○α ki Angle of line i in the image field This is described by (Figure 37).

[0296] The compensation lines adapted to the dotted row form a first bundle of lines (bunch0)36, and these lines are Y C It is parallel to the axis. The lines of the dot row form a second bundle of lines (bunch1)37, and these lines are parallel to the X C It is parallel to the axis. As a result, the position data of 15 × 15 = 225 dots is reduced to 15 + 15 = 30 data points of the compensation line.

[0297] Interpolation further enhances the compensation line's robustness to individual variations in the dot positions by performing averaging across the dot positions on each line. This improves the stability and accuracy of the 6D position values.

[0298] Figure 36 shows the compensation lines for a reading field with 15 × 15 dots, namely the line from bunch 0 as reference code 36 and the line from bunch 1 as reference code 37.

[0299] Step 420 - Identify bunch data In this step, the three linear parameters SX of each bundle of compensation lines 36 and 37 are ki SY ki ,α ki However, interpolation is performed using a quadratic polynomial. The interpolation method considers only valid lines. Invalid lines that have only a few valid dots available for interpolation are excluded from bunch interpolation. For this purpose, weighted interpolation based on minimizing the squared error is performed using standard mathematical methods. Valid lines are evaluated with weights 1,0, while invalid lines are evaluated with weights 0,0 and are therefore hidden.

[0300] The interpolation function approach is, ·SX k (i) = psx k0 +i·psx k1 +i 2 PSX k2 ·SY k (i) = psy k0 +i·psy k1 +i 2 ·psy k2 ·α k (i) = pα k0 +i·pα k1 +i 2 ·pα k2 That is the case.

[0301] In this case, i is the number of the next line in each line bundle (line index).

[0302] The CE approach for interpolating linear parameters is P k (i) = pk0 +i·p k1 +i 2 ·p k2 However, P k (i) = SX k (i), SY k (i) or α k (i) That is the case.

[0303] To describe a straight line mathematically in a unique way, one of the two axis intersections SX ki or SY ki Only the straight line angle α ki It is sufficient to show this along with the axis. Ideally, the intersection with the axis as perpendicular as possible to the line should be shown. That is, for a line with a small slope, the intersection with the Y-axis is preferably shown, and for a steep line, the intersection with the X-axis is shown. Therefore, the following method is used: In bundle 1, a reference line located near the center 32 of the reading field 22 is selected; • X when the angle of the reference line is within the range of [(-1 / 4)π, (1 / 4)π] or [(3 / 4)π, (5 / 4)π] (Figure 38a) I The sector on the axis is a straight line with a small slope. In that case, for all straight lines in bundle 0(36), the intersection point SX with the X-axis is 0i The specified value is given to all lines in bundle 1 (37) and the intersection point with the Y-axis SY 1i This is specified (Figure 38b). As a marker, rot_status is set to 0; • Y when the angle of the reference line is outside the range described above (Figure 38a) I The sector on the axis is a steep straight line. In that case, for all straight lines in bundle 0(36), the intersection point with the Y-axis SY 0i The specified value is given to all lines in bundle 1(37) and the intersection point SX with the X-axis. 1i This is specified (Figure 38c). As a marker, rot_status is set to 1.

[0304] This step further reduces the amount of data. That is, the bundle of lines is described by only six parameters, namely α ki Three parameters related to SX, and one of the axis intersections, i.e., SX ki or SY ki It is described by three parameters. In the case of two bundles, twelve parameters and the marker rot_status are used.

[0305] Therefore, the two bundles 36 and 37 are always described by 12 parameters and the boolean variable rot_status, regardless of the number of lines or the size of the measurement field. From these parameters, the 6D position is calculated in the next step.

[0306] Interpolation further enhances the robustness of the interpolated values ​​against individual variations in each line or dot by performing averaging across the lines of each bundle and, consequently, across all the dots. This improves the stability and accuracy of the 6D position values.

[0307] Step 600 - Calculate the 6D camera position. From the 12 interpolation parameters of the line bundle (Table 9.1), the six coordinates of the camera position are calculated in the following order: • Partial step 500: Position r X and r Y • Partial step 710: Calculate the characteristic values ​​of the distorted grid. • Partial step 600: Camera angle φ Ζ • Partial step 700: Position r z • Partial step 700: Camera angle φ X and φ Y • Partial step 700: φ X , φ Y and r z Iterative algorithm for solving a system of equations related to This calculation involves deriving characteristic values ​​that describe the measured grid at the location of optical axis 5. These characteristic values ​​are linked to the 6D camera position via a system of equations derived from the imaging model and the laws of optical optics.

[0308] The camera position is determined by solving a system of equations.

[0309] Additionally, the effectiveness of the positional values ​​can be estimated by evaluating numerous diagnostic results from individual steps in the program flow.

[0310] Regarding partial step 500 - camera position r X and r Y Calculate The camera position in X and Y is an integer coarse position (r X,int ,r Y,int It is calculated by multiplying the sum of ) and a real fraction [0···1] (fine position) by the dot grid distance: r X =(r X,int +r X,fract )·c grid Unit: [m] r Y =(r Y,int +r Y,fract )·c grid Unit: [m] however, (r X,int ,r Y,int ): The integer position of the reading field in units [dots], as identified in step 300. (r X,fract ,r Y,fract ): A fraction of the real number of the read field position in units [dots] c grid : Dot grid distance in units [m / dot] That is the case.

[0311] The reference point for measuring the camera position is the intersection point 4 of the optical axis 5 and the code array 2, and within the camera image 3, it is the intersection point of the optical axis 5 and the imaging device 12 (point (4) in Figure 2 or the image of the intersection point 4 of the image center 4 in Figure 39). This image center IC(4) is identified by the orientation of the objective lens 9 relative to the camera chip, and the image center IC(4) does not have to be the same as the center of the camera chip. The image center IC(4) is identified in the calibration process and is represented by a certain two-dimensional vector (IC X ,I C Y It is stored in the program as a fraction r. X,fract and r Y,fract This is calculated as the intersection point of the interpolated line and the image center 4. For two ray beams k=0 and k=1 (36,37), the interpolation equation for the axis intersection is: S k (i) = ps k0 +i·ps k1 +i 2 · ps k2 However, S depends on rot_status. k (i) = SX k (i) or SY k (i) This holds true.

[0312] Figure 39 shows the central line for two bundles of lines k=0 and k=1.

[0313] The following holds true; ·If rot_status=0:

number

number

number

number

number

number

number

[0314] Regarding sub-step 710 - Calculate the characteristic values ​​of the distorted grid. From the interpolation equation, further characteristic values ​​that characterize the distorted grid can be obtained. To obtain characteristic values ​​at the center of the image, the fine position r X,fract and r Y,fract These are substituted into the interpolation equation. These characteristic values ​​are given by dimension r Z ,φ X ,φ Y , and φ Ζ It is necessary to determine the location.

[0315] a) Grid dimensions of the axis intersections of the lines for both line bundles: g C0 ,g C1 .

[0316] The grid dimensions are determined according to the dimensionless linear index i at the center of the image, and the axis intersection S k This is determined by deriving the interpolation function for (i): bunch k In contrast:

number

number

[0317] b) Linear angle at the center of the image: α C0 ,α C1 (See Figure 37) is obtained from the interpolation function with respect to the linear angle: bunch k In contrast:

number

[0318] c) Angular divergence of the line bundle at the center of the image: dα C0 ,dα C1 .

[0319] Angular divergence should be understood as the angular difference between adjacent lines near the center of an image. Angular divergence can be obtained by deriving an interpolation function for the line angle, according to a dimensionless line index i at the image center: bunch k In contrast:

number

[0320] Regarding partial step 600 - camera angle φ Z Calculate Straight angle φ Ζ This is identified from the imaging equation derived in 2.4.2:

number

[0321] Image coordinate B according to code grid coordinate x0X and B Y By deriving,

number

number

[0322] The gradient at the center when y0=0 is considered, and m1| y0=0 = -tan(φ Z ) can be obtained. m1| y0=0 This is the gradient of the line of bunch1 at the center of the image. This gradient is derived from the characteristic value α from 6.2b). C1 This corresponds to, that is, m1| y0=0 =tan(α C1 )

[0323] This allows the camera angle φ Ζ Equation relating to φ Ζ =α C1 This will result.

[0324] Regarding partial step 700 - camera position r Z Calculate Camera position r Z This is a bundle of two straight lines i The axis intersection g Ci It is determined from the grid distance. Since there are two line bundles in each image, there are basically two r Z The position can be determined. When rot_status=k, there are four values ​​r Zik However, (i,k=0···1) is obtained. The imaging equation for Figure 9 holds:

number

number

number

number

number

number

number

number

number

number

number

number

[0325] Table: Camera position value r Zik Calculation By introducing the angle β = rot_status·(π / 2), these equations can be expressed as follows:

number

[0326] One value r per image Z To output only this, the average of the two z values ​​is calculated: r Z =(r Z0k +r Z1k ) / 2 Regarding partial step 700 - camera angle φ X ,φ Y Calculate Angle φ X ,φ Y This is a bundle of two lines at the center of the image. k The angle divergence ∂αk (i) / ∂ i This is derived from the following. In step 600, the linear gradient m1 with respect to the bundle of lines bunch1 is derived as a function of the linear index y0:

number

[0327] From here, we calculate the linear angle α1(y0)=tan(m1(y0)), and then,

number

number

[0328] • At the center of the image y0=0, the angle diverges.

number

number

[0329] Similarly, angle φ Χ This is obtained from the gradient m0 ​​of the straight line at bunch0. Similar to the calculation at 600,

number

[0330] · Angle α0(x0) = tan(m0(x0))

number

number

[0331] • At the center of the image x0=0, the angle diverges.

number

[0332] • Measured angular divergence dα C0 If we identify it with φ Χ Equations relating to:

number

[0333] Regarding partial step 700 - camera position Z and camera angle φ X ,φ Y Iteratively calculate function φ X , φ Y There is an interdependence between and Z: φ Χ =f(φ y ,r Z );φ Y =f(φ X ,r Z );r Z =f(φ X ,φ y ) Therefore, these equations are solved in an iterative sequence. Each iterative step solves the camera position Z and the camera angle φ. Χ and φ Y This improves accuracy. This algorithm is shown in the flowchart in Figure 40.

[0334] Lower step 700.1: Real fraction of the field position read in units [dots] (r X,fract ,r Y,fract ) calculate Lower step 700.2: position r x and r y Calculate Lower step 700.3: Camera rotation / camera angle φ Ζ Calculate Lower step 700.4: Initialize camera angles phix=0;phiy=0 Sub-step 700.5: Initialize the counter for iteration, for example, by 4. Lower step 700.6:r Z0k and r Z1k Calculate and finally r Z =(r Z0k +r Z1k Calculate ) / 2 Lower step 700.7: Calculate camera angles phix and phiy. Lower step 700.8: Counter = 0? Recalculate or otherwise abort the iteration. Lower step 700.9: Error response This algorithm converges quickly, and the results become sufficiently stable after, for example, four iterations. Finally, the results are as follows: • Camera position (r X ,r Y ,r Z ) • Camera angle (φ X ,φ Y ,φ Z ) • Validity information The following will be output. [Explanation of symbols]

[0335] 1 Camera 2 Code Arrays 3 images 4. Intersection point of the optical axis 5 and the code array 2; reference point; 5 Optical axis Number 6 is missing. 7. Intersection of the optical axis and the imaging device 8. An object with at least one code array 2. 9. Objective lens 10 Focal point of objective lens 9 11 Focal point of objective lens 9 12 Imaging device 13 Optical center of objective lens 9 14 Observation rays 15 The coordinate system (X) of the imaging device 12 in the plane of the imaging device 12 I ,Y I ) 16. Coordinate system of code array 2 in the plane of code array 2 (X C ,Y C ) 17. Coordinate system of camera 1 (X, Y, Z) 18 Virtual image plane 19 parcels 20 Basic Symbols 21 Parcel Symbols 22 Read Fields 23 Center position 24X parcel area 25 Y parcel area 26 Parcel area for additional data 27 Intersection Patterns 28. Starting Field / Starting Base / Launchpad 29 Dot Matrix 30 starting position 31 Search Ray 32 Center position of reading field 22 33 Unparallelized point grid 34 Parallelized point grid 35 Reading track in the first code direction 36 Reading track in the second code direction 37 First line bundle / bunch0 38 Second bundle of straight lines / bunch1

Claims

1. A method for identifying at least one degree of freedom of camera (1) relative to a code array (2) from a camera image of camera (1), The code array (2) has a point grid comprising a plurality of basic symbols (20), the code array (2) in particular has the coarse positions of the basic symbols (20) encoded, the basic symbols (20) define a first principal direction along the point grid and a second principal direction independent of the first principal direction, and the camera image is in coordinate system (X Ｉ , Y Ｉ ) define, Within the camera image, a reference point formed as the intersection of the optical axis of the camera (1) and the imaging device and / or the camera image is located. A first linear function having a first function argument in the coordinate system of the camera image is provided for a first line oriented parallel to the first principal direction in the coordinate system of the code array, and by changing the first function argument, the first line is shifted parallel to the second principal direction in the coordinate system of the code array, and / or a second linear function having a second function argument in the coordinate system of the camera image is provided for a second line oriented parallel to the second principal direction in the coordinate system of the code array, and by changing the second function argument, the second line is shifted parallel to the first principal direction in the coordinate system of the code array. The first function argument and / or the second function argument are determined such that the first line or the second line intersects the reference point. A method in which the rotation angle (phiz) of the camera (1) around the optical axis is derived as the degree of freedom of the camera (1) relative to the code array (2) based on the first straight line and / or the second straight line.

2. A basic symbol matrix is ​​identified from the camera image, the positions of the basic symbols in the camera image are entered into the basic symbol matrix, and the first linear function and / or the second linear function are derived based on the basic symbol matrix. The method according to claim 1.

3. The aforementioned linear function is the linear angle function of the linear angle and the coordinate system (X) of the camera image. Ｉ , Y Ｉ It is formed in combination with one of the axes, depending on the function argument and the axis intersection function, The aforementioned axis intersection function, depending on the first function argument, determines the coordinate system (X) of the camera image. Ｉ , Y Ｉ The axial intersection point along the axis extending through the aforementioned reference point is determined. The method according to claim 1 or 2.

4. The rotation angle phiz about the optical axis is determined independently over two linear functions and is formed as the average value of the two values. The method according to any one of claims 1 to 3.

5. From the camera image, a basic symbol matrix is ​​identified in which the positions of the basic symbols (20) in the camera image are entered. Based on the basic symbol matrix, a first linear function having the first function argument in the coordinate system of the camera image is derived for the first line, and / or Based on the aforementioned basic symbol matrix, a second linear function having the second function argument in the coordinate system of the camera image is derived for the second line. The method according to any one of claims 1 to 4.

6. Based on the basic symbols (20) of the basic symbol matrix or one subset thereof, the coarse position of at least one of the basic symbols in the code sequence is determined, and based on the first linear function and the second linear function, the fine positions of the two degrees of freedom of the camera relative to the code sequence are determined. The method according to any one of claims 1 to 5.

7. A reference point is placed in the aforementioned camera image. A first axis intersection function is provided for the first straight line in the coordinate system of the camera image, and the first straight line is in the coordinate system (X) of the code array. Ｃ , Y Ｃ ) is oriented parallel to the first principal direction (2) of the point grid, A first function argument is provided, and by changing the first function argument, the first straight line is shifted parallel to the second main direction in the coordinate system (X Ｃ , Y Ｃ ) of the code array. According to the first axis intersection function, depending on the first function argument, a first axis intersection is determined along a first axis extending through the reference point in the coordinate system (X Ｉ , Y Ｉ ) of the camera image. A second axis intersection function is provided for the second straight line in the coordinate system of the camera image, and the second straight line is in the coordinate system (X) of the code array. Ｃ , Y Ｃ ) is oriented parallel to the second principal direction, A second function argument is provided, and by changing the second function argument, the second line is in the coordinate system (X) of the code array. Ｃ , Y Ｃ ) is shifted parallel to the first principal direction, and the second axis intersection function, depending on the second function argument, the coordinate system (X) of the camera image is shifted parallel to the first principal direction, and the second axis intersection function is shifted parallel to the second function argument. Ｉ , Y Ｉ A second axial intersection is determined along a second axis extending through the aforementioned reference point of ), Based on the aforementioned axis intersection function, the first and second function arguments are determined such that the reference point forms the first axis intersection and the second axis intersection. Based on the first function argument, the second function argument, and the coarse position, the coordinate system (X) of the code sequence (2) in the plane of the code sequence (2) Ｃ , Y Ｃ The precise position of the reference point in ) is determined as at least one further degree of freedom. The method according to claim 6.

8. A reference point or the aforementioned reference point is positioned in the camera image, and the reference point is formed as the intersection of the optical axis of the camera and the imaging device and / or the camera image. From the camera image at the aforementioned reference point, the following values ​​are obtained, namely: - One rotation angle (phiz) of the camera with respect to the optical axis of the camera, - The first local grid dimension (gc0) of the point grid in the camera image in the first principal direction and / or the second local grid dimension (gc1) of the point grid in the camera image in the second principal direction, - The first local angular divergence (dalphac0) of the point grid in the camera image in the first principal direction, which describes the difference angle between two adjacent lines in the first principal direction within the point grid, - The second local angular divergence (dalphac1) of the point grid in the camera image in the second principal direction, which describes the difference angle between two adjacent lines in the second principal direction within the point grid, and This is identified as a characteristic value, Based on the aforementioned characteristic values, three further degrees of freedom of the camera relative to the code sequence are determined, namely, - The distance (rz) between the code array (2) and the camera (1), - With respect to the code array (2), the two independent pitch angles (phix; phiy) of the optical axis and The decision is made. The method according to any one of claims 1 to 7.

9. A starting field (28) having basic symbols (20) is determined, and the starting field (28) has at least three basic symbols (20). Through the basic symbols (20) of the start field (28), two independent principal directions along the point grid in the camera image are estimated, and the basic symbols (20) of the start field form a valid basic symbol. In the search step, further basic symbols (20) are searched for based on at least one valid basic symbol (20) along at least one of the principal directions, and if the search is successful, the further basic symbol (20) is characterized as a valid basic symbol (20). The search step is performed multiple times, and a basic symbol matrix is ​​identified based on the valid basic symbol (20) or a subset thereof. The method according to any one of claims 1 to 8.

10. An electronic control unit, or an automated device equipped with an electronic control unit, The control unit is configured programmatically and / or circuitmatically to carry out the method described in any one of claims 1 to 9. An electronic control unit, or an automated device equipped with an electronic control unit.

11. It is a computer program, The computer program is configured to carry out the method described in any one of claims 1 to 9 when the computer program is executed in a computer or in the control unit described in claim 10. Computer program.

12. A machine-readable storage medium storing the computer program described in claim 11.

Images