Method for calibrating a set of cameras for optical tracking

Abstract

The invention relates to methods for calibrating a system of cameras for the optical tracking of objects in a space. The present method includes the steps of scanning a file system to calculate the number of cameras connected, receiving consecutive frames from all of the connected cameras, locating optical markers on each frame of an image, determining the centre of each marker, generating an array of the centres of the markers, sending a packet to a computing device, converting the local coordinates of a marker into global coordinates, identifying overlaps, filtering, and sending the resulting packet containing the coordinates of an object by means of a UDP request to a socket of a receiving computing device. The invention allows more accurate calibration of cameras with different focal lengths, resolutions and lens distortions.

Description

[0001] METHOD FOR CALIBRATION OF A SET OF CAMERAS FOR OPTICAL TRACKING

[0002] AREA OF TECHNOLOGY

[0003] The present technical solution relates to the field of computer technology, in particular, to methods for calibrating a set of cameras for optical tracking of objects in space.

[0004] LEVEL OF TECHNOLOGY

[0005] A solution selected as the closest analogue is known from the prior art, WO 2019 / 122950 A1, 27.06.2019. This solution relates to the field of computer engineering, namely to a method and system for inverse optical-inertial tracking of a mobile object. The method includes the steps of: ensuring that a plurality of markers emit light in the infrared range; reading, using a first data processing device, an image from an optical sensor that is physically connected to the mobile object, and identifying in the read image pixels that satisfy predetermined conditions; identifying, based on the identified pixels and using the first data processing device, images of markers and determining the parameters of said marker images; reading, using the first data processing device, from at least one inertial sensor that is physically connected to the mobile object;creating, using a first data processing device, a combined data stream containing parameters of marker images and data from said at least one inertial sensor, and transmitting said data stream via a first device message to a second communication device; receiving the combined data stream via a second communication device and determining, based on said data stream, positional data of the tracker in the coordinate system of the mobile object's movement zone; checking, using the second data processing device and using the positional data of the tracker, a mathematical model of the tracker's movement trajectory; extrapolating, using the second data processing device and using the verified mathematical model of the tracker's movement trajectory, the positional data of the tracker to the extrapolation horizon.

[0006] The proposed solution is aimed at eliminating the shortcomings of the current level of technology and differs from known solutions in that the proposed technical solution allows for a significant improvement in the ability to use cameras with very different focal lengths, resolutions and lens distortions in one system.

[0007] ESSENCE OF THE INVENTION

[0008] The technical problem that the proposed solution aims to solve is the creation of a method for calibrating a set of cameras for optical tracking of objects in space.

[0009] The technical result coincides with the purpose of the invention, and additionally consists in improving the possibility of using cameras with very different focal lengths, resolutions and lens distortions in one system.

[0010] The claimed result is achieved by implementing a method for calibrating a set of cameras for optical tracking of objects in space, which contains the following stages: counting the number of connected cameras by scanning the file system; continuous sequential reading of frames from all connected cameras, with images from each camera being obtained in different streams; searching for images of optical markers on each read frame; determining the geometric center for each marker image; forming a packet with an array of geometric centers of each found marker for sending to a computing device; sending the packet to the computing device-server via a socket connection using a UDP packet, with sending being performed to a known local IP address of the computing device;wherein each computing device-aggregator at each cycle of its operation sends to the local IP address and port of the computing device-server a UDP request containing a list of coordinates of the centers of the marker images on the frames of the connected cameras; the transformation of the local coordinates of the marker image in the camera coordinate system into global coordinates, by means of a function at the input of which the calibration matrix of the camera and the coordinates of the point on the frame are received, and at the output - three-dimensional coordinates of the point in space in the global coordinate system;finding intersections, filtering and sending the resulting packet, whereby intersections are found by calculating pairwise intersections of all lines constructed on the basis of the obtained marker coordinates on camera frames, and the obtained intersections are filtered according to the quality criterion and the obtained 3D coordinates are linked to the tracking objects, whereby the sending of the resulting packet with the object coordinates is performed using a UPD request to the socket of the receiving computing device.

[0011] In a particular embodiment of the described solution, the result of the search on each read frame of images of optical markers is an array of two-dimensional coordinates of the marker centers in the frame space, as well as their colors, while filtering of the found markers by geometric parameters is used to cut off false highlights.

[0012] DESCRIPTION OF DRAWINGS

[0013] The implementation of the invention will be described further in accordance with the attached drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

[0014] Fig. 1 illustrates an example of a Pinhole camera model.

[0015] Fig. 2 illustrates the arrangement of dotted stickers on the wall.

[0016] Fig. 3 illustrates a map indicating the orientations of the cameras and their viewing angles.

[0017] Fig. 4 illustrates the camera calibration process.

[0018] Fig. 5 illustrates a general block diagram of the method.

[0019] Fig. 6 illustrates a block diagram of the operation of the computing device-server.

[0020] DETAILED DESCRIPTION OF THE INVENTION

[0021] In the following detailed description of the embodiment of the invention, numerous implementation details are set forth in order to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail in order not to unnecessarily obscure the features of the present invention.

[0022] In addition, it will be clear from the above description that the invention is not limited to the embodiment shown. Numerous possible modifications, changes, variations and substitutions that retain the essence and form of the present invention will be obvious to those skilled in the art. The following terms and definitions are used in this document.

[0023] Tracking is the process of determining the location of moving objects over time using a camera.

[0024] Calibration is the establishment of a relationship between the readings of a measuring instrument (device) and the size of the measured (input) value.

[0025] Space is a set of objects, given in human perception simultaneously, forms a complex spatial image of the world, which is a necessary condition for the orientation of any human activity

[0026] A camera is an optical device for recording images.

[0027] Optical tracking is a special technology that underlies human interaction with the virtual world. It is aimed at accurately determining the coordinates and position of a real object (for example, a hand, head or device) in a virtual environment using three coordinates (x, y, z) of its location and three angles (a, b, d) that define its orientation in space.

[0028] An optical marker is three points that are the vertices of a conditional non-equilateral triangle, one of the angles of which is equal to 90°.

[0029] Recently, object tracking systems using a set of cameras have become increasingly relevant, as they allow you to determine the absolute position of an object in space. This technology is critical to achieving the effect of immersion in virtual reality, which is increasingly gaining popularity due to the growth of the market for games with virtual reality glasses. The majority of sales in the Russian market are made up of virtual reality (VR) technologies.

[0030] The main goal of virtual reality is to transfer the user from the real world to a new one generated by the computer. That is why the maximum immersion effect is felt in a special helmet, completely hiding the user from the surrounding world. However, the player's capabilities in such games were small due to the small number of degrees of freedom of head movements.

[0031] In order to allow the player to move freely in space, see their hands and feet, interact with the environment, other players and real interactive objects, it is necessary to use a tracking system that tracks players and interactive objects in 3D space.

[0032] For the task of tracking players in space, optical cameras and LED markers mounted on VR glasses are usually used. The cameras are mounted in the room so that each point in space is viewed by at least two cameras.

[0033] The tasks that are solved in the process of creating a tracking method for finding and identifying a marker in frames from a large number of cameras: creating a method for searching for an object in a frame; conducting a load test to guarantee the technological quality of the information system after the deployment of the information system and ensuring compliance with the declared technological requirements during the operation of the information system; classifying the marker in frames by its color with a large number of cameras.

[0034] To create tracking in space, a pinhole camera model is used (Figure 1), in which all rays fall on the matrix through a point located at a focal distance from the matrix.

[0035] The Pinhole camera model describes the mathematical relationship between the coordinates of a point in three-dimensional space and its projection onto the image plane of an ideal Pinhole camera, where the camera lens is described as a point and no lenses are used to focus the light.

[0036] According to this model, the two-dimensional coordinates of a point on a frame are transformed into three-dimensional equations of a straight line in space. To track the 3D coordinates of a marker, it is necessary to carry out a calibration procedure. To do this, the following is performed:

[0037] - tracking calibration;

[0038] - tracking testing;

[0039] - camera calibration;

[0040] - calculation of marker coordinates.

[0041] To calibrate the tracking, the cameras were connected to a computing device and directed in such a way as to cover the maximum game volume. Then, using special equipment (laser level and rangefinder), the coordinates of all cameras and the coordinates of points on the object were measured.

[0042] To calculate the orientations and focal lengths of the cameras, the coordinates of points on the object were measured. For this purpose, paper stickers with points drawn on them with a marker were placed on the wall so that when viewing the frame through the interface, these points would fall on some object inside the room (Fig. 2).

[0043] Using a numerical solution on gradient descent, the focal length in pixels was calculated. As a result of camera calibration, a 2D interface with a map was developed, on which the orientation of the cameras and viewing angles were indicated (Fig. 3).

[0044] To test the tracking, a visualization was launched that showed ■ the orientation of the cameras and the interpretation of the frames - the lines coming from the camera icons should intersect at one point. However, during the first test, the accuracy was not high enough due to distortion in the camera lenses

[0045] In order to compensate for distortions in camera lenses, it is necessary to carry out a camera calibration procedure.

[0046] The calibration process consists of obtaining all the parameters that the user needs to know to eliminate projection effects in the camera. These parameters are divided into two types: internal and external. The former are typical for the camera and do not change. As a rule, this is the focal length, pixel size and the main point. External ones are the position and orientation of the camera in three dimensions (the absolute coordinate system in the world is taken as a basis).

[0047] During the calibration process, the camera took a set of photographs of the chessboard, in which the angles between the squares were recognized with subpixel accuracy (Fig. 4).

[0048] The result of the calibration was a matrix containing focal lengths in pixels along two axes and the matrix offset relative to the optical center, as well as a vector of distortion coefficients, which allows for compensation of lens distortions using pixel coordinate transformations.

[0049] By applying transformations with these coefficients to the marker coordinates on the frame, the system was brought to the model of an ideal pinhole camera. Then a new tracking test was performed.

[0050] As a result, a simpler method for calculating the marker coordinates was discovered. A tracking algorithm was formed that pairwise calculates the intersections of lines from cameras, finds the best one and uses it as the marker coordinates.

[0051] The developers were tasked with developing a method that would implement the reception and interpretation of data from aggregator devices with subsequent visualization of optical marker tracking.

[0052] Figure 5 illustrates a general block diagram of the proposed method.

[0053] This technical solution can operate in two modes:

[0054] • tracking mode;

[0055] • calibration mode.

[0056] In tracking mode, the following algorithm of actions is carried out: • Receiving packets with data on two-dimensional coordinates of the centers of marker images in camera frames from aggregator devices.

[0057] • Use of calibration matrices, determining the position and orientation of cameras, forming equations of lines corresponding to each point obtained from aggregator devices;

[0058] • Calculation of the coordinates of intersections of lines in pairs and assessment of the quality of intersections (based on the shortest distance between lines);

[0059] • Filtering and binding the obtained 3D intersection coordinates to tracking objects;

[0060] • Sending packets with arrays of final 3D coordinates of objects to the user's address in real time for further use.

[0061] In calibration mode, the following algorithm of actions is carried out:

[0062] • Receiving packets of data on two-dimensional coordinates of the centers of images of the calibration marker on camera frames from aggregator devices.

[0063] • Iterative generation of calibration matrices for each camera, determining their positions and orientations based on the assumption that there is a single marker in the tracking space during calibration and all rays intersect on it;

[0064] • Saving the obtained calibration matrices in the permanent memory of the server device for subsequent use in tracking mode.

[0065] Basic functions of the server device algorithm.

[0066] The server device algorithm provides the following functions:

[0067] - receiving data from aggregator devices;

[0068] - calibration of cameras (calculation of the matrix of internal (intrinsic) and external (extrinsic) parameters of each camera);

[0069] - interpretation of data based on calibration data;

[0070] - tracking of optical markers in space;

[0071] - visualization of optical marker tracking.

[0072] Below is a detailed description of how these functions are implemented on the server device:

[0073] - Receiving data from aggregator devices;

[0074] Data is received from aggregator computers by receiving UDP requests from the local network to the server computer port via a socket connection. Each aggregator computer at each cycle of its operation sends a UDP request to the local IP address and port of the server computer, containing a list of coordinates of the centers of marker images on the frames of the connected cameras. - Calibration of cameras (calculation of the matrix of internal (intrinsic) and external (extrinsic) parameters of each camera);

[0075] To perform the calibration procedure, you need to run the algorithm in calibration mode and start moving a special calibration marker in the tracking space.

[0076] In this case, packets with data on the two-dimensional coordinates of the centers of the images of the calibration marker in the camera frames are received from aggregator devices.

[0077] After data collection is complete, the iterative formation of calibration matrices for each camera begins, determining their positions and orientations based on the assumption that there was only one marker in the tracking space during calibration and all rays intersect on it.

[0078] The data obtained during the matrix calibration process are saved in the memory of the server device in the form of files with the extension .pr for subsequent use in tracking mode.

[0079] - Interpretation of data based on calibration data;

[0080] When the algorithm is launched in tracking mode, the first step is to read files with calibration matrices from the device memory. The data from the files are interpreted as arrays of floating point numbers of size 4*4 in the numpy.array format. The incoming arrays of local 2D coordinates of the centers of marker images in the frames from the cameras are transformed into equations of straight lines in the global coordinate system using calibration matrices.

[0081] - Tracking of optical markers in space;

[0082] Based on the obtained equations of lines in the global coordinate system, the coordinates of their intersections are calculated in pairs and the quality of the intersections is assessed. The quality of the intersections is assessed by the shortest distance between the lines (the length of the segment perpendicular to both lines). The center of this segment is taken as the coordinates of the intersection.

[0083] The obtained intersections are filtered by the quality parameter and the obtained 3D coordinates are linked to the tracking objects.

[0084] - Visualization of optical marker tracking;

[0085] Tracking visualization occurs in the form of two-dimensional rendering of projections of lines and intersection points on a horizontal plane. Rendering is performed using the pyQt library and OpenCV.

[0086] The marker position tracking system consists of a distributed network of cameras and computing systems. It determines the position of each object in space with sub-centimeter accuracy. The adaptive tracking algorithm works with an arbitrary number and location of cameras, which makes the system easy to install and easily scalable to any area of ​​the simulation zone.

[0087] One of the key features of this algorithm is its resistance to differences in optical parameters of cameras. During the calibration process, the location, orientation, focal length, and lens distortion parameters of each camera are automatically calculated.

[0088] This is achieved by multidimensional gradient descent algorithms for each camera, the space of which is made up of the relative focal length of the lens and distortion coefficients. The linear and angular coordinates of the cameras are also calculated by gradient descent, but independently to reduce the dimensionality of the problem.

[0089] Compensation for growing errors is performed by decomposing the main matrix of the stereo pair and calculating the coordinates of one camera in the coordinate system of the other for each pair of cameras. Calculations are performed in the form of matrices of internal and external parameters of cameras, which allows vectorizing the task and distributing calculations over independent streams to reduce calibration time. Thus, the system allows using any cameras with unknown and different optical parameters.

[0090] Main characteristics:

[0091] - fixed parameters of camera calibration: linear and angular coordinates in 3D space, focal length, lens distortion parameters;

[0092] - frame rate: from 30 fps;

[0093] - sensitivity of marker coordinate determination: <5 mm;

[0094] - maximum number of cameras: unlimited;

[0095] - maximum coverage area: unlimited;

[0096] - camera matrix resolution: 640x480.

[0097] Fig. 6 illustrates a block diagram of the operation of the computing device-server (server).

[0098] The computing server device receives and interprets data from computing aggregator devices with subsequent visualization of optical marker tracking. The server algorithm ensures the following functions: - receiving data from aggregator computers; - calibration of cameras (calculation of the matrix of internal (intrinsic) and external (extrinsic) parameters of each camera); - interpretation of data based on calibration data; - tracking of optical markers in space; - visualization of optical marker tracking.

[0099] The decomposition of the algorithm processes is presented below.

[0100] 1) Receiving packets from aggregator computers: Receiving data from aggregators is performed by receiving UDP requests from the local network to the server port via a socket connection. Each aggregator, at each cycle of its operation, sends a UDP request to the local IP address and port of the server computer, containing a list of coordinates of the centers of marker images on the frames of the connected cameras.

[0101] In each cycle of the server operation, the presence of UDP packets in the receiving buffer is checked using the sock.recvfrom function (4096). If there are unread packets in the buffer, the function returns an array of bytes for further processing.

[0102] 2) Construction of equations of lines, conversion to the global coordinate system: Each point on the camera frame defines a line in space passing through the center of the camera and the marker whose image is displayed on the frame. To construct the equation, it is necessary to convert the local coordinates of the marker image (in the camera coordinate system) into global coordinates. This is done using the frame_to_global(cam_params, frame dot) function. The input of this function is the camera calibration matrix and the coordinates of the point on the frame, and the output is the three-dimensional coordinates of the point in space in the global coordinate system. Thus, the line in space is defined by this point and the 3D coordinates of the camera.

[0103] 3) Finding intersections, filtering and sending the resulting packet: to determine the coordinates of the tracked object, you need to find the intersection of two lines coming from two different cameras. To do this, the algorithm calculates pairwise intersections of all lines constructed on the basis of the received coordinates of markers on the camera frames. The calculation is performed in the crossing(line1, Iine2) function. The function receives two pairs of points on the lines as input, and the output is the coordinates of the center of the segment perpendicular to both lines. The quality of the intersections is estimated by the length of this segment (the shortest distance between the lines). The obtained intersections are filtered by the quality parameter and the obtained 3D coordinates are bound to the tracking objects. The resulting packet with the object coordinates is sent using a UPD request to the socket of the receiving computer using the sock.sendto(pack, obj.target_ip_port) function. The function receives a packet to send and an IP address and port on the receiving computer as input.Thus, due to the present technical solution, highly efficient calibration of a set of cameras for optical tracking of objects in space is carried out.

[0104] The present application materials present a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

Formula 1. A method for calibrating a set of cameras for optical tracking of objects in space, comprising the following stages: counting the number of connected cameras by scanning the file system; continuous sequential reading of frames from all connected cameras, with images from each camera being obtained in different streams; searching for images of optical markers on each read frame; determining the geometric center for each marker image; forming a packet with an array of geometric centers of each found marker for sending to a computing device; sending the packet to the computing device-server via a socket connection using a UDP packet, with sending being performed to a known local IP address of the computing device;wherein each computing device-aggregator at each cycle of its operation sends to the local IP address and port of the computing device-server a UDP request containing a list of coordinates of the centers of the marker images on the frames of the connected cameras; the transformation of the local coordinates of the marker image in the camera coordinate system into global coordinates, by means of a function at the input of which the calibration matrix of the camera and the coordinates of the point on the frame are received, and at the output - three-dimensional coordinates of the point in space in the global coordinate system;finding intersections, filtering and sending the resulting packet, whereby intersections are found by calculating pairwise intersections of all lines constructed on the basis of the obtained marker coordinates on camera frames, and the obtained intersections are filtered according to the quality criterion and the obtained 3D coordinates are linked to the tracking objects, whereby the sending of the resulting packet with the object coordinates is performed using a UPD request to the socket of the receiving computing device.

2. The method according to item 1, in which the result of the search on each read frame of images of optical markers is an array of two-dimensional coordinates of the centers of the markers in the frame space, as well as their colors, wherein filtering of the found markers by geometric parameters is used to cut off false highlights.

Images