Data processing method, data processing device and tracking scanning system

Abstract

The invention relates to a data processing method, a data processing device and a tracking scanning system.The data processing method is used for the tracking scanning system, and the tracking scanning system comprises a tracker and a scanner; the method comprises the following steps: acquiring tracking data generated by tracking the pose of a scanner in a scanning process by a tracker; using the tracking data to calculate the camera parameters of the tracker, and adjusting the tracking pose of the tracker according to the calculation result; and unifying scanning data acquired by the scanner to a tracking coordinate system of the tracker by using the adjusted tracking pose. According to the invention, the problem of low scanning efficiency of the tracking scanning system is solved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional scanning, and in particular to a data processing method, a data processing device and a tracking scanning system. Background Art

[0002] In the related art, a tracking and scanning system is usually used for three-dimensional scanning. The tracking and scanning system includes a scanner and a tracker, wherein the tracker is mainly used to locate the scanning posture of the scanner. When in use, it is often necessary to regularly use calibration rods and other standards to calibrate the external parameters of the tracker (the posture relationship of the tracker camera pair) to ensure the tracking accuracy of the tracker, otherwise the tracking scanning accuracy will be affected by temperature and device state changes. The conventional calibration method is to calibrate the external parameters of each tracker before use, and then make the scale or a set of marker points with a priori geometric distance relationship appear in the field of view of the tracking device during the tracking scan, and use the scale and other prior marker point sets to correct the tracker parameters in real time.

[0003] In the prior art, when using a calibration rod to calibrate the external parameters of the trackers in the tracking and scanning system, they need to be calibrated one by one. For a multi-tracking and scanning system, as the number of trackers used increases, the measurement efficiency decreases. Moreover, for some trackers fixed in special positions, the field of view of the tracker may not be calibrated directly, and the equipment needs to be disassembled for calibration, which will cause the calibration to take a long time, further affecting the scanning efficiency.

[0004] With respect to the problem of low scanning efficiency of the tracking scanning system existing in the related art, no effective solution has been proposed yet. Summary of the invention

[0005] In this embodiment, a data processing method, a data processing device and a tracking and scanning system are provided to solve the problem of low scanning efficiency of the tracking and scanning system in the related art.

[0006] In a first aspect, a data processing method is provided in this embodiment for use in a tracking and scanning system, wherein the tracking and scanning system includes a tracker and a scanner, and the method includes:

[0007] Acquire the tracker, and generate tracking data by tracking the position and posture of the scanner during scanning;

[0008] Calculating camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation results;

[0009] The scan data acquired by the scanner is unified into the tracking coordinate system of the tracker using the adjusted tracking posture.

[0010] In some embodiments, obtaining the tracking data generated by the tracker by tracking the position and posture of the scanner during the scanning process includes:

[0011] The tracking data is generated by tracking the position and posture of the scanner during a scanning process using the tracker without calibrating the camera parameters.

[0012] In some embodiments, calculating the camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes:

[0013] During the scanning process, the tracking data of the currently generated target frame is used to calculate the camera parameters of the tracker, and the tracking posture of the target frame is adjusted according to the calculation result.

[0014] In some embodiments, calculating the camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes:

[0015] After the scanning is completed, the camera parameters of the tracker are calculated using the tracking data generated during the scanning process, and the tracking posture is adjusted according to the parameter calculation results.

[0016] In some embodiments, calculating the camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes:

[0017] According to the image information acquired by each camera of the tracker in the tracking data, a reprojection residual or a point-to-polar line distance residual is constructed, the camera parameters of the tracker are calculated, and according to the calculation results, the tracking posture of the tracker is adjusted.

[0018] In some embodiments, calculating the camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes:

[0019] The camera parameters of the tracker are set to initial values, and a loss function is constructed by using reprojection and coordinate transformation reprojection according to the image information acquired by each camera of the tracker in the tracking data;

[0020] Minimizing the loss function using a nonlinear optimization algorithm to calculate optimized values ​​of the camera parameters of the tracker;

[0021] The tracking posture of the tracker is updated according to the optimized value of the camera parameter.

[0022] In some embodiments, calculating the camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes:

[0023] Based on the frame tracking poses of each camera of the tracker during the scanning process contained in the tracking data and the historical camera parameters of the tracker, calculating the distances from the image points to the corresponding epipolar lines of the camera;

[0024] According to the distance from the image point to each corresponding camera epipolar line, an optimization function is constructed to minimize the distance from all image points to each corresponding camera epipolar line, and the optimization function is solved to obtain the optimized value of the camera parameter of the tracker;

[0025] According to the optimized values ​​of the camera parameters of the tracker, the tracking pose of the tracker is updated. In some embodiments, the method further includes:

[0026] The calculated results of the camera parameters of the tracker are constrained using a preset standard.

[0027] In some embodiments, the method further comprises:

[0028] During the scanning process, based on the current frame acquired by the tracker, determining whether there is a camera that has not obtained the tracking posture of the scanner;

[0029] If it exists, the tracking information posture of the camera that has not obtained the scanner tracking posture is supplemented according to the information of the camera that has obtained the scanner tracking posture and the historical camera parameters of the tracker.

[0030] In some embodiments, the method of completing the tracking information posture of the camera that has not obtained the scanner tracking posture according to the information of the camera that has obtained the scanner tracking posture and combining the historical camera parameters of the tracker includes:

[0031] The tracking pose of the camera that tracks the tracking pose of the scanner is converted into the camera coordinate system of the camera that does not obtain the tracking pose of the scanner according to the camera parameters of the historical tracker to obtain the prior tracking pose;

[0032] According to the prior tracking posture, using the historical camera parameters of the tracker, obtaining the re-projected image points obtained by re-projecting the image points of the scanner onto the image of the camera that does not obtain the scanner tracking posture;

[0033] Calculating the correspondence between the historical image points recorded by the camera that does not obtain the scanner tracking posture and the re-projected image points;

[0034] According to the corresponding relationship, the tracking information posture of the camera that has not obtained the scanner tracking posture is completed.

[0035] In some of the embodiments, in the process of calculating the camera parameters of the tracker, when the camera parameters between two cameras are recalculated in a certain frame, the historical camera parameters of the two cameras are retained; or the camera parameters of the two cameras are directly overwritten; or based on the tracking error of the tracking posture of the scanner in the coordinate system of the two cameras, it is determined whether the camera parameters of the two cameras need to be overwritten; or based on preset weights, the historical camera parameters of the two cameras and the newly calculated camera parameters of the two cameras are fused.

[0036] In some embodiments, the method further comprises:

[0037] Acquire a parameter compensation model of the tracker, a parameter compensation coefficient of the tracker, a parameter compensation model of the scanner, and a parameter compensation coefficient of the scanner;

[0038] During the scanning process, the parameters of the tracker are compensated in real time using the parameter compensation model of the tracker and the parameter compensation coefficient of the tracker, and the parameters of the scanner are compensated in real time using the parameter compensation model of the scanner and the parameter compensation coefficient of the scanner, and the parameters of the tracker after real-time compensation are recorded for each frame.

[0039] In some embodiments, calculating the pose relationship of the camera pair of the tracker using the tracking data, and adjusting the tracking pose of the tracker according to the calculation result, includes:

[0040] The parameter values ​​of the tracker after real-time compensation, the frame tracking poses of each camera of the tracker, and the historical camera parameters of the tracker are set as initial values, and a loss function is constructed by reprojection and coordinate transformation reprojection according to the image information acquired by each camera of the tracker;

[0041] Minimizing the loss function using a nonlinear optimization algorithm, and calculating optimized values ​​of parameters of real-time compensation of the tracker, optimized values ​​of frame tracking poses of each camera of the tracker, and optimized values ​​of camera parameters;

[0042] According to the optimized values ​​of the parameters of the real-time compensation of the tracker, the optimized values ​​of the frame tracking poses of the various cameras of the tracker, and the optimized values ​​of the camera parameters, the parameter values ​​of the real-time compensation of the tracker are updated, the frame tracking poses of the various cameras of the tracker are updated, and the camera parameters of the tracker are updated.

[0043] In some embodiments, the method further comprises:

[0044] After the scan data acquired by the scanner is unified into the tracking coordinate system of the tracker, the scan data is sent to an associated display terminal for display.

[0045] In a second aspect, in this embodiment, a data processing device is provided for a tracking and scanning system, wherein the tracking and scanning system includes a tracker and a scanner, and the device includes a data acquisition module, a tracking posture optimization module, and a scanning data unification module;

[0046] The data acquisition module is used to acquire the tracking data generated by the tracker by tracking the position and posture of the scanner during the scanning process;

[0047] The tracking posture optimization module is used to calculate the camera parameters of the tracker using the tracking data, and adjust the tracking posture of the tracker according to the calculation result;

[0048] The scanning data unification module is used to unify the scanning data acquired by the scanner into the tracking coordinate system of the tracker using the adjusted tracking posture.

[0049] In a third aspect, a tracking and scanning system is provided in this embodiment, wherein the tracking and scanning system includes a tracker and a scanner, and an optimization device;

[0050] The optimization device stores a computer program, and when the computer program is executed by the processor, the steps of the data processing method described in the first aspect are implemented.

[0051] In a fourth aspect, a tracking scanning method is provided in this embodiment, wherein the tracking scanning system described in the third aspect is used to perform three-dimensional scanning on a scanning object, and the method includes:

[0052] Starting the tracking and scanning system;

[0053] The tracker is used to track the position and posture of the scanner and execute the computer program stored in the optimization device.

[0054] Compared with the related art, a data processing method, a data processing device and a tracking scanning system are provided in this embodiment. The data processing method is used for the tracking scanning system, and the tracking scanning system includes a tracker and a scanner; the tracking data is generated by tracking the position and posture of the scanner during the scanning process by acquiring the tracker; the camera parameters of the tracker are calculated using the tracking data, and the tracking position and posture of the tracker are adjusted according to the calculation results; the scanning data acquired by the scanner is unified into the tracking coordinate system of the tracker using the adjusted tracking posture, thereby solving the problem of low scanning efficiency of the tracking scanning system.

[0055] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 is a hardware structure block diagram of a terminal of the data processing method of this embodiment;

[0058] Figure 2 is a flow chart of the data processing method of this embodiment;

[0059] Figure 3 is a structural block diagram of the tracking and scanning device of this embodiment;

[0060] Figure 4 is a flow chart of the tracking scanning method of this embodiment. DETAILED DESCRIPTION

[0061] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0063] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the data processing method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0064] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the tracking scanning method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0066] In this embodiment, a tracking and scanning method is provided, which can be used in a tracking and scanning system. The tracking and scanning system includes a tracker and a scanner. Each tracker and each tracker and each scanner can be wirelessly connected and communicated through a network, or can be wired through a USB or the like.

[0067] Specifically, the tracking and scanning system may include one tracker, two trackers, or multiple trackers. The main function of the tracker is to obtain the scanning posture of the scanner. The tracker may be a binocular vision system or a multi-eye vision system. In the embodiment of the present application, a binocular vision system is taken as an example.

[0068] The tracking and scanning system may also include 1 scanner, 2 scanners, or multiple scanners. The number of scanners and the number of trackers are both greater than or equal to 1. The scanner is mainly used to generate local three-dimensional surface data. The scanner may be a laser scanner, a structured light scanner, a light pen, or other devices. In the embodiment of the present application, a multi-line laser scanner is used as an example.

[0069] Figure 2 is a flow chart of the data processing method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0070] Step S210, obtaining a tracker to track the position and posture of the scanner during the scanning process to generate tracking data.

[0071] Among them, at least one tracker is executed for each frame to track the scanning posture of at least one scanner. The process of the tracker tracking the posture of the scanner is specifically that the tracker is usually equipped with a variety of sensors, such as cameras, inertial measurement units, laser rangefinders, etc., for real-time collection of environmental information and the status of the scanner. During the data collection process, the tracker obtains feature points arranged on the surface of the scanner through its sensors, such as reflective marking points or luminous marking points, so as to obtain the position, direction, speed and other posture information of the scanner in real time.

[0072] Alternatively, the tracking and scanning system may further include a stereoscopic target. Feature points are set on the stereoscopic target, and the stereoscopic target is associated with the scanner. The coordinate system conversion relationship between the stereoscopic target and the scanner associated with it is fixed. The feature points of the stereoscopic target fastened to the scanner have image point information on at least two cameras of at least one tracker. The stereoscopic target is mainly used to locate the scanning posture of the scanner. The stereoscopic target can be tightly connected to the scanner and fastened to the scanner; or it can be independent in the middle of the space except for the stereoscopic target that must be used for the tracking scanner, and the stereoscopic target is ensured to be in the field of view of the tracker during scanning. It is sufficient to ensure that the coordinate system conversion relationship between the stereoscopic target and the scanner associated with it is fixed, so that the coordinate system conversion relationship between the stereoscopic target and the scanner can be known in advance. The stereoscopic target is associated with the scanner, that is, one scanner can be associated with one stereoscopic target. If there are multiple scanners, each of the multiple scanners can be associated with one stereoscopic target separately, and multiple scanners can also be associated with the same stereoscopic target. The feature points on the stereo target can be captured by the tracker, that is, they can leave information on the tracker's camera, which is convenient for the tracker to observe and ensure that the tracker can normally reconstruct the feature points on the stereo target in three dimensions and complete the tracking operation. The shape of the stereo target can be a regular shape such as a sphere, or other irregular shapes.

[0073] Then, when the above-mentioned stereo target is arranged, the process of the tracker tracking the position and posture of the scanner can also be specifically that when the same feature point has image point information on at least two cameras at the same time, the three-dimensional point data in the at least two camera coordinate systems of the tracker is obtained according to the tracker parameters. When at least one tracker reconstructs the three-dimensional coordinates of a sufficient number of feature points in three dimensions, the three-dimensionally reconstructed feature point set and the prefabricated feature point library of the stereo target are stereo matched to obtain the conversion relationship between the at least two camera coordinate systems of at least one tracker and the feature point library coordinate system. The coordinate system conversion relationship is the tracking position and posture of the stereo target of at least one scanner in the at least two camera coordinate systems of at least one tracker. According to the known coordinate system conversion relationship between at least one scanner and its stereo target, the tracking position and posture of at least one scanner in the at least two camera coordinate systems of at least one tracker is obtained. The feature point library of the stereo target is made in advance, such as by photogrammetry or pre-established and imported feature point library data.

[0074] The above-mentioned tracking data may specifically include the tracking posture of the scanner tracked by the tracker, and the image points of the above-mentioned feature points on the camera of the tracker. Optionally, the coordinates of the feature points can be three-dimensionally reconstructed based on the image points that match the feature points on the left and right cameras of the tracker, and the calibrated camera parameters of the tracker itself, and the parameters between the left and right cameras. Afterwards, the tracking posture of the feature points under each tracker is calculated based on the coordinates of the feature points after three-dimensional reconstruction and the preset feature point coordinate library, thereby obtaining the above-mentioned tracking data. Among them, the image point is the pixel coordinate generated by the scanner feature point being captured by the camera.

[0075] Step S220, using the tracking data, calculating the camera parameters of the tracker, and adjusting the tracking posture of the tracker according to the calculation result.

[0076] The camera parameters of the tracker include camera intrinsic parameters, camera extrinsic parameters, camera distortion, etc. In this example, the parameters of the tracker can be solved and optimized using at least one or more frames of tracking data.

[0077] The following example illustrates the optimization process based on tracking data. For the camera external parameters, the relative pose relationship between the first camera and the second camera of the tracker in this frame can be calculated by the following formula:

[0078]

[0079]

[0080] in, Represents the rotation matrix in the relative pose relationship between the first camera and the second camera of the tracker in the i-th frame data, The translation matrix representing the relative position relationship between the first camera and the second camera of the tracker in the i-th frame data, represents the rotational position of the stereo target in the first camera coordinate system of the tracker in the i-th frame data, The translation pose of the stereo target in the i-th frame in the first camera coordinate system of the tracker, represents the rotational position of the stereo target in the second camera coordinate system of the tracker in the i-th frame data, The translation pose of the stereo target in the i-th frame in the tracker's second camera coordinate system.

[0081] And / or, the camera's internal parameters (such as focal length, principal point coordinates) and possible distortion parameters (such as radial distortion coefficient) can be calculated based on the tracking data. Taking the tracking data including feature point data on a set stereo target as an example, a correspondence is established between the image coordinates of the feature points in the tracking data and the world coordinates of the corresponding points on the calibration target. A calibration algorithm (such as Zhang Zhengyou calibration method) is used to estimate the camera's internal parameters and distortion parameters. These algorithms usually solve the parameters by minimizing the reprojection error of the feature points.

[0082] Next, the tracking posture is optimized and adjusted based on the calculated camera parameter results, which may specifically include obtaining optimized values ​​of parameters of real-time compensation of at least one tracker and optimized values ​​of frame tracking postures of each camera of at least one tracker, updating the parameter values ​​of real-time compensation of at least one tracker and updating the frame tracking posture values ​​of each camera of at least one tracker, so as to improve the posture tracking accuracy of at least one camera of at least one tracker while optimizing the parameters.

[0083] It should be understood that the above steps can be performed at any time during the scanning process. For example, if the scanning time is from t=0 to t=100, this step can be performed at t=20, using the multi-frame tracking data in the time period of t=0 to 20 for solving, or it can be performed at t=50, using the multi-frame tracking data in the time period of t=0 to 50 for solving. Alternatively, the above steps can also be performed after the scanning is completed, which is not limited here.

[0084] Step S230: Using the adjusted tracking posture, the scan data acquired by the scanner is unified into the tracking coordinate system of the tracker.

[0085] During the scanning process, the tracking posture of the successfully tracked scanner in the corresponding camera coordinate system of the tracker is recorded, and the local three-dimensional surface data collected by the successfully tracked scanner is recorded. According to the coordinate system conversion relationship between the cameras of the tracker in the same frame, and the tracking posture of the scanner in the coordinate system of at least two cameras of the tracker, the local three-dimensional surface data collected by the scanner in the frame can be unified into the same coordinate system through coordinate system conversion. Among them, the conversion process of this unified coordinate system can be expressed as follows:

[0086]

[0087] In the above formula, The matrix represents the rotation coordinate matrix of the coordinate system, T x T y T z represents the translation transformation of the coordinate system, represents the three-dimensional coordinates of the original coordinate system, Represents the 3D coordinates of the target coordinate system.

[0088] In the prior art, it is necessary to calibrate the tracker's internal and external parameters before scanning, use the calibrated tracker to track the scanner's position, and unify the scanner's scanning data into the tracking coordinate system in real time to end the scan and obtain a complete scanning result. However, when the tracker is used for a long time or the use environment changes, the tracker's camera parameters will change, and recalibration is required to ensure the scanning accuracy, which affects the scanning efficiency.

[0089] The tracking scanning method provided in the above embodiment of the present application obtains the tracking data of the tracker during the scanning process to adjust its tracking posture, and finally uses the adjusted tracking posture to unify the scanning data obtained by the scanner into the tracking coordinate system of the tracker. This method is suitable for the use scenarios of single and multiple tracking devices. During the scanning process or after the scanning, the camera parameters of the tracker can be calculated using the tracking data generated during the actual scanning process, and then the tracking posture can be adjusted to optimize the camera parameters of the tracker. No calibration is required in advance, thereby eliminating the need to calibrate all tracker parameters before use or regularly, which can greatly improve the scanning efficiency.

[0090] In some embodiments, the tracking data generated by the tracking device during the scanning process by tracking the position of the scanner includes:

[0091] Get the tracking data generated by tracking the position of the scanner during the scanning process using a tracker with uncalibrated camera parameters.

[0092] It should be noted that the embodiment of the present application can use a tracker with uncalibrated parameters to track the position and posture of the scanner, and then use the tracking data to solve and optimize the parameters of the tracker in the subsequent steps. It can be seen that for such trackers with uncalibrated parameters, the above method can achieve optimization during or after scanning, so that there is no need for calibration before scanning, which leads to a longer overall scanning time, which is conducive to improving scanning efficiency.

[0093] Alternatively, in another embodiment, tracking data acquisition and new parameter optimization can be performed for a tracker with calibrated parameters. At this time, although the tracker has been calibrated, due to long-term use, environmental changes, etc., the calibration accuracy of the tracker has decreased or it is difficult to reach the required accuracy; therefore, the steps described in the above embodiment can also be used to obtain the tracking data of the calibrated parameter tracker during the scanning process for optimization, and update the original parameters to more accurate camera parameters. It can be seen that for this type of tracker with calibrated parameters, the optimization method during or after scanning can also be used, so as to further improve its tracking accuracy while ensuring high scanning efficiency.

[0094] In some embodiments, the tracking and scanning method further includes: obtaining tracking data generated by the tracker in real time during the scanning process, and scanning data generated by the scanner in synchronization with the tracking data. The tracking data and the scanning data are recorded in real time during the scanning process to prepare for subsequent optimization. It should be understood that the synchronization between the tracking data and the scanning data can be achieved by synchronizing timestamps, etc., so as to ensure the accuracy of the scanning.

[0095] Optionally, in one embodiment, based on the above step S220, the camera parameters of the tracker are calculated using the tracking data, and the tracking posture of the tracker is adjusted according to the calculation result, which may specifically include:

[0096] During the scanning process, the tracking data of the currently generated target frame is used to calculate the camera parameters of the tracker, and the tracking posture of the target frame is adjusted according to the calculation results.

[0097] Among them, in this embodiment, the tracking data currently generated during the scanning process can be used to continuously update the camera parameters. For example, during the scanning process, according to the tracking posture of the i-th frame tracked by the scanner by at least one tracker, the camera parameters corresponding to the i-th frame data, including the relative posture relationship between at least two cameras of the tracker, are solved. After that, the historical camera parameters are adjusted using the calculation results corresponding to the i-th frame data. After obtaining the tracking data of the i+1th frame, the above-mentioned camera parameter calculation and adjustment process is repeated. In this way, the real-time calculation and update of the tracker camera parameters can be achieved during the scanning process.

[0098] In addition, the camera parameters of at least one tracker may be calculated based on the tracking data generated within a certain period of time during the scanning process, and the tracking posture of the tracker may be adjusted according to the current calculation result.

[0099] In this embodiment, the camera parameters are calculated synchronously and in real time based on the tracking data generated during the scanning process, and the tracking posture adjustment is achieved during the scanning process.

[0100] Optionally, in one embodiment, based on the above step S220, using the tracking data, calculating the camera parameters of the tracker, and adjusting the tracking posture of the tracker according to the calculation result may include:

[0101] After the scan is completed, the tracking data generated during the scan process is used to calculate the camera parameters of the tracker, and the tracking posture is adjusted according to the parameter calculation results.

[0102] Specifically, after the scan is completed, the tracking data generated by at least one tracker during the entire scanning process can be read, and then the camera parameters of the tracker can be optimized based on the tracking data of the entire scanning process to obtain the final tracking posture of the tracker.

[0103] In this embodiment, the tracking accuracy of the tracker can be improved by calculating the tracker camera parameters based on the tracking data generated during the entire scanning process after the scanning is completed, and then adjusting the tracking posture of the tracker.

[0104] In some of the embodiments, the method of calculating the camera parameters of the tracker using the tracking data and adjusting the tracking posture of the tracker according to the calculation result may further include the following steps:

[0105] According to the image information obtained by each camera of the tracker in the tracking data, the reprojection residual or the point-to-polar line distance residual is constructed, the camera parameters of the tracker are calculated, and the tracking posture of the tracker is adjusted according to the calculation results.

[0106] Among them, after using the tracking data of the tracker on the scanner obtained during the scanning process and calculating the camera parameters of the tracker, the tracking posture of at least one camera of at least one tracker on at least one scanner in each frame can be improved in accuracy and the parameters of the tracker can be optimized.

[0107] Specifically, the entire process can be constructed as an optimization problem, or in other ways. Taking the construction of an optimization problem as an example, the optimized time window can be the entire scanning process or a partial scanning process. If the optimized time window is a partial scanning process, the optimization problem is also independent of multiple ones. More specifically, the construction of the optimization problem can be implemented based on residual construction and other methods. For example, with the goal of minimizing the reprojection residual or the point-to-polar line distance residual, an objective function of the optimization problem is constructed. Then, the constructed objective function is used to optimize the parameters. The optimized parameters may include the camera parameters of the tracker (such as focal length, optical center, distortion coefficient, etc.) and the tracking pose of each frame. Finally, the tracking pose of each frame is adjusted according to the optimized parameters to improve its accuracy.

[0108] Through the above embodiments, an optimization problem is constructed based on constructing a reprojection residual or a point-to-polar line distance residual and the tracking posture of the tracker is adjusted, so that the posture of the tracker can be accurately improved, which is beneficial to improving the accuracy of scanning.

[0109] In some of the embodiments, the tracking data is used to calculate the camera parameters of the tracker, and the tracking posture of the tracker is adjusted according to the calculation result, including:

[0110] Step S221, the camera parameters of the tracker are set to initial values, and a loss function is constructed by reprojection and coordinate transformation reprojection according to the image information acquired by each camera of the tracker in the tracking data.

[0111] In this step, the optimization problem is constructed as follows: the parameter value of the real-time compensation of at least one tracker, the frame tracking pose of each camera of at least one tracker, and the camera parameters of the historical tracker are set as initial values ​​and are to be optimized. The parameter optimization amount can be all parameters or part of the above parameters. For example, for the frame tracking pose, the optimization amount can be RT, or R or T; R represents a three-dimensional rotation matrix, and T represents a one-dimensional translation vector. And / or, for the camera parameters of the historical tracker, the optimization amount can be RT, or R or T.

[0112] The optimization problem is converted into a loss function, and the loss function is minimized by adjusting the parameters to be optimized. The loss function can be composed of a reprojection loss function and a coordinate system change reprojection loss function. Other loss functions can also be added, such as the epipolar geometry loss function.

[0113] One expression of the reprojection loss function is as follows:

[0114]

[0115] In the above formula, J rep is the reprojection residual term, g is the camera index, i is the frame number index, j is the feature point index of the stereo target associated with the scanner, and P g,i,j is the image point of the jth feature point of the stereo target observed by the gth camera in the i-th frame, rep(K g ,R g,i ,T g,i ,(P M ) g,i,j ) is the process of calculating the re-projected image point, and specifically refer to the above formula (1). g is the internal reference of camera g, R g,i is the tracking rotation pose of the scanner by camera g in frame i, T g,i is the tracking translation pose of the scanner by camera g in frame i, (P M ) g,i,j It is the three-dimensional coordinate of the j-th feature point of the scanner's stereo target recorded by the g camera in the i-frame in the stereo target coordinate system.

[0116] P g,i,j The image points and rep(K g ,R g,i ,T g,i ,(P M ) g,i,j ) to determine the two-dimensional distance residual. The reprojected image point is a pixel coordinate value calculated in the camera pixel coordinate system based on the scanner feature points, scanning posture, and camera parameters.

[0117] The coordinate system transformation reprojection loss function is defined as follows:

[0118]

[0119] Where k is the total number of cameras, m is the total number of tracking data frames, and n gi is the total number of scanner feature points observed by the g camera in the i-th frame. g and h represent the camera index, i represents the frame number index, and j represents the feature point index of the stereo target of at least one scanning head. is the reprojection residual after the camera coordinate system transformation, P h,i,j is the image point of the jth 3D point of the stereo target recorded by camera h in frame i, rep(K h ,R g,i ,T g,i ,R gh ,T gh ,(P M ) h,i,j ) represents the reprojected image point after calculating the camera coordinate transformation.

[0120] By Rg,i and T g,i The transformed g camera coordinate system is the position and posture of the stereo target, (P M ) h,i,j Convert to the g camera coordinate system through R gh and T gh The relative position relationship between the g camera and the h camera is (P M ) h,i,j Convert to the h camera coordinate system, and then use the internal parameter matrix K i Project and obtain the re-projected image point. h,i,j K is the image point of the j-th feature point of the stereo target of at least one scanner recorded by the h camera in the i frame. h is the internal parameter of camera h, R g,i is the tracking rotation pose of at least one scanner by camera g in frame i, T g,i is the tracking translation pose of at least one scanner by camera g in frame i, R gh is the rotation matrix of the camera coordinate system from g camera to h camera in the history tracker camera parameters; T gh is the translation matrix of the camera coordinate system from g camera to h camera in the history tracker camera parameters; (P M ) h,i,j is the three-dimensional coordinate of the j-th feature point of the stereo target of at least one scanner recorded by the h camera in the i-th frame in the stereo target coordinate system. h,i,j The image points and rep(K h ,R g,i ,T g,i ,R gh ,T gh ,(P M ) h,i,j ) The distance between the transformed and reprojected image points calculated by the coordinate system determines the two-dimensional distance residual.

[0121] Step S222: Minimize the loss function using a nonlinear optimization algorithm to calculate the optimized values ​​of the camera parameters of the tracker.

[0122] Specifically, the LM (Levenberg-Marquardt) algorithm, Gaussian method, maximum likelihood optimization algorithm, or nonlinear algorithms such as parameter optimization based on machine learning can be used to adjust the parameters to be optimized to reduce the above-mentioned two-dimensional distance residual sum.

[0123] Step S223: updating the tracking posture of the tracker according to the optimized values ​​of the camera parameters.

[0124] Based on the above steps, the optimization value of the parameter of the real-time compensation of at least one tracker, the optimization value of the frame tracking pose of each camera of at least one tracker, and the optimization value of the camera parameter of the historical tracker can be obtained. According to the optimization value, the parameter value of the real-time compensation of at least one tracker is updated, the frame tracking pose value of each camera of at least one tracker is updated, and the value of the camera parameter of the historical tracker is updated.

[0125] Through the above steps S221 to S223, the tracking and scanning method provided in the present application is optimized and solved by constructing the reprojection residual using image information, which is beneficial to improving the optimization accuracy; and since the reprojection residual is based on image points, the parallelism of image processing can be used to improve computational efficiency.

[0126] Additionally, in one embodiment, based on the above step S220, the camera parameters of the tracker are calculated using the tracking data, and the tracking posture of the tracker is adjusted according to the calculation result, which may specifically include:

[0127] Based on the frame tracking poses of each camera of the tracker during the scanning process contained in the tracking data and the historical camera parameters of the tracker, the distance from the image point to each polar line of the camera is calculated, wherein each polar line of the camera is obtained by the polar line equation calculated by matching the image points of other cameras; according to the distance from the image point to each polar line of the camera, an optimization function is constructed, with the goal of minimizing the distance from all image points to each polar line of the camera, the optimization function is solved, and the optimized values ​​of the camera parameters of the tracker are obtained; according to the optimized values ​​of the camera parameters of the tracker, the tracking pose of the tracker is updated.

[0128] In this step, the feature points (such as the above-mentioned marking points) used for tracking on the scanner can be reprojected to the camera planes of all trackers based on the tracking data of the scanner by the tracker and the historical camera parameters of the tracker to obtain reprojected image points. The same feature point has a pair of matching image points under at least two cameras of at least one tracker.

[0129] According to the historical camera parameters of the tracker, the relative position relationship RT between the cameras of the tracker is obtained. Between a pair of cameras, there is a common observed image point matching pair. For the same image point matching pair, one image point A in the matching pair is on the pixel plane of the first camera a of the tracker, and the other image point B is on the pixel plane of the second camera b of the tracker.

[0130] Using the phase and pose relationship RT between the cameras, the epipolar line b_line of the second camera b can be obtained through the image point A and RT. There is a point-to-line distance between the epipolar line b_line and the image point B. Similarly, the epipolar line a_line of the first camera a is obtained through the image point B and RT, and there is a point-to-line distance between the epipolar line a_line and the image point A.

[0131] Based on this, taking the epipolar equation of the second camera of the tracker as an example, the epipolar equation can be constructed as follows:

[0132]

[0133] Where b_line is the epipolar line equation of the second camera b, and coeff_a, coeff_b, and coeff_c are the coefficients of the equation. A_L v A_L 1” is the coordinate of the image point in the first camera a of the tracker. 3×3 The matrix calculation process is as follows:

[0134]

[0135] Among them, IM_b and IM_a are the intrinsic parameter matrices of the first camera a and the second camera b respectively, and E is the essential matrix, which includes the relative position relationship RT of the cameras.

[0136] Afterwards, based on the epipolar line equation b_line of the second camera b, the distance from the image point of the first camera on the epipolar line of the second camera to the image point B of the second camera b can be calculated.

[0137] Based on the above expression for solving the distance between the feature point and the corresponding epipolar lines of each camera of the tracker, the optimization function can be constructed as follows:

[0138]

[0139] L gh,i,j Indicates the distance from the j image point of camera g in frame i to the j image point of camera h on the epipolar line of camera h.

[0140] Therefore, in this embodiment, for the above-mentioned optimization function, the relative pose relationship between the cameras of the tracker can be solved so that the distance between all feature points and the corresponding epipolar lines is minimized, thereby achieving global optimization of the relative pose relationship between the cameras of the tracker.

[0141] In some of the embodiments, for the above optimization process, the data processing method further includes the following step: using a preset standard to constrain the calculation results of the camera parameters of the tracker.

[0142] In this step, an additional standard can be used to assist the modified optimization process; in other words, in the aforementioned optimization process, an additional scale constraint is added. One of the expressions of the constructed standard is shown in the following formula:

[0143]

[0144] D gh,i，j,q = rec(R gh ,T gh ,K g ,K h ,u gj ,v gj ,u gq ,v gq ,u hj ,v hj ,u hq ,v hq )

[0145] Among them, D gh,i，j,q is the distance between the feature point of the standard j and the feature point of the standard q. In this calculation process, the relative position relationship R of the camera can be used to calculate the distance between the feature point of the standard j and the feature point of the standard q. gh ,T gh , the intrinsic parameter matrix of the g camera, the intrinsic parameter matrix of the h camera, the image point u of the j and q feature points under the g camera gj ,v gj ,u gq ,v gq , j and q feature points at image point u under camera h hj ,v hj ,u hq ,v hq , thereby calculating the distance between two feature points. truth,gh,i,j,q is the prior value of the distance. By adjusting the tracker parameters, such as R gh ,T gh ,K g ,K h , so that the residual of is reduced by J scale , in order to constrain the scale accuracy of the optimization, effectively improving the accuracy of data processing and scanning.

[0146] It should be noted that the rec function is the process of calculating the distance from feature point j to feature point q. For this function, its calculation process can be expressed as follows:

[0147] rec(R gh ,T gh ,K g ,K h ,u gj ,v gj ,u gq ,vgq ,u hj ,v hj ,u hq ,v hq )=||rec3D(R gh ,T gh ,K g ,K h ,u gj ,v gj ,u hj ,v hj )-rec3D(R gh ,T gh ,K g ,K h ,u gq ,v gq ,u hq ,v hq )||

[0148]

[0149]

[0150] In the above formula, rec3D(R gh ,T gh ,K g ,K h ,u gj ,v gj ,u hj ,v hj ) Based on the relative position relationship R of the camera gh ,T gh , g camera's intrinsic parameter matrix K g ,h The intrinsic parameter matrix K of the camera h , the pixel coordinates u of feature point j on camera g gj ,v gj , the pixel coordinates u of feature point j on camera h hj ,v hj , the three-dimensional coordinates of feature point j can be calculated rec3D(R gh ,T gh ,K g ,K h ,u gq ,v gq ,u hq ,v hq ) Based on the relative position relationship R of the camera gh ,T gh , g camera's intrinsic parameter matrix K g ,h The intrinsic parameter matrix K of the camera h , the pixel coordinates u of feature point j on camera g gq,v gq , the pixel coordinates u of feature point j on camera h hq ,v hq , the three-dimensional coordinates of the q feature point can be calculated

[0151] In some embodiments, the data processing method may further include the following steps:

[0152] During the scanning process, based on the current frame acquired by the tracker, it is determined whether there is a camera that has not obtained the scanner's tracking posture; if so, the tracking information posture of the camera that has not obtained the scanner's tracking posture is completed based on the information of the camera that has obtained the scanner's tracking posture and the historical camera parameters of the tracker.

[0153] It should be noted that when using at least two cameras of the tracker to track the scanner, there are situations where all cameras successfully track the scanner's posture, and there are also situations where some cameras fail to track the scanner. Based on this, in this embodiment, it is first detected whether there is a camera that fails to track. If so, it is determined whether the camera can find the coordinate transformation relationship from the camera to the camera with the tracking posture of at least one scanner through the camera parameters of the historical tracker. If so, based on the camera information of the tracking posture of at least one scanner and combined with the camera parameters of the historical tracker, the tracking information posture of the camera without tracking posture information is supplemented. The purpose of the supplement is to provide relevant data when the information camera without tracking posture regains the tracking posture of the scanner.

[0154] Through the above embodiment, by completing the posture of the tracker that has not successfully acquired the scanner tracking posture, it is avoided that when one of the cameras of the tracker fails in tracking, the required scanner tracking data cannot be obtained, resulting in the inability to generate comprehensive scanning data, thereby effectively improving the accuracy of the scanning.

[0155] In some embodiments, the method of completing the tracking information posture of the camera that has not obtained the scanner tracking posture according to the information of the camera that has obtained the scanner tracking posture and combining the historical camera parameters of the tracker also includes the following steps:

[0156] The tracking pose of the camera that tracks the tracking pose of the scanner is converted into a coordinate system according to the camera parameters of the historical tracker, and converted into the camera coordinate system of the camera that has not obtained the scanner tracking pose, so as to obtain a priori tracking pose; according to the prior tracking pose, the camera parameters of the historical tracker are used to obtain the re-projected image points obtained by re-projecting the image points of the scanner onto the image of the camera that has not obtained the scanner tracking pose; the correspondence between the historical image points and the re-projected image points recorded by the camera that has not obtained the scanner tracking pose is calculated; and according to the corresponding relationship, the tracking information pose of the camera that has not obtained the scanner tracking pose is completed.

[0157] Specifically, the tracking posture of the camera with at least one scanner tracking posture is converted to the camera coordinate system without tracking information through one or more coordinate system conversions using the camera parameters of the historical tracker; this coordinate system conversion process is shown in the following formula:

[0158]

[0159] In the above formula, represents the calculated reprojected image point, rep(K,R,T,(P M ) g,i,j ) is the calculation process of the reprojected image point, K is the internal parameter matrix, (P M ) g,i,j is the j-th 3D coordinate of the stereo target used to calculate the reprojected image point of the i-th frame of the g camera.

[0160] The tracking pose of the above coordinate transformation is called the prior tracking pose. According to the prior tracking pose, all feature points of the stereo target of at least one scanner can be re-projected onto the camera image using the camera parameters to obtain the re-projected image points. The calculation process of the re-projected image points is as follows:

[0161]

[0162] in, is the internal parameter matrix, R is the rotational posture of the stereo target in the i-frame of the g camera, T is the translational posture of the stereo target in the i-frame of the g camera, XM is the x-component of the three-dimensional coordinates of a feature point of the scanner, YM is the y-component of the three-dimensional coordinates of a feature point of the scanner, and ZM is the z-component of the three-dimensional coordinates of a feature point of the scanner.

[0163] Next, the correspondence between the recorded image points and the calculated re-projected image points is determined by a distance threshold. If a correspondence is found, the correspondence between the recorded image points and the feature points of the stereoscopic target of at least one scanner can be obtained. The correspondence between the recorded image points and the feature points of the stereoscopic target of at least one scanner needs to be recorded for subsequent optimization.

[0164] It is understandable that when the number of trackers is determined, the maximum value of the number of camera parameters is upper bounded. Based on this, as an implementable method, in the process of calculating the camera parameters of the tracker, when the camera parameters between the two cameras are recalculated in a certain frame, the historical camera parameters of the two cameras are retained; or the camera parameters of the two cameras are directly overwritten; or according to the tracking error of the tracking posture of the scanner in the coordinate system of the two cameras, it is determined whether the camera parameters of the two cameras need to be overwritten; or according to the preset weights, the historical camera parameters of the two cameras and the newly calculated camera parameters of the two cameras are fused. The preset weights can be set in combination with actual conditions. Through the above embodiments, by retaining historical parameters, directly overwriting or processing based on weight fusion and other different methods, the number of camera parameters can be limited to a feasible range, effectively expanding its application scenarios.

[0165] In some embodiments, the data processing method further includes the following steps:

[0166] Obtain the parameter compensation model of the tracker, the parameter compensation coefficient of the tracker, the parameter compensation model of the scanner, and the parameter compensation coefficient of the scanner. Among them, each parameter compensation model and parameter compensation coefficient can be set according to the initialization parameters of each device. The initialization parameters of the tracker are used to generate three-dimensional data of the characteristics of the tracked target, which can be the factory initialization value or the value calibrated again after leaving the factory. If the scanner is a laser binocular scanner, the parameters of the scanner include camera intrinsic parameters, distortion, and the relative posture relationship between the camera pair. The parameter compensation model of the scanner can be temperature compensation, gravity compensation, etc.

[0167] During the scanning process, the tracker's parameter compensation model and the tracker's parameter compensation coefficient are used to perform real-time compensation for the tracker's parameters, and the scanner's parameter compensation model and the scanner's parameter compensation coefficient are used to perform real-time compensation for the scanner's parameters, and the tracker's parameters after real-time compensation are recorded for each frame.

[0168] In the above scanning process, the parameter compensation model and parameter compensation coefficient are used to compensate the parameters of the corresponding tracker and scanner in real time. The parameter compensation model and parameter compensation coefficient are used to compensate the parameters of the tracker in real time when the tracker performs tracking work to ensure the tracking accuracy and success rate.

[0169] Compensation can be performed periodically to update the parameter values ​​used. The number of data frames in each period can be one frame or multiple frames. The compensation period can be different for multiple trackers. During the scanning process, the actual parameters used by at least one tracker per frame can be recorded for subsequent use. The compensation of scanner parameters is similar to the compensation of tracker parameters, which is also compensated periodically. The compensation period between different scanners can be inconsistent.

[0170] Through the above embodiment, the calculated camera parameters are periodically compensated, thereby avoiding the problem of parameter optimization being easily affected by environmental factors during the scanning process, and further improving the accuracy anomalies generated during the scanning process, which is beneficial to improving the accuracy of the tracking scanning system.

[0171] In some embodiments, the above-mentioned calculation of the camera pose relationship of the tracker using the tracking data and adjusting the tracking pose of the tracker according to the calculation result includes the following steps:

[0172] The parameter values ​​of the tracker after real-time compensation, the frame tracking poses of each camera of the tracker, and the camera parameters of the historical tracker are set as initial values, and the loss function is constructed by reprojection and coordinate transformation reprojection based on the image information obtained by each camera of the tracker;

[0173] The nonlinear optimization algorithm is used to minimize the loss function, and the optimized values ​​of the parameters of the real-time compensation of the tracker, the optimized values ​​of the frame tracking poses of each camera of the tracker, and the optimized values ​​of the camera parameters are calculated;

[0174] According to the optimized values ​​of the real-time compensation parameters of the tracker, the optimized values ​​of the frame tracking poses of each camera of the tracker, and the optimized values ​​of the camera parameters, the parameter values ​​of the real-time compensation of the tracker are updated, the frame tracking poses of each camera of the tracker are updated, and the camera parameters of the tracker are updated.

[0175] It is understandable that the method of constructing the reprojection residual of the compensated parameter value to optimize the tracking posture is the same as the method of constructing the reprojection residual and optimizing the camera parameters, so the specific process is not repeated here. In this embodiment, by constructing the residual of the compensated parameter value to further optimize the parameter and posture, it is beneficial to further improve the accuracy of data processing and scanning, and effectively ensure the accuracy and reliability of the scanning process.

[0176] In some of the embodiments, the data processing method further includes the following steps: after the scan data acquired by the scanner is unified to the tracking coordinate system of the tracker, the scan data is sent to an associated display terminal for display. The display terminal includes but is not limited to various display screens, personal computers, laptops, smart phones and tablet computers. The display terminal can be directly connected to the tracker and scanner, and the user terminal can also be deployed in the same local area network as the scanner; alternatively, the display terminal can also be connected to the scanner wirelessly to realize data interaction and processing between the display terminal and the scanner.

[0177] Specifically, the display terminal may be equipped with scanning control software for realizing the scanning function; the scanning control software displays a graphical user interface (GUI) on the display terminal. Then, each frame of local three-dimensional surface data after the coordinates of at least one scanner are unified during the scanning process, and / or the scanning data after the coordinates of at least one scanner are unified after the scanning is completed, can be sent to the display terminal and displayed in real time on the GUI interface, so as to facilitate the user to check the scanning status in time or interact with it.

[0178] In this embodiment, a tracking scanning device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0179] Figure 3 is a structural block diagram of the tracking and scanning device of this embodiment, such as Figure 3 As shown, the device is applied to a tracking and scanning system, which includes a tracker and a scanner; the tracking and scanning device includes a data acquisition module 31, a tracking posture optimization module 32, and a scanning data unification module 33;

[0180] A data acquisition module 31 is used to acquire tracking data generated by a tracker by tracking the position and posture of a scanner during scanning;

[0181] A tracking posture optimization module 32 is used to calculate the camera parameters of the tracker using the tracking data, and adjust the tracking posture of the tracker according to the calculation results;

[0182] The scanning data unification module 33 is used to unify the scanning data acquired by the scanner into the tracking coordinate system of the tracker using the adjusted tracking posture.

[0183] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination. For specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0184] In addition, in combination with the data processing method provided in the above embodiments, a tracking and scanning system is also provided in this embodiment. The tracking and scanning system includes a tracker and a scanner, and an optimization device; a computer program is stored on the optimization device, and when the computer program is executed by the processor, the steps of any one of the tracking and scanning methods in the above embodiments are implemented.

[0185] This embodiment also provides a tracking scanning method, using the tracking scanning system described in the above embodiment to perform three-dimensional scanning on the scanning data. Figure 4 , the method comprises the following steps:

[0186] Step S410, starting the tracking and scanning system;

[0187] Step S420, using a tracker to track the position and posture of the scanner, and executing a computer program stored in the optimization device.

[0188] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0189] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0190] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0191] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0192] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: Used in a tracking and scanning system, the tracking and scanning system includes a tracker and a scanner, the method includes: Acquire the tracker, and generate tracking data by tracking the position and posture of the scanner during scanning; Calculating camera parameters of the tracker using the tracking data, and adjusting the tracking posture of the tracker according to the calculation results; The scan data acquired by the scanner is unified into the tracking coordinate system of the tracker using the adjusted tracking posture.

2. The data processing method according to claim 1, characterized in that: The obtaining of the tracking data generated by the tracker by tracking the position and posture of the scanner during the scanning process includes: The tracking data is generated by tracking the position and posture of the scanner during a scanning process using the tracker without calibrating the camera parameters.

3. The data processing method according to claim 1, characterized in that: The step of calculating camera parameters of the tracker by using the tracking data and adjusting the tracking posture of the tracker according to the calculation result includes: During the scanning process, the tracking data of the currently generated target frame is used to calculate the camera parameters of the tracker, and the tracking posture of the target frame is adjusted according to the calculation result.

4. The data processing method according to claim 1, characterized in that: The step of calculating camera parameters of the tracker by using the tracking data and adjusting the tracking posture of the tracker according to the calculation result includes: After the scanning is completed, the camera parameters of the tracker are calculated using the tracking data generated during the scanning process, and the tracking posture is adjusted according to the parameter calculation results.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The step of calculating camera parameters of the tracker by using the tracking data and adjusting the tracking posture of the tracker according to the calculation result includes: According to the image information acquired by each camera of the tracker in the tracking data, a reprojection residual or a point-to-polar line distance residual is constructed, the camera parameters of the tracker are calculated, and according to the calculation results, the tracking posture of the tracker is adjusted.

6. The data processing method according to any one of claims 1 to 4, characterized in that: The step of calculating camera parameters of the tracker by using the tracking data and adjusting the tracking posture of the tracker according to the calculation result includes: The camera parameters of the tracker are set to initial values, and a loss function is constructed by using reprojection and coordinate transformation reprojection according to the image information acquired by each camera of the tracker in the tracking data; Minimizing the loss function using a nonlinear optimization algorithm to calculate optimized values ​​of the camera parameters of the tracker; The tracking posture of the tracker is updated according to the optimized value of the camera parameter.

7. The data processing method according to any one of claims 1 to 4, characterized in that: The step of calculating the camera parameters of the tracker by using the tracking data, and adjusting the tracking posture of the tracker according to the calculation result, includes: Based on the frame tracking poses of each camera of the tracker during the scanning process contained in the tracking data and the historical camera parameters of the tracker, calculating the distances from the image points to the corresponding epipolar lines of the camera; According to the distance from the image point to each corresponding camera epipolar line, an optimization function is constructed to minimize the distance from all image points to each corresponding camera epipolar line, and the optimization function is solved to obtain the optimized value of the camera parameter of the tracker; The tracking posture of the tracker is updated according to the optimized values ​​of the camera parameters of the tracker.

8. The data processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: The calculated results of the camera parameters of the tracker are constrained using a preset standard.

9. The data processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: During the scanning process, based on the current frame acquired by the tracker, determining whether there is a camera that has not obtained the tracking posture of the scanner; If it exists, the tracking information posture of the camera that has not obtained the scanner tracking posture is supplemented according to the information of the camera that has obtained the scanner tracking posture and the historical camera parameters of the tracker.

10. The data processing method according to claim 9, characterized in that: The method of completing the tracking information posture of the camera that has not obtained the tracking posture of the scanner according to the information of the camera that has obtained the tracking posture of the scanner and combining the historical camera parameters of the tracker includes: The tracking pose of the camera that tracks the tracking pose of the scanner is converted into the camera coordinate system of the camera that does not obtain the tracking pose of the scanner according to the camera parameters of the historical tracker to obtain the prior tracking pose; According to the prior tracking posture, using the historical camera parameters of the tracker, obtaining the re-projected image points obtained by re-projecting the image points of the scanner onto the image of the camera that does not obtain the scanner tracking posture; Calculating the correspondence between the historical image points recorded by the camera that does not obtain the scanner tracking posture and the re-projected image points; According to the corresponding relationship, the tracking information posture of the camera that has not obtained the scanner tracking posture is completed.

11. The data processing method according to any one of claims 1 to 4, characterized in that: In the process of calculating the camera parameters of the tracker, when the camera parameters between the two cameras are recalculated in a certain frame, the historical camera parameters of the two cameras are retained; or the camera parameters of the two cameras are directly overwritten; or according to the tracking error of the tracking posture of the scanner in the coordinate system of the two cameras, it is determined whether the camera parameters of the two cameras need to be overwritten; or according to the preset weights, the historical camera parameters of the two cameras and the newly calculated camera parameters of the two cameras are fused.

12. The data processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a parameter compensation model of the tracker, a parameter compensation coefficient of the tracker, a parameter compensation model of the scanner, and a parameter compensation coefficient of the scanner; During the scanning process, the parameters of the tracker are compensated in real time using the parameter compensation model of the tracker and the parameter compensation coefficient of the tracker, and the parameters of the scanner are compensated in real time using the parameter compensation model of the scanner and the parameter compensation coefficient of the scanner, and the parameters of the tracker after real-time compensation are recorded for each frame.

13. The data processing method according to claim 12, characterized in that: The step of calculating the position and posture relationship of the camera pair of the tracker by using the tracking data, and adjusting the tracking position and posture of the tracker according to the calculation result, comprises: The parameter values ​​of the tracker after real-time compensation, the frame tracking poses of each camera of the tracker, and the historical camera parameters of the tracker are set as initial values, and a loss function is constructed by reprojection and coordinate transformation reprojection according to the image information acquired by each camera of the tracker; Minimizing the loss function using a nonlinear optimization algorithm, and calculating optimized values ​​of parameters of real-time compensation of the tracker, optimized values ​​of frame tracking poses of each camera of the tracker, and optimized values ​​of camera parameters; According to the optimized values ​​of the parameters of the real-time compensation of the tracker, the optimized values ​​of the frame tracking poses of the various cameras of the tracker, and the optimized values ​​of the camera parameters, the parameter values ​​of the real-time compensation of the tracker are updated, the frame tracking poses of the various cameras of the tracker are updated, and the camera parameters of the tracker are updated.

14. The data processing method according to any one of claims 1 to 4, characterized in that: The method further comprises: After the scan data acquired by the scanner is unified into the tracking coordinate system of the tracker, the scan data is sent to an associated display terminal for display.

15. A data processing device, characterized in that: Used in a tracking and scanning system, the tracking and scanning system includes a tracker and a scanner, and the device includes a data acquisition module, a tracking posture optimization module, and a scanning data unification module; The data acquisition module is used to acquire the tracking data generated by the tracker by tracking the position and posture of the scanner during the scanning process; The tracking posture optimization module is used to calculate the camera parameters of the tracker using the tracking data, and adjust the tracking posture of the tracker according to the calculation result; The scanning data unification module is used to unify the scanning data acquired by the scanner into the tracking coordinate system of the tracker using the adjusted tracking posture.

16. A tracking and scanning system, characterized in that: The tracking and scanning system includes a tracker and a scanner, and an optimization device; A computer program is stored on the optimization device, and when the computer program is executed by a processor, the steps of the data processing method described in any one of claims 1 to 14 are implemented.

17. A tracking scanning method, characterized in that: Using the tracking scanning system of claim 16 to perform three-dimensional scanning on a scanning object, the method comprises: Starting the tracking and scanning system; The tracker is used to track the position and posture of the scanner and execute the computer program stored in the optimization device.

Images