A data processing method, apparatus, and tracking scanning system

Abstract

The application relates to a data processing method and device and a tracking scanning system, wherein the data processing method comprises the following steps: acquiring at least two trackers, tracking the pose of a scanner in a scanning process to generate tracking data; calculating the relative pose relationship of the at least two trackers by using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers according to the calculation result; and unifying the scanning data acquired by the scanner to a global coordinate system by using the adjusted relative pose relationship of the at least two trackers. In the use scene of the multiple tracking devices, the relative pose relationship between the trackers is calculated in real time based on the tracking of the pose of the scanner by the trackers in the scanning process, the relative pose relationship between the trackers is calibrated in advance, and the scanning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of stereo vision measurement technology, and in particular to a data processing method, apparatus, and tracking scanning system. Background Technology

[0002] Surface data detection technology has wide applications in parts size and position measurement, robot guidance, industrial design, defect detection, and reverse engineering. It primarily involves scanning the spatial shape, structure, and color of objects to obtain their spatial coordinates. In related technologies, tracking scanning systems (also known as tracking scanners) are typically used to detect surface data. A tracking scanning system generally includes a tracker and a scanner. The tracker is mainly used to position the scanner, while the scanner is primarily used to detect surface data. Therefore, the scanning range is limited to the field of view that the tracking device can track. Based on this, multi-tracker scanning systems, including multiple trackers, have emerged to meet the detection needs of larger scenarios.

[0003] Multi-tracker scanning systems often require calibration of the relative poses between multiple trackers to ensure final scanning accuracy. Related technologies typically calibrate the relative poses of the trackers before use. This is done using specific calibration rods or plates to calibrate the trackers after placement, thus obtaining the pose relationships between them. However, this method requires recalibration after the trackers move, resulting in low efficiency.

[0004] There is currently no effective solution to the problem of low scanning efficiency in multi-tracker scanning systems in related technologies. Summary of the Invention

[0005] This embodiment provides a data processing method, apparatus, and tracking scanning system to solve the problem of low scanning efficiency in multi-tracker scanning systems in related technologies.

[0006] In a first aspect, this embodiment provides a data processing method for a tracking scanning system, the tracking scanning system including a scanner and at least two trackers, the method comprising the following steps:

[0007] Acquire the at least two trackers, track the pose of the scanners during the scanning process, and generate tracking data;

[0008] Using the tracking data, the relative pose relationship of the at least two trackers is calculated, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results;

[0009] Using the adjusted relative pose relationship of the at least two trackers, the scanning data acquired by the scanner is unified to the global coordinate system.

[0010] In some embodiments, the acquisition of the at least two trackers, tracking the pose of the scanners during the scanning process, and generating tracking data include:

[0011] The at least two trackers with uncalibrated relative poses are acquired, and the poses of the scanners are tracked during the scanning process to generate tracking data.

[0012] In some embodiments, the step of using the tracking data to calculate the relative pose relationship of the at least two trackers, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes:

[0013] During the scanning process, the relative pose relationship of the at least two trackers is calculated using the tracking data of the currently generated target frame, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

[0014] In some embodiments, the step of using the tracking data to calculate the relative pose relationship of the at least two trackers, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes:

[0015] After the scan is completed, the relative pose relationship of the at least two trackers is calculated using the tracking data generated during the scan, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

[0016] In some embodiments, the step of using the tracking data to calculate the relative pose relationship of the at least two trackers, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes:

[0017] Based on the image information obtained by the at least two trackers in the tracking data, a reprojection residual or a point-to-epidial distance residual is constructed to calculate the relative pose relationship of the at least two trackers, and the historical relative pose relationship of the at least two trackers is optimized based on the calculation results.

[0018] In some embodiments, the tracking data is used to calculate the relative pose relationship of the at least two trackers, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results, including:

[0019] Based on the common tracking pose of the at least two trackers for at least one frame and the image information of the scanner in at least one camera of the at least two trackers for the corresponding frame, the reprojection of the at least two trackers and the reprojection loss function after the tracker coordinate system transformation are obtained.

[0020] The loss function is minimized using a nonlinear optimization algorithm to calculate the optimized value of the relative pose relationship between the at least two trackers;

[0021] The historical relative pose relationship of the at least two trackers is adjusted based on the optimized value of the relative pose relationship of the at least two trackers.

[0022] In some embodiments, the step of using the tracking data to calculate the relative pose relationship of the at least two trackers, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes:

[0023] Acquire the relative pose relationships of at least two trackers in history and the tracked pose of the successfully tracked scanner;

[0024] Based on the historical relative pose relationship between at least two trackers and the successfully tracked scanner pose, the distance from the image point to each epipolar line of the corresponding camera is calculated, wherein each epipolar line of the camera is obtained by calculating the epipolar line equation through the camera-matched image points of other trackers;

[0025] Based on the distances from the image points to the corresponding epipolar lines of the camera, an optimization function is constructed with the objective of minimizing the distances from all image points to the corresponding epipolar lines of the camera. The optimization function is then solved to obtain the optimized values ​​of the relative pose relationship between the at least two trackers.

[0026] The historical relative pose relationship of the at least two trackers is adjusted based on the optimized value of the relative pose relationship of the at least two trackers.

[0027] In some embodiments, the method further includes:

[0028] Acquire frame information or time information where the relative pose relationship of at least two trackers changes abruptly during the scanning process.

[0029] In some embodiments, the step of using the tracking data to calculate the relative pose relationship of the at least two trackers, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes:

[0030] The entire scanning process is divided into at least one time window, and the relative pose relationship of the at least two trackers within the same time window remains unchanged.

[0031] In each time window, the relative pose relationship of the at least two trackers is calculated using the tracking data, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

[0032] In some embodiments, the method further includes:

[0033] Based on the relative pose relationship of the at least two trackers optimized within each of the scanning time windows, the local 3D scanning data generated by the corresponding scanners are unified to the global coordinate system.

[0034] In some embodiments, the method further includes:

[0035] After unifying the local 3D scan data generated by the scanner into a global coordinate system, it is sent to the associated display terminal for display.

[0036] In some embodiments, the method further includes:

[0037] During the scanning process, when the scanner is successfully tracked by only one of the at least two trackers and there is a historical relative pose relationship between the at least two trackers, the tracking pose of the currently successfully tracked tracker on the feature points of the scanner, the relative pose relationship between the two cameras of the currently untracked tracker, and the historical relative pose relationship between the at least two trackers are combined to complete the tracking pose of the currently untracked tracker, so as to obtain the actual tracking pose of the currently untracked tracker.

[0038] Based on the tracking pose of the feature points of the scanner by the currently successfully tracked tracker and the actual tracking pose of the currently failed tracker, the real-time relative pose relationship of the at least two trackers is calculated, and the historical relative pose relationship of the at least two trackers is updated based on the real-time relative pose relationship of the at least two trackers.

[0039] In some embodiments, the step of calculating the real-time relative pose relationship of the at least two trackers based on the tracking pose of the feature points of the scanner by the currently successfully tracked tracker and the actual tracking pose of the currently untracked tracker, and updating the historical relative pose relationship of the at least two trackers based on the real-time relative pose relationship of the at least two trackers, includes:

[0040] Based on the tracking pose of the scanner's feature points by the currently successfully tracked tracker, the relative pose relationship of the camera of the currently untracked tracker, and combined with the historical relative pose relationship of the at least two trackers, the predicted tracking pose of the scanner's feature points in the coordinate system of at least one camera of the currently untracked tracker is obtained.

[0041] Based on the predicted tracking pose, the feature points of the scanner are projected onto the image plane of at least one camera of the tracker that is currently failing to track, generating reprojected image points.

[0042] Based on the correspondence between the reprojected image points and the image points, the correspondence between the image points and the feature points of the scanner is obtained;

[0043] Based on the correspondence between the image points and the feature points of the scanner, and according to the preset optimization algorithm, the predicted tracking pose is optimized with the minimum distance between the reprojected image points as the optimization objective, and the optimization result is used as the actual tracking pose in at least one camera coordinate system of the tracker that is currently failing to track.

[0044] The actual tracking pose of the tracker that has failed to track is obtained based on the actual tracking pose in at least one camera coordinate system of the tracker that has failed to track and the relative pose relationship between the two cameras of the tracker that has failed to track.

[0045] Secondly, this embodiment provides a data processing device for a tracking scanning system, the tracking scanning system including a scanner and at least two trackers, the device including a data acquisition module, a tracker relative pose optimization module, and a scanning data unification module;

[0046] The data acquisition module is used to acquire the at least two trackers, track the pose of the scanners during the scanning process, and generate tracking data;

[0047] The tracker relative pose optimization module is used to calculate the relative pose relationship of the at least two trackers using the tracking data, and to adjust the historical relative pose relationship of the at least two trackers based on the calculation results.

[0048] The scan data unification module is used to unify the scan data acquired by the scanner to a global coordinate system using the adjusted relative pose relationship of the at least two trackers.

[0049] Thirdly, this embodiment provides a tracking scanning system, which includes a scanner, at least two trackers, and an optimization device;

[0050] The optimization device stores a computer program, which, when executed by a processor, implements the steps of the data processing method described in the first aspect above.

[0051] Fourthly, this embodiment provides a tracking scanning method, which uses the tracking scanning system described in the third aspect above to perform a three-dimensional scan of the object being scanned. The method includes:

[0052] Turn on the tracking and scanning system;

[0053] The at least two trackers are used to track the pose of the scanner and execute the computer program stored in the optimization device.

[0054] Compared with related technologies, this embodiment provides a data processing method, apparatus, and tracking scanning system. The data processing method first acquires data from at least two trackers, tracking their poses during the scanning process to generate tracking data. Then, using the tracking data, it calculates the relative pose relationship between the at least two trackers and adjusts their historical relative pose relationship based on the calculation results. Finally, it uses the adjusted relative pose relationship between the at least two trackers to unify the scanning data acquired by the scanner into a global coordinate system. In scenarios with multiple tracking devices, this method calculates the real-time relative pose relationship between trackers based on their tracking of the scanner's pose during the scanning process, eliminating the need for pre-calibrating the relative pose relationship between trackers and improving scanning efficiency.

[0055] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0057] Figure 1 This is a hardware structure block diagram of the terminal of the data processing method in this embodiment;

[0058] Figure 2 This is a flowchart of the data processing method in this embodiment;

[0059] Figure 3 This is a structural block diagram of the data processing device in this embodiment;

[0060] Figure 4 This is a schematic diagram of the tracking and scanning system in this embodiment;

[0061] Figure 5 This is a flowchart of the tracking and scanning method in this embodiment. Detailed Implementation

[0062] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0063] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0064] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the data processing method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown are illustrated.

[0065] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0067] This embodiment provides a data processing method for a tracking scanning system, which includes a scanner and at least two trackers. Figure 2 This is a flowchart of the data processing method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0068] Step S210: Acquire at least two trackers, track the pose of the scanners during the scanning process, and generate tracking data.

[0069] The aforementioned tracker can be implemented based on a binocular vision system or a multi-view vision system, and its main function is to obtain the scanning posture of the scanner. This embodiment uses a binocular vision system as the tracker. In the tracking scanning system, there is at least one scanner. The scanner can be a laser scanner, a structured light scanner, a light pen, or other types of scanners suitable for 3D scanning scenarios. This embodiment uses a multi-line laser scanner as an example.

[0070] During scanning, the tracker can track the scanner's pose by tracking a stereo target associated with the scanner, thereby obtaining tracking data. For example, the stereo target is mainly used to determine the scanner's scanning posture. The stereo target can be tightly connected to the scanner, or it can be an independent target in space, separate from the stereo targets necessary for tracking the scanner, ensuring the stereo target is within the tracker's field of view during scanning. As long as the coordinate system transformation relationship between the stereo target and its associated scanner is fixed, this transformation relationship can be known in advance. The stereo target and scanner association settings allow one scanner to be associated with one stereo target. If there are multiple scanners, each scanner can be associated with a separate stereo target, or multiple scanners can be associated with the same stereo target. Feature points on the stereo target can be reflective or luminous markers, as long as they can be captured by the tracker and leave information on the tracker's camera, facilitating observation and ensuring the tracker can correctly reconstruct the feature points on the stereo target in 3D to complete the tracking operation. The shape of a three-dimensional target can be a regular shape such as a sphere, or it can be other irregular shapes.

[0071] Furthermore, during the scanning process, the tracker can also track the scanner's pose by tracking feature points fixedly set on the scanner's surface, thereby generating tracking data. This can be achieved by either attaching markers to the scanner's surface or by placing the scanner within a pre-set rigid frame with markers on the frame. Those skilled in the art can choose the method for tracking the scanner's pose according to the needs of the actual application scenario; this embodiment does not impose specific limitations on this. The feature points set on the scanner's surface can be reflective markers or luminous markers, as long as they can be captured by the tracker, i.e., leaving information on the tracker's camera, facilitating observation by the tracker and ensuring that the tracker can correctly reconstruct the feature points on the stereo target in three dimensions, thus completing the tracking operation.

[0072] The aforementioned tracking data may specifically include the tracking pose tracked by the tracker to the scanner, and the image points of the aforementioned feature points on the tracker's cameras. Optionally, the feature point coordinates can be reconstructed in 3D based on the image points of the feature points matched on the left and right cameras of the tracker, the camera parameters already calibrated by the tracker, and the parameters between the left and right cameras. Then, the reconstructed feature point coordinates are matched with a preset feature point coordinate library to calculate the tracking pose of the feature points under each tracker, thereby obtaining the aforementioned tracking data.

[0073] For example, when using a first tracker and a second tracker to track the pose of a scanner, based on the image points acquired by the two trackers respectively, the corresponding feature point coordinates can be reconstructed in 3D. By matching these coordinates with a preset feature point coordinate library, the tracking pose of the scanner obtained by the first tracker can be obtained. The tracking pose obtained by the second tracker currently tracking the scanner Where i represents the i-th frame in which the tracker tracks the pose of the scanner.

[0074] Subsequently, before scanning, the camera intrinsic parameters, distortion, and other parameters of each tracker can be pre-calibrated, along with the relative pose relationship between the left and right cameras of each tracker. Additionally, the scanner's intrinsic parameters and the relative pose of its left and right cameras can be calibrated. When using a stereo target to track the scanner's pose, the coordinate transformation relationship between the scanner's coordinate system and the stereo target's coordinate system can also be pre-calibrated before scanning.

[0075] After completing the above calibration, multiple trackers can be deployed together in the tracking scanning scene. When setting the positions of multiple trackers, each tracker can be configured to track the scanner's pose simultaneously with at least one other tracker in at least one scan frame. That is, it is guaranteed that at least two trackers can achieve joint tracking with each other in at least one scan frame.

[0076] Step S220: Using the tracking data, calculate the relative pose relationship of at least two trackers, and adjust the relative pose relationship of at least two trackers based on the calculation results.

[0077] After obtaining the aforementioned tracking data, the relative pose relationship between at least two trackers can be calculated based on the tracking poses tracked by the scanner from the at least two trackers included therein. For example, during the scanning process, in at least one frame of data, based on the poses of the scanner successfully tracked simultaneously by two trackers (e.g., the first tracker and the second tracker), and The relative pose relationship between the two trackers in this frame can be calculated using the following formula:

[0078]

[0079] in, This represents the rotation matrix in the relative pose relationship between the first and second trackers in the i-th frame of data. The translation matrix represents the relative pose relationship between the first and second trackers in the i-th frame of data. This represents the rotational pose of the scanner in the first tracker coordinate system of the i-th frame of data. This represents the translational pose of the scanner in the first tracker coordinate system in the i-th frame. This represents the rotational pose of the stereo target in the second tracker coordinate system in the i-th frame of data. The translational pose of the stereo target in the second tracker coordinate system in the i-th frame.

[0080] Therefore, if multiple trackers share at least one frame of common tracking data, their relative pose relationships can be calculated in real time based on this data. This real-time calculation result can then be used to update previously calculated historical relative pose relationships. These historical relative pose relationships can be either calculated during the scanning process based on previously generated tracking data, or they can be pre-calibrated before scanning using other methods. Understandably, if there is no common tracking in the current frame, the historical relative pose relationships will continue to be used.

[0081] Step S230: Using the adjusted relative pose relationship of at least two trackers, the scanning data acquired by the scanner is unified to the global coordinate system.

[0082] After obtaining the adjusted relative pose relationship of at least two trackers, the scanning data of the scanner in the coordinate systems of at least two trackers are integrated into the global coordinate system to obtain scanning data in the same coordinate system, thereby obtaining the final 3D scanning result of the scanned object. This global coordinate system can be a pre-selected master tracker coordinate system. The master tracker coordinate system can be a coordinate system selected from multiple tracker coordinate systems. Furthermore, the global coordinate system is not limited to the established tracker coordinate system; it can also be the workpiece coordinate system or the scene coordinate system of background marker points in the scanning scene, etc. This embodiment does not specifically limit the global coordinate system.

[0083] In existing related technologies, multiple trackers are often calibrated before scanning using specific calibration rods or calibration plates as calibration objects to obtain the relative pose relationships between the trackers. However, this method of calibration requires recalibration when the tracker position moves, resulting in low scanning efficiency.

[0084] Based on this, this embodiment, through the above steps, can directly determine the relative pose relationships of the trackers during the scanning process by tracking the scanner's pose, without needing to pre-determine the relative pose relationships between multiple trackers before scanning. Furthermore, even if the tracker's position moves during scanning, the real-time relative pose relationships can be determined based on the real-time tracking data, thereby updating the historical relative pose relationships. Therefore, compared to related technologies, this embodiment improves scanning efficiency. It should also be noted that in obtaining the aforementioned relative pose relationships, it is not necessary for each tracker to simultaneously reconstruct the same 3D feature point; it is only necessary for them to simultaneously track the scanner's pose. Therefore, this embodiment offers more flexible tracker position settings.

[0085] Steps S210 to S230 above involve acquiring at least two trackers, tracking the scanner's pose during the scanning process, generating tracking data, calculating the relative pose relationship between the at least two trackers using the tracking data, adjusting the historical relative pose relationship between the at least two trackers based on the calculation results, and using the adjusted relative pose relationship between the at least two trackers to unify the scanning data acquired by the scanner into a global coordinate system. In multi-tracker application scenarios, this method calculates the real-time relative pose relationship between trackers based on their tracking of the scanner's pose during the scanning process, eliminating the need for pre-calibrating the relative pose relationship between trackers and improving scanning efficiency.

[0086] In one embodiment, based on step S210 above, at least two trackers are acquired, and the pose of the scanners is tracked during the scanning process. The generated tracking data may include:

[0087] Acquire at least two trackers with uncalibrated relative poses, track the poses of the scanners during the scanning process, and generate tracking data.

[0088] It should be further noted that the embodiments of this application may use multiple trackers whose relative pose relationships have not been calibrated to track the pose of the scanner, and then use the tracking data to calculate and optimize the relative pose relationships between the trackers in subsequent steps. It is evident that for such trackers with uncalibrated relative pose relationships, calibration can be achieved during or after scanning using the above method, thus eliminating the need for pre-scan calibration, which would otherwise result in a longer overall scanning time and improve scanning efficiency.

[0089] Alternatively, in another embodiment, the tracking data acquisition and optimization of new relative pose relationships can be performed on trackers with pre-calibrated relative pose relationships. Although the relative pose relationships between trackers have been pre-calibrated, changes in usage time, environmental variations, or human movement of the trackers can cause these relationships to change. Therefore, the steps provided in the above embodiments can be used to acquire the tracking data of the trackers with pre-calibrated relative pose relationships during the scanning process for further optimization. It is evident that for trackers with pre-calibrated relative pose relationships, global optimization during or after scanning can be used to further improve scanning accuracy while maintaining high scanning efficiency.

[0090] In another embodiment, the above data processing method may further include:

[0091] Acquire tracking data generated in real time during the scanning process, as well as scan data generated by the scanner in sync with the tracking data.

[0092] The scanning process records tracking and scanning data in real time, preparing for subsequent optimization. It should be understood that synchronization between tracking and scanning data can be achieved through methods such as synchronizing timestamps or frame number markers, thereby ensuring scanning accuracy.

[0093] Optionally, in one embodiment, based on the above step S220, the relative pose relationship of at least two trackers is calculated using tracking data, and the historical relative pose relationship of at least two trackers is adjusted according to the calculation result, which may specifically include:

[0094] During the scanning process, the relative pose relationship of at least two trackers is calculated using the tracking data of the currently generated target frame, and the historical relative pose relationship of at least two trackers is adjusted based on the calculation results.

[0095] In this embodiment, the relative pose relationship can be continuously updated using the tracking data generated during the scanning process. For example, during the scanning process, based on the tracking pose of the first tracker and the second tracker on the i-th frame tracked by the scanner, the relative pose relationship between the first and second trackers corresponding to the i-th frame data is calculated. Then, the historical relative pose relationship is adjusted using this relative pose relationship corresponding to the i-th frame data. After obtaining the tracking data for the (i+1)-th frame, the adjustment process of the relative pose relationship is repeated. Thus, real-time calculation and updating of the relative pose relationship between the trackers can be achieved during the scanning process.

[0096] Alternatively, based on the tracking data generated during a certain period of time during the scanning process, the relative pose relationship between at least two trackers can be calculated, and the historical relative pose relationship can be adjusted according to the currently calculated relative pose relationship.

[0097] In this embodiment, by calculating the relative pose relationship based on the generated tracking data during the scanning process and adjusting the historical relative pose relationship during the scanning process, the scanning efficiency when multiple trackers are tracking simultaneously can be further improved.

[0098] Optionally, in one embodiment, based on the above step S220, calculating the relative pose relationship of at least two trackers using tracking data, and adjusting the historical relative pose relationship of at least two trackers according to the calculation result, may include:

[0099] After the scan is completed, the relative pose relationship of at least two trackers is calculated using the tracking data generated during the scan, and the historical relative pose relationship of at least two trackers is adjusted based on the calculation results.

[0100] Specifically, after the scan is completed, the tracking data generated by each tracker during the entire scan process can be read. Then, based on the tracking data of the entire scan process, the relative pose relationship between the trackers can be optimized to obtain the final relative pose relationship between the trackers.

[0101] In this embodiment, by calculating the relative pose relationship between at least two trackers based on the tracking data generated throughout the scanning process after the scanning is completed, and then adjusting the historical relative pose relationship, the accuracy of the pose relationship can be improved.

[0102] Furthermore, in one embodiment, based on the above step S220, calculating the relative pose relationship of at least two trackers using tracking data, and adjusting the historical relative pose relationship of at least two trackers according to the calculation result, may include:

[0103] Based on the image information obtained from at least two trackers in the tracking data, reprojection residuals or point-to-epidural distance residuals are constructed to calculate the relative pose relationship of at least two trackers. Based on the calculation results, the historical relative pose relationship of at least two trackers is optimized.

[0104] In addition, by using the tracking data of the tracker to the scanner obtained during the scanning process and calculating the relative pose relationship of the trackers, the accuracy of the tracking pose of at least one camera of each tracker to at least one scanner can be improved, and the relative pose relationship between each tracker can be optimized.

[0105] Specifically, the entire process can be structured as an optimization problem, or in other ways. Taking the construction of an optimization problem as an example, the optimization time window can be the entire scanning process or a partial scanning process. If the optimization time window is a partial scanning process, then the optimization problem is also divided into multiple independent problems. More specifically, the construction of the optimization problem can be based on residual construction, etc. For example, the objective function of the optimization problem can be constructed with the goal of minimizing the reprojection residual or the point-to-epidural distance residual. Then, the constructed objective function is used to optimize the relative pose relationships between trackers, as well as the tracking poses of each frame. Finally, the historical relative pose relationships between each tracker are adjusted based on the optimized results.

[0106] Through the above embodiments, optimization problems are constructed based on methods such as constructing reprojection residuals or point-to-epidial distance residuals, and the relative pose relationships of each tracker are adjusted, thereby improving the accuracy of the relative pose relationships between trackers and improving the scanning accuracy.

[0107] In one embodiment, based on step S220 above, calculating the relative pose relationship of at least two trackers using tracking data, and adjusting the historical relative pose relationship of at least two trackers according to the calculation results, may include:

[0108] Based on the common tracking pose of at least two trackers for at least one frame and the image information of the scanner in the corresponding frame on at least one camera of at least two trackers, the reprojection of at least two trackers and the reprojection loss function after tracker coordinate system transformation are obtained; the loss function is minimized using a nonlinear optimization algorithm to calculate the optimized value of the relative pose relationship of at least two trackers; based on the optimized value of the relative pose relationship of at least two trackers, the historical relative pose relationship of at least two trackers is adjusted.

[0109] In this embodiment, the accuracy of the relative pose relationship between trackers can be improved through optimization. During the scanning process, after each tracker is placed, the normal scanning process begins. Once the relative pose of the trackers changes significantly, such as due to manual movement, the gradient of the change in the relative pose relationship between the trackers will change abruptly. Therefore, the change in the relative pose relationship between the trackers can be determined based on the gradient of the relative pose relationship or by other means. Within the time window when the relative pose relationship between the trackers remains unchanged, the real-time relative pose of each tracker at different times is recorded. Not entirely. In this case, the change in real-time relative pose over time is mainly caused by measurement uncertainties and other factors. Therefore, in order to improve the accuracy of the final relative pose relationship between the trackers, at least one frame of common tracking pose is taken within this time window, and the image point information of the stored feature points in at least one camera of each tracker is recorded to construct an optimization problem for the relative pose relationship of the trackers.

[0110] The optimization problem is transformed into constructing a loss function, resulting in the reprojection loss function of at least two trackers and the reprojection loss function after coordinate system transformation. During the optimization process, the loss function is minimized by adjusting the relative pose relationships to be optimized. The loss function can be composed of the reprojection loss function and the coordinate system transformation reprojection loss function. Other loss functions, such as epipolar geometry loss functions, can also be added.

[0111] One expression for the reprojection loss function is shown below:

[0112]

[0113] Among them, J rep Let g be the reprojection residual term, i be the tracker index, i be the frame number index, and j be the feature point index of the scanner. For the g-th tracker, the left camera observes the j-th feature point in the i-th frame. k is the total number of trackers, m is the total number of frames of tracking data, and n gi It is the total number of scanner feature points that the g-th tracker can observe in the i-th frame. This describes the process of calculating the reprojected image point of the j-th feature point on the left camera of the g-th tracker. It is the intrinsic parameter matrix of the left camera of the g-th tracker. It is the rotational pose of the left camera of the scanner in the g-th tracker at frame i. The scanner's translation pose on the left camera of the g-th tracker in frame i, (P M ) g,i,j It is the three-dimensional coordinate of feature point j of the scanner in the scanner coordinate system, recorded by the g-th tracker in frame i. It is the j-th feature point recorded by the right camera of the g tracker in the i-th frame. It calculates the reprojected image point of the j-th feature point on the right camera of tracker g. It is the intrinsic parameter matrix of the right camera of the g tracker. It is the rotational attitude of the scanner in the i-th frame of the right camera of the tracking head. This represents the translational posture of the scanner in the g-th frame of the right camera. It should also be noted that the image points are the pixel coordinates generated by the camera capturing the scanner's feature points; the projected image points are pixel coordinate values ​​calculated in the camera pixel coordinate system based on the scanner's feature points, scanning posture, and camera parameters.

[0114] The coordinate system transformation reprojection loss function is defined as follows:

[0115]

[0116] Where g and h represent tracker indices, i represents frame number index, and j represents feature point index of at least one scanner. It is the reprojection residual after the tracker coordinate system transformation. It is the j-th feature point of the scanner recorded by the left camera of the h-th tracker in the i-th frame; This represents the calculation of the reprojected image points after camera coordinate transformation, through... The pose of the scanner in the left camera coordinate system of the g-th tracker will be (P M ) h,i,j Transform to the left camera coordinate system of the g-th tracker, via R gh ,T gh The relative pose relationship between the g-th tracker and the h-th tracker, (P M ) h,i,j Transform to the left camera coordinate system of the h-th tracker, and then use the intrinsic parameter matrix. Project the image to obtain the reprojected image points; It is the j-th feature point of the scanner recorded by the right camera of the h-th tracker in the i-th frame. This represents the calculation of the reprojected image points after camera coordinate transformation, through... The pose of the scanner in the right camera coordinate system of the g-th tracker, will be (P M ) h,i,j Transform to the right camera coordinate system of the g-th tracker, via R gh ,T gh The relative pose relationship between the g-th tracker and the h-th tracker, (P M ) h,i,j Transform to the coordinate system of the right camera of the h-th tracker, and then use the intrinsic parameter matrix. Project the image to obtain the reprojected image points.

[0117] Based on the reprojection of the tracked pose and the reprojection after relative pose coordinate transformation using the tracker, the two reprojection errors of at least one camera of each tracker can be calculated using the above process: and

[0118] During the reprojection process, the following formula can be used for calculation:

[0119]

[0120] in, This represents the focal length of the left camera of the g-th tracker in the x-direction. This represents the focal length of the left camera of the g-th tracker in the y-direction. This represents the focal length of the left camera of the h-th tracker in the y-direction. This represents the principal point in the x-direction of the left camera of the g-th tracker. Let X represent the principal point of the left camera of the g-th tracker in the y-direction, (X M ) g,i,j Let x represent the x-coordinate of the j-th scanner feature point observed by the g-th tracker in the i-th frame, (y = x - y) M ) g,i,j Let y = (Z) represent the y-coordinate of the j-th scanner feature point observed by the g-th tracker in the i-th frame. M ) g,i,j Let represent the z-coordinate of the j-th scanner feature point observed by the g-th tracker in the i-th frame. This represents the focal length of the left camera of the h-th tracker in the x-direction. This represents the focal length of the left camera of the h-th tracker in the y-direction. This represents the principal point in the x-direction of the left camera of the h-th tracker. This represents the principal point in the y-direction of the left camera of the h-th tracker.

[0121] J can be constructed using two reprojection errors. rep and The loss function can be specifically adjusted using nonlinear algorithms such as least squares, Bayesian optimization, maximum likelihood optimization, or parameter optimization based on machine learning, to adjust the relative pose relationships between the trackers to be optimized. If there are multiple scanning time windows in the scanning process, the data is divided according to the time windows to construct multiple independent optimization problems. Under different time windows, the optimized relative pose relationships for that time window are obtained.

[0122] Optimizing the tracking scanning method provided in this application by constructing reprojection residuals using image information is beneficial to improving optimization accuracy; furthermore, since the reprojection residuals are based on image points, the parallelism of image processing can be used to improve computational efficiency.

[0123] In another embodiment, based on step S220 above, the relative pose relationship of at least two trackers is calculated using tracking data, and the historical relative pose relationship of at least two trackers is adjusted according to the calculation result. Specifically, this may include:

[0124] Obtain the historical relative pose relationships of at least two trackers and the tracking poses of successfully tracked scanners; based on the historical relative pose relationships of at least two trackers and the tracking poses of successfully tracked scanners, calculate the distances from image points to the epipolar lines of the corresponding cameras, where each epipolar line is obtained by calculating the epipolar line equations through camera-matched image points of other trackers; construct an optimization function based on the distances from image points to the corresponding epipolar lines of the cameras, with the objective of minimizing the distances from all image points to the corresponding epipolar lines of the cameras, solve the optimization function, and obtain the optimized values ​​of the relative pose relationships of at least two trackers; adjust the historical relative pose relationships of at least two trackers based on the optimized values ​​of the relative pose relationships of at least two trackers.

[0125] When using at least two trackers to track a scanner, there are cases where all trackers successfully track the scanner's pose, and cases where some trackers fail to track the scanner. For example, if the left and right cameras of a certain tracker do not simultaneously observe feature points, 3D reconstruction will fail. However, in this case, the tracker still has image point information from at least one camera that may contain the feature point. Therefore, based on the tracking data of the successfully tracked trackers and combined with the historical relative pose relationships between the trackers, the feature points (such as the aforementioned marker points) used for tracking on the scanner can be reprojected onto the camera planes of all trackers to obtain reprojected image points. Here, the same feature point will have a pair of matching image points on the cameras of two trackers.

[0126] Based on the historical relative pose relationships between the trackers, the distance from the aforementioned image point on any camera of any tracker to the corresponding epipolar lines of that camera can be calculated. These corresponding epipolar lines are obtained by calculating the epipolar equations using matching image points from other cameras. For example, for the same feature point, the image point on the pixel plane of the left camera of the first tracker is A_L, and the image point on the pixel plane of the right camera of the first tracker is A_R. Similarly, the image point on the pixel plane of the left camera of the second tracker is B_L, and the image point on the pixel plane of the right camera of the second tracker is B_R.

[0127] By utilizing the relative pose relationships between the trackers, the left camera epipolar line b_line_L1 and the right camera epipolar line b_line_R1 of the second tracker can be obtained from image point A_L. Correspondingly, the left camera epipolar line b_line_L2 and the right camera epipolar line b_line_R2 of the second tracker can also be obtained from image point A_R.

[0128] Correspondingly, the relative pose relationships between the trackers can also be used to obtain the left camera epipolar line a_line_L1 and the right camera epipolar line a_line_R1 of the first tracker through image point B_L. Furthermore, the left camera epipolar line a_line_L2 and the right camera epipolar line a_line_R2 of the first tracker can be obtained through image point B_R.

[0129] The epipolar lines b_line_L1, b_line_L2 and image point B_L have distances from the point to the line; the epipolar lines b_line_R1, b_line_R2 and image point B_R have distances from the point to the line; the epipolar lines a_line_L1, a_line_L2 and image point A_L have distances from the point to the line; the epipolar lines a_line_R1, a_line_R2 and image point A_R have distances from the point to the line.

[0130] Based on this, taking the epipolar equation of the left camera of the second tracker as an example, the epipolar equation can be constructed as follows:

[0131]

[0132] Where b_line_L1 is the epipolar equation of the left camera of the second tracker, and coeff_a, coeff_b, and coeff_c are the equation coefficients. A_L v A_L "1" represents the coordinates of the image point on the left camera of the first tracker. F 3×3 The matrix calculation process is as follows:

[0133]

[0134] Wherein, IM_b_L and IM_a_L are the intrinsic parameter matrices of the left camera of the second tracker and the left camera of the first tracker, respectively, and E is the essential matrix, which includes the relative pose relationship between the left camera of the second tracker and the left camera of the first tracker.

[0135] Then, based on the epipolar equation b_line_L1 of the second tracker's left camera, the distance L from the image point of the first tracker's left camera to the epipolar line of the second tracker's left camera can be calculated. a_L,b_L,B_L :

[0136]

[0137] Based on the above expression for solving the distance between feature points and the epipolar lines of at least two corresponding tracker cameras, the optimization function can be constructed as follows:

[0138]

[0139] L g_L,h_L,i,j_L L represents the distance from the j-image point of the left camera of tracker g in frame i to the epipolar line of the left camera of tracker h. g_L,h_R,i,j_R Let j be the epipolar line of the left camera of tracker g in frame i, and the distance to j is the distance from the right camera of tracker h.

[0140] L g_R,h_L,i,j_L L represents the distance from the j-image point of the right camera of tracker g in frame i to the j-image point of the left camera of tracker h, located on the epipolar line. g_R,h_R,i,j_R Let j be the epipolar line of the right camera of tracker g in frame i, and the distance to j is the distance from the right camera of tracker h.

[0141] Therefore, in this embodiment, for the above-mentioned optimization function, the relative pose relationship between each tracker can be solved to minimize the distance between all feature points and the corresponding epipolar lines, thereby achieving global optimization of the relative pose relationship between each tracker.

[0142] In particular, in one embodiment, the above data processing method may further include:

[0143] Acquire frame or time information when the relative pose relationship of at least two trackers undergoes abrupt changes during the scanning process. Specifically, when trackers are moved manually or due to other factors, causing significant changes in their positions, abrupt changes in the relative pose relationship between the trackers will occur. Therefore, by observing the real-time changes in the relative pose relationship, when abrupt changes in the relative pose relationship of at least two trackers are determined, the corresponding frame or time information is recorded to pinpoint the time point of tracker movement or change, providing supplementary information for subsequent data processing and analysis.

[0144] In one embodiment, based on step S220 above, calculating the relative pose relationship of at least two trackers using tracking data, and adjusting the historical relative pose relationship of at least two trackers according to the calculation results, may include:

[0145] The entire scanning process is divided into at least one time window, and the relative pose relationship of at least two trackers within the same time window remains unchanged. In each time window, the relative pose relationship of at least two trackers is calculated using tracking data, and the historical relative pose relationship of at least two trackers is adjusted based on the calculation results.

[0146] For example, based on the frame information and time information of the recorded relative pose relationships of each tracker that change abruptly, the entire scanning process can be divided into multiple sub-scanning processes. Each sub-scanning process is a time window, and in each sub-scanning process, global optimization of the relative pose relationships will be performed within the corresponding time window.

[0147] Based on this, in this embodiment, by dividing the time window, the relative pose relationship of at least two trackers within the same time window remains unchanged. Then, global optimization and updates are performed on the relative pose relationship within each time window, which can improve the accuracy of the relative pose relationship throughout the scanning process.

[0148] In one embodiment, the above data processing method may further include:

[0149] Based on the relative pose relationship of at least two trackers optimized within each scanning time window, the local 3D scanning data generated by the corresponding scanners are unified into the global coordinate system.

[0150] Specifically, the relative pose relationships of each tracker obtained in the final optimization within each time window can be obtained, and the local 3D scanning data generated by the scanner can be unified to a unified global coordinate system pre-selected within that time window, such as the coordinate system of the main tracker corresponding to the pre-selected time window.

[0151] In this embodiment, by unifying the global coordinate system of the local 3D scans generated by the scanner within each time window based on the divided time windows, the influence of sudden changes in the tracker position on the final scan results can be avoided, the accuracy of the tracker's relative pose relationship can be improved, and the accuracy of data integration can be improved.

[0152] Furthermore, in one embodiment, the above data processing method may further include:

[0153] After unifying the local 3D scan data generated by the scanner into the global coordinate system, it is sent to the associated display terminal for display.

[0154] The display terminal includes, but is not limited to, various displays, personal computers, laptops, smartphones, and tablets. The display terminal can communicate directly with the aforementioned trackers and scanners, or it can be deployed on the same local area network as the scanner; alternatively, the display terminal can also connect wirelessly to the scanner, enabling data interaction and processing between the display terminal and the scanner.

[0155] Specifically, the aforementioned display terminal can be equipped with scanning control software that implements scanning functions; this scanning control software displays a graphical user interface (GUI) on the display terminal. Thus, for each frame of local three-dimensional surface data of at least one scanner during the scanning process, and / or, for the scan data of at least one scanner unified to the global coordinate system after the scanning is completed, the data can be sent to the display terminal and displayed in real time on the GUI interface, thereby facilitating users to promptly check the scanning status or interact with it.

[0156] In another embodiment, the above data processing method may further include:

[0157] During the scanning process, when the scanner is successfully tracked by only one of at least two trackers, and there is a historical relative pose relationship between at least two trackers, the tracking pose of the currently successfully tracked tracker on the feature points of the scanner, the relative pose relationship between the two cameras of the currently untracked tracker, and the historical relative pose relationship between at least two trackers are combined to complete the tracking pose of the currently untracked tracker, thus obtaining the actual tracking pose of the currently untracked tracker. Based on the tracking pose of the currently successfully tracked tracker on the feature points of the scanner and the actual tracking pose of the currently untracked tracker, the real-time relative pose relationship between at least two trackers is calculated, and the historical relative pose relationship between at least two trackers is updated based on the real-time relative pose relationship between at least two trackers.

[0158] When the left and right cameras on the tracker do not simultaneously observe the scanner, resulting in the failure of 3D reconstruction of feature points, the tracker will fail to track the scanner. When there are historical relative pose relationships between the trackers, and there is a current situation where a tracker fails to track the scanner, the feature points may have image point information on at least one camera of the tracker that failed to track. In this case, based on the tracking pose of the tracker that successfully tracked the scanner (e.g., the first tracker) and the relative pose relationships between the two cameras of the second tracker that failed to track, combined with the historical relative pose relationships between the trackers, the predicted tracking pose of the scanner in the coordinate system of at least one camera of the second tracker can be obtained, thus obtaining the aforementioned actual tracking pose.

[0159] Furthermore, the tracking pose of the first tracker's feature points on the scanner, which successfully tracked them, is combined with the actual tracking pose of the second tracker's tracking points on the scanner, which failed to track as calculated above, to obtain the relative pose relationship between the first and second trackers. Based on this, the calculation of the relative pose relationship between the trackers can continue even if one tracker fails to track.

[0160] In this embodiment, when there are trackers that fail to track, the actual tracking pose of the scanner under the camera of the tracker that failed to track is calculated based on the historical relative pose relationship and the tracking pose of the feature points of the scanner obtained by the tracker that successfully tracked. This allows for the determination of the relative pose relationship between the trackers, which can improve the stability of the relative pose optimization and reduce the impact of accidental tracking failures of the trackers on the optimization results.

[0161] In one embodiment, based on the tracking poses of the currently successfully tracked tracker's feature points on the scanner and the actual tracking poses of the currently unsuccessfully tracked trackers, the relative pose relationship between at least two trackers in real time is calculated. Then, based on the real-time relative pose relationship between the at least two trackers, the historical relative pose relationship between at least two trackers is updated. Specifically, this may include:

[0162] Based on the tracking pose of the scanner's feature points by the currently successfully tracked tracker, the relative pose relationship of the cameras of the currently failed tracker, and the historical relative pose relationships of at least two trackers, the predicted tracking pose of the scanner's feature points in the coordinate system of at least one camera of the currently failed tracker is obtained. Based on the predicted tracking pose, the scanner's feature points are projected onto the image plane of at least one camera of the currently failed tracker, generating reprojected image points. Based on the correspondence between the reprojected image points and the image points, the correspondence between the image points and the scanner's feature points is obtained. Based on the correspondence between the image points and the scanner's feature points, and according to a preset optimization algorithm, with the minimum distance between the reprojected image points and the image points as the optimization objective, the predicted tracking pose is optimized, and the optimization result is used as the actual tracking pose in the coordinate system of at least one camera of the currently failed tracker. Based on the actual tracking pose in the coordinate system of at least one camera of the currently failed tracker and the relative pose relationship between the two cameras of the currently failed tracker, the actual tracking pose of the currently failed tracker is obtained.

[0163] Continuing with the example of the first tracker that successfully tracked the scanner and the second tracker that failed to track it, the following steps are taken. First, based on the tracking pose obtained by tracking the feature points of the scanner using the first tracker that successfully tracked the scanner, and the historical relative pose relationship between the first and second trackers, the predicted tracking pose of the scanner in at least one camera coordinate system of the second tracker is calculated.

[0164] Next, by predicting the tracking pose, the feature points of the scanner are projected onto the image plane of at least one camera of the second tracker using the reprojection formula, forming reprojected image points. Then, utilizing the correspondence between the image points and the reprojected image points, and combining optimization algorithms such as least squares, gradient descent, and Newton's method, the predicted tracking pose in the coordinate system of at least one camera of the second tracker is used as the optimization quantity to solve for the predicted tracking pose that minimizes the distance between the reprojected image points and the image points. This yields the actual tracking pose of the scanner's feature points in the coordinate system of at least one camera of the second tracker where tracking has failed. Furthermore, based on the actual tracking pose in the coordinate system of at least one camera of the currently failed second tracker and the relative pose relationship between the two cameras of the second camera, the actual tracking pose of the scanner's feature points corresponding to the currently tracked tracker is obtained.

[0165] In this embodiment, by using reprojection and optimization algorithms, and combining the historical relative poses between trackers with the tracking results of successfully tracked trackers, the relative pose relationships between trackers can still be optimized even in the case of trackers that fail to track, thereby further improving the stability and accuracy of relative pose relationship optimization.

[0166] This embodiment also provides a data processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0167] Figure 3 This is a structural block diagram of the data processing device 30 in this embodiment. The data processing device 30 is used in a tracking scanning system, which includes a scanner and at least two trackers. Figure 3 As shown, the data processing device 30 includes: a data acquisition module 32, a tracker relative pose optimization module 34, and a scan data unification module 36; wherein:

[0168] The data acquisition module 32 is used to acquire data from at least two trackers, track the pose of the scanner during the scanning process, and generate tracking data. The tracker relative pose optimization module 34 is used to calculate the relative pose relationship of at least two trackers using the tracking data, and adjust the historical relative pose relationship of at least two trackers based on the calculation results. The scan data unification module 36 is used to unify the scan data acquired by the scanner to the global coordinate system using the adjusted relative pose relationship of at least two trackers.

[0169] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination. Specific examples in this embodiment can be found in the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0170] This embodiment also provides a tracking and scanning system. Figure 4 This is a schematic diagram of the tracking and scanning system 40 in this embodiment, as shown below. Figure 4 As shown, the tracking and scanning system 40 includes a scanner 42 and at least two trackers 44, as well as an optimization device 46; the optimization device 46 stores a computer program, which, when executed by a processor, implements the steps of the data processing method provided in any of the above embodiments.

[0171] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0172] In one embodiment, a tracking scanning method is also provided, which uses the tracking scanning system provided in the above embodiment to perform three-dimensional scanning of the object to be scanned. Figure 5 This is a flowchart of the tracking and scanning method in this embodiment, as follows: Figure 5 As shown, the tracking and scanning method includes the following steps:

[0173] Step S501: Start the tracking and scanning system;

[0174] Step S502: Using at least two trackers, the pose of the scanner is tracked, and the computer program stored in the optimization device is executed.

[0175] The above steps S501 to S502 can calculate the relative pose relationship of the trackers in real time based on the tracking data tracked by the scanner in the use scenario of multiple tracking devices, eliminating the need to pre-calibrate the relative pose relationship between each tracker, improving scanning efficiency and ensuring scanning accuracy.

[0176] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0178] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0179] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0180] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, For a tracking scanning system, the tracking scanning system including a scanner and at least two trackers, the method includes the following steps: Acquire the at least two trackers, track the pose of the scanners during the scanning process, and generate tracking data; Using the tracking data, the relative pose relationship of the at least two trackers is calculated, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results; Using the adjusted relative pose relationship of the at least two trackers, the scanning data acquired by the scanner is unified to the global coordinate system.

2. The data processing method according to claim 1, characterized in that, The acquisition of the at least two trackers, tracking the pose of the scanners during the scanning process, and generating tracking data include: The at least two trackers with uncalibrated relative poses are acquired, and the poses of the scanners are tracked during the scanning process to generate tracking data.

3. The data processing method according to claim 1, characterized in that, The step of calculating the relative pose relationship of the at least two trackers using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes: During the scanning process, the relative pose relationship of the at least two trackers is calculated using the tracking data of the currently generated target frame, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

4. The data processing method according to claim 1, characterized in that, The step of calculating the relative pose relationship of the at least two trackers using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes: After the scan is completed, the relative pose relationship of the at least two trackers is calculated using the tracking data generated during the scan, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

5. The data processing method according to any one of claims 1 to 4, characterized in that, The step of calculating the relative pose relationship of the at least two trackers using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes: Based on the image information obtained by the at least two trackers in the tracking data, a reprojection residual or a point-to-epidial distance residual is constructed to calculate the relative pose relationship of the at least two trackers, and the historical relative pose relationship of the at least two trackers is optimized based on the calculation results.

6. The data processing method according to any one of claims 1 to 4, characterized in that, Using the tracking data, the relative pose relationship of the at least two trackers is calculated, and based on the calculation results, the historical relative pose relationship of the at least two trackers is adjusted, including: Based on the common tracking pose of the at least two trackers for at least one frame and the image information of the scanner in at least one camera of the at least two trackers for the corresponding frame, the reprojection of the at least two trackers and the reprojection loss function after the tracker coordinate system transformation are obtained. The loss function is minimized using a nonlinear optimization algorithm to calculate the optimized value of the relative pose relationship between the at least two trackers; The historical relative pose relationship of the at least two trackers is adjusted based on the optimized value of the relative pose relationship of the at least two trackers.

7. The data processing method according to any one of claims 1 to 4, characterized in that, The step of calculating the relative pose relationship of the at least two trackers using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes: Acquire the relative pose relationships of at least two trackers in history and the tracked pose of the successfully tracked scanner; Based on the historical relative pose relationship between at least two trackers and the successfully tracked scanner pose, the distance from the image point to each epipolar line of the corresponding camera is calculated, wherein each epipolar line of the camera is obtained by calculating the epipolar line equation through the camera-matched image points of other trackers; Based on the distances from the image points to the corresponding epipolar lines of the camera, an optimization function is constructed with the objective of minimizing the distances from all image points to the corresponding epipolar lines of the camera. The optimization function is then solved to obtain the optimized values ​​of the relative pose relationship between the at least two trackers. The historical relative pose relationship of the at least two trackers is adjusted based on the optimized value of the relative pose relationship of the at least two trackers.

8. The data processing method according to any one of claims 1 to 4, characterized in that, The method further includes: Acquire frame information or time information where the relative pose relationship of at least two trackers changes abruptly during the scanning process.

9. The data processing method according to any one of claims 1 to 4, characterized in that, The step of calculating the relative pose relationship of the at least two trackers using the tracking data, and adjusting the historical relative pose relationship of the at least two trackers based on the calculation results, includes: The entire scanning process is divided into at least one time window, and the relative pose relationship of the at least two trackers within the same time window remains unchanged. In each time window, the relative pose relationship of the at least two trackers is calculated using the tracking data, and the historical relative pose relationship of the at least two trackers is adjusted based on the calculation results.

10. The data processing method according to claim 9, characterized in that, The method further includes: Based on the relative pose relationship of the at least two trackers optimized within each of the scanning time windows, the local 3D scanning data generated by the corresponding scanners are unified to the global coordinate system.

11. The data processing method according to any one of claims 1 to 4, characterized in that, The method further includes: After unifying the local 3D scan data generated by the scanner into a global coordinate system, it is sent to the associated display terminal for display.

12. The data processing method according to any one of claims 1 to 4, characterized in that, The method further includes: During the scanning process, when the scanner is successfully tracked by only one of the at least two trackers and there is a historical relative pose relationship between the at least two trackers, the tracking pose of the currently successfully tracked tracker on the feature points of the scanner, the relative pose relationship between the two cameras of the currently untracked tracker, and the historical relative pose relationship between the at least two trackers are combined to complete the tracking pose of the currently untracked tracker, so as to obtain the actual tracking pose of the currently untracked tracker. Based on the tracking pose of the feature points of the scanner by the currently successfully tracked tracker and the actual tracking pose of the currently failed tracker, the real-time relative pose relationship of the at least two trackers is calculated, and the historical relative pose relationship of the at least two trackers is updated based on the real-time relative pose relationship of the at least two trackers.

13. The data processing method according to claim 12, characterized in that, The step of calculating the real-time relative pose relationship of the at least two trackers based on the tracking pose of the feature points of the scanner by the currently successfully tracked tracker and the actual tracking pose of the currently untracked tracker, and updating the historical relative pose relationship of the at least two trackers based on the real-time relative pose relationship of the at least two trackers, includes: Based on the tracking pose of the scanner's feature points by the currently successfully tracked tracker, the relative pose relationship of the camera of the currently untracked tracker, and combined with the historical relative pose relationship of the at least two trackers, the predicted tracking pose of the scanner's feature points in the coordinate system of at least one camera of the currently untracked tracker is obtained. Based on the predicted tracking pose, the feature points of the scanner are projected onto the image plane of at least one camera of the tracker that is currently failing to track, generating reprojected image points. Based on the correspondence between the reprojected image points and the image points, the correspondence between the image points and the feature points of the scanner is obtained; Based on the correspondence between the image points and the feature points of the scanner, and according to the preset optimization algorithm, the predicted tracking pose is optimized with the minimum distance between the reprojected image points as the optimization objective, and the optimization result is used as the actual tracking pose in at least one camera coordinate system of the tracker that is currently failing to track. The actual tracking pose of the tracker that has failed to track is obtained based on the actual tracking pose in at least one camera coordinate system of the tracker that has failed to track and the relative pose relationship between the two cameras of the tracker that has failed to track.

14. A data processing apparatus, characterized in that, For a tracking scanning system, the tracking scanning system includes a scanner and at least two trackers, and the device includes a data acquisition module, a tracker relative pose optimization module, and a scanning data unification module; The data acquisition module is used to acquire the at least two trackers, track the pose of the scanners during the scanning process, and generate tracking data; The tracker relative pose optimization module is used to calculate the relative pose relationship of the at least two trackers using the tracking data, and to adjust the historical relative pose relationship of the at least two trackers based on the calculation results. The scan data unification module is used to unify the scan data acquired by the scanner to a global coordinate system using the adjusted relative pose relationship of the at least two trackers.

15. A tracking and scanning system, characterized in that, The tracking and scanning system includes a scanner and at least two trackers, as well as an optimization device; The optimization device stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 13.

16. A tracking scanning method, characterized in that, The method of performing three-dimensional scanning of a scanning object using the tracking scanning system of claim 15 includes: Start the tracking and scanning system; The at least two trackers are used to track the pose of the scanner and execute the computer program stored in the optimization device.

Images