A multi-tof camera point cloud fusion processing method based on task flow parallelism

Abstract

The application discloses a kind of multi-TOF camera point cloud fusion processing methods based on task flow parallel, belong to computer vision technical field.The method uses task flow parallel architecture, and the data processing of multi-TOF camera is divided into four parallel tasks of data acquisition, external parameter transformation and preprocessing, point cloud registration and result storage.Ordered execution of each task is guaranteed by task flow design and timing start, and system resource utilization and processing efficiency are significantly improved.The application realizes the real-time acquisition and processing of point cloud data, guarantees the synchronism of multi-camera data, and improves the scalability of system.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a multi-TOF camera point cloud fusion processing method based on task flow parallelism. Background Technology

[0002] A Time-of-Flight (TOF) camera is an imaging device capable of acquiring depth information and has wide applications in fields such as 3D reconstruction and robot vision. Using multiple TOF cameras can expand the observation range or increase the system's frame rate, but the following technical problems exist: the acquisition, processing, and fusion of multi-camera data involve large computational loads, making it difficult to guarantee real-time performance; traditional serial processing methods are inefficient and cannot fully utilize system resources; and system resource utilization is low, resulting in large processing delays.

[0003] In existing technologies, data from multiple cameras is typically processed serially, meaning that data acquisition, preprocessing, and registration for each camera are performed sequentially. This approach results in low system resource utilization, high processing latency, and difficulty in meeting the demands of real-time applications. Furthermore, the lack of an effective parallel processing mechanism severely limits the overall processing efficiency of the system. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to provide a multi-TOF camera point cloud fusion processing method based on task flow parallelism, which improves system resource utilization and processing efficiency through a reasonable parallel processing mechanism.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention provides a high frame rate point cloud fusion processing method based on task flow parallelism using multiple TOF cameras, characterized by the following steps:

[0007] Number the multiple TOF cameras sequentially and create a thread for each TOF camera, and let the number of threads be n;

[0008] The entire process is broken down into four sub-tasks: data acquisition, external parameter transformation and preprocessing, point cloud registration, and result storage.

[0009] At the start of time period w, the following tasks are executed in parallel:

[0010] For the TOF camera numbered k, the k-th thread is started to perform the data acquisition task and generate a data packet, where k = (w mod n) + 1, w ≥ 0, and mod represents the modulo operation; the data packet contains the camera number, acquisition timestamp, and point cloud data; w mod n represents the remainder when w is divided by n;

[0011] Start the m-th thread for the TOF camera numbered m. Based on the extrinsic calibration matrix, perform extrinsic transformation and preprocessing tasks on the point cloud data in the data packet generated by the previous task of the thread, and update the data packet, where m = ((w-1)mod n) + 1, w ≥ 1;

[0012] For the TOF camera numbered p, start the p-th thread, perform point cloud registration on the data packet that has undergone extrinsic parameter transformation and preprocessing in the previous task of the thread, and output the transformation matrix. Update the data packet according to the transformation matrix, where p = ((w-2)modn)+1, w≥2; here, the transformation matrix is ​​used to represent the registration result;

[0013] Start the q-th thread for the TOF camera numbered q, and perform a storage task on the data packet updated by the previous task of the thread, where q = ((w-3)mod n)+1, w≥3.

[0014] Furthermore, the data acquisition task includes:

[0015] Record the serial number of the TOF camera;

[0016] The system time of point cloud acquisition is recorded as a timestamp;

[0017] Acquire point cloud data;

[0018] The camera ID, timestamp, and point cloud data are packaged together to generate a data package.

[0019] Furthermore, the extrinsic parameter transformation and preprocessing task includes three steps: extrinsic parameter transformation, point cloud downsampling, and outlier removal, and updating the data packet.

[0020] Furthermore, the extrinsic parameter transformation step includes:

[0021] Based on the TOF camera number in the data packet, the corresponding extrinsic calibration matrix is ​​obtained from the pre-established extrinsic parameter matrix mapping table;

[0022] The obtained extrinsic calibration matrix is ​​used to perform coordinate transformation on the point cloud data;

[0023] Maintain the association between point cloud data and camera ID and timestamp, and update the data package.

[0024] Furthermore, the result storage task includes:

[0025] Create a storage path based on the camera number and timestamp in the data packet;

[0026] Save the corresponding point cloud data as a file in the specified format;

[0027] Output the transformation matrix sorted by camera number and timestamp.

[0028] Furthermore, the extrinsic parameter matrix mapping table adopts a hash table data structure.

[0029] Furthermore, the time interval between the start times of two adjacent time periods is dynamically adjusted based on the system's processing capacity.

[0030] Furthermore, the time interval between the start times of two adjacent time periods is at least the longest time required to execute four subtasks individually.

[0031] The present invention has the following advantages over the prior art:

[0032] This invention employs a task-flow parallel architecture, dividing the data processing from multiple TOF cameras into four parallel tasks: data acquisition, extrinsic parameter transformation and preprocessing, point cloud registration, and result storage. Through task flow design and parallel processing mechanisms, the system's data processing efficiency is improved, enabling real-time acquisition and processing of point cloud data, increasing system resource utilization, and ensuring the synchronization of data from multiple cameras. Attached Figure Description

[0033] Figure 1 This is a sequence diagram showing the task flow startup in this embodiment. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0035] See Figure 1 This embodiment provides a technical solution: a multi-TOF camera point cloud fusion processing method based on parallel task flow. Taking four TOF cameras and four threads as an example, the following is defined:

[0036] The first thread acquires task G1;

[0037] First thread extrinsic parameter transformation and preprocessing task T1;

[0038] First thread point cloud registration task R1;

[0039] First thread result saving task S1;

[0040] The second thread acquires task G2;

[0041] Second thread extrinsic parameter transformation and preprocessing task T2;

[0042] Second thread point cloud registration task R2;

[0043] The second thread results are saved in task S2;

[0044] The third thread acquires task G3;

[0045] Third thread extrinsic parameter transformation and preprocessing task T3;

[0046] Third thread point cloud registration task R3;

[0047] Third thread result saving task S3;

[0048] The fourth thread retrieves task G4;

[0049] Fourth thread extrinsic parameter transformation and preprocessing task T4;

[0050] Fourth thread point cloud registration task R4;

[0051] The fourth thread's result is saved in task S4.

[0052] After the program starts, four threads are assigned to the four cameras. Then, at the beginning of time period w, the following tasks are executed:

[0053] When w = 0, k = (w mod 4) + 1 = 1; start the first thread to execute the data acquisition task for the TOF camera with sequence number 1, that is, start the first thread acquisition task G1.

[0054] When w = 1, k = (w mod 4) + 1 = 2, m = ((w - 1) mod 4) + 1 = 1; for the TOF camera with sequence number k = 2, start the second thread to perform the data acquisition task, that is, start the second thread acquisition task G2; at the same time, for the TOF camera with sequence number m = 1, start the first thread to acquire the corresponding extrinsic calibration matrix, and perform extrinsic transformation and preprocessing tasks on its point cloud data, that is, start the first thread extrinsic transformation and preprocessing task T1.

[0055] When w = 2, k = (w mod 4) + 1 = 3, m = ((w-1) mod 4) + 1 = 2, p = ((w-2) mod 4) + 1 = 1; for the TOF camera with serial number k = 3, the third thread is started to perform the data acquisition task, that is, the third thread acquisition task G3 is started; at the same time, for the TOF camera with serial number m = 2, the second thread is started to obtain the corresponding extrinsic calibration matrix, and the extrinsic parameter transformation and preprocessing task is performed on its point cloud data, that is, the second thread extrinsic parameter transformation and preprocessing task T2 is started; at the same time, for the TOF camera with serial number p = 1, the first thread is started to perform the point cloud registration task, that is, the first thread point cloud registration task R1 is started.

[0056] When w = 3, k = (w mod 4) + 1 = 4, m = ((w - 1) mod 4) + 1 = 3, p = ((w - 2) mod 4) + 1 = 2, q = ((w - 3) mod 4) + 1 = 1; For the TOF camera with serial number k = 4, the fourth thread is started to perform the data acquisition task, that is, the fourth thread acquisition task G4 is started; At the same time, for the TOF camera with serial number m = 3, the third thread is started to obtain the corresponding extrinsic calibration matrix and perform extrinsic parameter transformation and preprocessing tasks on its point cloud data, that is, the third thread extrinsic parameter transformation and preprocessing task T3 is started; At the same time, for the TOF camera with serial number p = 2, the second thread is started to perform the point cloud registration task, that is, the second thread point cloud registration task R2 is started; At the same time, for the TOF camera with serial number q = 1, the first thread is started to perform the result saving task, that is, the first thread result saving task S1 is started.

[0057] When w = 4, k = (w mod 4) + 1 = 1, m = ((w-1) mod 4) + 1 = 4, p = ((w-2) mod 4) + 1 = 3, q ​​= ((w-3) mod 4) + 1 = 2; For the TOF camera with serial number k = 1, the first thread is started to perform the data acquisition task, that is, the first thread acquisition task G1 is started; At the same time, for the TOF camera with serial number m = 4, the fourth thread is started to obtain the corresponding extrinsic calibration matrix, and the extrinsic parameter transformation and preprocessing task is performed on its point cloud data, that is, the fourth thread extrinsic parameter transformation and preprocessing task T4 is started; At the same time, for the TOF camera with serial number p = 3, the third thread is started to perform the point cloud registration task, that is, the third thread point cloud registration task R3 is started; At the same time, for the TOF camera with serial number q = 2, the second thread is started to perform the result saving task, that is, the second thread result saving task S2 is started.

[0058] The diagram also illustrates the task flow startup sequence when w=5 and w=6, which will not be described further here.

[0059] Upon receiving a command to stop the program, the program will exit after all four threads have completed their execution. This command can be generated automatically after all point clouds from the multiple TOF cameras have been fused, or it can be manually stopped.

[0060] Table 1 shows the experimental results of the multi-TOF camera point cloud fusion processing method based on parallel task flow as illustrated in this embodiment. In this embodiment, a C++ program was written to implement this method, and a pipelined serial fusion processing program was written for comparison. The program simulates four cameras each acquiring 100 frames of point cloud data, for a total of 400 frames. The total processing time was recorded, and the processing frame rate was calculated. The results show that the processing frame rate of this invention is 2.57 times higher than that of serial processing.

[0061] Table 1

[0062] method Total number of frames processed Total processing time Processing frame rate This method 400 197ms 2030.46fps Comparison Methods 400 507ms 788.95fps

[0063] In summary, this invention divides the processing flow into four processing tasks. By adopting a task-flow parallel architecture, multiple threads are launched within the same time slice to process different tasks, achieving parallel processing of point cloud data. While conventional methods use serial processing in multi-camera fusion processing, this method uses multi-threaded parallel processing without involving parallel processing of algorithms within each task. This improves the overall processing frame rate of the multi-camera system, reduces the parallel requirements of point cloud algorithms within tasks, expands its applicability, and has practical application value.

[0064] The specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, these descriptions should not be construed as limiting the scope of the present invention. The scope of protection of the present invention is defined by the appended claims, and any modifications based on the claims of the present invention are within the scope of protection of the present invention.

Claims

1. A method for high frame rate point cloud fusion processing using multiple TOF cameras based on parallel task flow, characterized in that, Includes the following steps: Number the multiple TOF cameras sequentially and create a thread for each TOF camera, and let the number of threads be n; The entire process is broken down into four sub-tasks: data acquisition, external parameter transformation and preprocessing, point cloud registration, and result storage. At the start of time period w, the following tasks are executed in parallel: For the TOF camera numbered k, the k-th thread is started to perform the data acquisition task and generate a data packet, where k = (wmod n) + 1, w ≥ 0, and mod represents the modulo operation; the data packet contains the camera number, acquisition timestamp, and point cloud data; Start the m-th thread for the TOF camera numbered m. Based on the extrinsic calibration matrix, perform extrinsic transformation and preprocessing tasks on the point cloud data in the data packet generated by the previous task of the thread, and update the data packet, where m = ((w-1)mod n) + 1, w ≥ 1; Start the p-th thread for the TOF camera numbered p. Perform point cloud registration on the data packet that has undergone extrinsic parameter transformation and preprocessing in the previous task of the thread and output the transformation matrix. Update the data packet according to the transformation matrix, where p = ((w-2)mod n)+1, w≥2; Start the q-th thread for the TOF camera numbered q, and perform a storage task on the data packet updated by the previous task of the thread, where q = ((w-3)mod n)+1, w≥3.

2. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 1, characterized in that, The data acquisition task includes: Record the serial number of the TOF camera; The system time of point cloud acquisition is recorded as a timestamp; Acquire point cloud data; The camera ID, timestamp, and point cloud data are packaged together to generate a data package.

3. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 1, characterized in that, The extrinsic parameter transformation and preprocessing task includes three steps: extrinsic parameter transformation, point cloud downsampling, and outlier removal, and updating the data packet.

4. The method according to claim 3, characterized in that, The external parameter transformation steps include: Based on the TOF camera number in the data packet, the corresponding extrinsic calibration matrix is ​​obtained from the pre-established extrinsic parameter matrix mapping table; The obtained extrinsic calibration matrix is ​​used to perform coordinate transformation on the point cloud data; Maintain the association between point cloud data and camera ID and timestamp, and update the data package.

5. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 1, characterized in that, The result storage task includes: Create a storage path based on the camera number and timestamp in the data packet; Save the corresponding point cloud data as a file in the specified format; Output the transformation matrix sorted by camera number and timestamp.

6. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 4, characterized in that, The extrinsic parameter matrix mapping table adopts a hash table data structure.

7. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 1, characterized in that, The time interval between the start times of two adjacent time periods is dynamically adjusted based on the system's processing capacity.

8. The method for high frame rate point cloud fusion processing based on task flow parallelism using multiple TOF cameras according to claim 7, characterized in that, The time interval between the start times of two adjacent time periods is at least the longest time required to execute four subtasks individually.

Images