Multi-TOF camera point cloud fusion processing method based on task flow parallelism

Abstract

The invention discloses a multi-TOF camera point cloud fusion processing method based on task flow parallelism, and belongs to the technical field of computer vision. According to the method, a task flow parallel architecture is adopted, and data processing of a multi-TOF camera is divided into four parallel tasks of data acquisition, external parameter transformation and preprocessing, point cloud registration and result storage. Ordered execution of all tasks is guaranteed through task flow design and time sequence starting, and the system resource utilization rate and the processing efficiency are remarkably improved. According to the invention, real-time acquisition and processing of point cloud data are realized, synchronism of multi-camera data is ensured, and expandability of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a multi-TOF camera point cloud fusion processing method based on task flow parallelism. BACKGROUND

[0002] A TOF (Time of Flight) camera is an imaging device capable of acquiring depth information and has wide application in three-dimensional reconstruction, robot vision and other fields. Using multiple TOF cameras can expand the observation range or improve the system acquisition frame rate, but there are the following technical problems: the calculation amount of the acquisition, processing and fusion process of multi-camera data is large, and it is difficult to ensure real-time performance; the traditional serial processing method is inefficient and cannot fully utilize system resources; the system resource utilization rate is not high, and the processing delay is large.

[0003] In the prior art, the data of multiple cameras is usually processed in a serial manner, that is, the data acquisition, preprocessing, registration and other steps of each camera are completed one by one. This processing method results in low system resource utilization, large processing delay and difficulty in meeting real-time application requirements. In addition, due to the lack of effective parallel processing mechanism, the overall processing efficiency of the system is severely limited. SUMMARY

[0004] To solve the above problems, the present application provides a multi-TOF camera point cloud fusion processing method based on task flow parallelism, which improves the system resource utilization and processing efficiency through a reasonable parallel processing mechanism.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] The present application provides a multi-TOF camera high frame rate point cloud fusion processing method based on task flow parallelism, characterized by the following steps:

[0007] The multiple TOF cameras are sequentially numbered and a thread is created for each TOF camera, and the number of threads is n;

[0008] The whole process is divided into four sub-tasks, including data acquisition task, external parameter transformation and preprocessing task, point cloud registration task and result storage task;

[0009] At the beginning of the w time period, the following tasks are executed in parallel:

[0010] The kth thread is started for the TOF camera numbered k to execute the data acquisition task and generate a data packet, where k=(w mod n)+1, w>=0, and mod represents the modulo operation; the data packet contains the camera number, acquisition timestamp and point cloud data; w mod n represents the remainder of w divided by n;

[0011] starting the mth thread for the TOF camera numbered m, performing extrinsic transformation and preprocessing tasks on the point cloud data in the data packet generated by the previous task of the thread based on an extrinsic calibration matrix, and updating the data packet, wherein m = ((w-1)mod n)+1, w≥1;

[0012] starting the pth thread for the TOF camera numbered p, performing point cloud registration tasks on the data packet that has undergone extrinsic transformation and preprocessing by the previous task of the thread and outputting a transformation matrix, and updating the data packet according to the transformation matrix, wherein p = ((w-2)mod n)+1, w≥2; here, the transformation matrix is used to represent the registration result;

[0013] starting the qth thread for the TOF camera numbered q, performing storage tasks on the data packet updated by the previous task of the thread, wherein q = ((w-3)mod n)+1, w≥3.

[0014] Further, the data acquisition task comprises:

[0015] recording the number of the TOF camera;

[0016] recording the system time of point cloud acquisition as a timestamp;

[0017] acquiring point cloud data;

[0018] packing the camera number, the timestamp and the point cloud data to generate a data packet.

[0019] Further, the extrinsic transformation and preprocessing task comprises three steps of extrinsic transformation, point cloud downsampling and outlier removal, and updating the data packet.

[0020] Further, the extrinsic transformation step comprises:

[0021] acquiring the corresponding extrinsic calibration matrix from a pre-established extrinsic matrix mapping table according to the number of the TOF camera in the data packet;

[0022] performing coordinate transformation on the point cloud data by using the acquired extrinsic calibration matrix;

[0023] maintaining the association between the point cloud data and the camera number and the timestamp, and updating the data packet.

[0024] Further, the result storage task comprises:

[0025] creating a storage path according to the camera number and the timestamp in the data packet;

[0026] saving the corresponding point cloud data as a file in a specified format;

[0027] outputting the transformation matrix sorted according to the camera number and the timestamp.

[0028] Further, the extrinsic matrix mapping table adopts a hash table data structure.

[0029] Further, the time interval of the start time of two adjacent time periods is dynamically adjusted according to the system processing capability.

[0030] Further, the time interval of the start time of two adjacent time periods is at least the longest time of executing four sub-tasks.

[0031] Compared with the prior art, the present application has the following advantages:

[0032] The present application adopts a task flow parallel architecture, and divides the data processing of the multi-TOF camera into four parallel tasks of data acquisition, extrinsic transformation and preprocessing, point cloud registration, and result storage. Through the task flow design and parallel processing mechanism, the data processing efficiency of the system is improved, the real-time acquisition and processing of the point cloud data are realized, the utilization rate of the system resources is improved, and the synchronism of the multi-camera data is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The task flow start timing diagram shown in the present embodiment is shown. DETAILED DESCRIPTION

[0034] The present application will be further described in detail below in combination with the drawings and embodiments.

[0035] Referring to Figure 1 The present embodiment provides a technical solution: a multi-TOF camera point cloud fusion processing method based on task flow parallelism, taking four TOF cameras and four threads as an example, defining:

[0036] The first thread acquires task G1;

[0037] The first thread performs extrinsic transformation and preprocessing task T1;

[0038] The first thread performs point cloud registration task R1;

[0039] The first thread saves result S1;

[0040] The second thread acquires task G2;

[0041] The second thread performs extrinsic transformation and preprocessing task T2;

[0042] The second thread performs point cloud registration task R2;

[0043] The second thread saves result S2;

[0044] The third thread acquires task G3;

[0045] The third thread performs extrinsic transformation and preprocessing task T3;

[0046] the third thread point cloud registration task R3;

[0047] the third thread result saving task S3;

[0048] the fourth thread acquisition task G4;

[0049] the fourth thread extrinsic transformation and preprocessing task T4;

[0050] the fourth thread point cloud registration task R4;

[0051] the fourth thread result saving task S4.

[0052] After the program is started, four threads are allocated for the four cameras, and then at the start time of the wth time period, the following tasks are executed:

[0053] when w = 0, k = (w mod 4) + 1 = 1; the first thread is started to execute the data acquisition task for the TOF camera with the serial number 1, that is, the first thread acquisition task G1 is started.

[0054] when w = 1, k = (w mod 4) + 1 = 2, m = ((w-1) mod 4) + 1 = 1; the second thread is started to execute the data acquisition task for the TOF camera with the serial number k = 2, that is, the second thread acquisition task G2 is started; at the same time, the first thread is started to acquire the extrinsic calibration matrix of the TOF camera with the serial number m = 1, and the extrinsic transformation and preprocessing task is executed for the point cloud data, that is, the first thread extrinsic transformation and preprocessing task T1 is started.

[0055] when w = 2, k = (w mod 4) + 1 = 3, m = ((w-1) mod 4) + 1 = 2, p = ((w-2) mod 4) + 1 = 1; the third thread is started to execute the data acquisition task for the TOF camera with the serial number k = 3, that is, the third thread acquisition task G3 is started; at the same time, the second thread is started to acquire the extrinsic calibration matrix of the TOF camera with the serial number m = 2, and the extrinsic transformation and preprocessing task is executed for the point cloud data, that is, the second thread extrinsic transformation and preprocessing task T2 is started; at the same time, the first thread is started to execute the point cloud registration task for the TOF camera with the serial number p = 1, that is, the first thread point cloud registration task R1 is started.

[0056] w=3, k=(w mod 4)+1=4, m=((w-1)mod 4)+1=3, p=((w-2)mod 4)+1=2, q=((w-3)mod 4)+1=1; the TOF camera with the serial number k=4 starts the fourth thread to execute the data acquisition task, that is, the fourth thread acquires the task G4; at the same time, the third thread is started to acquire the extrinsic calibration matrix of the TOF camera with the serial number m=3, and the point cloud data is executed to perform the extrinsic transformation and preprocessing task, that is, the third thread extrinsic transformation and preprocessing task T3 is started; at the same time, the second thread is started to execute the point cloud registration task of the TOF camera with the serial number p=2, that is, the second thread point cloud registration task R2 is started; at the same time, the first thread is started to execute the result saving task of the TOF camera with the serial number q=1, that is, the first thread result saving task S1 is started.

[0057] w=4, k=(w mod 4)+1=1, m=((w-1)mod 4)+1=4, p=((w-2)mod 4)+1=3, q=((w-3)mod 4)+1=2; the TOF camera with the serial number k=1 starts the first thread to execute the data acquisition task, that is, the first thread acquires the task G1; at the same time, the fourth thread is started to acquire the extrinsic calibration matrix of the TOF camera with the serial number m=4, and the point cloud data is executed to perform the extrinsic transformation and preprocessing task, that is, the fourth thread extrinsic transformation and preprocessing task T4 is started; at the same time, the third thread is started to execute the point cloud registration task of the TOF camera with the serial number p=3, that is, the third thread point cloud registration task R3 is started; at the same time, the second thread is started to execute the result saving task of the TOF camera with the serial number q=2, that is, the second thread result saving task S2 is started.

[0058] The task flow start timing when w=5 and w=6 is also shown in the figure, which will not be described here.

[0059] After receiving the instruction to stop running the program, the program is exited after the four threads are completed. Here, the instruction to stop running the program can be automatically generated after the multi-TOF camera point cloud fusion is completed, or it can be manually stopped.

[0060] Referring to Table 1, the experimental results of the multi-TOF camera point cloud fusion processing method based on task flow parallelism shown in the embodiment are obtained, a c++ program is written to realize the method, and a program of pipeline serial fusion processing is written as a comparison. The program simulates four cameras to collect 100 frames of point cloud respectively, a total of 400 frames of point cloud, records the total processing time and calculates the processing frame rate. From the results, it can be seen that the processing frame rate of the present application is improved by 2.57 times compared with the serial processing.

[0061] Table 1

[0062] Method Total processing frames Total processing time Processing frame rate The method 400 197 ms 2030.46 fps Comparative method 400 507 ms 788.95 fps

[0063] In summary, the application divides the processing flow into four processing tasks, and uses a task flow parallel architecture to start multiple thread processing different tasks in the same time slice to achieve parallel processing of point cloud data. The general method uses a serial processing method when processing multi-camera fusion, and the present method uses a multi-thread parallel processing method and does not involve parallel processing of the algorithm in each task, which is beneficial to improve the overall processing frame rate of the multi-camera system, while reducing the parallel requirements of the point cloud algorithm in the task, improving the application range, and having practical application value.

[0064] The specific embodiments of the application are described above with reference to the accompanying drawings, but these descriptions cannot be understood as limiting the scope of the application, and the protection scope of the application is defined by the appended claims, and any modification based on the claims of the application is within the protection scope of the application.

Claims

1. A method for high frame rate point cloud fusion processing of multi-TOF cameras based on task flow parallelism, characterized in that, The method comprises the following steps: sequentially numbering the multiple TOF cameras and creating a thread for each TOF camera, with the number of threads being n; splitting the whole process into four sub-tasks, including a data acquisition task, an extrinsic transformation and preprocessing task, a point cloud registration task, and a result storage task; at the beginning of the wth time period, performing the following tasks in parallel: starting the kth thread for the TOF camera numbered k to perform the data acquisition task and generate a data packet, where k=(w mod n)+1, w≥0, and mod represents the modulo operation; the data packet contains the camera number, the acquisition timestamp, and the point cloud data; starting the mth thread for the TOF camera numbered m, performing the extrinsic transformation and preprocessing task on the point cloud data in the data packet generated by the previous task of the thread based on the extrinsic calibration matrix, and updating the data packet, where m=((w-1) mod n)+1, w≥1; starting the pth thread for the TOF camera numbered p, performing the point cloud registration task on the data packet that has undergone the extrinsic transformation and preprocessing in the previous task of the thread, and outputting a transformation matrix, and updating the data packet according to the transformation matrix, where p=((w-2) mod n)+1, w≥2; starting the qth thread for the TOF camera numbered q, performing the storage task on the data packet updated in the previous task of the thread, where q=((w-3) mod n)+1, w≥3.

2. The multi-TOF camera high frame rate point cloud fusion processing method based on task flow parallelism according to claim 1, characterized in that, The data acquisition task comprises: recording the number of the TOF camera; recording the system time of point cloud acquisition as a timestamp; acquiring point cloud data; packing the camera number, the timestamp, and the point cloud data to generate a data packet.

3. The method of claim 1, wherein the method further comprises: The extrinsic transformation and preprocessing task comprises three steps of extrinsic transformation, point cloud downsampling, and outlier removal, and updates the data packet.

4. The method of claim 3, wherein, The extrinsic transformation step comprises: obtaining the corresponding extrinsic calibration matrix from a pre-established extrinsic matrix mapping table according to the number of the TOF camera in the data packet; performing coordinate transformation on the point cloud data using the obtained extrinsic calibration matrix; maintaining the association between the point cloud data and the camera number and the timestamp, and updating the data packet.

5. The method of claim 1, wherein the method further comprises: The result storage task comprises: creating a storage path according to the camera number and the timestamp in the data packet; saving the corresponding point cloud data as a file in a specified format; outputting the transformation matrix sorted by the camera number and the timestamp.

6. The method of claim 4, wherein the method further comprises: The extrinsic matrix mapping table adopts a hash table data structure.

7. The method of claim 1, wherein the method further comprises: The time interval between the start times of adjacent two time periods is dynamically adjusted according to the system processing capacity.

8. The multi-TOF camera high frame rate point cloud fusion processing method based on task flow parallelism according to claim 7, characterized in that, The time interval between the start times of adjacent two time periods is at least the longest time for separately performing the four sub-tasks.

Images