Line laser 3D camera data processing method based on ARM+GPU architecture

Abstract

The present invention provides a line laser 3D camera data processing method based on ARM+GPU architecture, which belongs to the field of 3D vision technology, including: setting a picture ring buffer and a point cloud ring buffer based on SVM technology; the line laser 3D camera collects picture data, and the CPU immediately writes the acquired picture data into the picture ring buffer; the GPU reads the image data of the SVM address currently associated in the picture ring buffer, processes the image data to obtain the light bar pixel point set where the center of the image light bar is located, and writes the light bar pixel point set into the point cloud ring buffer; the CPU cyclically reads the light bar pixel point set corresponding to each frame of the picture from the point cloud ring buffer, and converts it into camera coordinates, and the camera coordinates are cached to a sequence to be sent; when the number of camera coordinates in the sending sequence reaches a preset threshold, all data in the sending sequence are packaged and sent to the client. The present invention can effectively improve the overall processing speed, enhance timeliness and stability, and reduce costs.

Description

Technical Field

[0001] The present invention belongs to the field of 3D vision technology, and in particular relates to a line laser 3D camera data processing method based on ARM+GPU architecture. Background Art

[0002] With the rapid development of 3D vision technology, line laser 3D cameras have shown great application potential in industrial inspection, robot welding and other fields. This type of camera can generate a depth image containing the shape information of the object surface by projecting a laser line onto the surface of the object and capturing its reflected light, which is then converted into useful 3D information. However, the amount of data generated by line laser 3D cameras is usually large, and this data needs to be processed in a complex manner to obtain useful 3D information of the object.

[0003] Traditional data processing methods often rely on high-performance computers or servers to complete these complex computing tasks. Although these devices perform well in processing power, they are large, power-hungry, and expensive, which limits their application in resource-constrained embedded environments. Therefore, how to improve and enhance computing efficiency, power consumption control, and system integration on embedded platforms to achieve efficient processing of line laser 3D camera data will help promote the application and development of 3D vision technology in more fields. Summary of the invention

[0004] The purpose of the present invention is to propose a line laser 3D camera data processing method based on ARM+GPU architecture, which can effectively improve the overall processing speed, enhance timeliness and stability, and reduce costs.

[0005] The present invention is achieved through the following technical solutions:

[0006] The data processing method of the line laser 3D camera based on the ARM+GPU architecture includes the following steps:

[0007] Step S1, setting a picture ring buffer and a point cloud ring buffer based on the SVM technology, the picture ring buffer is used to store the collected picture data, and the point cloud ring buffer is used to store the light strip pixel point set where the center line of the image light strip is located;

[0008] Step S2, setting the camera callback function, the line laser 3D camera collects image data, the ARM CPU instantly obtains each frame of image data called back by the camera, and writes the obtained image data into the image ring buffer;

[0009] Step S3, the GPU reads the image data of the SVM address currently associated in the picture ring buffer, processes the image data to obtain a light strip pixel point set where the center of the image light strip is located, and writes the light strip pixel point set into the point cloud ring buffer;

[0010] Step S4, the CPU cyclically reads the light bar pixel point set corresponding to each frame of the picture from the point cloud ring buffer, and converts the light bar pixel points into camera coordinates, and caches the camera coordinates into a sequence to be sent;

[0011] Step S5: When the number of camera coordinates in the sending sequence reaches a preset threshold, all data in the sending sequence are packaged and sent to the client.

[0012] Furthermore, in the step S1, a space of a set size is pre-allocated in the SVM as an image ring buffer and a point cloud ring buffer. Both the image ring buffer and the point cloud ring buffer follow a first-in-first-out strategy. Both the image ring buffer and the point cloud ring buffer include a write index and a read index. When writing, data is written to a corresponding position according to a current write index value. After writing is successful, the write index value is updated. When reading, data is read from a corresponding position according to a current read index value, and then the read index value is updated.

[0013] Furthermore, in step S2, when collecting image data, the laser transmitter projects a laser beam onto the surface of the target object, and the line laser 3D camera captures the laser stripes reflected by the surface of the target object according to a preset trigger frame rate to generate image data, and the image data is a two-dimensional grayscale image.

[0014] Furthermore, the CPU determines the number of pending images in the image ring buffer through a loop. When the number is greater than 0, the SVM function provided by OpenCL is used to associate the memory area corresponding to the current read index value with the SVM address of the GPU, so that the GPU reads the pending image data.

[0015] Furthermore, in step S3, the CPU determines the global work item size and the local work item size according to the resolution of the image data and the computing power of the GPU to determine the number of column pixels processed by each work group of the GPU and the number of parallel work groups. After the GPU reads the image data, it divides the image data according to the global work item size and the local work item size and allocates it to the work group, wherein the size of each work group is equal to the global work item in the row dimension.

[0016] Further, in step S3, each work group calculates the image area it is responsible for according to its work group ID and the number of column pixels processed by each work group, traverses each column in the image area, and calculates the image area according to the public Calculate the col The grayscale centroid of the column is used as the pixel point of the light bar in the column ( u , v ),in, i Indicates the firsti OK, rows Indicates the total number of rows in the calculated column. g i Indicates the calculated column i Grayscale value of the row pixels.

[0017] Furthermore, in step S3, a grayscale threshold is set. Among the column pixels processed by the working group, if the pixels of each row in a column are smaller than the grayscale threshold, the column is discarded and does not participate in the calculation of the grayscale centroid.

[0018] Furthermore, in step S5, when the number of camera coordinates in the sending sequence reaches a preset threshold, the data in the sending sequence is packaged into a data packet and sent to the client through the TCP network. After the data packet is sent, the sending sequence is cleared.

[0019] The present invention has the following beneficial effects:

[0020] The present invention first sets up a picture ring buffer and a point cloud ring buffer based on SVM (shared virtual memory) technology. The picture data collected by the line laser 3D camera is directly stored in the picture ring buffer. The GPU reads the image data of the SVM address currently associated in the picture ring buffer, processes the image data to obtain a light bar pixel point set where the center of the image light bar is located, and writes the light bar pixel point set into the point cloud ring buffer. The CPU cyclically reads the light bar pixel point set corresponding to each frame of the picture from the point cloud ring buffer, and converts the light bar pixel points into camera coordinates. The camera coordinates are cached in a sequence to be sent. When the number of camera coordinates in the sending sequence reaches a preset threshold, all the data in the sending sequence are packaged and sent to the client. By deeply integrating the excellent performance of the ARM processor in task scheduling and data flow control with the powerful ability of the GPU in parallel computing and large-scale data processing, the overall processing speed of the line laser 3D camera data, the timeliness and stability of the processing data are effectively improved. By setting the ring buffer, the utilization of memory and computing resources is further optimized, the memory access delay and data transmission overhead are reduced. At the same time, on the basis of maintaining high performance, a hardware configuration with relatively low cost can be selected, thereby reducing the overall hardware cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention is further described in detail below with reference to the accompanying drawings.

[0022] Figure 1 It is a flow chart of the present invention.

[0023] Figure 2 A schematic diagram of the structure of setting up a picture ring buffer and a point cloud ring buffer in the present invention.

[0024] Figure 3It is a schematic diagram of the division of global work items and local work items of the present invention.

[0025] Figure 4 This is a diagram showing the effect of extracting the center of the light strip of the present invention. DETAILED DESCRIPTION

[0026] like Figure 1 As shown, the line laser 3D camera data processing method based on ARM+GPU architecture includes the following steps:

[0027] Step S1, setting a picture ring buffer and a point cloud ring buffer based on the SVM technology, the picture ring buffer is used to store the collected picture data, and the point cloud ring buffer is used to store the light strip pixel point set where the center line of the image light strip is located;

[0028] Specifically, Figure 2 As shown, a space of a set size is pre-allocated in the SVM as the image ring buffer and the point cloud ring buffer, and both the image ring buffer and the point cloud ring buffer follow a first-in-first-out strategy to ensure the orderly flow of data. In this embodiment, the image ring buffer and the point cloud ring buffer are divided into 6 even blocks.

[0029] The image ring buffer and point cloud ring buffer both include pointer array, image frame array, write index and read index. The pointer array is used to indicate the starting point of each data storage area to facilitate the reading of data. The image frame array is used to indicate the image frame corresponding to the stored data to facilitate the verification of frame drop or error. When writing, the data is written to the corresponding position according to the current write index value. After the write is successful, the write index value is updated and the number of data to be processed is increased by 1. When reading, the data is read from the corresponding position according to the current read index value, and then the read index value is updated and the number of data to be processed is reduced by 1.

[0030] The dual ring buffer design not only optimizes memory usage, but also significantly reduces memory access latency and data transmission overhead, providing a strong guarantee for the high-performance operation of the system. When the image ring buffer is full (that is, the number of images to be processed in the buffer is equal to the buffer size), the system will automatically stop inserting image data and return a frame loss error code prompt, thereby effectively avoiding the risk of data overflow and data confusion. And by separating image data and point cloud data, it can further reduce memory access conflicts and improve the parallelism and efficiency of data processing.

[0031] Step S2, setting the camera callback function, the line laser 3D camera collects image data, the ARM CPU instantly obtains each frame of image data called back by the camera, and writes the obtained image data into the image ring buffer;

[0032] Specifically, when collecting image data, the laser transmitter projects a laser beam onto the surface of the target object, and the line laser 3D camera captures the laser stripes reflected by the surface of the target object according to a preset trigger frame rate to generate image data, and the image data is a two-dimensional grayscale image.

[0033] In this embodiment, the two-dimensional grayscale image is not preprocessed, but is directly written into the image ring buffer in sequence, thereby reducing unnecessary data processing links and ensuring fast storage and transmission of image data.

[0034] The CPU determines the number of images to be processed in the image ring buffer through a loop. When the number is greater than 0, the SVM function provided by OpenCL is used to associate the memory area corresponding to the current read index value with the SVM address of the GPU, so that the GPU can access the corresponding memory area and then read the image data to be processed. In this way, there is no need to copy data between the ARM and the GPU, which significantly improves data processing efficiency.

[0035] Step S3, the GPU reads the image data of the SVM address currently associated in the picture ring buffer, processes the image data to obtain a light strip pixel point set where the center of the image light strip is located, and writes the light strip pixel point set into the point cloud ring buffer;

[0036] Specifically, the CPU determines the global work item size and the local work item size according to the resolution of the image data and the computing power of the GPU to determine the number of column pixels processed by each work group of the GPU and the number of parallel work groups. When the resolution of the image data remains unchanged, the above data only needs to be determined once. After the GPU reads the image data, it divides the image data according to the global work item size and the local work item size and allocates it to the work group, where the size of each work group is equal to the global work item in the row dimension.

[0037] In this embodiment, the picture data with a resolution of 1280*1024 is used for illustration. The number of work items supported by the GPU in the two-dimensional index space used in this embodiment is 384*384. Figure 3As shown, the global work item size is set to {320,32}, where "320" corresponds to the number of work items divided by the width (X direction) of the image data, and "32" corresponds to the number of work items divided by the height (Y direction) of the image data. The local work item is the basic unit that makes up the work group. The local work item size is set to {1,32}, which represents the number of work items in each work group. This means that each work group processes 4 columns of pixels, and a column of pixels has a total of 1024 pixels. Each work group processes 4*1024 pixels, and a total of 320 work groups process in parallel. Therefore, the GPU can be called once to complete the processing of a frame of image data with a size of 1280*1024 and extract the center of the light bar.

[0038] Each work group calculates the image area it is responsible for based on its work group ID and the number of column pixels processed by each work group, traverses each column in the image area, and calculates the image area according to the formula Calculate the col The grayscale centroid of the column is used as the pixel point of the light bar in the column ( u , v ),in, i Indicates the first i OK, rows Indicates the total number of rows in the calculated column. g i Indicates the calculated column i When all working groups have completed the grayscale centroid calculation, the center of the light stripe of a frame of image is extracted. The extraction effect is as follows: Figure 4 shown.

[0039] To further improve efficiency, a grayscale threshold is set. Among the column pixels processed by the working group, if the pixels of each row in a column are all smaller than the grayscale threshold, it means that the column has no optical center, that is, it is considered that the laser has not illuminated the column, and the column is discarded and does not participate in the calculation of the grayscale center of gravity.

[0040] Step S4, the CPU cyclically reads the light bar pixel point set corresponding to each frame of the picture from the point cloud ring buffer, and converts the light bar pixel points into camera coordinates, and caches the camera coordinates into a sequence to be sent;

[0041] Specifically, the points stored in the point cloud ring buffer are the pixel coordinates corresponding to the center of the light strip, so the pixel coordinates need to be converted into camera coordinates with depth information. This conversion process relies on a pre-calibrated transformation matrix that can map points on the image plane into three-dimensional space.

[0042] In this embodiment, the CPU reads the light bar pixel points from the point cloud ring buffer ( u , v ) and then the light bar pixel (u , v ) is converted to camera coordinates ( x , y , z ), the conversion process converts the pixel points ( u , v ) is converted to camera coordinates ( x , z ), when scanning an object for 3D point cloud reconstruction, y Calculated based on actual location information provided by external mobile devices.

[0043] Step S5: When the number of camera coordinates in the sending sequence reaches a preset threshold, all data in the sending sequence are packaged and sent to the client;

[0044] When processing light stripe point data extracted from an image, especially when the camera frame rate is high, directly sending the point set corresponding to each frame to the client frame by frame may result in a significant reduction in network transmission efficiency. The reason is that although the amount of light stripe point data in a single frame may not be large, the data accumulates very quickly at a high frame rate, and frequent transmission of small data packets will occupy a large amount of network resources, increase latency and reduce throughput.

[0045] Therefore, in this embodiment, a point cloud waiting to be sent linked list buffer is established as a sending list. When the number of camera coordinates in the sending sequence reaches a preset threshold, the data in the sending sequence is packaged into a data packet and sent to the client through the TCP network. After the data packet is sent, the sending sequence is cleared. In this way, network congestion and resource waste caused by frequent transmission of small data packets can be significantly reduced, and the performance and response speed of the overall system can be improved.

[0046] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A line laser 3D camera data processing method based on ARM+GPU architecture, characterized in that: The steps include: Step S1, setting a picture ring buffer and a point cloud ring buffer based on the SVM technology, the picture ring buffer is used to store the collected picture data, and the point cloud ring buffer is used to store the light strip pixel point set where the center line of the image light strip is located; Step S2, setting the camera callback function, the line laser 3D camera collects image data, the ARM CPU instantly obtains each frame of image data called back by the camera, and writes the obtained image data into the image ring buffer; Step S3, the GPU reads the image data of the SVM address currently associated in the picture ring buffer, processes the image data to obtain a light strip pixel point set where the center of the image light strip is located, and writes the light strip pixel point set into the point cloud ring buffer; Step S4, the CPU cyclically reads the light bar pixel point set corresponding to each frame of the picture from the point cloud ring buffer, and converts the light bar pixel points into camera coordinates, and caches the camera coordinates into a sequence to be sent; Step S5: When the number of camera coordinates in the sending sequence reaches a preset threshold, all data in the sending sequence are packaged and sent to the client.

2. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 1, characterized in that: In the step S1, a space of a set size is pre-allocated in the SVM as an image ring buffer and a point cloud ring buffer. Both the image ring buffer and the point cloud ring buffer follow a first-in-first-out strategy. Both the image ring buffer and the point cloud ring buffer include a write index and a read index. When writing, data is written to a corresponding position according to a current write index value. After the write is successful, the write index value is updated. When reading, data is read from a corresponding position according to a current read index value, and then the read index value is updated.

3. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 2, characterized in that: In step S2, when collecting image data, the laser transmitter projects a laser beam onto the surface of the target object, and the line laser 3D camera captures the laser stripes reflected by the surface of the target object according to a preset trigger frame rate to generate image data, and the image data is a two-dimensional grayscale image.

4. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 1, 2 or 3, characterized in that: The CPU determines the number of pending images in the image ring buffer through a loop. When the number is greater than 0, the SVM function provided by OpenCL is used to associate the memory area corresponding to the current read index value with the SVM address of the GPU, so that the GPU reads the pending image data.

5. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 4, characterized in that: In step S3, the CPU determines the global work item size and the local work item size according to the resolution of the image data and the computing power of the GPU to determine the number of column pixels processed by each work group of the GPU and the number of parallel work groups. After reading the image data, the GPU divides the image data according to the global work item size and the local work item size and distributes it to the work groups. Each work group calculates the image area it is responsible for according to its work group ID and the number of column pixels processed by each work group, traverses each column in the image area, and calculates the image area according to the formula Calculate the col The grayscale centroid of the column is used as the pixel point of the light bar in the column ( u , v ),in, i Indicates the first i OK, rows Indicates the total number of rows in the calculated column. g i Indicates the calculated column i Grayscale value of the row pixels.

6. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 5, characterized in that: In step S3, a grayscale threshold is set. Among the column pixels processed by the working group, if the pixels of each row in a column are smaller than the grayscale threshold, the column is discarded and does not participate in the calculation of the grayscale centroid.

7. The line laser 3D camera data processing method based on ARM+GPU architecture according to claim 1, 2 or 3, characterized in that: In step S5, when the number of camera coordinates in the sending sequence reaches a preset threshold, the data in the sending sequence is packaged into a data packet and sent to the client through the TCP network. After the data packet is sent, the sending sequence is cleared.

Images