PCD format point cloud data visualization processing software framework building method

Abstract

The invention provides a PCD format point cloud data visualization processing software framework construction method, and provides a point cloud data visualization processing program development mode capable of being transplanted and operated between platforms such as Linux and Windows aiming at the limitation that only simple reading of PCD format files is supported generally in the prior art. Various functions of visual editing, modification, storage and the like of the point cloud data can be comprehensively realized, so that the application scene and the processing depth of the PCD-format point cloud data are greatly expanded.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud data processing and visualization, and specifically relates to a method for building a software framework for visualization processing of point cloud data in PCD format. Background Art

[0002] In the field of 3D point cloud data processing, the PCD (Point Cloud Data) file format is widely used due to its advantages in storing ordered point cloud data sets, fast binary data type reading and writing, and support for different data types. However, the PCD file format has relatively limited functions in visual editing. Currently, most PCD files only support basic creation and reading operations, and lack the ability to perform in-depth editing and operations on existing 3D point cloud data. For example, although users can create PCD files and store the XYZ coordinates, intensity information (Intensity), and color information (RGB) of point sets, existing tools are often unable to cope with more complex editing tasks, such as adding, deleting, modifying, and measuring distances of point clouds, as well as performing advanced point cloud filtering and coordinate transformation.

[0003] In addition, existing PCD file processing tools usually do not support dynamic visualization and real-time editing, which limits the user's interactive experience and the depth of application scenarios in point cloud data processing. Users can only use some complex and expensive professional software and hardware, which is not conducive to the popularization and application of 3D point cloud data. Therefore, it is necessary to develop a new software tool to overcome the limitations of existing technologies and provide richer and deeper point cloud data processing functions. Summary of the invention

[0004] In view of this, in order to solve the technical problems existing in this field, the present invention provides a method for building a software framework for visualizing point cloud data in PCD format, which specifically includes the following steps:

[0005] Step 1: Design the software user interface UI based on the QT framework and QT components, use C++ as the development programming language, use PCL library and VTK library as point cloud data processing tools and point cloud visualization framework, and use Eigen library as the software built-in algorithm mathematical solution tool; first, create a new Qt Widgets Application class project file under the QT framework, and use the pro file as the project configuration file to configure QT, project files, and third-party libraries including PCL, VTK, Eigen and Boost libraries to realize the overall framework Transplantation and operation between Linux and Windows platforms;

[0006] Step 2. According to the multi-threaded optimization design plan, create an instance of the StartThread class to start the software. First, create a QApplication class object in the StartThread class constructor to manage the control flow and main settings of the GUI program; then instantiate the CtrlWindow class to register and manage the software user interaction interface, that is, the UI interface; then create a PCDEditor class object for point cloud processing class function registration and management; then create a PCDEditorThread class object to perform sub-thread packaging of the point cloud processing program; then create a ViewerThread class object to register and manage point cloud visualization class functions, and perform sub-thread packaging at the same time. Finally, call the show() function of the CtrlWindow parent class QWidget to display the software UI, and call the start() function of the QThread class to start the parallel operation of the two sub-threads;

[0007] Step 3: Design the UI window in the QT framework through the ctrlwindow.ui layout file. First, visualize the ctrlwindow.ui window in QTDesign, and build UI components according to actual functional requirements to implement related functions including operation, processing and display of PCD.

[0008] Step 4: Establish a communication mechanism between components and threads based on the signal-slot mechanism, including responding and binding with the UI components in the ctrlwindow.ui layout file through the slot function in the CtrlWindow class of the main thread, ensuring that the interaction between components is achieved through the predefined event triggering strategy;

[0009] Step 5. Design the PCDEditor class that runs under the PCL, VTK, and Eigen library framework and serves as the core of the point cloud data processing sub-thread, so that it can communicate with the CtrlWindow main thread and the ViewerThread sub-thread during runtime to achieve response processing for user interaction operations and instant visualization update of point cloud data;

[0010] Step 6. Design the ViewerThread class as the core of the point cloud visualization and user interaction mechanism. The ViewerThread class implements two-way inter-thread communication by subscribing to the class objects of CtrlWindow and PCDEditor in the constructor, which is used to realize interactive viewing, modification and editing of point cloud data in the visualization window.

[0011] Furthermore, the relevant functions implemented by building UI components in step three specifically include: using Push Button components to load PCD, save PCD, globally reset, reset perspective, screenshot, export, globally downsample, filter out discrete points, pass-through filtering, global coordinate transformation, reset two-point distance measurement, coordinate axis visualization, and process print clearing functions; using Check Box components to design and apply point cloud deletion, point cloud addition, point cloud review, two-point distance measurement, and fixed top-down functions; using Table Widget components to review point clouds and display the results of two-point distance measurement functions; using Horizontal Slider components to change the size of a single point and develop and apply coordinate axis visualization functions; using SpinBox and Double Spin Box components to set hyperparameters for global downsampling, filtering out discrete points, and pass-through filtering; using Combo Box components to select the mode of pass-through filtering and global coordinate transformation; using Text Edit components to display UI window description text and process print output of some functional modules.

[0012] Furthermore, in step 4, the CtrlWindow class simultaneously maintains a series of function flags (Flag) and corresponding read (get) and set (set) functions, through which the flags and other class member variables are queried and updated, thereby realizing inter-thread communication.

[0013] Furthermore, in step 5, the PCDEditor class declares a pcl::PointCloud <pointt>::Ptr type member variable pe_cloud is used to store global point cloud data; implement the sub-thread encapsulation of PCDEditor instance pe in the StartThread class constructor, and at the same time, PCDEditor's Run() function can respond to the start() method of PCDEditorThread's parent class QThread, execute the Run() method of PCDEditorThread class, and then enable PCDEditor's Run() function to run in the sub-thread to achieve parallel operation;

[0014] The point cloud processing task is continuously executed by implementing an infinite loop function in the Run() function. Its point cloud processing function includes three data management functions: LoadPCDFile() is used to load point clouds, SavePCDFile() is used to store point clouds, and SaveAsFile() supports exporting point clouds to multiple formats, including PLY, PCDBinary, OBJ, and VTK; 4 point cloud filtering modes: AllPointsDownsize() performs global downsampling, allowing users to set the voxel edge length; AllPointsOutlierRemoval() is used to filter out discrete points based on the user-defined hyperparameters of the average distance estimation number of points and the standard deviation multiple; AllPointsPassT hrough() is used to perform through filtering based on the filtering direction and range specified by the user; AllPointsRemoveGround() is used to filter out ground points based on the ground height and elevation threshold specified by the user; the global coordinate transformation capability is provided through the TfTrans() function, and user-defined three-axis transformation is supported; based on the infinite loop nature of the Run() function, the child thread periodically calls the get function of a series of related functions in the CtrlWindow main thread, and monitors user needs through the Flag flag. Once a user request is detected, the child thread immediately executes the corresponding function to meet the user's needs, and resets the Flag by calling the set function of CtrlWindow, thereby completing this communication cycle.

[0015] Furthermore, when AllPointsRemoveGround() performs ground point filtering, it specifically calls the getAllPointsRemoveGroundFlag() function of the CtrlWindow class instance cw through inter-thread communication to detect the user request corresponding to this function in real time; when the user request is captured, the user interface class instance ui_ is captured by calling the getUi() function, and the ground height groundZ and elevation threshold ground_remove_threshold parameters set by the user are directly received from ui_; then, the boundary of the point cloud data is determined, and then rasterization is performed at the specified resolution on the XY plane, and the maximum and minimum Z values ​​of each grid cell are initialized; through By traversing the point cloud and updating the Z value of the grid cell, and calculating the Z value change of each cell; if the Z value change of the cell is less than the elevation threshold ground_remove_threshold, and the height of the point is close to groundZ, the cell is marked as ground; then, the ground and non-ground point clouds are separated according to the marking results of the grid to filter out the ground points; finally, the setAllPointsRemoveGroundFlag() function is called to reset and update the ground point filtering function flag to implement the user's request to filter out the ground points; for point cloud data with different densities and terrain features, the resolution and threshold are adjusted to adapt, and the actual ground height of the sensor during data collection is considered when setting groundZ.

[0016] Furthermore, in step six, the ViewerThread class is a subclass of QThread, and directly calls the start() method of its parent class QThread in StartThread to execute the run() method of the ViewerThread class, thereby realizing parallel operation of different threads; the ViewerThread class declares and maintains a member variable viewer__ of the pcl::visualization::PCLVisualizer type, which is used to create a parametrically configured visualization window; in the above run() method, the viewer__ object is first initialized, and the basic visualization window parameters are configured, including the window name, position, size, background color, presented point cloud data, various point cloud properties, view camera parameters, etc.; then the child thread enters an infinite loop function, which loops at a frequency of 10Hz and detects the health of viewer_. When viewer_ stops running, the loop ends automatically to avoid invalid visualization refresh and waste of platform computing power; in the infinite loop, the corresponding point cloud rendering and visualization control, user interaction processing, cross-thread parameter synchronization and view export functions are implemented according to different user requests.

[0017] Furthermore, when implementing the point cloud rendering function, the point cloud rendering parameters are initialized in the run() method. When the user triggers the "single point size" change request, ViewerThread captures the current user request flag and the "point size" parameter set by the user through the Horizontal Slider component through communication with the main thread, executes the changePointSize() function, and resets the point cloud rendering parameters according to the user-set parameters. After completing this function and subsequent other function functions, it enters the next loop, and the VTK visualization window object viewer_ is re-rendered according to the newly set parameters, thereby realizing the dynamic point cloud rendering function; the visualization control is specifically the "show coordinate axis" function, which enables the user to dynamically adjust the coordinate axis size through the Horizontal Slider component in the UI interface, and can also trigger the function through the "show coordinate axis / hide coordinate axis" button, and implements related functions through the showAxis() function. Its basic process is similar to point cloud rendering.

[0018] Furthermore, the user interaction processing function includes four functions: adding point clouds, checking point clouds, two-point distance measurement, and deleting point clouds. When implementing these functions, first register and bind the interactive callback functions in the run() function based on the PCL framework, including keyboard registerKeyboardCallback(), mouse registerMouseCallback(), point selection registerPointPickingCallback(), and area selection registerAreaPickingCallback(); these callback functions also introduce the event triggering mechanism (Event) of the PCL library, which refers to a specified action triggered by a certain behind-the-scenes event, such as a mouse double-click event; the events applied to the user interaction processing function include: keyboard event KeyboardEvent, mouse event MouseEvent, single point selection event PointPickingEvent, area selection event AreaPickingEvent; through different event response mechanisms and related callback functions, the user can interact with the point cloud data in real time in the visual interface;

[0019] For the point cloud deletion function, the "X" and "x" keys are bound to the on / off function of the point cloud deletion function through the KeyboardEvent event in the pclKeyBoardCallBack() callback function; then the user request is monitored through inter-thread communication, and when the request is captured, the UI process information flow outputs prompt information to assist the user to realize the secondary on / off of the function through the "X" and "x" keys; when the user turns on the function for the second time, the user's mouse selection action will trigger the above-mentioned area selection callback function and start executing the pclAreaPickingCallBack() function; the getPointsIndices() function of the AreaPickingEvent event can extract the serial number of the point in the pe_cloud container selected by the frame, and then combine the ExtractIndices filter to delete the selected point set, and finally output the UI process information flow and reset and update the relevant flag bits to realize the user request;

[0020] To add point cloud function, first call PCL's getCameraParameters() function to get the camera parameters under the current viewing angle, and then use computeViewMatrix() and computeProjectionMatrix() functions to calculate the camera's view matrix M respectively. view and the projection matrix M proj ; Among them, M view Represents the transformation from the point cloud coordinate system to the camera coordinate system, M proj It represents the projection transformation from the camera coordinate system to the screen coordinate system; through the product of the following two matrices M composite The complete transformation relationship from the point cloud coordinate system to the screen coordinate system is defined:

[0021] M composite =M view *M proj

[0022] Then, read two reference points from the pe_cloud container across threads, and obtain their coordinates P1(x1, y1, z1) and P2(x2, y2, z2) in the point cloud coordinate system respectively; convert these two points into two-dimensional coordinates to eliminate the effects of rotation and perspective, and obtain P1'(x1, y1, 0) and P2'(x2, y2, 0); convert the two points into the same coordinate system according to M composite The coordinate system is transformed according to the transformation relationship to obtain the pixel coordinates P1" (a1, b1, 0) and P2" (a2, b2, 0) in the screen coordinate system; with the help of the mouse event MouseEvent, the clicked screen position P0" (a0, b0, 0) is captured; according to the principle of similarity, there is a proportional formula:

[0023]

[0024] Thus, we get:

[0025]

[0026] At this point, the 2D coordinates of the mouse click position in the point cloud coordinate system are obtained:

[0027]

[0028] Then, the intensity and elevation value z of the point are customized to form a complete point cloud point, the point is added to the pe_cloud container, and visually rendered in the next rendering cycle, thereby realizing the user's request to add a point cloud.

[0029] Furthermore, the cross-thread parameter synchronization function includes two parts: perspective reset and global reset. The perspective reset is implemented by the doViewReset() function, which captures user requests through inter-thread communication, and then keeps the perspective camera position unchanged, only resetting its observation angle completely in the opposite direction of the elevation to achieve a co-located bird's-eye view effect. When this function needs to be implemented once, a request is sent by clicking the "Perspective Reset" button. If a high-frequency loop is required, a request is sent by checking the "Fixed Bird's Eye" CheckBox component. Global reset is implemented by the doAllReset() function. First, the point cloud is reset across threads to replace the current point cloud with the initially loaded point cloud. Then, the UI component parameters are reset across threads to reset them all to the state when the software is just started. Finally, the perspective reset of this thread is executed to reset the visualization window.

[0030] Furthermore, the view export function is specifically implemented using the saveScreenShot() function. After capturing the user request through inter-thread communication, the function uses the saveScreenshot() function provided by the VTK framework to capture the current visualization window image information, and initiates a pop-up request to the main thread through the getScreenShotPathOut() function of the CtrlWindow class. The CtrlWindow class invokes the QFileDialog pop-up window to assist the user in specifying the image save path and naming the image. Finally, the function flag is reset and updated through the set function to meet user needs.

[0031] The method for building a software framework for visualizing point cloud data in PCD format provided by the present invention addresses the limitation that the prior art generally only supports simple reading of PCD format files, and provides a method for developing a point cloud data visualization processing program that can be ported and run between Linux, Windows and other platforms. It can more comprehensively realize multiple functions such as visual editing, modification and storage of point cloud data, thereby greatly expanding the application scenarios and processing depth of PCD format point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A software system framework diagram constructed by the method provided by the present invention;

[0033] Figure 2 This is a diagram of an optional software user interface built based on the method provided by the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The PCD format point cloud data visualization processing software framework developed based on the present invention is as follows Figure 1 As shown, the construction method specifically includes the following steps:

[0036] Step 1: Design the software user interface UI based on the QT framework and QT components, use C++ as the development programming language, use PCL library and VTK library as point cloud data processing tools and point cloud visualization framework, and use Eigen library as the software built-in algorithm mathematical solution tool; first, create a new Qt Widgets Application class project file under the QT framework, and use the pro file as the project configuration file to configure QT, project files, and third-party libraries including PCL, VTK, Eigen and Boost libraries to realize the overall framework Transplantation and operation between Linux and Windows platforms;

[0037] Step 2. According to the multi-threaded optimization design plan, create an instance of the StartThread class to start the software. First, create a QApplication class object in the StartThread class constructor to manage the control flow and main settings of the GUI program; then instantiate the CtrlWindow class to register and manage the software user interaction interface, that is, the UI interface; then create a PCDEditor class object for point cloud processing class function registration and management; then create a PCDEditorThread class object to perform sub-thread packaging of the point cloud processing program; then create a ViewerThread class object to register and manage point cloud visualization class functions, and perform sub-thread packaging at the same time. Finally, call the show() function of the CtrlWindow parent class QWidget to display the software UI, and call the start() function of the QThread class to start the parallel operation of the two sub-threads;

[0038] Step 3: Design the UI window in the QT framework through the ctrlwindow.ui layout file. First, visualize the ctrlwindow.ui window in QTDesign, and build UI components according to actual functional requirements to implement related functions including operation, processing and display of PCD.

[0039] Step 4: Establish a communication mechanism between components and threads based on the signal-slot mechanism, including responding and binding with the UI components in the ctrlwindow.ui layout file through the slot function in the CtrlWindow class of the main thread, ensuring that the interaction between components is achieved through the predefined event triggering strategy;

[0040] Step 5. Design the PCDEditor class that runs under the PCL, VTK, and Eigen library framework and serves as the core of the point cloud data processing sub-thread, so that it can communicate with the CtrlWindow main thread and the ViewerThread sub-thread during runtime to achieve response processing for user interaction operations and instant visualization update of point cloud data;

[0041] Step 6. Design the ViewerThread class as the core of the point cloud visualization and user interaction mechanism. The ViewerThread class implements two-way inter-thread communication by subscribing to the class objects of CtrlWindow and PCDEditor in the constructor, which is used to realize interactive viewing, modification and editing of point cloud data in the visualization window.

[0042] In a preferred embodiment of the present invention, the relevant functions implemented by building UI components in step three specifically include: using Push Button components to load PCD, save PCD, globally reset, reset view angle, screenshot, export, globally downsample, filter out discrete points, pass-through filtering, global coordinate transformation, reset two-point distance measurement, coordinate axis visualization, and process print clearing functions; using Check Box components to design and apply point cloud deletion, point cloud addition, point cloud review, two-point distance measurement, and fixed top-view functions; using Table Widget components to display the results of point cloud review and two-point distance measurement functions; using Horizontal Slider components to develop and apply single point size change and coordinate axis visualization functions; using Spin Box and Double Spin Box components to set hyperparameters for global downsampling, filtering out discrete points, and pass-through filtering; using Combo Box components to select pass-through filtering and global coordinate transformation modes; using Text Edit components to display UI window description text and process print output of some functional modules. Figure 2 An optional software user interface designed through the above steps is shown in FIG.

[0043] In a preferred embodiment of the present invention, in step 4, the CtrlWindow class simultaneously maintains a series of function flags (Flag) and corresponding read (get) and set (set) functions, and query and update the flags and other class member variables are realized through these functions, thereby realizing inter-thread communication. Take the PCD file reading function as an example: when the user clicks the "Load PCD" button (Push Button) in the UI interface, the slot function on_loadPCD_View_clicked() bound to the button in the CtrlWindow class will immediately respond to the execution, first executing a specific function, that is, calling the QFileDialog component pop-up window, so that the user can select a PCD file under a certain path in the window for loading; then perform a PCD file path logic detection, and further extract the absolute path after the detection passes, and encapsulate it in the class member variable pcdFilePathIn, and update the flag loadPCDViewFlag of the function to True, and output the UI process information flow, so that the user can know the current function execution status. If the above file path logic detection is not passed, the flag is not updated, and the relevant alarm information is output in the UI process information flow, prompting the user that the function has not been successfully executed.

[0044] In the corresponding read (get) and set (set) function design, the PCD file reading function is still taken as an example: the getLoadPCDViewFlag() function is designed so that the PCDEditor child thread can monitor the function flag loadPCDViewFlag. When the latter is activated and updated, the relevant program of the PCDEditor child thread starts to execute, and the absolute path of the PCD file to be read is captured through the getPcdFilePathIn() function. Then, specific PCD file reading, assignment, logic checking, UI process information flow output and other functions are executed. Finally, the PCDEditor child thread calls the setLoadPCDViewFlag() function to reset and update the function flag, thereby completing the thread communication and specific function implementation between the main thread and the child thread.

[0045] In a preferred embodiment of the present invention, in step 5, the PCDEditor class declares a pcl::PointCloud <pointt>::Ptr type member variable pe_cloud is used to store global point cloud data; implement the sub-thread encapsulation of PCDEditor instance pe in the StartThread class constructor, and at the same time, PCDEditor's Run() function can respond to the start() method of PCDEditorThread's parent class QThread, execute the Run() method of PCDEditorThread class, and then enable PCDEditor's Run() function to run in the sub-thread to achieve parallel operation;

[0046] The point cloud processing task is continuously executed by implementing an infinite loop function in the Run() function. Its point cloud processing function includes three data management functions: LoadPCDFile() is used to load point clouds, SavePCDFile() is used to store point clouds, and SaveAsFile() supports exporting point clouds to multiple formats, including PLY, PCDBinary, OBJ, and VTK; 4 point cloud filtering modes: AllPointsDownsize() performs global downsampling, allowing users to set the voxel edge length; AllPointsOutlierRemoval() is used to filter out discrete points based on the user-defined hyperparameters of the average distance estimation number of points and the standard deviation multiple; AllPointsPassT hrough() is used to perform through filtering based on the filtering direction and range specified by the user; AllPointsRemoveGround() is used to filter out ground points based on the ground height and elevation threshold specified by the user; the global coordinate transformation capability is provided through the TfTrans() function, and user-defined three-axis transformation is supported; based on the infinite loop nature of the Run() function, the child thread periodically calls the get function of a series of related functions in the CtrlWindow main thread, and monitors user needs through the Flag flag. Once a user request is detected, the child thread immediately executes the corresponding function to meet the user's needs, and resets the Flag by calling the set function of CtrlWindow, thereby completing this communication cycle.

[0047] In a preferred embodiment of the present invention, when AllPointsRemoveGround() performs ground point filtering, the getAllPointsRemoveGroundFlag() function of the CtrlWindow class instance cw is called through inter-thread communication to detect the user request corresponding to the function in real time; when the user request is captured, the getUi() function is called to capture the user interface class instance ui_, and the ground height groundZ and elevation threshold ground_remove_threshold parameters set by the user are directly received from ui_; thereafter, the boundary of the point cloud data is determined, and then rasterization is performed at the specified resolution on the XY plane, and the maximum and minimum values ​​of each grid cell are initialized. Small Z value; traverse the point cloud and update the Z value of the grid cell, and calculate the Z value change of each cell; if the Z value change of the cell is less than the elevation threshold ground_remove_threshold, and the height of the point is close to groundZ, then mark the cell as ground; then, separate the ground and non-ground point clouds according to the marking results of the grid to filter out the ground points; finally, call the setAllPointsRemoveGroundFlag() function to reset and update the ground point filtering function flag to implement the user's request to filter out the ground points; for point cloud data with different densities and terrain features, adapt by adjusting the resolution and threshold, where the actual ground height of the sensor during data collection is considered when setting groundZ.

[0048] In a preferred embodiment of the present invention, in step six, the ViewerThread class is a subclass of QThread, and the start() method of its parent class QThread is directly called in StartThread to execute the run() method of the ViewerThread class, thereby realizing parallel operation of sub-threads; the ViewerThread class declares and maintains a member variable viewer__ of type pcl::visualization::PCLVisualizer, which is used to create a parameterized visualization window; in the above run() method, the viewer__ object is first initialized, and basic visualization window parameters are configured, including window name, position, size, background color, presented point cloud data, various point cloud attributes, view camera parameters, etc.; then the child thread enters an infinite loop function, which loops at a frequency of 10Hz and detects the health of viewer__. When viewer__ stops running, the loop ends automatically to avoid invalid visualization refresh and waste of platform computing power; in the infinite loop, corresponding point cloud rendering and visualization control, user interaction processing, cross-thread parameter synchronization and view export functions are respectively implemented according to different user requests.

[0049] In a preferred embodiment of the present invention, when implementing the point cloud rendering function, the point cloud rendering parameters are specifically initialized in the run() method. When the user triggers the "single point size" change request, ViewerThread captures the current user request flag and the "point size" parameter set by the user through the Horizontal Slider component through communication with the main thread, executes the changePointSize() function, and resets the point cloud rendering parameters according to the user-set parameters. After completing this function and subsequent other function functions, it enters the next loop, and the VTK visualization window object viewer_ is re-rendered according to the newly set parameters, thereby realizing the dynamic point cloud rendering function; the visualization control is specifically the "show coordinate axis" function, which enables the user to dynamically adjust the coordinate axis size through the Horizontal Slider component in the UI interface, and can also trigger the function through the "show coordinate axis / hide coordinate axis" button, and realizes related functions through the showAxis() function, and its basic process is similar to point cloud rendering.

[0050] In a preferred embodiment of the present invention, the user interaction processing function includes four functions: adding point clouds, checking point clouds, two-point distance measurement, and deleting point clouds. When implementing these functions, firstly, based on the PCL framework, the interactive callback functions are registered and bound in the run() function, including keyboard registerKeyboardCallback(), mouse registerMouseCallback(), point selection registerPointPickingCallback(), and area selection registerAreaPickingCallback(); these callback functions also introduce the event triggering mechanism (Event) of the PCL library, which refers to a specified action triggered by a certain behind-the-scenes event, such as a mouse double-click event, etc.; the events applied to the user interaction processing function include: keyboard event KeyboardEvent, mouse event MouseEvent, single point selection event PointPickingEvent, area selection event AreaPickingEvent; through different event response mechanisms and related callback functions, the real-time interaction between the user and the point cloud data in the visualization interface is realized;

[0051] For the point cloud deletion function, the "X" and "x" keys are bound to the on / off function of the point cloud deletion function through the KeyboardEvent event in the pclKeyBoardCallBack() callback function; then the user request is monitored through inter-thread communication, and when the request is captured, the UI process information flow outputs prompt information to assist the user to realize the secondary on / off of the function through the "X" and "x" keys; when the user turns on the function for the second time, the user's mouse selection action will trigger the above-mentioned area selection callback function and start executing the pclAreaPickingCallBack() function; the getPointsIndices() function of the AreaPickingEvent event can extract the serial number of the point in the pe_cloud container selected by the frame, and then combine the ExtractIndices filter to delete the selected point set, and finally output the UI process information flow and reset and update the relevant flag bits to realize the user request;

[0052] To add point cloud function, first call PCL's getCameraParameters() function to get the camera parameters under the current viewing angle, and then use computeViewMatrix() and computeProjectionMatrix() functions to calculate the camera's view matrix M respectively. view and the projection matrix M proj ; Among them, M view Represents the transformation from the point cloud coordinate system to the camera coordinate system, M proj It represents the projection transformation from the camera coordinate system to the screen coordinate system; through the product of the following two matrices M composite The complete transformation relationship from the point cloud coordinate system to the screen coordinate system is defined:

[0053] M composite =M view *M proj

[0054] Then, read two reference points from the pe_cloud container across threads, and obtain their coordinates P1(x1, y1, z1) and P2(x2, y2, z2) in the point cloud coordinate system respectively; convert these two points into two-dimensional coordinates to eliminate the effects of rotation and perspective, and obtain P1'(x1, y1, 0) and P2'(x2, y2, 0); convert the two points into the same coordinate system according to M composite The coordinate system is transformed according to the transformation relationship to obtain the pixel coordinates P1" (a1, b1, 0) and P2" (a2, b2, 0) in the screen coordinate system; with the help of the mouse event MouseEvent, the clicked screen position P0" (a0, b0, 0) is captured; according to the principle of similarity, there is a proportional formula:

[0055]

[0056] Thus, we get:

[0057]

[0058] At this point, the 2D coordinates of the mouse click position in the point cloud coordinate system are obtained:

[0059]

[0060] Then, the intensity and elevation value z of the point are customized to form a complete point cloud point, the point is added to the pe_cloud container, and visually rendered in the next rendering cycle, thereby realizing the user's request to add a point cloud.

[0061] In a preferred embodiment of the present invention, the cross-thread parameter synchronization function includes two parts: perspective reset and global reset. The perspective reset is implemented by the doViewReset() function, which captures the user request through inter-thread communication, and then keeps the perspective camera position unchanged, only resets its observation angle completely in the opposite direction of the elevation to achieve a co-located bird's-eye view effect. When this function needs to be implemented once, a request is sent by clicking the "Perspective Reset" button. If a high-frequency loop is required, a request is sent by checking the "Fixed Bird's Eye" Check Box component. Global reset is implemented by the doAllReset() function. First, a point cloud reset is performed across threads to replace the current point cloud with the initially loaded point cloud. Then, the UI component parameters are reset across threads to reset them all to the state when the software is just started. Finally, the perspective reset of this thread is executed to reset the visualization window.

[0062] In a preferred embodiment of the present invention, the view export function is specifically implemented using the saveScreenShot() function. After capturing the user request through inter-thread communication, the function uses the saveScreenshot() function provided by the VTK framework to capture the current visualization window image information, and initiates a pop-up request to the main thread through the getScreenShotPathOut() function of the CtrlWindow class. The CtrlWindow class invokes the QFileDialog pop-up window to assist the user in specifying the image save path and naming the image. Finally, the function flag is reset and updated through the set function to meet user needs.

[0063] It should be understood that the size of the serial numbers of the steps in the embodiment of the present invention does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.< / pointt> < / pointt>

Claims

1. A method for building a software framework for visualizing point cloud data in PCD format, characterized by: The specific steps include: Step 1: Design the software user interface UI based on the QT framework and QT components, use C++ as the development programming language, use PCL library and VTK library as point cloud data processing tools and point cloud visualization framework, and use Eigen library as the software built-in algorithm mathematical solution tool; first, create a new Qt Widgets Application class project file under the QT framework, and use the pro file as the project configuration file to configure QT, project files, and third-party libraries including PCL, VTK, Eigen and Boost libraries to realize the overall framework Transplantation and operation between Linux and Windows platforms; Step 2. According to the multi-threaded optimization design scheme, create an instance of the StartThread class to start the software. First, create a QApplication class object in the StartThread class constructor to manage the control flow and main settings of the GUI program; then instantiate the CtrlWindow class to register and manage the software user interaction interface, that is, the UI interface; then create a PCDEditor class object for point cloud processing class function registration and management; then create a PCDEditorThread class object to perform sub-thread packaging of the point cloud processing program; Then create a ViewerThread class object to register and manage point cloud visualization class functions and encapsulate child threads. Finally, call the show() function of the CtrlWindow parent class QWidget to display the software UI, and call the start() function of the QThread class to start the parallel operation of the two child threads. Step 3: Design the UI window in the QT framework through the ctrlwindow.ui layout file. First, visualize the ctrlwindow.ui window in QT Design, and build UI components according to actual functional requirements to implement related functions including operation, processing and display of PCD. Step 4: Establish a communication mechanism between components and threads based on the signal-slot mechanism, including responding and binding with the UI components in the ctrlwindow.ui layout file through the slot function in the CtrlWindow class of the main thread, ensuring that the interaction between components is achieved through the predefined event triggering strategy; Step 5. Design the PCDEditor class that runs under the PCL, VTK, and Eigen library framework and serves as the core of the point cloud data processing sub-thread, so that it can communicate with the CtrlWindow main thread and the ViewerThread sub-thread during runtime to achieve response processing for user interaction operations and instant visualization update of point cloud data; Step 6. Design the ViewerThread class as the core of the point cloud visualization and user interaction mechanism. The ViewerThread class implements two-way inter-thread communication by subscribing to the class objects of CtrlWindow and PCDEditor in the constructor, which is used to realize the interactive viewing, modification and editing functions of point cloud data in the visualization window.

2. The method according to claim 1, characterized in that: The relevant functions implemented by building UI components in step three specifically include: using Push Button components to load PCD, save PCD, global reset, view reset, screenshot, export, global downsampling, filtering out discrete points, through filtering, global coordinate transformation, two-point distance measurement reset, coordinate axis visualization, and process print clearing functions; using Check Box components to design and apply point cloud deletion, point cloud addition, point cloud review, two-point distance measurement, and fixed top-down functions; using Table Widget components to review point clouds and display the results of two-point distance measurement functions; using Horizontal Slider components to change the size of a single point and develop and apply coordinate axis visualization functions; using Spin Box and Double Spin Box components to set hyperparameters for global downsampling, filtering out discrete points, and through filtering; using Combo Box components to select the mode of through filtering and global coordinate transformation; using Text Edit components to display the UI window description text and process print output of some functional modules.

3. The method according to claim 1, characterized in that: In step 4, the CtrlWindow class also maintains a series of function flags Flag and corresponding read get and set functions. These functions are used to query and update flags and other class member variables, thereby realizing inter-thread communication.

4. The method according to claim 1, characterized in that: In step 5, the PCDEditor class declares a pcl::PointCloud <pointt> ::Ptr type member variable pe_cloud is used to store global point cloud data; implement the sub-thread encapsulation of PCDEditor instance pe in the StartThread class constructor, and at the same time, PCDEditor's Run() function can respond to the start() method of PCDEditorThread's parent class QThread, execute the Run() method of PCDEditorThread class, and then enable PCDEditor's Run() function to run in the sub-thread to achieve parallel operation;< / pointt> The point cloud processing task is continuously executed by implementing an infinite loop function in the Run() function. Its point cloud processing function includes three data management functions: LoadPCDFile() is used to load point clouds, SavePCDFile() is used to store point clouds, and SaveAsFile() supports exporting point clouds to multiple formats, including PLY, PCDBinary, OBJ, and VTK; 4 point cloud filtering modes: AllPointsDownsize() performs global downsampling, allowing users to set the voxel edge length; AllPointsOutlierRemoval() is used to filter out discrete points based on the user-defined hyperparameters of the average distance estimation number of points and the standard deviation multiple; AllPointsPassT hrough() is used to perform through filtering based on the filtering direction and range specified by the user; AllPointsRemoveGround() is used to filter out ground points based on the ground height and elevation threshold specified by the user; the global coordinate transformation capability is provided through the TfTrans() function, and user-defined three-axis transformation is supported; based on the infinite loop nature of the Run() function, the child thread periodically calls the get function of a series of related functions in the CtrlWindow main thread, and monitors user needs through the Flag flag. Once a user request is detected, the child thread immediately executes the corresponding function to meet the user's needs, and resets the Flag by calling the set function of CtrlWindow, thereby completing this communication cycle.

5. The method according to claim 4, characterized in that: When AllPointsRemoveGround() performs ground point filtering, it specifically calls the getAllPointsRemoveGroundFlag() function of the CtrlWindow class instance cw through inter-thread communication to detect the user request corresponding to this function in real time; when the user request is captured, the user interface class instance ui is captured by calling the getUi() function, and the ground height groundZ and elevation threshold ground_remove_threshold parameters set by the user are directly received from ui_; then, the boundary of the point cloud data is determined, and then rasterization is performed at the specified resolution on the XY plane, and the maximum and minimum Z values ​​of each grid cell are initialized; by traversing The point cloud updates the Z value of the grid cell and calculates the Z value change of each cell; if the Z value change of the cell is less than the elevation threshold ground_remove_threshold, and the height of the point is close to groundZ, the cell is marked as ground; then, the ground and non-ground point clouds are separated according to the marking results of the grid to filter out the ground points; finally, the setAllPointsRemoveGroundFlag() function is called to reset and update the ground point filtering function flag to implement the user's request to filter out the ground points; for point cloud data with different densities and terrain features, the resolution and threshold are adjusted to adapt, and the actual ground height of the sensor during data collection is considered when setting groundZ.

6. The method according to claim 1, characterized in that: In step 6, the ViewerThread class is a subclass of QThread. In StartThread, the start() method of its parent class QThread is directly called to execute the run() method of the ViewerThread class, thereby realizing parallel operation of different threads. The ViewerThread class declares and maintains a member variable viewer__ of the pcl::visualization::PCLVisualizer type, which is used to create a parametrically configured visualization window. In the above run() method, the viewer__ object is first initialized to configure basic visualization window parameters, including window name, position, size, background color, presented point cloud data, various point cloud properties, and viewing angle camera parameters. The child thread then enters an infinite loop function, which loops at a frequency of 10Hz and detects the health of viewer_. The loop ends automatically when viewer_ stops running to avoid invalid visualization refreshes that waste platform computing power. In this infinite loop, the corresponding point cloud rendering and visualization control, user interaction processing, cross-thread parameter synchronization and view export functions are implemented according to different user requests.

7. The method according to claim 6, characterized in that: When implementing the point cloud rendering function, the point cloud rendering parameters are initialized in the run() method. When the user triggers the "single point size" change request, ViewerThread captures the current user request flag and the "point size" parameter set by the user through the Horizontal Slider component through communication with the main thread, executes the changePointSize() function, and resets the point cloud rendering parameters according to the user-set parameters. After completing this function and subsequent function functions, it enters the next loop, and the VTK visualization window object viewer_ is re-rendered according to the newly set parameters, thereby realizing the dynamic point cloud rendering function. The visualization control is specifically the "show axis" function, which enables the user to dynamically adjust the axis size through the Horizontal Slider component in the UI interface, and can also trigger the function through the "show axis / hide axis" button, and implement related functions through the showAxis() function. Its basic process is similar to point cloud rendering.

8. The method according to claim 6, characterized in that: The user interaction processing functions include adding point clouds, checking point clouds, two-point distance measurement, and deleting point clouds. When implementing these functions, we first register and bind the interactive callback functions in the run() function based on the PCL framework, including registerKeyboardCallback() for keyboard, registerMouseCallback() for mouse, registerPointPickingCallback() for point selection, and registerAreaPickingCallback() for area selection. These callback functions also introduce the event triggering mechanism Event of the PCL library, which refers to a specified action triggered by a certain behind-the-scenes event. The events applied to the user interaction processing function include: keyboard event KeyboardEvent, mouse event MouseEvent, single point selection event PointPickingEvent, area selection event AreaPickingEvent; through different event response mechanisms and related callback functions, the user can realize real-time interaction with point cloud data in the visualization interface; For the point cloud deletion function, the "X" and "x" keys are bound to the on / off function of the point cloud deletion function through the KeyboardEvent event in the pclKeyBoardCallBack() callback function; then the user request is monitored through inter-thread communication, and when the request is captured, the UI process information flow outputs prompt information to assist the user to realize the secondary on / off of the function through the "X" and "x" keys; when the user turns on the function for the second time, the user's mouse selection action will trigger the above-mentioned area selection callback function and start executing the pclAreaPickingCallBack() function; the getPointsIndices() function of the AreaPickingEvent event can extract the serial number of the point in the pe_cloud container selected by the frame, and then combine the ExtractIndices filter to delete the selected point set, and finally output the UI process information flow and reset and update the relevant flag bits to realize the user request; To add point cloud function, first call PCL's getCameraParameters() function to get the camera parameters under the current viewing angle, and then use computeViewMatrix() and computeProjectionMatrix() functions to calculate the camera's view matrix M respectively. view and the projection matrix M proj ; Among them, M view Represents the transformation from the point cloud coordinate system to the camera coordinate system, M proj It represents the projection transformation from the camera coordinate system to the screen coordinate system; through the product of the following two matrices M composite The complete transformation relationship from the point cloud coordinate system to the screen coordinate system is defined: M composite =M view *M proj Then, read two reference points from the pe_cloud container across threads, and obtain their coordinates P1(x1, y1, z1) and P2(x2, y2, z2) in the point cloud coordinate system respectively; convert these two points into two-dimensional coordinates to eliminate the effects of rotation and perspective, and obtain P1'(x1, y1, 0) and P2'(x2, y2, 0); convert the two points into the same coordinate system according to M composite The coordinate system is transformed according to the transformation relationship to obtain the pixel coordinates P1" (a1, b1, 0) and P2" (a2, b2, 0) in the screen coordinate system; with the help of the mouse event MouseEvent, the clicked screen position P0" (a0, b0, 0) is captured; according to the principle of similarity, there is a proportional formula: Thus, we get: At this point, the 2D coordinates of the mouse click position in the point cloud coordinate system are obtained: Then, the intensity and elevation value z of the point are customized to form a complete point cloud point, the point is added to the pe_cloud container, and visually rendered in the next rendering cycle, thereby realizing the user's request to add a point cloud.

9. The method according to claim 1, characterized in that: The cross-thread parameter synchronization function includes two parts: perspective reset and global reset. The perspective reset is implemented by the doViewReset() function, which captures user requests through inter-thread communication, then keeps the perspective camera position unchanged, and only resets its observation angle completely in the opposite direction of the elevation to achieve a co-located bird's-eye view effect. When this function needs to be implemented once, send a request by clicking the "Perspective Reset" button. If high-frequency loop execution is required, send a request by checking the "Fixed Bird's Eye" Check Box component. Global reset is implemented by the doAllReset() function. First, the point cloud is reset across threads to replace the current point cloud with the initially loaded point cloud. Then, the UI component parameters are reset across threads to reset them all to the state when the software is just started. Finally, the perspective reset of this thread is executed to reset the visualization window.

10. The method according to claim 1, characterized in that: The view export function is specifically implemented using the saveScreenShot() function. After capturing the user request through inter-thread communication, the function uses the saveScreenshot() function provided by the VTK framework to capture the current visualization window image information, and initiates a pop-up request to the main thread through the getScreenShotPathOut() function of the CtrlWindow class. The CtrlWindow class evokes the QFileDialog pop-up window to assist the user in specifying the image save path and naming the image. Finally, the function flag is reset and updated through the set function to meet user needs.

Images