Multi-mode collaborative laser radar target detection result visual analysis platform

Abstract

The invention relates to a multi-modal collaborative laser radar target detection result visualization analysis platform, which comprises a data layer, a cache layer, a visualization layer and an interaction layer, and is characterized in that the visualization layer is connected with the cache layer and comprises a 3D point cloud visualization module, a multi-view image display module and a comparative analysis module, the 3D point cloud visualization module is constructed based on PyVista and VTK, and the interaction layer is connected with the interaction layer. And the multi-view image display module supports storage and recovery of point cloud color coding and camera attitude, adopts a vectorization generation algorithm of a bounding box, and performs batch processing of bounding box vertex calculation through matrix operation to form a topological data structure of a direct input rendering pipeline. According to the method, abstract detection errors are converted into intuitive visual contrast in a three-dimensional space, prediction frames and true value frames with different colors are matched with selectable category labels, so that the problems of missing detection, false detection, positioning deviation, size errors, direction errors and the like are clear at a glance, and the depth and accuracy of problem diagnosis are improved.

Description

Technical Field

[0001] This invention relates to the field of lidar target detection technology, and in particular to a multimodal collaborative lidar target detection result visualization and analysis platform. Background Technology

[0002] In the field of 3D object detection, especially in the research and evaluation of autonomous driving perception models based on laser point clouds, developers typically rely on quantitative metrics such as precision, recall, and mAP (mean precision) to evaluate model performance. However, these numerical metrics cannot provide intuitive information about why the model fails in specific scenarios. For example, they cannot show the specific spatial deviation between the predicted bounding box and the real object, the morphology of incorrect associations, or missed detections. While some existing general-purpose point cloud visualization tools (such as Open3D's basic viewer) can render point clouds and individual detection results, their functionality is limited to displaying data from a single source, lacking the ability to synchronize and compare model predictions with labeled ground truth values ​​from the same perspective. Furthermore, existing tools generally do not support frame-synchronized interactive browsing with images from multiple cameras, and lack optimization mechanisms (such as intelligent caching and preloading) for efficient rendering and smooth interaction of large-scale point cloud data. This results in low efficiency and sluggish operation when analyzing massive amounts of real-world data, making it difficult to quickly locate and diagnose specific problems with the model in complex scenarios.

[0003] In the prior art, Chinese patent application (publication number: CN117542022A) discloses a method, apparatus, device, medium, and product for evaluating target detection models. This method determines a pre-set ground truth bounding box for an obstacle vehicle in a target image, and detects a predicted bounding box for the obstacle vehicle in the target image based on a target detection model with known parameters. The target image is acquired by an image sensor mounted on the target vehicle. Based on the relative spatial position of the target vehicle and the obstacle vehicle, visible edges within the ground truth bounding box are determined. Visible edges refer to edges within the ground truth bounding box that are within the scanning range of the image sensor. Error calculation is performed based on the visible edges to obtain the boundary error of the target image. The evaluation result of the target detection model is determined based on the boundary error of the target image. However, this approach only focuses on the calculation of numerical indicators and completely lacks intuitive visualization capabilities. Researchers cannot directly observe and understand the model's specific performance in various scenarios through visual means. When the model malfunctions, engineers find it difficult to quickly locate the root cause of the problem and must rely on abstract numerical results for speculation, which greatly reduces the efficiency of model optimization.

[0004] Secondly, existing technologies fail to achieve collaborative display of multimodal data. In autonomous driving scenarios, understanding object detection results requires considering multiple pieces of information simultaneously, such as 3D point cloud data, multi-view 2D images, and the spatial relationship between predicted bounding boxes and ground truth boxes. However, existing evaluation schemes process this information in a fragmented manner, failing to provide a unified perspective. This forces analysts to frequently switch between multiple tools, distracting them and increasing their cognitive burden.

[0005] Third, existing technologies perform poorly when processing large-scale point cloud data. Real-world LiDAR point clouds typically contain hundreds of thousands to millions of points, and existing tools lack effective caching mechanisms and multi-resolution strategies, leading to sluggish interaction and a poor user experience. Particularly during inter-frame navigation, most tools reset the view state of both the 3D perspective and the 2D image, forcing users to repeatedly adjust and reducing analysis efficiency.

[0006] Finally, existing solutions lack the ability to compare and analyze prediction results with ground truth labels. While numerical metrics can reflect overall performance, they struggle to reveal specific spatial relationships and error types (such as missed detections, false detections, positioning offsets, dimensional errors, and orientation errors). In actual R&D, engineers need to directly observe these differences to make targeted improvements to the model architecture or training strategies, and purely numerical evaluation methods cannot provide this intuitive understanding. Summary of the Invention

[0007] In view of the above problems, the present invention provides a multimodal collaborative LiDAR target detection result visualization analysis platform, which not only transforms the abstract detection error into an intuitive visual comparison in three-dimensional space, but also makes problems such as missed detection, false detection, positioning deviation, size error and orientation error clear at a glance. Moreover, the integrated view greatly reduces the cognitive burden, accelerates the problem localization and solution formulation process, and improves the depth and accuracy of problem diagnosis.

[0008] To achieve the above and other related objectives, the present invention provides the following technical solution: A multimodal collaborative lidar target detection result visualization and analysis platform includes a data layer, a caching layer, a visualization layer, and an interaction layer. The data layer is used to automatically match multi-view camera images and point cloud data. Through an intelligent catalog scanning mechanism, it identifies the catalog structure of different datasets and automatically builds a data index. The cache layer, connected to the data layer, is used to adopt a two-level LRU caching strategy, setting up independent cache spaces for point cloud data and image data respectively. The point cloud cache uses the voxel size as part of the cache key and supports multi-resolution data management. The image cache uses the file path as the key to store QImage objects. Through an intelligent prefetching mechanism, the background thread automatically loads image data of adjacent frames according to the position of the current frame. The visualization layer, connected to the cache layer, includes a 3D point cloud visualization module, a multi-view image display module, and a comparison analysis module. The 3D point cloud visualization module is built on PyVista and VTK, and supports point cloud color encoding and the saving and restoration of camera pose. The multi-view image display module adopts a bounding box vectorization generation algorithm and processes bounding box vertex calculations in batches through matrix operations to form a topology data structure that is directly input into the rendering pipeline. The comparison analysis module distinguishes between predicted boxes and ground truth boxes by different colors and supports dynamically displaying / hiding category labels to display the detection quality evaluation results. The interaction layer, connected to the visualization layer, provides intuitive frame navigation control, view state maintenance, and result export functionality.

[0009] Furthermore, the frame navigation control includes left and right arrow keys and previous / next frame buttons.

[0010] Furthermore, the view state is maintained as the 3D camera pose and the 2D image zoom / pan state, which is automatically saved during inter-frame navigation.

[0011] Furthermore, the method of generating bounding boxes using vectorization and batch processing of bounding box vertex calculations through matrix operations to form a topology data structure that is directly input into the rendering pipeline includes: M1. Organize the geometric parameters of the n bounding boxes into a tensor structure: center coordinate tensor C∈R n×3 Size tensor S∈R n×3 and the yaw angle tensor θ∈R n×3 Where n is a positive integer, in the local coordinate system, the eight vertices of the bounding box follow a standard cuboid topology, and its symbolic pattern can be represented as a symbolic matrix M∈R. 8×3 , ; The M2 algorithm fuses size parameters and symbol patterns through a tensor broadcasting mechanism. It decomposes the size tensor S into semi-size vectors along the coordinate axes: dx = width / 2, dy = length / 2, dz = height / 2, forming a column vector of shape (n, 1). This vector is then broadcast multiplied with the symbol matrix to generate the local vertex coordinate tensor V. local ∈R n×8×3 : , Where ∙ represents the Hadamard product, i = 1, 2, 3, ..., n; M3. The rotation transformation is implemented using a standard rotation matrix around the Z-axis. For the i-th target bounding box, its yaw angle is θ. i The rotation operation can be broken down into: , , Z r =Z.

[0012] Furthermore, the method of using a vectorized bounding box generation algorithm and batch processing bounding box vertex calculations through matrix operations to form a topology data structure that is directly input into the rendering pipeline also includes: M4. Rotated vertex tensor V rot Broadcast addition is performed with the center point coordinate tensor C to complete the translation transformation, ultimately obtaining the complete vertex set V in the world coordinate system. world ∈R n×8×3 : , Among them, V rot [i,j,:] represents the local coordinates of the j-th vertex of the i-th bounding box after rotation, and C[i,:] represents the absolute coordinates of the center point of the i-th bounding box in the world coordinate system; M5. The topology connection relationship construction adopts the index offset broadcasting technique. The algorithm predefines the connection pattern of 12 edges of a standard cuboid as E={(0,1),(1,2),(2,3),(3,0),(4,5),(5,6),(6,7),(7,4),(0,4),(1,5),(2,6),(3,7)}, which is represented as an index matrix of shape (12,2). To generate the edge connection relationship of n bounding boxes in batches, the system constructs the base index offset tensor B=reshape([0,8,16,...,8(n-1)],(n,1,1)), expands E to the shape (1, 12, 2), and adds it to B to obtain the batch edge index tensor E. batch ∈R n×12×2 The flattened tensor is combined with the VTK line sequence format [2,i,j] to form the topology data structure for direct input rendering pipeline.

[0013] Furthermore, the three-dimensional parameters of the prediction box and the ground truth box include center point coordinates, size, yaw angle, and category information.

[0014] Furthermore, the workflow of the visualization layer includes: Parse the label file, extract the 3D parameters of the ground truth bounding box and the predicted bounding box, and store the processed point cloud and bounding box data in the cache for quick access later. Update the 3D visualization view, including rendering point clouds, drawing ground truth bounding boxes, drawing predicted bounding boxes, and displaying the corresponding category labels near the bounding boxes; The system loads and updates image views in six surrounding directions simultaneously. It prioritizes reading from the image cache. If the image is not found, it loads the image from the disk and updates the cache. Each image view supports independent zoom and pan operations, and its display state can be saved and restored when switching between frames. Enter the user interaction loop, wait for and respond to user actions.

[0015] Furthermore, the workflow of the interaction layer includes: If a user triggers a frame switching operation, the system updates the current timestamp index and asynchronously preloads the data of adjacent frames through a background thread; If the user adjusts the viewing angle of the 3D view, the system updates the camera parameters and saves the current viewing angle state to ensure that the viewing angle can be quickly restored in subsequent analysis; If the user adjusts the display state of the 2D image view, the system records the state and maintains consistency during frame switching; If the user switches display options, the system updates the rendering status of the 3D scene in real time.

[0016] Furthermore, the interaction layer workflow also includes exporting all associated data of the current frame to the user-specified directory if the user triggers a save operation.

[0017] Furthermore, the workflow of the interaction layer also includes, if the user chooses to exit, the system safely shuts down the 3D renderer, clears all cached data, terminates background threads, and ends program execution.

[0018] The present invention has the following positive effects: 1. This invention transforms abstract detection errors into intuitive visual comparisons in three-dimensional space. Different colored prediction boxes and ground truth boxes, along with optional category labels, make issues such as missed detections, false detections, positioning deviations, size errors, and orientation errors readily apparent. This allows researchers to quickly identify systemic defects in the model (such as insufficient detection capability for specific categories, distances, or poses), greatly accelerating the process of locating and understanding model defects and providing direct evidence for targeted model improvements.

[0019] 2. This invention significantly improves model evaluation efficiency through multimodal data fusion and display. Researchers can simultaneously observe 3D point clouds, multi-view 2D images, prediction boxes, and ground truth boxes within a single interface without switching between multiple tools, gaining a comprehensive understanding of the model's performance characteristics in various scenarios. Especially when analyzing complex scenes (such as distant targets, partial occlusion, and dense objects), this integrated view greatly reduces cognitive burden, accelerates problem localization and solution development, and improves the depth and accuracy of problem diagnosis.

[0020] 3. This invention, through caching, preloading, and optional voxel downsampling, enables the system to smoothly process large-scale real-world scene datasets containing hundreds of thousands of points, achieving a "zero-wait" inter-frame browsing experience. The view state preservation function significantly improves the user experience. When analyzing multiple frames consecutively, the 3D camera pose and 2D image scaling / translation states are automatically saved, eliminating the need for users to repeatedly adjust the viewpoint and allowing them to focus on analyzing the changing patterns of target detection results. This design is particularly suitable for tracking the performance of specific targets over time or analyzing the stability of models during continuous scene transitions.

[0021] 4. This invention employs a modular design and an scalable architecture to support multiple data formats and custom function plugins. Through simple configuration, the system can adapt to the directory structure and data format of different autonomous driving datasets (such as KITTI, nuScenes, Waymo, etc.), and easily adapt to the data formats output by different 3D detection models or access new sensor data types. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the system framework of the present invention; Figure 2 This is a schematic diagram illustrating the operating results of the platform of the present invention; Figure 3 This is a flowchart illustrating the vectorization generation algorithm for the bounding box of the present invention. Figure 4 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0024] Example 1: As Figure 1 or Figure 2As shown, a multimodal collaborative lidar target detection result visualization and analysis platform includes a data layer, a caching layer, a visualization layer, and an interaction layer. The data layer is used to automatically match multi-view camera images and point cloud data. Through an intelligent catalog scanning mechanism, it identifies the catalog structure of different datasets and automatically builds a data index. The cache layer, connected to the data layer, is used to adopt a two-level LRU caching strategy, setting up independent cache spaces for point cloud data and image data respectively. The point cloud cache uses the voxel size as part of the cache key and supports multi-resolution data management. The image cache uses the file path as the key to store QImage objects. Through an intelligent prefetching mechanism, the background thread automatically loads image data of adjacent frames according to the position of the current frame. The visualization layer, connected to the cache layer, includes a 3D point cloud visualization module, a multi-view image display module, and a comparison analysis module. The 3D point cloud visualization module is built on PyVista and VTK, and supports point cloud color encoding and the saving and restoration of camera pose. The multi-view image display module adopts a bounding box vectorization generation algorithm and processes bounding box vertex calculations in batches through matrix operations to form a topology data structure that is directly input into the rendering pipeline. The comparison analysis module distinguishes between predicted boxes and ground truth boxes by different colors and supports dynamically displaying / hiding category labels to display the detection quality evaluation results. The interaction layer, connected to the visualization layer, provides intuitive frame navigation control, view state maintenance, and result export functionality.

[0025] In this embodiment, the frame navigation control includes left and right arrow keys and previous / next frame buttons.

[0026] In this embodiment, the view state is maintained as the 3D camera pose and the 2D image scaling / panning state, which is automatically saved during inter-frame navigation.

[0027] In this embodiment, as Figure 3 As shown, the algorithm for generating bounding boxes using vectorization, and the batch processing of bounding box vertex calculations through matrix operations to form a topology data structure that is directly input into the rendering pipeline, includes: M1. Organize the geometric parameters of the n bounding boxes into a tensor structure: center coordinate tensor C∈R n×3 Size tensor S∈R n×3 and the yaw angle tensor θ∈R n×3 Where n is a positive integer, in the local coordinate system, the eight vertices of the bounding box follow a standard cuboid topology, and its symbolic pattern can be represented as a symbolic matrix M∈R. 8×3 , ; The M2 algorithm fuses size parameters and symbol patterns through a tensor broadcasting mechanism. It decomposes the size tensor S into semi-size vectors along the coordinate axes: dx = width / 2, dy = length / 2, dz = height / 2, forming a column vector of shape (n, 1). This vector is then broadcast multiplied with the symbol matrix to generate the local vertex coordinate tensor V. local ∈R n×8×3 : , Where ∙ represents the Hadamard product, i = 1, 2, 3, ..., n; M3. The rotation transformation is implemented using a standard rotation matrix around the Z-axis. For the i-th target bounding box, its yaw angle is θ. i The rotation operation can be broken down into: , , Z r =Z.

[0028] In this embodiment, the step of employing a bounding box vectorization generation algorithm and batch processing bounding box vertex calculations through matrix operations to form a topology data structure directly input to the rendering pipeline further includes: M4. Rotated vertex tensor V rot Broadcast addition is performed with the center point coordinate tensor C to complete the translation transformation, ultimately obtaining the complete vertex set V in the world coordinate system. world ∈R n×8×3 : , Among them, V rot [i,j,:] represents the local coordinates of the j-th vertex of the i-th bounding box after rotation, and C[i,:] represents the absolute coordinates of the center point of the i-th bounding box in the world coordinate system; M5. The topology connection relationship construction adopts the index offset broadcasting technique. The algorithm predefines the connection pattern of 12 edges of a standard cuboid as E={(0,1),(1,2),(2,3),(3,0),(4,5),(5,6),(6,7),(7,4),(0,4),(1,5),(2,6),(3,7)}, which is represented as an index matrix of shape (12,2). To generate the edge connection relationship of n bounding boxes in batches, the system constructs the base index offset tensor B=reshape([0,8,16,...,8(n-1)],(n,1,1)), expands E to the shape (1, 12, 2), and adds it to B to obtain the batch edge index tensor E. batch ∈R n×12×2The flattened tensor is combined with the VTK line sequence format [2, i, j] to form the topology data structure for direct input rendering pipeline.

[0029] In this embodiment, during the visualization update phase, the system innovatively introduces an incremental update mechanism based on content signatures, significantly reducing unnecessary redrawing overhead. For the bounding box set B={b1,b2,...,b...} n The system extracts its geometric feature tensor F∈R. n×7 Each bounding box contains its center coordinates (3D), size parameters (3D), and yaw angle (1D). To eliminate misjudgments caused by minor differences in floating-point calculations, the system rounds the feature tensor to 4 decimal places. F rounded [i,j] = round(F[i,j], 4) Then, the BLAKE2B hash algorithm is used to generate a 16-byte content signature σ: σ=BLAKE2B(flatten(F rounded ), digest size =16) The predicted bounding boxes and ground truth bounding boxes employ a differentiated signature strategy, generating independent signature spaces by adding different prefixes to ensure that the two types of bounding boxes manage their update states independently. During each frame update, the system compares the current signature with the cached signature, triggering bounding box redrawing and label update operations only when the signature changes, thereby reducing unnecessary GPU rendering calls.

[0030] The label display employs color differentiation and spatial layering strategies to enhance visual differentiation. This design effectively avoids label overlap in dense scenes, enabling users to quickly identify typical error types such as missed detections (only truth boxes), false positives (only predicted boxes), and positioning errors (box position offset). Label content is dynamically generated based on data source differences: truth labels include the category name, target ID, and ghost attribute identifier, while predicted labels only display the category name, further strengthening the visual differentiation effect.

[0031] The computational complexity analysis of the vectorized algorithm shows that the time complexity of the traditional loop method is O(n×k) (n is the number of boxes, k is the sum of the number of vertices and edges of a single box, k=20); while the vectorized algorithm transforms the main computation into tensor broadcasting operations, reducing the time complexity to O(n) with a very small constant factor.

[0032] In this embodiment, the three-dimensional parameters of the prediction box and the truth box include the center point coordinates, size, yaw angle, and category information.

[0033] In this embodiment, the workflow of the visualization layer includes: Parse the label file, extract the 3D parameters of the ground truth bounding box and the predicted bounding box, and store the processed point cloud and bounding box data in the cache for quick access later. Update the 3D visualization view, including rendering point clouds, drawing ground truth bounding boxes, drawing predicted bounding boxes, and displaying the corresponding category labels near the bounding boxes; The system loads and updates image views in six surrounding directions simultaneously. It prioritizes reading from the image cache. If the image is not found, it loads the image from the disk and updates the cache. Each image view supports independent zoom and pan operations, and its display state can be saved and restored when switching between frames. Enter the user interaction loop, wait for and respond to user actions.

[0034] In this embodiment, the workflow of the interaction layer includes: If a user triggers a frame switching operation, the system updates the current timestamp index and asynchronously preloads the data of adjacent frames through a background thread; If the user adjusts the viewing angle of the 3D view, the system updates the camera parameters and saves the current viewing angle state to ensure that the viewing angle can be quickly restored in subsequent analysis; If the user adjusts the display state of the 2D image view, the system records the state and maintains consistency during frame switching; If the user switches display options, the system updates the rendering status of the 3D scene in real time.

[0035] In this embodiment, the interaction layer workflow also includes exporting all associated data of the current frame to the user-specified directory if the user triggers a save operation.

[0036] In this embodiment, the workflow of the interaction layer also includes, if the user chooses to exit, the system safely shuts down the 3D renderer, clears all cached data, terminates background threads, and ends program execution.

[0037] Example 2: Based on the multimodal collaborative lidar target detection result visualization analysis platform of Example 1, the present invention will be further explained and described below.

[0038] like Figure 1 or Figure 2 As shown, a multimodal collaborative lidar target detection result visualization and analysis platform includes a data layer, a caching layer, a visualization layer, and an interaction layer. The data layer is used to automatically match multi-view camera images and point cloud data. Through an intelligent catalog scanning mechanism, it identifies the catalog structure of different datasets and automatically builds a data index. The cache layer, connected to the data layer, is used to adopt a two-level LRU caching strategy, setting up independent cache spaces for point cloud data and image data respectively. The point cloud cache uses the voxel size as part of the cache key and supports multi-resolution data management. The image cache uses the file path as the key to store QImage objects. Through an intelligent prefetching mechanism, the background thread automatically loads image data of adjacent frames according to the position of the current frame. The visualization layer, connected to the cache layer, includes a 3D point cloud visualization module, a multi-view image display module, and a comparison analysis module. The 3D point cloud visualization module is built on PyVista and VTK, and supports point cloud color encoding and the saving and restoration of camera pose. The multi-view image display module adopts a bounding box vectorization generation algorithm and processes bounding box vertex calculations in batches through matrix operations to form a topology data structure that is directly input into the rendering pipeline. The comparison analysis module distinguishes between predicted boxes and ground truth boxes by different colors and supports dynamically displaying / hiding category labels to display the detection quality evaluation results. The interaction layer, connected to the visualization layer, provides intuitive frame navigation control, view state maintenance, and result export functionality.

[0039] In this embodiment, the platform adopts a layered architecture design, including a data layer, a caching layer, a visualization layer, and an interaction layer, which together realize efficient multimodal data collaborative display and analysis functions. At the data layer, the system supports efficient reading of .bin format point cloud data, is compatible with .txt format annotation and prediction result parsing, and can automatically match multi-view camera images with point cloud data. Through an intelligent directory scanning mechanism, the system can identify the directory structure of different datasets, automatically build data indexes, and achieve fast data loading and access.

[0040] At the caching layer, the system employs a two-level LRU caching strategy, setting up independent cache spaces for point cloud data and image data respectively. The point cloud cache uses voxel size as part of the cache key, supporting multi-resolution data management; the image cache uses file path as the key to store QImage objects. The system also implements an intelligent prefetch mechanism, where a background thread automatically loads adjacent frame data based on the current frame position, significantly reducing user waiting time. To ensure thread safety, all caching operations are protected by mutex locks to avoid data inconsistency issues caused by concurrent access.

[0041] The visualization layer is the core of this invention, comprising a 3D point cloud visualization module, a multi-view image display module, and a comparative analysis module. The 3D point cloud visualization module is built on PyVista and VTK, supporting point cloud color encoding (height, intensity, or RGB) and the saving and restoration of camera pose. In particular, this invention innovatively implements a vectorized bounding box generation algorithm, batch processing bounding box vertex calculations through matrix operations, significantly improving rendering efficiency. The comparative analysis module distinguishes between predicted and ground truth boxes using different colors (ground truth boxes use red, predicted boxes use blue) and supports dynamically showing / hiding category labels, making detection quality assessment readily apparent.

[0042] At the interaction layer, the system provides intuitive frame navigation controls (left and right arrow keys, previous / next frame buttons), view state preservation (3D camera pose and 2D image zoom / pan state are automatically saved during inter-frame navigation), and result export functionality. Users can quickly switch view modes, adjust point cloud density (voxel downsampling), and save the current frame's 3D screenshot or the complete dataset using keyboard shortcuts. The interface layout is meticulously designed, with multi-view 2D images displayed on the left and right sides (front / back / left front / right front / left back / right back) respectively, and the central area presenting 3D point clouds and bounding boxes, creating a spatially connected observation experience.

[0043] This invention also implements a superior resource management mechanism. When the application is closed, the system sets an exit flag, clears the task queue, waits for background tasks to complete, and safely releases 3D renderer resources, avoiding memory leaks and state inconsistencies. This design ensures the stability and reliability of the system, maintaining good performance even when processing large-scale datasets.

[0044] A method for synchronously comparing and visualizing predictions and ground truth: The core algorithm and implementation process of performing differentiated rendering (color, label) on 3D bounding boxes from different sources (model predictions and manual annotations) and synchronously comparing and displaying them in the same 3D coordinate system.

[0045] Multimodal data frame synchronization and view linkage mechanism: realizes high-precision alignment between LiDAR point cloud sequence and multi-camera image sequence based on timestamp, and ensures the state linkage and collaborative update of 3D view and 2D image view during the interaction process.

[0046] Performance-optimized architecture for smooth browsing: A comprehensive performance optimization scheme combining LRU caching, background asynchronous preloading, and configurable point cloud simplification (voxel downsampling) ensures real-time interactive capabilities for massive sequence data.

[0047] Interactive context preservation technology: enables the saving and restoration of 3D camera parameters (position, viewpoint, zoom) and 2D image view state during frame switching, maintaining the continuity of user analysis tasks.

[0048] In this embodiment, as Figure 4 As shown, the system starts up, and the user selects the root directory containing point cloud, label, and image data through the graphical interface.

[0049] The data indexing module automatically scans each subfolder under the root directory (including velodyne, label_2, label, and image), extracts the timestamp information of each file, and establishes a unified time series index by calculating the intersection of timestamps to ensure the alignment of multimodal data in the time dimension.

[0050] The system initializes the main interface, creating a 3D visualization view and six 2D image views (corresponding to the six surrounding view directions: front left, front, back left, front right, back right, and back right).

[0051] Load the frame data corresponding to the currently selected timestamp. The system first checks the LRU cache; if processed data exists in the cache, it reads it directly.

[0052] The system reads point cloud files (.bin format), truth label files (.txt format), and prediction label files (.txt format) from the disk in parallel. For the point cloud data, voxel downsampling is performed depending on whether the user has enabled "Fast Mode".

[0053] The label file is parsed to extract the 3D parameters (including center point coordinates, dimensions, yaw angle, and category information) of the ground truth bounding boxes and predicted bounding boxes. The processed point cloud and bounding box data are then cached for later quick access.

[0054] Update the 3D visualization view. This includes rendering the point cloud (using height or intensity shading), drawing the ground truth bounding box (displayed in a specific color such as white), drawing the predicted bounding box (displayed in a striking color such as red), and displaying the corresponding category label near the box.

[0055] The system synchronously loads and updates image views in six panoramic directions. It prioritizes reading from the image cache; if the image is not found in the cache, it loads the image from disk and updates the cache. Each image view supports independent zoom and pan operations, and its display state can be saved and restored during frame transitions.

[0056] Enter the user interaction loop, wait for and respond to user actions.

[0057] If a user triggers a frame switching operation (previous frame / next frame), the system updates the current timestamp index and asynchronously preloads data from adjacent frames via a background thread to improve browsing smoothness.

[0058] If the user adjusts the viewing angle of the 3D view, the system updates the camera parameters and saves the current viewing angle state to ensure that the viewing angle can be quickly restored in subsequent analysis.

[0059] If a user adjusts the display state of a 2D image view (zoom, pan), the system records the state and maintains consistency during frame switching.

[0060] If the user switches display options (such as showing / hiding bounding box labels), the system updates the rendering status of the 3D scene in real time.

[0061] If the user triggers a save operation, the system will export all associated data of the current frame (point cloud file, label file, multi-view image and 3D scene screenshot) to the user-specified directory.

[0062] If the user chooses to exit, the system will safely shut down the 3D renderer, clear all cached data, terminate background threads, and end the program.

[0063] In summary, this invention not only transforms abstract detection errors into intuitive visual comparisons in three-dimensional space, with different colored prediction boxes and truth boxes, along with optional category labels, making issues such as missed detections, false detections, positioning deviations, size errors, and orientation errors readily apparent, but also greatly reduces cognitive burden by integrating the view, accelerating the problem localization and solution development process, and improving the depth and accuracy of problem diagnosis.

[0064] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multimodal collaborative lidar target detection result visualization and analysis platform, comprising a data layer, a cache layer, a visualization layer, and an interaction layer, characterized in that: The data layer is used to automatically match multi-view camera images and point cloud data. Through an intelligent catalog scanning mechanism, it identifies the catalog structure of different datasets and automatically builds a data index. The cache layer, connected to the data layer, is used to adopt a two-level LRU caching strategy, setting up independent cache spaces for point cloud data and image data respectively. The point cloud cache uses the voxel size as part of the cache key and supports multi-resolution data management. The image cache uses the file path as the key to store QImage objects. Through an intelligent prefetching mechanism, the background thread automatically loads image data of adjacent frames according to the position of the current frame. The visualization layer, connected to the cache layer, includes a 3D point cloud visualization module, a multi-view image display module, and a comparison analysis module. The 3D point cloud visualization module is built on PyVista and VTK, and supports point cloud color encoding and the saving and restoration of camera pose. The multi-view image display module adopts a bounding box vectorization generation algorithm and processes bounding box vertex calculations in batches through matrix operations to form a topology data structure that is directly input into the rendering pipeline. The comparison analysis module distinguishes between predicted boxes and ground truth boxes by different colors and supports dynamically displaying / hiding category labels to display the detection quality evaluation results. The interaction layer, connected to the visualization layer, provides intuitive frame navigation control, view state maintenance, and result export functionality.

2. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that: The frame navigation control includes left and right arrow keys and previous / next frame buttons.

3. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that: The view state is maintained as the 3D camera pose and the 2D image zoom / pan state, and is automatically saved during inter-frame navigation.

4. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that, The method employs a vectorized bounding box generation algorithm, and uses matrix operations to batch process bounding box vertex calculations to form a topology data structure that is directly input into the rendering pipeline. M1. Organize the geometric parameters of the n bounding boxes into a tensor structure: center coordinate tensor C∈R n×3 Size tensor S∈R n×3 and the yaw angle tensor θ∈R n×3 Where n is a positive integer, in the local coordinate system, the eight vertices of the bounding box follow a standard cuboid topology, and its symbolic pattern can be represented as a symbolic matrix M∈R. 8×3 , ； The M2 algorithm fuses size parameters and symbol patterns through a tensor broadcasting mechanism. It decomposes the size tensor S into semi-size vectors along the coordinate axes: dx = width / 2, dy = length / 2, dz = height / 2, forming a column vector of shape (n, 1). This vector is then broadcast multiplied with the symbol matrix to generate the local vertex coordinate tensor V. local ∈R n×8×3 : , Where ∙ represents the Hadamard product, i = 1, 2, 3, ..., n; M3. The rotation transformation is implemented using a standard rotation matrix around the Z-axis. For the i-th target bounding box, its yaw angle is θ. i The rotation operation can be broken down into: ， ， WITH r =Z。 5. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 4, characterized in that, The method of using a vectorized bounding box generation algorithm and batch processing bounding box vertex calculations through matrix operations to form a topology data structure that is directly input into the rendering pipeline also includes: M4. Rotated vertex tensor V rot Broadcast addition is performed with the center point coordinate tensor C to complete the translation transformation, ultimately obtaining the complete vertex set V in the world coordinate system. world ∈R n×8×3 : ， Among them, V rot [i,j,:] represents the local coordinates of the j-th vertex of the i-th bounding box after rotation, and C[i,:] represents the absolute coordinates of the center point of the i-th bounding box in the world coordinate system; M5. The topology connection relationship construction adopts the index offset broadcasting technique. The algorithm predefines the connection pattern of 12 edges of a standard cuboid as E={(0,1),(1,2),(2,3),(3,0),(4,5),(5,6),(6,7),(7,4),(0,4),(1,5),(2,6),(3,7)}, which is represented as an index matrix of shape (12,2). To generate the edge connection relationship of n bounding boxes in batches, the system constructs the base index offset tensor B=reshape([0,8,16,...,8(n-1)],(n,1,1)), expands E to the shape (1, 12, 2), and adds it to B to obtain the batch edge index tensor E. batch ∈R n×12×2 The flattened tensor is combined with the VTK line sequence format [2, i, j] to form the topology data structure of the direct input rendering pipeline.

6. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that: The three-dimensional parameters of the prediction box and the truth box include the center point coordinates, size, yaw angle, and category information.

7. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that, The workflow of the visualization layer includes: Parse the label file, extract the 3D parameters of the ground truth bounding box and the predicted bounding box, and store the processed point cloud and bounding box data in the cache for quick access later. Update the 3D visualization view, including rendering point clouds, drawing ground truth bounding boxes, drawing predicted bounding boxes, and displaying the corresponding category labels near the bounding boxes; The system loads and updates image views in six surrounding directions simultaneously. It prioritizes reading from the image cache. If the image is not found, it loads the image from the disk and updates the cache. Each image view supports independent zoom and pan operations, and its display state can be saved and restored when switching between frames. Enter the user interaction loop, wait for and respond to user actions.

8. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 1, characterized in that, The workflow of the interaction layer includes: If a user triggers a frame switching operation, the system updates the current timestamp index and asynchronously preloads the data of adjacent frames through a background thread; If the user adjusts the viewing angle of the 3D view, the system updates the camera parameters and saves the current viewing angle state to ensure that the viewing angle can be quickly restored in subsequent analysis; If the user adjusts the display state of the 2D image view, the system records the state and maintains consistency during frame switching; If the user switches display options, the system updates the rendering status of the 3D scene in real time.

9. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 8, characterized in that, The interaction layer workflow also includes exporting all associated data of the current frame to the user-specified directory if the user triggers a save operation.

10. The multimodal collaborative lidar target detection result visualization and analysis platform according to claim 9, characterized in that, The workflow of the interaction layer also includes, if the user chooses to exit, the system safely shuts down the 3D renderer, clears all cached data, terminates background threads, and ends program execution.

Images