A web-based 3D point cloud semantic segmentation method and system
By projecting 3D point cloud coordinates onto a 2D screen space and combining a two-level collision detection strategy of axis-aligned bounding box pre-screening and off-screen Canvas precise judgment, the computational complexity and interactivity issues of large-scale point clouds in web browsers are solved, achieving efficient semantic segmentation and a smooth interactive experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 广州祺宸科技有限公司
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-29
Smart Images
Figure CN122115859A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of semantic segmentation methods, specifically relating to a web-based 3D point cloud semantic segmentation method and system. Background Technology
[0002] With the popularization of 3D sensing technology, 3D point cloud data has become a core data form in many fields. Semantic segmentation, as a fundamental task of point cloud understanding, aims to assign semantic labels to each 3D point. Current point cloud processing technologies include traditional algorithms that perform geometric calculations and neighborhood searches directly in 3D space, as well as efficient algorithms that rely on parallel computing frameworks such as CUDA, which provide a foundation for realizing semantic segmentation.
[0003] However, when the above-mentioned solutions are applied to interactive point cloud semantic segmentation on the Web, the complexity of direct geometric calculations in 3D space is extremely high, making it difficult to handle large-scale point clouds. Furthermore, as a lightweight platform, the Web browser has limited computing and graphics processing capabilities, making it impossible to directly port algorithms that rely on specific hardware acceleration. Moreover, the intuitive and accurate interactive design of 3D data on a 2D screen is complex, resulting in cumbersome operation, response delays, and low user experience and annotation efficiency of existing solutions. Summary of the Invention
[0004] The technical problem to be solved by this invention is to overcome the problems of high computational complexity, limited performance, and poor interactive experience when performing real-time, interactive semantic segmentation of large-scale 3D point clouds in a web browser environment.
[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a web-based 3D point cloud semantic segmentation method, the method comprising the following steps:
[0006] Transform the coordinates of the 3D point cloud in the world coordinate system to the 2D screen coordinate system using the camera view matrix and projection matrix;
[0007] Based on the marked shapes drawn by the user on the two-dimensional screen, a two-level progressive collision detection strategy is adopted to filter the converted screen coordinate points. The two-level progressive collision detection strategy includes: firstly, using the axis alignment bounding box of the marked shape for fast pre-screening, and then using off-screen Canvas technology to accurately judge the pre-screened points in order to filter out the points located inside the marked shape.
[0008] Semantic labels are assigned to points that pass collision detection, and the semantic labels and display status codes are stored together in the dual-attribute field of each point. The segmentation results are visualized by real-time rendering through the GPU vertex shader.
[0009] As an optional implementation, in the first aspect of the present invention, the transformation of the coordinates in the world coordinate system of the three-dimensional point cloud to the two-dimensional screen coordinate system through the camera view matrix and the projection matrix includes:
[0010] The coordinates in the world coordinate system are transformed to the normalized device coordinate space using the view matrix and projection matrix, where the normalized device coordinates are obtained through the formula... calculate, Let the world coordinates of the point be... For camera view matrix, The projection matrix;
[0011] Map normalized device coordinates to screen pixel coordinates;
[0012] Screen X coordinate through calculate;
[0013] Screen Y coordinate through calculate;
[0014] in, and This refers to the canvas size.
[0015] As an optional implementation, in the first aspect of the present invention, the use of off-screen Canvas technology for accurate judgment specifically includes the following steps:
[0016] Create an off-screen Canvas element in memory that is the same size as the displayed view;
[0017] Use the CanvasRenderingContext2D API to fill the user-drawn annotation shape onto the off-screen Canvas with a specific color;
[0018] For each point to be judged, the color value of the corresponding pixel on the off-screen Canvas is read using the getImageData method;
[0019] If the color value matches the specific color, then the point is determined to be inside the marked shape.
[0020] As an optional implementation, in the first aspect of the invention, the dual-attribute field is a property attribute defined for each point;
[0021] The property.x file stores semantic tag IDs, which correspond to a predefined semantic category table.
[0022] The property.y file stores the display status code, which is used to control the rendering behavior of the point.
[0023] As an optional implementation, the first aspect of the present invention further includes a frame-overlay annotation mode for synchronously annotating time-series point clouds, specifically including:
[0024] Align multiple frame point clouds to a unified reference coordinate system;
[0025] Based on the labeled shapes drawn by the user on the reference frame, collision detection and semantic information storage are performed synchronously on the point clouds of all frames.
[0026] As an optional implementation, in the first aspect of the invention, the annotation shape includes a shape generated by a lasso tool, polygon tool, rectangle tool, or point tool;
[0027] The method also supports overwrite, non-overwrite, update, or delete annotation processing modes.
[0028] As an optional implementation, the first aspect of the invention also includes point cloud picking optimization using a ray casting method:
[0029] Calculate the 3D ray in reverse based on the screen coordinates of the mouse click, find the intersection of the ray with the point cloud, select the nearest point, and highlight the corresponding segmented object.
[0030] As an optional implementation, in the first aspect of the present invention, the ray projection method specifically includes:
[0031] The calculated 3D ray is geometrically intersected with the point cloud, and the index of the nearest point intersecting the ray is returned. After finding the corresponding segmented object based on the index, the bounding box of the object is highlighted, and the opacity of other point clouds is reduced.
[0032] A second aspect of this invention discloses a web-based 3D point cloud semantic segmentation system for implementing the web-based 3D point cloud semantic segmentation method described in any of the above embodiments. The system includes:
[0033] The coordinate transformation module is used to transform the coordinates of a 3D point cloud in the world coordinate system to a 2D screen coordinate system through the camera view matrix and projection matrix.
[0034] The collision detection module uses a two-level progressive collision detection strategy to filter the converted screen coordinate points based on the marked shapes drawn by the user on the two-dimensional screen.
[0035] The rendering module is used to assign semantic labels to points that pass collision detection and store the semantic labels and display status codes in the dual-attribute field of each point. The GPU vertex shader renders the data in real time to visualize the segmentation results.
[0036] A third aspect of this invention discloses another web-based 3D point cloud semantic segmentation system, the system comprising:
[0037] Memory containing executable program code;
[0038] A processor coupled to the memory;
[0039] The processor calls the executable program code stored in the memory to execute the web-based 3D point cloud semantic segmentation method disclosed in the first aspect of the present invention.
[0040] The fourth aspect of this invention discloses a computer-readable storage medium storing computer instructions, which, when invoked by a processor, are used to execute a web-based 3D point cloud semantic segmentation method disclosed in the first aspect of this invention.
[0041] Compared with the prior art, the beneficial effects of the present invention are:
[0042] By projecting the coordinates of a 3D point cloud onto a 2D screen space, this invention transforms complex 3D geometric calculations into efficient 2D graphics operations. Combined with a two-level collision detection strategy—axis-aligned bounding box pre-screening and off-screen Canvas precise judgment—it significantly reduces computational load, enabling real-time processing of millions of point clouds in ordinary web browsers. This solves the performance bottleneck problem on the web. The system is entirely built on standard web technologies such as WebGL, requiring no specific plugins or computational frameworks, and possesses excellent versatility and scalability. Furthermore, this invention provides intuitive interactive tools and modes similar to 2D image annotation, and combined with real-time rendering using GPU vertex shaders, ensures immediate response and a smooth experience, improving annotation efficiency. In addition, the innovative frame-stacked annotation mode, through coordinate system unification and batch projection, achieves synchronous annotation of time-series point clouds, not only improving the efficiency of dynamic target annotation by an order of magnitude but also ensuring the spatiotemporal consistency of annotation results through mathematical transformations. Attached Figure Description
[0043] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:
[0044] Figure 1 This is a flowchart illustrating a web-based 3D point cloud semantic segmentation method disclosed in an embodiment of the present invention.
[0045] Figure 2 This is a schematic diagram of the structure of a web-based 3D point cloud semantic segmentation system disclosed in an embodiment of the present invention;
[0046] Figure 3This is a schematic diagram of another web-based 3D point cloud semantic segmentation system disclosed in an embodiment of the present invention. Detailed Implementation
[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.
[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0050] This invention discloses a web-based 3D point cloud semantic segmentation method and system. By projecting the coordinates of a 3D point cloud onto a 2D screen space, this invention transforms complex 3D geometric calculations into efficient 2D graphics operations. Combined with a two-level collision detection strategy of axis-aligned bounding box pre-screening and off-screen Canvas precise judgment, the computational load is significantly reduced, making it possible to process millions of point clouds in real time in a regular web browser. This solves the performance bottleneck problem of the web. The system is built entirely on standard web technologies such as WebGL, requiring no specific plugins or computing frameworks, and possesses excellent versatility and scalability. At the same time, this invention provides intuitive interactive tools and modes similar to 2D image annotation, and combined with real-time rendering using GPU vertex shaders, ensures immediate response and a smooth experience, improving annotation efficiency.
[0051] Example 1
[0052] Please see Figure 1 , Figure 1 This is a flowchart illustrating a web-based 3D point cloud semantic segmentation method disclosed in an embodiment of the present invention. Wherein, Figure 1 The described web-based 3D point cloud semantic segmentation method is applied to a data processing chip, processing terminal, or processing server, which can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 1 As shown, this web-based 3D point cloud semantic segmentation method can include the following operations:
[0053] 101. Transform the coordinates of the 3D point cloud in the world coordinate system to the 2D screen coordinate system using the camera view matrix and projection matrix.
[0054] Specifically, this invention transforms complex geometric calculations traditionally performed in three-dimensional space into operations in two-dimensional screen space. In particular, the calculation process is simplified from the high-complexity problem of dealing with the relationships between points and points and complex three-dimensional shapes in three-dimensional space to coordinate mapping and pixel-level judgment on a two-dimensional plane. This transformation, combined with a two-level strategy of fast pre-screening with axis-aligned bounding boxes and accurate judgment on off-screen Canvas, significantly reduces the computational overhead of a single judgment, making it possible to process large-scale point cloud data in real time in a standard web browser environment, effectively overcoming the bottleneck of limited computing resources on the web.
[0055] As can be seen, the coordinate projection-based process relies entirely on the standard WebGL and JavaScript technology stack, without the need for additional plugins or specific computing frameworks. This not only gives the system a lightweight architecture but also ensures its good compatibility and portability in various modern web browsers. At the same time, this process provides a unified data foundation for subsequent intuitive interactions based on two-dimensional screen coordinates (such as lasso selection) and provides technical support for a smooth interactive experience through real-time rendering via GPU vertex shaders.
[0056] 102. Based on the annotation shapes drawn by the user on the two-dimensional screen, a two-level progressive collision detection strategy is adopted to filter the converted screen coordinate points.
[0057] Specifically, this collision detection strategy optimizes the computation process through a two-stage progressive processing mechanism. The first stage uses axis-aligned bounding boxes of labeled shapes for rapid pre-screening. A large number of obviously irrelevant screen coordinate points can be eliminated through four simple coordinate comparisons, reducing the computational base for subsequent accurate judgment. The second stage uses off-screen Canvas technology to fill the user-drawn polygonal region in the memory Canvas with a specific color. Geometric position is determined by querying the pixel color value of the screen coordinates corresponding to the query point. This transforms the high-complexity point judgment within the polygon in traditional algorithms into a one-time color filling and a pixel query operation with constant time complexity. This combined strategy effectively solves the performance bottleneck caused by directly performing geometric calculations when the amount of 3D point cloud data is huge.
[0058] As can be seen, this strategy fully utilizes Web standard graphics APIs to achieve efficient computation, avoiding dependence on dedicated computing hardware. Off-screen Canvas technology is based on the browser's natively supported CanvasRenderingContext2D interface, and completes accurate collision detection through color filling and pixel query, which not only ensures the accuracy of the judgment, but also ensures that the solution has good compatibility in various browsers that support modern Web standards. Combined with the real-time rendering of the detection results by the GPU vertex shader, this invention can still maintain the instant response of interactive operations in a resource-limited environment, and provide a smooth visual feedback experience for point cloud annotation in complex scenes.
[0059] 103. Assign semantic labels to points that pass collision detection, and store the semantic labels and display status codes together in the dual-attribute field of each point. Then, use the GPU vertex shader to render in real time to visualize the segmentation results.
[0060] Specifically, by designing dual-attribute fields for each point to store semantic labels and display status codes, high efficiency in data storage and flexibility in rendering control are achieved. The property.x field encodes semantic categories in the form of integer indices, while the property.y field independently controls the rendering status of the point. This compact two-dimensional vector structure minimizes storage overhead while decoupling semantic information from display logic. These attributes are directly read and parsed by the GPU vertex shader, and the color, size, and transparency of the point are dynamically calculated. The status judgment and rendering tasks of a large number of points are transferred to the graphics hardware for parallel processing, ensuring that visual updates remain smooth in real time even when facing millions of point clouds.
[0061] As can be seen, this invention supports complex annotation workflows and interactive feedback. For example, during the annotation process, the selected area can be highlighted instantly by setting property.y to a high-brightness status code, or the background can be faded by a semi-transparent status code to focus on the key points, providing users with clear visual confirmation. The independent control of semantic tags and display status allows for multiple rounds of iterative annotation, result comparison, and error correction, improving the accuracy and functionality of annotation operations. At the same time, it reserves flexible interfaces for future expansion of more semantic categories or interactive states.
[0062] As an optional embodiment, the step described above, transforming the coordinates in the world coordinate system of the 3D point cloud to the 2D screen coordinate system using a camera view matrix and a projection matrix, includes:
[0063] The coordinates in the world coordinate system are transformed to the normalized device coordinate space using the view matrix and projection matrix, where the normalized device coordinates are obtained through the formula... calculate, Let the world coordinates of the point be... For camera view matrix, The projection matrix;
[0064] Map normalized device coordinates to screen pixel coordinates;
[0065] Screen X coordinate through calculate;
[0066] Screen Y coordinate through calculate;
[0067] in, and This refers to the canvas size.
[0068] In this embodiment of the invention, in terms of computational efficiency, the invention shifts the processing focus from three-dimensional Euclidean space to two-dimensional projection space. World coordinates are transformed into normalized device coordinate space through view matrix and projection matrix, and then converted into screen pixel coordinates through linear mapping. This process transforms the high-complexity calculations that rely on three-dimensional spatial relationship judgments in traditional methods into deterministic matrix multiplication and linear interpolation operations. This transformation fundamentally avoids the computational overhead caused by directly performing proximity search or geometric relationship judgments in three-dimensional space, paving the way for efficient collision detection on the two-dimensional plane, thereby effectively solving the performance bottleneck problem in large-scale point cloud data processing.
[0069] As can be seen, in terms of system versatility, the coordinate transformation process is entirely based on standard linear algebra operations, without relying on any specific hardware acceleration instruction set or private computing library. The matrices (view matrix, projection matrix) and parameters (canvas size) used in the transformation process can be obtained and set through standard graphics APIs such as WebGL, ensuring the high consistency and portability of the solution across different browsers and hardware platforms. The design based on general mathematical principles enables the entire point cloud semantic segmentation solution to run seamlessly in various computing environments that support modern Web standards, reducing the threshold and complexity of technology deployment.
[0070] Furthermore, ProjectionMatrix is the projection matrix, defining the mapping from the view frustum to the cube clipping space. For perspective projection, its typical form is as follows:
[0071]
[0072] Using the projection matrix of the current camera view and the inverse of the world matrix (matrixWorldInverse), the coordinates of each point cloud point are combined and transformed (usually by matrix multiplication projectionMatrix *matrixWorldInverse). This transformation projects the point from the three-dimensional world space to a normalized clipping space, and its coordinate range is normalized to the interval [-1, 1] in each dimension.
[0073] The above conversion formula transforms NDC coordinates into screen pixel coordinates, mapping the normalized coordinates to specific pixel positions in the browser viewport (Canvas). After this step, each 3D point obtains a unique 2D screen coordinate representation, providing a basis for subsequent operations on the 2D plane.
[0074] Furthermore, ndcX and ndcY are the X and Y components of the normalized device coordinates, respectively; screenX and screenY are the mapped screen pixel coordinates; and canvasWidth and canvasHeight are the width and height dimensions of the rendering canvas.
[0075] As an optional embodiment, the step described above, which involves using off-screen Canvas technology for accurate judgment, specifically includes the following steps:
[0076] Create an off-screen Canvas element in memory that is the same size as the displayed view;
[0077] Use the CanvasRenderingContext2D API to fill the user-drawn annotation shape onto the off-screen Canvas with a specific color;
[0078] For each point to be judged, the color value of the corresponding pixel on the off-screen Canvas is read using the getImageData method;
[0079] If the color value matches the specific color, then the point is determined to be inside the marked shape.
[0080] In this embodiment of the invention, the off-screen Canvas accurate judgment technology transforms the complex geometric judgment problem of "points inside polygons" into efficient image pixel operations, thereby optimizing computational performance. Traditional algorithms require repeated geometric calculations for each point to be judged and each edge of the polygon, and the computational complexity increases linearly with the number of polygon edges. However, this invention only needs to fill the user-drawn shape with a specific color on the off-screen Canvas in memory once, and then use the getImageData method to perform a pixel color query for each point with constant time complexity, which greatly reduces the computational overhead. This enables real-time processing of large-scale point cloud data in a web browser and effectively overcomes the performance bottleneck caused by direct 3D calculation.
[0081] As can be seen, this technology is based on the standard CanvasRenderingContext2D API, ensuring the universality and compatibility of the solution in the Web environment. Since it does not rely on specific hardware acceleration or external computing libraries, this invention can run stably in any browser that supports the modern HTML5 standard. Combined with the real-time rendering of detection results by the GPU vertex shader, this method provides immediate visual feedback for point cloud semantic segmentation, improves the smoothness and response speed of interactive operations, and provides a reliable technical foundation for complex annotation workflows.
[0082] Furthermore, it is determined whether the converted screen coordinates are located inside the marked shape drawn by the user on the 2D view. To improve detection efficiency, this invention adopts a two-level progressive judgment strategy:
[0083] 1. Quick Pre-Filtering of Bounding Boxes (AABB Culling): Before proceeding with precise calculations, the axis-aligned bounding box (AABB) of the user-drawn shape is first calculated. For a point to be judged, only four simple coordinate comparisons are needed (point.x >= minX && point.x <= maxX && point.y >= minY && point.y <= maxY), which can quickly eliminate a large number of points that are clearly not within the bounding box. This step has extremely low computational cost, yet it can filter out typically 70% of irrelevant points.
[0084] 2. Off-screen Canvas Precise Judgment: For points pre-filtered by bounding boxes, precise position determination is performed. This invention innovatively adopts off-screen Canvas technology: an invisible Canvas element of the same size as the display view is created in memory. On this Canvas, the user-drawn polygon area is filled with a specific unique color (such as red) using the CanvasRenderingContext2D API. Then, for the screen coordinates of each point to be judged, the color value of the corresponding pixel on the off-screen Canvas is read by the getImageData() method. If the color is the preset fill color, the point is determined to be inside the polygon; otherwise, it is outside. This method transforms the complex geometric problem of "point inside polygon" judgment into a one-time polygon drawing and a pixel color query with O(1) complexity, completely avoiding the performance bottleneck of traditional ray casting, angle summation, or vector cross product methods that require repeated calculations for each point and each edge of the polygon. This is especially advantageous when the polygon is complex or the point cloud is dense.
[0085] As an optional embodiment, in the above steps, the dual-attribute field is a property attribute defined for each point;
[0086] The property.x file stores semantic tag IDs, which correspond to a predefined semantic category table.
[0087] The property.y file stores the display status code, which is used to control the rendering behavior of the point.
[0088] In this embodiment of the invention, the dual-attribute field design uses a two-dimensional vector structure to store the attribute information of each point, encoding semantic labels and display states in a compact form. The property.x field records the semantic category in the form of an integer index, and the property.y field independently controls the rendering state. This design reduces the data storage overhead of a single point. By directly reading and parsing these attribute values through the GPU vertex shader, the color, size, and transparency of the points are dynamically calculated, and the rendering logic is transferred to the parallel processing of the graphics hardware. This ensures that real-time and smooth visual feedback can still be achieved when facing millions of point clouds, effectively improving the efficiency of data processing and the system response speed.
[0089] As can be seen, the decoupling design of semantic information and display status supports complex annotation workflows and interactive control. Users can independently operate semantic label allocation and visual representation. For example, by setting property.y to a highlight status code, the selected area can be highlighted in real time, or the background can be faded with a semi-transparent status code to focus on the key points. This meets the needs of multi-round iterative annotation, result comparison and error correction. This flexible attribute management mechanism reserves interfaces for expanding more semantic categories or interactive states, enhancing the system's functionality and scalability.
[0090] Furthermore, to achieve the storage and real-time visualization of the segmentation results, this invention designs a compact and efficient attribute storage structure for each point in the point cloud, and accelerates rendering using a GPU:
[0091] 1. Dual-attribute field design: In the point cloud data structure, a property field is defined for each point. The two floating-point numbers in this two-dimensional vector encode different information:
[0092] property.x: Stores the semantic tag ID, which is an integer index value corresponding to a predefined semantic category table.
[0093] Tag ID Semantic categories describe 0 Unmarked Unlabeled origin points 1 vehicle Various types of cars, trucks, etc. 2 pedestrian pedestrian 3 building Houses and buildings 4 vegetation Trees, grassland 5 ground Roads and sidewalks
[0094] `property.y`: Stores the display status code, used to control the rendering behavior of this point in the current view. Multiple states can be defined:
[0095] status codes meaning 0 Default display 1 hide 2 Completely transparent 3 Highlight 4 translucent
[0096] 2. Real-time GPU Shader Rendering: In WebGL's vertex shader, the program reads the property attribute of each point and dynamically calculates the final output color, size, and transparency of that point based on its value. For example, when property.y == 3 is detected, the shader ignores the original color, directly outputs red, and increases gl_PointSize to achieve a highlight effect; when property.y == 4, the transparency of the point's color is reduced. This method of transferring the state judgment and rendering logic of a large number of points to the GPU for parallel execution fully utilizes the parallel computing capabilities of graphics hardware, ensuring that visual updates remain smooth and real-time even when facing millions of point clouds.
[0097] As an optional embodiment, the above steps also include a frame-overlay annotation mode for synchronous annotation of time-series point clouds, specifically including:
[0098] Align multiple frame point clouds to a unified reference coordinate system;
[0099] Based on the labeled shapes drawn by the user on the reference frame, collision detection and semantic information storage are performed synchronously on the point clouds of all frames.
[0100] In this embodiment of the invention, the frame-overlapping annotation mode transforms the annotation operation of dynamic point cloud sequences from repetitive labor into a one-time batch processing task through a unified reference coordinate system and a synchronous processing mechanism. This mode pre-aligns multiple frames of point clouds to a unified coordinate system. After the user draws the annotation shape on the reference frame view, the system, based on the same view and projection matrix, transforms each frame of point cloud to the same screen coordinate system and performs collision detection and semantic label allocation in parallel. This process reduces the N interactions required for traditional frame-by-frame annotation to one interaction plus one batch calculation, avoiding repeated drawing operations for the same moving target on each frame. This improves the overall efficiency of time-series data annotation by an order of magnitude while reducing the operator's workload.
[0101] As can be seen, this model ensures the consistency of annotation results in the spatiotemporal dimension through strict mathematical transformation. The point clouds of all frames are projected and judged based on the same set of coordinate transformation parameters, ensuring that the three-dimensional spatial position of the annotated area of the same physical entity in different frames strictly corresponds. This effectively eliminates problems such as position offset and shape inconsistency caused by manual frame-by-frame operation. This consistency guaranteed by the algorithm provides a more reliable data foundation for applications such as autonomous driving model training that require high-precision temporal annotation data, and improves the quality and usability of the annotation data. This processing flow is a key component of the overall method.
[0102] Furthermore, this mode aims to address the need for efficient and consistent annotation of time-series point clouds (such as continuous frames of LiDAR scanning). Its core idea is "one-time drawing, multiple frames take effect simultaneously". Through coordinate system unification and batch projection, the annotation operation of the user on the reference frame is synchronized in real time to multiple aligned other frames.
[0103] 1. Core principle: Alignment of point cloud coordinates across multiple frames;
[0104] A unified reference coordinate system is established, typically chosen as the base coordinate system for the first frame (Frame_0). Point clouds from all other frames are then transformed to this unified coordinate system. Alignment is performed during point cloud preprocessing.
[0105] 2. Implementation process: Synchronous collision detection and attribute update;
[0106] For the same 2D polygon selection drawn by the user on the reference frame view, the screen coordinates obtained from the point cloud transformation of each frame are used to perform collision detection in parallel (unified AABB pre-screening and off-screen canvas precise judgment), and the annotation results are stored. During rendering, only the segmentation data of the current frame is rendered. For points that pass the detection in other frames, the rendering is updated based on the semantic information in the above steps when the frame is sliced.
[0107] As an optional embodiment, in the above steps, the annotation shape includes shapes generated by the lasso tool, polygon tool, rectangle tool, or point tool;
[0108] The method also supports overwrite, non-overwrite, update, or delete annotation processing modes.
[0109] In this embodiment of the invention, the diverse annotation toolset provides lasso, polygon, rectangle, and point tools, enabling annotation operations to adapt to target objects with different geometric shapes. The lasso tool allows users to freely draw irregular boundaries, suitable for objects with complex contours; the polygon tool defines vertices through continuous clicks, suitable for annotating structures with sharp edges; the rectangle tool enables quick selection of bounding boxes, for approximately rectangular targets; and the point tool is used to precisely select discrete points. This combination of tools ensures that users can choose the most suitable interaction method based on the actual shape characteristics of objects in the point cloud, directly reducing operational complexity and learning costs, and improving annotation efficiency.
[0110] As can be seen, the four annotation processing modes supported—overwrite, non-overwrite, update, and delete—provide fine-grained control over multi-round iterative annotation workflows. Overwrite mode completely replaces old annotations; non-overwrite mode ensures existing annotations are not destroyed by new operations and is suitable for incremental annotation; update mode allows batch modifications to specific categories; and delete mode is used to erase erroneous annotations. These modes enable users to flexibly manage the logical relationships between new and old annotations, effectively avoiding annotation conflicts and data chaos, and supporting complex annotation processes from coarse to fine, with annotation and modification occurring simultaneously, thereby improving the fault tolerance of the annotation process and the accuracy of the final results.
[0111] Furthermore, the application of coverage patterns:
[0112] How it works: When a user adds a new annotation to an already annotated area, regardless of the old annotation, the semantic labels (property.x) and display status (property.y) of all points in that area will be completely replaced by the content of the new annotation.
[0113] Application scenarios: Used to correct incorrect annotations or redefine object boundaries. For example, an area originally labeled "vegetation" is actually a car obscured by leaves. Users can use this mode to directly draw a new area to cover it as "vehicle".
[0114] Applications of non-overwrite mode:
[0115] How it works: The new annotation only applies to points that have not yet been annotated (i.e., "unannotated" points with property.x = 0). For points that have already been annotated (property.x ≠ 0), the new annotation will ignore them and retain their original semantic labels and states.
[0116] Application scenario: Used for incremental annotation to avoid destroying existing results. In complex scenarios, users can first annotate all "vehicles" and then switch to "non-overlapping" mode to annotate "pedestrians". This way, even if the drawn area overlaps with the vehicle, the vehicle points will not be mistakenly changed to pedestrians.
[0117] Application of update mode:
[0118] How it works: This mode is typically used to modify the labels of objects of a specific category. The user needs to specify a target semantic label (e.g., "vehicle") before drawing. The system only updates the labels of points within the drawing area whose current semantic label matches the target label to the new semantic label.
[0119] Application scenario: Used for batch correction of errors of the same category. For example, if a batch of "buildings" are found to be incorrectly labeled as "vegetation", users can select the "update" mode, set the target label to "vegetation" and the new label to "buildings", and then draw in the incorrectly labeled area to quickly complete the batch correction without affecting the areas that are actually "vegetation".
[0120] Application of deletion mode:
[0121] Working method: Reset the semantic labels of all points in the user-drawn area to "unlabeled" (i.e., property.x = 0) and restore the display state to the default (property.y = 0).
[0122] Application scenario: Used to erase incorrect or unwanted annotations, which is one of the most common correction operations in the annotation process.
[0123] As an optional embodiment, the above steps also include point cloud picking optimization, using ray casting:
[0124] Calculate the 3D ray in reverse based on the screen coordinates of the mouse click, find the intersection of the ray with the point cloud, select the nearest point, and highlight the corresponding segmented object.
[0125] In this embodiment of the invention, the ray casting method calculates a three-dimensional ray from the screen coordinates in reverse and performs geometric intersection, achieving accurate and efficient selection of specific objects in a three-dimensional point cloud. The invention calculates a three-dimensional ray from the viewpoint through the click position based on the two-dimensional screen coordinates of the mouse click and camera parameters. Then, the nearest intersection point is quickly located through the intersection algorithm of the ray and the point cloud. This mathematical geometry-based picking mechanism avoids the high overhead of global search or complex neighborhood calculation in dense point clouds, directly and accurately selects the target object, and improves the picking accuracy when processing large-scale point clouds in the Web environment.
[0126] As can be seen, this invention enhances the intuitiveness and responsiveness of the interaction through real-time visual feedback. Once the nearest point is selected, the system associates it with the corresponding segmented object based on its index and highlights the bounding box of the object. At the same time, by setting attribute fields to reduce the transparency of other point clouds, this real-time state change provides users with clear operation confirmation, making the selection and editing of objects in the 3D scene more intuitive and controllable, and effectively improving the overall experience of 3D data interaction on a 2D interface.
[0127] Furthermore, the present invention includes:
[0128] 1. Diverse Annotation Tools and Modes: Provides four annotation tools: lasso, polygon, rectangle, and point, to accommodate the segmentation needs of objects with different shapes. It also offers four annotation processing modes: cover, don't cover, update, and delete, allowing users to flexibly manage the relationship between old and new annotations and meet the needs of multi-round annotation and correction in complex scenarios.
[0129] 2. Efficient Point Cloud Picking Scheme (Ray Projection Method): When a user needs to select a segmented object for subsequent operations (such as viewing, editing, or deleting), this invention uses the ray projection method for efficient picking. The specific process is as follows: Based on the screen coordinates of the mouse click, a 3D ray originating from the camera and passing through the click position is calculated in reverse using camera parameters; using the geometric intersection algorithm between the ray and the point cloud, the point closest to the ray is quickly calculated and the index of that point is returned; then, the corresponding segmented object is found based on the index, the bounding box of the segmented object is highlighted, and the transparency of other point clouds is reduced (property.y is set to 4).
[0130] As an optional embodiment, the ray projection method in the above steps specifically includes:
[0131] The calculated 3D ray is geometrically intersected with the point cloud, and the index of the nearest point intersecting the ray is returned. After finding the corresponding segmented object based on the index, the bounding box of the object is highlighted, and the opacity of other point clouds is reduced.
[0132] In this embodiment of the invention, the ray casting method optimizes the accuracy and efficiency of point cloud object selection through precise geometric calculations. The invention calculates a three-dimensional spatial ray based on the screen coordinates of mouse clicks and quickly locates the nearest intersection point index through a geometric intersection algorithm between the ray and the point cloud data. This picking mechanism based on a mathematical model avoids the high overhead of global search or complex neighborhood calculation in dense point clouds, and directly determines the target position through geometric relationships, thereby improving the calculation accuracy and response speed of interactive selection of specific point cloud objects in a Web environment.
[0133] As can be seen, this invention enhances the intuitiveness and user experience of 3D data operations through an instant visual feedback mechanism. The system associates the returned point index with the corresponding segmented object and automatically highlights the 3D bounding box of the object. At the same time, it reduces the transparency of other point clouds by adjusting attribute fields. This instant visual state change provides users with clear operation confirmation, making object selection and subsequent editing operations in the 3D scene more intuitive and controllable, and effectively improving the overall operation efficiency of 3D data interaction on a 2D interface.
[0134] Example 2
[0135] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a web-based 3D point cloud semantic segmentation system disclosed in an embodiment of the present invention. Figure 2 The web-based 3D point cloud semantic segmentation system described herein can be applied to data processing chips, processing terminals, or processing servers, and the processing server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 2 As shown, this web-based 3D point cloud semantic segmentation system can include the following operations:
[0136] The coordinate transformation module 201 is used to transform the coordinates in the world coordinate system of the 3D point cloud to the 2D screen coordinate system through the camera view matrix and projection matrix.
[0137] Specifically, this invention transforms complex geometric calculations traditionally performed in three-dimensional space into operations in two-dimensional screen space. In particular, the calculation process is simplified from the high-complexity problem of dealing with the relationships between points and points and complex three-dimensional shapes in three-dimensional space to coordinate mapping and pixel-level judgment on a two-dimensional plane. This transformation, combined with a two-level strategy of fast pre-screening with axis-aligned bounding boxes and accurate judgment on off-screen Canvas, significantly reduces the computational overhead of a single judgment, making it possible to process large-scale point cloud data in real time in a standard web browser environment, effectively overcoming the bottleneck of limited computing resources on the web.
[0138] As can be seen, the coordinate projection-based process relies entirely on the standard WebGL and JavaScript technology stack, without the need for additional plugins or specific computing frameworks. This not only gives the system a lightweight architecture but also ensures its good compatibility and portability in various modern web browsers. At the same time, this process provides a unified data foundation for subsequent intuitive interactions based on two-dimensional screen coordinates (such as lasso selection) and provides technical support for a smooth interactive experience through real-time rendering via GPU vertex shaders.
[0139] The collision detection module 202 uses a two-level progressive collision detection strategy to filter the converted screen coordinate points based on the marked shapes drawn by the user on the two-dimensional screen.
[0140] Specifically, this collision detection strategy optimizes the computation process through a two-stage progressive processing mechanism. The first stage uses axis-aligned bounding boxes of labeled shapes for rapid pre-screening. A large number of obviously irrelevant screen coordinate points can be eliminated through four simple coordinate comparisons, reducing the computational base for subsequent accurate judgment. The second stage uses off-screen Canvas technology to fill the user-drawn polygonal region in the memory Canvas with a specific color. Geometric position is determined by querying the pixel color value of the screen coordinates corresponding to the query point. This transforms the high-complexity point judgment within the polygon in traditional algorithms into a one-time color filling and a pixel query operation with constant time complexity. This combined strategy effectively solves the performance bottleneck caused by directly performing geometric calculations when the amount of 3D point cloud data is huge.
[0141] As can be seen, this strategy fully utilizes Web standard graphics APIs to achieve efficient computation, avoiding dependence on dedicated computing hardware. Off-screen Canvas technology is based on the browser's natively supported CanvasRenderingContext2D interface, and completes accurate collision detection through color filling and pixel query, which not only ensures the accuracy of the judgment, but also ensures that the solution has good compatibility in various browsers that support modern Web standards. Combined with the real-time rendering of the detection results by the GPU vertex shader, this invention can still maintain the instant response of interactive operations in a resource-limited environment, and provide a smooth visual feedback experience for point cloud annotation in complex scenes.
[0142] The rendering module 203 is used to assign semantic labels to points that pass collision detection and store the semantic labels and display status codes in the dual attribute field of each point. The GPU vertex shader renders the data in real time to visualize the segmentation results.
[0143] Specifically, by designing dual-attribute fields for each point to store semantic labels and display status codes, high efficiency in data storage and flexibility in rendering control are achieved. The property.x field encodes semantic categories in the form of integer indices, while the property.y field independently controls the rendering status of the point. This compact two-dimensional vector structure minimizes storage overhead while decoupling semantic information from display logic. These attributes are directly read and parsed by the GPU vertex shader, and the color, size, and transparency of the point are dynamically calculated. The status judgment and rendering tasks of a large number of points are transferred to the graphics hardware for parallel processing, ensuring that visual updates remain smooth in real time even when facing millions of point clouds.
[0144] As can be seen, this invention supports complex annotation workflows and interactive feedback. For example, during the annotation process, the selected area can be highlighted instantly by setting property.y to a high-brightness status code, or the background can be faded by a semi-transparent status code to focus on the key points, providing users with clear visual confirmation. The independent control of semantic tags and display status allows for multiple rounds of iterative annotation, result comparison, and error correction, improving the accuracy and functionality of annotation operations. At the same time, it reserves flexible interfaces for future expansion of more semantic categories or interactive states.
[0145] Example 3
[0146] Please see Figure 3 , Figure 3 This is a schematic diagram of another web-based 3D point cloud semantic segmentation system disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include:
[0147] Memory 301 storing executable program code;
[0148] Processor 302 coupled to memory 301;
[0149] The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in the web-based 3D point cloud semantic segmentation method disclosed in Embodiment 1 of the present invention.
[0150] Example 4
[0151] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in the web-based 3D point cloud semantic segmentation method disclosed in Embodiment 1 of this invention.
[0152] Example 5
[0153] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of a web-based 3D point cloud semantic segmentation method described in Embodiment 1.
[0154] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0155] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0156] Finally, it should be noted that the web-based 3D point cloud semantic segmentation method and system disclosed in the embodiments of this invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention.
Claims
1. A web-based 3D point cloud semantic segmentation method, characterized in that, The method includes the following steps: Transform the coordinates of the 3D point cloud in the world coordinate system to the 2D screen coordinate system using the camera view matrix and projection matrix; Based on the marked shapes drawn by the user on the two-dimensional screen, a two-level progressive collision detection strategy is adopted to filter the converted screen coordinate points. The two-level progressive collision detection strategy includes: firstly, using the axis alignment bounding box of the marked shape for fast pre-screening, and then using off-screen Canvas technology to accurately judge the pre-screened points in order to filter out the points located inside the marked shape. Semantic labels are assigned to points that pass collision detection, and the semantic labels and display status codes are stored together in the dual-attribute field of each point. The segmentation results are visualized by real-time rendering through the GPU vertex shader.
2. The web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, The process of transforming the coordinates of the 3D point cloud in the world coordinate system to the 2D screen coordinate system through the camera view matrix and projection matrix includes: The coordinates in the world coordinate system are transformed to the normalized device coordinate space using the view matrix and projection matrix, where the normalized device coordinates are obtained through the formula... calculate, Let the world coordinates of the point be... For camera view matrix, The projection matrix; Map normalized device coordinates to screen pixel coordinates; Screen X coordinate through calculate; Screen Y coordinate through calculate; in, and This refers to the canvas size.
3. The web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, The precise determination using off-screen Canvas technology specifically includes the following steps: Create an off-screen Canvas element in memory that is the same size as the displayed view; Use the CanvasRenderingContext2D API to fill the user-drawn annotation shape onto the off-screen Canvas with a specific color; For each point to be judged, the color value of the corresponding pixel on the off-screen Canvas is read using the getImageData method; If the color value matches the specific color, then the point is determined to be inside the marked shape.
4. The web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, The dual-attribute field defines a property attribute for each point; The property.x file stores semantic tag IDs, which correspond to a predefined semantic category table. The property.y file stores the display status code, which is used to control the rendering behavior of the point.
5. A web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, It also includes a frame-overlay annotation mode for synchronous annotation of time-series point clouds, specifically including: Align multiple frame point clouds to a unified reference coordinate system; Based on the labeled shapes drawn by the user on the reference frame, collision detection and semantic information storage are performed synchronously on the point clouds of all frames.
6. The web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, The labeled shape includes shapes generated by the lasso tool, polygon tool, rectangle tool, or point tool; The method also supports overwrite, non-overwrite, update, or delete annotation processing modes.
7. The web-based 3D point cloud semantic segmentation method according to claim 1, characterized in that, It also includes point cloud picking optimization, using ray casting: Calculate the 3D ray in reverse based on the screen coordinates of the mouse click, find the intersection of the ray with the point cloud, select the nearest point, and highlight the corresponding segmented object.
8. A web-based 3D point cloud semantic segmentation method according to claim 7, characterized in that, The ray projection method specifically includes: The calculated 3D ray is geometrically intersected with the point cloud, and the index of the nearest point intersecting the ray is returned. After finding the corresponding segmented object based on the index, the bounding box of the object is highlighted, and the opacity of other point clouds is reduced.
9. A web-based 3D point cloud semantic segmentation system, used to implement the web-based 3D point cloud semantic segmentation method according to any one of claims 1-8, characterized in that, The system includes: The coordinate transformation module is used to transform the coordinates of a 3D point cloud in the world coordinate system to a 2D screen coordinate system through the camera view matrix and projection matrix. The collision detection module uses a two-level progressive collision detection strategy to filter the converted screen coordinate points based on the marked shapes drawn by the user on the two-dimensional screen. The rendering module is used to assign semantic labels to points that pass collision detection and store the semantic labels and display status codes in the dual-attribute field of each point. The GPU vertex shader renders the data in real time to visualize the segmentation results.
10. A web-based 3D point cloud semantic segmentation system, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute a web-based 3D point cloud semantic segmentation method as described in any one of claims 1-8.