Crowdsourcing method for generating point cloud semantics, medium, device and product
By combining IMU pre-integration with point cloud registration, and integrating image semantic segmentation and localization data, a global point cloud semantic map is constructed. This solves the problems of high cost and insufficient consistency in point cloud semantic annotation, and achieves efficient and accurate semantic map generation and updating.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JISHU TECHNOLOGY (WUHAN) CO LTD
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-28
AI Technical Summary
In existing technologies, point cloud semantic annotation relies on professional annotators, resulting in high costs and long cycles. Furthermore, the annotation results under the crowdsourcing model vary greatly in quality and lack semantic consistency, making it difficult to achieve large-scale, wide-ranging semantic map construction.
A local map is constructed by combining IMU pre-integration with point cloud registration. Based on the location data, it is unified to the absolute world coordinate system. The image semantic segmentation algorithm is used to obtain pixel semantic labels and map them to point cloud data. The semantic initial values are assigned by combining the number of observations and angle selection. A four-dimensional confidence model is used for dynamic updating to realize the semantic generation and automatic labeling of the global map.
It improves the accuracy and consistency of semantic annotation, reduces annotation costs, broadens scenario coverage, improves data processing efficiency, and achieves adaptive optimization of stable semantic regions.
Smart Images
Figure CN122473374A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of map semantic generation technology, and in particular to point cloud semantic generation methods, media, devices and products using a crowdsourcing approach. Background Technology
[0002] 3D point cloud semantic maps serve as a core foundation for fields such as autonomous driving, drone inspection, and robot navigation. The efficiency, cost, and accuracy of their generation and updating directly impact the practical application of related technologies. Current point cloud semantic annotation services primarily rely on manual annotation by professional annotators, which has the following significant drawbacks: First, the training cost of professional annotators is high, resulting in expensive labor costs and a persistently high overall cost for the annotation business. Second, the manual annotation process is cumbersome, with lengthy project cycles, making it difficult to meet the needs of rapidly iterating business processes. Third, limited by the scope and time available to annotators, the annotation scenario coverage is limited, hindering the construction of large-scale, wide-ranging semantic maps.
[0003] To address these issues, some solutions have attempted to utilize crowdsourcing for annotation. However, this approach faces new technical bottlenecks: the varying levels of expertise among crowdsourcing participants lead to significant differences in annotation quality and insufficient semantic consistency; the lack of an effective semantic fusion and update mechanism makes it difficult to efficiently integrate multi-source crowdsourced annotation data into a unified and accurate global semantic map; furthermore, duplicate annotations and invalid data consume substantial computing resources, resulting in low data processing efficiency, which further hinders the application and promotion of crowdsourcing in the field of point cloud semantic annotation.
[0004] Therefore, there is an urgent need for a point cloud semantic generation and update method that can reduce annotation costs, broaden scene coverage, and improve semantic consistency and data processing efficiency to address the shortcomings of existing technologies. Summary of the Invention
[0005] The purpose of this invention is to address the problem of integrating multi-source crowdsourced labeled data into a global semantic map, and to propose a crowdsourced point cloud semantic generation method, comprising the following steps: S1. Obtain point cloud data, image data, IMU data and positioning data from crowdsourced data collected from multiple mobile acquisition sources, and construct multiple local maps by combining IMU pre-integration and point cloud registration. S2. Based on the positioning data, the data collected by multiple mobile devices are unified into the absolute world coordinate system. Under the absolute world coordinate system, the pose of each local map is optimized, and the optimized local maps are stitched together into a global map. S3. Obtain the pixel semantic labels of the image data and map the pixel semantic labels to the corresponding single-frame point cloud data; S4. Using the localization information of each frame of point cloud and the pose relationship of the local map, the semantic information of the single frame of point cloud is projected onto the local map to obtain the local semantic map. S5. Project the local semantic map onto the global map so that each point in the global map contains multi-dimensional information including multiple observation angles, multiple observation times, and multiple observation semantics. S6. When a target point in the global map meets the preset conditions, assign an initial semantic value to the target point; the preset conditions are: the number of observations of the target point's semantics is ≥ a preset number threshold, and the observation angle is ≥ a preset angle threshold. S7. For points with initial semantic values in the global map, if they are observed by a new data source, update the observation angle, observation count, and observation semantics. At the same time, record the number of semantic changes and the number of updates. Calculate the semantic confidence of the points in the global map based on the observation angle, observation count, number of semantic changes, and number of updates. When the confidence value is above the set minimum limit, stop updating the semantics of the points and obtain a stable semantic region. S8. When new crowdsourced data contains stable semantic regions, the point cloud semantics of the new crowdsourced data are assigned values through matching to achieve automatic generation of point cloud semantics.
[0006] Furthermore, multiple local maps are constructed by combining IMU pre-integration with point cloud registration.
[0007] Furthermore, the feature-matching-based point cloud registration algorithm optimizes the pose of each local map.
[0008] Furthermore, S3 specifically refers to: Image semantic segmentation algorithm is used to obtain pixel semantic labels of image data, depth estimation algorithm is used to obtain pixel depth information, and combined with the intrinsic and extrinsic parameter calibration parameters of the acquisition device, the pixel semantic labels are mapped to the corresponding point cloud data within the depth limit range, thus completing the semantic assignment of a single frame point cloud.
[0009] Furthermore, the preset conditions mentioned in S6 are: the number of observations of the target point semantics is greater than or equal to a preset number threshold, and the observation angle is greater than or equal to a preset angle threshold; Furthermore, the formula for calculating semantic confidence is:
[0010] Where C represents semantic confidence and N represents the number of current observations. The maximum number of observations is preset; θ is the observation angle. The maximum preset angle is K; K represents the number of semantic changes. This represents the maximum allowed number of changes; U represents the number of updates. The maximum number of updates is preset; α, β, γ and δ are weight coefficients, and satisfy α+β+γ+δ=1.
[0011] The present invention also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0012] The present invention also proposes an electronic device, including a processor and a memory, wherein the processor is interconnected with the memory, the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described method.
[0013] The present invention also proposes a computer program product, including a computer program / instructions that, when executed by a processor, implement the above-described method.
[0014] The beneficial effects of the technical solution provided by this invention are: (1) This invention stitches local maps into a global map, combines IMU data and GNSS positioning to achieve accurate fusion of multi-source data, and assigns semantic initial values through dual screening of observation number and observation angle, thereby improving the reliability of semantic initial values; (2) A four-dimensional confidence calculation model including the number of observations, observation angle, number of semantic changes, and number of updates is proposed. By dynamically updating the confidence and setting a stable threshold, the adaptive optimization and locking of semantics are realized, which effectively solves the problem of insufficient semantic consistency in the crowdsourcing mode and improves the accuracy of semantic annotation. Attached Figure Description
[0015] Figure 1 This is a flowchart of a crowdsourcing-based point cloud semantic generation method according to an example of the present invention; Figure 2 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0017] A flowchart of a crowdsourcing-based point cloud semantic generation method according to an example of the present invention is shown below. Figure 1 Specifically, it includes: S1. Obtain point cloud data, image data, IMU data, and positioning data from crowdsourced data collected from multiple mobile data sources. Construct multiple local maps by combining IMU pre-integration with point cloud registration. The mobile data sources can be data collection vehicles, drones, or mobile robots.
[0018] The specific steps for constructing multiple local maps are as follows: (1) Continuously collect IMU data measurements of angular velocity and acceleration and point cloud data. The frequency of IMU data is usually between 100Hz and 1000Hz. The data contains serious noise and zero bias, which needs to be adjusted. The frequency of point cloud data is lower (e.g., 10Hz), and distortion correction is required because the lidar itself is also moving during the scanning process (using IMU data or estimated motion can help with distortion correction).
[0019] (2) Based on certain strategies (such as when the movement distance exceeds the threshold, the rotation angle exceeds the threshold, or the scene features change significantly), the current point cloud frame is marked as a key frame. The key frame is the basic unit for building the map and performing optimization.
[0020] (3) Calculate the key frame times of two consecutive frames. arrive The relative motion increment, extract the timestamp in [ All raw IMU data between [a certain point in time] are integrated based on the IMU kinematic model. The integration process involves three key quantities: Relative rotation increment: from arrive Posture changes; Relative velocity increment: in The velocity changes observed in the corresponding frame coordinate system; Relative position increment: in The positional changes observed in the corresponding frame coordinate system.
[0021] (4) For a new keyframe, its point cloud is registered with the point cloud of the previous keyframe. Common algorithms include feature-based matching methods (such as edge point and planar point matching in LOAM). The point cloud registration algorithm outputs a pose transformation (relative to the previous keyframe) and forms a point cloud matching constraint. This constraint measures the alignment error between the point cloud and the map under the current estimated pose (such as the sum of the distances from a point to a surface or from a point to a line).
[0022] (5) Construct a factor graph. The IMU pre-integration factor connects adjacent keyframes, constraining the relative motion between these two keyframes to be consistent with the IMU pre-integration result, while also considering the change in zero bias. The lidar matching factor connects the keyframes and the local map, constraining the pose of the keyframes to ensure the best alignment between their point cloud and the local map. Solve this factor graph using a nonlinear optimization library to minimize the sum of the residuals (i.e., errors) of all factors. After optimization, the optimal and consistent poses and states of all keyframes are obtained.
[0023] (6) Use the optimized keyframe poses to construct a local map.
[0024] S2. Based on positioning data (such as GNSS positioning data), the data collected by multiple mobile devices are unified into the absolute world coordinate system. Under the absolute world coordinate system, a point cloud registration algorithm based on feature matching is used to optimize the pose between various local maps, eliminate the cumulative error between local maps, and stitch the optimized local maps into a global map.
[0025] S3. Obtain the pixel semantic labels of the image data and map the pixel semantic labels to the corresponding single-frame point cloud data.
[0026] Each acquisition source's single pass of data contains image files and point cloud files of the same frame. Image semantic segmentation algorithms are used to obtain pixel semantic labels for image data, and depth estimation algorithms are used to obtain pixel depth information. The purpose of depth estimation is mainly to prevent image semantic misprojection caused by density differences between point clouds and images. Combined with the intrinsic and extrinsic calibration parameters of the acquisition device (such as a camera), pixel semantic labels are mapped to the corresponding point cloud data within the depth limit range, thus completing the semantic assignment of a single frame point cloud.
[0027] In this embodiment of the invention, the image semantic segmentation algorithm is the InternImage framework, and the depth estimation algorithm is the Metric3D algorithm.
[0028] S4. The 3D point cloud of the current frame, where each point, in addition to its coordinates, also has a semantic label (e.g., vehicle, pedestrian, road, building, vegetation, etc.). Using the pose of the current frame, transform all point clouds of the frame from the sensor coordinate system to the local map coordinate system. For each point in the current frame, find one or more points / voxels in the local map with the closest Euclidean distance, and project the semantic information of the single-frame point cloud onto the corresponding one or more points / voxels in the local map to obtain the local semantic map.
[0029] S5. Project the local semantic map onto the global map. Since the local map is constructed from multiple frames of point cloud data, each point in the local map will be observed by multiple frames of point cloud. Therefore, each point in the global map contains multi-dimensional information including multiple observation angles, multiple observation times, and multiple observation semantics.
[0030] The observation angle refers to the angle or direction of a sensor (such as a camera or lidar) relative to that point or the world coordinate system when acquiring data at that point. This helps in understanding from which perspective the information about that point was obtained and can be used for the fusion of multi-view information and 3D reconstruction.
[0031] The observation count refers to the number of times a point has been successfully detected or recorded by a sensor. This value reflects the stability and reliability of the information about that point. A higher observation count usually indicates that the point's location is static and persistent, rather than transient noise.
[0032] Observational semantics refers to the understanding of the category of the object or region to which the point belongs. For example, it might be label information such as "road," "building," "pedestrian," or "vehicle," usually derived from image semantic segmentation or object detection models. This endows maps with a high level of environmental understanding.
[0033] The combination of these three elements can create a rich, stable, and understandable environmental model.
[0034] S6. For each point in the global map, count all associated observation semantics and the corresponding number of observations and observation angle coverage. When a target point in the global map meets the preset conditions, assign an initial semantic value to the target point. The preset conditions are: the number of observations of the target point semantics is ≥ the preset number threshold, and the observation angle is ≥ the preset angle threshold. The preset number threshold is 3-5 times. Divide the observation angle into intervals. When the interval span of the angle exceeds 120 degrees, the angle coverage requirement is met.
[0035] S7. For points with initial semantic values in the global map, if they are observed by a new data source, update the observation angle, observation count, and observation semantics accordingly. At the same time, record the number of semantic changes and the number of updates. Calculate the semantic confidence of points in the global map based on the observation angle, observation count, number of semantic changes, and number of updates. When the confidence value is above the set minimum limit, stop updating the semantics of the points and obtain a stable semantic region.
[0036] The semantic change count refers to the total number of times the semantic meaning of a point changes from its initial value to the current observed semantic meaning. If the observed semantic meaning is different in two adjacent updates, the semantic change count is incremented by 1. If the observed semantic meaning remains the same for n updates, the change count is 0. The update count is incremented by 1 for each new observation received from a new data source.
[0037] The formula for calculating semantic confidence is:
[0038] Where C represents semantic confidence and N represents the number of current observations. The maximum number of observations is preset; θ is the observation angle. The maximum preset angle is K; K represents the number of semantic changes. This represents the maximum allowed number of changes; U represents the number of updates. The maximum number of updates is preset; α, β, γ, and δ are weight coefficients, and satisfy α + β + γ + δ = 1. In this embodiment of the invention, The value is 180°. The value is taken 2-3 times. The values are 5-8, α, β, δ [0.2,0.4], γ [0.1,0.3]; The minimum confidence level is limited to 0.7-0.85. When the calculated confidence level C ≥ the minimum confidence level limit, the semantics of the point are considered to be correct and stable, the semantics of the point are locked, and it will not be updated in subsequent collection processes.
[0039] S8. When new crowdsourced data contains stable semantic regions, the point cloud semantics of the new crowdsourced data are assigned values through matching. As the amount of crowdsourced data continues to increase, new data sources will continuously cover unlabeled areas, causing the global map to expand continuously. At the same time, the semantics of the covered areas tend to stabilize through multiple rounds of iterative updates. When new data passes through semantically stable areas in the global map again, the newly collected point cloud data is matched with stable semantic regions in the global map through a point cloud registration algorithm. The semantic labels of the successfully matched areas are directly reused, without having to repeat the image semantic segmentation and depth estimation process, thus achieving automated annotation of point cloud semantics.
[0040] CUDA operators are used to accelerate the processes of local map construction, global map stitching, semantic projection, and confidence calculation. Parallel computing is used to improve data processing efficiency and shorten the cycle of semantic map generation and updating.
[0041] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the method described above.
[0042] Please see Figure 2 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.
[0043] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through a communication bus to execute the above method.
[0044] In one exemplary embodiment, a computer program product is proposed, including a computer program / instructions that, when executed by a processor, implement the method described above.
[0045] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A crowdsourcing-based method for generating point cloud semantics, characterized in that, Includes the following steps: S1. Obtain point cloud data, image data, IMU data, and positioning data from crowdsourced data collected from multiple mobile data sources, and construct multiple local maps; S2. Based on the positioning data, the data collected by multiple mobile devices are unified into the absolute world coordinate system. Under the absolute world coordinate system, the pose of each local map is optimized, and the optimized local maps are stitched together into a global map. S3. Obtain the pixel semantic labels of the image data and map the pixel semantic labels to the corresponding single-frame point cloud data; S4. Using the localization information of each frame of point cloud and the pose relationship of the local map, the semantic information of the single frame of point cloud is projected onto the local map to obtain the local semantic map. S5. Project the local semantic map onto the global map so that each point in the global map contains multi-dimensional information including multiple observation angles, multiple observation times, and multiple observation semantics. S6. When the number of observations and the observation angle of the target point semantics in the global map meet the preset conditions, assign the target point an initial semantic value; S7. For points with initial semantic values in the global map, if they are observed by a new data source, update the observation angle, observation count, and observation semantics. At the same time, record the number of semantic changes and the number of updates. Calculate the semantic confidence of the points in the global map based on the observation angle, observation count, number of semantic changes, and number of updates. When the confidence value is above the set minimum limit, stop updating the semantics of the points and obtain a stable semantic region. S8. When new crowdsourced data contains stable semantic regions, the point cloud semantics of the new crowdsourced data are assigned values through matching to achieve automatic generation of point cloud semantics.
2. The point cloud semantic generation method based on crowdsourcing according to claim 1, characterized in that, Multiple local maps are constructed by combining IMU pre-integration with point cloud registration.
3. The point cloud semantic generation method based on crowdsourcing according to claim 1, characterized in that, The feature-matching-based point cloud registration algorithm optimizes the pose of each local map.
4. The point cloud semantic generation method based on crowdsourcing according to claim 1, characterized in that, S3 specifically refers to: Image semantic segmentation algorithm is used to obtain pixel semantic labels of image data, depth estimation algorithm is used to obtain pixel depth information, and combined with the intrinsic and extrinsic parameter calibration parameters of the acquisition device, the pixel semantic labels are mapped to the corresponding point cloud data within the depth limit range, thus completing the semantic assignment of a single frame point cloud.
5. The point cloud semantic generation method based on crowdsourcing according to claim 1, characterized in that, The preset conditions mentioned in S6 are: the number of observations of the target point semantics is greater than or equal to a preset number threshold, and the observation angle is greater than or equal to a preset angle threshold.
6. The point cloud semantic generation method based on crowdsourcing according to claim 1, characterized in that, The formula for calculating semantic confidence is: Where C represents semantic confidence and N represents the number of current observations. The maximum number of observations is preset; θ is the observation angle. The maximum preset angle is K; K represents the number of semantic changes. This represents the maximum allowed number of changes; U represents the number of updates. The maximum number of updates is preset; α, β, γ and δ are weight coefficients, and satisfy α+β+γ+δ=1.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.
8. An electronic device, characterized in that, The device includes a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1 to 6.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method described in any one of claims 1 to 6.