Method and system for labeling of model training data, electronic device and crowd-sourced data labeling platform

Abstract

The application relates to the technical field of data labeling, and discloses a labeling method for model training data, which comprises the following steps: obtaining a multi-modal original data packet for model training; constructing a projection mapping matrix from a three-dimensional point cloud coordinate system to a two-dimensional image pixel coordinate system by using an internal parameter matrix of a camera acquisition device and an external parameter matrix of the camera acquisition device relative to a point cloud acquisition device; based on the projection mapping matrix, aligning an image frame in image data with a point cloud frame in point cloud data by using a timestamp sequence of the image data and the point cloud data, generating a spatio-temporal alignment fusion data frame; and labeling the fusion data frame. Based on the projection mapping matrix and the timestamp sequence of the image data and the point cloud data, the spatio-temporal alignment fusion data frame is generated, an objective geometric basis for data labeling is constructed, and the labeling accuracy of the labeled data is significantly improved. The application further discloses a labeling system for model training data, an electronic device and a crowdsourcing data labeling platform.

Description

Technical Field

[0001] This application relates to the field of data annotation technology, such as a method and system for annotating model training data, electronic devices, and crowdsourced data annotation platforms. Background Technology

[0002] Data annotation is a core and fundamental step in training artificial intelligence models. With the rapid development of technologies such as autonomous driving, smart healthcare, and multimodal large-scale models, the objects to be annotated have evolved from single two-dimensional images or text to multimodal fusion data including LiDAR point clouds, camera images, video streams, and speech-text. This type of data typically requires precise cleaning, spatiotemporal alignment, and semantic labeling of heterogeneous information collected from different sensors to form supervisory signals that can be effectively used by the algorithm model for learning.

[0003] In current technical practices, the annotation of such multimodal data (especially 2D images and 3D point clouds) mainly relies on manual operation and generally adopts a "split-screen independent annotation" interaction mode. The annotation system front end provides independent rendering viewports for 2D image data and 3D point cloud data. The annotation process typically includes: the annotator first loads camera data in the 2D image viewport, identifies the target object (such as a vehicle), and then uses the rectangle tool to draw a 2D bounding box; subsequently, the annotator needs to manually switch to the 3D point cloud viewport, and in a complex scene composed of hundreds of thousands of discrete points, rely entirely on visual observation and spatial imagination to find the point cloud clusters corresponding to the target in the 2D image; after finding the roughly corresponding point cloud region, the annotator uses the 3D bounding box tool to manually adjust its position, length, width, height, and rotation angle to complete the 3D annotation.

[0004] In the process of implementing the embodiments of this disclosure, at least the following problems were found in the related art: Manual annotation of data relies entirely on human visual experience to find the corresponding target in two separate viewports. Annotators find it difficult to accurately determine the real physical boundary and position of a 2D bounding box in 3D depth space, which easily leads to parallax errors and results in low accuracy of the annotated data.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a method and system for labeling model training data, an electronic device, and a crowdsourced data labeling platform to improve the accuracy of labeled data.

[0008] In some embodiments, the annotation method for model training data includes: obtaining a multimodal raw data packet for model training; wherein the multimodal raw data packet includes image data obtained by a camera acquisition device and point cloud data obtained by a point cloud acquisition device; constructing a projection mapping matrix from a three-dimensional point cloud coordinate system to a two-dimensional image pixel coordinate system using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device; based on the projection mapping matrix, aligning image frames in the image data with point cloud frames in the point cloud data using the timestamp sequence of the image data and the point cloud data to generate a spatiotemporally aligned fused data frame; and annotating the fused data frame.

[0009] Optionally, based on the projection mapping matrix, the image frames in the image data and the point cloud frames in the point cloud data are aligned using the timestamp sequences of the image data and point cloud data to generate spatiotemporally aligned fused data frames. This includes: using the timestamp sequences of the image data and point cloud data to temporally align the image frames and point cloud frames to form temporally matched image frame-point cloud frame pairs; and based on the projection mapping matrix, spatially aligning the image frame-point cloud frame pairs to generate spatiotemporally aligned fused data frames.

[0010] Optionally, the image frames and point cloud frames are time-aligned using the timestamp sequences of the image data and point cloud data to form time-matched image frame and point cloud frame pairs. This includes: extracting the first timestamp of the image frame and the second timestamp of the point cloud frame; using the first timestamp as a reference, employing a nearest neighbor interpolation algorithm, and using the second timestamp to match the point cloud frame with the smallest absolute time difference for each image frame, thus forming an image frame-point cloud frame pair.

[0011] Optionally, based on the projection mapping matrix, spatial alignment processing is performed on the image frame-point cloud frame pair to generate a spatiotemporally aligned fused data frame, including: loading the projection mapping matrix to map the point cloud frame in the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system; and encapsulating the image frame-point cloud frame pair and the projection mapping matrix used for spatial association together into a fused data frame.

[0012] Optionally, labeling the fused data frame includes pre-labeling the fused data frame, wherein the pre-labeling operation includes: calling a visual segmentation model to perform semantic segmentation on the image frames in the fused data frame to obtain a semantic segmentation mask; performing semantic matching between each candidate region in the semantic segmentation encoding and a predefined set of text labels to calculate the semantic similarity of each candidate region; and generating a two-dimensional initial annotation box corresponding to the target candidate region for the target candidate region whose semantic similarity exceeds a preset similarity threshold.

[0013] Optionally, the pre-annotation operation further includes: back-projecting the two-dimensional initial annotation box to three-dimensional space based on the projection mapping matrix to determine the three-dimensional view frustum; filtering point cloud data located within the three-dimensional view frustum from the point cloud frame corresponding to the current image frame in time and space; performing cluster analysis on the filtered point cloud data, and generating the three-dimensional initial annotation box based on the clustering results.

[0014] Optionally, the fused data frame is labeled, including scheduling operations for the labeling tasks of the fused data frame. The scheduling operations include: constructing a user capability profile containing a consistency coefficient based on the historical labeling records of crowdsourcing users; wherein the consistency coefficient reflects the degree of consistency between the user's historical labeling results and the standard labeling results; calculating the labeling task difficulty coefficient of the fused data frame using the average confidence and target density output by the pre-labeling operation; and distributing the labeling task of the fused data frame to crowdsourcing users according to the labeling task difficulty coefficient and the user capability profile.

[0015] Optionally, the annotation method for model training data further includes: after completing the annotation of the fused data frame, calling a rule base containing cross-modal logical mutual exclusion rules; comparing the annotation results of the fused data frame with the logical mutual exclusion rules in the rule base, and intercepting the annotation results that violate the logical mutual exclusion rules.

[0016] In some embodiments, a labeling system for model training data includes: an acquisition module configured to acquire a multimodal raw data packet for model training; wherein the multimodal raw data packet includes image data acquired by a camera acquisition device and point cloud data acquired by a point cloud acquisition device; a construction module, communicatively connected to the acquisition module, configured to construct a projection mapping matrix from a three-dimensional point cloud coordinate system to a two-dimensional image pixel coordinate system using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device; a generation module, communicatively connected to the construction module, configured to align image frames in the image data with point cloud frames in the point cloud data based on the projection mapping matrix and using the timestamp sequence of the image data and point cloud data to generate a spatiotemporally aligned fused data frame; and a labeling module, communicatively connected to the generation module, configured to label the fused data frame.

[0017] In some embodiments, the electronic device includes a processor, a memory, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements a method for labeling model training data.

[0018] In some embodiments, the crowdsourced data annotation platform integrates the aforementioned annotation system for model training data, or the aforementioned electronic device.

[0019] The annotation method and system, electronic device, and crowdsourced data annotation platform for model training data provided in this disclosure can achieve the following technical effects: In this disclosed technical solution, an intrinsic parameter matrix of the camera acquisition device and an extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device are used to construct a projection mapping matrix from the 3D point cloud coordinate system to the 2D image pixel coordinate system. This establishes a definite and computable geometric correspondence between the 2D image pixels and the 3D point cloud space, completely replacing the traditional spatial matching method that relies entirely on manual visual observation and subjective experience. This eliminates parallax and misalignment caused by human spatial imagination errors at the source. Simultaneously, frame-level alignment based on timestamp sequences ensures strict synchronization of multimodal data in the temporal dimension, forming a spatiotemporally consistent fused data frame. Annotation is performed on the fused data frame, enabling 2D annotations to automatically and accurately derive 3D annotations through predetermined projection relationships, thus ensuring that the annotation results of the two modalities are geometrically highly consistent and conform to physical reality. In this way, based on the projection mapping matrix and the timestamp sequences of image data and point cloud data, a spatiotemporally aligned fused data frame is generated, systematically constructing the objective geometric foundation for data annotation, thereby significantly improving the annotation accuracy of the annotated data.

[0020] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0021] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart illustrating a method for labeling model training data provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating another method for labeling model training data provided in this embodiment of the disclosure; Figure 3 This is a flowchart illustrating another method for labeling model training data provided in this embodiment of the disclosure; Figure 4This is a flowchart illustrating another method for labeling model training data provided in this embodiment of the disclosure; Figure 5 This is a flowchart illustrating another method for labeling model training data provided in this embodiment of the disclosure; Figure 6 This is a schematic diagram of the structure of a labeling device for model training data provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0022] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0023] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0024] Unless otherwise stated, the term "multiple" means two or more. In embodiments of this disclosure, the character " / " indicates that the preceding and following objects are in an "OR" relationship. For example, A / B means: A or B. The term "and / or" describes an association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B. The term "correspondence" can refer to an association or binding relationship; A corresponding to B means that there is an association or binding relationship between A and B.

[0025] Combination Figure 1 As shown in the embodiments of this disclosure, a method for labeling model training data is provided, including the following steps: S101, obtain the multimodal raw data package for model training; wherein, the multimodal raw data package includes image data obtained by the camera acquisition device and point cloud data obtained by the point cloud acquisition device.

[0026] Here, multimodal raw data packets refer to the unlabeled raw data sets generated by at least two different types of acquisition devices (e.g., sensors) synchronously or quasi-synchronously acquiring the same physical scene within the same or similar time periods.

[0027] In some specific application scenarios, the multimodal raw data packet includes image data acquired by a camera acquisition device and point cloud data acquired by a point cloud acquisition device; in other specific application scenarios, the multimodal raw data packet includes image data acquired by a camera acquisition device, point cloud data acquired by a point cloud acquisition device, and audio data acquired by an audio acquisition device. In practical applications, the multimodal raw data packet also includes timestamp sequences and device identification information from the acquisition process.

[0028] S102, using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device, constructs a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system.

[0029] Here, the intrinsic parameter matrix of the camera acquisition device is used to describe the internal optical characteristics of the camera. The intrinsic parameter matrix projects three-dimensional points in camera coordinates onto a two-dimensional imaging plane, and further transforms them into pixel coordinates. The intrinsic parameter matrix contains the core parameters of camera imaging, which are parameters that are fixedly related to the camera itself and do not change with the spatial position of the camera.

[0030] The extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device describes the spatial transformation relationship between the coordinate system of the point cloud acquisition device and the coordinate system of the camera acquisition device. The extrinsic parameter matrix includes a rotation matrix and a translation vector. The rotation matrix describes how the coordinate system of the point cloud acquisition device needs to be rotated to align with the direction of the camera coordinate system, and the translation vector describes the three-dimensional coordinate position of the origin of the point cloud coordinate system in the camera coordinate system.

[0031] In practical applications, the projection mapping matrix is ​​constructed according to the following formula:

[0032] Where P is the projection mapping matrix, K is the intrinsic parameter matrix, T is the extrinsic parameter matrix, R is the rotation matrix, and t is the translation vector.

[0033] For example, K is a 3x3 matrix, T is a 4x4 homogeneous transformation matrix, where R is a 3x3 orthogonal matrix and t is a 3x1 vector. When constructing the projection mapping matrix P, the T matrix is ​​multiplied by the intrinsic parameter matrix K, thereby establishing a complete geometric link that transforms any 3D point in the LiDAR coordinate system to the camera coordinate system and then projects it onto image pixels.

[0034] S103, based on the projection mapping matrix, uses the timestamp sequence of image data and point cloud data to align the image frames in the image data with the point cloud frames in the point cloud data, generating spatiotemporally aligned fused data frames.

[0035] Optionally, based on the projection mapping matrix, the image frames in the image data and the point cloud frames in the point cloud data are aligned using the timestamp sequences of the image data and point cloud data to generate spatiotemporally aligned fused data frames. This includes: using the timestamp sequences of the image data and point cloud data to temporally align the image frames and point cloud frames to form temporally matched image frame-point cloud frame pairs; and based on the projection mapping matrix, spatially aligning the image frame-point cloud frame pairs to generate spatiotemporally aligned fused data frames.

[0036] Through a two-stage progressive alignment mechanism, heterogeneous data from different sensors are integrated into a strictly unified representation in the spatiotemporal dimensions. Temporal alignment utilizes a precise timestamp sequence to find the point cloud frame with the closest acquisition time for each image frame, forming a temporally synchronized data pair and resolving the temporal misalignment problem caused by different sensor sampling rates. Spatial alignment calls upon a pre-calibrated projection mapping matrix to establish a deterministic mathematical mapping from 3D points to 2D pixels for each pair of temporally synchronized data. This provides a unified spatiotemporal reference frame for the data pairs, thereby generating a fused data frame that combines image texture information and point cloud spatial structure, providing a precise and consistent underlying data foundation for all subsequent processing.

[0037] In this way, the traditional association method relying on manual visual observation and experience-based guesswork is completely abandoned. An accurate correspondence between 2D and 3D data is automatically established through objective mathematical transformations, fundamentally eliminating parallax and mismatch errors. This ensures that the generated 2D and 3D annotations are geometrically self-consistent and conform to the physical constraints of the real world. Simultaneously, spatiotemporal alignment ensures that the data of each labeled object in different modalities points to the same entity at the same time, providing high-quality, consistent supervision signals for training the perception model.

[0038] In some possible implementations, the image frames and point cloud frames are time-aligned using the timestamp sequence of image data and point cloud data to form time-matched image frame and point cloud frame pairs. This includes: extracting the first timestamp of the image frame and the second timestamp of the point cloud frame; using the first timestamp as a reference, employing the nearest neighbor interpolation algorithm, and using the second timestamp to match the point cloud frame with the smallest absolute time difference for each image frame, thus forming an image frame-point cloud frame pair.

[0039] In practical applications, assume that an autonomous vehicle simultaneously acquires image and point cloud data. The camera acquires data at a frequency of 30Hz, i.e., 30 frames per second, and the LiDAR acquires data at a frequency of 10Hz, i.e., 10 frames per second. Within a 1-second segment, the generated timestamp sequence is as follows (unit: milliseconds ms): Image timestamp (first timestamp): [1000, 1033, 1066, 1100, 1133, 1166, 1200, 1233, 1266, 1300]; Point cloud timestamp (second timestamp): [1005, 1105, 1205, 1305]. These two timestamp sequences are extracted, and matching is performed based on the timestamps of the image frames. The timestamps of each image frame are iterated through, and the absolute time difference between the image frame and all point cloud frame timestamps is calculated. The point cloud frame with the smallest difference is selected as its pairing object. For example, for image frame t... img = 1000 ms, calculate the time difference with point cloud frames [1005, 1105, 1205, 1305]: [5, 105, 205, 305]. The minimum difference is 5 ms, so match point cloud frame 1005 ms, and so on. Finally, the following paired image frame-point cloud frame pairs are generated: (image frame @ 1000ms, point cloud frame @ 1005ms), (image frame @ 1033ms, point cloud frame @ 1005ms), (image frame @ 1066ms, point cloud frame @ 1105ms), (image frame @ 1100ms, point cloud frame @ 1105ms), ... (subsequent pairings follow the same rules).

[0040] By using automated timestamp nearest neighbor matching, the fundamental problem of native time asynchrony caused by different sensors' varying hardware sampling frequencies is solved. This ensures that each pair of image and point cloud data, which is subsequently spatially aligned and labeled, represents the physical world state closest to the same moment in time, thus providing a true and reliable foundation for temporal consistency in multimodal data. This not only avoids spatial discrepancies in labeled objects caused by temporal misalignment (such as a vehicle having moved), but also guarantees the effectiveness of the labeled data for model training from the outset—the model learns the multimodal feature associations of the same object at the same moment, rather than erroneous signals introduced by time differences.

[0041] In some possible implementations, spatial alignment processing is performed on image frame-point cloud frame pairs based on the projection mapping matrix to generate spatiotemporally aligned fused data frames. This includes: loading the projection mapping matrix to map point cloud frames in the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system; and encapsulating the image frame-point cloud frame pairs and the projection mapping matrix used for spatial association together into a fused data frame.

[0042] In some specific applications, based on the projection mapping matrix, the point cloud frame in the three-dimensional point cloud coordinate system is mapped to the two-dimensional image pixel coordinate system according to the following formula:

[0043] in, The coordinates are 3D point cloud coordinates, representing a physical point in a point cloud frame within the 3D point cloud coordinate system. These are two-dimensional image pixel coordinates, representing the position of a point in physical space in the two-dimensional image pixel coordinate system after projection mapping.

[0044] In practical applications, Let (x, y, z, 1) be a homogeneous coordinate system. T The column vector form, where x, y, z represent the three-dimensional spatial coordinates of the physical space points of the point cloud frame in the coordinate system centered on the lidar, in meters; the fourth component 1 is the identifier of homogeneous coordinates, used to uniformly represent affine transformations (rotation R and translation t) in three-dimensional space as matrix multiplication operations. Let (u, v, 1) be a two-dimensional homogeneous coordinate system. T The image is represented by a column vector, where u and v represent the column (pixel column index) and row (pixel row index) corresponding to a point in physical space projected onto the image, in pixels; the third component 1 is used to maintain the homogeneous coordinate representation. After the projection calculation is completed, the actual pixel coordinates (u, v) can be obtained by perspective division (dividing the first two components by the third component). For example, P 3d = [10.2, -1.5, 45.3, 1] T (Unit: meter), through Calculate P 2d = [u', v', w'] T = [112300, 53200, 45.3] T The actual pixel coordinates are (u, v) = (u' / w', v' / w') ≈ (2479, 1174).

[0045] In practical applications, we obtain the camera's intrinsic parameter matrix K (3×3) and the lidar's extrinsic parameter matrix T (4×4). Reading K and T, we calculate the projection mapping matrix P = K × T. The resulting P is a 3×4 matrix that encapsulates the complete transformation relationship from 3D point cloud coordinates to 2D image pixel coordinates. This transformation can be achieved through P... 2d = P × P 3dCalculate its projected 2D homogeneous coordinates. The fused data frame includes: the pixel coordinate matrix of the image frame, the coordinate matrix of the point cloud frame, the projection mapping matrix P, and timestamp pairs. The fused data frame can project 3D points onto the image, or backproject 2D boxes into 3D view frustums.

[0046] By encapsulating the projection mapping matrix together with the original data, a definite and repeatable projection relationship is ensured between each pixel in the image and the 3D points in the point cloud. This fundamentally eliminates spatial misalignment and subjective errors caused by manual viewport switching and visual matching in traditional methods. Simultaneously, the complex coordinate transformation process is abstracted into a built-in function of the fused data frame, enabling subsequent intelligent pre-annotation algorithms to directly and efficiently utilize this relationship for cross-modal inference, significantly improving the automation level of annotation.

[0047] S104, annotate the fused data frames.

[0048] By intelligently analyzing and recognizing images using vision and multimodal large models, 2D bounding boxes with high-confidence labels are automatically generated. Then, these 2D bounding boxes are automatically and accurately back-projected into 3D space using inherent projection relationships. Initial 3D bounding boxes are generated through point cloud clustering, realizing automatic derivation of cross-modal annotations and thus obtaining annotation data.

[0049] The annotation method for model training data provided in this disclosure utilizes the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device to construct a projection mapping matrix from the 3D point cloud coordinate system to the 2D image pixel coordinate system. This establishes a definite and computable geometric correspondence between the 2D image pixels and the 3D point cloud space, completely replacing the traditional spatial matching method that relies entirely on manual visual observation and subjective experience. This eliminates parallax and misalignment caused by human spatial imagination errors at the source. Simultaneously, frame-level alignment based on timestamp sequences ensures strict synchronization of multimodal data in the temporal dimension, forming a spatiotemporally consistent fused data frame. Annotation is performed on the fused data frame, enabling 2D annotations to automatically and accurately derive 3D annotations through predetermined projection relationships, thus ensuring that the annotation results of the two modalities are geometrically highly consistent and conform to physical reality. In this way, based on the projection mapping matrix and the timestamp sequences of image data and point cloud data, a spatiotemporally aligned fused data frame is generated, systematically constructing the objective geometric foundation for data annotation, thereby significantly improving the annotation accuracy of the data.

[0050] In some embodiments, labeling the fused data frame includes pre-labeling the fused data frame, wherein the pre-labeling operation includes: calling a visual segmentation model to perform semantic segmentation on the image frames in the fused data frame to obtain a semantic segmentation mask; performing semantic matching between each candidate region in the semantic segmentation encoding and a predefined set of text labels to calculate the semantic similarity of each candidate region; and generating a two-dimensional initial annotation box corresponding to the target candidate region for the target candidate region whose semantic similarity exceeds a preset similarity threshold.

[0051] In specific application scenarios, a deployed visual segmentation model (such as Meta's SegmentAnything Model, SAM) is invoked. The visual segmentation model infers from the input image frame and outputs a semantic segmentation mask. This mask categorizes each pixel in the image frame into different candidate regions. For example, the visual segmentation model outputs multiple irregular mask regions, corresponding to "Vehicle A," "Vehicle B," "Pedestrian," "Traffic Light," "Road," and "Sidewalk," respectively. A predefined set of text labels ["Car," "Pedestrian," "Traffic Light," "Bicycle," ...] and each mask region obtained in the previous step (e.g., the pixel block corresponding to "Vehicle A") are simultaneously input into a multimodal semantic model (such as OpenAI's CLIP). The multimodal semantic model encodes the text labels and image regions into feature vectors, and calculates the cosine similarity (e.g., 0.92) between the feature vector of the "Vehicle A" image region and the feature vector of each text label. The preset similarity threshold is 0.85. Since the similarity between the "Vehicle A" region and the "Car" label is 0.92 > 0.85, this region is determined to be a candidate target region. The minimum bounding rectangle that can tightly enclose the "Vehicle A" mask region is calculated, and a two-dimensional initial bounding box is generated, with coordinates as shown in (x... min =450, y min =300, width=120, height=80), and automatically associate the label "car". In this way, 2D bounding boxes with category labels are automatically generated for the vast majority of recognizable objects in the image frame.

[0052] By introducing visual segmentation models and multimodal semantic matching models, the tedious initial steps of manual selection and classification are replaced by traditional manual methods. AI completes object localization and preliminary recognition in batches and with high accuracy, effectively reducing initial errors caused by differences in human skill levels or subjective judgment. This lays a solid and efficient foundation for the rapid construction of subsequent cross-modal annotation (such as 3D bounding box generation) and high-quality training datasets, and is a key link in the scaling and intelligentization of multimodal data annotation.

[0053] Optionally, the pre-annotation operation further includes: back-projecting the two-dimensional initial annotation box to three-dimensional space based on the projection mapping matrix to determine the three-dimensional view frustum; filtering point cloud data located within the three-dimensional view frustum from the point cloud frame corresponding to the current image frame in time and space; performing cluster analysis on the filtered point cloud data, and generating the three-dimensional initial annotation box based on the clustering results.

[0054] In some specific application scenarios, the inverse transformation principle of the projection mapping matrix is ​​used to backproject the four corner points of the 2D bounding box (each point is regarded as a ray originating from the optical center of the camera) into 3D space. This forms a 3D view frustum in 3D space, with the optical center of the camera as its vertex and the four rays passing through the 2D bounding box as its edges. The 3D view frustum defines all possible spatial locations of the target object (e.g., a car) in the image in the real world. Fast geometric calculations are then performed from the point cloud frame that is strictly spatiotemporally aligned with this image frame, traversing all point clouds to determine the physical space point P. 3d Whether it is located within the aforementioned view frustum. For the selected physical space points, a clustering algorithm (such as DBSCAN) is performed to identify that these physical space points mainly belong to a large dense cluster (corresponding to the vehicle itself), possibly with several smaller clusters or discrete points (corresponding to noise or protruding components such as rearview mirrors). The main point cloud clusters are extracted, and their minimum bounding box in 3D space is calculated. Through fitting, an initial 3D bounding box is automatically generated, with parameters such as: center position (x=15.2m, y=-1.5m, z=0.9m), dimensions (length=4.5m, width=1.8m, height=1.5m), and orientation (yaw angle) yaw=0.2 rad. The 3D bounding box then forms a precise geometric correspondence with the 2D bounding box in the image.

[0055] By utilizing precise projective geometry, identified objects in 2D images can be automatically and accurately located in 3D point cloud space. Relevant point cloud data is then intelligently aggregated to generate 3D bounding boxes with reasonable physical dimensions and orientations. This not only completely liberates annotators from the arduous task of manually searching, rotating, and scaling 3D boxes among hundreds of thousands of discrete points, achieving an order-of-magnitude improvement in efficiency, but more importantly, it ensures that the generated 3D annotations are geometrically strictly aligned with the 2D annotations and are physically realistic and reliable. This provides high-quality, highly consistent key data for training models requiring precise 3D spatial perception, such as those for autonomous driving, fundamentally improving the reliability and usability of the annotation results.

[0056] Combination Figure 2 As shown, the annotation method for model training data includes the following steps: S201, Obtain the multimodal raw data package for model training; wherein, the multimodal raw data package includes image data obtained by the camera acquisition device and point cloud data obtained by the point cloud acquisition device.

[0057] S202, using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device, constructs a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system.

[0058] S203, based on the projection mapping matrix, uses the timestamp sequence of image data and point cloud data to align the image frames in the image data with the point cloud frames in the point cloud data, generating spatiotemporally aligned fused data frames.

[0059] S204, invoke the visual segmentation model to perform semantic segmentation on the image frames in the fused data frame to obtain the semantic segmentation mask.

[0060] S205, perform semantic matching between each candidate region in the semantic segmentation encoding and the predefined text label set, and calculate the semantic similarity of each candidate region.

[0061] S206. For target candidate regions whose semantic similarity exceeds a preset similarity threshold, generate a two-dimensional initial bounding box corresponding to the target candidate region.

[0062] S207, based on the projection mapping matrix, backprojects the two-dimensional initial annotation box to three-dimensional space to determine the three-dimensional view frustum.

[0063] S208: Select point cloud data located within the three-dimensional view frustum from the point cloud frame that corresponds to the spatiotemporal location of the current image frame.

[0064] S209. Perform cluster analysis on the selected point cloud data and generate initial 3D annotation boxes based on the clustering results.

[0065] The annotation method for model training data in this disclosure first establishes an objective and unified geometric and temporal basis for annotation by aligning sensor calibration with timestamps, completely eliminating the subjective errors and spatiotemporal inconsistencies caused by traditional manual visual matching. Then, based on the aligned data, it automatically and accurately generates 2D initial annotations using vision and multimodal large models, and intelligently derives strictly corresponding 3D initial annotations through geometric backprojection and point cloud clustering, forming high-precision initial annotation data. While ensuring geometric consistency and conformity to physical laws between 2D and 3D annotation results, it greatly accelerates the production process of high-quality training datasets.

[0066] In some embodiments, labeling the fused data frame includes scheduling the labeling task of the fused data frame, wherein the scheduling operation includes: constructing a user capability profile containing a consistency coefficient based on the historical labeling records of crowdsourcing users; wherein the consistency coefficient reflects the degree of consistency between the user's historical labeling results and the standard labeling results; calculating the labeling task difficulty coefficient of the fused data frame using the average confidence and target density output by the pre-labeling operation; and distributing the labeling task of the fused data frame to crowdsourcing users according to the labeling task difficulty coefficient and the user capability profile.

[0067] The average confidence score is the arithmetic mean of the semantic matching similarity scores of all automatically identified valid target objects in the fused data frame during the pre-annotation stage. A higher average confidence score indicates a higher level of credibility with the annotation results.

[0068] Target density is the ratio of the number of valid target objects identified through pre-annotation in the fused data frame to the area of ​​the corresponding image pixels in the fused data frame. A higher target density indicates that the current frame contains more objects that need to be labeled.

[0069] Alternatively, the consistency coefficient (Kappa coefficient) can be calculated using the following formula:

[0070] Where K is the consistency coefficient. The observation consistency ratio refers to the proportion of samples where user-annotated results are completely identical to standard annotation results (such as expert annotation results) out of the total sample size. The random expected consistency ratio is the theoretical probability that users and standard annotations will accidentally reach a consensus, calculated statistically under the assumption that users and standard annotations are completely random and independent.

[0071] Optionally, the annotation task difficulty coefficient of the fused data frame is calculated using the average confidence and target density output by the pre-annotation operation, including calculating the annotation task difficulty coefficient according to the following formula:

[0072] in, To label the task difficulty coefficient, C represents the average confidence level. This is the confidence threshold (e.g., 0.6). For the target density, Density threshold , These are the weighting coefficients.

[0073] In practical application scenarios, user capability profiles include core capability indicators (including consistency coefficient, historical average accuracy, and domain of expertise tags), real-time status indicators (including the number of tasks currently being processed, average task completion time, recent login and activity time, etc.), and statistical indicators (including the total number of completed tasks, success rate and quantity distribution in various scenario types, etc.).

[0074] Optionally, based on the annotation task difficulty coefficient and user capability profile, the annotation tasks of the fused data frame are distributed to crowdsourcing users, including: obtaining a candidate user list based on the user capability profile; calculating the comprehensive matching score between the candidate user and the annotation task of the fused data frame for the candidate user in the candidate user list; pushing the annotation tasks with an annotation task difficulty coefficient greater than the first difficulty coefficient threshold (e.g., 0.8) and less than the second difficulty coefficient threshold (e.g., 0.85) to the first target user with the highest comprehensive matching score; pushing the annotation tasks with an annotation task difficulty coefficient greater than the second difficulty threshold to the second target users ranked in the top N of the comprehensive matching score, and using the Kappa coefficient of the second target user as the confidence weight to perform a weighted average of the coordinate parameters of multiple annotation boxes to determine the optimal annotation box.

[0075] After the pre-annotation operation is completed, the user (annotator) receives the annotation task and only needs to make minor adjustments to the annotation boxes with low confidence or positional deviations on the front-end interface. When the annotator drags the initial two-dimensional annotation box on the 2D image, the corresponding initial three-dimensional annotation box in the 3D view is updated in real time through the projection matrix, realizing "one operation, two-end linkage".

[0076] Quantitative analysis of crowdsourcing users' historical performance generates user capability profiles centered on consistency coefficients, reflecting their annotation reliability and areas of expertise in real time. Simultaneously, based on the confidence level and target density of pre-annotated operation results, the complexity of the current task is objectively calculated, forming a task difficulty coefficient. The scheduling engine, according to preset matching rules, automatically directs high-difficulty tasks to high-ability users and sets differentiated acceptance thresholds for tasks of varying difficulty, thereby achieving precise matching between task characteristics and user capabilities and optimizing human resource allocation through a distribution mechanism.

[0077] By assigning challenging tasks to users with high consistency, the platform significantly reduces labeling errors and rework rates caused by insufficient personnel capabilities, ensuring the accuracy of complex data labeling. At the same time, through difficulty grading and dynamic matching, it avoids experienced users being overwhelmed by simple tasks while providing appropriate challenges for budding users, optimizing the platform's overall productivity and user experience, and providing sustainable intelligent scheduling support for the large-scale production of high-quality training data.

[0078] Combination Figure 3 As shown, the method for large model simulation training includes the following steps: S301, Obtain the multimodal raw data packet for model training; wherein, the multimodal raw data packet includes image data obtained by the camera acquisition device and point cloud data obtained by the point cloud acquisition device.

[0079] S302 uses the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device to construct a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system.

[0080] S303, based on the projection mapping matrix, uses the timestamp sequence of image data and point cloud data to align the image frames in the image data with the point cloud frames in the point cloud data, generating spatiotemporally aligned fused data frames.

[0081] S304. Based on the historical annotation records of crowdsourcing users, construct a user capability profile including a consistency coefficient; whereby the consistency coefficient reflects the degree of consistency between the user's historical annotation results and the standard annotation results.

[0082] S305 uses the average confidence level and target density output by the pre-annotation operation to calculate the annotation task difficulty coefficient of the fused data frame.

[0083] S306, based on the difficulty coefficient of the annotation task and the user's ability profile, distributes the annotation task of the fused data frame to the crowdsourcing users.

[0084] This embodiment abandons the traditional task-grabbing model for annotation. Instead, it dynamically evaluates users' actual abilities based on indicators such as the Kappa coefficient and objectively quantifies task difficulty according to the confidence and complexity of pre-annotation results. This data-driven matching mechanism accurately assigns high-difficulty tasks to high-ability users, effectively solving problems such as unstable annotation quality, high rework rates, and resource waste caused by mismatches between tasks and personnel capabilities in traditional crowdsourcing. It ensures the accuracy and consistency of annotation results from the source. Simultaneously, by optimizing human resource allocation, it significantly accelerates the production process of large-scale, high-quality training datasets, providing an efficient and reliable data supply chain foundation for training artificial intelligence models. Thus, by constructing a dynamic user ability profile and task difficulty assessment system, intelligent matching and precise scheduling of crowdsourced annotation tasks are achieved, significantly improving the overall efficiency and scalability of the annotation system while ensuring data quality.

[0085] In some embodiments, the method for large model simulation training further includes: after completing the annotation of the fused data frame, calling a rule base containing cross-modal logical mutual exclusion rules; comparing the annotation results of the fused data frame with the logical mutual exclusion rules in the rule base, and intercepting the annotation results that violate the logical mutual exclusion rules.

[0086] For example, cross-modal logical mutual exclusion rules in the rule base include: <image feature: strong light> and <text label: night> are mutually exclusive; <2D image: no vehicles> and <3D point cloud: obstacles> are mutually exclusive. These cross-modal logical mutual exclusion rules can be defined manually using pre-defined physical common sense, or automatically generated by mining and summarizing error logs from historical stored data using a Large Language Model (LLM).

[0087] In practical applications, after obtaining the annotation results of the fused data frame, the intelligent review engine is automatically triggered. The engine parses the submitted annotation results, extracting structured information such as the category, size, and position of all 2D / 3D bounding boxes, as well as related attribute labels (such as "rainy day" and "daytime"). Simultaneously, it rapidly analyzes the original fused data frame, extracting key unstructured visual features, such as using a lightweight model to determine the overall lighting conditions (bright / dim) and weather conditions (sunny / rainy). The parsed annotation results and extracted visual features are logically compared with each rule in the rule base. For example, matching the annotation "time: night" with the image analysis result "lighting: strong light" triggers a mutual exclusion rule. Once any logical conflict is detected, the system immediately marks the annotation result as "logical verification failed" and automatically rejects it, preventing it from entering the subsequent manual review or delivery stage.

[0088] By introducing an automated verification layer based on physical common sense and logical rules, semantic and physical consistency errors that traditional methods cannot detect can be identified in a deep and automatic manner, thereby greatly improving the logical rationality and overall credibility of the labeled data. At the same time, it frees reviewers from the massive and tedious task of searching for low-level logical errors, allowing them to focus on reviewing more complex semantic and edge cases. This significantly reduces the cost and false negative rate of manual quality inspection, while achieving a dimensional upgrade and a multiplier effect in data quality control.

[0089] Combination Figure 4 As shown, the method for large model simulation training includes the following steps: S401, Obtain the multimodal raw data packet for model training; wherein, the multimodal raw data packet includes image data obtained by the camera acquisition device and point cloud data obtained by the point cloud acquisition device.

[0090] S402 uses the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device to construct a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system.

[0091] S403, based on the projection mapping matrix, uses the timestamp sequence of image data and point cloud data to align the image frames in the image data with the point cloud frames in the point cloud data, generating spatiotemporally aligned fused data frames.

[0092] S404, annotate the fused data frames.

[0093] S405, after completing the annotation of the fused data frame, calls the rule library containing cross-modal logical mutual exclusion rules.

[0094] S406 compares the annotation results of the fused data frame with the logical mutual exclusion rules in the rule base and intercepts the annotation results that violate the logical mutual exclusion rules.

[0095] In this embodiment, during the critical step after alignment and annotation, the annotation results are automatically compared with a predefined or machine learning-mined physical common sense rule base. This enables efficient and accurate automatic identification and interception of deep logical contradictions and semantic errors that are difficult to detect using traditional methods. This not only fundamentally avoids injecting data containing physical logical fallacies into model training, ensuring the quality and consistency of the training dataset and improving the reliability and security of subsequent AI model inference, but also transforms quality review from an uncertain mode that highly relies on manual random sampling to precise and automated rule-based processing. This greatly improves the efficiency and coverage of quality inspection and provides key technical support for constructing large-scale, highly reliable multimodal training data.

[0096] In some embodiments, combined with Figure 5 As shown, the method for large model simulation training includes the following steps: S501, Obtain the multimodal raw data package for model training; wherein, the multimodal raw data package includes image data obtained by the camera acquisition device and point cloud data obtained by the point cloud acquisition device.

[0097] S502 utilizes the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device to construct a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system.

[0098] S503, based on the projection mapping matrix, uses the timestamp sequence of image data and point cloud data to align the image frames in the image data with the point cloud frames in the point cloud data, generating spatiotemporally aligned fused data frames.

[0099] S504 performs pre-annotation operations on the fused data frames.

[0100] S505 performs scheduling operations on the annotation tasks of the fused data frames.

[0101] S506, after completing the annotation of the fused data frame, calls the rule library containing cross-modal logical mutual exclusion rules.

[0102] S507 compares the annotation results of the fused data frame with the logical mutual exclusion rules in the rule base and intercepts the annotation results that violate the logical mutual exclusion rules.

[0103] The annotation method for model training data provided in this disclosure first ensures spatiotemporal consistency of data from the source, based on sensor calibration and time synchronization, establishing an objective geometric benchmark for data annotation. Second, AI-driven cross-modal pre-annotation transforms manual operations into efficient human-machine collaborative fine-tuning, significantly improving annotation efficiency. Third, intelligent scheduling based on dynamic capability profiles and task difficulty matching optimizes the allocation of crowdsourced resources, ensuring task completion quality. Finally, automated deep validation using a cross-modal logical rule base intercepts semantic and physical contradictions that are difficult for humans to detect, forming a reliable quality closed loop. This not only significantly accelerates the production of high-quality training data but also provides a solid and reliable data foundation for training multimodal large models from multiple dimensions, including geometric accuracy, logical rationality, and physical realism.

[0104] Combination Figure 6 As shown, this disclosure provides a labeling system 600 for model training data, including an acquisition module 601, a construction module 602, a generation module 603, and a labeling module 604. The acquisition module 601 is configured to acquire a multimodal raw data packet for model training. The multimodal raw data packet includes image data acquired by a camera acquisition device and point cloud data acquired by a point cloud acquisition device. The construction module 602 is communicatively connected to the acquisition module 601 and is configured to construct a projection mapping matrix from a three-dimensional point cloud coordinate system to a two-dimensional image pixel coordinate system using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device. The generation module 603 is communicatively connected to the construction module 602 and is configured to align image frames in the image data with point cloud frames in the point cloud data based on the projection mapping matrix and using the timestamp sequences of the image data and point cloud data, generating a spatiotemporally aligned fused data frame. The labeling module 604 is communicatively connected to the generation module 603 and is configured to label the fused data frame.

[0105] In some embodiments, the annotation system 600 for model training data further includes an auditing module 605, which is communicatively connected to the annotation module 604 and is configured to, after completing the annotation of the fused data frame, call a rule base containing cross-modal logical mutual exclusion rules; compare the annotation results of the fused data frame with the logical mutual exclusion rules in the rule base, and intercept annotation results that violate the logical mutual exclusion rules.

[0106] The annotation device for model training data provided in this disclosure utilizes the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device to construct a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system. This establishes a definite and computable geometric correspondence between the two-dimensional image pixels and the three-dimensional point cloud space, completely replacing the traditional spatial matching method that relies entirely on manual visual observation and subjective experience. This eliminates parallax and misalignment caused by human spatial imagination errors at the source. Simultaneously, frame-level alignment based on timestamp sequences ensures strict synchronization of multimodal data in the temporal dimension, forming a spatiotemporally consistent fused data frame. Annotation is performed on the fused data frame, enabling two-dimensional annotations to automatically and accurately derive three-dimensional annotations through predetermined projection relationships, thereby ensuring that the annotation results of the two modalities are geometrically highly consistent and conform to physical reality. Thus, based on the projection mapping matrix and the timestamp sequences of image data and point cloud data, a spatiotemporally aligned fused data frame is generated, systematically constructing the objective geometric foundation for data annotation, thereby significantly improving the annotation accuracy of the annotated data.

[0107] Combination Figure 7 As shown, this embodiment of the disclosure provides an electronic device (e.g., a computer, controller, etc.) 700, which includes a memory 701, a processor 702, a communication interface 703, and a bus 704. The memory 701, processor 702, and communication interface 703 are interconnected via the bus 704.

[0108] The memory 701 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0109] The memory 701 can store a program. When the program stored in the memory 701 is executed by the processor 702, the processor 702 and the communication interface 703 are used to execute the various steps of the annotation method for model training data in the embodiments of this application.

[0110] Processor 702 is a circuit with signal processing capabilities. In one implementation, processor 702 can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, processor 702 can implement certain functions through the logical relationships of hardware circuits. These logical relationships of hardware circuits are fixed or reconfigurable. For example, processor 702 can be a hardware circuit implemented as an ASIC or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a type of ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), or deep learning processing unit (DPU). The processor 702 is used to execute related programs to implement the functions required by the units in the annotation apparatus for model training data in the embodiments of this application, or to execute the annotation method for model training data in the method embodiments of this application.

[0111] As can be seen, each module in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0112] Furthermore, the modules in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these modules are integrated together as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or for implementing the functions of the modules of the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0113] The communication interface 703 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the electronic device 700 and other devices or communication networks. For example, data can be acquired through the communication interface 703.

[0114] Bus 704 may include a pathway for transmitting information between various components of electronic device 700 (e.g., memory 701, processor 702, communication interface 703).

[0115] It should be noted that, although Figure 7 The illustrated electronic device 700 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 700 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 700 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device 700 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 7 All the devices shown.

[0116] In some embodiments, the crowdsourced data annotation platform integrates the aforementioned annotation system for model training data, or the aforementioned electronic device.

[0117] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0118] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

[0119] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0120] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0121] The foregoing description and accompanying drawings fully illustrate embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included or substituted for parts and features of other embodiments. The scope of the embodiments of this disclosure includes the entire scope of the claims and all available equivalents of the claims. While the terms “first,” “second,” etc., may be used in this application to describe elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be called a second element without changing the meaning of the description, and similarly, a second element may be called a first element, provided that all occurrences of “first element” are consistently renamed and all occurrences of “second element” are consistently renamed. First and second elements are both elements, but may not be the same element. Moreover, the terminology used in this application is only for describing embodiments and is not intended to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. Similarly, the term “and / or” as used herein means including one or more of the associated listed any and all possible combinations. Additionally, when used herein, the terms “comprise” and its variations “comprises” and / or “comprising” refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase “comprising an…” does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0122] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0123] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for labeling model training data, characterized in that, include: Obtain multimodal raw data packets for model training; wherein, the multimodal raw data packets include image data obtained through a camera acquisition device and point cloud data obtained through a point cloud acquisition device; By using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device, a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system is constructed. Based on the projection mapping matrix, the image frames in the image data and the point cloud frames in the point cloud data are aligned using the timestamp sequence of image data and point cloud data to generate spatiotemporally aligned fused data frames. Label the fused data frames.

2. The annotation method according to claim 1, characterized in that, Based on the projection mapping matrix, and utilizing the timestamp sequences of image data and point cloud data, image frames in the image data are aligned with point cloud frames in the point cloud data to generate spatiotemporally aligned fused data frames, including: By using the timestamp sequence of image data and point cloud data, image frames and point cloud frames are time-aligned to form time-matched image frame-point cloud frame pairs; Based on the projection mapping matrix, spatial alignment processing is performed on image frame-point cloud frame pairs to generate spatiotemporally aligned fused data frames.

3. The annotation method according to claim 1, characterized in that, Using the timestamp sequences of image data and point cloud data, image frames and point cloud frames are time-aligned to form temporally matched image frame and point cloud frame pairs, including: Extract the first timestamp of the image frame and the second timestamp of the point cloud frame; Based on the first timestamp, the nearest neighbor interpolation algorithm is used to match the point cloud frame with the smallest absolute time difference for each image frame using the second timestamp, thus forming an image frame-point cloud frame pair. And / or, Based on the projection mapping matrix, spatial alignment processing is performed on image frame-point cloud frame pairs to generate spatiotemporally aligned fused data frames, including: Load the projection mapping matrix to map the point cloud frame in the 3D point cloud coordinate system to the 2D image pixel coordinate system; The image frame-point cloud frame pair and the projection mapping matrix used for spatial correlation are encapsulated together into a fused data frame.

4. The annotation method according to claim 1, characterized in that, Labeling the fused data frame includes pre-labeling the fused data frame, wherein the pre-labeling operation includes: The visual segmentation model is invoked to perform semantic segmentation on the image frames in the fused data frame to obtain the semantic segmentation mask; Each candidate region in the semantic segmentation encoding is semantically matched with a predefined set of text labels, and the semantic similarity of each candidate region is calculated. For target candidate regions whose semantic similarity exceeds a preset similarity threshold, generate the corresponding two-dimensional initial bounding box.

5. The annotation method according to claim 4, characterized in that, Pre-annotation operations also include: Based on the projection mapping matrix, the initial two-dimensional bounding box is back-projected into three-dimensional space to determine the three-dimensional view frustum; Filter point cloud data located within the 3D view frustum from the point cloud frames that correspond to the current image frame in time and space. Cluster analysis is performed on the selected point cloud data, and initial 3D annotation boxes are generated based on the clustering results.

6. The annotation method according to any one of claims 1 to 5, characterized in that, Labeling the fused data frames includes scheduling the labeling tasks for the fused data frames, wherein the scheduling operations include: Based on the historical annotation records of crowdsourcing users, a user capability profile including a consistency coefficient is constructed; whereby the consistency coefficient reflects the degree of consistency between the user's historical annotation results and the standard annotation results. The labeling difficulty coefficient of the fused data frame is calculated using the average confidence and target density output by the pre-labeling operation. Based on the difficulty level of the annotation task and the user's ability profile, the annotation task of the fused data frame is distributed to crowdsourcing users.

7. The annotation method according to any one of claims 1 to 5, characterized in that, Also includes: After completing the annotation of the fused data frames, the rule base containing cross-modal logical mutual exclusion rules is invoked; The annotation results of the fused data frames are compared with the logical mutual exclusion rules in the rule base, and the annotation results that violate the logical mutual exclusion rules are blocked.

8. A labeling system for model training data, comprising: The acquisition module is configured to acquire multimodal raw data packets for model training; wherein the multimodal raw data packets include image data acquired by a camera acquisition device and point cloud data acquired by a point cloud acquisition device; The construction module, which communicates with the acquisition module, is configured to construct a projection mapping matrix from the three-dimensional point cloud coordinate system to the two-dimensional image pixel coordinate system using the intrinsic parameter matrix of the camera acquisition device and the extrinsic parameter matrix of the camera acquisition device relative to the point cloud acquisition device. The generation module, which communicates with the building module, is configured to align image frames in the image data with point cloud frames in the point cloud data based on the projection mapping matrix and using the timestamp sequence of image data and point cloud data to generate spatiotemporally aligned fused data frames. The annotation module, which communicates with the generation module, is configured to annotate the fused data frames.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the annotation method for model training data as described in any one of claims 1 to 7.

10. A crowdsourced data annotation platform, characterized in that, The device integrates the annotation system for model training data as described in claim 8, or the electronic device as described in claim 9.

Images