A multi-modal fusion production-education integration data labeling practical training platform

Abstract

This invention discloses a multimodal fusion-based industry-education integrated data annotation training platform, belonging to the field of computer-aided teaching and information processing technology. The platform includes a heterogeneous data access and high-precision spatiotemporal synchronization module for achieving high-precision alignment of multi-source sensor data on the time axis and unified transformation of spatial coordinate systems; a cross-modal logical semantic consistency verification module for real-time semantic comparison and temporal correlation checks of annotation results for video, point clouds, and audio based on domain knowledge graphs; and a trainee fatigue prediction module based on long short-term memory networks for predicting fatigue states by monitoring operational behavior sequences and automatically triggering intervention mechanisms. This invention significantly improves annotation efficiency and data quality, and shortens the training cycle for advanced annotation personnel.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided teaching and information processing technology, specifically involving a multimodal fusion industry-education integrated data annotation training platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, high-quality data annotation has become the core foundation for deep learning model training and algorithm iteration. Industry-education integrated training platforms, serving as a link between educational resources and industrial needs, aim to cultivate high-level data processing talent by simulating real-world industrial environments and simultaneously producing standardized datasets. These platforms cover processing workflows for multiple data dimensions, including video, audio, and sensor data, and are key carriers for realizing a closed-loop data value system and the synergistic evolution of talent cultivation.

[0003] Multimodal fusion annotation technology, as an advanced direction in the current data processing field, requires the simultaneous processing and correlation analysis of heterogeneous data from different sensors. This technology aims to capture the interactive features of cross-dimensional information, providing accurate semantic support for complex application scenarios such as autonomous driving and smart healthcare. Effective multimodal annotation not only requires accurate parsing of individual modalities but also emphasizes seamless integration across time and spatial poses to ensure that the fused dataset possesses high consistency and logical rigor.

[0004] Current technologies for handling industry-education integration training tasks typically separate multimodal data annotation at the physical level, resulting in isolated annotation processes for video, audio, and various sensor data. This approach lacks real-time cross-modal correlation and synchronous verification mechanisms, easily leading to timeline misalignment between video action and audio description, or significant deviations between image detection bounding boxes and LiDAR point cloud positions. Existing quality control methods heavily rely on large-scale manual sampling in the later stages, which is not only inefficient overall but also unable to capture and correct annotation logic errors made by trainees in real time during the training phase. This results in inconsistent data quality in the final output, failing to meet the stringent training requirements of industrial-grade models for multimodal fusion data. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion-based industry-education integrated data annotation training platform to solve the problems mentioned in the background art, such as the lack of correlation, spatiotemporal asynchrony, lack of real-time logical verification in the training process, and excessive reliance on manual sampling for quality control caused by the physical splitting of multimodal data annotation.

[0006] The technical solution of this invention is a multimodal fusion industry-education integrated data annotation training platform, comprising: The heterogeneous data access and high-precision spatiotemporal synchronization module is used to receive video streams, audio streams, LiDAR point cloud streams, and inertial measurement unit data in parallel. It extracts the original timestamps of each data frame, selects the time reference of the high-frequency sensor as the global reference clock, and uses interpolation or matching algorithms to align the time axes of each data stream, controlling the time deviation within 10 milliseconds. It also uses the extrinsic parameter matrices of each sensor to uniformly transform the labeled vectors and point cloud coordinates to a shared world coordinate system. At the same time, based on the attitude compensation parameters provided by the inertial measurement unit, it constructs a homogeneous transformation matrix to compensate for sensor motion distortion, and performs motion distortion compensation on the point cloud spatial coordinates to achieve spatial centimeter-level alignment. The cross-modal logical semantic consistency verification module is used to perform real-time semantic comparison of the annotation results of video, point cloud and audio modalities based on the pre-built domain knowledge graph; extract the feature vectors of video image regions and LiDAR point cloud regions, calculate the Euclidean distance between them as the confidence of cross-modal association; and perform temporal correlation checks on long sequence data. When the confidence is lower than the preset threshold or a temporal logical conflict is detected, a warning is triggered and the annotation task is locked. The fatigue prediction module for trainees based on long short-term memory networks is used to monitor the trainees' operational behavior on the annotation interface in real time. It collects behavioral sequence data, including mouse movement trajectory, dwell time, view zoom frequency, and command call order. The behavioral sequence over a period of time is input into the trained long short-term memory neural network to predict the current fatigue probability. When the fatigue probability exceeds a preset threshold, an intervention mechanism is automatically triggered.

[0007] Furthermore, in the heterogeneous data access and high-precision spatiotemporal synchronization module, the time axis alignment operation is specifically as follows: the inertial measurement unit with the highest sampling frequency is selected as the time reference source; the low-frequency video frames or point cloud frames are resampled using a linear interpolation algorithm, or the nearest neighbor matching algorithm is used to uniformly map the sampling time of the low-frequency data stream onto the global reference clock sequence; The motion distortion compensation is specifically as follows: extract the angular velocity and linear acceleration output by the inertial measurement unit, calculate the instantaneous rotation compensation and instantaneous translation compensation, combine the rotation compensation and translation compensation into a homogeneous transformation matrix, and for any point in the original point cloud, perform calculations on its homogeneous coordinates through the homogeneous transformation matrix to obtain the new coordinates after motion distortion compensation.

[0008] Furthermore, in the cross-modal logical semantic consistency verification module, the feature vector of the video image region is obtained by extracting the features of the image region where the target is located through a fine-tuned ResNet-50 network, and the feature vector of the lidar point cloud region is obtained by extracting the features of the target point cloud region through a fine-tuned PointNet++ network; the confidence score is calculated based on the Euclidean distance of the feature vectors. The temporal correlation check is specifically as follows: maintain a temporal label sequence for each labeled target. When a change in the category label of a target within a preset time threshold that does not conform to the law of physical evolution is detected, it is determined to be a temporal logical conflict.

[0009] Furthermore, it also includes a multimodal panoramic perception annotation and interaction module, which is used to build a unified virtual training space, provide a multi-window linkage interactive interface, and realize the synchronous visualization of video, audio, point cloud and sensor data; When a target selection operation is performed in a view, according to the preset projection transformation model, the two-dimensional pixel coordinates are back-projected into the three-dimensional coordinate system of the LiDAR using the pre-calibrated camera intrinsic and extrinsic parameter matrices, and an associated bounding box is generated at the corresponding three-dimensional spatial position. It adopts a publish-subscribe message synchronization mechanism. When the annotation attribute of any modality changes, a message containing the modified content, timestamp and target identifier is generated and published to the global message bus. Listeners of other modalities receive the message and update their respective metadata copies in real time.

[0010] Furthermore, the multimodal panoramic perception annotation and interaction module has built-in automated auxiliary tools; For target tracking annotation, after manually annotating the target position in the starting frame, the system automatically provides two optional algorithms: optical flow or a pre-trained 3D target tracking neural network, to extrapolate the annotation position in subsequent frames. The 3D target tracking neural network adopts the Siamese network structure based on point cloud feature extraction. It takes the target point cloud template of the starting frame and the complete point cloud of the current frame as input and outputs the 3D bounding box position of the target in the current frame.

[0011] Furthermore, it also includes a dynamic scheduling module for industry-education integration tasks, which is used to construct a multi-dimensional capability matrix that includes four dimensions: two-dimensional image recognition, three-dimensional point cloud segmentation, audio semantic transcription, and cross-modal comprehensive judgment. Continuously capture historical behavioral data of trainees to generate dynamically updated capability profiles; break down industrial tasks entering the platform into multiple atomic annotation task units, with each atomic task unit corresponding to an independent annotation operation; Using a heuristic search algorithm based on task-ability matching, combined with greedy search and local backtracking, the optimal trainee is matched for each atomic task. When the success rate of a task at a certain difficulty level reaches a preset threshold, a higher difficulty task is automatically unlocked.

[0012] Furthermore, it also includes an industrial-grade quality lifecycle assessment module, which includes a quality detection sub-module based on generative adversarial networks; The quality detection submodule employs a generator network with a U-Net structure and a discriminator network with a convolutional neural network structure, which are jointly optimized through adversarial training. During the quality inspection phase, the annotation results submitted by the trainees are compared with the annotation results predicted by the generator based on the original data. The spatial overlap between the two is calculated, and the semantic category labels are compared. When the spatial overlap is lower than the preset threshold or the semantic categories are inconsistent, it is judged as an abnormal sample and transferred to the manual review process.

[0013] Furthermore, the industrial-grade quality lifecycle assessment module also includes a blockchain-based quality traceability sub-module; When each annotation action occurs, a record is generated that includes the action timestamp, trainee identifier, data sample identifier, annotation operation type, hash value of the snapshot of the state before and after the operation, electronic signature of the reviewer, and review conclusion. Records are organized into blocks in chronological order, with each block containing the hash value of the previous block, forming an immutable hash chain used to provide a complete production trajectory query for data samples.

[0014] Furthermore, the heterogeneous data access and high-precision spatiotemporal synchronization module also employs a hierarchical voxelization filtering strategy to process point cloud data. The point cloud is divided into layers based on a spatial octree structure. In each layer, the point closest to the voxel center is retained as the representative point, and the normal vector distribution of each point in the voxel is calculated. If the normal vector variance exceeds a preset threshold, all points in the voxel are forcibly retained, so as to maintain the spatial topological features of the object surface while reducing the point cloud density.

[0015] Furthermore, it also includes a sample selection module for running an active learning-based sample selection algorithm to select high-value samples from the pool of data to be labeled; The sample selection module is configured to calculate the cross-modal conflict severity score and annotation difficulty score for each unlabeled sample, where the conflict severity score is negatively correlated with the cross-modal association confidence, and the annotation difficulty score is determined by the variance of the probability score output by the discriminator in the generative adversarial network. The two scores are weighted and summed to obtain a comprehensive sampling score. The comprehensive sampling scores are sorted from high to low, and the top-ranked samples are selected as marginal samples and pushed to personnel whose cross-modal comprehensive judgment ability scores exceed a preset threshold for processing. After processing, the conflicting modal pairs and the final correct semantic relations in the sample are extracted and transformed into new entity-relation triples and incorporated into the knowledge graph.

[0016] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention addresses the mismatch between multi-source sensor data in time and space by integrating heterogeneous data access and a high-precision spatiotemporal synchronization module. It employs high-precision global reference clock mapping and extrinsic parameter matrix coordinate transformation to integrate previously isolated video, point cloud, and audio data into a unified spatiotemporal framework. This reduces annotation bias from seconds to less than 10 milliseconds and improves spatial accuracy to centimeter level, fundamentally ensuring the underlying consistency and scientific validity of the multimodal fusion dataset.

[0017] 2. This invention's innovative cross-modal logical semantic consistency verification module changes the traditional annotation platform's lagging management model that relies on manual sampling checks later. By introducing industry knowledge graphs and semantic conflict detection algorithms, it achieves real-time, automated quality inspection of the annotation process. This immediate logical correction mechanism not only greatly improves the initial pass rate of annotation results, but more importantly, through real-time correction guidance, it enables trainees to quickly understand the inherent logical connections of multimodal data in annotation practice, significantly shortening the training cycle for advanced annotation talents.

[0018] 3. The industry-education integration task dynamic scheduling module and competency profiling system constructed in this invention achieve precise matching between industrial needs and educational resources. Through fine-grained quantification of trainees' abilities and scientific modeling of task complexity, the platform can assign the most suitable task to the most suitable person at the most suitable time. This dynamic optimization mechanism not only ensures the delivery quality and efficiency of industrial orders but also provides trainees with a tiered growth path, maximizing both production efficiency and teaching effectiveness.

[0019] 4. This invention establishes a rigorous system for monitoring annotation behavior and quantitatively analyzing results through an industrial-grade quality lifecycle assessment module. It utilizes deep learning models to predict annotation fatigue and monitor attention, combining anomaly detection based on generative adversarial networks with blockchain-based quality traceability technology to construct a comprehensive, multi-dimensional quality firewall. This lifecycle management approach ensures that every piece of data produced in the training meets the training requirements of industrial-grade models, providing high-quality data support for the development of the artificial intelligence industry.

[0020] 5. The distributed microservice architecture and highly integrated multimodal panoramic perception annotation interaction module adopted by the platform described in this invention greatly improve the work efficiency of trainees and the scalability of the platform. Multi-window linkage, automated auxiliary annotation tools, and edge sample screening based on active learning liberate trainees from inefficient repetitive work, allowing them to focus more on cross-modal deep understanding and complex logical judgment. This improves work efficiency by more than 1.5 times while also enhancing the social value and economic benefits of the industry-education integration platform. Attached Figure Description

[0021] Figure 1This is a schematic diagram of the overall solution for a multimodal integrated industry-education data annotation training platform; Figure 2 This is a schematic diagram of the core principle framework of the cross-modal logical semantic consistency verification module; Figure 3 This is the logic flowchart of the heterogeneous data access and high-precision spatiotemporal synchronization module; Figure 4 This is a schematic diagram of the multi-level interaction relationships and data flow of the multimodal panoramic perception annotation interaction module; Figure 5 It is a logical flowchart of the dynamic scheduling of industry-education integration tasks and the full life cycle evaluation of quality. Detailed Implementation

[0022] Please refer to the attached document. Figure 1 This embodiment provides a multimodal fusion-based industry-education integrated data annotation training platform. Its system architecture operates within a distributed computing framework environment, achieving efficient flow and accurate annotation of heterogeneous data through a layered logical structure. The system's underlying resource pool supports the storage, preprocessing, and high-speed distribution of heterogeneous data; the intermediate service layer carries the core algorithm logic for synchronization, verification, scheduling, and evaluation; and the upper interaction layer provides trainees with a multimodal panoramic perception annotation interface.

[0023] Combined with appendix Figure 3 The heterogeneous data access and high-precision spatiotemporal synchronization module in this platform first performs the acquisition and parsing of raw data.

[0024] The system receives raw data streams from multiple sensors in parallel via a network interface or local bus. These data streams specifically include video streams, audio streams, LiDAR point cloud streams, and inertial measurement unit data. Upon receiving the data, the module immediately initiates a deep analysis program. The core task of this program is to identify and extract the hardware-generated raw timestamp information embedded in the frame structure of each data stream.

[0025] Next, the module performs time axis alignment. To eliminate asynchronous issues caused by differences in sampling frequencies between different sensors, the system selects a high-frequency sensor as the global reference clock. In practice, the inertial measurement unit (IMU) typically has the highest data sampling frequency and is therefore often set as this time reference source.

[0026] Subsequently, the system uses a linear interpolation algorithm to resample low-frequency video frames or point cloud frames; alternatively, it employs a nearest neighbor matching algorithm to uniformly map the sampling times of the low-frequency data stream onto a global reference clock sequence. Through this timeline alignment process, the system can control the time deviation between video, audio, and point cloud data to within 10 milliseconds.

[0027] After completing time synchronization, the module then aligns the spatial dimensions. The system pre-acquires and loads the pre-calibrated extrinsic parameter matrices for each sensor. This extrinsic parameter matrix defines the rotation and translation relationship of each sensor's local coordinate system relative to the device's central coordinate system.

[0028] Based on this, the system uses coordinate transformation operators to transform all labeled vectors and point cloud spatial coordinates from their respective local coordinate systems to a shared world coordinate system.

[0029] Considering the potential dynamic jitter that may occur during sensor movement, the system extracts attitude compensation parameters from the inertial measurement unit in real time. For each frame of point cloud data, the system calculates the instantaneous motion compensation matrix at the time of acquisition. The construction and operation of this compensation matrix are as follows: Let the angular velocity output by the inertial measurement unit be ω, the linear acceleration be a, and the sampling time interval be Δt. Then the formula for calculating the instantaneous rotational compensation ΔR is: ,in, This indicates an antisymmetric matrix operator, that is, for a three-dimensional vector. ,have exp(·) is a matrix exponential function used to convert the antisymmetric matrix corresponding to the angular velocity increment into a three-dimensional rotation matrix. This calculation is the standard method for inertial navigation system (INS) solutions in this field.

[0030] The formula for calculating the instantaneous translation compensation ΔT is: Where v is the velocity estimate from the previous moment. This velocity estimate is obtained through the linear acceleration of the real-time integrating inertial measurement unit: the system starts from rest (initial velocity is zero), and each sampling period is based on... Recursive update, where v k-1 Let q be the velocity at the previous moment. k-1 This is the linear acceleration from the previous moment. This recursive process is executed synchronously at each point cloud frame to ensure the real-time nature of the velocity estimation.

[0031] The rotation compensation amount ΔR and the translation compensation amount ΔT are combined into a 4×4 homogeneous transformation matrix M, which has the following form: , where 0 is the three-dimensional zero vector. For any point in the original point cloud, its coordinates in the local coordinate system are . (homogeneous coordinate form).

[0032] New coordinates P after motion distortion compensation comp Calculate P using the following formula: comp =M·P raw This involves first applying rotation compensation, followed by translation compensation. This operation is performed independently on each point in the point cloud, thereby eliminating point cloud stretching or distortion caused by sensor movement.

[0033] Ultimately, this module ensures that multimodal data have a unique correspondence in spatial location, and its spatial consistency accuracy can reach the centimeter level.

[0034] The above process begins with the reception of raw data and sequentially completes hardware timestamp extraction, time axis alignment based on a high-frequency reference source, unified spatial coordinate transformation based on the extrinsic parameter matrix, and motion distortion compensation using inertial measurement unit data.

[0035] The section on motion distortion compensation details the calculation formulas for rotation and translation compensation, the recursive method for obtaining velocity estimates, and the specific way the compensation matrix affects point cloud coordinates.

[0036] Combined with appendix Figure 4 The multimodal panoramic perception annotation and interaction module constructs a unified virtual training space. This space provides an interactive interface with multi-window linkage capabilities, and its design goal is to achieve synchronous visualization of video, audio, point cloud, and sensor data.

[0037] When trainees use a mouse or stylus to select targets in the video view, this module will generate a bounding box in real time at the corresponding 3D spatial position in the LiDAR point cloud view based on the preset projection transformation model.

[0038] The projection transformation model here specifically uses a pre-calibrated camera intrinsic parameter matrix K and a camera-to-LiDAR extrinsic parameter matrix. 2D pixel coordinates in video images The image is projected back onto the 3D coordinate system of the lidar.

[0039] The back projection relationship is given by the formula The description is as follows, where λ is the depth scale factor, obtained by measuring the distance to the corresponding pixel in the LiDAR point cloud.

[0040] This linkage is not limited to the visual level. The system will also automatically locate the corresponding spectral segment in the audio waveform: based on the timestamp of the video frame, it will locate the waveform area at the same time point on the time axis of the audio waveform and highlight it, making it easier for trainees to determine whether the target is emitting sound and the physical characteristics of the sound.

[0041] When trainees edit the attributes of a specific modality, such as modifying the target category, motion state, or semantic label, the metadata records of all other modalities will be updated in real time through the system's internal message synchronization mechanism.

[0042] The message synchronization mechanism adopts a publish-subscribe pattern: when the annotation attribute of any modality changes, the system generates a message containing the modified content, timestamp, and target unique identifier, and publishes the message to the global message bus.

[0043] Upon receiving a message, listeners of other modalities immediately update their respective copies of metadata. Message transmission uses the Trainer Datagram Protocol to reduce latency, and a sequence number mechanism ensures ordered message processing.

[0044] To improve interaction efficiency, this module allows trainees to customize shortcut key combinations. In addition, the module includes built-in automated auxiliary tools for 12 common annotation tasks, such as object tracking, semantic segmentation, and keypoint detection. The following uses object tracking annotation as an example to illustrate its implementation.

[0045] When performing target tracking and annotation, trainees only need to manually annotate the target position in the first frame, and the system can automatically extrapolate the annotation positions for the subsequent 10 to 20 frames. The system provides two optional algorithms: optical flow and point cloud tracking.

[0046] The optical flow method is based on the Lucas-Kanade algorithm and does not require training. For two consecutive frames of images, the optical flow vectors of feature points within the target region in the first frame are calculated, and then the position of the target in the next frame is estimated.

[0047] The point cloud tracking algorithm is based on a pre-trained 3D target tracking neural network. The training process of this network is as follows: the training data consists of a sequence of consecutive frames of LiDAR point clouds, with the target's 3D bounding box labeled in each frame. The network structure adopts a Siamese network based on point cloud feature extraction. The input is the target point cloud template of the first frame and the complete point cloud of the current frame, and the output is the position of the target's 3D bounding box in the current frame.

[0048] The loss function uses smoothed L1 loss, calculating the center point coordinate error, size error, and orientation angle error between the predicted bounding box and the ground truth bounding box. The training objective is to minimize these errors, enabling the network to accurately predict the target's position in subsequent frames. After training, the network uses the template from frame 1 as a reference to predict each subsequent frame, thus achieving automatic tracking.

[0049] Trainees can choose to use either optical flow or point cloud tracing algorithms based on the characteristics of the data. The system defaults to prioritizing point cloud tracing algorithms, and automatically reverts to optical flow if the point cloud quality is poor. After automatic extrapolation, trainees only need to quickly verify and fine-tune the automatically generated results, without needing to manually annotate frame by frame.

[0050] The modules described above construct an efficient and intuitive annotation interaction environment through multi-window linkage, cross-modal attribute synchronization, and built-in automated auxiliary tools. Specifically, the projection transformation model clearly defines the coordinate mapping relationship from 2D to 3D, the message synchronization mechanism adopts a mature publish-subscribe pattern, and the automated auxiliary tools provide specific algorithms and their training details.

[0051] Combined with appendix Figure 2 This is a cross-modal logical semantic consistency verification module. As the core quality assurance engine of the platform, it performs real-time semantic comparison of annotation results between different modalities based on a pre-built domain knowledge graph.

[0052] This knowledge graph covers information such as the physical attributes, behavioral logic, and common sense of the target. It is constructed as follows: entity, attribute, and relation triples are extracted from industry standards, annotation specifications, and expert experience, such as "vehicles have speed attributes" and "braking sound and deceleration behavior are strongly correlated". It is stored in a graph structure, where each node represents a semantic concept and each edge represents the logical relationship between concepts.

[0053] When a trainee labels a vehicle as moving at high speed in a video modal, and the LiDAR point cloud analysis shows that the vehicle's displacement vector remains 0 for five consecutive frames, the verification module will immediately trigger a spatial logic conflict warning. Specifically, the point cloud displacement analysis involves calculating the difference between the center coordinates of the 3D bounding box of the same target in consecutive frames. If the absolute value of the change in the center coordinates within five consecutive frames is less than a preset spatial threshold (e.g., 0.05 meters), the displacement vector is determined to be 0.

[0054] If the audio modality identifies a sudden braking sound as the ambient sound, but the vehicle in the video annotation is still labeled as moving at a constant speed, the system will display a semantic logic anomaly warning. The audio modality recognition is based on a pre-trained audio event detection network, whose output includes the "braking sound" category and its confidence level; the driving status in the video annotation is manually labeled by trainees or generated by auxiliary tools.

[0055] The verification module queries the knowledge graph for strong association rules between "brake sound" and "deceleration" and finds that "constant speed driving" and "brake sound" constitute a logical contradiction, thus triggering a warning.

[0056] To quantify the consistency of annotation results across different modalities, the verification module evaluates the confidence of cross-modal associations by calculating the Euclidean distance between the feature vectors extracted from each modality in the feature space. The feature vectors are extracted as follows: For the video modality, a ResNet-50 network pre-trained on the ImageNet dataset is used to extract features of the image region where the target is located, and the output of the penultimate layer is taken as the video feature vector V; for the LiDAR point cloud modality, a PointNet++ network is used to extract features of the target point cloud region, and the output of the penultimate layer is also taken as the point cloud feature vector P.

[0057] Both networks were fine-tuned before feature extraction. The training data for fine-tuning came from the platform's labeled multimodal data. The training objective was to minimize the cosine distance between different modal features of the same target, so that semantically similar targets would be close to each other in the feature space.

[0058] The formula for calculating confidence level is: ; Where C represents the confidence score of cross-modal association, ranging from 0 to 1; V i P represents the i-th feature component extracted from the video modality; i This represents the i-th dimension feature component extracted from the lidar point cloud mode; n represents the total dimension of the feature vector, which is 256 in this embodiment. The system compares the calculated confidence level C with a preset threshold of 0.75 in real time.

[0059] Once the confidence level falls below 0.75, the system will forcibly lock the current annotation task and highlight the conflicting modalities and regions on the interactive interface, requiring trainees to verify the authenticity of the target's attributes from a multi-dimensional perspective.

[0060] For long-sequence video data, the cross-modal logical semantic consistency verification module also performs temporal correlation checks. Specifically, the system maintains a temporal label sequence for each labeled target, recording its category label and motion state in each frame.

[0061] If a target's category label is "car" in frame 100 and changes to "pedestrian" in frame 105, with a time interval of only 0.2 seconds between these two frames (assuming a frame rate of 25 frames per second), the system determines that this change does not conform to the laws of physical evolution—an object cannot change from a car to a pedestrian in 0.2 seconds. In this case, the system will automatically backtrack and lock the time sequence from frame 100 to frame 105, prompting the trainee to check for labeling errors or confirm whether the target switching was caused by occlusion. The rules for temporal correlation checking are also derived from the knowledge graph, which defines the basic common sense that "category labels should remain stable within a short period of time."

[0062] In summary, the modules described above provide semantic rules through knowledge graphs, and combine them with spatial displacement analysis, audio event detection, feature vector confidence calculation, and temporal stability checks to achieve real-time logical verification of multimodal annotation results. The training data source, fine-tuning objective, and loss function of the feature extraction network are clearly defined, and the meaning of each letter in the confidence formula has been explained.

[0063] Combined with appendix Figure 5 This is a dynamic scheduling module for industry-education integration tasks, which optimizes the allocation of training resources by constructing a multi-dimensional capability matrix. This capability matrix is ​​a fine-grained quantitative evaluation system that covers the real-time scores of trainees in four key dimensions: 2D image recognition, 3D point cloud segmentation, audio semantic transcription, and cross-modal comprehensive judgment.

[0064] The system will continuously capture multiple behavioral data of trainees in historical tasks, including average completion time, accuracy of initial annotation, frequency of modification for the same task, and coping ability when dealing with extremely complex scenarios.

[0065] Based on this data, the system generates a dynamically updated competency profile for each trainee. This competency profile is not static; it adjusts in real time based on the trainee's performance in each task.

[0066] When real production tasks from industry enter the platform, this module first breaks down the task into multiple atomized labeled task units.

[0067] Each atomic task unit corresponds to a relatively independent and quantifiable annotation operation, such as labeling a target in a single frame of an image, drawing a bounding box for an object in a point cloud, or marking the start and end times of a specific event in an audio clip.

[0068] After the breakdown is completed, the module will use a heuristic search algorithm to find the optimal personnel and task matching solution based on the modal complexity, accuracy requirements and time urgency of each atomic task.

[0069] The heuristic search algorithm in this application specifically adopts a combination of greedy search based on task-ability matching degree and local backtracking: the algorithm first sorts the tasks from high to low difficulty, and then selects the personnel with the highest ability profile matching degree from the currently available trainees; if the matching degree is lower than the preset threshold, local backtracking is triggered, and the task is reassigned to other personnel with similar abilities and lower current load.

[0070] The matching degree is calculated by comprehensively considering the scores of personnel in each dimension and the weight of task requirements in each dimension, and is obtained through weighted Euclidean distance. The goal of the search is to maximize the total matching degree while meeting the time constraints.

[0071] Through this scheduling mechanism, the system can automatically assign high-difficulty industrial orders to senior training personnel with higher scores in the ability profile, and assign basic tasks to beginners for skill consolidation, thereby achieving double optimization of production quality and teaching effect.

[0072] In addition, this module also has an adaptive difficulty evolution mechanism. When a trainee's task success rate in tasks at a specific difficulty level exceeds 95% for three consecutive times, the system will determine that they have the ability to handle higher challenges and automatically unlock advanced tasks with more occlusions, weather interference, or sensor noise for them.

[0073] The task success rate described in this application refers to the proportion of the annotation results submitted by the trainee that are judged to be qualified after being detected by the industrial-level quality full-life cycle assessment module. The qualified judgment standard is: the spatial overlap or semantic deviation between the annotation result and the prediction result of the expert model does not exceed 2%.

[0074] In summary, the above-described production-education integration task dynamic scheduling module achieves precise adaptation of training tasks and personnel capabilities through a multi-dimensional ability matrix, dynamic ability profile, atomic task decomposition, and heuristic search matching. At the same time, the adaptive difficulty evolution mechanism ensures that trainees can obtain a stepped growth path. This scheduling mechanism provides a reasonable task allocation basis for the subsequent industrial-level quality full-life cycle assessment module, enabling quality assessment to more specifically monitor and feedback on the outputs of personnel at different ability levels, thus forming a closed-loop optimization among production, assessment, and scheduling.

[0075] Finally, the industrial-level quality full-life cycle assessment module is responsible for comprehensively quantifying and analyzing the behavior data and final annotation results during the annotation process. This module mainly includes three sub-functions: fatigue prediction based on behavior sequences, quality detection based on generative adversarial networks, and quality traceability based on blockchain.

[0076] For fatigue prediction based on behavior sequences, the module monitors the operation behaviors of trainees on the annotation interface in real time, and specifically collects the following four types of behavior data: mouse movement trajectory, the residence time of the mouse in each area of the interface, the frequency of view zoom operations, and the call order of various annotation instructions. These behavior data are recorded at a fixed sampling frequency to form a time series of the past 5 minutes.

[0077] The system uses a long short-term memory neural network to model these behavior sequences to predict the current fatigue probability of trainees. The training process of this network is as follows: Training data construction: Behavioral sequences of trainees were collected from the platform's historical operational data, and each sequence was manually labeled with a fatigue tag. The criteria for determining the fatigue tag were: whether the trainees experienced a significant surge in labeling errors or an abnormally prolonged reaction time within the time period corresponding to the behavioral sequence. Specifically, if the labeling error rate exceeded three times the normal level within a certain time period, and the average response time per operation exceeded twice the historical average, then that time period was marked as a fatigue state.

[0078] Network Structure and Input / Output: The input to the Long Short-Term Memory (LSTM) neural network is the aforementioned behavioral sequence matrix, where rows correspond to time steps and columns correspond to four types of behavioral feature values. The network output is a fatigue probability value between 0 and 1.

[0079] Loss function: The binary cross-entropy loss function is used, and the formula is as follows: ; Where: L represents the loss function value; N represents the total number of training samples; j represents the index number of the sample; y j This represents the true label of the j-th sample, with a value of 1 indicating a fatigued state and 0 indicating a non-fatigue state. This represents the fatigue probability predicted by the network for the j-th sample.

[0080] Training objective: To minimize the loss function L described above, enabling the network to accurately predict the fatigue state of trainees based on the input behavioral sequence. After training, the network parameters are fixed for real-time inference.

[0081] During the real-time prediction phase, the module inputs the behavioral sequence of the most recent 5 minutes into the trained network to obtain the fatigue probability value P. f The calculation formula is as follows: ;wherein: P f The value represents the predicted fatigue probability, ranging from 0 to 1; σ represents the Sigmoid nonlinear activation function, defined as... W h The weight matrix of the output layer is obtained through training; h t This represents the hidden state vector of the Long Short-Term Memory neural network at the current moment, which integrates the characteristics of personnel operations over the past 5 minutes; b f This represents the bias term of the output layer, which is obtained through training.

[0082] When P f When the threshold of 0.15 is exceeded, the system determines that the trainee is in a state of fatigue and automatically takes intervention measures: a prompt window suggesting a rest pops up on the interactive interface, and a higher-level auxiliary verification mechanism is temporarily introduced for the annotation results submitted by the trainee, such as forcing a second review.

[0083] For quality inspection based on generative adversarial networks (GANs), the module employs a GAN-based quality inspection model for automated quality checks on the labeled results submitted by trainees. This model consists of a generator network and a discriminator network, which are jointly optimized through adversarial training.

[0084] Training data construction: High-quality labeled samples that have passed expert review and verification are collected from the platform's historical data. Each sample contains a set of original sensor data and its corresponding expert annotation results, including bounding boxes, segmentation masks, or semantic labels. Simultaneously, an equal number of training labeled samples are collected from the historical submissions of trainees, including some samples with intentionally introduced defects.

[0085] The generator network employs a U-Net architecture, taking raw sensor data as input and predicting the labeled results as output. The training objective of the generator is to make its output labeled results as close as possible to expert annotations.

[0086] Discriminator network: It adopts a convolutional neural network. The input is a labeling result, and the output is a probability score between 0 and 1, which represents the probability that the input labeling result belongs to the expert labeling.

[0087] Loss function: The total loss function of a generative adversarial network is defined as follows: ; where: L GAN L represents the total loss value of the generative adversarial network; adv α represents the adversarial loss, used to measure the difference between the generator output and the distribution of expert annotations from the discriminator's perspective; α represents the weighting coefficient of the L1 loss; L L1 This represents the L1 distance between the generator output and the expert annotations, which is the sum of absolute errors per pixel or per point.

[0088] Combat loss L adv The specific form is: ; Where: G represents the generator network; D represents the discriminator network; x represents the distribution of real data p data (x) represents the expert annotation results sampled from the prior distribution p; z represents the result of sampling from the prior distribution p. z The raw sensor data sampled in (z), where p z (z) represents the distribution of the original data in the training data; denoted as the expectation operator, which is approximated by the mean of the in-batch samples in actual training; D(x) represents the probability score of the discriminator's output of the expert annotation x; D(G(z)) represents the probability score of the discriminator's judgment of the annotation result output by the generator.

[0089] Training objective: Minimize L GANThis allows the generator to output annotation results that are highly consistent with expert annotations, while making it impossible for the discriminator to effectively distinguish between the generator output and real expert annotations.

[0090] The quality control process is as follows: After training, the module uses the trained generator and discriminator to perform quality checks on the annotation results submitted by the trainees. The specific steps are as follows: The annotation results submitted by the trainees will be recorded as Y. student The original sensor data corresponding to the labeled result is input into the trained generator to obtain the labeled result Y predicted by the generator. gen ; Calculate Y student With Y gen The spatial overlap between two bounding boxes is known as the Cross-Union Ratio (CUI). For bounding box annotations, the CUI is the ratio of the intersection area to the union area of ​​the two boxes; for segmentation masks, the CUI is the pixel-by-pixel overlap ratio between the predicted and actual bounding boxes. Comparing Y... student With Y gen Are the semantic category labels consistent? If the intersection-union ratio is below 0.98 or the semantic categories are inconsistent, the semantic deviation of the sample is determined to exceed 2%, and it is marked as an abnormal sample, which is then transferred to the manual review process. Meanwhile, Y... gen The probability score obtained from the input discriminator is used as an auxiliary reference to further confirm the anomaly.

[0091] To ensure traceability of annotation quality based on blockchain, the platform integrates a blockchain recording mechanism. Each annotation action generates a record containing the following fields: timestamp of the action, unique identifier of the trainee, identifier of the data sample operated on, specific annotation operation type (e.g., creating a box, modifying a category, deleting a point), and hash values ​​of the annotation status snapshots before and after the operation. After the annotation task is reviewed, the system also records the final reviewer's electronic signature and review conclusion.

[0092] These records are organized into blocks in chronological order, with each block containing the hash value of the previous block, forming an immutable hash chain. Industrial customers can query the complete production trajectory of this data on the training platform using sample identifiers, including all personnel involved in annotation, the content of each modification, review records, and final qualification certification.

[0093] In summary, the industrial-grade quality lifecycle assessment module described above constructs a complete quality control system from three levels: behavior monitoring and fatigue prediction, automated quality inspection via generative adversarial networks (GANs), and blockchain-based quality traceability. Specifically, the fatigue prediction module provides the training data construction method, loss function, and prediction formula for LSTM networks, with all symbols clearly defined; the GAN module details the structure of the generator and discriminator, the source of training data, the specific form of the loss function, and the meaning of each symbol, and clarifies the judgment logic for the quality inspection stage; the blockchain module clearly defines the recording fields and storage mechanism. These three sub-functions work together to form a comprehensive quality assurance mechanism covering the entire process from pre-annotation to post-annotation, providing reliable anomaly detection input for subsequent real-time feedback and interactive correction systems.

[0094] The platform also integrates a real-time feedback and interactive correction system. When the cross-modal logical semantic consistency verification module detects a conflict or the quality assessment module identifies a low-quality sample, the system does not directly provide a standard answer. Instead, it guides trainees to independently analyze and discover the root cause of the error by highlighting the conflicting modalities, automatically retrieving related frames, and displaying logical contradiction descriptions in prominent positions on the interactive interface.

[0095] The system meticulously records the logical evolution of the trainees during the correction process, specifically recording the following: each modal view switching operation and its timestamp, each modification operation to the annotation attributes and their values ​​before and after the modification, and the final confirmation of the completion of the correction. These records are organized into a behavioral sequence in chronological order.

[0096] The system correlates the above behavioral sequence with the correction results: if the trainee successfully corrects the conflict or low-quality sample, a correction efficiency coefficient is calculated based on the total number of operations and total time spent completing the correction, using the following formula: Where E is the correction efficiency coefficient, and N ref The preset standard reference number of operations is T, where T is the total number of operations actually performed by the trainees to complete the correction. The value of E ranges from 0 to 1, with E being closer to 1 indicating higher correction efficiency. This coefficient serves as an important weighting factor and is used to update the "Cross-modal Comprehensive Judgment" dimension score in the trainees' competency profile. The update method is to take a weighted average of the original score of this dimension and E, and the weighting coefficient can be configured according to actual teaching needs.

[0097] This heuristic teaching model not only improves the quality of annotation outputs, but also effectively cultivates the professional skills of trainees in handling complex multimodal data.

[0098] To ensure smooth rendering and a real-time interactive experience when processing large-scale point cloud data, the heterogeneous data access and high-precision spatiotemporal synchronization module employs a hierarchical voxelization filtering strategy. This strategy is based on a spatial octree structure to hierarchically divide the point cloud: first, the point cloud space is recursively divided into eight octets, forming multiple levels of voxel meshes; in each level, the point closest to the voxel center within that level is retained as the representative point, and the normal vector distribution of each point within the voxel is calculated. If the normal vector variance exceeds a preset threshold, it is considered that geometric details exist within that voxel, and all points within that voxel are forcibly retained. By filtering layer by layer from coarse-grained to fine-grained layers, the original point cloud density is reduced to 30% while ensuring that the spatial topological features of the object surface are not lost, thereby significantly reducing the computational load on the graphics processor.

[0099] At the rendering layer, the system utilizes hardware acceleration technologies, such as OpenGL vertex buffer objects and shaders, to achieve millisecond-level synchronous refresh of multimodal data streams, ensuring that trainees do not experience noticeable delays when rotating and scaling the 3D viewpoint.

[0100] Furthermore, for audio annotation, the system developed a speech activity detection algorithm based on a Long Short-Term Memory (LSTM) network. This algorithm can automatically remove background noise, accurately locate the start and end times of human voices or specific mechanical sounds, and transform them into visualized waveform boundaries, greatly facilitating cross-modal timeline alignment. The algorithm is implemented as follows: First, Mel-frequency cepstral coefficients (MFCCs) are extracted from the original audio every 25 milliseconds as acoustic features, with adjacent frames overlapping by 10 milliseconds. The MFCC feature sequence of 200 consecutive frames is used as input to the LSM network. The network outputs a probability value between 0 and 1, indicating whether the current frame contains a human voice or a target mechanical sound. The network employs a two-layer LSM structure, with the output layer using the Sigmoid activation function. Training data comes from the platform's already labeled multimodal audio dataset, where each audio frame is manually labeled as a speech activity frame. The loss function uses binary cross-entropy loss, in the form of… Where M is the total number of training samples, k is the sample index, and y k This is the true label for the k-th frame, where 1 represents a frame with active speech and 0 represents a frame with no active speech. This represents the probability predicted by the network. The training objective is to minimize this loss function so that the network can accurately identify the start and end boundaries of speech activities. After training, the network predicts each frame of the audio, binarizes continuous probability sequences with a threshold of 0.5, merges isolated segments with a duration of less than 100 milliseconds, and finally outputs the start and end timestamps of the speech activities.

[0101] The cross-modal logical semantic consistency verification module also integrates a sample selection algorithm based on active learning. This algorithm employs an uncertainty-based active learning strategy to automatically select high-value samples from the unlabeled data pool.

[0102] The specific screening method is as follows: For each unlabeled multimodal sample, the system first calculates its cross-modal conflict severity score S. conflict =1-C, where C is the aforementioned cross-modal association confidence score; simultaneously, its annotation difficulty score S is calculated. difficulty This score is determined by the variance of the probability scores output by the discriminator in the generative adversarial network (GAN). Specifically, it is calculated by taking the standard deviation of the discriminator outputs obtained from multiple forward propagations of the same sample. A larger standard deviation indicates greater model uncertainty and higher annotation difficulty. The combined sampling score is obtained by weighted summation of the two scores. Where w1 and w2 are preset weights, both defaulting to 0.5. The system presses S... sample Sort the samples from highest to lowest and select the top 5% as marginal samples.

[0103] These samples often represent the most challenging extreme scenarios that algorithmic models struggle to cover. The system pushes these high-value samples to senior trainees or mentors for collaborative processing. The push rule is: for personnel whose "cross-modal comprehensive judgment" score in the retrieval ability profile exceeds a preset threshold of 0.85, the sample is assigned to the person with the shortest task queue. Through in-depth analysis of these samples, the platform's industry knowledge graph is continuously improved. Specifically, after senior personnel complete annotation and correction, the system extracts conflicting modal pairs and ultimately correct semantic relationships from the sample, transforming them into new entity-relation triples, such as "rainy environment - leads to - increased point cloud noise," and merging them into the knowledge graph. This closed-loop iterative mechanism allows the platform's verification intelligence to continuously improve with the accumulation of training tasks.

[0104] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0105] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multimodal fusion-based industry-education integrated data annotation training platform, characterized in that, include: The heterogeneous data access and high-precision spatiotemporal synchronization module is used to receive video streams, audio streams, LiDAR point cloud streams, and inertial measurement unit data in parallel. It extracts the original timestamps of each data frame, selects the time reference of the high-frequency sensor as the global reference clock, and uses interpolation or matching algorithms to align the time axes of each data stream, controlling the time deviation within 10 milliseconds. It also uses the extrinsic parameter matrices of each sensor to uniformly transform the labeled vectors and point cloud coordinates to a shared world coordinate system. At the same time, based on the attitude compensation parameters provided by the inertial measurement unit, it constructs a homogeneous transformation matrix to compensate for sensor motion distortion, and performs motion distortion compensation on the point cloud spatial coordinates to achieve spatial centimeter-level alignment. The cross-modal logical semantic consistency verification module is used to perform real-time semantic comparison of the annotation results of video, point cloud and audio modalities based on the pre-built domain knowledge graph; extract the feature vectors of video image regions and LiDAR point cloud regions, calculate the Euclidean distance between them as the confidence of cross-modal association; and perform temporal correlation checks on long sequence data. When the confidence is lower than the preset threshold or a temporal logical conflict is detected, a warning is triggered and the annotation task is locked. The fatigue prediction module for trainees based on long short-term memory networks is used to monitor the trainees' operational behavior on the annotation interface in real time. It collects behavioral sequence data, including mouse movement trajectory, dwell time, view zoom frequency, and command call order. The behavioral sequence over a period of time is input into the trained long short-term memory neural network to predict the current fatigue probability. When the fatigue probability exceeds a preset threshold, an intervention mechanism is automatically triggered.

2. The multimodal fusion-based industry-education integrated data annotation training platform according to claim 1, characterized in that, In the heterogeneous data access and high-precision spatiotemporal synchronization module, the time axis alignment operation is specifically as follows: the inertial measurement unit with the highest sampling frequency is selected as the time reference source; the low-frequency video frames or point cloud frames are resampled using a linear interpolation algorithm, or the nearest neighbor matching algorithm is used to uniformly map the sampling time of the low-frequency data stream onto the global reference clock sequence; The motion distortion compensation is specifically as follows: extract the angular velocity and linear acceleration output by the inertial measurement unit, calculate the instantaneous rotation compensation and instantaneous translation compensation, combine the rotation compensation and translation compensation into a homogeneous transformation matrix, and for any point in the original point cloud, perform calculations on its homogeneous coordinates through the homogeneous transformation matrix to obtain the new coordinates after motion distortion compensation.

3. The multimodal fusion-based industry-education integrated data annotation training platform according to claim 1, characterized in that, In the cross-modal logical semantic consistency verification module, the feature vector of the video image region is obtained by extracting the features of the image region where the target is located through a fine-tuned ResNet-50 network, and the feature vector of the LiDAR point cloud region is obtained by extracting the features of the target point cloud region through a fine-tuned PointNet++ network; the confidence score is calculated based on the Euclidean distance of the feature vectors. The temporal correlation check is specifically as follows: maintain a temporal label sequence for each labeled target. When a change in the category label of a target within a preset time threshold that does not conform to the law of physical evolution is detected, it is determined to be a temporal logical conflict.

4. The multimodal fusion industry-education integrated data annotation training platform according to claim 1, characterized in that, It also includes a multimodal panoramic perception annotation and interaction module, which is used to build a unified virtual training space, provide a multi-window linkage interactive interface, and realize the synchronous visualization of video, audio, point cloud and sensor data; When a target selection operation is performed in a view, according to the preset projection transformation model, the two-dimensional pixel coordinates are back-projected into the three-dimensional coordinate system of the LiDAR using the pre-calibrated camera intrinsic and extrinsic parameter matrices, and an associated bounding box is generated at the corresponding three-dimensional spatial position. It adopts a publish-subscribe message synchronization mechanism. When the annotation attribute of any modality changes, a message containing the modified content, timestamp and target identifier is generated and published to the global message bus. Listeners of other modalities receive the message and update their respective metadata copies in real time.

5. The multimodal fusion-based industry-education integrated data annotation training platform according to claim 4, characterized in that, The multimodal panoramic perception annotation and interaction module has built-in automated auxiliary tools; For target tracking annotation, after manually annotating the target position in the starting frame, the system automatically provides two optional algorithms: optical flow or a pre-trained 3D target tracking neural network, to extrapolate the annotation position in subsequent frames. The 3D target tracking neural network adopts the Siamese network structure based on point cloud feature extraction. It takes the target point cloud template of the starting frame and the complete point cloud of the current frame as input and outputs the 3D bounding box position of the target in the current frame.

6. The multimodal fusion industry-education integrated data annotation training platform according to claim 1, characterized in that, It also includes a dynamic scheduling module for industry-education integration tasks, which is used to construct a multi-dimensional capability matrix that includes four dimensions: two-dimensional image recognition, three-dimensional point cloud segmentation, audio semantic transcription, and cross-modal comprehensive judgment. Continuously capture historical behavioral data of trainees to generate dynamically updated capability profiles; break down industrial tasks entering the platform into multiple atomic annotation task units, with each atomic task unit corresponding to an independent annotation operation; Using a heuristic search algorithm based on task-ability matching, combined with greedy search and local backtracking, the optimal trainee is matched for each atomic task. When the success rate of a task at a certain difficulty level reaches a preset threshold, a higher difficulty task is automatically unlocked.

7. The multimodal fusion industry-education integrated data annotation training platform according to claim 1, characterized in that, It also includes an industrial-grade quality lifecycle assessment module, which contains a quality detection sub-module based on generative adversarial networks; The quality detection submodule employs a generator network with a U-Net structure and a discriminator network with a convolutional neural network structure, which are jointly optimized through adversarial training. During the quality inspection phase, the annotation results submitted by the trainees are compared with the annotation results predicted by the generator based on the original data. The spatial overlap between the two is calculated, and the semantic category labels are compared. When the spatial overlap is lower than the preset threshold or the semantic categories are inconsistent, it is judged as an abnormal sample and transferred to the manual review process.

8. The multimodal fusion industry-education integrated data annotation training platform according to claim 7, characterized in that, The industrial-grade quality lifecycle assessment module also includes a blockchain-based quality traceability sub-module; When each annotation action occurs, a record is generated that includes the action timestamp, trainee identifier, data sample identifier, annotation operation type, hash value of the snapshot of the state before and after the operation, electronic signature of the reviewer, and review conclusion. Records are organized into blocks in chronological order, with each block containing the hash value of the previous block, forming an immutable hash chain used to provide a complete production trajectory query for data samples.

9. The multimodal fusion industry-education integrated data annotation training platform according to claim 1, characterized in that, In the heterogeneous data access and high-precision spatiotemporal synchronization module, a hierarchical voxelization filtering strategy is also used to process point cloud data; The point cloud is divided into layers based on a spatial octree structure. In each layer, the point closest to the voxel center is retained as the representative point, and the normal vector distribution of each point in the voxel is calculated. If the normal vector variance exceeds a preset threshold, all points in the voxel are forcibly retained, so as to maintain the spatial topological features of the object surface while reducing the point cloud density.

10. The multimodal fusion-based industry-education integrated data annotation training platform according to claim 1, characterized in that, It also includes a sample selection module for running an active learning-based sample selection algorithm to select high-value samples from the pool of unlabeled data. The sample selection module is configured to calculate the cross-modal conflict severity score and annotation difficulty score for each unlabeled sample, where the conflict severity score is negatively correlated with the cross-modal association confidence, and the annotation difficulty score is determined by the variance of the probability score output by the discriminator in the generative adversarial network. The two scores are weighted and summed to obtain a comprehensive sampling score. The comprehensive sampling scores are sorted from high to low, and the top-ranked samples are selected as marginal samples and pushed to personnel whose cross-modal comprehensive judgment ability scores exceed a preset threshold for processing. After processing, the conflicting modal pairs and the final correct semantic relations in the sample are extracted and transformed into new entity-relation triples and incorporated into the knowledge graph.

Images