Big model multi-modal training data collection and intelligent recommendation model construction system

Abstract

The application discloses a big model multi-modal training data collection and intelligent recommendation model construction system, relates to the training data processing technical field, and is suitable for a high-risk post training scene; the method firstly collects, pre-processes and merges candidate steps of a video stream, a voice stream, operation logs, test results and simulator or device logs to form unified evidence objects; secondly, a cross-modal evidence credibility graph is constructed, and proficiency is updated according to the comprehensive credibility; thirdly, late evidence is executed to perform local difference correction and steady-state control; finally, step-level or process-level supplementary training sequences are generated, and supplementary training execution results, retest results and post results are written back to step nodes and skill nodes. The method can improve the accuracy of multi-modal evidence merging, reduce the interference of low-quality evidence on portrait updating, enhance the consistency of supplementary training recommendation and training state, and improve the closed-loop processing capability of the training process.

Description

Technical Field

[0001] This invention relates to the field of training data processing technology, specifically to a system for collecting multimodal training data and constructing intelligent recommendation models based on large models. Background Technology

[0002] Training platforms often need to simultaneously access video, audio, operation logs, test results, simulator or device logs, and then generate the next training plan based on the trainee's performance at each step or process. The typical technical approach involves collecting multimodal data at the front end or edge, writing learning activities to a learning record repository using a standardized interface, sorting events by time using streaming processing, and then combining content tags, skill tags, or learning path models to generate results. IEEE 9274.1.1-2023 specifies XAPI as a communication mechanism between learning activities and the learning record repository based on a JSON data model and a Rest interface. OpenMMLA also indicates that multimodal learning already has the toolchain-based data acquisition foundation.

[0003] The closest existing technologies can be summarized as "standardized learning records, multimodal acquisition, event timing processing, and content-skill association." IEEE 9274.1.1-2023 discloses the XAPI standard, which describes a JSON data model and Rest-style communication method between learning activities and learning record repositories; OpenMMLA discloses a multimodal learning analysis toolkit, providing pre-built pipelines and programming interfaces for audio analysis, video frame analysis, etc., to bring multi-source sensor and system data onto a single platform; US patent document US11531928B2 (publication date: December 20, 2022) discloses a method for automatically associating content items and skills through machine learning, providing a training process for text embedding, classification models, and skill labels to achieve automatic annotation of media content and skills; Apache FLink discloses an event timing and watermark mechanism, assigning timestamps and advancing watermarks to elements in a stream to set when to stop waiting for earlier events and trigger windows. According to the existing technology process, the typical process in this field is as follows: multi-source data is generated from the content acquisition terminal and log system, linked and stored through standardized interfaces or tools, then out-of-order and late events are organized using event timing mechanisms, and then associations are established with courses, questions or media content and skill tags to support course recommendations, learning path planning or learning analysis.

[0004] The aforementioned technologies are designed for general content recommendation or learning activity recording scenarios, but are not suitable for high-risk job training. Firstly, XAPI and similar standards primarily address how learning activities are recorded and exchanged, focusing on event representation and transmission rather than determining the step-by-step or process-level consistency of multimodal evidence. Secondly, toolchains like OpenMMLA mainly address multi-source acquisition and pre-built pipeline access, emphasizing "acquisition and integration," but not directly addressing the trustworthiness of evidence before it enters subsequent profiling and recommendation processes. Thirdly, Flink's event timing and watermarking primarily address time progression and window triggering in streaming computing, focusing on when to stop waiting for out-of-order events, rather than in video, audio, or logs. When modalities are non-uniform, have different granularities, and different arrival delays, can they collectively constitute valid evidence that can be used for training decisions? Fourth, the content-skill association scheme represented by US11531928B2 focuses on establishing a mapping between content and skills, which fundamentally relies on relatively stable content representations and skill labels. It does not carry out differentiated constraints on a large amount of step-level, temporal-level, and cross-modal evidence in high-risk training. Auditing studies also believe that AI education systems need verifiable claims, evidence, and technical means to verify evidence, but existing systems manage documents, original sources, and logs separately.

[0005] Therefore, in high-risk job training, when different modalities occur, arrive, and are processed at different times, and when some modalities have noise, missing information, delayed retransmission, or changes in authorization, step-level evidence mismatch, inaccurate skill judgment, and insufficient basis for subsequent recommendations can occur, leading to unreliable retraining arrangements, audit traceability, and training results. Therefore, how to improve the temporal organization, quality judgment, and traceability reliability of cross-modal learning evidence in high-risk job training is an urgent technical problem to be solved. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the shortcomings of existing technologies, this invention provides a system for constructing a large-scale multimodal training data acquisition and intelligent recommendation model. This system constructs a cross-modal evidence credibility graph and gates proficiency updates based on comprehensive credibility. It then performs local differential correction and steady-state control on late-arriving evidence. Finally, it generates step-level or process-level retraining sequences and writes the retraining results, retest results, and job descriptions back to the step and skill nodes. This method improves the accuracy of multimodal evidence merging, reduces the interference of low-quality evidence on profile updates, enhances the consistency between retraining recommendations and training status, and improves the closed-loop processing capability of the training process; thus solving the technical problems described in the background section.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A system for building a large-scale multimodal training data acquisition and intelligent recommendation model includes, in response to the start of a training task or the establishment of a session, acquiring at least three types of data from video streams, audio streams, operation logs, test results, and simulator or device logs, recording the event occurrence time, acquisition arrival time, and processing time, and encapsulating them into a unified evidence object containing segment boundaries, synchronization confidence, and usage status according to the candidate step window;

[0011] Based on a unified evidence object, a cross-modal evidence credibility graph is constructed, which includes attached step nodes, skill nodes, and trainee profile nodes. The overall credibility is calculated, and the proficiency of skill nodes is updated by gating according to the relationship between the overall credibility and low and high thresholds.

[0012] For late-arriving unified evidence objects, only the affected step nodes, skill nodes, and recommendation results are subject to local differential correction; supplementary training recommendation results are generated based on the updated proficiency profile, step risk level, and prerequisite ability satisfaction, and the supplementary training execution results or retest results are written back to the step nodes and skill nodes.

[0013] Furthermore, preprocessing includes performing original timestamp marking, basic denoising, local desensitization, and summary value generation on each modality of data, and maintaining a one-to-one correspondence between the summary value and the original fragment in the edge cache unit. Then, the access gateway combines the session identifier, personnel identifier, and job step identifier to supplement the collection arrival time and processing time.

[0014] Furthermore, the candidate step window is jointly determined by the job step identifier, process identifier, interaction boundary, and time sliding window. The evidence object construction unit only encapsulates the multimodal fragments belonging to the same candidate step window into a unified evidence object, and writes the fragment boundary, modal quality vector, missing mask, and synchronization confidence into the unified evidence object.

[0015] Furthermore, the cross-modal evidence credibility graph takes a unified evidence object as the central node and attaches step nodes, skill nodes, device nodes, and trainee profile nodes respectively. The step nodes are determined based on the training process library, the skill nodes are determined based on the mapping relationship between the step nodes and skill items, and the device nodes are located based on the modal source set.

[0016] Furthermore, the overall credibility is formed by the combination of time consistency, semantic consistency, device reliability, and usage permission. The gating update includes: when the overall credibility is higher than the high threshold and the usage permission meets the writing conditions, the unified evidence object is written to the skill node associated with the step node, and the corresponding record in the student profile node is updated synchronously.

[0017] Furthermore, when the overall credibility is between the low and high thresholds, the unified evidence object is transferred to the review zone and its connection with the step node and skill node is maintained; when the overall credibility is below the low threshold, only the object record, source record, and version record of the unified evidence object are retained, and the proficiency update is not performed.

[0018] Furthermore, high-risk steps are configured with higher and lower thresholds than low-risk steps. When the prerequisite capability satisfaction of the step node does not meet the preset conditions, the unified evidence object is prohibited from being directly used for the proficiency update of the advanced step, and is instead retained in the processing chain corresponding to the prerequisite step.

[0019] Furthermore, when a unified evidence object is determined to be a late unified evidence object after comparing its event occurrence time, collection arrival time, and current processing time, only the affected step nodes, skill nodes, and current recommendation results are subject to local differential correction, while the existing states of other unaffected step nodes and skill nodes remain unchanged.

[0020] Furthermore, when multiple unified evidence objects give inconsistent conclusions on the same step node, the unified evidence object that is consistent with the device node state and has a stable connection with the step node is retained first, and the remaining unified evidence objects are transferred to the review zone; when the missing key modality reaches the preset condition, the step node is switched to the evidence conservative mode.

[0021] Furthermore, the supplementary training recommendation results are generated in the order of supplementary training in the previous step, operation demonstration in the current step, re-operation task in the current step, and re-test task in the current step. The supplementary training execution results, re-test results, and job pass results are then written back to the step node, skill node, and trainee profile node in sequence to determine whether the current training session has ended or the next round of data collection, evaluation, and correction process has begun.

[0022] (III) Beneficial Effects

[0023] This invention provides a system for collecting multimodal training data and constructing intelligent recommendation models based on large models, which has the following beneficial effects:

[0024] By simultaneously writing the event occurrence time, collection arrival time, processing time, modal quality vector, missing mask, synchronization confidence, agreement status, and desensitization status into a unified evidence object, each piece of training evidence has a complete state description before entering the subsequent processing chain. This ensures that subsequent proficiency updates are based on objects with clear boundaries, clear sources, and clear states, rather than directly writing evidence fragments of unknown origin or incomplete states into the trainee profile node.

[0025] By creating a cross-modal evidence credibility graph, unified evidence objects are attached to step nodes, skill nodes, device nodes, and trainee profile nodes respectively. Credibility is judged based on time consistency, semantic consistency, device reliability, and compliance availability. Evidence objects first complete the credibility classification before entering the proficiency update link, thereby preventing low-quality evidence from contaminating step nodes and skill nodes and improving the relevance and stability of profile updates.

[0026] By enabling dual-threshold gating hysteresis, late correction grace periods and credible admission conditions are configured for high-risk steps. Unified evidence objects can be stratified and transferred between writing, review, and bypass when there are conflicts, omissions, or unstable boundaries. This avoids frequent flipping of step nodes and skill nodes in adjacent processing rounds, extending the processing order and judgment convergence of high-risk job training.

[0027] By generating step-level or process-level retraining sequences based on proficiency profiles, step-by-step risk levels, prerequisite competency satisfaction, evidence credibility, and recent retraining benefits, the retraining execution results, retest results, and job requirements are written back to the step and skill nodes. This creates a closed-loop link connecting retraining recommendations, training execution, and result feedback, ensuring that subsequent training arrangements are always based on real-world conditions, rather than a one-time recommendation or static record. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall architecture of the multimodal evidence processing system according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the process of merging multimodal fragments into a unified evidence object according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the unified evidence object structure according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram illustrating the construction of a cross-modal evidence credibility graph according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the comprehensive credibility calculation and dual-threshold gating flow splitting in an embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the late arrival correction and conflict resolution process according to an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the closed loop for generating supplementary training sequences and writing back results in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figures 1-7 This invention provides a system for collecting multimodal training data and constructing intelligent recommendation models based on large models, comprising:

[0037] Step 1: Without changing the on-site data collection method for high-risk job training, aggregate at least three types of modal data into the same candidate step or process window, so that a unified evidence object simultaneously retains the time source, fragment boundary, quality status and usage boundary, and provides a single continuous input carrier for subsequent proficiency updates.

[0038] In manufacturing assembly training, maintenance and disassembly training, or drug dispensing operation training, trainees often first perform a visible action, followed by a verbal confirmation, which then triggers an operation log or equipment report code. If the system only extracts fixed segments at a single sampling frequency, it will break the same action into multiple unrelated data segments; if the system only splices the logs according to their arrival order, it will mistakenly merge segments that occurred earlier and were subsequently transmitted into subsequent actions.

[0039] Therefore, before the raw data enters the image processing, a chain of mergeable fragments around the candidate steps is first established, so that each modality can enter the subsequent encapsulation with the same candidate step as a reference.

[0040] The edge preprocessing unit first performs raw timestamping and light cleaning on the input data in the video stream, audio stream, operation log, test results, and simulator / device log, respectively. Then, the access gateway adds the acquisition arrival time and processing time to each arriving segment. The use of three time records here is not to create redundant fields, but to decouple the three different events: action occurrence, segment arrival, and system processing.

[0041] The actions occur in the actual sequence of on-site behaviors, the segments arrive at the corresponding link states, and the system processes the corresponding computational position for subsequent merging. The candidate step segmentation unit then forms the candidate step boundary together with the job step identifier, process identifier, interaction boundary, and time sliding window. As long as a segment maintains a continuous coupling relationship with a candidate step boundary, the segment will not be immediately excluded due to retransmission, out-of-order delivery, or short-term missingness, but will first enter the set of candidates to be merged.

[0042] The edge preprocessing unit writes the event occurrence time to each raw fragment. The access gateway writes the acquisition arrival time when it receives the original fragment. The candidate step segmentation unit writes the processing time before performing the merge decision. .

[0043] Subsequently, the access gateway uses time-based traction. Should the order of events or the sequence of events be prioritized when determining which segments to prioritize?

[0044]

[0045] In the formula, time traction amount : Used to characterize the degree to which a certain segment is affected by link delay, with a value ranging from 0 to 1; event occurrence time. : Indicates the actual moment when the action corresponding to the segment occurred, in seconds; arrival time of acquisition. : Indicates the time when the segment arrives at the access gateway, in seconds;

[0046] Processing time The time unit for the access gateway to start merging and determining all segments is seconds; Link impact coefficient. The impact of queuing on fragment merging is calculated to be between 0.2 and 0.8; the buffer baseline time is... To avoid sudden changes in traction volume caused by extremely short time differences, the value is set to 0.5 to 3 seconds.

[0047] When time pull When the time traction is small, it indicates that the segment arrived later, but the closer it is to the original on-site sequence, the more likely the candidate step segmentation unit will retain it in the set to be merged for that candidate step; when the time traction... A larger value indicates that the segment is significantly affected by link lag, and the system does not discard it immediately but marks it as a link-driven segment.

[0048] When in use, the original fragment will not lose its on-site action attribution due to changes in the arrival order of the link; the delayed impact of edge buffer retransmission is made explicit in advance, and subsequent units process the same candidate step.

[0049] After completing the three-time recording, the candidate step segmentation unit will be merged based on the fixed time window and combined with the job step identifier, interaction boundary, and segment adjacency relationship. This is used to determine whether segments from different modalities converge in the same candidate step:

[0050]

[0051] In the formula, the merging potential : Used to indicate the degree of tendency for a fragment to be incorporated into the current candidate step, ranging from 0 to 1; boundary fitting amount : Indicates the degree of fit between the fragment boundary and the candidate step boundary, ranging from 0 to 1; boundary fit amount The overlap ratio between the segment boundary interval and the candidate step boundary interval is used to represent the overlap ratio, and the interval intersection-union ratio is selected.

[0052]

[0053] Where: Boundary fit amount This indicates the degree of fit between the current fragment boundary and the candidate step boundary, with a value ranging from 0 to 1; fragment boundary interval Indicates the start and end time interval of the current modal segment; step boundary interval. This indicates the start and end time interval of the current candidate step window.

[0054] Consistent quantity of steps : Indicates the degree of matching between the step identifier or process identifier in the fragment and the current candidate step, with a value ranging from 0 to 1; Modal cooperation quantity : The degree of consistency between a fragment and other modal fragments within the same class in terms of their sequential relationship, with a value ranging from 0 to 1; Modal coherence. The degree of matching reflecting the consistency of the sequential relationship of multimodal segments is preferably formed based on the consistency of video action tags, speech keyword order, and log time series; if the order of the three is consistent, then... Take the higher value; if there are order conflicts, decrease the value based on the number of conflicts.

[0055] Weighting coefficient Weighting coefficients Weighting coefficients Let these represent the influence strengths of boundary, step, and modal cooperation, respectively, all of which are greater than 0 and satisfy the following conditions: .

[0056] Angelica sinensis and its properties When the merging threshold is reached, the candidate step segmentation unit writes the segment into the set to be merged for the current candidate step; when the merging potential is reached... If a segment has not reached the merging threshold but has already been marked as a link-driven segment, the candidate step segmentation unit places it in the candidate step edge set, waiting for adjacent modes to be completed before further judgment; only when the merging potential is met... Only when a fragment remains below the threshold and has no clear step identifier will the system classify it into the free fragment set.

[0057] During intravenous puncture training for medical staff, nurses first disinfect, then verbally state the name of the medication, followed by an electronic training arm recording the needle tip's arrival. At this point, the boundaries of the disinfection image, the verbal statement of the medication name, and the training arm log of the needle tip's arrival do not coincide. (Merging potential) It's not about rigidly aligning these three categories, but rather comprehensively judging whether they converge around the candidate step of preparing for venipuncture. Thus, the same candidate step can accommodate a natural operational rhythm of video first, voice follow later, and log confirmation at the end, without artificially fragmenting continuous actions.

[0058] When used, the formation of candidate steps no longer depends on a single time window, but on the convergence of boundary alignment, step consistency and modal coordination. The resulting set to be merged is closer to the rhythm of on-site operations, which is convenient for subsequent encapsulation into a unified evidence object and reduces mismerging and omissions caused by differences in modal granularity.

[0059] After completing the fragment convergence at the candidate step level, two adjacent problems still need to be addressed: First, how to ensure that video clips, audio clips, operation logs, and device logs remain clear in terms of source, boundaries, and missing information within the same object; second, how to determine whether an object can subsequently enter the profile update at the time of object generation, rather than waiting until later steps to make corrections. The technical motivation lies in the fact that high-risk job training involves a training process with action consequences. If the source, missing information, and usage boundaries are not fully recorded within the object, then even if subsequent steps complete the proficiency update, it is difficult to determine whether the update is based on sufficient fragments.

[0060] The evidence object construction unit receives the set to be merged, first establishing the object skeleton according to the same candidate step or process window, and then writing in the modality source set, fragment boundaries, modality quality vector, missing mask, synchronization confidence, agreement status, de-identification status, original summary value, feature version information, and model version information one by one. The object skeleton here is not a database record template, but rather a transaction carrier jointly invoked by subsequent proficiency updates and recommendation control. Once a unified evidence object is formed, subsequent steps no longer make judgments based on scattered fragments, but only on the unified evidence object.

[0061] When creating a unified evidence object, the evidence object construction unit first writes the modality source set, then records the fragment boundaries, modality quality vectors, and missing masks for each modality fragment, and finally calculates the object completeness. :

[0062]

[0063] In the formula, object completeness : Used to characterize the acceptability of the current unified evidence object in the collected modalities, with a value ranging from 0 to 1; Modal number : The first person involved in the current unified evidence object encapsulation Modal class; total number of modalities : The number of modal categories included in the evaluation in this encapsulation, with a value being an integer greater than or equal to 3;

[0064] Modal weights : No. The load-bearing capacity of the modality in the current job step, with values ​​ranging from 0.1 to 1; modal mass. : No. The quality level of the modal fragment after preprocessing, with values ​​ranging from 0 to 1; modality presence quantity. : No. Does the class modality exist in the current unified evidence object? If it exists, return 1; if it does, return 0.

[0065] When object integrity When the object's encapsulation level falls below the lower bound, the evidence object construction unit does not terminate the encapsulation process. Instead, it completes the object skeleton writing and explicitly marks the source of the missing data in the missing mask. This results in subsequent steps obtaining an object with clear but insufficient boundaries. When the object completeness... When the value exceeds the lower bound of object encapsulation, the evidence object building unit continues to write the synchronization confidence and subsequent usage boundary.

[0066] In software operation and maintenance fault handling training, trainees enter troubleshooting commands in the console, then read out the alarm number, and finally return to the service recovery log. If the voice cannot be extracted due to environmental noise, the unified evidence object still retains the console command boundaries, service recovery log boundaries, and voice missing markers. Because no voice was used, but the source and nature of the missing information are consistent, there is no need to return to the underlying stream to confirm whether the voice was used; instead, the judgment is made directly based on the unified evidence object.

[0067] When used, the unified evidence object possesses three attributes upon its formation: source closure, boundary closure, and missing data closure. Subsequent steps reading the object can directly determine which modalities it comprises, the boundary positions of each modality in candidate steps, and the distribution of missing data, thereby avoiding fragment-level redundant judgments.

[0068] After the object skeleton is completed, the evidence object construction unit continues to write the consent status, de-identification status, original summary value, feature version information, and model version information, and registers the unified evidence object as the input carrier for subsequent processing chains. The core here is not simply adding management fields, but binding the usage boundaries to the object itself, so that objects received in subsequent steps naturally carry the discriminatory premise of whether they can enter the profile update. The original summary value is used to maintain the one-to-one correspondence between local fragments and encapsulated objects; the feature version information is used to describe the formation version of derived quantities such as the object's internal quality vector and synchronization confidence; the model version information is used to describe the version of the subsequent processing chain that the object is prepared to enter. If the consent status does not allow training, the evidence object construction unit only retains the object skeleton, original summary value, and version information, and does not send the unified evidence object into the subsequent profile update chain.

[0069] As a supplement, different observable quality indices are used to form modal mass components under different modes. Normalize to the 0-1 range. For video streams, this applies to video quality components. Image sharpness, key action visibility, and frame continuity are used to assign these parameters; for audio streams, these parameters are used to assign audio quality components. Assigned based on effective speech percentage, keyword completeness, and signal-to-noise discriminability;

[0070] For operation logs and emulator / device logs, log quality components are considered. The test results are weighted by timestamp continuity, field completeness, and session attribution consistency, and are considered as a test quality component. It is obtained by weighting the question completion rate, the answer structure completeness, and the time window matching rate.

[0071] The mass components of any mode are adopted in a uniform normalized form:

[0072]

[0073] Where: Modal mass components Indicates the first The quality level of the modality ranges from 0 to 1; the number of indicators Indicates the first The quality index used for the modality is an integer greater than or equal to 2; the weighting coefficient... Indicates the first In the class modality, the first The weights of each quality indicator are real numbers greater than 0; normalized indicator values... Indicates the first In the class modality, the first The normalized result of each quality indicator takes a value between 0 and 1.

[0074] For example, in field pipeline inspection training, the edge terminal first generates the original summary value locally and caches the original fragments. Once back in the site's network coverage area, the access gateway records the acquisition arrival time and processing time. For nursing training within a hospital LAN, video acquisition devices, voice acquisition devices, and nursing training stations directly send data to the access gateway, which then performs unified object encapsulation. For assembly training in a manufacturing workshop, torque tools, cameras, and workstation terminals can either send fragments separately or have the workstation edge terminal first collect the data before sending it up uniformly. Regardless of the deployment method, the unified evidence object maintains the same terminology and field system, and subsequent steps are always processed based on the unified evidence object.

[0075] In practice, multiple terminals are deployed in different scenarios. As long as a unified evidence object with the same field system is formed, the fragment chains to be merged are converged into a unified evidence object. When the object is generated, the source boundary, missing boundary, and usage boundary are written. Subsequent calls to the object ensure both clear multimodal source and preservation of object integrity. If the status and version are agreed upon, the proficiency update judgment will continue without changing the original stream.

[0076] In a preferred embodiment, the edge preprocessing unit is handled by an industrial edge terminal, the access gateway by a site server, and the candidate step segmentation unit and the evidence object construction unit by the same data processing server. The video stream is output from a head-mounted camera, the audio stream from a chest-mounted microphone, the operation log from a control terminal, and the simulator / device log from a training device control board. The edge preprocessing unit first writes the event occurrence time into the header of each segment. The gateway then generates the original summary value and records the arrival time of the data collection upon receipt. and processing time Candidate step segmentation unit utilizes time traction. Harmony and Integration A set to be merged is formed; the evidence object construction unit is then based on the object completeness. Complete the writing of the unified evidence object. If an alternative implementation is needed, the edge preprocessing unit can also be deployed within the head-mounted terminal, and the access gateway can also be deployed within the workstation terminal, as long as the three-time recording and merging potential are maintained. Judgment and object integrity By keeping these three processing chains unchanged, an equivalent implementation method can be formed.

[0077] Step 2: Establish a cross-modal evidence credibility graph centered on the unified evidence object, and limit which unified evidence objects can be written into the step node and skill node accordingly, thereby further reducing the collected and encapsulated objects to verified and usable objects.

[0078] In step one, the video stream, audio stream, operation log, test results, and simulator / device log have been consolidated into a unified evidence object. However, a unified evidence object only indicates that these segments belong to the same candidate step and does not automatically equate to these segments being sufficient to support subsequent profile writing. If the audio segment comes from ambient noise from an adjacent workstation, or the console operation log comes from a session that was not closed in the previous round, then even if the boundaries of the unified evidence object are complete, it cannot be directly written into the student profile node.

[0079] Therefore, a cross-modal evidence credibility map is inserted between the unified evidence object and the proficiency profile, which explicitly unfolds the relationship between the object, steps, skills, equipment and trainee profile, and makes the relationship itself the carrier of subsequent admission judgment.

[0080] The data processing server first reads the unified evidence object set output from step one, then reads the step node set and skill node set from the training process library, simultaneously reads the equipment node set from the equipment ledger, and reads the student profile node from the student file. The evidence credibility graph construction unit first uses the unified evidence object as the central node, placing step nodes upstream of its process link, skill nodes upstream of its capability mapping link, equipment nodes upstream of its source constraint link, and student profile nodes downstream of its write link. This constructed framework ensures that subsequent calculations are not performed directly on the one-sided mapping between objects and profiles, but rather within the constrained channels of objects, steps, skills, equipment, and profiles.

[0081] The evidence credibility graph construction unit first constructs the graph based on the job step identifiers, process identifiers, fragment boundaries, and merging potentials already written within the unified evidence object. The unified evidence object is attached to the current candidate step node or an adjacent step node; then, according to the step-skill mapping pre-maintained in the training process library, the unified evidence object is attached to the corresponding skill node.

[0082] The key here is not to simply use the candidate step conclusions from Step 1, but to rewrite those conclusions as restricted edges on a graph. If the unified evidence object crosses the boundaries of two adjacent steps simultaneously, the evidence credibility graph construction unit first preserves the bilateral connections, and then subsequent steps determine which edge is more stable.

[0083] To elevate the attachment process from rule-based assignment to restricted matching, the evidence credibility graph construction unit assigns values ​​to the calculation steps of each unified evidence object. :

[0084]

[0085] In the formula, the step is attached to the value. : Used to characterize the tightness with which the unified evidence object is attached to the current step node, with a value ranging from 0 to 1; merging potential : The degree to which the unified evidence object is converged to the current candidate step in step one, with a value ranging from 0 to 1; object completeness The degree of acceptance of each modality in the unified evidence object, with a value ranging from 0 to 1;

[0086] Total number of modes : The number of modal categories participating in the current unified evidence object evaluation, with a standard of an integer not less than 3; Modal weights : No. The load factor of the modality in the current job step, ranging from 0.1 to 1; an indicator exists. : No. Does the class modality exist in the current unified evidence object? If it exists, return 1; if it does, return 0.

[0087] Step attach value When the value is high, the evidence credibility graph construction unit will retain the main connection between the unified evidence object and the current step node; in the step attachment value When in the middle range, the double connection is preserved and semantic judgment is performed; when the step attachment value is low, the object will not be allowed to enter the skill node write link. Taking manufacturing assembly training as an example, the assembler first reads the work order, then picks up the parts, and then presses them. The head-mounted camera captures the hand action of picking up the parts, the workstation terminal captures the work order page turning, but the pressing equipment has not yet moved. When the unified evidence object and the step attachment value of the part picking step node are... The step attachment value is higher than that of the pressing and fitting step node. At that time, the evidence credibility graph construction unit will retain the main connection of the former.

[0088] When used, after the unified evidence object enters the evidence credibility graph, it first obtains clear step main connections and skill candidate connections. Subsequent calculations deal with the already converged graph edges, rather than a messy fully connected structure.

[0089] After connecting the step nodes and skill nodes, the evidence credibility graph construction unit continues to connect the unified evidence object to the device node and trainee profile node. The purpose here is not to increase the graph size, but to solidify the two boundary conditions: who generated the fragment and who the fragment is to be written onto. Because in high-risk job training scenarios, the same action is often recorded by different acquisition devices. If the source device is not clearly identified in the graph, it is impossible to distinguish between abnormal fragments caused by device offset and abnormal fragments caused by trainee misbehavior. Similarly, if trainee shift changes, session switching, or role handovers are not clearly identified in the graph's trainee profile node, the same unified evidence object, even if it matches the step node and skill node, may be written into the wrong profile.

[0090] The evidence credibility graph construction unit first locates the corresponding device node based on the modality source set and original summary value in the unified evidence object, and then locates the target student profile node based on the session identifier and personnel identifier. If a device node has a calibration and maintenance record in the current training period, the evidence credibility graph construction unit will retain the device node, but at the same time write a device status flag on its edge; if the personnel identifier corresponding to a unified evidence object is inconsistent with the student profile node in the current training session, the system will not allow it to enter the profile writing preparation chain, but will instead switch to the personnel mapping verification chain. In this way, the device node is responsible for limiting the reliable source range, and the student profile node is responsible for limiting the writing attribution range. The unified evidence object already has dual constraints of source and attribution before entering the subsequent credibility calculation.

[0091] When in use, the unified evidence object is placed on the graph skeleton composed of step nodes, skill nodes, equipment nodes, and student profile nodes. Object judgment no longer only involves the object itself field, but starts from the restricted relationship between the object and multiple types of nodes. This preserves the integrity of the object encapsulation in step one and provides a structured entry point for gating updates.

[0092] After the graph skeleton is established, assuming that multimodal sources, step boundaries, equipment status, and personnel affiliation are all linked, which unified evidence objects can be truly written into the skill nodes and trainee profile nodes, which unified evidence objects can only remain in a pending state, and which unified evidence objects should be blocked? If we still directly compare single quality components in the traditional way... Or just look at the completeness of the object This can confuse objects with complete boundaries but semantically deviated features with objects that are semantically correct but temporarily lack modality. Therefore, time consistency, semantic consistency, device support, and usability are all factored into the overall credibility, and then dual-threshold gating is used to divert objects within the graph to different subsequent channels.

[0093] The credibility calculation unit takes graph edges as input, first calculating temporal consistency, semantic fit, device support, and usability separately, and then combining the four into a comprehensive credibility score. Here, temporal consistency does not involve reverting to step one for temporal sorting, but rather evaluates the degree of temporal convergence of the unified evidence object at the current step node; semantic fit is not abstract semantic similarity, but rather evaluates whether the video actions, voice statements, operation logs, and test results within the object collectively point to the same skill action.

[0094] Device support score is not the overall device quality score, but rather an evaluation of whether the current device node is sufficient to support the unified evidence object in the profile; usage permission score follows the consent and de-identification status in step one, and is used to limit the subsequent use of the unified evidence object. Subsequently, the gating update unit performs write, review, or blocking based on the range of the comprehensive credibility score.

[0095] The credibility calculation unit first constructs the overall credibility. :

[0096]

[0097] In the formula, the overall credibility is... : Used to characterize the overall credibility of unified evidence objects entering the profile update chain, with a value ranging from 0 to 1; time consistency. : The degree of temporal convergence of the unified evidence object at the current step node, with a value ranging from 0 to 1; semantic fit The degree to which multimodal fragments within a unified evidence object collectively point to the current skill node, with a value ranging from 0 to 1;

[0098] Equipment support The current device node's source support for this unified evidence object ranges from 0 to 1; device support level. The equipment status table represents the equipment's status and includes at least the following: calibration status, online status, fault status, and status of the most recent maintenance. If the equipment is properly calibrated and online, its support level is [not specified]. The value is relatively large; if the equipment is not troubleshooted or is expired, the equipment support will be insufficient. Smaller.

[0099] Permissions The permissible value for existing unified evidence objects entering subsequent links under both consent and anonymized states ranges from 0 to 1; usage permissibility. The consent and anonymization states are determined by a state mapping table. The preferred state mapping relationship is: consent state allows, and anonymization state satisfies: Consent status is allowed, but desensitization status is insufficient: Take the median value; agree that the state is not allowed for training. Weighting coefficients Weighting coefficients Weighting coefficients Weighting coefficients All greater than And satisfy .

[0100] Among them, time consistency Based on the time of the event Time of collection arrival Processing time This is given together with the boundary of the current step node;

[0101] semantic fit Generated by a rule engine, time series aligner, or multimodal embedding solver; device support. Generated by device node status, object source modality, and current step requirements; usage permission. Following step one are the consent and desensitization states. If a unified evidence object has sufficient temporal convergence but semantic fit... Low overall credibility It will not be improved by the integrity of a certain boundary; if the semantic fit of a unified evidence object is... Higher and more permissible If the value is too low, its subsequent use will be limited to non-portrait update channels.

[0102] During medication verification training, trainees first check the label on the medication bag, then verbally state the medication name and dosage, and finally click to confirm on the terminal. If video clips, audio clips, and terminal click logs all revolve around the same medication bag, but the verbal description includes another patient's name, then the temporal consistency of the unified evidence object is considered. and equipment support The semantic fit remains high, while the semantic fit is still high. It will decrease. Overall credibility Therefore, it will not enter the writable region. The subsequent gating update unit will transfer the object to the verification instead of writing it to the skill node where the medicine preparation verification is completed.

[0103] When using it, consider overall credibility. It is not an amplified version of a single mass, but rather the result of a combined effect across time, semantics, devices, and usage boundaries.

[0104] After obtaining the overall credibility Then, the gating update unit updates based on the low threshold. and high threshold Triage is performed on the unified evidence object. Based on the overall credibility... Above the high threshold And allowance If the writing conditions are met, the gating update unit writes the unified evidence object to the skill node associated with the current step node and simultaneously updates the step record in the student profile node; if the overall credibility is... At a low threshold With high threshold In the interim, the gating update unit does not perform a direct write, but instead retains the unified evidence object in the review band; if the overall credibility is... Below the low threshold If the unified evidence object is not included in the profile update chain, then the unified evidence object will not enter the profile update chain. For high-risk steps, the gating update unit uses a higher threshold. For low-risk steps, the gated update unit uses a wider verification band to avoid temporary loss causing link interruption.

[0105] In rail transit maintenance training, pantograph inspection is considered a high-risk procedure, and the gating update unit will configure a high threshold for this type of procedure. This ensures that unified evidence objects are only written to skill nodes when the video action, spoken item, and tool status all align. In office software maintenance training, log viewing is a low-risk step, while the gated update unit allows unified evidence objects to enter the review zone even with partial audio loss. If an alternative implementation is used, semantic fit will be compromised. It can be formed by rule-based semantic slot matching or by multimodal embedding matching; device support. This can be provided by a device status table or by a static mapping between device nodes and source modalities; the gating update unit can run on a graph database or on a relational database with a cached mapping table. As long as the overall reliability is considered... Four-way synthesis and low threshold High threshold The dual-threshold shunt logic remains unchanged, meaning they are parallel implementations under the same inventive concept.

[0106] In practice, gating updates no longer treat the unified evidence object as a data entry to be written immediately upon collection, but rather as a restricted carrier to be written only after thorough calculation. Therefore, the writing of step nodes and skill nodes has clear preconditions, and the student profile node obtains filtered step records.

[0107] Step 3: After the unified evidence object has entered the cross-modal evidence credibility graph and completed the initial gating, the state offset caused by retransmission, out-of-order transmission and conflict is locally corrected so that the subsequent steps receive a set of step states that has been converged and does not fluctuate violently.

[0108] Step two has already categorized unified evidence objects into writable objects, verification objects, and bypass objects. However, this categorization is based on the set of objects that have already arrived at that time. Once the edge cache unit retransmits new unified evidence objects after the link is restored, or if a device node delays uploading a segment due to short-term jitter, the original traffic splitting results may be overturned. If all step nodes and skill nodes are recalculated at this point, not only will the state within the graph fluctuate repeatedly, but stable steps unrelated to the currently late-arriving objects will also be dragged into the recalculation.

[0109] The data processing server first reads the event occurrence time from the unified evidence object. Time of collection arrival Processing time Then retrieve the time traction amount generated in step one. and the overall credibility formed in step two The late arrival evidence detection unit does not solely rely on arrival time being later than the current time as the sole criterion. Instead, it simultaneously considers the step nodes, skill nodes, and student profile nodes currently connected to the unified evidence object to determine if the object has structurally missed its original update window. If the unified evidence object arrives late but maintains a stable connection with its original step nodes, the system does not immediately expand the correction scope; however, if the unified evidence object not only arrives late but also has a strong tendency to merge... The original steps are still relatively high, and the object completeness is high. If the level remains high, the system will prioritize treating it as a late arrival that should be replenished; if the unified evidence object arrives late and the merging potential is high... If the value has already decreased, the system treats it as an edge-late arrival object and only retains the ability to perform local bypass correction.

[0110] The late arrival evidence detection unit does not directly use the time of collection arrival. Whether the judgment is made later than a certain absolute moment, and using time-dependent measures. The overall credibility already established in step two Perform a late compression to obtain the compression reliability before entering the local correction chain:

[0111]

[0112] In the formula, the original comprehensive credibility This represents the overall credibility calculated in step two, with a value ranging from 0 to 1; the overall credibility after late compression. This represents the overall credibility after considering the impact of lateness, with a value ranging from 0 to 1; merging potential. The degree to which the unified evidence object maintains a convergent relationship with the original candidate step, with a value ranging from 0 to 1;

[0113] Object completeness The degree to which the modalities contained in the unified evidence object accept the action in this step, with a value ranging from 0 to 1; time traction. Continuing from the definition in step one, this is used to characterize the degree to which the unified evidence object is affected by link lag, and its value ranges from 0 to 1.

[0114] The overall credibility after late arrival compression Still at a high threshold When the late arrival evidence detection unit sends the unified evidence object into the core channel of the local correction chain, the overall credibility after late compression is... Between low threshold With high threshold In between, the system sends it to the buffer channel of the local correction chain; when the overall reliability after late compression... Below the low threshold At this time, the system will no longer trigger the active rewriting of step nodes and skill nodes, but will only retain the relationships in the historical record chain.

[0115] In one embodiment, trainees in substation maintenance training first complete the voltage testing procedure, then verbally confirm the information, and finally, the contact status is uploaded by the insulation tool. If the insulation tool's upload arrives later due to wireless obstruction within the substation, the data processing server will not re-push the entire session into segmentation and mapping; instead, it will first read the previous merging potential of the unified evidence object. Object completeness and overall credibility If these quantities indicate that the object was originally close to the power testing completion step node, then the object is sent to the core channel, and only the step nodes, skill nodes, and student profile nodes related to power testing completion are identified as affected subgraphs; nodes unrelated to subsequent steps such as tagging and locking or power restoration are not included in this round of correction chain.

[0116] When using it, once the unified evidence object enters step three, the process based on... , , and The compression decision limits subsequent corrections to local subgraphs where existing connections remain stable. This preserves the supplementary value of late arrivals to the original steps while avoiding global disturbances.

[0117] After identifying the affected subgraph, the incremental correction unit does not reconstruct the entire cross-modal evidence credibility graph. Instead, it generates a differential correction package along the existing link of unified evidence object - step node - skill node - student profile node. The differential correction package carries at least the object identifier of the late-arriving unified evidence object, the original connected step node, the current connected step node, the original write state, the state to be corrected, and the source device node. Subsequently, the incremental correction unit first updates the connection weights between the step node and the unified evidence object, then updates the skill nodes directly associated with that step node, and finally writes the changes to the student profile node. If the late-arriving unified evidence object only strengthens the original step conclusion, the differential correction package performs incremental overlay; if the late-arriving unified evidence object changes the original step conclusion, the differential correction package performs limited rollback before writing; if the late-arriving unified evidence object is in the buffer channel, the differential correction package only updates the review band state and does not directly rewrite the valid step records in the student profile node.

[0118] In software operations and maintenance training, trainees first execute log screening commands on the console, then send screenshots of the fault page via their browsers, and finally, the monitoring agent re-uploads service recovery logs. If the service recovery logs arrive later than the screenshots, the incremental correction unit will not recalculate the entire fault handling process, but will only perform a write-back within the log screening-anomaly location-service recovery chain. If the recovery logs prove that a recovery action has indeed been performed, the trainee profile record, which was previously stuck at the anomaly location step node, is advanced to the service recovery step node; if the recovery logs are inconsistent with the preceding browser screenshots, the system only places the object in the review zone, preventing it from directly rewriting the original conclusion. For example, the differential correction package can be generated by an edge updater on a graph database, or by a relational database with a cached index. As long as the update order maintains a single-chain structure of first the step node, then the skill node, and finally the trainee profile node, it belongs to an equivalent implementation path.

[0119] In practice, step three transforms the late arrival link into a local subgraph correction problem, while utilizing the overall reliability after late arrival compression. The constraint correction entry point is used. This ensures that the late arrival of the unified evidence object will not trigger a global recalculation; instead, it will be compressed, defined, and written back first, thus maintaining the same step-state.

[0120] After the late-arriving unified evidence has been identified and a local correction chain has been formed, another common scenario is that multiple unified evidence objects point to the same node in terms of time and steps, but their conclusions are inconsistent. For example, video footage shows that the trainee has completed the device reset, but the audio footage is still verbally describing the previous step's check items, or the operation log shows that a certain button has been pressed, but the device log does not provide corresponding feedback. If the system simply selects the latest arriving object or the highest quality object in such conflicting scenarios, the conflict resolution will degenerate into a single indicator trade-off, causing step nodes and skill nodes to frequently jump in high-risk scenarios.

[0121] Therefore, while maintaining the local correction range, conflict resolution and threshold hysteresis are used to make the state within the graph tend to converge rather than oscillate.

[0122] The data processing server first reads all unified evidence objects within the current local correction chain, checking one by one whether they are all attached to the same step node and point to the same skill node. If so, the conflict evidence resolution unit continues to compare the source support relationship of these unified evidence objects at the device node, the temporal convergence relationship at the step node, and the semantic fit relationship at the skill node. No new credibility is reinvented here; instead, the comprehensive credibility already established in step two is utilized. Simultaneously, combining the current delayed compression state and local correction state from step three, the low threshold is... and high threshold The implementation of hysteresis bundles prevents conflict unification evidence from immediately overturning its original state upon the arrival of a new object. For high-risk steps, hysteresis bundles tend to maintain the existing stable state; for low-risk steps, hysteresis bundles tend to allow new objects to enter the review zone.

[0123] The gating update unit does not always use the low threshold from step two. and high threshold Instead of directly addressing the conflict levels already present in the current local correction chain, a threshold convergence is performed to give the gating boundary a hysteresis characteristic. The processing method is as follows:

[0124]

[0125] In the formula, the high threshold : Represents the high threshold after conflict convergence, used to restrict conflict unification evidence objects from directly entering step nodes and skill nodes, with a value ranging from 0 to 1; Object completeness The degree of acceptance of the current unified evidence object in multimodal modes, with a value ranging from 0 to 1;

[0126] Merging power : The degree of convergence between the current unified evidence object and the target step node, with a value ranging from 0 to 1.

[0127]

[0128] In the formula, the low threshold After the conflict is resolved, the threshold for unified evidence objects that may remain within the review zone is between 0 and 1, representing the time traction. The degree of lag in the chain of unified evidence objects is rated from 0 to 1; overall credibility. The credibility of the unified evidence object is compressed, ranging from 0 to 1.

[0129] Threshold convergence is triggered only when there are two or more consistent evidence objects at the same step node and their conclusions are inconsistent; the high threshold after convergence and low threshold It is only effective in the current local correction chain and does not cover other step nodes.

[0130] When high threshold After being moved up, conflict unification evidence objects that were originally only slightly above the old high threshold will no longer be directly written to the step node, but will first enter the review zone; when the low threshold... After being moved upwards, conflict-related unification evidence objects that were previously hovering on the edge of the review zone will be bypassed more quickly. For high-risk steps, the gating update unit preferably moves the higher threshold upwards again based on the above convergence. For low-risk steps, maintain a low threshold. The length of the buffer band remains constant so that more boundary objects can remain within the core band.

[0131] During intravenous puncture training for medical staff, video clips show the needle tip already inserted into the vein, while audio clips still verbally describe disinfection procedures, and the training arm log simultaneously indicates blood return status. Without threshold hysteresis, the system will jump between "puncture complete" and "still in the preparation stage." With the above convergence, the unified evidence object formed by the training arm log and video clips, if the object completeness... Harmony and Integration If it is high enough, it can still maintain a high threshold. Above; while supported only by voice fragments, but with overall credibility Those with low levels of unified evidence were then pushed down to a low threshold. The process then moves to the verification zone. This prevents conflicting objects in the same step from simultaneously vying for write rights.

[0132] When used, the handling of conflict unification evidence objects no longer depends on the intuitive choice of which is newer or stronger. Instead, it compresses the conflict into a state range that can be handled in a hierarchical manner through threshold hysteresis. As a result, high-risk steps remain stable and low-risk steps are not prematurely blocked.

[0133] After completing the threshold hysteresis, the conflict evidence resolution unit continues to determine whether the current local correction chain has entered the evidence conservatism mode. If the missing mask in the unified evidence object shows a persistent absence of a key modality, or if the overall credibility of multiple unified evidence objects at the same step... Continuously falling at low threshold and high threshold In between, the gating update unit switches the current step node to evidence conservation mode. As a supplement: assuming the key modality missing ratio... for:

[0134]

[0135] Where: Key mode missing ratio This indicates the degree of missing key modalities in the current unified evidence object, with a value ranging from 0 to 1;

[0136] Modal weights and Existence Indicators Continuing with the previous definition, when the critical mode is missing more than... When the threshold for missing data is exceeded, the step node switches to the evidence conservative mode.

[0137] The evidence-conservative mode does not provide an automatic confirmation process, but only retains the processing chain of supplementary testing, review, or waiting for subsequent objects to arrive. When the training boundary changes, step three no longer allows the affected unified evidence objects to participate in the profile update, stops their subsequent writing eligibility, and only retains the historical version chain that has been connected.

[0138] If the unified evidence object at the same step node continues to fall within the current local correction chain... and If the number of consecutive occurrences between these thresholds reaches a preset number, the system will switch to the evidence-conserving mode.

[0139] In rail transit braking system inspection training, brake pad thickness images, distance sensing records, and verbal descriptions are often used simultaneously for judgment. If distance sensing records are temporarily missing, and there is a conflict between the verbal description and the image fragment, this step will switch to an evidence-conserving mode. The system will only retain subsequent actions such as supplementing brake pad thickness or manually reviewing the image, without advancing the current state to inspection completion. In another generalized implementation, the evidence-conserving mode can be implemented by a state machine or a rule engine; freezing the writing qualification can be achieved through graph edge state bits or by writing to a whitelist based on the profile. As long as the basic link remains unchanged—stopping automatic advancement when the missing data is too heavy or the conflict is not resolved, and stopping further writing after the boundary changes—this is a parallel implementation method for this step.

[0140] When in use, step three not only handles late-arriving unified evidence objects, but also conflicting unified evidence objects and boundary change objects. It will not force progress due to incomplete evidence, nor will it immediately reverse the state when a new object enters. The subsequent step four can also generate a supplementary training sequence based on the current state.

[0141] Step 4: Based on the stable state set formed in Step 3, generate a sequential training sequence and rewrite the subsequent execution results back into the original graph structure, so that the step nodes, skill nodes and trainee profile nodes form a complete closed loop, instead of staying at a one-time judgment result.

[0142] While step three has compressed late-arriving and conflicting unified evidence objects into a stable state set, this set still only represents a static result of which steps are currently reliable, which require review, and which skills are not yet satisfied. If the system directly lists all unsatisfied skills at once, trainees will receive a retraining task list lacking sequential relationships. If the system only outputs the current step nodes in chronological order, it will ignore the dependencies between high-risk steps and prerequisite capabilities. Therefore, converting the current state set into an executable retraining sequence allows for the assessment of weak skill levels, step risk levels, prerequisite capability satisfaction, and overall reliability. The sequence generation is influenced by both recent retraining gains and other factors, rather than being dominated by any single factor.

[0143] The data processing server first reads the current status, preceding step status, and risk level of each step from the step node. Then, it reads the current proficiency gap for the corresponding skill from the skill node, and the recent retraining gains and recent retest records from the student profile node. Subsequently, the risk perception and recommendation unit breaks down whether the current step needs retraining and how it should be retrained into two layers. The first layer calculates the retraining pull of the step or process within the current session, while the second layer, given the determined pull, arranges the micro-lessons, operation demonstrations, re-operation tasks, retest tasks, and preceding step retraining items in sequence according to their order of priority. The purpose of this approach is to first determine whether retraining is necessary, then how and first, ensuring the sequence generation process remains a single-chain closed loop.

[0144] The risk perception recommendation unit calculates the retraining traction value for each step node. This is used to characterize the strength of this step entering the pre-training sequence. (Training pull value) Based on overall credibility Step-by-step risk level Precondition satisfaction Weakness and recent refresher training benefits Together they form:

[0145]

[0146] In the formula, the supplementary training traction value : Used to characterize the priority of the current step entering the pre-training sequence, with a value greater than or equal to 0; Overall confidence level : Indicates the confidence level of the current step's status after late correction and conflict resolution, with a value ranging from 0 to 1; Step risk level The risk level corresponding to an error in the current step, ranging from 0 to 1;

[0147] Preconditions The completion level of the prerequisite steps and the satisfaction level of the prerequisite skills for the current step are both rated from 0 to 1; prerequisite satisfaction level. The current step is composed of the closed states of all preceding step nodes and the satisfied states of the preceding skills. Ideally, the prerequisite satisfaction level is achieved when all preceding steps are closed and all preceding skills meet preset thresholds. High; conversely, low.

[0148] Weakness The degree of gap between the current skill node and the target state is taken as 0 to 1, representing the weakness. It is represented by the difference between the current skill node state and the target skill state. The current skill node is based on a level system and is normalized according to the level difference. The current skill is a continuous score and is normalized according to the difference between the target value and the current value.

[0149] Recent Refresher Training Benefits The convergence of the most recent or several retraining sessions on similar steps, expressed as 0 to 1. Recent retraining benefits. The degree of improvement in the status of this step is determined by the most recent or several recent retraining sessions. For example, if the skill gap has significantly narrowed after the most recent similar retraining, then the recent retraining is considered effective. If the benefits increase, otherwise there will be no significant change in condition after supplementary training, then the benefits of recent retraining will be limited. It is a low value.

[0150] When supplementary training pull value When the risk perception recommendation unit is high, it places this step at the beginning of the retraining sequence; when the retraining traction value is high... When the value is in the middle range, the system further judges based on the status of the review band; when the supplementary training traction value is... When the level is low, the system does not prioritize this step for retraining in the current session. Overall reliability This is not about reducing the need for supplementary training, but rather about increasing the intensity of this supplementary training based on solid evidence; prior satisfaction. The lower the value, the more it indicates that although the current steps seem to have significant gaps, it is not advisable to directly proceed to advanced retraining; instead, priority should be given to retraining the previous steps. (Recent retraining benefits) The higher the value, the more likely it is that similar retraining has just occurred and has already concluded, so this step will not be repeated at the beginning of the sequence in the same round of sessions.

[0151] In one embodiment, rail transit vehicle door maintenance training includes four steps: power outage confirmation, mechanical locking, door disassembly, and gap retesting. The trainee's overall reliability in the mechanical locking step is considered important. High risk level, step-by-step High, prerequisite satisfaction High degree of weakness Clearly, this step will be placed at the beginning of the retraining sequence; although the disassembly of the gate also has weaknesses. However, its prerequisite satisfaction A low value indicates that the upstream mechanical lock has not yet been fully resolved. Therefore, the risk perception recommendation unit will not output the door disassembly and reoperation task first, but will instead prioritize the mechanical lock retraining item. This results in an order that is not simply ranked by risk or by the degree of weakness, but rather pulls risk, credibility, pre-requisites, and benefits into a traction chain.

[0152] When used, the order of the training sequence is no longer a flat arrangement, but rather determined by the training pull value. Make decisions in advance. This generates a sequence that more closely approximates the actual gaps in the current state set and avoids directly pushing steps that have not yet met the preconditions into execution.

[0153] After completing the supplementary training traction value After calculation, the risk perception recommendation unit continues to assemble the supplementary training content corresponding to each step into a supplementary training sequence. This assembly follows a single-chain relationship: if the prerequisite satisfaction is met... If the threshold is lower than the corresponding limit, the system first installs the supplementary training items for the preceding steps, and then installs the operation demonstration or re-operation task for the current step. If the current step is in the review zone, the system first installs the supplementary test task or manual review prompt, and then decides whether to install the supplementary training items for the current step based on the review return result. If the current step has already met the conditions for entering the execution phase, the system assembles the supplementary training items in the same step in the order of instruction materials - on-site demonstration - re-operation - re-test. After receiving the supplementary training sequence, the training terminal displays it in the form of a list of step items, a list of process items, or a task card with illustrations. Each item in the same sequence carries the recommendation basis, evidence chain, risk explanation, audit index, and model version information, which facilitates the closure of the subsequent write-back chain on the same object.

[0154] Taking hospital infusion verification training as an example, when the system determines the retraining traction value for the patient identity verification step... When the drip rate exceeds the confirmation step, the training terminal first displays supplementary training items to verify the patient's identity. If the current step is still within the verification zone, the training terminal first displays supplementary testing tasks, such as rescanning the wristband, rereading the medication bag information, and waiting for the results from the retesting terminal. If the current step has left the verification zone, the training terminal continues to display re-operation tasks, such as completing the wristband verification again and performing a second confirmation in a predetermined order on the operating table. For the same system concept, there are also parallel implementation methods: the supplementary training sequence can be generated by a rule engine, or by a graph searcher based on step dependencies, or by a sequence planner combining historical supplementary training benefits. The training terminal can be a fixed workstation terminal, a tablet terminal, or a head-mounted terminal. As long as the output object remains a training sequence with a step order and carries the same field system, it belongs to an equivalent implementation method.

[0155] Step four converts the stable state set from step three into an executable training sequence. Training pull value. Steps that are high-risk, high-reliability, and meet the prerequisites are allowed to enter the beginning of the sequence. Steps that do not meet the requirements or whose prerequisites are not met are automatically moved to the end or diverted. The training terminal provides a sequential execution chain.

[0156] After the supplementary training sequence has been distributed to the training terminal, if the system only remains in the state of "supplementary training items distributed," the entire solution is still just a one-time recommendation and cannot form a complete closed loop. What high-risk job training truly needs is for the system to know which steps have been completed due to the supplementary training, which steps are still stuck in the original state, and which steps need to return to the previous state due to retesting or job confirmation results.

[0157] Therefore, the execution results of the training terminal, the retest results of the retest terminal, and the job confirmation results of the job confirmation terminal are returned to the step node, skill node, and trainee profile node, so that the current session tends to close through continuous backwriting.

[0158] After the supplementary training sequence is distributed, the data processing server continuously receives execution results from the training terminals, retest results from the retest terminals, and job pass results from the job confirmation terminals. The profile writing unit does not directly pile these results into the trainee profile node one by one; instead, it first determines which step node, skill node, and unified evidence object chain segment each result corresponds to.

[0159] If the returned result only indicates that a certain supplementary training item has been executed, but has not yet corresponded to the retest conclusion, the profile writing-back unit only updates the execution status and not the skill status; if the retest result is consistent with the target conditions of the original step node, the profile writing-back unit writes back sequentially along the step node - skill node - trainee profile node; if the job pass result and the retest result both indicate that the same process has been closed, the profile writing-back unit continues to update the session-level completion status and prepares to trigger the exit judgment. The purpose of this processing is to write back the three types of results—"done," "tested," and "passed"—in a layered manner, rather than mixing them into a single, general completion marker.

[0160] The image write-back unit first generates a write-back gain for each executed supplementary training item. This is used to indicate the strength of the correction that the result of this round makes to the state of the original step:

[0161]

[0162] In the formula, the write-back gain : Used to characterize the correction strength of a certain supplementary training item's execution result to the current step node and skill node, with a value ranging from 0 to 1; result conformity. The degree of conformity between the training terminal execution result, the retest terminal retest result, or the job confirmation terminal job pass result and the target step conditions, with a value ranging from 0 to 1; result conformity. The result comes from one or a combination of three sources: training terminal execution result; retesting terminal retest result; and job confirmation terminal job pass result. The training terminal execution result primarily indicates whether the task was performed; the retesting terminal retest result primarily indicates whether it was performed correctly; and the job confirmation terminal job pass result primarily indicates whether the job delivery conditions have been met. When all three are present, the results from the retesting terminal and the job confirmation terminal take precedence. .

[0163] Overall credibility Continuing the definition from step three, the credibility of the original step state before entering the retraining phase is taken as 0 to 1; object completeness. : The degree of acceptance of the original unified evidence object in multimodal modes is represented by values ​​from 0 to 1; Supplementary training traction value The priority of this step in the training sequence is represented by a value from 0 to 0.

[0164] When write-back gain When the gain is high, the portrait write-back unit first advances the current step node to a converged state, then reduces the gaps in the skill nodes connected to that step, and finally updates the current round's step record in the student's portrait node; when the write-back gain is high... When in the middle range, the image write-back unit only narrows the skill node gap, but does not immediately mark the step node as closed; when the write-back gain At lower speeds, the system only records states that have been executed but not yet terminated, and sends that step back to the next round of candidate training chains. (Result conformity) The decision to write back is not made in isolation, but rather in conjunction with overall credibility. and object integrity Together they work; similarly, supplementary training pull value The higher the value, the more pronounced the original gap in that step, and therefore the greater the write-back gain per execution. It will not be exaggerated.

[0165] In factory robot training, trainees complete three supplementary training items based on terminal displays: zeroing confirmation, coordinate verification, and writing teaching points. If the retest result of coordinate verification shows that the target point error meets the requirements, the profile write-back unit will first update the coordinate verification step node, then narrow the position verification gap in its corresponding skill node, and finally advance the session record in the trainee profile node to the next state. If the teaching point writing operation is only completed but the retest result has not yet been returned, the system only records the execution status and does not immediately narrow the skill gap. In this way, the write-back link and the supplementary training link maintain the same order, avoiding the reversal of writing the profile first and then supplementing the steps.

[0166] When used, all feedback results after retraining are uniformly converted into write-back gain. The data is written back layer by layer in the order of first step nodes, then skill nodes, and finally student profile nodes. This allows local changes in the current session to accumulate continuously without forming isolated backhaul records.

[0167] After completing the layer-by-layer write-back, the data processing server continues to determine whether the current session has met the exit conditions. The exit conditions do not rely solely on whether there are still tasks, but also consider whether high-risk steps have reached the target state, whether there are still objects to be processed in the review band, and whether there are still unwritten results in the current retraining sequence. If all high-risk steps have reached the target state, and all steps in the review band have obtained a clear conclusion, the session is marked as closed; if there are still high-risk steps that are still in the executed but not yet concluded state, or if there are still objects in the review band that have not reached a conclusion, the data processing server resends these steps into the next round of retraining candidate chain. For sessions that consistently show high misjudgment and high review rates on the same type of steps, the data processing server will also write the result back into the template correction chain to adjust the segmentation parameters in step one or the credibility template in step two. However, this adjustment does not directly change the existing conclusion of the current session, but is used for loading in the next round of session.

[0168] In a preferred embodiment, high-risk steps in petrochemical inspection training include gas detection, valve position verification, and interlock confirmation. If, after supplementary training on valve position verification, the training terminal reports completion, the retest terminal reports passing retest, and the job confirmation terminal reports that the process has met handover conditions, then this step, along with its corresponding skill node, is closed. If interlock confirmation remains at the retest stage and no clear retest conclusion is obtained, the session is not closed, and the system puts the interlock confirmation back into the next round of supplementary training candidate chain. As a generalized implementation, session closure determination can be executed by a state machine or by a graph traverser executing the step chain; the template correction chain can be handled by an offline configurator or an online parameter manager. As long as the order of completing the write-back of the current session before deciding on closure or continuation remains unchanged, they belong to parallel implementations under the same inventive concept.

[0169] In practice, supplementary training is not a one-time, irreversible action, but rather it reverts to the original diagram structure after each execution, retest, and job confirmation. Therefore, session closure is based on the actual convergence of step and skill nodes, rather than on the superficial state of a completed task list.

[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A system for collecting large-scale multimodal training data and constructing intelligent recommendation models, characterized by: include, In response to the start of a training task or the establishment of a session, at least three of the following are collected: video stream, audio stream, operation log, test results, and simulator or device log. The event occurrence time, collection arrival time, and processing time are recorded and encapsulated into a unified evidence object containing segment boundaries, synchronization confidence, and usage status according to the candidate step window. Based on a unified evidence object, a cross-modal evidence credibility graph is constructed, which includes attached step nodes, skill nodes, and trainee profile nodes. The overall credibility is calculated, and the proficiency of skill nodes is updated by gating according to the relationship between the overall credibility and low and high thresholds. For late-arriving unified evidence objects, only the affected step nodes, skill nodes, and recommendation results are subject to local differential correction; supplementary training recommendation results are generated based on the updated proficiency profile, step risk level, and prerequisite ability satisfaction, and the supplementary training execution results or retest results are written back to the step nodes and skill nodes.

2. The model building system according to claim 1, characterized in that: Preprocessing includes performing original timestamp marking, basic denoising, local desensitization, and summary value generation on each modality of data. The summary value and the original fragment are maintained in the edge cache unit. Then, the access gateway combines the session identifier, personnel identifier, and job step identifier to supplement the collection arrival time and processing time.

3. The model building system according to claim 2, characterized in that: The candidate step window is jointly determined by the job step identifier, process identifier, interaction boundary, and time sliding window. The evidence object construction unit only encapsulates the multimodal fragments that belong to the same candidate step window into a unified evidence object, and writes the fragment boundary, modality quality vector, missing mask, and synchronization confidence into the unified evidence object.

4. The model building system according to claim 3, characterized in that: The cross-modal evidence credibility graph takes a unified evidence object as the central node and attaches step nodes, skill nodes, device nodes, and trainee profile nodes respectively. The step nodes are determined based on the training process library, the skill nodes are determined based on the mapping relationship between the step nodes and skill items, and the device nodes are located based on the modal source set.

5. The model building system according to claim 4, characterized in that: The overall credibility is formed by the combination of time consistency, semantic consistency, device reliability and usage permission. The gating update includes: when the overall credibility is higher than the high threshold and the usage permission meets the writing conditions, the unified evidence object is written to the skill node associated with the step node, and the corresponding record in the student profile node is updated synchronously.

6. The model building system according to claim 5, characterized in that: When the overall credibility is between the low and high thresholds, the unified evidence object is transferred to the review zone and its connection with the step node and skill node is maintained; when the overall credibility is below the low threshold, only the object record, source record and version record of the unified evidence object are retained, and the proficiency update is not performed.

7. The model building system according to claim 6, characterized in that: Configure higher and lower thresholds for high-risk steps than for low-risk steps, and prohibit the use of unified evidence objects for proficiency updates in advanced steps when the prerequisite capability satisfaction of the step node does not meet the preset conditions, but instead keep them in the processing chain corresponding to the prerequisite step.

8. The model building system according to claim 7, characterized in that: When a unified evidence object is determined to be a late unified evidence object after comparing its event occurrence time, collection arrival time, and current processing time, only the affected step nodes, skill nodes, and current recommendation results are subject to local differential correction, while the existing states of other unaffected step nodes and skill nodes remain unchanged.

9. The model building system according to claim 8, characterized in that: When multiple unified evidence objects give inconsistent conclusions on the same step node, the unified evidence object that is consistent with the device node state and has a stable connection with the step node is retained first, and the remaining unified evidence objects are transferred to the review zone; when the missing key modality reaches the preset condition, the step node is switched to the evidence conservative mode.

10. The model building system according to claim 9, characterized in that: The supplementary training recommendation results are generated in the following order: supplementary training of the previous step, operation demonstration of the current step, re-operation task of the current step, and re-test task of the current step. The supplementary training execution results, re-test results, and job pass results are then written back to the step node, skill node, and trainee profile node in sequence to determine whether the current training session ends or enters the next round of data collection, evaluation, and correction process.

Images