Man-machine interaction whole-process supervision and evaluation system for VR practical training of basic library station

Abstract

The invention belongs to the technical field of basic library station practical training supervision, and particularly relates to a basic library station VR practical training-oriented human-computer interaction full-process supervision evaluation system, which comprises a multi-modal interaction behavior capture and analysis module, an interaction behavior validity output module, a practical training compliance intelligent judgment module, a practical training task difficulty decision module and a practical training supervision terminal. The comprehensiveness and analysis precision of multi-modal interaction behavior information are guaranteed from the source through a multi-modal interaction behavior capturing and analysis module, a unified interaction specification evaluation dimension is constructed through an interaction behavior effectiveness output module, and a practical training compliance intelligent judgment module recognizes violation steps and quantitatively judges the compliance of a whole practical training task. The practical training task difficulty decision-making module accurately divides complex tasks and simple tasks, comprehensively covers the supervision and evaluation requirements of the whole process of practical training, and provides powerful technical support for standardized development of VR practical training of a basic library station.

Description

Technical Field

[0001] This invention relates to the field of grassroots warehouse station training and supervision technology, specifically a human-computer interaction full-process supervision and evaluation system for VR training in grassroots warehouse stations. Background Technology

[0002] As a key scenario for energy storage, transfer, and operation and maintenance management, the standardization of operating procedures and the emergency response capabilities of personnel at grassroots warehouses directly affect production safety and operational efficiency. VR training technology, with its advantages of zero risk and highly immersive scenario simulation, has become a core means for skills training and capability assessment of personnel at grassroots warehouses. However, as the industry continues to raise its requirements for training quality and regulatory precision, traditional VR training technology for grassroots warehouses is gradually revealing its systemic shortcomings.

[0003] Currently, VR training technologies mostly focus on single-dimensional functional optimization and lack customized full-process supervision design for grassroots warehouse training scenarios. For example, Chinese invention patent with publication number CN110174951A discloses a method for human action recognition based on VR training equipment. It establishes virtual quadrant separators in the VR engine, associates them to form synchronous separator groups, and combines collision judgment between colliders and separators to realize action recognition and tracking. This solves the problems of large computational load and complex algorithms in traditional VR device action recognition and has the advantages of strong versatility and accurate judgment.

[0004] However, the above-mentioned invention still has several limitations: First, the interaction dimension is singular, focusing only on the recognition and tracking of body movements, without involving the capture and analysis of multimodal interaction data such as key voice commands and eye-tracking trajectories in grassroots warehouse training. This results in a one-sided understanding of the trainees' operational behavior and cannot fully reflect their operational standardization and focus. Secondly, the assessment function is limited, only staying at the basic level of "whether the action is accurately identified". It has not established a step-by-step compliance quantitative judgment mechanism for the characteristics of the training process at the grassroots warehouses and stations, and cannot identify and warn against violations in real time. Furthermore, it does not address the difficulty of the training tasks themselves. Third, the lack of a closed-loop supervision mechanism, the absence of a linkage feedback mechanism with the training supervision terminal, and the failure to consider the impact of psychological factors on the training effect make it difficult to extend the training supervision from action recognition to the entire chain of "capability assessment - problem identification - optimization guidance", which is not conducive to improving the quality and efficiency of VR training at the grassroots warehouse stations.

[0005] To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations, in order to solve the technical defects mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations, including a multimodal interaction behavior capture and analysis module, an interaction behavior validity output module, a training compliance intelligent judgment module, a training task difficulty decision module, and a training supervision terminal; During VR training at grassroots warehouses, the multimodal interaction behavior capture and analysis module captures and analyzes the trainees' multimodal interaction behaviors in the VR training scenario, including body movements, voice commands, and eye tracking. The interaction behavior validity output module quantifies the validity of trainees' interaction behavior in the corresponding training steps and generates an interaction behavior validity coefficient. It then sends the interaction behavior validity coefficient of trainees in the corresponding training steps to the training compliance intelligent judgment module. The training compliance intelligent judgment module analyzes and identifies illegal steps in real time and generates early warning information. It also intelligently quantifies the compliance of training tasks and sends the judgment results to the training effectiveness multi-dimensional evaluation module and the training supervision terminal. The training task difficulty decision module will analyze the training performance of all trainees for the corresponding training task, mark the corresponding training task as a complex task or a simple task based on the analysis, and send the difficulty marking information of the corresponding training task to the training supervision terminal.

[0008] Furthermore, the multimodal interaction behavior capture and analysis module is equipped with multimodal sensors including motion capture cameras, voice acquisition microphones, and eye-tracking devices. When trainees conduct VR training, it simultaneously captures the trainees' hand gestures and walking paths, verbal commands for operating the devices, and raw data on the area and duration of their gaze on the devices. A multimodal fusion parsing algorithm is used to sequentially denoise, extract features, and semantically map the raw data. Specifically, for body movements, specific operational behaviors are identified through skeletal point tracking and operation action template matching. For speech data, speech recognition and intent classification models are used to parse the target and semantics of instructions. For eye movement data, gaze point clustering and dwell time analysis are used to determine the level of attention that personnel pay to key equipment and risk areas.

[0009] Furthermore, the method for obtaining the effectiveness coefficient of interactive behavior is as follows: Obtain all training steps of the training task in the current VR training scene and mark the corresponding training steps as i, i={1,2,…,m}; when the trainee performs training step i, collect the total number of limb movement frames Atotal of the trainee in the corresponding step training process through the motion capture camera, and mark the number of movement frames that match the standard operation movement as Amatch. Furthermore, the total duration Ltotal of voice commands during the corresponding training steps is recorded through a voice acquisition microphone, and the duration of commands that conform to semantic specifications is marked as the valid voice command duration Lvalid. The eye-tracking gaze conformity Ssemantic of the trainees during the corresponding training steps is obtained through eye-tracking gaze decision analysis. The interaction behavior effectiveness coefficient Ei of the trainees in training step i is calculated using the interaction behavior effectiveness calculation formula.

[0010] Furthermore, the specific analysis process of eye-tracking fixation decision analysis is as follows: The total fixation time (ttotal) of trainees during the corresponding training steps was collected. The duration of the key operation area or risk warning area of ​​the eye-tracking fixation device was marked as tkey. The matching degree (omatch) between the fixation sequence and the standard operation sequence was also obtained. The value of omatch is 0 or 1. If the fixation sequence matches the standard operation sequence, the value of omatch is 1. Otherwise, the value of omatch is 0. The eye-tracking fixation conformity (Ssemantic) was calculated by the semantic conformity calculation method of the key area of ​​eye-tracking fixation.

[0011] Furthermore, the specific operation process of the intelligent compliance judgment module for practical training includes: Retrieve the operation process standards of all pre-stored training steps. When the interaction behavior validity coefficient Ei is received from the interaction behavior validity output module, obtain the actual operation process order of the trainee performing training step i and assign the sequence compliance mark Iorder. The value of Iorder is 1 or 0. If the operation process order is compliant, the value of Iorder is 1, otherwise the value of Iorder is 0. Furthermore, the actual time spent by trainees on training step i (Tactual) is obtained, and the standard time spent on training step i is marked as Tstd; the step-level compliance coefficient Cstepi is calculated using the step-level compliance analysis formula, and the step-level compliance coefficient Cstepi is compared with the preset step-level compliance coefficient threshold Cmin. If Cstepi < Cmin, then training step i is judged to be a non-compliant step and a warning message is generated; the compliance of the training task is quantitatively determined based on the compliance performance of all training steps.

[0012] Furthermore, the quantitative determination process for the compliance of practical training tasks is as follows: The number of violations is obtained and its ratio to the total number of training steps m in the training task is calculated to obtain the violation status value. The average of the step-level compliance coefficients of all training steps in the training task is calculated to obtain the compliance assessment value. The violation step status value and compliance assessment value are compared with the preset violation step status threshold and the preset compliance assessment threshold respectively. If the violation step status value exceeds the preset violation step status threshold or the compliance assessment value does not exceed the preset compliance assessment threshold, the training task performed by the trainee is determined to be non-compliant and an early warning message is generated.

[0013] Furthermore, the specific analysis process of the training task difficulty decision module is as follows: The system obtains the total number of trainees who will undertake the corresponding training tasks, and marks the number of trainees whose training tasks are non-compliant as training failure test values. The system calculates the ratio of training failure test values ​​to the total number of trainees to obtain the training failure coefficient. The system compares the training failure coefficient with the preset training failure coefficient threshold. If the training failure coefficient exceeds the preset training failure coefficient threshold, the corresponding training task is marked as a complex task.

[0014] Furthermore, if the training discrepancy coefficient does not exceed the preset training discrepancy coefficient threshold, the average value of the violation steps of all trainees is used to calculate the violation characteristic value, and the ratio of the preset compliance assessment threshold to the corresponding compliance assessment value of the trainees is used to calculate the compliance inverse value, and the average value of the compliance inverse values ​​of all trainees is used to calculate the compliance characteristic value. The task difficulty decision value is calculated by weighting and summing the training difference coefficient, violation characteristic value, and compliance characteristic value. The task difficulty decision value is then compared with a preset task difficulty decision threshold. If the task difficulty decision value exceeds the preset task difficulty decision threshold, the corresponding training task is marked as a complex task; if the task difficulty decision value does not exceed the preset task difficulty decision threshold, the corresponding training task is marked as a simple task.

[0015] Furthermore, the training task difficulty decision module communicates with the psychological stability analysis module. After marking the corresponding trainees as having simple tasks, the training task difficulty decision module sends the information of trainees whose training tasks in simple tasks are non-compliant to the psychological stability analysis module. The psychological stability analysis module marks the trainees whose training tasks in simple tasks are non-compliant as target personnel and analyzes the psychological stability performance of the target personnel in the corresponding training tasks. Through analysis, it determines whether the target personnel are psychologically unqualified and sends the psychological stability analysis results of the target personnel to the training supervision terminal.

[0016] Furthermore, the specific analysis process of the psychological stability analysis module is as follows: The average heart rate fluctuation amplitude ΔHR of the target personnel during the training process is obtained and the heart rate fluctuation safety threshold is marked as ΔHRmax. The average skin conductance activity data of the target personnel during the training process is obtained and marked as EDAavg, and the skin conductance safety threshold is marked as EDAmax. The psychological stability coefficient is calculated and compared with a preset psychological stability coefficient threshold. If the psychological stability coefficient exceeds the preset psychological stability coefficient threshold, the corresponding target personnel are judged to be psychologically unqualified and an early warning message is generated.

[0017] Compared with the prior art, the beneficial effects of the present invention are: In this invention, the multimodal interaction behavior capture and analysis module ensures the comprehensiveness and accuracy of multimodal interaction behavior information from the source; the interaction behavior validity output module constructs a unified interaction standard evaluation dimension; the training compliance intelligent judgment module identifies illegal steps and quantifies the overall compliance of the training task; and the training task difficulty decision module accurately classifies complex and simple tasks. This comprehensively covers the supervision and evaluation needs of the entire training process, providing strong technical support for the standardized development of VR training in grassroots warehouses and stations. It has a high level of intelligence and significantly improves the accuracy, effectiveness, and pertinence of training supervision.

[0018] In this invention, a psychological stability analysis module is used to calculate the psychological stability coefficient for non-compliant trainees in simple tasks by combining the average heart rate fluctuation range and skin conductance data. This accurately identifies trainees who fail due to psychological factors, providing a clear basis for subsequent targeted psychological quality training. This helps these trainees improve their psychological state during training, further enhancing the pass rate and overall training ability of the trainees. Attached Figure Description

[0019] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a system block diagram of Embodiment 1 of the present invention; Figure 2 This is a system block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1

[0021] like Figure 1 As shown, the human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses proposed in this invention includes a multimodal interaction behavior capture and analysis module, an interaction behavior validity output module, a training compliance intelligent judgment module, a training task difficulty decision module, and a training supervision terminal. During VR training at grassroots warehouses, the multimodal interaction behavior capture and analysis module captures and analyzes the multimodal interaction behaviors of trainees in VR training scenarios, including body movements, voice commands, and eye movements. It can comprehensively and accurately analyze the multi-dimensional interaction behaviors of trainees, providing detailed evidence for subsequent "interaction effectiveness assessment" and "training compliance judgment". This ensures the accuracy of subsequent analysis results from the source and avoids evaluation bias caused by incomplete data or rough analysis, thus ensuring the accuracy of subsequent analysis results. Specifically, the multimodal interaction behavior capture and analysis module is deployed with a multimodal sensor including a motion capture camera, a voice acquisition microphone, and an eye-tracking device. When trainees conduct VR training, it simultaneously captures their raw data such as "gesture trajectory and walking path of operating the device" (body movements), "verbal instructions for operating the device, such as opening the oil valve" (voice), and "area and duration of eye gaze" (eye movement trajectory). Next, a multimodal fusion parsing algorithm is used to denoise, extract features, and semantically map the raw data sequentially. Specifically, for limb movements, specific operational behaviors are identified through skeletal point tracking and operation action template matching, such as identifying specific operational behaviors like "valve opening / closing" and "pipe connection / disconnection." For voice data, speech recognition and intent classification models are used to analyze the target and semantics of commands, such as analyzing the target and semantics of voice commands like "increase pressure to 1.2 MPa." For eye-tracking data, gaze point clustering and dwell time analysis are used to determine the level of attention personnel pay to key equipment and risk areas.

[0022] The interaction behavior effectiveness output module quantifies the effectiveness of trainees' interaction behavior in the corresponding training steps and generates an interaction behavior effectiveness coefficient. It then sends the interaction behavior effectiveness coefficient of trainees in the corresponding training steps to the training compliance intelligent judgment module, constructs a standardized interaction behavior evaluation dimension, reasonably defines the standardization level of interaction behavior in each step, and provides objective and quantifiable data support for the training compliance judgment module. This effectively avoids the problem of ambiguity in compliance judgment caused by the lack of a unified effectiveness judgment standard, and further consolidates the data analysis foundation for training supervision. The method for obtaining the effectiveness coefficient of interactive behavior is as follows: Obtain all training steps of the training task in the current VR training scene and mark the corresponding training steps as i, i={1,2,…,m}; when the trainee performs training step i, collect the total number of limb movement frames Atotal of the trainee in the corresponding step training process through the motion capture camera, and mark the number of movement frames that match the standard operation movement as Amatch. The process involves capturing limb motion frame sequences using a motion capture camera and matching them frame by frame with a "standard operation motion frame template library" (containing standard motion frame sequences such as valve opening and pipe connection). When the similarity is ≥0.8, the frame is considered to be successfully matched, and the cumulative number of successfully matched frames is Amatch. Furthermore, the total duration Ltotal of voice commands during the corresponding training steps is recorded using a voice acquisition microphone (calculated directly from the audio timestamp), and the duration of commands that conform to the semantic specifications is marked as the valid voice command duration Lvalid. The voice data is converted into text through speech recognition and then matched with the "standard instruction text library" (including standard instructions such as "open the oil valve to 90 degrees") using the BERT semantic similarity algorithm. When the similarity is ≥0.7, it is determined to be a valid instruction, and the cumulative duration of valid instructions is Lvalid). The total fixation time ttotal of trainees during the corresponding step training process is collected, the duration of key operation area or risk warning area of ​​eye-tracking fixation device is marked as tkey, and the matching degree omatch between fixation order and standard operation order is obtained. The value of omatch is 0 or 1. If the fixation order matches the standard operation order, the value of omatch is 1, otherwise the value of omatch is 0. The eye-tracking gaze conformity score (Ssemantic) is calculated using a semantic conformity calculation method for key eye-tracking gaze regions. The specific formula is as follows: Where η is a preset weight value, preferably η=0.6; The effectiveness coefficient Ei (dimensionless, value range [0,1], the closer to 1, the more compliant the interaction behavior is with the standard) of the trainee's interaction behavior in step i is calculated using the following formula: Where w1, w2, and w3 are preset weight values, and w1 + w2 + w3 = 1.

[0023] The intelligent compliance judgment module for practical training analyzes and identifies violations in real time and generates early warning information. It also intelligently quantifies the compliance of practical training tasks and sends the judgment results to the multi-dimensional evaluation module for practical training effectiveness and the practical training supervision terminal. This realizes the transformation of practical training performance evaluation from "human subjective judgment" to "intelligent quantitative judgment", which greatly improves the timeliness of violation identification and the efficiency and accuracy of compliance judgment. It completely avoids the lag and subjectivity problems of human judgment and makes practical training supervision more objective and timely. The identification process for violations is as follows: retrieve the operation process standards for all pre-stored training steps; when the interaction behavior validity coefficient Ei is received from the interaction behavior validity output module, obtain the actual operation process sequence of the trainee performing training step i, and assign a sequence compliance identifier Iorder; where Iorder takes the value of 1 or 0. If the operation process sequence is compliant, that is, the operation process sequence is correct, then Iorder takes the value of 1; if the operation process sequence is non-compliant, then Iorder takes the value of 0. Furthermore, the actual time spent by the trainees on training step i (Tactual) is obtained, and the standard time spent on training step i is marked as Tstd; the step-level compliance coefficient Cstepi (dimensionless, value range [0,1], the closer to 1, the more compliant the corresponding trainee's current training process is with the standard) is calculated using the step-level compliance analysis formula, as follows: Where α, β, and γ are preset weight values, and α+β+γ=1; the step-level compliance coefficient Cstepi is compared with the preset step-level compliance coefficient threshold Cmin. If Cstepi<Cmin, then the training step i is judged to be a violation step and an early warning message is generated; the compliance of the training task is quantitatively determined based on the compliance performance of all training steps.

[0024] The quantitative judgment process of the compliance of the training task is as follows: obtain the number of non-compliant steps and calculate the ratio with the total number of training steps m in the training task to obtain the non-compliant step status value, and calculate the average of the step-level compliance coefficients of all training steps in the training task to obtain the compliance evaluation value. The violation step status value and compliance assessment value are compared with the preset violation step status threshold and the preset compliance assessment threshold respectively. If the violation step status value exceeds the preset violation step status threshold or the compliance assessment value does not exceed the preset compliance assessment threshold, the training task performed by the trainee is determined to be non-compliant and an early warning message is generated. Specifically, an early warning message is generated with "violation step name and violation type" (such as 'incorrect operation sequence', 'non-standard operation action', etc.).

[0025] The training task difficulty decision module analyzes the performance of all trainees on a given training task. Based on this analysis, the module marks the task as either complex or simple and sends this difficulty level information to the training monitoring terminal. The terminal displays this information, accurately distinguishing task difficulty levels. This facilitates targeted reinforcement of complex tasks with training and guidance, optimizes training resource allocation, avoids unreasonable training arrangements due to unclear task difficulty definitions, and improves overall training efficiency and effectiveness. The specific analysis process of the training task difficulty decision module is as follows: The total number of trainees participating in the corresponding training tasks is obtained, and the number of trainees whose training tasks are non-compliant is marked as the training failure test value. The ratio of the training failure test value to the total number of trainees is calculated to obtain the training failure coefficient. The training failure coefficient is compared with the preset training failure coefficient threshold. If the training failure coefficient exceeds the preset training failure coefficient threshold, it indicates that the overall training performance for the corresponding training task is poor and the execution difficulty of the training task is high. In this case, the corresponding training task is marked as a complex task.

[0026] Furthermore, if the training discrepancy coefficient does not exceed the preset training discrepancy coefficient threshold, the average value of the violation steps of all trainees is used to calculate the violation characteristic value, and the ratio of the preset compliance assessment threshold to the corresponding compliance assessment value of the trainees is used to calculate the compliance inverse value, and the average value of the compliance inverse values ​​of all trainees is used to calculate the compliance characteristic value. The task difficulty decision value is obtained by weighted summation of the training difference coefficient, violation characteristic value and compliance characteristic value. Specifically, the training difference coefficient, violation characteristic value and compliance characteristic value are assigned corresponding preset weight coefficients, and the training difference coefficient, violation characteristic value and compliance characteristic value are multiplied by the corresponding preset weight coefficients. The sum of the three sets of product results is marked as the task difficulty decision value. It should be noted that the larger the value of the task difficulty decision value, the worse the overall training performance for the corresponding training task, and the greater the difficulty of executing the training task. The task difficulty decision value is compared with the preset task difficulty decision threshold. If the task difficulty decision value exceeds the preset task difficulty decision threshold, it indicates that the overall training performance for the corresponding training task is poor, that is, the training task is difficult to execute, and the corresponding training task is marked as a complex task. If the task difficulty decision value does not exceed the preset task difficulty decision threshold, it indicates that the overall training performance for the corresponding training task is relatively good, that is, the training task is relatively easy to execute, and the corresponding training task is marked as a simple task. Example 2

[0027] like Figure 2 As shown, the difference between this embodiment and embodiment one is that the training task difficulty decision module is connected to the psychological stability analysis module. After marking the corresponding trainees as having simple tasks, the training task difficulty decision module sends the information of trainees whose training tasks in simple tasks are non-compliant to the psychological stability analysis module. The psychological stability analysis module marks the trainees whose training tasks in simple tasks are non-compliant as target personnel. Furthermore, the psychological stability performance of target personnel during corresponding training tasks is analyzed. This analysis determines whether a target personnel is psychologically unstable, and the results are sent to the training monitoring terminal. This allows for the accurate identification of personnel whose training non-compliance is due to psychological factors, providing a clear basis for subsequent targeted psychological quality training. This helps these personnel improve their psychological state during training, ensuring they can complete the training with a stable mindset, and further enhancing the overall training ability and pass rate of trainees. The specific analysis process of the psychological stability analysis module is as follows: The average heart rate fluctuation range ΔHR of the target personnel during the training process was obtained (excessive heart rate fluctuation indicates that the target personnel are too nervous and have poor psychological quality), and the safe threshold for heart rate fluctuation was marked as ΔHRmax. The average value of the skin conductance activity data of the target personnel during the training process was also obtained and marked as EDAavg (skin conductance activity reflects changes in skin conductance, and the measurement value of skin conductance activity is usually expressed in micro-Siemens μS), and the safe threshold for skin conductance was marked as EDAmax (preset = 5μS, referring to physiological and psychological research, exceeding 5μS indicates excessive tension). Through formula The psychological stability coefficient is calculated, where the larger the value of the psychological stability coefficient Qscore, the worse the psychological stability of the corresponding target personnel during the training process. The psychological stability coefficient Qscore is compared with the preset psychological stability coefficient threshold. If the psychological stability coefficient exceeds the preset psychological stability coefficient threshold, it indicates that the psychological stability of the corresponding target personnel during the training process is poor. In this case, the corresponding target personnel are judged to be psychologically unqualified and an early warning message is generated.

[0028] The working principle of this invention is as follows: In use, the multimodal interaction behavior capture and analysis module ensures the comprehensiveness and analysis accuracy of multimodal interaction behavior information from the source. The interaction behavior validity output module generates the interaction behavior validity coefficient of each training step by quantifying it, and constructs a unified interaction standard evaluation dimension to provide objective and quantifiable data support for compliance judgment. The training compliance intelligent judgment module identifies the non-compliant steps and quantifies the overall training task compliance by calculating the step-level compliance coefficient, which greatly improves the timeliness of non-compliance identification and the accuracy of compliance judgment, and avoids the lag and subjectivity of manual judgment. Furthermore, by analyzing the training performance of all trainees through the training task difficulty decision module, tasks are accurately marked as complex or simple tasks. This facilitates targeted training and guidance for complex tasks, optimizes the allocation of training resources, comprehensively covers the supervision and evaluation needs of the entire training process, significantly improves the intelligence level of training supervision, effectively ensures the quality of training and the skill improvement effect of trainees, and provides strong technical support for the standardized development of VR training in grassroots warehouses and stations.

[0029] In this invention, the threshold, preset value, or preset range settings are for result comparison and analysis to determine whether the result is good or bad. The magnitude of these values ​​is determined by a combination of large-scale model analysis of sample data and human experience, and can also be appropriately adjusted based on seasonal or common-sense influence conditions. Similarly, the preset weight coefficients and influence factors are assigned specific values ​​based on the magnitude of each parameter's influence on the result, ultimately reflecting the impact on the result. These settings are also determined by a combination of large-scale model analysis of sample data and human experience, and can also be appropriately adjusted based on seasonal or common-sense influence conditions.

[0030] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize it. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A human-computer interaction-based full-process monitoring and evaluation system for VR training at grassroots warehouse stations, characterized in that: It includes a multimodal interaction behavior capture and analysis module, an interaction behavior validity output module, a training compliance intelligent judgment module, a training task difficulty decision module, and a training supervision terminal; During the VR training at the grassroots warehouse station, the multimodal interaction behavior capture and analysis module captures and analyzes the multimodal interaction behavior of trainees in the VR training scenario, and the interaction behavior effectiveness output module quantifies the effectiveness of the trainees' interaction behavior in the corresponding training steps and generates the interaction behavior effectiveness coefficient. The training compliance intelligent judgment module analyzes and identifies illegal steps in real time and generates early warning information. It also intelligently quantifies the compliance of training tasks and sends the judgment results to the training effectiveness multi-dimensional evaluation module and the training supervision terminal. The training task difficulty decision module will analyze the training performance of all trainees for the corresponding training task, mark the corresponding training task as a complex task or a simple task based on the analysis, and send the difficulty marking information of the corresponding training task to the training supervision terminal.

2. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations as described in claim 1, characterized in that, The multimodal interaction behavior capture and analysis module deployment includes a multimodal sensor consisting of a motion capture camera, a voice acquisition microphone, and an eye-tracking device.

3. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations as described in claim 1, characterized in that, The method for obtaining the effectiveness coefficient of interactive behavior is as follows: When trainees perform training step i, a motion capture camera is used to collect the total number of frames of the trainees' limb movements during the corresponding training step, and the number of frames that match the standard operation movements are marked as Amatch; a voice capture microphone is used to record the total duration of the voice commands during the corresponding training step, and the duration of commands that conform to the semantic specifications are marked as the duration of valid voice commands; and eye-tracking gaze conformity of trainees during the corresponding training step is obtained through eye-tracking gaze decision analysis; the effectiveness coefficient of the trainees' interactive behavior in training step i is calculated using the interactive behavior effectiveness calculation formula.

4. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations as described in claim 3, is characterized in that, The specific analysis process of eye-tracking fixation decision analysis is as follows: the total fixation time of trainees in the corresponding step training process is collected, the time of the key operation area or risk warning area of ​​the eye-tracking fixation device is marked as tkey, and the matching degree between the fixation sequence and the standard operation sequence is obtained. The eye-tracking gaze conformity is calculated using a semantic conformity calculation method for key eye-tracking regions.

5. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations as described in claim 1, characterized in that, The specific operation process of the intelligent compliance judgment module for practical training includes: Retrieve the pre-stored operation process standards for all training steps. When the interaction behavior validity coefficient Ei is received from the interaction behavior validity output module, obtain the actual operation process sequence of the trainees performing training step i and assign the sequence compliance identifier Iorder. Furthermore, the actual time spent by trainees on training step i (Tactual) is obtained, and the standard time spent on training step i is marked as Tstd; the step-level compliance coefficient Cstepi is calculated through the step-level compliance analysis formula. If Cstepi < Cmin, then training step i is judged as a violation step and a warning message is generated; the compliance of the training task is quantitatively determined based on the compliance performance of all training steps.

6. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations according to claim 5, characterized in that, The quantitative determination process for the compliance of practical training tasks is as follows: The number of violations is obtained and its ratio to the total number of training steps m in the training task is calculated to obtain the violation status value. The average of the step-level compliance coefficients of all training steps in the training task is calculated to obtain the compliance assessment value. If the violation step status value exceeds the preset violation step status threshold or the compliance assessment value does not exceed the preset compliance assessment threshold, the training task performed by the trainee is deemed non-compliant and an early warning message is generated.

7. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations as described in claim 1, characterized in that, The specific analysis process of the training task difficulty decision module is as follows: the ratio of the non-successful test value to the total number of trainees is calculated to obtain the training difference coefficient. If the training difference coefficient exceeds the preset training difference coefficient threshold, the corresponding training task is marked as a complex task.

8. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations according to claim 7, characterized in that, If the training difference coefficient does not exceed the preset training difference coefficient threshold, the task difficulty decision value is calculated by weighted summation of the training difference coefficient, violation feature value and compliance feature value. If the task difficulty decision value exceeds the preset task difficulty decision threshold, the corresponding training task is marked as a complex task; otherwise, the corresponding training task is marked as a simple task.

9. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations according to claim 1, characterized in that, The training task difficulty decision module communicates with the psychological stability analysis module. After the training task difficulty decision module marks the corresponding trainees as having simple tasks, the psychological stability analysis module marks the trainees who do not meet the requirements of the simple tasks as target personnel, and analyzes the psychological stability performance of the target personnel in the corresponding training tasks. Through analysis, it is determined whether the target personnel are psychologically unqualified.

10. The human-computer interaction full-process supervision and evaluation system for VR training at grassroots warehouses and stations according to claim 9, characterized in that, The specific analysis process of the psychological stability analysis module is as follows: The psychological stability coefficient is calculated. If the psychological stability coefficient exceeds the preset psychological stability coefficient threshold, the corresponding target personnel are judged to be unqualified in terms of psychological stability and an early warning message is generated.

Images