VR practical training learning effect evaluation method and system based on large language model

Abstract

The invention provides a VR training learning effect evaluation method and system based on a large language model, and belongs to the technical field of fusion of a large language model technology and a virtual reality technology, and the method comprises the steps: obtaining user behavior original data, converting the user behavior original data into encrypted standardized transmission data, and uploading the encrypted standardized transmission data to a server; receiving the encrypted standardized transmission data, performing data standardization processing, and generating an LLM input data set; inputting the LLM input data set into a large language model for multi-dimensional feature importance evaluation, error pattern classification and learning trend prediction, and obtaining an LLM analysis result; generating a multi-dimensional ability map based on the LLM analysis result; generating a dynamic optimization strategy based on the multi-dimensional capability atlas; obtaining an optimized new LLM input data set and a new multi-dimensional capability graph based on a dynamic optimization strategy, performing evaluation, obtaining a practical training evaluation report, and judging whether all practical training tasks are completed or not; the problems of insufficient real-time performance, guidance generalization and single data dimension of a traditional evaluation method are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of large language model technology and virtual reality technology fusion, more specifically, relates to a VR practical training learning effect evaluation method and system based on a large language model. BACKGROUND

[0002] With the wide application of virtual reality technology in the fields of vocational skill training and education practical training, VR practical training gradually becomes an important means to improve the practical operation ability of learners due to the advantages of simulating high-risk scenes, supporting repeated practice, and controllable cost. However, as the core closed-loop link of the VR practical training system, the effectiveness of learning effect evaluation directly affects the quality of practical training. The current mainstream evaluation method still relies on traditional modes and cannot adapt to the evaluation needs of learning process refinement and real-time in the VR practical training scene, resulting in a gap between the actual training effect and the expected target.

[0003] The existing VR practical training learning effect evaluation indicators mainly focus on static result data such as the number of operation completions and the total task time, which can only reflect the final operation results and cannot capture subtle changes in the learning process such as operation sequence deviation, key step hesitation time, and local area attention omission. Dynamic behavior characteristics, it is difficult to locate the deep reasons for operation errors, and the evaluation results rely on manual analysis, the evaluator needs to sort out the data and subjectively judge the learning weaknesses after the event, which not only leads to feedback lag, but also may cause different standards due to the experience difference of the evaluators, lack of objective and unified analysis basis, and the data collection dimension is relatively single, mostly only focusing on operation result related data, ignoring the psychological and physiological dimension data such as the attention distribution and emotional state of learners, and these data often have a significant impact on learning effect, resulting in that the evaluation conclusion cannot fully reflect the real ability state of learners. SUMMARY

[0004] In order to solve the above technical problems, the application provides a VR practical training learning effect evaluation method and system based on a large language model to solve the technical problems in the prior art that the traditional evaluation method cannot locate the deep reasons for operation errors and ignores the psychological and physiological dimension data of learners, resulting in that the evaluation conclusion cannot fully reflect the real ability state of learners.

[0005] The purpose and effect of the VR practical training learning effect evaluation method and system based on a large language model of the application are achieved by the following specific technical means:

[0006] A VR practical training learning effect evaluation method based on a large language model, comprising the following steps:

[0007] S1: obtaining user behavior raw data based on a data acquisition controller;

[0008] S2: converting the user behavior raw data into encrypted standardized transmission data and uploading to a server;

[0009] S3: receiving the encrypted standardized transmission data, performing data standardization processing based on the encrypted standardized transmission data, and generating LLM input data sets;

[0010] S4: inputting the LLM input data sets into a large language model to perform multi-dimensional feature importance evaluation, error mode classification, and learning trend prediction, and obtaining LLM analysis results;

[0011] S5: generating a multi-dimensional capability map based on the LLM analysis results;

[0012] S6: generating a dynamic optimization strategy based on the multi-dimensional capability map;

[0013] S7: obtaining an optimized new LLM input data set and a new multi-dimensional capability map based on the dynamic optimization strategy, performing evaluation to obtain a practical training evaluation report, and determining whether the user has completed all practical training tasks based on the practical training evaluation report.

[0014] According to a preferred embodiment, the user behavior raw data is obtained, including:

[0015] In the VR practical training scene, the user's specific operation path in the VR practical training is captured in real time based on the VR handle, and the trajectory data is preliminarily de-duplicated and time-stamped by a data acquisition controller to obtain operation trajectory raw data;

[0016] The eye tracking instrument records the user's gaze in the VR scene in real time, and preliminarily associates the gaze data with the scene coordinates through a data acquisition controller to obtain attention distribution raw data;

[0017] The physiological sensor collects heart rate variability in real time, and converts the physiological signal into a transmissible digital signal through a data acquisition controller to obtain emotional state raw data;

[0018] Based on the operation trajectory raw data, attention distribution raw data, and emotional state raw data, the data acquisition controller eliminates invalid data according to the principle of minimum necessity, and integrates them into user behavior raw data in time sequence.

[0019] According to a preferred embodiment, the user behavior raw data is converted into encrypted standardized transmission data and uploaded to a server, including:

[0020] Based on user behavior raw data, the sensitive data in the user behavior raw data is desensitized by using the AES-256 encryption algorithm, and the non-sensitive data is packaged in the JSON format, and the packaging field includes user ID, timestamp, operation type, operation result, error code, decision time, attention distribution, and emotion state, and encrypted standardized transmission data is obtained;

[0021] Based on the encrypted standardized transmission data, the connection with the analysis server is established by operating the HTTPS encryption API interface, and only the encrypted model parameters are uploaded by using the federated learning technology, so that the sensitive information is avoided to be leaked.

[0022] According to a preferred embodiment, the encrypted standardized transmission data is received, data standardization processing is performed based on the encrypted standardized transmission data, and an LLM input data set is generated, including:

[0023] The operation trajectory raw data in the encrypted standardized transmission data is extracted, the operation trajectory raw data is converted into a vector sequence in a unified three-dimensional coordinate system, the differences between different device coordinate systems are eliminated, and standardized operation trajectory data is obtained;

[0024] The decision time in the encrypted standardized transmission data is extracted, the time interval from triggering an operation instruction in a VR scene to executing an operation by a user is calculated in seconds, abnormal data exceeding a preset range is eliminated, and standardized decision time data is obtained;

[0025] The operation result in the encrypted standardized transmission data is extracted, the user operation result is compared with preset practical training metadata, the error number of each type of operation is counted, the error probability is calculated, and standardized error rate data is obtained;

[0026] The attention distribution raw data in the encrypted standardized transmission data is extracted, the attention distribution raw data is mapped to the three-dimensional coordinates of the VR scene, the proportion of the stay time of the user in the preset key area and the non-key area is counted, and standardized attention distribution data is obtained;

[0027] Based on the standardized operation trajectory data, the standardized decision time data, the standardized error rate data, and the standardized attention distribution data, data association and integration are performed according to the timestamp, and a structured LLM input data set is generated.

[0028] According to a preferred embodiment, the LLM input data set is input into a large language model to perform multi-dimensional feature importance evaluation, error mode classification, and learning trend prediction, and LLM analysis results are obtained, including:

[0029] Based on the LLM input data set, the large language model analyzes the influence weight of each data in the LLM input data set on the learning effect by calling a pre-trained practical training field feature evaluation algorithm, identifies the data with the highest weight, and obtains multi-dimensional feature importance evaluation results.

[0030] The standardized error rate data is extracted from the LLM input data set, user operation errors are divided into knowledge errors, skill errors and psychological errors, specific reasons are labeled for each type of error, and error mode classification results are obtained;

[0031] User historical training data is obtained, and based on the LLM input data set, error mode classification results and user historical training data, learning bottlenecks of the user in subsequent training are predicted, and learning trend prediction results are obtained;

[0032] Based on the multi-dimensional feature importance evaluation result, the error mode classification result and the learning trend prediction result, a text description is integrated to obtain an LLM analysis result.

[0033] According to a preferred embodiment, the multi-dimensional ability map is generated based on the LLM analysis result, comprising:

[0034] The multi-dimensional feature importance evaluation result is extracted from the LLM analysis result, and based on the LLM analysis result and the multi-dimensional feature importance evaluation result, key ability dimensions are extracted around the core target of the training, each dimension is associated with a corresponding behavior index, and an ability dimension division result is obtained;

[0035] Based on the ability dimension division result and the LLM input data set, for each ability dimension, a score is calculated by combining the feature weight, the ability level is quantified by adopting the percentage system, and the ability score data of each dimension in the interval [0, 100] is obtained;

[0036] The error mode classification result is extracted from the LLM analysis result, and based on the error mode classification result and the ability score data of each dimension, the dimensions with scores lower than 60 are marked as ability weaknesses, and the corresponding error reasons are associated, and an ability weakness labeling result is obtained;

[0037] The learning trend prediction result is extracted from the LLM analysis result, and based on the ability weakness labeling result and the learning trend prediction result, targeted improvement suggestions are generated and the implementation cycle and evaluation standard of the suggestions are associated, and ability improvement suggestion data is obtained;

[0038] Based on the ability dimension division result, the ability score data of each dimension, the ability weakness labeling result and the ability improvement suggestion data, the ability distribution is visualized by using the radar chart, all information is integrated to form a complete ability map, and a multi-dimensional ability map is obtained.

[0039] According to a preferred embodiment, the dynamic optimization strategy is generated based on the multi-dimensional ability map, comprising:

[0040] The matching degree in the [0, 1] interval is obtained based on the multi-dimensional ability graph calculation of the current ability of the user and the difficulty of the VR practical training scene; if the matching degree is less than 0.6, the scene difficulty is reduced, if the matching degree is greater than 0.8, the scene difficulty is increased, otherwise the scene difficulty remains unchanged;

[0041] The scene parameter is updated in real time by the VR rendering engine after adjustment, and the scene difficulty adjustment parameter is obtained;

[0042] The standardized error rate data is extracted from the LLM input data set, and the knowledge push intensity is adjusted according to the error rate based on the standardized error rate data, and the knowledge push frequency optimization parameter is obtained;

[0043] The standardized attention distribution data is extracted from the LLM input data set, and the interaction weight is set to visual 50%, auditory 30%, and tactile 20% based on the multi-dimensional ability graph and the standardized attention distribution data;

[0044] If the environmental noise is greater than the threshold, the auditory weight is reduced by 15% and the weights of vision and touch are increased, and the new weight is applied in real time by the VR interaction engine after adjustment, and the interaction mode optimization parameter is obtained;

[0045] Based on the scene difficulty adjustment parameter, the knowledge push frequency optimization parameter and the interaction mode optimization parameter, the dynamic optimization strategy is formed by classifying and integrating the practical training scene nodes.

[0046] According to a preferred embodiment, the optimized new LLM input data set and the new multi-dimensional ability graph are obtained based on the dynamic optimization strategy, and the evaluation report is obtained based on the evaluation of the new LLM input data set and the new multi-dimensional ability graph, and whether the user has completed all the practical training tasks is determined based on the practical training evaluation report, including:

[0047] Based on the dynamic optimization strategy, the steps S1-S5 are repeated to obtain the optimized new LLM input data set and the new multi-dimensional ability graph, and the evaluation report is obtained based on the evaluation of the LLM input data set, the multi-dimensional ability graph, and the new LLM input data set and the new multi-dimensional ability graph, and whether the user has completed all the practical training tasks is determined.

[0048] According to a preferred embodiment, the method further comprises:

[0049] If the practical training evaluation report indicates that it is not completed, a new dynamic optimization strategy is obtained based on the practical training evaluation report, and the steps S1-S7 are repeated;

[0050] If the practical training evaluation report indicates that it is completed, the practical training evaluation report is archived, and the evaluation process is ended.

[0051] A VR practical training learning effect evaluation system based on a large language model, comprising:

[0052] A user behavior data collection module is configured to collect user behavior raw data, and the user behavior data collection module is in communication connection with a data collection controller, which is configured to preliminarily integrate and screen the collected multi-source data;

[0053] A data encryption transmission module is in communication connection with the user behavior data collection module, and is configured to convert the user behavior raw data into encrypted standardized transmission data, and upload the encrypted standardized transmission data to a server;

[0054] A data standardization processing module is in communication connection with the server, and is configured to receive the encrypted standardized transmission data, and perform data standardization processing based on the encrypted standardized transmission data to generate an LLM input data set;

[0055] An LLM multi-dimensional feature analysis module is in communication connection with the data standardization processing module, and is configured to receive the LLM input data set, input the LLM input data set into a large language model, and perform multi-dimensional feature importance evaluation, error mode classification and learning trend prediction to output an LLM analysis result;

[0056] A capability map generation module is in communication connection with the LLM multi-dimensional feature analysis module, and is configured to generate a multi-dimensional capability map based on the LLM analysis result;

[0057] A dynamic optimization decision module is in communication connection with the capability map generation module, and is configured to generate a dynamic optimization strategy based on the multi-dimensional capability map;

[0058] An evaluation cycle and report generation module is in communication connection with the dynamic optimization decision module, the user behavior data collection module, the data standardization processing module, the LLM multi-dimensional feature analysis module and the capability map generation module, and is configured to trigger a new round of data collection and analysis based on the dynamic optimization strategy, obtain an optimized new LLM input data set and a new multi-dimensional capability map, generate a practical training evaluation report based on the LLM input data set, the multi-dimensional capability map, the new LLM input data set and the new multi-dimensional capability map, and determine whether a user has completed all practical training tasks.

[0059] Compared with the prior art, the present application has the following beneficial effects:

[0060] 1. Capture user operation trajectory through VR handle, record attention distribution through eye tracking instrument, collect emotional data such as heart rate variability through physiological sensor, and integrate and screen by the principle of minimum necessity through data acquisition controller to ensure that dynamic behavior and psychophysiological state data in the operation process are covered; after the standardized multi-dimensional data is input into LLM, LLM will combine the knowledge in the practical training field, subdivide the operation errors into knowledge errors, skill errors and psychological errors, and label the reasons for each type of error; this process from data acquisition, deep analysis and error classification can excavate the knowledge blind area, skill short board or psychological interference factors behind the surface operation error, and solve the problem that the traditional evaluation method is difficult to locate the deep reason of the error.

[0061] 2. The eye tracking instrument can record the user's gaze in the VR scene in real time, and the dwell area and time can reflect the learner's attention to the key operation area; the heart rate variability data collected by the physiological sensor is converted into emotional score after standardization, which can reflect the emotional fluctuation of the learner in the practical training; these psychological and physiological data, operation trajectory, decision time, error rate and other data together generate LLM input data set, and the multi-dimensional ability graph generated after the analysis of the large language model includes the quantitative score of the ability dimension, and the ability weakness is also labeled combined with attention and emotional data; this multi-dimensional integrated evaluation method avoids the one-sidedness of the conclusion caused by ignoring the psychological and physiological data in the traditional evaluation, and ensures that the evaluation result can fully reflect the learner's knowledge mastery, skill level and psychological adaptability.

[0062] 3. After generating the multi-dimensional ability graph, the matching degree of user ability and scene difficulty will be calculated based on the graph, the VR scene parameters will be dynamically adjusted, and the knowledge push frequency will be optimized according to the ability weakness and error rate; based on the dynamic optimization strategy, the data acquisition, analysis and graph generation process is repeated to obtain new data and new ability graph after optimization, and the training evaluation report is generated by comparing the evaluation results before and after; if the learner has not completed the training, the report will be updated based on the optimization strategy to continue the cycle until the report is archived after completing the training. This closed-loop process of evaluation, optimization and re-evaluation enables the training scheme to continuously adapt to the learner's ability changes, avoids the lack of targeting problem of traditional fixed mode training, and enables the learner to gradually make up for the shortcomings in the training environment that adapts to their own state, improving learning effect and operation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a step flowchart of a VR practical training learning effect evaluation method based on a large language model of the present application;

[0064] Figure 2 is a structural schematic diagram of a VR practical training learning effect evaluation system based on a large language model of the present application. DETAILED DESCRIPTION

[0065] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the technical solutions of the present application, but cannot be used to limit the protection scope of the present application.

[0066] Embodiments:

[0067] As shown in the accompanying Figure 1 :

[0068] The present application provides a VR practical training learning effect evaluation method based on a large language model, comprising the following steps:

[0069] S1: obtaining user behavior original data based on a data acquisition controller;

[0070] In this embodiment, the user carries out VR practical training such as fire equipment operation, industrial equipment maintenance, etc. The user holds a VR handle to perform practical training operation. The VR handle captures the specific operation path of the user in the virtual scene in real time through the built-in motion sensor, such as the spatial movement trajectory of picking up and putting down the fire extinguisher, the rotation angle change of the valve switch, the click sequence of the equipment button, etc. These trajectory data are transmitted to the data acquisition controller in real time. After the data acquisition controller receives the trajectory data, it automatically identifies and deletes repeated records such as multiple invalid data of the same position caused by accidental triggering of the VR handle due to hand shaking, and adds a millisecond-level timestamp to each valid trajectory data to mark the specific time when the data is generated. Through this series of operations, the operation trajectory original data is obtained, which can clearly record the spatial sequence and time node of the user's operation, and can be used for subsequent analysis of whether the user's operation conforms to the practical training specification process and whether there is a step omission or sequence error.

[0071] In the VR practical training process, the lens of the eye tracking instrument is aimed at the user's eye area, and the user moves the line of sight following the calibration point on the screen to complete the matching of the device and the user's eye position, ensuring accurate line of sight capture. After calibration, the eye tracking instrument records the user's line of sight in the VR scene in real time, such as whether the user's line of sight is focused on the fire extinguisher pressure instrument panel, device fault warning sign, operation step prompt box, etc. key area, while counting the user's stay time in each area, such as 2 seconds in the pressure instrument panel and 5 seconds in the non-key background area. These line of sight data are continuously transmitted to the data acquisition controller. The data acquisition controller preliminarily associates the line of sight data with the three-dimensional coordinates of the VR scene, such as mapping the line of sight stay point to the coordinates X1, Y1, Z1 of the pressure instrument panel, the coordinates X2, Y2, Z2 of the emergency stop button, and the coordinates of specific objects, etc. Through this series of operations, the attention distribution original data is obtained, which can intuitively reflect the user's attention focusing situation in the practical training, and can be used for subsequent judgment of whether the user is distracted.

[0072] If the physiological sensor is a chest patch sensor, it needs to be attached to the user's chest skin and fixed; if it is a wrist-worn sensor, it needs to be worn on the inner side of the user's wrist to ensure good contact between the sensor and the skin to accurately collect physiological signals; during VR practical training, the physiological sensor will collect the user's heart rate variability data in real time, which can indirectly reflect the user's emotional fluctuations. A decrease in heart rate variability may indicate that the user is in a state of tension, and stable heart rate variability indicates a calm mood. The collected simulated physiological signals will be transmitted to the data collection controller in real time; after the operation trajectory original data, attention distribution original data and emotional state original data are transmitted to the data collection controller, the data collection controller will start the data screening and integration process: according to the principle of minimum necessity, automatically identify and eliminate invalid data, such as chaotic trajectory data generated when the VR device suddenly powers off, meaningless operation data generated when the user touches the VR handle unintentionally before practical training, and abnormal signal data generated when the physiological sensor is not in good contact, only keeping the valid data directly related to learning effect evaluation; Subsequently, the data collection controller will integrate the data into unified user behavior original data in time sequence.

[0073] S2: Based on the user behavior original data, convert it into encrypted standardized transmission data and upload to the server;

[0074] In this embodiment, based on the user behavior raw data, the classification processing is carried out to distinguish sensitive data and non-sensitive data. The sensitive data includes the heart rate variability raw value in the emotional state raw data which can be directly associated with the physiological characteristics of the user, the associated field in the user ID which may imply personal identity information, etc. The non-sensitive data covers the timestamp, operation type, operation result, error code, decision time, attention distribution, and emotional state. For the sensitive data, the AES-256 encryption algorithm is used for desensitization processing. When operating, the AES-256 encryption program is first loaded in the data processing unit, a unique and regularly updated encryption key is set, the sensitive data is input into the program in segments, and the original sensitive data is converted into ciphertext which cannot be directly interpreted through algorithm operation. For the non-sensitive data, the encapsulation is carried out according to the syntax rules of the JSON format, each non-sensitive data field and the corresponding content are organized in the form of key-value pairs, the non-sensitive data JSON segment with unified structure is formed, and then the desensitized sensitive data ciphertext and the non-sensitive data JSON segment are integrated to ensure that each data field corresponds correctly. Finally, the encrypted standardized transmission data is obtained, which not only realizes the privacy protection of sensitive information, but also avoids the problem of data parsing confusion when the subsequent server receives due to the unified format. Based on the obtained encrypted standardized transmission data, the connection initialization function of the HTTPS encryption API interface calling interface is operated in the data transmission unit, the connection request to the analysis server is initiated, the HTTPS protocol automatically establishes the SSL encryption channel, all data in the transmission process is encrypted in real time, the digital certificate of the analysis server is verified to confirm the legality of the server identity and prevent data leakage caused by connecting to a false server. After the connection is established, the complete encrypted standardized transmission data is not directly uploaded, but the federal learning technology is used for data screening and uploading. In the local data processing unit, the model parameters related to the training and learning effect are trained based on the user behavior raw data, the same AES-256 encryption algorithm as the sensitive data is used to process these model parameters, and only the encrypted model parameters are uploaded to the analysis server through the established HTTPS encryption API interface. The original encrypted standardized transmission data is still complete and retained in the local device and is not uploaded. In this way, the analysis server can obtain the model parameters used for overall training effect analysis, and the sensitive information of the user will not be separated from the local device, effectively avoiding the risk of data leakage in the uploading process or the server storage stage, and taking into account the usability and privacy security of the data.

[0075] S3: receiving the encrypted standardized transmission data, performing data standardization processing based on the encrypted standardized transmission data to generate an LLM input data set;

[0076] In this embodiment, the analysis server receives the encrypted standardized transmission data uploaded from the local device through the HTTPS encrypted API interface, verifies the integrity of the data during the receiving process, confirms that the data is undamaged, calls the built-in AES-256 decryption program of the server, inputs the decryption key consistent with the local device, decrypts the sensitive data ciphertext in the data, restores it to readable sensitive data, and parses the JSON format fragments of non-sensitive data, extracts the operation trajectory original data, decision time, operation result, attention distribution original data, and other field contents, ensures that all data to be processed can be extracted completely and without parsing errors, and provides a basis for subsequent standardized processing.

[0077] The operation trajectory original data in the decrypted encrypted standardized transmission data is extracted, and a coordinate system conversion program is started in the data processing module of the server. A unified three-dimensional coordinate system is established with the preset fixed origin in the VR training scene as the origin of the three-dimensional coordinate system, the X-axis along the horizontal direction of the scene, the Y-axis along the vertical direction of the scene, and the Z-axis along the depth direction of the scene. The original trajectory data collected by different brands of VR handles is converted into coordinate values in the unified coordinate system point by point, and then these coordinate values are arranged in chronological order to form a continuous vector sequence. Through this series of operations, the trajectory data deviation caused by the difference between different device coordinate systems is eliminated, and standardized operation trajectory data is obtained. This data can ensure that the subsequent large language model analysis of operation trajectories collected by different users and different devices can judge whether the operation is standard based on a unified standard, avoiding analysis errors caused by non-uniform coordinate systems.

[0078] The decision time related information in the decrypted encrypted standardized transmission data is extracted, and the time interval is calculated in the data processing module: the decision time original value is obtained by subtracting the timestamp of the VR scene triggering operation instruction from the timestamp of the user starting to perform the operation in seconds; at the same time, an abnormal data screening program is called, and the decision time original value exceeding the preset range set according to the training scene is determined as abnormal data and excluded, only the data within the reasonable range is retained, and standardized decision time data is obtained. This data can accurately reflect the thinking and reaction speed of the user after receiving the operation instruction.

[0079] The operation result in the encrypted standardized transmission data after decryption is extracted, a comparison program is called in the data processing module, and each item of the user operation result is compared with the preset practical training metadata to determine whether each operation is in line with the standard: if the operation result is consistent with the practical training metadata, it is marked as correct operation; if there is deviation or step loss, it is marked as error operation and the error type is recorded; then the total operation times and error operation times of the same operation type are counted, the error probability of each type of operation is obtained through the calculation method of error probability = error operation times / total operation times, and the standardized error rate data is obtained, which can quantify the operation accuracy of the user in different practical training links and help the large language model quickly locate the high-frequency error link of the user.

[0080] The attention distribution raw data in the encrypted standardized transmission data after decryption is extracted, the coordinate mapping program is started in the data processing module, the pixel coordinates of the line of sight are converted into three-dimensional coordinates of the VR scene, and the key areas and non-key areas are preset according to the practical training target; the staying time of the user's line of sight in each key area and non-key area is counted, the attention distribution proportion is obtained through the calculation method of key area staying time proportion = key area total staying time / total line of sight staying time, and non-key area staying time proportion = non-key area total staying time / total line of sight staying time, and the standardized attention distribution data is obtained, which can clearly reflect the attention focusing direction of the user in the practical training process.

[0081] Based on the obtained standardized operation trajectory data, standardized decision time data, standardized error rate data and standardized attention distribution data, the time stamp association program is started in the data integration module of the server, the millisecond level time stamp at the time of data collection is taken as the reference, each dimension data under the same time stamp is matched, the associated data is organized in time sequence into structured format, and the structured LLM input data set is generated.

[0082] S4: inputting the LLM input data set into the large language model to perform multi-dimensional feature importance evaluation, error mode classification and learning trend prediction, and obtaining LLM analysis result;

[0083] In this embodiment, based on the LLM input data set, the large language model imports the LLM input data set, which contains standardized operation trajectory data, standardized decision time data, standardized error rate data, standardized attention distribution data and corresponding timestamps; then start the feature evaluation process of the large language model, the large language model will automatically call the pre-trained training field feature evaluation algorithm, which is generated in advance based on a large amount of same type VR training data such as fire equipment operation and industrial equipment maintenance training data, and the algorithm is built-in correlation analysis logic of each data feature and learning effect such as operation specification, task completion rate and error correction ability. The model calculates the influence weight of each type of data in the LLM input data set by algorithm, and sorts the weight from high to low, finally identifies the top 1 or 2 types of data with the highest weight, and obtains the multi-dimensional feature importance evaluation result. This result can help subsequent analysis focus on key data, avoid the problem of blurred focus caused by scattered attention to all data, and ensure that the evaluation of learning effect can grasp the core influencing factors.

[0084] After completing the multi-dimensional feature importance evaluation, the large language model automatically switches to the error mode classification process, first extracts all standardized error rate data from the LLM input data set, and then calls the built-in error classification logic of the model. This logic divides error categories based on training field knowledge: knowledge errors refer to errors caused by users' lack of understanding of training operation standards and principles, skill errors refer to errors caused by users' lack of operation proficiency and precision, and psychological errors refer to errors caused by users' emotional fluctuations such as nervousness and impatience. The model will match the category of each error operation and label the specific reason for each type of error, while calculating the proportion of each type of error, and obtaining the error mode classification result. This result can clearly reveal the root cause of user operation errors and avoid the problem of traditional evaluation that only knows there are errors but does not know why.

[0085] Through the data interface, the user's historical training data is obtained from the historical database of the server, which contains the LLM input data set of the user's past 3 or more times of the same type of VR training, error mode classification results, ability scores and other information; the model will perform correlation analysis on the current LLM input data set, error mode classification results and historical training data, focusing on two types of information: one is the coincidence degree of current error mode and historical error mode, and the other is the trend of change of current key data and historical data; based on these analyses, the model predicts the learning bottleneck that the user may face in subsequent training, and obtains the learning trend prediction result. This result can predict potential problems in the user's learning process in advance.

[0086] Finally, the large language model automatically integrates the multi-dimensional feature importance evaluation results, error mode classification results, and learning trend prediction results to organize the content in the form of text description: first, explain the multi-dimensional feature importance evaluation results, then describe the error mode classification results, skill error times, and psychological error times, and finally present the learning trend prediction results, forming a structured and complete text description, and obtaining the LLM analysis results.

[0087] S5: Generate multi-dimensional ability atlas based on LLM analysis results;

[0088] In this embodiment, the multi-dimensional feature importance evaluation results are extracted from the LLM analysis results, and then combined with the core objectives of VR practical training, the key ability dimensions are extracted around the objectives, each ability dimension needs to correspond to the core data in the multi-dimensional feature importance evaluation results, ensuring that the dimension setting is strongly related to the learning effect, and at the same time, associating the corresponding behavior indicators for each dimension, through a series of operations to clarify the definition and evaluation basis of each ability dimension, and obtain the ability dimension division results, which can avoid the problem of fuzzy ability evaluation dimensions.

[0089] For example, in fire training, fire extinguisher operation ability, pressure value detection ability, emergency scene judgment ability, and VR handle control ability are key ability dimensions, and the behavior indicators are represented as the handle movement fluency and operation step integrity associated with standardized operation trajectory data for fire extinguisher operation ability, the detection result accuracy and error probability associated with standardized error rate data for pressure value detection ability, the instruction response speed and judgment correctness associated with standardized decision time data for emergency scene judgment ability, and the action precision and angle control accuracy associated with standardized operation trajectory data for VR handle control ability.

[0090] Based on the ability dimension division results and the LLM input data set, start the scoring calculation program in the ability atlas generation module: first, assign feature weights to each behavior indicator corresponding to the ability dimension; then extract the specific performance data of each behavior indicator from the LLM input data set; obtain the original score of each ability dimension through the calculation method of "dimension score = (index 1 performance x index 1 weight) + (index 2 performance x index 2 weight) + …", and then convert the original score to a percentage score in the [0, 100] interval, ensuring that the score calculation logic matches the practical training standards during the calculation process, and finally obtaining the dimension ability score data, which directly reflects the user's level in each core ability dimension in a quantitative form, avoiding the problem of traditional evaluation mainly relying on qualitative description and lacking quantitative basis.

[0091] After the score calculation is completed, the error pattern classification result in the LLM analysis result is extracted in the ability atlas generation module, and the error pattern is associated and matched with the dimension ability score data: if the score of a certain ability dimension is lower than 60 points, the dimension is marked as an ability weakness; at the same time, the error causes directly related to the weakness are screened out from the error pattern classification result and marked beside the corresponding dimension. Through a series of operations, the correspondence between the ability short board and the error root is determined, and the ability weakness marking result is obtained. This result can help users quickly locate their core weaknesses and avoid fuzzy understanding of the ability weakness.

[0092] Subsequently, the learning trend prediction result in the LLM analysis result is extracted in the ability atlas generation module, and the improvement suggestion generation program is started in combination with the ability weakness marking result: for each ability weakness, specific suggestions are made by referring to the error causes and trend prediction; at the same time, the implementation period and evaluation standard are associated for each suggestion to ensure that the suggestions are executable and verifiable, and the ability improvement suggestion data is obtained. This data provides a specific ability improvement path for the user and avoids the problem of vague improvement direction and lack of landing.

[0093] Finally, the ability distribution is presented in the form of a radar chart. Each axis of the radar chart corresponds to a key ability dimension, and the score scale in the [0, 100] interval is marked on the axis. According to the dimension ability score data, the score points are plotted on the radar chart, and then the score points are connected to form a closed figure, which intuitively shows the balance of the ability. At the same time, a text explanation area is added beside the radar chart, which integrates the ability dimension division result, the ability weakness marking result, and the ability improvement suggestion data to form a complete ability atlas containing visual ability distribution and textual supplementary explanation. The multi-dimensional ability atlas is obtained.

[0094] S6: Generating a dynamic optimization strategy based on the multi-dimensional ability atlas;

[0095] In this embodiment, the matching degree of the user's current ability and the VR training scene difficulty in the [0, 1] interval is calculated based on the multi-dimensional ability atlas; if the matching degree < 0.6, the scene difficulty is reduced, and the key parameters in the VR training scene are adjusted, such as reducing the smoke density in fire training, lowering the virtual environment temperature, and simplifying the fault triggering logic; if the matching degree > 0.8, the scene difficulty is increased, such as increasing the smoke density, increasing the temperature, and increasing the multi-device cooperation link; if the matching degree is in the [0.6-0.8] interval, the scene difficulty remains unchanged; after the difficulty adjustment instruction is generated, it is transmitted to the VR rendering engine in real time through a data interface. The VR rendering engine updates the scene parameters immediately after receiving the instruction and feeds back a parameter update completion signal. The scene difficulty adjustment parameter is obtained, which can ensure that the VR training scene difficulty is adapted to the user's ability, avoiding the problem of user frustration due to too high difficulty or low learning efficiency due to too low difficulty.

[0096] After completing the scene difficulty adjustment, the standardized error rate data is extracted from the LLM input data set in the dynamic optimization decision module, the knowledge push frequency optimization is performed, and the error rate is adjusted according to the same proportion. The logic runs, that is, the higher the error rate of a certain ability dimension, the greater the corresponding knowledge push strength. For example, if the error rate of the stress value detection is 23% higher than the average error rate, the push frequency of the related knowledge of this dimension will be adjusted according to the same proportion of error rate from the basic interval, and the push form includes text prompt box, voice explanation, and animation demonstration in the VR scene. After the frequency adjustment is completed, the adjusted push interval, content type and other information are recorded, and the knowledge push frequency optimization parameter is obtained. This parameter can ensure that the knowledge push can accurately match the user's error-prone link and avoid information overload or key knowledge omission caused by indiscriminate push.

[0097] Subsequently, the standardized attention distribution data is extracted from the LLM input data set in the dynamic optimization decision module, and the interactive mode optimization is performed in combination with the ability weaknesses in the multi-dimensional ability map. First, the initial interactive weight is set to visual 50%, auditory 30%, and tactile 20%. If the multi-dimensional ability map shows that the user's attention is concentrated in the visual area and the auditory information reception efficiency is low, the visual weight is appropriately increased and the auditory weight is appropriately reduced. At the same time, the environment sensor is called to obtain the noise data of the current practical training environment. If the environmental noise is greater than the preset threshold, it is determined that the auditory information is susceptible to interference, the auditory weight is reduced by 15%, and the visual weight and tactile weight are correspondingly increased to ensure that the user can clearly obtain the practical training information through vision and touch. After the interactive weight adjustment is completed, the new weight parameter is transmitted to the VR interaction engine, the VR interaction engine immediately applies the new weight, and a weight application success signal is returned, and the interactive mode optimization parameter is obtained.

[0098] The scene difficulty adjustment parameter, knowledge push frequency optimization parameter, and interactive mode optimization parameter are classified and integrated according to the VR practical training scene node, the scene difficulty is adjusted synchronously, and the dynamic optimization strategy covering the entire practical training process is formed. This strategy can realize real-time adaptive adjustment of VR practical training, so that the practical training process always fits the user's ability changes and environmental needs, and improves the pertinence and effectiveness of the practical training.

[0099] S7: Based on the dynamic optimization strategy, the optimized new LLM input data set and new multi-dimensional ability map are obtained and evaluated to obtain a practical training evaluation report. Based on the practical training evaluation report, it is determined whether the user has completed all the practical training tasks.

[0100] In this embodiment, first, the generated dynamic optimization strategy is loaded in the evaluation cycle and report generation module, the running parameters of the VR practical training system and the data acquisition equipment are adjusted synchronously according to the strategy instructions, the VR rendering engine applies new scene difficulty parameters, the VR interaction engine applies new interaction weights, the knowledge pushing module pushes content at a new frequency, and the running state of the data acquisition equipment is adapted to the new scene; then the complete process of steps S1-S5 is repeated, and finally the optimized new LLM input data set and new multi-dimensional ability graph are obtained.

[0101] After the new data and the new ability graph are obtained, the evaluation program is started in the evaluation cycle and report generation module, and the LLM input data set before optimization, the multi-dimensional ability graph and the new LLM input data set after optimization, and the new multi-dimensional ability graph are compared and analyzed in multiple dimensions: the standardized error rate data are compared, the correlation between the error rate reduction amplitude and the dynamic optimization strategy is analyzed, the dimensional ability score data are compared, the ability change curve is drawn, and the improvement of each dimension is intuitively presented, the error mode classification results are compared, and the main error types still existing at present are analyzed; based on these comparison data, the preset training target is combined to judge whether the user's current ability meets the standard, and then a training evaluation report is generated,

[0102] Based on the training evaluation report, subsequent operations are performed, if the report determines that the entire training task is not completed, a new dynamic optimization strategy is generated, the new dynamic optimization strategy is transmitted to the dynamic optimization decision module, and then the process of steps S1-S7 is restarted, and the new data is continuously collected, the new ability graph is generated, and the evaluation is performed until the user's ability meets the standard;

[0103] If the report determines that the entire training task has been completed, the training evaluation report, the LLM input data set at each stage, the multi-dimensional ability graph at each version, and all dynamic optimization strategy parameters are stored in the evaluation cycle and report generation module according to the naming rules of user ID, training date and training type, and uploaded to the historical training database of the server, and the evaluation process is ended.

[0104] Please refer to FIG. 1 Figure 2 The application also provides a VR practical training learning effect evaluation system based on a large language model, which comprises:

[0105] A user behavior data acquisition module is used to acquire user behavior raw data, the user behavior data acquisition module is in communication connection with a data acquisition controller, and the data acquisition controller is used to preliminarily integrate and screen the collected multi-source data;

[0106] A data encryption transmission module is in communication connection with the user behavior data acquisition module, and is used to convert the user behavior raw data into encrypted standardized transmission data, and upload the encrypted standardized transmission data to a server.

[0107] The data standardization processing module is in communication connection with the server, configured to receive the encrypted standardized transmission data, and perform data standardization processing based on the encrypted standardized transmission data to generate an LLM input data set;

[0108] The LLM multi-dimensional feature analysis module is in communication connection with the data standardization processing module, configured to receive the LLM input data set, input the LLM input data set into the large language model, and perform multi-dimensional feature importance evaluation, error mode classification and learning trend prediction to output an LLM analysis result;

[0109] The capability map generation module is in communication connection with the LLM multi-dimensional feature analysis module, configured to generate a multi-dimensional capability map based on the LLM analysis result;

[0110] The dynamic optimization decision module is in communication connection with the capability map generation module, configured to generate a dynamic optimization strategy based on the multi-dimensional capability map;

[0111] The evaluation cycle and report generation module is in communication connection with the dynamic optimization decision module, the user behavior data acquisition module, the data standardization processing module, the LLM multi-dimensional feature analysis module and the capability map generation module, respectively, configured to trigger a new round of data acquisition and analysis based on the dynamic optimization strategy, acquire a new LLM input data set and a new multi-dimensional capability map after optimization, generate a practical training evaluation report based on the LLM input data set, the multi-dimensional capability map, the new LLM input data set and the new multi-dimensional capability map, and determine whether the user has completed all practical training tasks.

[0112] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for evaluating the learning effectiveness of VR practical training based on a large language model, characterized in that, It includes the following steps: S1: Acquire raw user behavior data based on the data acquisition controller; S2: Based on the user behavior raw data, convert it into encrypted and standardized transmission data and upload it to the server; S3: Receive encrypted and standardized transmission data, perform data standardization processing based on the encrypted and standardized transmission data, and generate an LLM input dataset; S4: Input the LLM input dataset into a large language model to perform multidimensional feature importance evaluation, error pattern classification, and learning trend prediction, and obtain the LLM analysis results; S5: Generate a multi-dimensional capability map based on LLM analysis results; S6: Generate dynamic optimization strategies based on multi-dimensional capability maps; S7: Based on the dynamic optimization strategy, obtain the optimized new LLM input dataset and the new multi-dimensional capability map, evaluate them, and obtain the training evaluation report. Based on the training evaluation report, determine whether the user has completed all training tasks.

2. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The acquisition of raw user behavior data includes: In VR training scenarios, the system uses VR controllers to capture the user's specific operation path in real time and uses a data acquisition controller to perform preliminary deduplication and timestamp marking on the trajectory data to obtain the original operation trajectory data. Based on eye trackers, the area where the user's gaze stays and the duration of the gaze in the VR scene are recorded in real time. The gaze data is initially correlated with the scene coordinates through a data acquisition controller to obtain raw data on attention distribution. Based on real-time acquisition of heart rate variability using physiological sensors, the physiological signals are converted into transmittable digital signals by a data acquisition controller to obtain raw data on emotional state. Based on the raw data of operation trajectory, raw data of attention distribution, and raw data of emotional state, the operation data acquisition controller removes invalid data according to the principle of minimum necessity and integrates them into raw user behavior data according to time series.

3. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The process of converting raw user behavior data into encrypted and standardized transmission data and uploading it to the server includes: Based on the raw user behavior data, sensitive data in the raw user behavior data is desensitized using the AES-256 encryption algorithm, and non-sensitive data is encapsulated in JSON format. The encapsulated fields include user ID, timestamp, operation type, operation result, error code, decision time, attention distribution, and emotional state, and the encrypted standardized transmission data is obtained. Based on encrypted standardized data transmission, the system establishes a connection with the analysis server using the HTTPS encrypted API interface, and employs federated learning technology to upload only encrypted model parameters, thus avoiding the leakage of sensitive information.

4. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The process of receiving encrypted and standardized transmission data, performing data standardization processing based on the encrypted and standardized transmission data, and generating an LLM input dataset includes: Extract the original operation trajectory data from the encrypted and standardized transmission data, convert the original operation trajectory data into a vector sequence in a unified three-dimensional coordinate system, eliminate the differences in coordinate systems of different devices, and obtain standardized operation trajectory data. Extract the decision time from the encrypted standardized transmission data, calculate the time interval from the VR scene triggering the operation command to the user executing the operation in seconds, remove abnormal data that exceeds the preset range, and obtain standardized decision time data; Extract the operation results from the encrypted standardized transmission data, compare the user operation results with the preset training metadata, count the number of errors for each type of operation, calculate the error probability, and obtain standardized error rate data; Extract the raw attention distribution data from the encrypted standardized transmission data, map the raw attention distribution data to the three-dimensional coordinates of the VR scene, and count the percentage of time users spend in preset key areas and non-key areas to obtain standardized attention distribution data; Based on standardized operation trajectory data, standardized decision time data, standardized error rate data, and standardized attention distribution data, data are linked and integrated according to timestamps to generate a structured LLM input dataset.

5. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The process involves inputting the LLM input dataset into a large language model for multidimensional feature importance evaluation, error pattern classification, and learning trend prediction to obtain LLM analysis results, including: Based on the LLM input dataset, the large language model analyzes the influence weight of each data in the LLM input dataset on the learning effect by calling the pre-trained training domain feature evaluation algorithm, identifies the data with the highest weight, and obtains the multi-dimensional feature importance evaluation results. Standardized error rate data is extracted from the LLM input dataset. User operation errors are classified into knowledge errors, skill errors, and psychological errors. Specific reasons are labeled for each type of error, and error pattern classification results are obtained. Obtain users’ historical training data, and based on the LLM input dataset, error pattern classification results and users’ historical training data, predict users’ learning bottlenecks in subsequent training and obtain learning trend prediction results. The results of LLM analysis are obtained by integrating the multidimensional feature importance assessment results, error pattern classification results, and learning trend prediction results into a text description.

6. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The generation of a multi-dimensional capability map based on LLM analysis results includes: The multidimensional feature importance assessment results are extracted from the LLM analysis results. Based on the LLM analysis results and the multidimensional feature importance assessment results, key capability dimensions are extracted around the core training objectives. Each dimension is associated with corresponding behavioral indicators to obtain capability dimension classification results. Based on the capability dimension segmentation results and the LLM input dataset, a score is calculated for each capability dimension by combining feature weights. The capability level is quantified using a percentage system, and capability score data for each dimension in the range of [0,100] are obtained. Error pattern classification results are extracted from the LLM analysis results. Based on the error pattern classification results and the ability score data of each dimension, the dimensions with scores below 60 are marked as ability weaknesses, and the corresponding error causes are associated to obtain the ability weakness labeling results. Learning trend prediction results are extracted from the LLM analysis results. Based on the capability weakness labeling results and learning trend prediction results, targeted improvement suggestions are generated and the implementation cycle and evaluation criteria of the suggestions are associated to obtain capability improvement suggestion data. Based on the results of capability dimension segmentation, capability score data for each dimension, capability weakness annotation results, and capability improvement suggestion data, a radar chart is used to visualize the capability distribution. All information is integrated to form a complete capability map and obtain a multi-dimensional capability map.

7. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The dynamic optimization strategy based on multi-dimensional capability graph generation includes: The matching degree between the user's current ability and the difficulty of the VR training scene is calculated based on the multi-dimensional ability graph. The matching degree is calculated in the interval [0, 1]. If the matching degree is < 0.6, the scene difficulty is reduced; if the matching degree is > 0.8, the scene difficulty is increased; otherwise, the scene difficulty remains unchanged. After adjustment, the scene parameters are updated in real time through the VR rendering engine to obtain the scene difficulty adjustment parameters; Extract standardized error rate data from the LLM input dataset, adjust the knowledge push intensity according to the error rate based on the standardized error rate data, and obtain the knowledge push frequency optimization parameters. Normalized attention distribution data were obtained from the LLM input dataset. Based on the multidimensional ability map and the normalized attention distribution data, the interaction weights were set to 50% for visual, 30% for auditory, and 20% for tactile. If the ambient noise exceeds the threshold, the auditory weight is reduced by 15%, and the visual and tactile weights are increased. After adjustment, the new weights are applied in real time through the VR interaction engine to obtain the interaction optimization parameters. Based on the parameters for adjusting the difficulty of the scenario, optimizing the frequency of knowledge delivery, and optimizing the interaction method, the parameters are integrated according to the training scenario nodes to form a dynamic optimization strategy.

8. The VR training learning effect evaluation method based on a large language model according to claim 1, characterized in that, The process involves obtaining an optimized new LLM input dataset and a new multi-dimensional capability map based on a dynamic optimization strategy, evaluating them, and generating a training evaluation report. Based on this report, it is determined whether the user has completed all training tasks, including: Based on the dynamic optimization strategy, repeat steps S1-S5 to obtain the optimized new LLM input dataset and the new multi-dimensional capability map. Based on the LLM input dataset, the multi-dimensional capability map, the new LLM input dataset, and the new multi-dimensional capability map, perform an evaluation, obtain a training evaluation report, and determine whether the user has completed all training tasks.

9. The VR training learning effect evaluation method based on a large language model according to claim 8, characterized in that, The method further includes: If the training evaluation report indicates that it is incomplete, obtain a new dynamic optimization strategy based on the training evaluation report and repeat steps S1-S7. If the training evaluation report indicates that it is completed, archive the training evaluation report and end the evaluation process.

10. A VR training learning effectiveness evaluation system based on a large language model, characterized in that, include: The user behavior data acquisition module is used to acquire raw user behavior data. The user behavior data acquisition module is connected to the data acquisition controller, which is used to perform preliminary integration and filtering of the acquired multi-source data. The data encryption transmission module communicates with the user behavior data collection module and is used to convert the raw user behavior data into encrypted and standardized transmission data, and then upload the encrypted and standardized transmission data to the server. The data standardization processing module communicates with the server to receive encrypted and standardized transmitted data, and performs data standardization processing based on the encrypted and standardized transmitted data to generate the LLM input dataset. The LLM multidimensional feature analysis module communicates with the data standardization processing module. It is used to receive the LLM input dataset, input the LLM input dataset into the large language model, perform multidimensional feature importance evaluation, error pattern classification and learning trend prediction, and output the LLM analysis results. The capability map generation module communicates with the LLM multidimensional feature analysis module and is used to generate multidimensional capability maps based on LLM analysis results. The dynamic optimization decision-making module communicates with the capability map generation module and is used to generate dynamic optimization strategies based on multi-dimensional capability maps. The evaluation cycle and report generation module communicates with the dynamic optimization decision-making module, user behavior data collection module, data standardization processing module, LLM multidimensional feature analysis module, and capability map generation module, respectively. It is used to trigger a new round of data collection and analysis based on the dynamic optimization strategy, obtain the optimized new LLM input dataset and the new multidimensional capability map, generate a training evaluation report based on the LLM input dataset, the multidimensional capability map, the new LLM input dataset, and the new multidimensional capability map, and determine whether the user has completed all training tasks.

Images