Model data validity determination method, electronic device, and storage medium

Abstract

This invention discloses a method, electronic device, and storage medium for determining the validity of model data, relating to the field of model training technology. The method includes: acquiring sensor data and user language commands; generating visual data based on the sensor data; obtaining predicted action sequence data based on the visual data, the user language commands, and a visual-language action model; acquiring actual action sequence data based on the predicted action sequence data and a virtual reality device; and calculating a comprehensive validity score representing multiple evaluation dimensions to determine the validity of the model data. The model data validity determination method provided in this application can effectively reduce the cost of collecting sample data for model training and improve the generalization of the trained system.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a method for determining the validity of model data, an electronic device, and a storage medium. Background Technology

[0002] Existing model data validity determination schemes fall into two categories: one is the traditional offline data cleaning scheme, where robots or autonomous vehicles record all sensor data, and invalid data is removed afterward (offline) through manual annotation or script filtering; the other is active learning based on uncertainty, where data collection is conducted by determining whether to save data based solely on the confidence level of the model output. For example, the "shadow mode" data feedback mechanism in the field of autonomous driving: a new version of the algorithm runs in the background while the vehicle is in motion, and data feedback is triggered when the algorithm's predicted path differs significantly from the driver's actual operation.

[0003] Although existing technologies have solved the model data selection problem to some extent, they have the following shortcomings when facing Visual Language Action Models (VLA models) and Embodied AI scenarios: lack of real-time visualization of multimodal interactions, single evaluation dimension, difficulty in adapting to VLA models, low data storage and annotation efficiency, and long loop cycle.

[0004] Therefore, there is an urgent need to develop a model data screening scheme that can effectively reduce data annotation costs and improve data purity. Summary of the Invention

[0005] The present invention aims to solve the above-mentioned technical problems. Specifically, the present invention provides a method, electronic device and storage medium for determining the validity of model data, which can solve the problem that VLA models cannot verify the difference between model intent and user operation in real time and intuitively during the data acquisition stage; solve the problem that relying solely on a single action error index cannot effectively evaluate the value of complex interactive data involving language and semantics; and solve the problem of storage waste and low annotation efficiency caused by the lag in screening high-value data (such as model failure and OOD scenarios) in massive data acquisition.

[0006] In a first aspect, the present invention provides a method for determining the validity of model data, the method comprising: Acquire sensor data and user voice commands; Visual data is generated based on the sensor data, wherein the visual data is data obtained by reconstructing a three-dimensional scene from the sensor data; The visual data and the user's language commands are input into the visual language action model to obtain predicted action sequence data. Based on the predicted action sequence data, actual action sequence data is obtained, wherein the actual action sequence data is action sequence data corresponding to human-computer interaction generated by sending the predicted action sequence data to a virtual reality device for display and obtaining it from the virtual reality device; A comprehensive effectiveness score representing multiple evaluation dimensions is calculated based on the visual data, the user's language instructions, the actual action sequence data, and the predicted action sequence data. The validity of the sensor data is determined based on the comprehensive validity score.

[0007] Preferably, the multiple evaluation dimensions include action difference, semantic consistency, and model uncertainty, and the calculation of the comprehensive effectiveness score representing the multiple evaluation dimensions based on the visual data, the user language instructions, the actual action sequence data, and the predicted action sequence data includes: The action difference score is calculated based on the actual action sequence data and the predicted action sequence data; the semantic consistency score is calculated based on the visual data and the user language instructions; and the model uncertainty score is calculated based on the original numerical values ​​output by the visual language action model. The comprehensive effectiveness score is obtained by weighting the action difference score, the semantic consistency score, and the model uncertainty score.

[0008] Preferably, the motion difference score is calculated as follows: the Euclidean distance and rotation error of the motion trajectory are calculated based on the actual motion sequence data and the predicted motion sequence data, and the motion difference score is determined based on the Euclidean distance and / or rotation error.

[0009] Preferably, the semantic consistency score is calculated as follows: input data is determined based on the visual data and the user language instruction; the input data is input into the visual language pre-trained model to determine whether the model's attention area matches the target object specified by the user language instruction; and the semantic consistency score is determined based on the matching result.

[0010] Preferably, the model uncertainty score is calculated by: calculating the entropy value of the original score output by the visual language action model, and determining the model uncertainty score based on the entropy value.

[0011] Preferably, before calculating the comprehensive effectiveness score representing multiple evaluation dimensions based on the visual data, the user language instructions, the actual action sequence data, and the predicted action sequence data, the method further includes: aligning the predicted action sequence data and the actual action sequence data with timestamps on the timeline.

[0012] Preferably, the determination of model data validity based on the comprehensive validity score includes: when the comprehensive validity score exceeds a preset threshold, marking the sensor data within the time window corresponding to the comprehensive validity score as valid data.

[0013] Preferably, the method further includes: adding tag information to the valid data marked within the time window based on real-time display of the virtual reality device and manual operation, and / or playback display of the virtual reality device and voice operation. In a second aspect, the present invention provides an electronic device including at least one processor and at least one memory, the memory being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the model data validity determination method described in any of the above technical solutions.

[0014] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of program codes adapted to be loaded and run by a processor to perform the model data validity determination method described in any of the above technical solutions.

[0015] By adopting the above technical solution, the present invention can achieve the following beneficial effects: After acquiring sensor data and user language commands, visual data is generated in a virtual reality environment. Based on the visual data, the user language commands, and the visual language action model, predicted action sequence data is obtained. That is, the shadow trajectory is visualized using VR. The prediction results of the VLA model are superimposed in 3D form on the VR operation interface in real time, and can be intuitively compared with the real operation. The validity is judged from multiple evaluation dimensions such as action difference, semantic consistency, and model uncertainty, realizing the screening of high-value (model failure, OOD scenario) data, thereby improving the purity of model training data and training efficiency. By using VR interaction, data cleaning and preliminary cause labeling are completed in the data acquisition stage, which can reduce the difficulty for the professional labeling team to understand the scene later, thereby effectively reducing the data labeling cost. Attached Figure Description

[0016] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein: Figure 1 This is a schematic diagram illustrating an application scenario of a model data validity determination method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the main implementation steps of a model data validity determination method provided in this application embodiment; Figure 3 This is provided by the embodiments of this application. Figure 2 A flowchart illustrating one specific implementation of step S15; Figure 4 This is a schematic diagram of the main structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0018] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0019] The technical terms used in the embodiments of this application are explained as follows: VR: The full English name is Virtual Reality. VR can use computers to simulate a three-dimensional virtual world, providing users with simulations of their senses such as sight, hearing, and touch.

[0020] VLA stands for Vision-Language-Action Model. It is a multimodal large model that can simultaneously process visual input and language commands, and directly output the control actions (actions) of the robot or intelligent agent.

[0021] CLIP stands for Contrastive Language-Image Pre-training, which is a pre-training method or model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. CLIP's training data consists of text-image pairs: an image and its corresponding text description. Through contrastive learning, the model can learn the matching relationship between text-image pairs.

[0022] OOD stands for Out-of-Distribution, which refers to data from scenarios outside the model's training dataset that the model has never seen or is difficult to process.

[0023] Valid data: In this application, it refers to data that has high value for model training.

[0024] Traditional shadow model-based effective data screening typically operates in a "black box" manner, where operators cannot see in real time what the model "intends to do." This makes it difficult to intuitively determine why a model fails during data collection—for example, whether it's a visual recognition error, a language understanding deviation, or a motion planning failure—and hinders timely adjustments to the collection strategy to capture such long-tail scenarios. Existing technologies often focus on single motion deviations (such as steering wheel angle errors) when judging effective data. However, VLA models involve language understanding and complex robotic arm operations. Simple motion errors cannot reflect semantic alignment. For example, if the instruction is "take the red apple," and the model's action is "take the green apple," the motion trajectories may be similar, but the semantics are completely wrong. Existing technologies struggle to detect such high-value model data. Furthermore, existing technologies often screen model data only after full collection or based solely on confidence levels. This results in storing a large amount of repetitive, simple scenario data (low-value model data) or missing samples where the model is overconfident (high-confidence but incorrect), leading to extremely high manual annotation costs later.

[0025] This application provides a real-time model data validity assessment scheme based on virtual reality interaction. It implements a visual shadow mode, allowing the prediction results of the VLA model to be overlaid in 3D on the VR interface in real time for intuitive comparison with real-world operations. It also implements a multimodal data value scoring algorithm based on geometric trajectory error (action), semantic attention (visual / language), and model confidence (probability). Furthermore, it implements a human-machine collaborative screening mechanism, supporting both automatic and manual data screening processes triggered by VR. The model data validity assessment scheme provided in this application can effectively reduce the cost of collecting VLA model data and improve the generalization of the trained model or system.

[0026] like Figure 1The diagram illustrates an application scenario of a model data validity determination method provided in this application. The model data validity determination method provided in this application can be understood as code or software configured in an electronic device, which can be a robot or an intelligent agent. The model data validity determination method of this application can be applied to... Figure 1 The hardware environment shown in the diagram consists of a data acquisition device, a VR interactive terminal, and electronic devices. In this application scenario, the VR interactive terminal may include a head-mounted display and a controller, the data acquisition device may include a camera and / or sensors, and the electronic devices mainly include a data acquisition module, a VLA shadow model inference engine module, a real-time evaluation module, and a data management module.

[0027] The data acquisition module is used to acquire sensor data and user language commands, and generate visual data based on the sensor data; the VLA shadow model inference engine module is used to input the visual data and user language commands into a visual language action model to obtain predicted action sequence data, and send the predicted action sequence data to a VR interactive terminal for display; the real-time evaluation module is used to acquire human-computer interaction operation data from the VR interactive terminal as actual action sequence data, and calculate a comprehensive validity score representing multiple evaluation dimensions based on the visual data, user language commands, actual action sequence data, and predicted action sequence data, and determine the model data validity of the sensor data based on the comprehensive validity score; the data management module is used to label and hierarchically store and manage the data according to the results of the model data validity determination.

[0028] See appendix Figure 2 , Figure 2 This is a flowchart illustrating the main implementation steps of a model data validity determination method provided in this application embodiment. As shown in the figure, the method mainly includes the following steps S11 to S13: Step S11: Acquire sensor data and user voice commands; In this embodiment, taking the application of the VLA model in a robot or intelligent agent as an example, the sensor data can be understood as sensor data collected by a visual sensor connected to the robot or intelligent agent, such as image data collected by an RGB-D camera or point cloud data collected by a LiDAR. It should be understood that image data collected by an RGB-D camera includes RGB image information and depth information. The user language instruction (Prompt) refers to the natural language instruction used to drive the VLA model to perform inference. For example, the Prompt can be "grab the red apple on the table" or "place the object in the specified position," etc.

[0029] Step S12: Generate visual data based on the sensor data; In this embodiment, the visual data mentioned in this step can be understood as data obtained by reconstructing a three-dimensional scene from the sensor data. Specifically, the three-dimensional scene reconstruction can be achieved through software configured in the electronic device, such as using digital twin software or 3D simulation software to generate the visual data based on the sensor data. It should be understood that the visual data can specifically be a digital twin scene or a video stream.

[0030] Step S13: Input the visual data and the user language command into the visual language action model to obtain predicted action sequence data; In this embodiment, a multimodal input is constructed based on the visual data and the user's language commands, and is input to the Visual Language Action Model (VLA model) to output predicted action sequence data. It should be understood that the predicted action sequence data is used to characterize the action predictions made for the reconstructed current scene. In practical applications, the multimodal input is transmitted to a robot or intelligent agent, which is equipped with a VLA shadow model inference engine. The VLA shadow model inference engine can be understood as an inference module that runs the VLA model in shadow mode. Based on the VLA shadow model inference engine, the predicted action sequence output by the VLA model can be rendered only as a shadow trajectory in the VR view, and the robot or intelligent agent does not execute the action sequence. For example, the predicted action sequence data may include the six-degree-of-freedom (6-DoF) pose trajectory of the end effector, the gripper's opening and closing state, etc.

[0031] Step S14: Obtain actual action sequence data based on the predicted action sequence data; In this embodiment, the actual action sequence data mentioned in this step can be understood as data corresponding to human-computer interaction operations obtained by sending the predicted action sequence data to a virtual reality device for display and acquiring the data from the virtual reality device. Specifically: the predicted action sequence output by the VLA model in step S13 is transmitted to the virtual reality device (i.e., the VR interactive terminal) for rendering in the VR field of view, and the actual actions performed by the operator through the VR controller or teach pendant are captured to obtain the actual action sequence data. It should be understood that the actual action sequence data is used to characterize the actual actions performed by the operator in the reconstructed current scene.

[0032] Furthermore, this step may also include: aligning the predicted action sequence data and the actual action sequence data with timestamps on the timeline.

[0033] Step S15: Calculate a comprehensive effectiveness score representing multiple evaluation dimensions based on the visual data, the user language instructions, the actual action sequence data, and the predicted action sequence data; In this embodiment, the multiple evaluation dimensions include action difference, semantic consistency, and model uncertainty.

[0034] In one specific implementation, such as Figure 3 As shown, step S15 can be specifically implemented by steps 151 to 154: Step 151: Calculate the action difference score based on the actual action sequence data and the predicted action sequence data; Specifically, the Euclidean distance and rotation error of the motion trajectory are calculated based on the actual motion sequence data and the predicted motion sequence data, and the motion difference score is determined based on the Euclidean distance and / or rotation error. It should be understood that a greater motion difference indicates that the model cannot currently handle the scene, and thus the higher the value of the original sensor data corresponding to that scene.

[0035] In practical applications, the choice between using Euclidean distance and / or rotational error to determine the motion difference score can be flexibly selected based on the specific type of the end effector of the robot or intelligent agent. For example, for tasks sensitive to end effector position, Euclidean distance alone can be used to determine the motion difference score; for tasks requiring high posture accuracy, rotational error alone can be used; and in some high-precision operation scenarios, a weighted combination of both can be used to determine the motion difference score.

[0036] Step 152: Calculate the semantic consistency score based on the visual data and the user's semantic instructions; Specifically, input data can be determined based on the visual data and the user language instruction. The input data is then input into a visual language pre-trained model (such as the CLIP model) to determine whether the model's region of interest matches the target object specified by the user language instruction. The semantic consistency score is then determined based on the matching result.

[0037] In practical applications, the input data for the CLIP model can be composed of the current RGB image or VR rendered view as image input and the corresponding Prompt as text input. Using the CLIP model or its interpretable extension methods (such as Grad-CAM and Attention Rollout), the following can be achieved: a visual attention heatmap can be generated, and this heatmap can be mapped back to the image input (RGB image or VR rendered view) to determine whether the model's region of interest is spatially or semantically consistent with the target object specified in the Prompt. If they are inconsistent, they are considered high-value negative samples. For example, if the similarity score output by the CLIP model is lower than a preset score, it indicates inconsistency. The semantic consistency score can be divided into 0 and 1, where 0 represents consistency and 1 represents inconsistency; or the similarity score can be directly used as the semantic consistency score.

[0038] Step 153: Calculate the model uncertainty score based on the original numerical output of the visual language action model; Specifically, the entropy value of the raw scores (logits) output by the Visual Language Action Model (VLA model) is calculated, and the model uncertainty score is determined based on the entropy value. For example, the entropy value can be directly used as the model uncertainty score, where a high entropy value indicates that the model is confused about the current scene, that is, the original sensor data corresponding to the scene is high-value data.

[0039] Step 154: Perform a weighted calculation on the action difference score, the semantic consistency score, and the model uncertainty score to obtain the comprehensive effectiveness score.

[0040] It should be understood that the weights in the weighted calculation can be set according to the actual application needs.

[0041] Step S16: Determine the validity of the sensor data as model data based on the comprehensive validity score.

[0042] In this embodiment, this step may specifically include: when the comprehensive validity score exceeds a preset threshold, marking the sensor data within the time window corresponding to the comprehensive validity score as valid data. Preferably, in practical applications, the time window corresponding to the comprehensive validity score can be a time window obtained by adding T seconds to the time window corresponding to the current scene. For example, if the time window corresponding to the current scene is 50 seconds and T is 10 seconds, this step may involve taking sensor data within 60 seconds, marking it as valid data, and saving it.

[0043] Furthermore, this step, when saving effective data, may also include: real-time display based on virtual reality devices and manual operation, and / or playback display based on virtual reality devices and voice operation, to add label information to the effective data marked within the time window. In practical applications, when an operator sees a significant error in the shadow trajectory in the VR field of view using the VR interactive terminal's headset (such as clipping or falling), i.e., a clear inconsistency between the model's predicted action and the actual operation, the operator can manually mark the current scene segment in the VR field of view using VR controller buttons (such as the "capture button") to add label information to the data corresponding to the current scene segment. In addition, the operator can also quickly replay the previous segment in VR and add descriptive labels instantly through voice input (such as "light reflection caused recognition error"). Then, the effective data with added label information is used as label text to generate structured metadata and written into the dataset. For example, the structured metadata may include timestamps, label text, and operator IDs. This embodiment utilizes VR interaction to complete part of the cleaning and preliminary cause labeling during the data acquisition stage (as mentioned above through voice and buttons), reducing the difficulty for the professional labeling team to understand the scene later, thereby effectively reducing data labeling costs.

[0044] Furthermore, in the model data validity determination method provided in this application, after filtering and labeling valid data from the original sensor data, a step of hierarchical storage of the data can be included for later model training. For example, the data can be divided into high-value data, ordinary data, and low-value data according to their value and stored separately. In one specific embodiment, sensor data marked as valid data, predicted action sequence data corresponding to the valid data, and label information can be saved as high-value data, state data or compressed time-frequency data can be saved as ordinary data, and other data can be directly discarded to release the cache. It should be understood that the stored high-value data can be directly obtained and used during later model training. Compared with the original sensor data, a large amount of invalid and simple repetitive data has been eliminated, making the collected model data rich in edge scenarios that the model is not currently aware of. This can achieve the effect of accelerating model convergence and improving training efficiency by improving data purity. In addition, the model data filtering scheme provided in the embodiments of this application supports a fast closed loop of "collection-verification-training". Operators can deliberately construct scenarios where the model is prone to errors based on real-time feedback to collect targeted model data.

[0045] Those skilled in the art will understand that all or part of the processes in the methods of any of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0046] Furthermore, the present invention also provides an electronic device. (See appendix.) Figure 4 , Figure 4 This is a schematic diagram of the main structure of an electronic device according to an embodiment of this application. In one embodiment of the electronic device according to this application, the electronic device includes a processor 301 and a memory 302. The memory 302 can be configured to store a program for executing the model data validity determination method of the above-described method embodiments. The processor 301 can be configured to execute the program in the memory, which includes, but is not limited to, a program for executing the model data validity determination method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention.

[0047] In some possible embodiments of this application, the electronic device may include multiple processors 301 and multiple memories 302. The program executing the model data validity determination method of the above method embodiments can be divided into multiple subroutines. Each subroutine can be loaded and run by a processor 301 to execute different steps of the model data validity determination method of the above method embodiments. Specifically, each subroutine can be stored in a different memory 302, and each processor 301 can be configured to execute programs in one or more memories 302 to jointly implement the model data validity determination method of the above method embodiments. That is, each processor 301 executes different steps of the model data validity determination method of the above method embodiments to jointly implement the model data validity determination method of the above method embodiments.

[0048] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the model data validity determination method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described model data validity determination method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0049] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for determining the validity of model data, characterized in that, include: Acquire sensor data and user voice commands; Visual data is generated based on the sensor data, wherein the visual data is data obtained by reconstructing a three-dimensional scene from the sensor data; The visual data and the user's language commands are input into the visual language action model to obtain predicted action sequence data. Based on the predicted action sequence data, actual action sequence data is obtained, wherein the actual action sequence data is action sequence data corresponding to human-computer interaction operations generated by sending the predicted action sequence data to a virtual reality device for display and obtaining it from the virtual reality device. A comprehensive effectiveness score representing multiple evaluation dimensions is calculated based on the visual data, the user's language instructions, the actual action sequence data, and the predicted action sequence data. The validity of the sensor data is determined based on the comprehensive validity score.

2. The method for determining the validity of model data according to claim 1, characterized in that: The multiple evaluation dimensions include action difference, semantic consistency, and model uncertainty. The comprehensive effectiveness score representing these multiple evaluation dimensions, calculated based on the visual data, the user's language instructions, the actual action sequence data, and the predicted action sequence data, includes: The action difference score is calculated based on the actual action sequence data and the predicted action sequence data; the semantic consistency score is calculated based on the visual data and the user language instructions; and the model uncertainty score is calculated based on the original numerical values ​​output by the visual language action model. The comprehensive effectiveness score is obtained by weighting the action difference score, the semantic consistency score, and the model uncertainty score.

3. The method for determining the validity of model data according to claim 2, characterized in that, The motion difference score is calculated as follows: the Euclidean distance and rotation error of the motion trajectory are calculated based on the actual motion sequence data and the predicted motion sequence data, and the motion difference score is determined based on the Euclidean distance and / or rotation error.

4. The method for determining the validity of model data according to claim 2, characterized in that, The semantic consistency score is calculated as follows: input data is determined based on the visual data and the user language instruction; the input data is input into the visual language pre-trained model to determine whether the model's attention area matches the target object specified by the user language instruction; and the semantic consistency score is determined based on the matching result.

5. The method for determining the validity of model data according to claim 2, characterized in that, The model uncertainty score is calculated by: calculating the entropy value of the original score output by the visual language action model, and determining the model uncertainty score based on the entropy value.

6. The method for determining the validity of model data according to claim 1, characterized in that, Before calculating the comprehensive effectiveness score representing multiple evaluation dimensions based on the visual data, the user language instructions, the actual action sequence data, and the predicted action sequence data, the method further includes: aligning the predicted action sequence data and the actual action sequence data with timestamps on the timeline.

7. The model data validity determination method according to claim 6, characterized in that, The determination of model data validity based on the comprehensive validity score includes: when the comprehensive validity score exceeds a preset threshold, the sensor data within the time window corresponding to the comprehensive validity score is marked as valid data.

8. The method for determining the validity of model data according to claim 7, characterized in that, The method further includes: adding tag information to the valid data marked within the time window based on real-time display and manual operation of virtual reality devices, and / or playback display and voice operation based on virtual reality devices.

9. An electronic device comprising at least one processor and at least one memory, said memory being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to perform the model data validity determination method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the model data validity determination method according to any one of claims 1 to 8.

Images