New energy equipment VR training system based on visual three-dimensional reconstruction

Abstract

The invention discloses a new energy equipment VR practical training system based on visual three-dimensional reconstruction, and relates to the technical field of VR practical training, and the system comprises a data collection module which carries out the surrounding shooting of target new energy equipment; the three-dimensional reconstruction module is used for generating a 3D Gaussian scene model file according to the surrounding image sequence by adopting a motion recovery structure and a 3D Gaussian sputtering hybrid algorithm; the VR interaction module is used for building a virtual practical training scene and carrying out gesture control and operation process guidance; the function module is used for carrying out operation normative detection, skill quantitative statistics and evaluation report generation; the cross-platform adaptation module is used for providing a multi-terminal operation environment; and the core control unit is used as a system server for overall planning and scheduling. According to the invention, by combining the surrounding image sequence, the motion recovery structure, the 3D Gaussian sputtering hybrid algorithm, VR interaction and cross-platform adaptation, the precision, interactivity and openness of new energy equipment VR practical training are improved, and the modeling cost is reduced.

Description

Technical Field

[0001] This application relates to the field of VR training technology, and in particular to a VR training system for new energy equipment based on visual 3D reconstruction. Background Technology

[0002] Against the backdrop of the rapid development of the new energy industry, the demand for highly skilled personnel has increased dramatically. However, traditional training models can no longer meet this demand. Vocational schools and corporate training face challenges such as high investment in real equipment, difficulty in conducting high-risk operations, and low teaching efficiency. Although virtual reality (VR) technology has been introduced into the education field, in highly specialized and hands-on fields like new energy, existing VR training solutions suffer from prominent problems such as high modeling costs, low model fidelity, interaction design detached from actual workflows, closed systems, and inability to quantify and evaluate performance.

[0003] In the prior art, Chinese patent CN106526850A discloses a method for constructing a chemical experimental equipment based on VR operation, including: creating a 3D model: creating a 3D model in 3D model making software according to the design drawings of the chemical experimental equipment, or directly scanning the equipment using a 3D scanning device to generate a 3D model, and then setting corresponding real materials in the corresponding parts of the 3D model according to the materials and colors of each component of the chemical experimental equipment to form a 3D model file; constructing a virtual reality scene: putting the 3D model file into a real-time rendering engine, and adding corresponding 3D environment models and related material and equipment models, and adding virtual reality observation components to the 3D environment model; realizing virtual reality operation of the chemical experimental equipment: adding corresponding control programs to the switches, moving mechanisms, display panels and control components in the equipment model according to the working principle of the chemical experimental equipment, so that they can interact with the user's controller or motion capture operation.

[0004] However, the existing technologies mentioned above use design drawings for modeling or 3D scanning to generate models. The modeling methods are limited and lack precision. The interaction is only for simple control of chemical equipment and does not achieve multi-terminal adaptation. The cumbersome modeling process leads to high costs and cannot meet the actual needs of training new energy equipment. The precision, interactivity, and openness of VR training for new energy equipment need to be improved, and the modeling cost needs to be reduced. Summary of the Invention

[0005] This application provides a VR training system for new energy equipment based on visual 3D reconstruction, which addresses the issues of insufficient accuracy, interactivity, and openness, as well as the need to reduce modeling costs in existing VR training systems for new energy equipment.

[0006] On the one hand, this application provides a VR training system for new energy equipment based on visual 3D reconstruction, including: a data acquisition module, a 3D reconstruction module, a VR interaction module, a functional module, a cross-platform adaptation module, and a core control unit.

[0007] The data acquisition module is configured to perform surround shooting of the target new energy equipment to obtain a surround image sequence.

[0008] The 3D reconstruction module is configured to generate a 3D Gaussian scene model file based on the surrounding image sequence by employing a motion recovery structure and a 3D Gaussian sputtering hybrid algorithm.

[0009] The VR interaction module is configured to: call the 3D Gaussian scene model file to build a virtual training scene for students to control with gestures and guide students through the work process.

[0010] The functional module is configured to: monitor VR interaction events of trainees in the virtual training scenario in real time, perform operation standardization checks, quantify and statistically analyze skills, and generate evaluation reports.

[0011] The cross-platform adaptation module is configured to package the training content corresponding to the VR interaction module and the functional module into an independently runnable application, providing a multi-terminal operating environment for the VR interaction module and the functional module.

[0012] The core control unit is configured as a system server to coordinate and schedule the data acquisition module, the 3D reconstruction module, the VR interaction module, the functional modules, and the cross-platform adaptation module.

[0013] In one possible implementation, the data acquisition module is further configured to: after obtaining the surround image sequence, perform noise reduction, exposure optimization, and white balance correction on the surround image sequence.

[0014] In one possible implementation, a motion recovery structure and a 3D Gaussian sputtering hybrid algorithm are used to generate a 3D Gaussian scene model file based on the surrounding image sequence, including: For the surrounding image sequence, feature points are extracted and matched using the scale-invariant feature transform algorithm, and the camera pose is calculated and a sparse point cloud is generated using incremental motion recovery structure.

[0015] The sparse point cloud is used as the center of the initial Gaussian distribution of the 3D Gaussian sputtering model. The current 3D Gaussian scene is rendered into a 2D image using differentiable rendering technology. The rendering loss between the 2D image and the surrounding image sequence is calculated, and the 3D Gaussian sputtering model is trained.

[0016] Adaptive density control is performed during the training and iteration of the 3D Gaussian sputtering model.

[0017] The optimizer iteratively optimizes all Gaussian point properties of the 3D Gaussian sputtering model to minimize rendering loss and generate a 3D Gaussian scene model file.

[0018] In one possible implementation, the adaptive density control includes: periodically pruning Gaussian points in regions where the transparency is lower than a preset transparency threshold in the 3D Gaussian sputtering model, and cloning and splitting Gaussian points in regions where the pose gradient is greater than a preset gradient threshold.

[0019] In one possible implementation, the gesture control includes: the trainee using a VR controller to grab virtual tools, disassemble and assemble parts within the virtual training scenario.

[0020] The work process guidance includes: setting up standard work procedures in the virtual training scenario, and using highlighting, arrows, and graphic prompts to guide students to complete the work process step by step.

[0021] In one possible implementation, the operational compliance detection includes: extracting the sequence of student operation steps and corresponding timestamps from the VR interaction event, comparing the sequence of student operation steps with the standard operating procedure using a dynamic time warping algorithm, and calculating the sequence compliance score.

[0022] In one possible implementation, the skill quantification statistics include: calculating skill quantification indicators from the VR interaction events, wherein the skill quantification indicators include total task duration, tool usage accuracy, component recognition accuracy, and number of erroneous operations.

[0023] The assessment report generation includes: a visual assessment report generated based on a three-level assessment model of memory-understanding-application and various quantitative indicators of skills.

[0024] The VR training system for new energy equipment based on visual 3D reconstruction disclosed in this application has the following advantages: By combining surround image sequences, motion recovery structures with 3D Gaussian sputtering hybrid algorithms, VR interaction, and cross-platform adaptation, the accuracy, interactivity, and openness of VR training for new energy equipment have been improved, while reducing modeling costs.

[0025] By employing a hybrid algorithm combining motion recovery structure and 3D Gaussian sputtering, feature point extraction and matching, sparse point cloud generation, differentiable rendering technology, adaptive density control, and iterative optimization are performed based on the surrounding image sequence to generate a 3D Gaussian scene model file. This improves the fidelity and rendering efficiency of 3D reconstruction of new energy equipment while balancing model lightweighting and real-time performance.

[0026] By implementing adaptive density control through pruning low-transparency Gaussian points and cloning and splitting Gaussian points in high-gradient regions during model training, the structural rationality and resource utilization efficiency of the 3D Gaussian model are improved, further enhancing the rendering smoothness of the virtual model.

[0027] Using VR controllers to grasp virtual tools, disassemble and assemble parts, and guided by highlights, arrows, and graphic prompts, trainees are guided step by step through the work process. This enhances the immersiveness and operational standardization of VR training for new energy equipment and reduces the learning difficulty of complex training processes.

[0028] By extracting the operation sequence and timestamp, and using a dynamic time warping algorithm to compare the standard operating procedure and calculate the compliance score, the accuracy and objectivity of the practical operation standardization detection are improved, and the real-time and accurate judgment of out-of-order and dangerous operations is achieved.

[0029] By statistically analyzing multiple quantitative indicators of skills and based on a three-level assessment model of memory-comprehension-application, a visual assessment report is generated according to each quantitative indicator of skills. This improves the comprehensiveness and quantitative level of the assessment of students' practical training effectiveness and provides an intuitive basis for teaching feedback and improvement. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a VR training system for new energy equipment based on visual 3D reconstruction, provided as an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] like Figure 1 As shown in the figure, this application provides a VR training system for new energy equipment based on visual 3D reconstruction, including: a data acquisition module, a 3D reconstruction module, a VR interaction module, a functional module, a cross-platform adaptation module, and a core control unit.

[0034] The data acquisition module is configured to perform surround shooting of the target new energy equipment to obtain a surround image sequence.

[0035] The 3D reconstruction module is configured to generate a 3D Gaussian scene model file based on the surrounding image sequence by employing a motion recovery structure and a 3D Gaussian sputtering hybrid algorithm.

[0036] The VR interaction module is configured to: call the 3D Gaussian scene model file to build a virtual training scene for students to control with gestures and guide students through the work process.

[0037] The functional module is configured to: monitor VR interaction events of trainees in the virtual training scenario in real time, perform operation standardization checks, quantify and statistically analyze skills, and generate evaluation reports.

[0038] The cross-platform adaptation module is configured to package the training content corresponding to the VR interaction module and the functional module into an independently runnable application, providing a multi-terminal operating environment for the VR interaction module and the functional module.

[0039] The core control unit is configured as a system server to coordinate and schedule the data acquisition module, the 3D reconstruction module, the VR interaction module, the functional modules, and the cross-platform adaptation module.

[0040] For example, the data acquisition module is further configured to: after obtaining the surround image sequence, perform noise reduction, exposure optimization and white balance correction on the surround image sequence.

[0041] Specifically, in this embodiment, the data acquisition module uses a handheld monocular consumer-grade camera to perform surround shooting of the target new energy device, acquiring a high-resolution image sequence covering key parts such as the device's appearance, interfaces, and nameplate, i.e., a surround image sequence. The camera has a built-in image signal processor that automatically performs noise reduction, exposure optimization, and white balance correction on the surround image sequence to ensure image quality.

[0042] In other possible embodiments, a video stream captured around the camera can be used instead of a sequence of images around the camera, and the initial pose and point cloud can be obtained through video frame extraction and synchronous localization and mapping techniques.

[0043] In this embodiment, the data acquisition module also collects the model, specifications, and environmental information of the target new energy equipment simultaneously through an auxiliary recording terminal, and performs spatiotemporal correlation with the surrounding image sequence.

[0044] For example, a motion recovery structure and 3D Gaussian sputtering hybrid algorithm is used to generate a 3D Gaussian scene model file based on the surrounding image sequence, including: For the surrounding image sequence, feature points are extracted and matched using the scale-invariant feature transform algorithm, and the camera pose is calculated and a sparse point cloud is generated using incremental motion recovery structure.

[0045] The sparse point cloud is used as the center of the initial Gaussian distribution of the 3D Gaussian sputtering model. The current 3D Gaussian scene is rendered into a 2D image using differentiable rendering technology. The rendering loss between the 2D image and the surrounding image sequence is calculated, and the 3D Gaussian sputtering model is trained.

[0046] Specifically, each Gaussian point has attributes such as position, covariance (controlling shape), opacity, and spherical harmonic coefficients (controlling color).

[0047] Adaptive density control is performed during the training and iteration of the 3D Gaussian sputtering model.

[0048] The optimizer iteratively optimizes all Gaussian point properties of the 3D Gaussian sputtering model to minimize rendering loss and generate a 3D Gaussian scene model file.

[0049] In other possible embodiments, 3D Gaussian sputtering can be replaced with neural radiation fields or their variants for neural rendering reconstruction, which can also achieve high-fidelity effects, but each has its own emphasis on real-time rendering efficiency.

[0050] For example, the adaptive density control includes: periodically pruning Gaussian points in regions where the transparency is lower than a preset transparency threshold in the 3D Gaussian sputtering model, and cloning and splitting Gaussian points in regions where the pose gradient is greater than a preset gradient threshold. This adaptively optimizes the distribution and number of Gaussian points.

[0051] Specifically, in this embodiment, the 3D reconstruction module ultimately generates a lightweight, high-fidelity 3D Gaussian scene model file that supports real-time rendering, and stores it in the new energy equipment digital model library.

[0052] Specifically, in this embodiment, the VR interaction module is developed based on the Unity engine. It calls the 3D Gaussian scene model from the digital model library of new energy equipment, configures physical colliders, bakes light maps, creates an immersive environment, and completes the construction of the virtual training scene.

[0053] For example, the gesture control includes: the trainee using a VR controller to grab virtual tools, disassemble and assemble parts in the virtual training scenario.

[0054] The work process guidance includes: setting up standard work procedures in the virtual training scenario, and using highlighting, arrows, and graphic prompts to guide students to complete the work process step by step.

[0055] Specifically, when performing gesture control, the system achieves precise operation through ray detection and object binding logic.

[0056] In other possible embodiments, in addition to VR controllers, data gloves or visual hand tracking can be combined to achieve more natural gesture control.

[0057] In one possible embodiment, the VR interaction module is also equipped with a voice recognition-based intelligent assistant. This intelligent assistant can receive voice commands from trainees to quickly retrieve virtual tools, query training steps by voice, and provide voice feedback for operation confirmation. During the disassembly and assembly training, trainees can replace manual operation through voice interaction, freeing their hands and improving the efficiency of training operations. This is suitable for training scenarios involving multiple processes and high concentration required for new energy equipment.

[0058] For example, the operation conformity detection includes: extracting the sequence of student operation steps and the corresponding timestamps from the VR interaction event, comparing the sequence of student operation steps with the standard operating procedure using a dynamic time warping algorithm, and calculating the sequence conformity score.

[0059] Specifically, in this embodiment, the dynamic time warping algorithm is used to perform temporal alignment and similarity calculation between the trainee's operation sequence and the standard operating procedure, adapting to the sequence mismatch problem caused by the difference in operation time; the sequence conformity score adopts a percentage system, based on the standard operating procedure, and calculates the score by combining the step order and execution logic matching degree. The higher the score, the more standardized the operation sequence, providing a quantitative basis for judging the standardization of operation.

[0060] Specifically, in this embodiment, during the operation compliance check, real-time alarms and point deductions are issued for omissions, out-of-order operations, or dangerous operations.

[0061] For example, the skill quantification statistics include: calculating skill quantification indicators from the VR interaction events, the skill quantification indicators including total task duration, tool usage accuracy, component recognition accuracy, and number of erroneous operations.

[0062] The assessment report generation includes: a visual assessment report generated based on a three-level assessment model of memory-understanding-application and various quantitative indicators of skills.

[0063] Specifically, in this embodiment, the visual evaluation report includes a radar chart, timeline, deduction details, and improvement suggestions, which are then fed back to students and teachers.

[0064] In this embodiment, the cross-platform adaptation module utilizes the cross-platform publishing capabilities of the Unity engine to package the training content corresponding to the VR interaction module and the functional modules into independently runnable applications, which are then published to VR all-in-one devices (Android system), Windows computers, and iOS tablets. Through a network synchronization framework, students on multiple terminals can access the same virtual scene for collaborative work or competitions.

[0065] In this embodiment, the core control unit acts as a system server, coordinating and scheduling the data acquisition module, the 3D reconstruction module, the VR interaction module, the functional modules, and the cross-platform adaptation module. It is responsible for task scheduling, data distribution, status synchronization, and instruction coordination, ensuring the smoothness and consistency of online training for multiple users.

[0066] In one possible embodiment, the system may adopt a cloud rendering mode, placing the core 3D rendering and computing tasks on the server side, while the terminal only receives streaming media and uploads interactive instructions, thereby further reducing the performance requirements of the terminal device.

[0067] This application's embodiments improve the accuracy, interactivity, and openness of VR training for new energy equipment by combining surround image sequences, motion recovery structures, and 3D Gaussian sputtering hybrid algorithms, VR interaction, and cross-platform adaptation, while reducing modeling costs.

[0068] By employing a hybrid algorithm combining motion recovery structure and 3D Gaussian sputtering, feature point extraction and matching, sparse point cloud generation, differentiable rendering technology, adaptive density control, and iterative optimization are performed based on the surrounding image sequence to generate a 3D Gaussian scene model file. This improves the fidelity and rendering efficiency of 3D reconstruction of new energy equipment while balancing model lightweighting and real-time performance.

[0069] By implementing adaptive density control through pruning low-transparency Gaussian points and cloning and splitting Gaussian points in high-gradient regions during model training, the structural rationality and resource utilization efficiency of the 3D Gaussian model are improved, further enhancing the rendering smoothness of the virtual model.

[0070] Using VR controllers to grasp virtual tools, disassemble and assemble parts, and guided by highlights, arrows, and graphic prompts, trainees are guided step by step through the work process. This enhances the immersiveness and operational standardization of VR training for new energy equipment and reduces the learning difficulty of complex training processes.

[0071] By extracting the operation sequence and timestamp, and using a dynamic time warping algorithm to compare the standard operating procedure and calculate the compliance score, the accuracy and objectivity of the practical operation standardization detection are improved, and the real-time and accurate judgment of out-of-order and dangerous operations is achieved.

[0072] By statistically analyzing multiple quantitative indicators of skills and based on a three-level assessment model of memory-comprehension-application, a visual assessment report is generated according to each quantitative indicator of skills. This improves the comprehensiveness and quantitative level of the assessment of students' practical training effectiveness and provides an intuitive basis for teaching feedback and improvement.

[0073] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0074] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A VR training system for new energy equipment based on visual 3D reconstruction, characterized in that, include: Data acquisition module, 3D reconstruction module, VR interaction module, functional module, cross-platform adaptation module, core control unit; The data acquisition module is configured to: perform surround shooting of the target new energy equipment to obtain a surround image sequence; The 3D reconstruction module is configured to generate a 3D Gaussian scene model file based on the surrounding image sequence by employing a hybrid algorithm of motion recovery structure and 3D Gaussian sputtering. The VR interaction module is configured to: call the 3D Gaussian scene model file to build a virtual training scene for students to perform gesture control and guide students through the work process; The functional module is configured to: monitor the VR interaction events of trainees in the virtual training scenario in real time, perform operation standardization detection, skill quantification statistics, and generate evaluation reports; The cross-platform adaptation module is configured to package the training content corresponding to the VR interaction module and the functional module into an independently runnable application, providing a multi-terminal operating environment for the VR interaction module and the functional module; The core control unit is configured as a system server to coordinate and schedule the data acquisition module, the 3D reconstruction module, the VR interaction module, the functional modules, and the cross-platform adaptation module.

2. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 1, characterized in that, The data acquisition module is further configured to perform noise reduction, exposure optimization, and white balance correction on the surrounding image sequence after obtaining it.

3. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 1, characterized in that, A hybrid algorithm combining motion reconstruction structure and 3D Gaussian sputtering is used to generate a 3D Gaussian scene model file based on the surrounding image sequence, including: For the surrounding image sequence, feature points are extracted and matched using the scale-invariant feature transform algorithm, and the camera pose is calculated and a sparse point cloud is generated using incremental motion reconstruction structure. The sparse point cloud is used as the center of the initial Gaussian distribution of the 3D Gaussian sputtering model. The current 3D Gaussian scene is rendered into a 2D image using differentiable rendering technology. The rendering loss between the 2D image and the surrounding image sequence is calculated, and the 3D Gaussian sputtering model is trained. Adaptive density control is performed during the training and iteration of the 3D Gaussian sputtering model; The optimizer iteratively optimizes all Gaussian point properties of the 3D Gaussian sputtering model to minimize rendering loss and generate a 3D Gaussian scene model file.

4. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 3, characterized in that, The adaptive density control includes: periodically pruning Gaussian points in regions where the transparency is lower than a preset transparency threshold in the 3D Gaussian sputtering model, and cloning and splitting Gaussian points in regions where the pose gradient is greater than a preset gradient threshold.

5. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 1, characterized in that, The gesture control includes: trainees using VR controllers to grab virtual tools, disassemble and assemble parts in the virtual training scene; The work process guidance includes: setting up standard work procedures in the virtual training scenario, and using highlighting, arrows, and graphic prompts to guide students to complete the work process step by step.

6. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 5, characterized in that, The operational compliance detection includes: extracting the sequence of student operation steps and corresponding timestamps from the VR interaction events, comparing the sequence of student operation steps with the standard operating procedure using a dynamic time warping algorithm, and calculating the sequence compliance score.

7. The VR training system for new energy equipment based on visual 3D reconstruction according to claim 1, characterized in that, The skill quantification statistics include: calculating skill quantification indicators from the VR interaction events, including total task duration, tool usage accuracy, component recognition accuracy, and number of erroneous operations; The assessment report generation includes: a visual assessment report generated based on a three-level assessment model of memory-understanding-application and quantitative indicators of various skills.

Images