Practical training method and system based on multi-mode Internet of Things perception and virtual-real symbiosis

Abstract

The invention provides a practical training method and system based on multi-modal Internet of Things perception and virtual-real symbiosis, and relates to the technical field of virtual-real symbiosis, and the method comprises the steps: collecting the multi-modal perception data of a practical training environment through a plurality of heterogeneous Internet of Things devices arranged in the practical training environment; constructing a three-dimensional digital twinborn model of the practical training environment according to the category and position information of the object in the practical training environment, the voice control instruction and the comprehensive environment parameters; the method comprises the following steps: setting training items and evaluation indexes in a virtual-real symbiotic training space, receiving a training item selection instruction input by training personnel, and loading a corresponding training item according to the selection instruction, and sending a control instruction of the practical training project to a virtual equipment model in the three-dimensional digital twinborn model and Internet of Things equipment mapped with the virtual equipment model in the practical training environment for practical training, and evaluating the practical training process and the practical training result of the practical training project according to the evaluation index after the practical training project is finished. And a training evaluation report is generated and fed back to the training management system.

Description

Technical Field

[0001] The present invention relates to virtual-reality symbiosis technology, and in particular to a training method and system based on multimodal Internet of Things perception and virtual-reality symbiosis. Background Art

[0002] In modern industry, medicine, military and education, practical training and simulation training have become key methods to improve skills and safety. However, traditional practical training methods usually rely on physical models and simulation equipment, which have problems such as high cost, difficulty in repeating, and lack of realistic scenes. In order to solve these problems, virtual reality (VR) and augmented reality (AR) technologies have been widely used in the field of practical training in recent years, and training and evaluation are carried out by creating virtual scenes and simulation environments. However, these systems usually face problems such as inaccurate virtual and real fusion, untimely real-time data updates, and insufficient dynamic response capabilities to actual equipment and environmental parameters.

[0003] In order to address these challenges, a training method and system based on multimodal IoT perception and virtual-real symbiosis is proposed. It can not only provide a more realistic and dynamic training experience, but also significantly improve the effect and quality of training through real-time data updates and precise virtual-real integration. Summary of the invention

[0004] The embodiments of the present invention provide a training method and system based on multimodal Internet of Things perception and virtual-reality symbiosis, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provide a practical training method based on multimodal IoT perception and virtual-real symbiosis, including:

[0007] By arranging a plurality of heterogeneous IoT devices in the training environment, multimodal perception data of the training environment is collected, wherein the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment, image segmentation processing is performed on the video data to obtain segmented target image data, the target image data is processed using a target detection algorithm to obtain the category and position information of the object in the training environment, feature extraction and voice recognition processing are performed on the audio data to obtain voice control instructions, and data fusion processing is performed on the temperature and humidity data, light intensity data and device status data to obtain comprehensive environmental parameters of the training environment;

[0008] According to the categories and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters, a three-dimensional digital twin model of the training environment is constructed, wherein the three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. The three-dimensional digital twin model is matched and mapped with the training environment in real time by using augmented reality technology to obtain a virtual-real symbiotic training space;

[0009] Training projects and evaluation indicators are set up in the virtual-real symbiotic training space, the selection instructions of the training projects input by the trainees are received, and the corresponding training projects are loaded according to the selection instructions, and the control instructions of the training projects are sent to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and after the training projects are completed, the training process and training results of the training projects are evaluated according to the evaluation indicators, and a training evaluation report is generated and fed back to the training management system.

[0010] In an optional embodiment,

[0011] According to the category and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters, a three-dimensional digital twin model of the training environment is constructed. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices, and attribute information of each three-dimensional model, including:

[0012] Acquire physical space data of the training environment, pre-process the physical space data to obtain a simplified three-dimensional geometric model, classify and label static geometric objects, set initial positions and target moving areas for movable objects, and obtain a three-dimensional geometric model of the training environment;

[0013] According to the category of objects in the training environment, the objects are matched with the preset 3D model library to obtain the matching 3D models of the objects. According to the category and position information of the objects, the corresponding 3D models of the objects are dynamically loaded into the 3D geometric model of the training environment to set the motion trajectory and interaction mode for the dynamic objects.

[0014] Acquire real-time data of heterogeneous IoT devices in a training environment, create a three-dimensional model of a heterogeneous IoT device including geometric shapes and functional attributes for each heterogeneous IoT device based on the real-time data, and map the real-time data of the heterogeneous IoT device to the corresponding three-dimensional model of the heterogeneous IoT device;

[0015] The three-dimensional geometric model of the training environment, the three-dimensional model of the object and the three-dimensional model of the heterogeneous Internet of Things are integrated to obtain a complete three-dimensional digital twin model.

[0016] In an optional embodiment,

[0017] Using augmented reality technology, the three-dimensional digital twin model is matched and mapped with the training environment in real time, and the training space with virtual and real coexistence is obtained, including:

[0018] Arrange visual signs in the training environment, capture them in real time through cameras, use image processing algorithms to identify and calculate the position and posture of the visual signs, and obtain the spatial position and posture information of the objects to which the visual signs are attached;

[0019] For unmarked objects, the corresponding visual feature points are extracted, and feature matching and pose estimation are performed in continuous video frames to track and locate the unmarked objects in real time, and obtain the spatial position and pose information of the unmarked objects;

[0020] The spatial position and posture information of the object to which the visual mark is attached and the spatial position and posture information of the object without the mark are transmitted to the three-dimensional digital twin model in real time. Through coordinate transformation and posture mapping, the movement and changes of the object to which the visual mark is attached and the object without the mark in the real environment are synchronously updated to the virtual objects of the three-dimensional digital twin model.

[0021] The motion data of the AR device is obtained, and combined with the visual odometer calculation method, the position and posture changes of the AR device in space are estimated, the motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device, and the rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space.

[0022] In an optional embodiment,

[0023] The motion data of the AR device is obtained, and the position and posture change of the AR device in space are estimated by combining the visual odometer calculation method. The motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device. The rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space, including:

[0024] Acquire and preprocess the training environment image data, input the preprocessed training environment image data into the pre-trained image segmentation model for semantic segmentation, and obtain the semantic segmentation results containing different object categories and boundary information. At the same time, use multi-eye vision equipment and stereo matching technology to analyze the parallax information of the left and right cameras to generate a depth map representing the scene depth information;

[0025] Based on the semantic segmentation results and the obtained depth map, a 3D model containing the object and plane information in the training environment is constructed through 3D reconstruction technology;

[0026] Acquire the IMU sensor data of the AR device, extract the feature points in the continuous training environment image frames, calculate the relative motion trajectory of the AR device by matching the feature points between frames, obtain the visual mileage calculation result, and optimize the acquired IMU sensor data and the visual mileage calculation result through the data fusion algorithm to obtain the pose estimation result of the AR device;

[0027] Based on the obtained three-dimensional model and the pose estimation results of the AR device, the optimal placement position and posture of the virtual object in the real environment are determined. According to the optimal placement position and posture of the virtual object, the rendering position and viewing angle of the virtual object are adjusted in real time. The display of the virtual object in the real environment is dynamically updated through the rendering engine to obtain the final virtual-real symbiotic training space.

[0028] In an optional embodiment,

[0029] Receiving a selection instruction of a training project input by a trainee, loading a corresponding training project according to the selection instruction, and sending a control instruction of the training project to a virtual device model in a three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in a training environment for training, including:

[0030] Design a project database containing multiple practical training projects, each of which includes task objectives, operation steps and evaluation criteria;

[0031] Receive a selection instruction of a training project input by a trainee, load a basic training project that meets the trainee's needs from a project database according to the selection instruction, send a control instruction of the loaded basic training project to a virtual device model in the three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in the training environment, simulate the operation and operating status of the device through the three-dimensional digital twin model, and generate training environment data;

[0032] The operational behavior data, body posture, and cognitive load information of the trainees are collected, combined with the training environment data, to generate comprehensive monitoring data of the training process, model and analyze the comprehensive monitoring data, and generate a personalized user portrait of the trainee. If it is detected according to the personalized user portrait that the trainee needs personalized training, a customized training project including task objectives, detailed operating steps, guidance information, and a virtual scene matching the task is automatically generated through a generative AI model, and the real-time operational performance of the trainees is continuously tracked. The content difficulty of the personalized training project and the detailed level of the detailed operating steps are dynamically adjusted, and training is carried out according to the adjusted personalized training project.

[0033] In an optional embodiment,

[0034] Customized training projects that automatically generate task objectives, detailed operation steps, guidance information, and virtual scenes that match the tasks through generative AI models include:

[0035] Constructing a knowledge graph for the practical training field, wherein the knowledge graph for the practical training field includes task objectives, operation steps, learning resources, and technical standards;

[0036] Use a generative AI model to reason about the structured data in the knowledge graph of the training field, identify the knowledge blind spots and skill advantages of trainees, generate initial personalized training projects by comparing the trainees' historical performance with the technical standards in the knowledge graph, and combine the trainees' current skill level and training needs, and set overall task goals and learning directions;

[0037] During the implementation of the personalized training project, sensors and biofeedback equipment are used to collect the trainees' operation data, posture and physiological state in real time, and the collected data are analyzed using data analysis technology and machine learning algorithms. According to the analysis results, the training content and difficulty of the customized training project are automatically adjusted;

[0038] Generate personalized operation steps and guidance information using a generative AI model based on the trainee's skill level and performance in a specific task. The guidance information includes text instructions, image recognition results, and interactive teaching tools. At the same time, define the design requirements of the virtual scene based on the set overall task objectives, collect and pre-process material data related to the virtual scene, and input the pre-processed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model.

[0039] The trained virtual scene generation model is used to generate a virtual scene, and the generated virtual scene is post-processed. The processed virtual scene, overall task objectives, personalized operation steps and guidance information are integrated to generate a complete customized training project.

[0040] In an optional embodiment,

[0041] Collecting and preprocessing material data related to the virtual scene, the material data including images, videos and three-dimensional models of the actual environment, and inputting the preprocessed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model includes:

[0042] The pre-processed material data is input into a predetermined generative adversarial network architecture, and the generator network and the discriminator network are optimized through an adversarial training process; wherein the generator network accepts random noise as input and generates a virtual scene image through a multi-layer convolutional neural network; the discriminator network evaluates the authenticity of the virtual scene image through a multi-layer convolution operation, and inputs the evaluation result as a feedback signal to the generator network;

[0043] During the adversarial training process, a loss function including adversarial loss and content loss is used to optimize the generator network and the discriminator network. The loss function formula is as follows:

[0044]

[0045] Where L represents the loss function, E[·] represents the expectation, D represents the discriminator, x represents the real sample, G(z) represents the generated sample, M represents the number of multi-scales, and λ i Represents the weight coefficient of content loss at the i-th scale, G(z) i represents the generated image at the i-th scale, T i represents the real image at the i-th scale.

[0046] According to a second aspect of the embodiments of the present invention,

[0047] Provide a training system based on multi-modal IoT perception and virtual-real symbiosis, including:

[0048] The first unit is used to collect multimodal perception data of the training environment through multiple heterogeneous Internet of Things devices arranged in the training environment, wherein the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment, perform image segmentation processing on the video data to obtain segmented target image data, process the target image data using a target detection algorithm to obtain the category and position information of the object in the training environment, perform feature extraction and voice recognition processing on the audio data to obtain voice control instructions, and perform data fusion processing on the temperature and humidity data, light intensity data and device status data to obtain comprehensive environmental parameters of the training environment;

[0049] The second unit is used to construct a three-dimensional digital twin model of the training environment according to the category and location information of the objects in the training environment, voice control instructions and comprehensive environmental parameters. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of the object, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. The three-dimensional digital twin model is matched and mapped with the training environment in real time by using augmented reality technology to obtain a virtual-real symbiotic training space.

[0050] The third unit is used to set up training projects and evaluation indicators in the virtual-real symbiotic training space, receive the selection instructions of the training projects input by the trainees, and load the corresponding training projects according to the selection instructions, and send the control instructions of the training projects to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and after the training projects are completed, evaluate the training process and training results of the training projects according to the evaluation indicators, generate a training evaluation report and feed it back to the training management system.

[0051] According to a third aspect of the embodiments of the present invention,

[0052] An electronic device is provided, comprising:

[0053] processor;

[0054] a memory for storing processor-executable instructions;

[0055] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the embodiments of the present invention is:

[0057] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0058] In this embodiment, through the combination of multimodal perception data and augmented reality technology, the virtual environment and the actual training environment can be seamlessly connected to provide a highly realistic training experience. The virtual and real symbiotic environment can reflect the changes in the actual environment in real time, enhancing the user's immersion and interactive experience. The application of image recognition technology and target detection algorithm can accurately identify the object category and location information in the training environment, dynamically load and control the corresponding three-dimensional model and virtual equipment, and ensure that the objects in the virtual environment are consistent with the objects in the actual environment. By setting training projects and evaluation indicators in the virtual and real symbiotic training space, a more objective and scientific training evaluation can be provided. The system can generate a detailed evaluation report based on the training process and results to help analyze and improve the training plan. Multi-sensor data fusion technology makes the processing of comprehensive environmental parameters more accurate, and the virtual scene can accurately reflect the environmental conditions in reality, thereby improving the authenticity and effectiveness of virtual training. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of a training method based on multi-modal IoT perception and virtual-real symbiosis according to an embodiment of the present invention;

[0060] Figure 2It is a structural schematic diagram of a training system based on multimodal Internet of Things perception and virtual-reality symbiosis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0063] Figure 1 FIG. 1 is a flow chart of a training method based on multimodal IoT perception and virtual-real symbiosis according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] S101. By arranging multiple heterogeneous Internet of Things devices in the training environment, multimodal perception data of the training environment is collected, and the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment. The video data is subjected to image segmentation processing to obtain segmented target image data, and the target image data is processed using a target detection algorithm to obtain the category and position information of objects in the training environment. The audio data is subjected to feature extraction and voice recognition processing to obtain voice control instructions. At the same time, the temperature and humidity data, light intensity data and device status data are subjected to data fusion processing to obtain comprehensive environmental parameters of the training environment.

[0065] For example, first, multiple heterogeneous IoT devices are deployed in the training environment, including video cameras, microphones, temperature and humidity sensors, light sensors, and device status monitors. Each device is responsible for collecting different types of sensory data to comprehensively monitor and record various states of the training environment.

[0066] The processing of video data begins with image segmentation of a real-time video stream. The goal of image segmentation is to separate different objects and regions in each frame. First, the video stream is decomposed into individual frames for processing. Each frame is preprocessed before processing, such as noise removal and contrast enhancement, to improve the accuracy of subsequent segmentation. Image segmentation algorithms (such as watershed algorithms or region growing) are applied to distinguish different regions in the image and generate independent image regions for each object.

[0067] Next, the segmented image is analyzed using an object detection algorithm to identify objects in the image and obtain their categories and locations. An appropriate object detection algorithm (such as YOLO or Faster R-CNN) is selected and the algorithm is trained to recognize various objects. During the detection process, the processed image is input into the object detection model, and the model returns the category label and location coordinates of the object.

[0068] The processing of audio data includes feature extraction and speech recognition of audio signals. First, the audio data is preprocessed, such as denoising and normalization. Then, feature extraction technology is used to extract the key features of the audio signal (such as Mel-frequency cepstral coefficients MFCC). Next, speech recognition technology is used to convert these features into text and parse the voice control instructions.

[0069] Temperature and humidity data, light intensity data, and equipment status data need to be fused. The data fusion process involves integrating data from different sensors to generate a comprehensive description of the environmental status. These data sources are synthesized into a comprehensive environmental parameter through a data fusion algorithm to provide an integrated environmental view.

[0070] In this embodiment, by arranging multiple heterogeneous Internet of Things devices in the training environment, the multimodal perception data of the training environment can be fully collected, including video, audio, temperature and humidity, light intensity, and device status data. The collection of such multimodal data can fully and accurately reflect the state of the training environment, and provide a rich information basis for subsequent analysis and model building. By performing image segmentation and target detection on video data, various objects in the training environment can be accurately identified and located. Image segmentation helps to separate specific object areas, while target detection provides the categories and locations of these objects, effectively supporting object recognition and tracking. Feature extraction and speech recognition of audio data can realize automatic parsing of the experimenter's voice instructions, and support operation and adjustment of the training environment through voice control. The implementation of such voice control simplifies the human-computer interaction process and improves the convenience and efficiency of operation. Data fusion processing of temperature and humidity, light intensity, and device status data can provide a comprehensive view of environmental parameters. This fusion processing can provide a comprehensive understanding of environmental changes and help to monitor and adjust experimental conditions in real time. Finally, by building a 3D digital twin model of the training environment, a virtual-real symbiotic training space can be realized. This model can accurately reflect the state of objects, equipment, and environment in the actual environment, and interact with the trainees' operating instructions and environmental data in real time. This virtual-real symbiotic training space not only enhances the immersion and sense of reality of the training, but also provides an efficient training and testing platform.

[0071] S102. Construct a three-dimensional digital twin model of the training environment based on the categories and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. Use augmented reality technology to perform real-time matching and mapping between the three-dimensional digital twin model and the training environment to obtain a virtual-real symbiotic training space.

[0072] Among them, the 3D geometric model is a digital representation of the physical space in the training environment, which can include the layout of the room, walls, floors and other basic structures. The model can be constructed by a 3D scanner, laser rangefinder or manual modeling software. Through augmented reality technology, the virtual and real symbiotic training space is presented to users. Users can view and interact with the 3D digital twin model through virtual reality or augmented reality devices, and operate with objects and equipment in the actual training environment.

[0073] In an optional embodiment,

[0074] According to the category and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters, a three-dimensional digital twin model of the training environment is constructed. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices, and attribute information of each three-dimensional model, including:

[0075] Acquire physical space data of the training environment, pre-process the physical space data to obtain a simplified three-dimensional geometric model, classify and label static geometric objects, set initial positions and target moving areas for movable objects, and obtain a three-dimensional geometric model of the training environment;

[0076] According to the category of objects in the training environment, the objects are matched with the preset 3D model library to obtain the matching 3D models of the objects. According to the category and position information of the objects, the corresponding 3D models of the objects are dynamically loaded into the 3D geometric model of the training environment to set the motion trajectory and interaction mode for the dynamic objects.

[0077] Acquire real-time data of heterogeneous IoT devices in a training environment, create a three-dimensional model of a heterogeneous IoT device including geometric shapes and functional attributes for each heterogeneous IoT device based on the real-time data, and map the real-time data of the heterogeneous IoT device to the corresponding three-dimensional model of the heterogeneous IoT device;

[0078] The three-dimensional geometric model of the training environment, the three-dimensional model of the object and the three-dimensional model of the heterogeneous Internet of Things are integrated to obtain a complete three-dimensional digital twin model.

[0079] For example, first, the physical space data of the training environment is obtained. Such data generally include the structural information of the environment, the position and size of objects, etc. Such data can be collected by laser scanning, rangefinder or other sensor equipment.

[0080] The acquired physical space data is preprocessed to simplify and optimize the 3D geometric model. This step includes removing unnecessary details, simplifying geometric shapes, and reducing the amount of data to facilitate subsequent processing and modeling. The simplified 3D geometric model represents the basic spatial structure of the training environment.

[0081] Classify and label static geometric objects. These static objects may include tables, chairs, walls, etc. These objects are fixed in the training environment and do not move. During the labeling process, you need to assign a category to each static object and record its location and size information.

[0082] For movable objects, it is necessary to set their initial position and possible moving area. Movable objects may include robots, mobile devices, etc., which can move in the environment. Setting the initial position is to determine the current position of the object, and the target moving area refers to the area that the object may reach, which is crucial for dynamic modeling and simulation.

[0083] Based on the above information, a complete three-dimensional geometric model of the training environment is obtained. This model includes the spatial distribution of all static objects and movable objects, and constitutes the basic structure of the training environment.

[0084] Next, the objects in the training environment are matched with the preset 3D model library according to their categories. These preset 3D model libraries contain 3D models of various objects, such as experimental equipment, tools, etc. Through matching, the object categories in the training environment are compared with the corresponding models in the model library to obtain the matching object 3D models.

[0085] According to the category and position information of the object, the acquired 3D model of the object is dynamically loaded into the 3D geometric model of the training environment. This step involves embedding the matching 3D model into the environment model to ensure that the location and category information of the model correctly reflect the actual environment. For dynamic objects, it is necessary to set the motion trajectory and interaction mode for them. This includes defining the object's movement path and possible interactive operations, such as the path planning of the robot or the operation of the tool.

[0086] In addition, real-time data of heterogeneous IoT devices in the training environment are obtained. These devices may include sensors, controllers, monitoring instruments, etc., which can provide real-time status data in the environment.

[0087] Based on these real-time data, a 3D model is created for each heterogeneous IoT device, including geometric form and functional attributes. These models need to reflect the actual appearance and functional characteristics of the device. The real-time data of heterogeneous IoT devices is mapped to these 3D models, so that the virtual model can display the status of the device in real time, such as operating conditions, sensor readings, etc.

[0088] Finally, the 3D geometric model of the training environment, the 3D object model, and the 3D model of heterogeneous IoT devices are integrated. This step combines the various models into a complete 3D digital twin model, allowing all components and information to be displayed in a unified virtual environment. The integrated model includes the geometry, object model, and device model of the entire training environment, providing an accurate virtual mirror for training, monitoring, and control.

[0089] In this embodiment, a simplified but accurate three-dimensional geometric model can be generated by acquiring and preprocessing the physical space data of the training environment. This model can effectively represent the spatial structure of the training environment and provide a basis for further virtual modeling and analysis. Classifying and labeling static geometric objects, and setting the initial position and target moving area for movable objects can ensure accurate classification and position tracking of objects in the training environment. This enables the objects in the model to reflect the physical properties and functions of the actual environment, thereby enhancing the authenticity and accuracy of the simulation. According to the category of the object, the preset three-dimensional model library is matched, and the corresponding object three-dimensional model can be dynamically loaded. Such a matching mechanism ensures that each object in the training environment has a corresponding three-dimensional model, and is dynamically updated according to the category and location information of the object, so that the three-dimensional digital twin model is always kept up to date. Acquiring and creating a three-dimensional model of heterogeneous IoT devices including geometric morphology and functional attributes, and mapping it with real-time data, can accurately reflect the actual state of the device. This real-time data mapping function supports effective monitoring and control of IoT devices, and improves the intelligence and automation level of the system. It realizes accurate modeling, dynamic updating and intelligent management of the training environment, improves the operability, authenticity and interactivity of the training system, and thus enhances the user's training experience and training effect.

[0090] In an optional embodiment,

[0091] Using augmented reality technology, the three-dimensional digital twin model is matched and mapped with the training environment in real time, and the training space with virtual and real coexistence is obtained, including:

[0092] Arrange visual signs in the training environment, capture them in real time through cameras, use image processing algorithms to identify and calculate the position and posture of the visual signs, and obtain the spatial position and posture information of the objects to which the visual signs are attached;

[0093] For unmarked objects, the corresponding visual feature points are extracted, and feature matching and pose estimation are performed in continuous video frames to track and locate the unmarked objects in real time, and obtain the spatial position and pose information of the unmarked objects;

[0094] The spatial position and posture information of the object to which the visual mark is attached and the spatial position and posture information of the object without the mark are transmitted to the three-dimensional digital twin model in real time. Through coordinate transformation and posture mapping, the movement and changes of the object to which the visual mark is attached and the object without the mark in the real environment are synchronously updated to the virtual objects of the three-dimensional digital twin model.

[0095] The motion data of the AR device is obtained, and combined with the visual odometer calculation method, the position and posture changes of the AR device in space are estimated, the motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device, and the rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space.

[0096] Exemplarily, visual identifiers (such as barcodes, QR codes or pattern labels) are arranged in a training environment, and these visual identifiers are used to provide identification information of objects. Images of these visual identifiers are captured in real time using a camera. Image processing algorithms (including image segmentation, feature extraction and pattern recognition) are applied to identify the specific content and position of the visual identifier. Through these processes, the spatial position and posture information of the object to which the visual identifier is attached can be obtained. The spatial position refers to the specific coordinates of the object in three-dimensional space, and the posture information includes the rotation angle and direction of the object.

[0097] For objects without visual identification, the system extracts the visual feature points of the object (such as corners, edges, etc.) and performs feature matching and pose estimation in consecutive video frames. The feature matching process involves finding the same feature points in different image frames and estimating the pose of the object through the movement of these matching points. Pose estimation refers to calculating the position and orientation of an object in three-dimensional space. In this way, the spatial position and posture information of unidentified objects can be tracked and located in real time.

[0098] The spatial position and posture information of objects with visual markers and unmarked objects are transmitted to the 3D digital twin model in real time. Through coordinate transformation and pose mapping, these real-time data are synchronously updated to the virtual objects in the 3D digital twin model. Coordinate transformation is the process of converting the coordinate system in the actual environment to the coordinate system in the virtual environment. Pose mapping is the process of converting the real posture of the object into the corresponding posture in the virtual environment.

[0099] Obtain the motion data of the AR device, including its position and posture changes. Use Visual Odometry to estimate the position and posture changes of the AR device in space. Visual Odometry infers the displacement and rotation of the device by analyzing the motion of feature points in continuous image frames. Fuse the motion data of the AR device with the visual odometry results to obtain accurate pose estimation results. Based on this result, dynamically adjust the rendering position and perspective of the virtual object to match the actual perspective of the AR device, and obtain the final virtual-real symbiotic training space.

[0100] In this embodiment, through visual identification and feature point tracking, both marked and unmarked objects can be accurately tracked and located in real time. This improves the ability to manage and control objects in the training environment. The position and posture information of visual identification and unmarked objects can be updated to the three-dimensional digital twin model in real time to ensure that the data in the virtual environment is consistent with the actual environment, thereby improving the authenticity and interactivity of the virtual environment. The motion data of the AR device is integrated with the visual mileage calculation results, and the rendering position and viewing angle of the virtual object can be dynamically adjusted, thereby enhancing the immersion and interactivity of the virtual and real symbiotic training space. Through the virtual and real symbiotic training space, users can obtain operations and feedback that are consistent with the real environment in the virtual environment, improving the actual effect and training quality of the training process.

[0101] In an optional embodiment,

[0102] The motion data of the AR device is obtained, and the position and posture change of the AR device in space are estimated by combining the visual odometer calculation method. The motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device. The rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space, including:

[0103] Acquire and preprocess the training environment image data, input the preprocessed training environment image data into the pre-trained image segmentation model for semantic segmentation, and obtain the semantic segmentation results containing different object categories and boundary information. At the same time, use multi-eye vision equipment and stereo matching technology to analyze the parallax information of the left and right cameras to generate a depth map representing the scene depth information;

[0104] Based on the semantic segmentation results and the obtained depth map, a 3D model containing the object and plane information in the training environment is constructed through 3D reconstruction technology;

[0105] Acquire the IMU sensor data of the AR device, extract the feature points in the continuous training environment image frames, calculate the relative motion trajectory of the AR device by matching the feature points between frames, obtain the visual mileage calculation result, and optimize the acquired IMU sensor data and the visual mileage calculation result through the data fusion algorithm to obtain the pose estimation result of the AR device;

[0106] Based on the obtained three-dimensional model and the pose estimation results of the AR device, the optimal placement position and posture of the virtual object in the real environment are determined. According to the optimal placement position and posture of the virtual object, the rendering position and viewing angle of the virtual object are adjusted in real time. The display of the virtual object in the real environment is dynamically updated through the rendering engine to obtain the final virtual-real symbiotic training space.

[0107] Exemplarily, first, image data of the training environment is obtained. This step uses a multi-eye camera (usually two mutually calibrated left and right cameras) to simultaneously shoot the environmental scene and capture two left and right images. In order to improve the quality of the data, the image needs to be preprocessed. Preprocessing includes denoising, standardization and geometric correction of the image so that the parallax between the left and right images can be accurately extracted. The purpose of image denoising is to eliminate noise that may be caused by camera sensors or changes in ambient light; standardization ensures that the brightness and contrast of different images are consistent, which is convenient for subsequent processing; geometric correction is to reduce problems such as camera lens distortion. Preprocessed images are more easily adapted to deep learning models.

[0108] Next, the preprocessed image data is input into a pre-trained image segmentation model for semantic segmentation. Semantic segmentation is a technique that uses deep learning technology to classify each pixel in an image into a specific object category. For example, pixels in a scene such as a table, floor, wall, etc. are classified into different categories to generate a semantic segmentation map. This step uses advanced convolutional neural network (CNN) models such as Mask R-CNN, DeepLab V3+, etc., which are able to identify object categories in images at the pixel level. The semantic segmentation results provide object category, location, and boundary information, allowing the system to accurately know every object in the training environment.

[0109] To further understand the three-dimensional structure of the scene, the system uses multi-viewing equipment to analyze the parallax information of the left and right cameras through stereo matching technology. Stereo matching calculates the distance between the feature point and the camera by comparing the displacement (called parallax) of the same feature point in the left and right images. The result of this step is a depth map, in which the depth value of each pixel in the depth map represents the distance of the point from the camera. The depth map can tell the position of each object and surface in three-dimensional space.

[0110] Based on the semantic segmentation results and depth map, the system uses 3D reconstruction technology to generate a 3D model of the training environment. The key to 3D reconstruction is to combine 2D image information with depth information to build a model that can reflect the position, size and shape of objects in space. The 3D model combines the classification information of semantic segmentation with depth information, allowing the system to not only identify each object in the environment, but also understand their 3D position and geometry.

[0111] Next, in order to track the position and posture of the AR device in space in real time, the system obtains motion data from the IMU sensor of the AR device. The IMU sensor includes an accelerometer and a gyroscope, which can measure the acceleration, angular velocity and other data of the device to describe the movement and rotation of the device. IMU data can provide the short-term motion trajectory of the device, but due to the cumulative error of the sensor, relying solely on IMU data will cause the estimation error to grow over time.

[0112] To improve accuracy, the system combines visual odometry. Visual odometry extracts feature points in the scene through continuous image frames, and matches and tracks these feature points between different frames. By analyzing the motion trajectory of these feature points, the system can infer the relative position change of the AR device. This process can compensate for the accumulated errors in the IMU data, making the pose estimation of the AR device more accurate.

[0113] IMU data and visual odometer calculation results are optimized through fusion algorithms. Commonly used algorithms are extended Kalman filter (EKF) or graph-based optimization methods. These algorithms can fuse the short-term high-frequency information provided by IMU data with the long-term position information provided by visual odometer calculation, eliminate noise and error accumulation, and ultimately obtain more stable and accurate AR device pose estimation results.

[0114] Based on the pose estimation results of the AR device and the generated 3D environment model, the system can determine the optimal placement and posture of the virtual object. The position and posture of the virtual object must not only take into account the physical limitations of the scene (such as the height of the desktop or the ground), but also be adjusted according to the user's perspective and the properties of the virtual object to ensure the natural integration of the virtual object with the real environment.

[0115] Finally, the rendering position and viewing angle of the virtual object will be adjusted in real time according to the pose estimation results of the AR device. The rendering engine is responsible for dynamically updating the display of virtual objects in the real environment, so that users can always see the correct position and status of virtual objects from different viewing angles. Through this dynamic update and real-time adjustment, the system realizes the seamless combination of virtual objects and real scenes, and builds a virtual and real symbiotic training space.

[0116] Suppose that equipment maintenance training is conducted in an industrial workshop. First, real-time images are acquired through cameras in the workshop, and these images are preprocessed to enhance their quality. Next, an image segmentation model is used to identify various equipment in the workshop, such as robotic arms, tools, and workbenches, and mark their locations and categories. At the same time, a depth map is acquired through a dual-camera system to show the specific distance of each device from the camera, thereby helping to build a three-dimensional model.

[0117] AR devices (such as AR glasses) are used to track the user's perspective and position. The visual mileage results calculated by IMU sensors and feature point matching are used to accurately estimate the current position and direction of the AR device. Combined with this data, virtual equipment (such as maintenance tools or manuals) are accurately displayed in the AR environment next to the location of the actual equipment in the workshop, ensuring seamless integration of the virtual and real environments.

[0118] In this embodiment, by combining semantic segmentation, depth map and 3D reconstruction technology, an accurate 3D model can be generated to describe in detail the objects and their spatial layout in the training environment. By fusion of the IMU sensor data of the AR device and the visual mileage calculation results, the position and posture of the device can be accurately estimated, thereby ensuring the correct display of virtual objects in the real environment. · By updating the position and perspective of virtual objects in real time, a training space with virtual and real symbiosis is provided, allowing users to interact naturally with virtual objects in an augmented reality environment. During the training process, users can obtain a more intuitive and realistic operating experience, enhance their understanding of the equipment and environment, and improve the training effect and learning efficiency.

[0119] S103. Set up training projects and evaluation indicators in the virtual-real symbiotic training space, receive the selection instructions of the training projects input by the trainees, and load the corresponding training projects according to the selection instructions, send the control instructions of the training projects to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and evaluate the training process and training results of the training projects according to the evaluation indicators after the training projects are completed, generate a training evaluation report and feed it back to the training management system.

[0120] In the virtual-real symbiosis training space, the first thing to do is to set up training projects and evaluation indicators. Training projects refer to specific tasks or operating procedures designed to cultivate specific skills, which may include equipment operation, fault diagnosis, process simulation, etc. Evaluation indicators are the standards used to measure the effectiveness of training, which may include operation accuracy, completion time, and compliance with safety regulations.

[0121] The system needs to provide a user interface that allows trainees to browse the list of available training projects and select the training project they want to conduct through clicking, voice commands or other interactive methods. This selection process will generate a selection instruction that contains the unique identifier of the selected training project.

[0122] After receiving the selection instruction, the system will load the corresponding training project from the pre-prepared training project database according to the project identifier in the instruction. The loading process includes reading project description, operation steps, relevant equipment information, evaluation criteria and other data.

[0123] After loading is complete, the system will generate a series of control instructions, which are used to guide the virtual device model in the 3D digital twin model to perform corresponding operations. At the same time, the system will convert these control instructions into a format suitable for IoT devices and send them to the actual IoT devices corresponding to the virtual device model in the training environment through the network.

[0124] A virtual device model is a digital representation of an actual device in a three-dimensional digital twin model, which can simulate the appearance, function and behavior of a real device. An IoT device is a physical device that can connect and communicate over the network, receive instructions and perform corresponding operations.

[0125] During the training process, the system will monitor and record the trainees’ operating behaviors, equipment status changes, environmental parameters and other data in real time. These data will be used for subsequent evaluation and analysis.

[0126] After the training project is completed, the system will conduct a comprehensive evaluation of the entire training process based on the preset evaluation indicators. The evaluation process includes analyzing the correctness of the operation, calculating the completion time, checking the compliance with safety regulations, etc. The system will compare the performance of the trainees with the preset standards and obtain a quantitative scoring result.

[0127] Based on the evaluation results, the system will automatically generate a detailed training evaluation report. The report usually includes an overview of the training project, a review of the operation process, scores of various indicators, an overall evaluation, improvement suggestions, etc. The generated report uses an easy-to-understand format and may include text descriptions, data charts, operation playback videos, etc.

[0128] Finally, the system will send the generated training evaluation report to the training management system. The training management system is a software platform that centrally manages training data, student information, course schedules, etc. It will associate the received evaluation report with the corresponding student profile to track the student's learning progress, analyze training results, and develop personalized learning plans.

[0129] Through this series of steps, the virtual-real symbiotic training space can provide trainees with an immersive learning experience, while also providing training institutions with accurate and objective evaluation tools, which helps improve the effectiveness and efficiency of training.

[0130] In an optional embodiment,

[0131] Receiving a selection instruction of a training project input by a trainee, loading a corresponding training project according to the selection instruction, and sending a control instruction of the training project to a virtual device model in a three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in a training environment for training, including:

[0132] Design a project database containing multiple practical training projects, each of which includes task objectives, operation steps and evaluation criteria;

[0133] Receive a selection instruction of a training project input by a trainee, load a basic training project that meets the trainee's needs from a project database according to the selection instruction, send a control instruction of the loaded basic training project to a virtual device model in the three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in the training environment, simulate the operation and operating status of the device through the three-dimensional digital twin model, and generate training environment data;

[0134] The operational behavior data, body posture, and cognitive load information of the trainees are collected, combined with the training environment data, to generate comprehensive monitoring data of the training process, model and analyze the comprehensive monitoring data, and generate a personalized user portrait of the trainee. If it is detected according to the personalized user portrait that the trainee needs personalized training, a customized training project including task objectives, detailed operating steps, guidance information, and a virtual scene matching the task is automatically generated through a generative AI model, and the real-time operational performance of the trainees is continuously tracked. The content difficulty of the personalized training project and the detailed level of the detailed operating steps are dynamically adjusted, and training is carried out according to the adjusted personalized training project.

[0135] For example, first of all, the design of the project database is one of the core foundations of the program. The project database stores multiple pre-defined training projects, each of which contains task objectives, which clearly describe the ultimate purpose of the training, such as proficiently operating a certain equipment, completing maintenance tasks, or optimizing system performance; operation steps, the execution steps of each task are listed in detail, guiding the trainees to complete the task step by step according to the process. The steps can be the operation of the machine, the start or stop of the equipment, or the diagnosis and treatment of a certain fault phenomenon; evaluation criteria, set specific indicators for measuring the performance of trainees, which may include the accuracy of completing the task, execution time, and standardization of the operation.

[0136] The design of the project database needs to be combined with actual needs to ensure that the content covers a wide range, is reasonably classified, and is scalable to facilitate the subsequent addition of new projects.

[0137] Before the training begins, the system receives the trainee's selection instructions through the interactive interface. The interactive forms can include a graphical user interface (GUI): a list of projects is displayed on the screen, and trainees can select by clicking; voice instructions: the system is equipped with a voice recognition module, and trainees can select projects through voice commands; gesture recognition: in augmented reality (AR) devices, gesture recognition is used for more natural interaction. When the selection instruction is entered, the system will load the corresponding basic training project from the project database according to the selection result. Loading includes retrieving the relevant task objectives, operation steps and evaluation criteria of the training project.

[0138] When the system determines the training project, it sends control instructions to the 3D digital twin model to map the virtual device model with the actual IoT device. The key is to simulate the operation and operating status of the device through the digital twin model, thereby generating real-time data for the training environment. The 3D digital twin model is an accurate virtualization of the actual physical environment, which keeps the status, location, and functional attributes of devices and objects synchronized in the virtual environment.

[0139] During the training process, the system collects the trainees' operational behavior data and physiological status through a variety of sensors and devices. The operational behavior data includes each step of the user's operation on the device, the order of execution of the steps, and the speed and accuracy of the operation. Body posture can be monitored through cameras, depth sensors or AR devices to help the system evaluate whether the trainees' operating posture meets the standards. Cognitive load information evaluates the trainees' mental stress and concentration during task execution by monitoring physiological indicators such as heart rate and eye movement trajectory.

[0140] Training environment data refers to information such as equipment status and operation results collected through three-dimensional digital twin models and IoT devices. These data are combined with trainees' operation behavior data, body posture and cognitive load information to form a comprehensive monitoring of the training process. Comprehensive monitoring data provides a comprehensive evaluation of trainees through modeling analysis. Modeling analysis will generate personalized user portraits based on dimensions such as operation accuracy, completion of operation steps, correctness of posture and cognitive load. Personalized user portraits are generated based on the performance of trainees during the training process, and they reflect the user's operating habits, operating skills, cognitive abilities and other information. If the system detects that the trainee's performance is not up to standard or has specific weaknesses through personalized user portrait analysis, the system will automatically determine that the user needs personalized training.

[0141] Based on personalized user portraits, the system automatically generates personalized training plans for the needs of trainees through generative AI models. The generative AI model adjusts the goals, operation steps, and guidance information of the training tasks according to the user's operational weaknesses and the difficulty of the training project. This generation process can not only customize the project, but also dynamically adjust the project difficulty according to the real-time performance of the trainees. For example, if a certain operation step fails multiple times, the system will provide more detailed guidance information or reduce the difficulty of the task. During the execution of the personalized training project, the system will continue to track the trainees' operational performance and adjust the difficulty and detail of the training content in real time. During this process, the AI ​​model will evaluate the user's progress, error feedback, and adaptability. For example, the system can gradually reduce the guidance information or increase the complexity of the task to ensure that the user's skills are improved. Finally, the results and performance of the personalized training project will be integrated into a training evaluation report.

[0142] In this embodiment, the system can dynamically generate personalized training programs based on the real-time performance of the trainees to ensure that everyone can gradually improve their skills at a pace that suits them. Through real-time monitoring of operating behavior, body posture, and cognitive load, the system can fully understand the status of the trainees and ensure the standardization and safety of operations. The generative AI model can automatically generate new training programs, reducing the burden of manually designed training tasks and improving the flexibility and pertinence of training. The difficulty and details of the training content will be dynamically adjusted according to user performance to ensure that each user can effectively improve their abilities. Combined with three-dimensional digital twin models and IoT devices, users can obtain a highly simulated training experience in a virtual and real environment to improve learning outcomes.

[0143] In an optional embodiment,

[0144] Customized training projects that automatically generate task objectives, detailed operation steps, guidance information, and virtual scenes that match the tasks through generative AI models include:

[0145] Constructing a knowledge graph for the practical training field, wherein the knowledge graph for the practical training field includes task objectives, operation steps, learning resources, and technical standards;

[0146] Use a generative AI model to reason about the structured data in the knowledge graph of the training field, identify the knowledge blind spots and skill advantages of trainees, generate initial personalized training projects by comparing the trainees' historical performance with the technical standards in the knowledge graph, and combine the trainees' current skill level and training needs, and set overall task goals and learning directions;

[0147] During the implementation of the personalized training project, sensors and biofeedback equipment are used to collect the trainees' operation data, posture and physiological state in real time, and the collected data are analyzed using data analysis technology and machine learning algorithms. According to the analysis results, the training content and difficulty of the customized training project are automatically adjusted;

[0148] Generate personalized operation steps and guidance information using a generative AI model based on the trainee's skill level and performance in a specific task. The guidance information includes text instructions, image recognition results, and interactive teaching tools. At the same time, define the design requirements of the virtual scene based on the set overall task objectives, collect and pre-process material data related to the virtual scene, and input the pre-processed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model.

[0149] The trained virtual scene generation model is used to generate a virtual scene, and the generated virtual scene is post-processed. The processed virtual scene, overall task objectives, personalized operation steps and guidance information are integrated to generate a complete customized training project.

[0150] Exemplarily, first construct a knowledge graph in the field of practical training, then use the generative AI model to design and dynamically adjust personalized practical training projects, and finally generate a complete customized practical training project. The construction of the knowledge graph in the field of practical training is the basis of the entire method. Knowledge graph is a structured way of representing knowledge, using nodes to represent entities and edges to represent the relationship between entities. The knowledge graph in the field of practical training contains four main components: task objectives, operating steps, learning resources and technical standards. Task objectives refer to the specific results that the practical training project hopes to achieve; operating steps are the specific sequence of actions required to complete the task; learning resources include relevant teaching materials, videos, cases and other auxiliary materials; technical standards are the objective basis for measuring the quality of practical training. By integrating and organizing this information, a comprehensive and systematic knowledge network can be constructed to provide a rich knowledge base for the subsequent design of personalized practical training projects.

[0151] After the knowledge graph is built, the next step is to use the generative AI model to infer the structured data in the knowledge graph to identify the trainees' knowledge blind spots and skill advantages. Generative AI models, such as the GPT (Generative Pre-trained Transformer) series of models, have powerful natural language understanding and generation capabilities. By inputting the trainees' historical performance data into the model and comparing it with the technical standards in the knowledge graph, the model can analyze the trainees' mastery of each skill point. At the same time, combined with the trainees' current skill level and training needs, the generative AI model can generate an initial personalized training project, including setting overall task goals and learning directions. This process not only takes into account the individual differences of trainees, but also ensures the pertinence and practicality of the training content.

[0152] During the implementation of personalized training projects, the system will use various sensors and biofeedback devices to collect the trainees' operation data, posture and physiological state in real time. These devices may include motion capture systems, eye trackers, heart rate monitors, etc. The collected data is processed and analyzed through data analysis technology and machine learning algorithms. For example, time series analysis can be used to identify abnormal patterns in operations, or clustering algorithms can be used to discover performance differences between different trainees. Based on the analysis results, the system can automatically adjust the training content and difficulty of the customized training project to achieve dynamic optimization of the training process.

[0153] Based on the trainees' skill level and performance in specific tasks, the system uses a generative AI model to generate personalized operating steps and guidance information. This information includes text descriptions, image recognition results, and interactive teaching tools. The text description may be a detailed explanation of the operating steps; the image recognition results can help trainees identify tools or parts; the interactive teaching tool may be a virtual 3D model that allows trainees to practice virtual operations. At the same time, the system will define the design requirements of the virtual scene based on the set overall task objectives. This includes aspects such as the complexity, interactivity, and realism of the scene. The system collects material data related to the virtual scene, such as images, videos, and three-dimensional models of the actual environment, and pre-processes the data, such as denoising, standardization, and enhancement.

[0154] The pre-processed material data is input into the predetermined Generative Adversarial Network (GAN) architecture to train the virtual scene generation model. GAN consists of two parts: the generator and the discriminator. It generates high-quality virtual scenes through adversarial learning. After training, the system uses this virtual scene generation model to generate virtual scenes that meet the design requirements. The generated virtual scene also needs to be post-processed, such as detail optimization, light and shadow adjustment, to improve the realism and immersion of the scene.

[0155] Finally, the system integrates the processed virtual scenes, overall task objectives, personalized operation steps and guidance information to generate a complete customized training project. This project is a comprehensive, personalized and immersive learning environment that can provide the most suitable learning experience according to the characteristics and needs of trainees.

[0156] Among them, the process of using the generative adversarial network architecture to train the virtual scene generation model first requires the collection of material data related to the virtual scene. These material data mainly include three categories: images, videos and 3D models of the actual environment. Image data can be high-resolution static photos that capture the texture, color and details of the environment; video data can provide dynamic information to help understand the motion characteristics of objects in the environment; and 3D models provide geometric structure information of the environment. These material data should cover all aspects of the target virtual scene as comprehensively as possible to ensure the diversity and authenticity of the generated virtual scene.

[0157] The collected material data needs to be preprocessed. The preprocessing steps may include: denoising, color correction, and lighting equalization of images and videos; mesh simplification and texture optimization of 3D models. In addition, the data needs to be normalized to adapt it to the input requirements of the neural network. The preprocessed data should maintain the key features of the original material and have a unified format and appropriate scale to facilitate subsequent network training.

[0158] After preprocessing, the processed material data is input into the predetermined Generative Adversarial Network (GAN) architecture. GAN is a deep learning model consisting of two parts: the generator network and the discriminator network. The main task of the generator network is to generate virtual scene images. It accepts random noise as input, which can be regarded as the potential representation of the virtual scene. The generator network processes these noises through multiple layers of convolutional neural networks, gradually converting them into images with structure and semantics. Each layer of the convolutional neural network can learn features of different scales and abstract levels, so that it can generate virtual scene images with rich details.

[0159] The discriminator network is responsible for evaluating the authenticity of the generated virtual scene images. It also uses a multi-layer convolutional neural network structure, but its goal is to distinguish generated images from real images. The discriminator network extracts image features through a series of convolution operations and finally outputs a score indicating the degree of authenticity of the image. This score is input into the generator network as a feedback signal to guide the generator to continuously improve its generation ability.

[0160] The generator and discriminator are continuously optimized through the adversarial training process. In this process, the generator tries to generate more and more realistic virtual scene images to deceive the discriminator; while the discriminator strives to improve its recognition ability to distinguish between real images and generated images. Through this adversarial process, both networks can continuously improve their performance.

[0161] In order to better optimize the network, a composite loss function consisting of adversarial loss and content loss is used. Adversarial loss reflects the ability of the generator to deceive the discriminator, which encourages the generator to produce more realistic images. Content loss measures the similarity between the generated image and the target image in high-level features, which helps to maintain the semantic content of the generated image. By balancing these two losses, virtual scene images that are both realistic and consistent with the expected content can be generated.

[0162] In an optional embodiment,

[0163] Collecting and preprocessing material data related to the virtual scene, the material data including images, videos and three-dimensional models of the actual environment, and inputting the preprocessed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model includes:

[0164] The pre-processed material data is input into a predetermined generative adversarial network architecture, and the generator network and the discriminator network are optimized through an adversarial training process; wherein the generator network accepts random noise as input and generates a virtual scene image through a multi-layer convolutional neural network; the discriminator network evaluates the authenticity of the virtual scene image through a multi-layer convolution operation, and inputs the evaluation result as a feedback signal to the generator network;

[0165] During the adversarial training process, a loss function including adversarial loss and content loss is used to optimize the generator network and the discriminator network. The loss function formula is as follows:

[0166]

[0167] Where L represents the loss function, E[·] represents the expectation, D represents the discriminator, x represents the real sample, G(z) represents the generated sample, M represents the number of multi-scales, and λ i Represents the weight coefficient of content loss at the i-th scale, G(z) i represents the generated image at the i-th scale, T i represents the real image at the i-th scale.

[0168] In this embodiment, by accurately identifying the knowledge blind spots and skill advantages of each trainee, the system can provide highly targeted learning content and effectively improve learning efficiency. Through real-time data collection and analysis, the system can adjust the training content and difficulty according to the trainee's immediate performance to ensure that the learning process is always in the best state. By generating high-quality virtual scenes and personalized guidance information, trainees can learn and practice in a close-to-real environment to improve the practical application ability of skills. The trained model can not only reproduce the scenes in the training data, but also generate new virtual scenes that conform to the real distribution. This makes it possible to create a diverse training environment. The knowledge graph can be continuously updated and expanded, and the generative AI model can also continue to learn and optimize, so that the system can adapt to changing technical requirements and learning needs. By integrating advanced artificial intelligence technology and educational principles, an innovative solution is provided for practical training education, which is expected to significantly improve the training effect and learning efficiency.

[0169] Figure 2 FIG. 1 is a schematic diagram of a training system based on multimodal IoT perception and virtual-real symbiosis according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0170] The first unit is used to collect multimodal perception data of the training environment through multiple heterogeneous Internet of Things devices arranged in the training environment, wherein the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment, perform image segmentation processing on the video data to obtain segmented target image data, process the target image data using a target detection algorithm to obtain the category and position information of the object in the training environment, perform feature extraction and voice recognition processing on the audio data to obtain voice control instructions, and perform data fusion processing on the temperature and humidity data, light intensity data and device status data to obtain comprehensive environmental parameters of the training environment;

[0171] The second unit is used to construct a three-dimensional digital twin model of the training environment according to the category and location information of the objects in the training environment, voice control instructions and comprehensive environmental parameters. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of the object, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. The three-dimensional digital twin model is matched and mapped with the training environment in real time by using augmented reality technology to obtain a virtual-real symbiotic training space.

[0172] The third unit is used to set up training projects and evaluation indicators in the virtual-real symbiotic training space, receive the selection instructions of the training projects input by the trainees, and load the corresponding training projects according to the selection instructions, and send the control instructions of the training projects to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and after the training projects are completed, evaluate the training process and training results of the training projects according to the evaluation indicators, generate a training evaluation report and feed it back to the training management system.

[0173] According to a third aspect of the embodiments of the present invention,

[0174] An electronic device is provided, comprising:

[0175] processor;

[0176] a memory for storing processor-executable instructions;

[0177] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0178] A fourth aspect of the embodiments of the present invention is:

[0179] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0180] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method based on multimodal IoT perception and virtual-real symbiosis, characterized in that: include: By arranging a plurality of heterogeneous IoT devices in the training environment, multimodal perception data of the training environment is collected, wherein the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment, image segmentation processing is performed on the video data to obtain segmented target image data, the target image data is processed using a target detection algorithm to obtain the category and position information of the object in the training environment, feature extraction and voice recognition processing are performed on the audio data to obtain voice control instructions, and data fusion processing is performed on the temperature and humidity data, light intensity data and device status data to obtain comprehensive environmental parameters of the training environment; According to the categories and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters, a three-dimensional digital twin model of the training environment is constructed, wherein the three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. The three-dimensional digital twin model is matched and mapped with the training environment in real time by using augmented reality technology to obtain a virtual-real symbiotic training space; Training projects and evaluation indicators are set up in the virtual-real symbiotic training space, the selection instructions of the training projects input by the trainees are received, and the corresponding training projects are loaded according to the selection instructions, and the control instructions of the training projects are sent to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and after the training projects are completed, the training process and training results of the training projects are evaluated according to the evaluation indicators, and a training evaluation report is generated and fed back to the training management system.

2. The method according to claim 1, characterized in that According to the category and location information of objects in the training environment, voice control instructions and comprehensive environmental parameters, a three-dimensional digital twin model of the training environment is constructed. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of objects, a three-dimensional model of heterogeneous Internet of Things devices, and attribute information of each three-dimensional model, including: Acquire physical space data of the training environment, pre-process the physical space data to obtain a simplified three-dimensional geometric model, classify and label static geometric objects, set initial positions and target moving areas for movable objects, and obtain a three-dimensional geometric model of the training environment; According to the category of objects in the training environment, the objects are matched with the preset 3D model library to obtain the matching 3D models of the objects. According to the category and position information of the objects, the corresponding 3D models of the objects are dynamically loaded into the 3D geometric model of the training environment to set the motion trajectory and interaction mode for the dynamic objects. Acquire real-time data of heterogeneous IoT devices in a training environment, create a three-dimensional model of a heterogeneous IoT device including geometric shapes and functional attributes for each heterogeneous IoT device based on the real-time data, and map the real-time data of the heterogeneous IoT device to the corresponding three-dimensional model of the heterogeneous IoT device; The three-dimensional geometric model of the training environment, the three-dimensional model of the object and the three-dimensional model of the heterogeneous Internet of Things are integrated to obtain a complete three-dimensional digital twin model.

3. The method according to claim 1, characterized in that Using augmented reality technology, the three-dimensional digital twin model is matched and mapped with the training environment in real time, and the training space with virtual and real coexistence is obtained, including: Arrange visual signs in the training environment, capture them in real time through cameras, use image processing algorithms to identify and calculate the position and posture of the visual signs, and obtain the spatial position and posture information of the objects to which the visual signs are attached; For unmarked objects, the corresponding visual feature points are extracted, and feature matching and pose estimation are performed in continuous video frames to track and locate the unmarked objects in real time, and obtain the spatial position and pose information of the unmarked objects; The spatial position and posture information of the object to which the visual mark is attached and the spatial position and posture information of the object without the mark are transmitted to the three-dimensional digital twin model in real time. Through coordinate transformation and posture mapping, the movement and changes of the object to which the visual mark is attached and the object without the mark in the real environment are synchronously updated to the virtual objects of the three-dimensional digital twin model. The motion data of the AR device is obtained, and combined with the visual odometer calculation method, the position and posture changes of the AR device in space are estimated, the motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device, and the rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space.

4. The method according to claim 3, characterized in that The motion data of the AR device is obtained, and the position and posture change of the AR device in space are estimated by combining the visual odometer calculation method. The motion data of the AR device and the visual odometer calculation result are fused to obtain the pose estimation result of the AR device. The rendering position and viewing angle of the virtual object are dynamically adjusted according to the pose estimation result to obtain the final virtual-real symbiotic training space, including: Acquire and preprocess the training environment image data, input the preprocessed training environment image data into the pre-trained image segmentation model for semantic segmentation, and obtain the semantic segmentation results containing different object categories and boundary information. At the same time, use multi-eye vision equipment and stereo matching technology to analyze the parallax information of the left and right cameras to generate a depth map representing the scene depth information; Based on the semantic segmentation results and the obtained depth map, a 3D model containing the object and plane information in the training environment is constructed through 3D reconstruction technology; Acquire the IMU sensor data of the AR device, extract the feature points in the continuous training environment image frames, calculate the relative motion trajectory of the AR device by matching the feature points between frames, obtain the visual mileage calculation result, and optimize the acquired IMU sensor data and the visual mileage calculation result through the data fusion algorithm to obtain the pose estimation result of the AR device; Based on the obtained three-dimensional model and the pose estimation results of the AR device, the optimal placement position and posture of the virtual object in the real environment are determined. According to the optimal placement position and posture of the virtual object, the rendering position and viewing angle of the virtual object are adjusted in real time. The display of the virtual object in the real environment is dynamically updated through the rendering engine to obtain the final virtual-real symbiotic training space.

5. The method according to claim 1, characterized in that Receiving a selection instruction of a training project input by a trainee, loading a corresponding training project according to the selection instruction, and sending a control instruction of the training project to a virtual device model in a three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in a training environment for training, including: Design a project database containing multiple practical training projects, each of which includes task objectives, operation steps and evaluation criteria; Receive a selection instruction of a training project input by a trainee, load a basic training project that meets the trainee's needs from a project database according to the selection instruction, send a control instruction of the loaded basic training project to a virtual device model in the three-dimensional digital twin model and an Internet of Things device mapped to the virtual device model in the training environment, simulate the operation and operating status of the device through the three-dimensional digital twin model, and generate training environment data; The operational behavior data, body posture, and cognitive load information of the trainees are collected, combined with the training environment data, to generate comprehensive monitoring data of the training process, model and analyze the comprehensive monitoring data, and generate a personalized user portrait of the trainee. If it is detected according to the personalized user portrait that the trainee needs personalized training, a customized training project including task objectives, detailed operating steps, guidance information, and a virtual scene matching the task is automatically generated through a generative AI model, and the real-time operational performance of the trainees is continuously tracked. The content difficulty of the personalized training project and the detailed level of the detailed operating steps are dynamically adjusted, and training is carried out according to the adjusted personalized training project.

6. The method according to claim 5, characterized in that Customized training projects that automatically generate task objectives, detailed operation steps, guidance information, and virtual scenes that match the tasks through generative AI models include: Constructing a knowledge graph for the practical training field, wherein the knowledge graph for the practical training field includes task objectives, operation steps, learning resources and technical standards; Use a generative AI model to reason about the structured data in the knowledge graph of the training field, identify the knowledge blind spots and skill advantages of trainees, generate initial personalized training projects by comparing the trainees' historical performance with the technical standards in the knowledge graph, and combine the trainees' current skill level and training needs, and set overall task goals and learning directions; During the implementation of the personalized training project, sensors and biofeedback equipment are used to collect the trainees' operation data, posture and physiological state in real time, and the collected data are analyzed using data analysis technology and machine learning algorithms. According to the analysis results, the training content and difficulty of the customized training project are automatically adjusted; Generate personalized operation steps and guidance information using a generative AI model based on the trainee's skill level and performance in a specific task. The guidance information includes text instructions, image recognition results, and interactive teaching tools. At the same time, define the design requirements of the virtual scene based on the set overall task objectives, collect and pre-process material data related to the virtual scene, and input the pre-processed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model. The trained virtual scene generation model is used to generate a virtual scene, and the generated virtual scene is post-processed. The processed virtual scene, overall task objectives, personalized operation steps and guidance information are integrated to generate a complete customized training project.

7. The method according to claim 6, characterized in that Collecting and preprocessing material data related to the virtual scene, the material data including images, videos and three-dimensional models of the actual environment, and inputting the preprocessed material data into a predetermined generative adversarial network architecture to train a virtual scene generation model includes: The pre-processed material data is input into a predetermined generative adversarial network architecture, and the generator network and the discriminator network are optimized through an adversarial training process; wherein the generator network accepts random noise as input and generates a virtual scene image through a multi-layer convolutional neural network; the discriminator network evaluates the authenticity of the virtual scene image through a multi-layer convolution operation, and inputs the evaluation result as a feedback signal to the generator network; During the adversarial training process, a loss function including adversarial loss and content loss is used to optimize the generator network and the discriminator network. The loss function formula is as follows: Where L represents the loss function, E[·] represents the expectation, D represents the discriminator, x represents the real sample, G(z) represents the generated sample, M represents the number of multi-scales, and λ i Represents the weight coefficient of content loss at the i-th scale, G(z) i represents the generated image at the i-th scale, T i represents the real image at the i-th scale.

8. A training system based on multimodal IoT perception and virtual-real symbiosis, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect multimodal perception data of the training environment through multiple heterogeneous Internet of Things devices arranged in the training environment, wherein the multimodal perception data includes video data, audio data, temperature and humidity data, light intensity data and device status data of the training environment, perform image segmentation processing on the video data to obtain segmented target image data, process the target image data using a target detection algorithm to obtain the category and position information of the object in the training environment, perform feature extraction and voice recognition processing on the audio data to obtain voice control instructions, and perform data fusion processing on the temperature and humidity data, light intensity data and device status data to obtain comprehensive environmental parameters of the training environment; The second unit is used to construct a three-dimensional digital twin model of the training environment according to the category and location information of the objects in the training environment, voice control instructions and comprehensive environmental parameters. The three-dimensional digital twin model includes a three-dimensional geometric model of the training environment, a three-dimensional model of the object, a three-dimensional model of heterogeneous Internet of Things devices and attribute information of each three-dimensional model. The three-dimensional digital twin model is matched and mapped with the training environment in real time by using augmented reality technology to obtain a virtual-real symbiotic training space. The third unit is used to set up training projects and evaluation indicators in the virtual-real symbiotic training space, receive the selection instructions of the training projects input by the trainees, and load the corresponding training projects according to the selection instructions, and send the control instructions of the training projects to the virtual device models in the three-dimensional digital twin model and the Internet of Things devices mapped to the virtual device models in the training environment for training, and after the training projects are completed, evaluate the training process and training results of the training projects according to the evaluation indicators, generate a training evaluation report and feed it back to the training management system.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images