Multi-user collaborative power equipment maintenance simulation training system

Abstract

The invention belongs to the field of simulation training, and particularly relates to a multi-user collaborative power equipment maintenance simulation training system which comprises a virtual reality component, a posture recognition module, a virtual environment modeling module, a processing and feedback module and a multi-user collaborative module. A posture recognition module of the system adopts an OpenPose library to extract skeleton key points according to a user video and hand posture data. And the virtual environment modeling module is used for constructing a simulation training space and a user substitute by adopting a Unity3D plug-in and an OpenPoseUnity plug-in. And the processing and feedback module adopts a feature fusion Hausdorff distance to score the operation of the trainee and uses a maintenance operation identification rule base to automatically generate personalized guidance suggestions and overall guidance suggestions.

Description

Technical Field

[0001] The invention belongs to the field of simulation training, and specifically refers to a multi-user collaborative power equipment maintenance simulation training system. Background Art

[0002] As the infrastructure of modern society, power equipment maintenance is particularly important. With the development of computer technology, simulation technology has been widely used in the field of power equipment maintenance training. Multi-user collaborative simulation training can simulate the actual working environment and improve trainees' teamwork and problem-solving abilities. However, existing simulation training systems generally have deficiencies in multi-user collaborative mechanisms, limited training scenarios, and are unable to meet diverse training needs. They also lack sufficiently accurate and stable operation identification and scoring systems. Summary of the invention

[0003] The present invention overcomes the shortcomings of the prior art and provides a multi-user collaborative power equipment maintenance simulation training system. The feature-fused Hausdorff distance effectively improves the accuracy of operation recognition and also makes training scoring more reliable. The parameters of the simulation training space can be input or randomly initialized, which provides new possibilities for the construction of training scenarios. The personalized guidance opinions automatically generated by the maintenance operation recognition rule library assist trainees in continuously improving their operating skills. The overall guidance opinions can also help instructors adjust their teaching modes, which solves the problems of the existing simulation training system's lack of multi-user collaborative mechanism, limited training scenarios, and lack of sufficiently accurate and stable operation recognition and scoring systems.

[0004] The multi-user collaborative power equipment maintenance simulation training system includes a virtual reality component, a posture recognition module, a virtual environment modeling module, a processing and feedback module and a multi-user collaborative module;

[0005] The virtual reality component is electrically connected to the gesture recognition module and the virtual environment modeling module, the gesture recognition module is electrically connected to the processing and feedback module and the virtual environment modeling module, the processing and feedback module is electrically connected to the virtual environment modeling module and the multi-user collaboration module, and the multi-user collaboration module is electrically connected to the virtual environment modeling module;

[0006] The virtual environment modeling module uses Unity3D to build a simulation training space, and inputs the scenes and interaction data generated in the simulation training space into the virtual reality component;

[0007] The virtual reality component captures the user's hand gesture data and the user interaction data, the virtual reality component receives the scene and interaction data from the virtual environment modeling module and transmits the hand gesture data and the user interaction data to the gesture recognition module;

[0008] The posture recognition module receives and processes the hand posture data and user interaction data in real time using the OpenPose library, collects user videos in real time, samples the user videos and extracts skeleton key points, generates skeleton key point data, and refines the hand skeleton key points in combination with the hand posture data, and transmits the skeleton key point data to the processing and feedback module and the virtual environment modeling module;

[0009] The virtual environment modeling module uses the OpenPoseUnity plug-in to build a user virtual avatar in the simulation training space according to the skeleton key point data;

[0010] The multi-user collaboration module interacts with the virtual environment modeling module in real time, and communicates with each other between the power equipment maintenance simulation training systems of different users. The multi-user collaboration module uses the same timestamp to synchronize the simulation training spaces between different users.

[0011] The processing and feedback module constructs a maintenance operation recognition rule base according to the characteristics of the skeleton key points, scores the user's operation according to the Hausdorff distance, and automatically generates personalized guidance opinions and overall guidance opinions in combination with the maintenance operation recognition rule base.

[0012] The virtual environment modeling module uses Unity3D to build a simulation training space and the multi-user collaboration module uses the same timestamp to synchronize the simulation training spaces between different users, including the following steps:

[0013] Step E1: The instructor manually inputs or randomly initializes the parameters of the simulation training space in Unity3D, including lighting conditions, power equipment type, fault type, fault location, and deadline, and the virtual environment modeling module constructs the simulation training space;

[0014] Step E2: The virtual environment modeling module transmits the scene and interaction data of the simulation training space to the multi-user collaboration module, the multi-user collaboration module transmits the scene and interaction data of the simulation training space to the multi-user collaboration module of the trainee, the multi-user collaboration module of the trainee transmits the scene and interaction data of the simulation training space to the virtual environment modeling module of the trainee, and the bilateral multi-user collaboration module then ensures that the simulation training space of the instructor and the trainee is consistent.

[0015] The process in which the processing and feedback module identifies and scores the user's operation according to the Hausdorff distance and automatically generates personalized guidance and overall guidance in combination with the maintenance operation identification rule base includes the following steps:

[0016] Step T1: The instructor demonstrates the operation in the simulation training space, and then the trainees operate according to the demonstration. The virtual environment modeling module records the key skeletal data of the instructor and the trainees and transmits it to the processing and feedback module;

[0017] Step T2: The processing and feedback module records the skeleton key point data of the instructor and the trainee in the form of a matrix to obtain a skeleton key point matrix. Each skeleton key point corresponds to a 3×T skeleton key point matrix, where 3 rows represent the 3D coordinates of the skeleton key point and T represents the number of sampling times.

[0018] Step T3: The processing and feedback module performs feature fusion on the coordinates, velocity, joint angle and joint angular velocity of the skeleton key points to construct an 8×T skeleton key point matrix. After feature fusion, each column of the skeleton key point matrix represents all the features of the skeleton key point at the sampling time point, which is called the feature fusion point.

[0019] Step T4: The processing and feedback module uses the Hausdorff distance to calculate the similarity between the skeleton key point matrices. The Hausdorff distance formula is as follows:

[0020] ;

[0021] in, is the Hausdorff distance. The skeleton key point matrices of the teacher and the student are regarded as two sets of feature fusion points, namely set A and set B. is the maximum distance from each feature fusion point in set A to the nearest feature fusion point in set B, is the maximum distance from each feature fusion point in set B to the nearest feature fusion point in set A, and the formulas are:

[0022] ;

[0023] ;

[0024] in It represents the Euclidean distance between feature fusion point a and feature fusion point b, where feature fusion point a and feature fusion point b belong to set A and set B respectively. Due to feature fusion, the feature fusion points here exist in eight-dimensional space. represents the supremum, which specifically refers to the maximum value of the distance from this feature fusion point to all feature fusion points in another set. represents the infimum, which specifically refers to the minimum value of the distance from this feature fusion point to all feature fusion points in another set;

[0025] Step T5: The processing and feedback module performs weighted average of the Hausdorff distances corresponding to each skeleton key point matrix. The closer the final value is to 0, the more similar the two sets are, and the higher the score the trainee gets. The weight corresponding to each skeleton key point is determined by the maintenance operation recognition rule base according to the parameters of the simulation training space.

[0026] Step T6: The maintenance operation recognition rule base generates personalized guidance based on the scores of past students and the characteristics of the skeleton key points, and generates overall guidance based on the personalized guidance of each student.

[0027] Furthermore, in step T3, the coordinates, speeds and joint angles of the skeleton key points are feature fused, including the following steps:

[0028] Step T31: The OpenPose library pre-labels the attributes of each skeletal key point. For skeletal key points pre-labeled as joints, the joint angles are calculated based on the coordinates of the adjacent skeletal key points and filled into the 4th row of the skeletal key point matrix. For skeletal key points pre-labeled as non-joints, the 4th row of the skeletal key point matrix is ​​filled with 0.

[0029] Step T32: Take the inverse of the sampling frequency of the gesture recognition module to obtain the sampling period. For the skeletal key points pre-marked as joints, calculate the joint angular velocity according to the joint angles of the adjacent columns of the skeletal key point matrix and the sampling period, and fill the joint angular velocity into the 5th row of the skeletal key point matrix. For the skeletal key points pre-marked as non-joints, fill the 5th row of the skeletal key point matrix with 0.

[0030] Step T33: Calculate the speed of the skeleton key point in the three dimensions of the Unity3D coordinate axis according to the first three rows of the adjacent column and the sampling period and fill in the 6th, 7th and 8th rows of the skeleton key point matrix;

[0031] Step T34: Use a moving average method with a sliding window size of 5 to smooth the joint angular velocity in the 5th row of the skeleton key point matrix and the velocities in the 6th, 7th, and 8th rows of the skeleton key point matrix;

[0032] Step T35: Normalize the data in each row of the skeleton key point matrix.

[0033] The construction process of the simulation training space includes the following steps:

[0034] Step C1: Based on the parameters of the simulation training space, use the built-in tools of Unity3D to define the three-dimensional models of the scene and the objects in the scene and their working conditions;

[0035] Step C2: Use C# to write the interactive logic code in the scene;

[0036] Step C3: Define the visual effects in the scene using Unity3D built-in tools;

[0037] Step C4: Use Unity3D built-in tools to import audio data in the scene;

[0038] Step C5: Get the simulation training space.

[0039] The process of constructing a maintenance operation recognition rule base according to the characteristics of the skeleton key points by the processing and feedback module comprises the following steps:

[0040] Step D1: Extract features from the skeleton key point matrix using principal component analysis;

[0041] Step D2: using the extracted features to define rules based on expert knowledge, including rules for determining the weights of skeleton key points according to the parameters of the simulation training space and output rules for personalized guidance and overall guidance;

[0042] Step D3: Organize the defined rules into a maintenance operation identification rule base.

[0043] Furthermore, in step D1, principal component analysis is used to extract features in the skeleton key point matrix, including the following steps:

[0044] Step D11: Decentralize the skeleton key point matrix, that is, subtract the corresponding row mean from each row;

[0045] Step D12: Calculate the covariance matrix of the skeleton key point matrix. The formula is as follows:

[0046] ;

[0047] in is the covariance matrix, X is the skeleton key point matrix after decentralization, is the transposed X;

[0048] Step D13: Solving The eigenvalues ​​and eigenvectors of

[0049] Step D14: Select K eigenvectors with the largest eigenvalues ​​to form a projection matrix;

[0050] Step D15: multiplying the projection matrix by the skeleton key point matrix after decentralization to obtain the skeleton key point matrix after dimension reduction;

[0051] Step D16: Use prior knowledge to analyze the reduced-dimensional skeleton key point matrix to extract its features.

[0052] The process of constructing a user virtual avatar in a simulation training space comprises the following steps:

[0053] Step A1: Select a 3D virtual character model from the Unity3D built-in library;

[0054] Step A2: Mapping the skeleton key points extracted by the posture recognition module to the corresponding skeleton key points of the 3D virtual character model in real time;

[0055] Step A3: Use the built-in tools in Unity3D to fine-tune any inconsistencies in the mapping process.

[0056] The beneficial effects of the present invention are as follows:

[0057] (1) The feature fusion data processing method replaces the traditional skeleton key point description method that only includes coordinates with a skeleton key point matrix including coordinates, joint angles, joint angular velocities, and velocity vectors. This more rigorously and comprehensively describes the operational characteristics of power equipment maintenance, and also facilitates the construction of a maintenance operation identification rule base, reducing the system development cost and time. Carefully designed features are usually more stable and less susceptible to fluctuations, making the system more stable and reliable.

[0058] (2) Normalizing the skeleton key point matrix eliminates the influence of the distance between the user and the camera, making the skeleton key point matrix more focused on motion features. The moving average rule makes the data smoother and easier to analyze, making the system as a whole more tolerant to anomalies and noise, improving the robustness of the system, and making the system operate more smoothly.

[0059] (3) The traditional Hausdorff distance that only considers coordinate trajectories can only compare two isolated trajectory curves. It is completely unable to consider structural features such as joint angles, and is also insensitive to dynamic features such as speed and angular velocity. The feature-fused Hausdorff distance introduces these features into the scoring process, which not only overcomes the technical bias of the conventional Hausdorff distance that is limited to position features, but also makes the scoring more comprehensive and precise.

[0060] (4) The use of Hausdorff distance instead of the common convolutional neural network greatly reduces the computing power burden of the hardware and increases the fluency of the system.

[0061] (5) The parameters of the simulation training space can be input or randomly initialized, which provides new possibilities for the construction of training scenarios and meets diverse training needs.

[0062] (6) The personalized guidance automatically generated by the maintenance operation recognition rule library helps students continuously improve their operation skills. The overall guidance can also help instructors adjust their teaching methods.

[0063] (7) The multi-user collaboration module then uses the same timestamp to ensure that the simulation training space of instructors and trainees is consistent, solving the shortcomings of the multi-user collaboration mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a module diagram of the multi-user collaborative power equipment maintenance simulation training system proposed by the present invention;

[0065] Figure 2 Flowchart of the process of constructing the training space for the simulation.

[0066] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the examples given are only used to explain the present invention, and are not used to limit the scope of the present invention.

[0068] Example 1, see Figure 1 ,The present invention provides a multi-user collaborative power equipment maintenance simulation training system, including a virtual reality component, a gesture recognition module, a virtual environment modeling module, a processing and feedback module and a multi-user collaborative module;

[0069] The virtual reality component is electrically connected to the gesture recognition module and the virtual environment modeling module, the gesture recognition module is electrically connected to the processing and feedback module and the virtual environment modeling module, the processing and feedback module is electrically connected to the virtual environment modeling module and the multi-user collaboration module, and the multi-user collaboration module is electrically connected to the virtual environment modeling module;

[0070] The virtual environment modeling module uses Unity3D to build a simulation training space, and inputs the scenes and interaction data generated in the simulation training space into the virtual reality component;

[0071] The virtual reality component includes a virtual reality head mounted display and a virtual reality interactive glove, the virtual reality component captures the user's hand gesture data and the user interaction data, the virtual reality component receives the scene and interaction data from the virtual environment modeling module and transmits the hand gesture data and the user interaction data to the gesture recognition module;

[0072] The posture recognition module receives and processes the hand posture data and user interaction data in real time using the OpenPose library, collects user videos in real time, samples and extracts skeleton key points at a frequency of 30 frames per second, generates skeleton key point data, and refines the hand skeleton key points in combination with the hand posture data, and transmits the skeleton key point data to the processing and feedback module and the virtual environment modeling module;

[0073] The multi-user collaboration module interacts with the virtual environment modeling module in real time, communicates with each other between the power equipment maintenance simulation training systems of different users, and uses the same timestamp to ensure the consistency and synchronization of the simulation training space between different users;

[0074] The virtual environment modeling module uses the OpenPoseUnity plug-in to build a user virtual avatar in the simulation training space according to the skeleton key point data;

[0075] The multi-user collaboration module interacts with the virtual environment modeling module in real time, and communicates with each other between the power equipment maintenance simulation training systems of different users. The multi-user collaboration module uses the same timestamp to synchronize the simulation training spaces between different users.

[0076] The processing and feedback module constructs a maintenance operation recognition rule base according to the characteristics of the skeleton key points, scores the user's operation according to the Hausdorff distance, and automatically generates personalized guidance opinions and overall guidance opinions in combination with the maintenance operation recognition rule base.

[0077] Example 2, see Figure 2 This embodiment is based on the above embodiment, and the construction process of the simulation training space includes the following steps:

[0078] Step C1: Based on the parameters of the simulation training space, use the built-in tools of Unity3D to define the three-dimensional models of the scene and the objects in the scene and their working conditions;

[0079] Step C2: Use C# to write the interactive logic code in the scene;

[0080] Step C3: Define the visual effects in the scene using Unity3D built-in tools;

[0081] Step C4: Use Unity3D built-in tools to import audio data in the scene;

[0082] Step C5: Get the simulation training space.

[0083] Embodiment 3, based on the above embodiment, the virtual environment modeling module uses Unity3D to build a simulation training space and the multi-user collaboration module uses the same timestamp to synchronize the simulation training spaces between different users, including the following steps:

[0084] Step E1: The instructor manually inputs the parameters of the simulation training space in Unity3D, including lighting conditions, power equipment type, fault type, fault location and deadline, and the virtual environment modeling module constructs the simulation training space. The parameters in this embodiment are daytime, transformer, short circuit, winding interior and 1 hour;

[0085] Step E2: The virtual environment modeling module transmits the scene and interaction data of the simulation training space to the multi-user collaboration module, the multi-user collaboration module transmits the scene and interaction data of the simulation training space to the multi-user collaboration module of the trainee, the multi-user collaboration module of the trainee transmits the scene and interaction data of the simulation training space to the virtual environment modeling module of the trainee, and the bilateral multi-user collaboration module then uses the same timestamp to ensure that the simulation training space of the instructor and the trainee is consistent.

[0086] Embodiment 4: This embodiment is based on the above embodiment. The processing and feedback module identifies and scores the user's operation according to the Hausdorff distance, and automatically generates personalized guidance opinions and overall guidance opinions in combination with the maintenance operation identification rule base. The process includes the following steps:

[0087] Step T1: The instructor demonstrates the operation in the simulation training space, and then the trainees operate according to the demonstration. The virtual environment modeling module records the motion trajectories of the instructor's and trainees' skeleton key points and transmits them to the processing and feedback module;

[0088] Step T2: The processing and feedback module records the skeleton key point data of the instructor and the trainee in the form of a matrix to obtain a skeleton key point matrix. Each skeleton key point corresponds to a 3×T skeleton key point matrix, 3 rows represent the 3D coordinates of the skeleton key point, T represents the number of sampling times. In this embodiment, the operation lasts for two minutes, the sampling frequency of the posture recognition module is 30 frames per second, and the value of T is 3600;

[0089] Step T3: The processing and feedback module performs feature fusion on the coordinates, velocity, joint angle and joint angular velocity of the skeleton key points to construct an 8×3600 skeleton key point matrix. After feature fusion, each column of the skeleton key point matrix represents all the features of the skeleton key point at the sampling time point, which is called the feature fusion point.

[0090] Step T4: The processing and feedback module uses the Hausdorff distance to calculate the similarity between the skeleton key point matrices. The Hausdorff distance formula is as follows:

[0091] ;

[0092] in, is the Hausdorff distance. The skeleton key point matrices of the teacher and the student are regarded as two sets of feature fusion points, namely set A and set B. is the maximum distance from each feature fusion point in set A to the nearest feature fusion point in set B, is the maximum distance from each feature fusion point in set B to the nearest feature fusion point in set A, and the formulas are:

[0093] ;

[0094] ;

[0095] in It represents the Euclidean distance between feature fusion point a and feature fusion point b, where feature fusion point a and feature fusion point b belong to set A and set B respectively. Due to feature fusion, the feature fusion points here exist in eight-dimensional space. represents the supremum, which specifically refers to the maximum value of the distance from this feature fusion point to all feature fusion points in another set. represents the infimum, which specifically refers to the minimum value of the distance from this feature fusion point to all feature fusion points in another set;

[0096] Step T5: The processing and feedback module performs weighted averaging on the Hausdorff distances corresponding to each skeleton key point matrix according to the weights determined by the maintenance operation recognition rule base, and the obtained value is 0.125, and then the obtained value is mapped to a score. The specific mapping formula in this embodiment is as follows:

[0097] ;

[0098] in is the weighted average Hausdorff distance, The score of the students is between 0 and 10;

[0099] After the mapping was completed, the final score was 9.9 points;

[0100] Step T6: The maintenance operation recognition rule base generates personalized guidance opinions based on the scores of past students and the characteristics of the skeleton key point matrix, and generates overall guidance opinions based on the personalized guidance opinions of each student;

[0101] Through the above operations, in order to address the problem of lack of sufficiently accurate and stable operation recognition and scoring systems, this scheme introduces speed, joint angle and joint angular velocity into the scoring process to achieve more comprehensive and detailed operation recognition and scoring.

[0102] Embodiment 5, based on embodiment 4, in step T3, the coordinates, speed and joint angles of the skeleton key points are feature fused, including the following steps:

[0103] Step T31: The OpenPose library pre-labels the attributes of each skeletal key point. For skeletal key points pre-labeled as joints, the joint angles are calculated based on the coordinates of adjacent skeletal key points and filled into the 4th row of the skeletal key point matrix. For skeletal key points pre-labeled as non-joints, the 4th row of the skeletal key point matrix is ​​filled with 0. The calculation formula for the joint angle in this embodiment is:

[0104] ;

[0105] in, is the joint angle in radians, where is the coordinate of the key point of the bone where the joint is located, and are the coordinates of adjacent bone key points;

[0106] Step T32: Take the inverse of the sampling frequency of the gesture recognition module. For the skeletal key points pre-marked as joints, calculate the joint angular velocity according to the joint angles and sampling periods of the adjacent columns of the skeletal key point matrix, and fill the joint angular velocity into the 5th row of the skeletal key point matrix. For the skeletal key points pre-marked as non-joints, the 5th row of the skeletal key point matrix is ​​filled with 0. The calculation formula of the joint angular velocity in this embodiment is as follows:

[0107] ;

[0108] in, is the joint angular velocity, is the joint angle in radians from the previous column, is the joint angle of the second column expressed in radians, is the sampling period of the gesture recognition module. 1 / 30 second;

[0109] Step T33: According to the first three rows of the adjacent columns and the sampling period, the speed of the skeleton key point in the three dimensions of the Unity3D coordinate axis is calculated and filled into the 6th, 7th and 8th rows of the skeleton key point matrix. In this embodiment, the speed formula in a single dimension is as follows:

[0110] ;

[0111] in, is the velocity of the skeleton keypoint in a single dimension, is the coordinate value corresponding to the dimension in the previous column, is the coordinate value corresponding to the dimension in the next column. is the sampling period of the gesture recognition module, and in this embodiment, t is 1 / 30 second;

[0112] Step T34: Use a moving average method with a sliding window size of 5 to smooth the joint angular velocity in the 5th row and the velocities in the 6th, 7th, and 8th rows of the skeleton key point matrix;

[0113] Step T35: Perform 0-1 normalization processing on the data of each row in the skeleton key point matrix. The specific formula is as follows:

[0114] ;

[0115] in is the original value, is the minimum value in the matrix, is the maximum value in the matrix, It is the value after normalization from 0 to 1;

[0116] Through the above operations, in order to solve the problem of interference caused by noise and the distance between the user and the camera, this solution uses the moving average method and normalization to improve the robustness of the system and make the system operate more smoothly.

[0117] Embodiment 6, based on the above embodiment, the process of the processing and feedback module constructing a maintenance operation identification rule base according to the characteristics of the skeleton key points comprises the following steps:

[0118] Step D1: Extract features from the skeleton key point matrix using principal component analysis;

[0119] Step D2: using the extracted features to define rules based on expert knowledge, including rules for determining the weights of skeleton key points according to the parameters of the simulation training space and output rules for personalized guidance and overall guidance;

[0120] Step D3: Organizing the defined rules into a maintenance operation identification rule base;

[0121] Through the above operations, in order to address the problem of lack of sufficiently accurate and stable operation recognition and scoring systems, this scheme introduces speed, joint angle and joint angular velocity into the scoring process to achieve more comprehensive and detailed operation recognition and scoring.

[0122] Embodiment 7, based on embodiment 6, in step D1, principal component analysis is used to extract features in the skeleton key point matrix, comprising the following steps:

[0123] Step D11: Decentralize the skeleton key point matrix, that is, subtract the corresponding row mean from each row;

[0124] Step D12: Calculate the covariance matrix of the skeleton key point matrix. The formula is as follows:

[0125] ;

[0126] in is the covariance matrix, X is the skeleton key point matrix after decentralization, is the transposed X, in this embodiment, T is 3600;

[0127] Step D13: Solve using the Numpy library The eigenvalues ​​and eigenvectors of

[0128] Step D14: Select the three eigenvectors with the largest eigenvalues ​​to form a projection matrix;

[0129] Step D15: multiplying the projection matrix by the skeleton key point matrix after decentralization to obtain the skeleton key point matrix after dimension reduction;

[0130] Step D16: Use prior knowledge to analyze the reduced-dimensional skeleton key point matrix to extract its features.

[0131] Embodiment 8: This embodiment is based on the above embodiment, and the process of constructing a user virtual avatar in a simulation training space includes the following steps:

[0132] Step A1: Select a 3D virtual character model from the Unity3D built-in library;

[0133] Step A2: Mapping the skeleton key points extracted by the posture recognition module to the corresponding skeleton key points of the 3D virtual character model in real time;

[0134] Step A3: Use the built-in tools in Unity3D to fine-tune any inconsistencies in the mapping process.

[0135] Embodiment 9. This embodiment is based on the above embodiment and runs in the Windows operating system environment. The simulation training space is constructed by Unity3D, and the maintenance operation recognition rule library is constructed by Python. The posture recognition module, the virtual environment modeling module, and the processing and feedback module are implemented by a Lenovo Savior computer and the camera connected to it. The multi-user collaboration module is implemented by a VR Sync box. The virtual reality components include an Oculus headset and a NoitomHi52.0 motion capture gloves.

[0136] The present invention and its implementation are described above, and such description is not restrictive. The drawings show only one implementation of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by the present invention, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should fall within the protection scope of the present invention.

Claims

1. Multi-user collaborative power equipment maintenance simulation training system, characterized by: The multi-user collaborative power equipment maintenance simulation training system comprises: a virtual reality component, a posture recognition module, a virtual environment modeling module, a processing and feedback module and a multi-user collaborative module; The virtual reality component is electrically connected to the gesture recognition module and the virtual environment modeling module, the gesture recognition module is electrically connected to the processing and feedback module and the virtual environment modeling module, the processing and feedback module is electrically connected to the virtual environment modeling module and the multi-user collaboration module, and the multi-user collaboration module is electrically connected to the virtual environment modeling module; The virtual environment modeling module constructs a simulation training space and inputs the scenes and interaction data generated in the simulation training space into the virtual reality component; The virtual reality component captures the user's hand gesture data and the user interaction data, the virtual reality component receives the scene and interaction data from the virtual environment modeling module and transmits the hand gesture data and the user interaction data to the gesture recognition module; The gesture recognition module receives and processes hand gesture data and user interaction data in real time, collects user videos in real time, samples the user videos and extracts skeleton key points, generates skeleton key point data, and refines the hand skeleton key points in combination with the hand gesture data, and transmits the skeleton key point data to the processing and feedback module and the virtual environment modeling module; The processing and feedback module constructs a maintenance operation recognition rule base according to the characteristics of the skeleton key points, scores the user's operation according to the Hausdorff distance, and automatically generates personalized guidance opinions and overall guidance opinions in combination with the maintenance operation recognition rule base.

2. The multi-user collaborative power equipment maintenance simulation training system according to claim 1 is characterized in that: The process in which the processing and feedback module identifies and scores the user's operation according to the Hausdorff distance and automatically generates personalized guidance and overall guidance in combination with the maintenance operation identification rule base includes the following steps: Step T1: The instructor demonstrates the operation in the simulation training space, and then the trainees operate according to the demonstration. The virtual environment modeling module records the key skeletal data of the instructor and the trainees and transmits it to the processing and feedback module; Step T2: The processing and feedback module records the skeleton key point data of the instructor and the trainee in the form of a matrix to obtain a skeleton key point matrix, where each skeleton key point corresponds to a 3×T skeleton key point matrix; Step T3: The processing and feedback module performs feature fusion on the coordinates, velocity, joint angle and joint angular velocity of the skeleton key points to construct an 8×T skeleton key point matrix. After feature fusion, each column of the skeleton key point matrix represents all the features of the skeleton key point at the sampling time point, which is called the feature fusion point. Step T4: The processing and feedback module uses Hausdorff distance to calculate the similarity between the skeleton key point matrices; Step T5: The processing and feedback module performs weighted average of the Hausdorff distances corresponding to each skeleton key point matrix, and the weight corresponding to each skeleton key point is determined by the maintenance operation recognition rule base according to the parameters of the simulation training space; Step T6: The maintenance operation recognition rule base generates personalized guidance based on the scores of past students and the characteristics of the skeleton key points, and generates overall guidance based on the personalized guidance of each student.

3. The multi-user collaborative power equipment maintenance simulation training system according to claim 2 is characterized in that: The virtual environment modeling module uses the OpenPoseUnity plug-in to build a user virtual avatar in the simulation training space according to the skeleton key point data; The multi-user collaboration module interacts with the virtual environment modeling module in real time, and communicates with each other between the power equipment maintenance simulation training systems of different users. The multi-user collaboration module uses the same timestamp to synchronize the simulation training spaces between different users.

4. The multi-user collaborative power equipment maintenance simulation training system according to claim 3 is characterized in that: In step T2, each skeleton key point corresponds to a 3×T skeleton key point matrix, where 3 rows represent the 3D coordinates of the skeleton key point and T represents the number of sampling times.

5. The multi-user collaborative power equipment maintenance simulation training system according to claim 4 is characterized in that: In step T3, the coordinates, velocity and joint angles of the skeleton key points are fused, including the following steps: Step T31: For the skeleton key points pre-marked as joints, the joint angles are calculated according to the coordinates of the adjacent skeleton key points and filled into the 4th row of the skeleton key point matrix; for the skeleton key points pre-marked as non-joints, the 4th row of the skeleton key point matrix is ​​filled with 0; Step T32: Take the inverse of the sampling frequency of the gesture recognition module to obtain the sampling period. For the skeletal key points pre-marked as joints, calculate the joint angular velocity according to the joint angles of the adjacent columns of the skeletal key point matrix and the sampling period, and fill the joint angular velocity into the 5th row of the skeletal key point matrix. For the skeletal key points pre-marked as non-joints, fill the 5th row of the skeletal key point matrix with 0. Step T33: Calculate the speed of the skeleton key point in the three dimensions of the coordinate axis according to the first three rows of the adjacent column and the sampling period and fill in the 6th, 7th and 8th rows of the skeleton key point matrix; Step T34: Use a moving average method with a sliding window size of 5 to smooth the joint angular velocity in the 5th row of the skeleton key point matrix and the velocities in the 6th, 7th, and 8th rows of the skeleton key point matrix; Step T35: Normalize the data in each row of the skeleton key point matrix.

Images