Virtual reality tower climbing training method and system based on deep learning

Abstract

The invention discloses a virtual reality tower climbing training method and system based on deep learning, and the method comprises the steps: arranging nine-axis sensors at four limbs of a student, collecting motion data, employing a deep learning model to recognize the motion types, including the motions of climbing up a tower, climbing down the tower, tying a long rope, tying a short rope, and the like, and combining the types of towers to judge whether the motions are standard or not. According to the practice system, an unattended training scene is realized, trainees can complete the training process by themselves under the guidance of the system, and the system has the characteristics of automation, high efficiency and intelligence. Based on the practical link, the tower climbing training integrates the two aspects of virtual action learning and actual operation drilling, the training effect is enhanced, and the practicability and effectiveness of the system are improved.

Description

Technical Field

[0001] The invention belongs to the field of electric power virtual reality, and in particular relates to a virtual reality pole tower climbing training system. Background Art

[0002] The power training simulation system that combines virtual reality technology, 3D modeling technology, and computer simulation technology has been widely used in the field of power training due to its advantages of high interactivity, high immersion, low cost, safety and efficiency. Building a close-to-reality working environment for trainees based on sensors, vision, and behavioral interaction is an effective way to implement a virtual reality power training system.

[0003] In recent years, power training based on virtual reality has become the mainstream construction direction. Through VR supporting equipment, an interactive three-dimensional power scene virtual environment is generated, including outdoor primary equipment, outdoor scenes, secondary equipment rooms and other full-scene immersive simulations. Trainees see the same feeling of the virtual immersive scene as the real world. By wearing positioning technology equipment, trainees operate the equipment in the virtual environment through the handle, and the operation method of the equipment is the same as the real scene, which provides great convenience for equipment management, safe production, and emergency drills. Trainees can not only receive training individually, but also complete training with multiple people. Virtual training has several advantages: first, the VR virtual simulation training system for power special operations can improve the safety of training; second, the VR virtual simulation training system for power special operations can improve the efficiency of training; third, the VR virtual simulation training system for power special operations can improve the repeatability of training, and finally, the VR virtual simulation training system for power special operations can improve the interactivity and fun of training.

[0004] However, there are still some items that are difficult to cover in current virtual training. Climbing a pole tower is one of the basic operations that power operation and maintenance personnel need to be trained. However, since climbing movements are relatively complex and it is difficult to determine whether they are standardized, it is not included in the current training system. This means that climbing training can only be carried out through on-site teaching, which is inefficient and still in the traditional training mode.

[0005] The process of power maintenance personnel climbing towers is divided into normal tower climbing behavior and illegal tower climbing behavior. One of the most basic regulations in the tower climbing process is to tie a safety rope. When climbing up and down the tower, maintenance personnel generally need to stop several times to adjust the safety rope. If they do not wear a safety rope, they will climb up and down without stopping. In addition, climbing the tower from the correct position is also a routine operating specification for maintenance personnel. The tower is generally equipped with footrests specially for climbing the tower. Maintenance personnel need to climb from the designated footrests, but sometimes maintenance personnel will climb the tower from other places in order to save time and effort. The sensor data corresponding to different climbing actions has its own characteristics. The data waveform is as follows Figure 1 and Figure 2As shown in the figure, it can be seen that the sensor data obtained when climbing the tower from the correct position shows good regularity, while the data obtained from climbing from other wrong positions show chaotic characteristics. Summary of the invention

[0006] The present invention is aimed at complex applications in power training. It uses sensors to collect the trainees' limb movements, completes the judgment of movement norms based on deep learning models and data processing technology, and uses virtual reality technology to build a simulation scene for training, thereby constructing a complete virtual reality tower climbing training method and system based on deep learning.

[0007] The technical solution of the present invention is a virtual reality tower climbing training method based on deep learning, which is characterized by comprising virtual reality tower climbing training and physical tower climbing training practice links;

[0008] The tower climbing training content based on virtual reality includes choosing different types of poles to climb up and down, tying long ropes, tying short ropes, and wearing safety helmets; building tower climbing scenes through 3d max, displaying virtual poles in the virtual training system, and trainees wearing VR glasses to watch and learn different tower climbing movements under the requirements of program instructions; trainees wear 6 nine-axis Bluetooth sensors on their limbs, and after each data is collected, it is transmitted to the deep learning model calculation server through the communication link. The deep learning model identifies the trainee's action category, and gives a judgment score through similarity comparison, and the result is transmitted to the master control server; the master control server transmits the data back to the VR glasses on the one hand, so that the trainees can see their scores, and the trainees can correct them while practicing; on the other hand, the results are displayed on the display screen for public display;

[0009] The content of the actual tower climbing training practice session includes that students also need to wear 6 nine-axis Bluetooth sensors. During the tower climbing process, their motion data will be sent back to the deep learning model server for standardization analysis of their movements. At this time, students no longer need to wear VR glasses. The system will provide students with feedback on their movement scores through voice. Both the VR virtual training session and the actual tower climbing session will collect students' motion data through nine-axis Bluetooth sensors, and share the deep learning model to judge the standardization of movements and give operation scores.

[0010] Furthermore, the deep learning model selects Transformer as the benchmark network for tower climbing behavior recognition, and constructs a core algorithm for judging the standardization of tower climbing actions through the deep learning model. The core algorithm can classify the current action based on the acquired limb movement posture data, and then judge which actions are standard and which are not through comparison.

[0011] Furthermore, it is necessary to pre-process the raw data of the tower climbing behavior obtained by the nine-axis Bluetooth sensor, including

[0012] (1) Moving average filtering

[0013] Use the moving average filtering algorithm to preprocess the raw data of the sensor. The specific algorithm is:

[0014]

[0015] Wherein, A is the original data, T is the moving step, i is the current time point, and F is the data after moving average filtering. The present invention sets the moving step to 5, and moves one position in the same direction each time;

[0016] (2) Data Dimension Expansion

[0017] In order to associate data and better reflect the data changes caused by personnel behavior activities, acceleration is used as a new data type. Acceleration only describes the data size but not the data direction. The acceleration calculation is shown in formula (2):

[0018]

[0019] (3) Data segmentation

[0020] According to the characteristics of the maintenance personnel's behavior and movement rules, the original data stream is segmented to obtain the segmented data set. In addition, the appropriate window size needs to be set when segmenting the data to ensure that each segmented segment contains complete action information; the sensor acquisition frequency is 10Hz, and the data expanded in the previous step is added after each row of data. The segmentation sliding window size is set to 20 sampling points, and the window slides according to the overlap rate of 80%, and finally a 20×10 data segmentation segment set is obtained;

[0021] (4) Tower Climbing Behavior Dataset Annotation

[0022] According to different sensors detecting specific detection categories, we finally obtained 19,207 data slices of training samples and data sets of 32 behavior categories. The labeled tower climbing behavior data sets were divided into training set, validation set, and test set according to the ratio of 70%, 20%, and 10%, and then trained on the deep model.

[0023] A virtual reality pole climbing tower training system based on deep learning includes a set of VR glasses, 6 nine-axis Bluetooth sensors, a deep learning model calculation server, a master control server, a display screen, and a virtual reality software platform.

[0024] Furthermore, the nine-axis Bluetooth sensor includes an XYZ three-axis acceleration data sensor, an XYZ three-axis angular velocity data sensor, and an XYZ three-axis angle data sensor; they are worn on six positions of the human body: the left upper arm, right upper arm, left wrist, right wrist, left ankle, and right ankle; the sensor acquires limb movement posture data at a frequency of 10 Hz, and sends these data to the deep learning model to determine whether the trainees' movements are standardized and correct.

[0025] The present invention has the following advantages and significance:

[0026] 1. The practice system will realize unattended training scenarios. Trainees can complete the training process completely by themselves under the guidance of the system, which is automatic, efficient and intelligent. Based on this practice link, the tower climbing training integrates virtual action learning and actual operation drills, which enhances the effect of training and improves the practicality and effectiveness of the system.

[0027] 2. The developed nine-axis sensor can accurately acquire motion features, improve the purity of the data source, and enable the deep learning model to maximize its effectiveness, ultimately forming an accurate judgment of the students’ movements.

[0028] 3. The present invention adds a deep learning model to distinguish actions in real time. This process goes through several steps, including data collection and uploading, data processing, and result feedback. Among them, data processing is an algorithm based on the deep learning model, which is often time-consuming. The present invention improves the system speed from the aspects of high-speed data transmission, lightweight model, and process optimization, so as to be able to provide timely feedback on the standardization of students' actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the X-axis acceleration data waveform of the climbing tower + left upper arm, (a) is with a safety rope, (b) is without a safety rope;

[0030] Figure 2 This is the X-axis acceleration data waveform of the left ankle when climbing the tower. (a) is when the safety rope is tied, and (b) is when the safety rope is not tied.

[0031] Figure 3 is the sensor wearing position map;

[0032] Figure 4 Figure 1. Raw acceleration data before filtering;

[0033] Figure 5 is the acceleration data graph after filtering;

[0034] Figure 6 It is the Transformer network structure diagram;

[0035] Figure 7 It is the structure diagram of the Transformer algorithm model;

[0036] Figure 8 is the network structure diagram; a is the self-attention mechanism, b is the masked multi-head attention;

[0037] Fig. 9 is a schematic diagram of action recognition results, a is the short rope tying action, b is the long rope tying action;

[0038] Fig.10 It is a VR-based tower climbing action training system architecture;

[0039] Fig.11 It is a display of the operation diagram, a is a virtual training scene picture, b is a practical tower climbing operation picture;

[0040] Fig.12 It is a tower climbing normative software display platform that provides an interface for displaying tower climbing action evaluation results. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings.

[0042] 1. Tower climbing behavior data collection based on nine-axis Bluetooth sensor

[0043] In order to accurately judge the movements of trainees, data with sufficiently high accuracy must be collected. In view of the movement characteristics of tower climbing behavior, sensors that can describe the orientation, posture, movement and other related information of human behavior are selected as auxiliary supervision tools. The tower climbing behavior of maintenance personnel can be comprehensively described by the displacement and rotation change states of the joints of the limbs. The displacement change of the joints is determined by its acceleration and velocity, and the rotation change of the joints is determined by the angular velocity and angle. In other words, the tower climbing behavior of maintenance personnel can be comprehensively described by the acceleration, angular velocity, angle and other information of each joint in space. Therefore, the present invention selects a nine-axis Bluetooth sensor containing XYZ three-axis acceleration data, XYZ three-axis angular velocity data, and XYZ three-axis angle data as the acquisition device. In order to ensure that the sensor can be fixed on the maintenance personnel without affecting normal operations, and at the same time be able to obtain enough body movement data to meet the detection requirements, it is finally determined that the nine-axis Bluetooth sensor device will be worn on the left upper arm, right upper arm, left wrist, right wrist, left ankle and right ankle of the human body, respectively. Figure 3 As shown. The sensor acquires data at a frequency of 10 Hz. The data of the limbs' movements and postures are acquired and sent to the action discrimination model to determine whether the trainees' movements are standardized and correct.

[0044] 2. Data Preprocessing

[0045] The nine-axis Bluetooth sensor accurately collects the trainee's action posture data (the original data of the sensor device). Due to the influence of the inherent signal caused by the process deviation of the sensor equipment and the interference of the involuntary trembling signal of the maintenance personnel, these data will be mixed with some useless noise signals, affecting the authenticity of the data. Therefore, it is necessary to perform the following data preprocessing on the acquired raw behavior data.

[0046] (1) Moving average filtering

[0047] The moving average filter is similar to a low-pass filter. It moves the continuous data in one direction and averages it according to a given moving distance and a given overlap rate, and combines the average values ​​obtained at each step into new continuous data in order. The present invention uses a moving average filter algorithm to preprocess the raw data of the sensor, which can remove the influence of clutter peaks in the data as much as possible, reduce the interference caused by noise, and effectively smooth the data, thereby making the characteristics of the tower climbing behavior data more prominent, and indirectly improving the accuracy of the algorithm. The specific algorithm is:

[0048]

[0049] Wherein, A is the original data, T is the moving step, i is the current time point, and F is the data after moving average filtering. The present invention sets the moving step to 5, and moves one position in the same direction each time.

[0050] Figure 4 This is the original acceleration data of the sensor before filtering. Figure 5 The figure shows the acceleration data after filtering. It can be seen from the figure that there are more noise glitches and abnormal points in the original data waveform. After moving average filtering, the waveform becomes smoother and more regular, and the samples with large deviations are corrected, making the behavior characteristics more obvious and convenient for the next step of processing.

[0051] (2) Data Dimension Expansion

[0052] When describing a person's behavior, the data collected by the sensor device is related to the device's own posture, and the sensor itself is affected by noise. There is little information correlation between different data at the same time. In order to correlate data and better reflect the data changes caused by the person's behavior, acceleration is used as a new data type. The acceleration only describes the data size but not the data direction. The acceleration calculation is shown in formula (2):

[0053]

[0054] (3) Data segmentation

[0055] Data segmentation is the last step of data preprocessing. The original tower climbing behavior data collected by the sensor device is a set of continuous sampling points, which is the behavior data within a period of time. It is impossible to directly process large data segments to obtain behavior features and perform behavior recognition. The present invention segments the original data stream according to the characteristics of the maintenance personnel's behavioral movement rules to obtain a segmented data set. In addition, it is necessary to set a suitable window size when segmenting the data to ensure that each segmented fragment contains complete action information. If the window size is set too large, each segmented fragment contains multiple behavior information, which will confuse the features; if the window size is set too small, the behavior information cannot be fully obtained. After a large amount of data experiments, it is found that most of the behavior actions will be completed within two seconds. The acquisition frequency of the sensor in the acquisition system of the present invention is 10 Hz, so the data expanded in the previous step is added after each row of data. Note: There are 6 sensors, each of which has three measurement dimensions: angle, acceleration, and angular velocity. Each dimension has values ​​in the three directions of XYZ. Therefore, it can be understood as finding a combined value in the three directions of XYZ. That is to say, each dimension has values ​​in the three directions of XYZ and also contains a combined value. For example, the angle measurement value is (1, 2, 3). After the dimension is expanded according to formula (2), it becomes (1, 2, 3, 3.741657). The size of the segmentation sliding window is set to 20 sampling points, and the window is slid according to an overlap rate of 80%. Finally, a set of 20×10 data segmentation fragments is obtained.

[0056] (4) Tower Climbing Behavior Dataset Annotation

[0057] At present, pole towers are generally divided into the following three types: Tower 1 is a cylindrical cement tower, which can be inspected with special footrests; Tower 2 is a cylindrical metal tower with grabs on the left and right sides; Tower 3 is a high-voltage iron tower. Different towers have different corresponding tower climbing movements. For example, a cylindrical tower is climbed vertically with a large range of movement, while an iron tower is climbed diagonally with a small range of movement.

[0058] According to the different towers and tower climbing actions, the tower climbing actions of maintenance personnel are divided into climbing of tower 1, tower 2 and tower 3 and climbing of tower 3 from the wrong position. During the up and down climbing process, towers 1 and 3 need to move the short protective rope position regularly, and need to tie a long protective rope to fix it when climbing to the top of the tower. The distinction between rope tying actions mainly relies on upper arm and wrist sensors, and the distinction between tower climbing actions mainly relies on upper arm and ankle sensors. Therefore, specific detection categories are detected according to different sensors. Finally, 19,207 data slices of training samples and data sets of 32 behavior categories were obtained. The labeled tower climbing behavior data set was divided into training set, validation set, and test set according to the ratio of 70%, 20%, and 10%. It will then be trained on the deep model.

[0059] 3. Tower climbing action normative detection algorithm based on deep learning

[0060] The core algorithm for judging the standardization of tower climbing actions is constructed through a deep learning model. The core algorithm can classify the current actions based on these data, and then judge which actions are standard and which are not through comparison.

[0061] There are various network models that can be used for classification, but the mainstream networks for processing time-series data are recurrent convolutional networks or Transformer series networks, etc. The Transformer network model was initially applied to natural language processing tasks, and was subsequently successfully used in image recognition, target detection and other fields. Its structure completely abandoned other neural network structures and only used the Attention mechanism to process tasks, and achieved good results. In view of the excellent performance of the Transformer structure when processing time-series data, this paper selects Transformer as the benchmark network for tower climbing behavior recognition.

[0062] The Transformer network model consists of two parts: the encoder module and the decoder module. The specific structure is as follows: Figure 6 As shown in the figure, the left half is the encoder and the right half is the decoder. The workflow of the Transformer network model is as follows: when receiving the data input, the Embedding module first converts the data into the vector form required by the Transformer, and then uses positional encoding (Positional Embedding) to add absolute position information and relative position information to the input. Then, the assembled data information is input into the Encoder for feature extraction, and finally the result is input into the Decoder. The Decoder also inputs the vector after position encoding, and then performs a series of processing, and obtains the final output through the final linear layer.

[0063] (1) Positional encoding

[0064] The various sensor data collected during the tower climbing operation are correlated and linked. If they are combined randomly, the correlation of the data in the data segment may be affected. However, Transformer abandons the sequential structure of the recurrent convolutional network and cannot directly obtain the positional relationship in the time series data input. In order to make up for this part of the missing information, the input needs to be preprocessed before data processing to complete the position information. To this end, Transformer sets the Encoder input to include two parts: one is the word vector obtained after Embedding processing, and the other is to use the Positional Embedding module to add position information to each word in the input.

[0065] Transformer concatenates the sequence vectors into a (m, N) matrix, where m is the number of vectors, N is the dimension of a single vector, and the position of an element in the matrix is ​​represented by the posth row and the ith column. The calculation process of the position information matrix is ​​as follows:

[0066]

[0067] The matrix dimension is (m, N), which indicates the position information of each element in the matrix. Then the even columns of the matrix are activated with the sine function, and the odd columns of the matrix are activated with the cosine function. The formula is as follows:

[0068]

[0069] After the sine and cosine functions are calculated, the final result of the position encoding is obtained. Transformer generally sets the dimension of the word vector output and the position encoding result output to the same, and then concatenates and fuses the two to obtain the real input of the encoder. The process is as follows Figure 7 shown.

[0070] (2) Encoder structure

[0071] The encoder part has a relatively simple function. It is composed of N completely repeated sub-blocks stacked together. It is responsible for learning the relationship information between sequences to extract sequence features. Each sub-block contains two modules, the multi-head attention mechanism and the feed forward neural network (FFN). The modules are connected through the residual network.

[0072] The Multi-Head Attention module is a combination of multiple original self-attention mechanisms. The self-attention mechanism is a form of attention mechanism, with a structure like Figure 8 It studies the internal process of biological observation of the outside world and adds a mechanism for local observation precision. The definition formula is as described in (5).

[0073]

[0074] Where Q (query), K (key value), and V (value) are obtained by linearly transforming the input of the self-attention mechanism using the linear transformation matrix WQ, WK, and WV. k is the number of columns of the matrix Q, K, that is, the vector dimension. The formula first calculates the inner product of the Q and K matrices and divides it by d kThe square root of prevents the inner product from being too large, and the Softmax function is used to calculate the attention coefficient between words, and finally multiplied by the matrix V to get the output result Z.

[0075] Multi-Head Attention is formed by combining multiple self-attentions. Linear transformations are performed separately through h different linear transformation matrices, and then different attention results are combined to obtain the final output result. The Multi-Head Attention structure is as follows: Figure 8 (b) as shown.

[0076] The calculation formula of Multi-Head Attention is as follows:

[0077] MultiHead(Q,K,V)=Concat(head1,...head h )W o

[0078] head i =Attention(QW i Q ,KW i K ,VW i V ) (6)

[0079] Among them, Attention represents the self-attention mechanism, and Concat represents the fusion of many self-attentions. The calculation first uses the Linear layer to linearly change Q, K, and V, and then performs h rounds of independent dot product calculations to obtain the output MultiHead. This operation uses different attention mechanisms to transform separately and then splices the results to achieve input mapping in different dimensions, so that the network obtains features of different dimensions and enhances the performance of the network.

[0080] The Feed Forward layer is relatively simple, and its function is spatial transformation. FFN contains two layers of linear transformation layers, and the activation function in the middle is ReLu.

[0081] (3) Decoder structure

[0082] The decoding module has a similar structure to the encoder module, and is also composed of N completely repeated sub-blocks stacked together. Each sub-module includes a multi-head attention mechanism, a masked multi-head attention mechanism, and an FFN layer. The Masked Multi-Head Attention mechanism uses a Masked operation, the main purpose of which is to mask the value, so that the decoder focuses on the output within time t before the current moment, and is not allowed to see future information. The generally used method is to generate an upper triangular matrix, set the values ​​of the upper triangle to 0, and then apply the triangular matrix to each input sequence, so that the purpose of shielding future information can be achieved. The final output of the encoder is the probability distribution at the corresponding position.

[0083] 4. Construction of tower climbing training system based on virtual reality

[0084] The training venue is an indoor venue, and the overall system architecture is as follows Fig.10 The system includes a set of VR glasses, 6 nine-axis sensors, a deep learning model computing server, a master control server, a display screen, and a virtual reality software platform.

[0085] The tower climbing training content includes choosing different types of poles to climb up and down, tying long ropes, tying short ropes, wearing safety helmets, etc. The tower climbing scene is built through 3d max, and the virtual pole is displayed in the virtual training system. Trainees wear VR glasses to watch and learn different tower climbing movements under the program instructions. Trainees wear 6 motion acquisition sensors on their limbs. After each data is collected, it is transmitted to the computing device through the communication link to score the standardization of the action. Trainees can correct it while practicing.

[0086] The VR virtual scene system has a good foundation. Based on the previous work, the present invention proposes a deep learning model server. The server will be equipped with the latest GPU dedicated chip, with more than 3,000 core units, support CUDA, and support deep learning platform deployment. The deep learning model identifies the action category of the trainee, gives a judgment score through similarity comparison, and transmits the result to the master control server. On the one hand, the master control service transmits the data back to the VR glasses so that the trainees can see their scores, and on the other hand, displays the results on the display screen for public display.

[0087] 5. Construction of the Tower Climbing Practice System

[0088] This system constructs a tower climbing training practice link, and provides a platform for students to practice by building a tower entity. After completing the tower climbing action training in the VR virtual classroom, students will operate on the physical tower. In the practical link, students also need to wear sensors. During the tower climbing process, their action data will be sent back to the deep learning model server to conduct normative analysis of their actions. At this time, students no longer need to wear VR glasses. The system feedbacks the action score to students through voice. In both the VR virtual training link and the physical tower climbing link, the students' action data are collected through the nine-axis sensor, and the deep learning model is shared to judge the normativeness of the action and give the operation score.

[0089] The training scene for trainees is divided into two parts: one is a simulated tower climbing based on virtual reality, and the other is a practical tower climbing operation in the training center. When simulating tower climbing, trainees wear helmets and can see the tower in the virtual scene. Under the instructions, they learn the various operations of tower climbing through the movements of their limbs. When practicing tower climbing, the sensors on the trainees' limbs send back the tower climbing action data in real time, and the back-end deep learning model determines the standardization of the action and provides relevant prompts for the operation.

[0090] This practice system will realize unattended training scenarios, where trainees can complete the training process entirely by themselves under the guidance of the system, and it is automatic, efficient, and intelligent. Fig.11 (b) shows the practical tower climbing operation. Fig.12 The tower climbing action evaluation result display interface is given. Based on this practical link, the tower climbing training integrates virtual action learning and actual operation drills, which enhances the training effect and improves the practicality and effectiveness of the system.

Claims

1. A virtual reality tower climbing training method based on deep learning, characterized in that: Including virtual reality-based tower climbing training and physical tower climbing training practice; The tower climbing training content based on virtual reality includes choosing different types of poles to climb up and down, tying long ropes, tying short ropes, and wearing safety helmets; building tower climbing scenes through 3d max, displaying virtual poles in the virtual training system, and trainees wearing VR glasses to watch and learn different tower climbing movements under the requirements of program instructions; trainees wear 6 nine-axis Bluetooth sensors on their limbs, and after each data is collected, it is transmitted to the deep learning model calculation server through the communication link, and the deep learning model is used to identify the trainee's action category, and the evaluation score is given through similarity comparison, and the result is transmitted to the master control server; On the one hand, the master control server transmits the data back to the VR glasses so that students can see their scores and make corrections while practicing; on the other hand, the results are displayed on the display screen for public display; The content of the actual tower climbing training practice session includes that students also need to wear 6 nine-axis Bluetooth sensors. During the tower climbing process, their motion data will be sent back to the deep learning model server for standardization analysis of their movements. At this time, students no longer need to wear VR glasses. The system will provide students with feedback on their movement scores through voice. Both the VR virtual training session and the actual tower climbing session will collect students' motion data through nine-axis Bluetooth sensors, and share the deep learning model to judge the standardization of movements and give operation scores.

2. The virtual reality tower climbing training method based on deep learning as claimed in claim 1, characterized in that: The deep learning model selects Transformer as the benchmark network for tower climbing behavior recognition. The core algorithm for judging the standardization of tower climbing actions is constructed through the deep learning model. The core algorithm can classify the current action based on the acquired limb posture data, and then judge which actions are standard and which are not through comparison.

3. The virtual reality tower climbing training method based on deep learning as claimed in claim 1, characterized in that: The raw data of tower climbing behavior obtained by the nine-axis Bluetooth sensor needs to be preprocessed, including (1) Moving average filtering Use the moving average filtering algorithm to preprocess the raw data of the sensor. The specific algorithm is: Wherein, A is the original data, T is the moving step, i is the current time point, and F is the data after moving average filtering. The present invention sets the moving step to 5, and moves one position in the same direction each time; (2) Data Dimension Expansion In order to associate data and better reflect the data changes caused by personnel behavior activities, acceleration is used as a new data type. Acceleration only describes the data size but not the data direction. The acceleration calculation is shown in formula (2): (3) Data segmentation According to the characteristics of the maintenance personnel's behavior and movement rules, the original data stream is segmented to obtain the segmented data set. In addition, the appropriate window size needs to be set when segmenting the data to ensure that each segmented segment contains complete action information; the sensor acquisition frequency is 10Hz, and the data expanded in the previous step is added after each row of data. The segmentation sliding window size is set to 20 sampling points, and the window slides according to the overlap rate of 80%, and finally a 20×10 data segmentation segment set is obtained; (4) Tower Climbing Behavior Dataset Annotation According to different sensors detecting specific detection categories, we finally obtained 19,207 data slices of training samples and data sets of 32 behavior categories. The labeled tower climbing behavior data sets were divided into training set, validation set, and test set according to the ratio of 70%, 20%, and 10%, and then trained on the deep model.

4. A virtual reality tower climbing training system based on deep learning, characterized in that: It includes a set of VR glasses, 6 nine-axis Bluetooth sensors, a deep learning model computing server, a master control server, a display screen, and a virtual reality software platform.

5. The deep learning-based virtual reality climbing tower training system according to claim 4, characterized in that: The nine-axis Bluetooth sensor includes an XYZ three-axis acceleration data sensor, an XYZ three-axis angular velocity data sensor, and an XYZ three-axis angle data sensor; they are worn on six positions of the human body: the left upper arm, right upper arm, left wrist, right wrist, left ankle, and right ankle; the sensor acquires limb movement posture data at a frequency of 10Hz, and sends these data to the deep learning model to determine whether the trainees' movements are standardized and correct.

Images