Immersive virtual reality simulation training system for wind power generation equipment

Abstract

The invention relates to the field of intelligent training, and particularly discloses an immersive wind power generation equipment virtual reality simulation training system, which is characterized in that a simulation training monitoring camera deployed on an immersive VR helmet worn by a student collects a training operation action video of the student; carrying out key frame sampling on the training operation action video by adopting a video image analysis and processing technology based on artificial intelligence, extracting a practical training action semantic feature and each historical training action semantic feature from the training operation action video, and then carrying out prompt gating dynamic aggregation on each historical training action semantic feature; therefore, whether a training operation error exists or not is automatically judged according to a semantic metric value between the training action semantic time sequence aggregation feature and the practical training action semantic feature, and a corresponding early warning and a correct video are generated. In this way, a correct operation demonstration video can be provided, and therefore the training effect is improved, and interactivity and interestingness are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent simulation training, and more specifically, to an immersive wind power generation equipment virtual reality simulation training system. Background Art

[0002] As an important renewable energy technology, wind power generation plays an important role in promoting sustainable resource utilization and environmental protection. In recent years, with the development of the market, the wind power industry has grown rapidly and the number of wind farms has continued to increase. Despite the rapid development of the wind power industry, the shortage of wind power technical talents has gradually emerged. This not only restricts the further development of the industry, but also brings challenges to the daily operations of enterprises.

[0003] At present, the training methods for wind power workers generally have the following problems: First, the training content is too theoretical and lacks practical links combined with actual operations, which makes it difficult for trainees to effectively apply the knowledge they have learned to their work. Secondly, the training format is single, mostly using classroom lectures, which lack interactivity and fun, and cannot fully mobilize the learning enthusiasm of trainees. In addition, the lack of a systematic assessment mechanism makes it impossible to fully evaluate the trainees' actual operating capabilities and technical specifications. The existing training methods often rely on subjective evaluations and lack objective and quantitative evaluation standards, making it difficult to accurately reflect the trainees' true level. This not only affects the inspection of the training effect, but also makes it difficult to timely discover and correct problems and deficiencies in the training, which in turn affects the improvement of trainees' operating skills and safety awareness.

[0004] Therefore, an optimized wind turbine training system is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] The embodiment of the present application provides an immersive wind power generation equipment virtual reality simulation training system, which includes: The camera deployment module is used to deploy the simulation training monitoring camera on the immersive VR helmet worn by the trainees; A training operation video acquisition module is used to acquire the training operation action video of the trainee collected by the simulation training monitoring camera; A training action semantic feature extraction module is used to perform key frame sampling and semantic feature extraction on the training operation action video to obtain a time series of training action semantic features; A practical historical training action extraction module is used to extract the time series of practical training action semantic features and historical training action semantic features from the time series of the training action semantic features; A training action dynamic aggregation module, used for dynamically aggregating the time series of the semantic features of the historical training actions with feature prompt gating to obtain a semantic temporal aggregation reasoning representation of the training actions; A semantic measurement module, used to calculate the semantic similarity between the semantic temporal aggregation reasoning representation of the training action and the semantic features of the practical training action to obtain a semantic consistency measurement value of the training action; The training operation judgment module is used to determine whether there is a training operation error based on the semantic consistency measurement value of the training action, and determine whether to generate an early warning prompt and a correct operation video of the wind power generation equipment.

[0007] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the training action semantic feature extraction module includes: a training action key frame sampling unit, which is used to perform key frame sampling on the training operation action video to obtain a time series of training action key frames; a training action semantic feature extraction unit, which is used to pass each training action key frame in the time series of the training action key frames through a training action semantic feature extractor based on a void convolutional neural network model to obtain a time series of training action semantic feature vectors as the time series of the training action semantic features.

[0008] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the practical historical training action extraction module is used to: extract the training action semantic feature vector at the last position in the time series of the training action semantic feature vector to obtain the practical training action semantic feature vector as the practical training action semantic feature; use the other training action semantic feature vectors in the time series of the training action semantic feature vector except the practical training action semantic feature vector as the historical training action semantic feature vector to obtain the time series of historical training action semantic feature vectors as the time series of the historical training action semantic features.

[0009] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the training action dynamic aggregation module includes: a current training action semantic feature extraction unit, which is used to extract the current training action semantic feature vector from the time series of the historical training action semantic feature vector; a training action semantic reasoning unit, which is used to use the current training action semantic feature vector as a feature prompt gate to perform gated temporal dynamic aggregation reasoning on the time series of the historical training action semantic feature vectors to obtain the training action semantic temporal aggregation reasoning representation.

[0010] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the training action semantic reasoning unit includes: a historical training action intensity prompt value calculation subunit, which is used to use the current training action semantic feature vector as a feature prompt gate to calculate the feature association intensity prompt value of each historical training action semantic feature vector in the time series of the historical training action semantic feature vector relative to the current training action semantic feature vector to obtain a sequence of historical training action semantic feature association intensity prompt values; a historical training action association intensity gate threshold determination subunit, which is used to extract the median of the sequence of historical training action semantic feature association intensity prompt values ​​as the historical training action semantic feature association intensity gate threshold; a historical training action semantic feature modulation subunit, which is used to use the historical training action semantic feature association intensity gate threshold as the switching threshold of the gating unit, and input the time series of the historical training action semantic feature vector and the sequence of the historical training action semantic feature association intensity prompt values ​​into the gating unit to obtain a sequence of modulated historical training action semantic feature vectors; a historical training action semantic feature aggregation reasoning subunit, which is used to perform feature aggregation reasoning on the sequence of modulated historical training action semantic feature vectors to obtain a training action semantic temporal aggregation reasoning representation vector as the training action semantic temporal aggregation reasoning representation.

[0011] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the historical training action intensity prompt value calculation subunit is used to: calculate the position point division between the historical training action semantic feature vector and the current training action semantic feature vector to obtain the historical training action semantic association strength vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the historical training action semantic association strength vector to obtain the historical training action semantic association strength logarithmic vector; calculate the position point multiplication between the historical training action semantic feature vector and the historical training action semantic association strength logarithmic vector to obtain the point product vector, and calculate the position point of each eigenvalue in the point product vector to obtain the historical training action semantic association strength value; calculate the natural exponential function value with the natural constant e as the base and the historical training action semantic association strength value as the exponential power to obtain the historical training action semantic feature association strength prompt value.

[0012] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the historical training action semantic feature modulation subunit is used to: compare each historical training action semantic feature association strength prompt value in the sequence of historical training action semantic feature association strength prompt values ​​with the historical training action semantic feature association strength gate threshold to obtain a sequence of weights; use each value in the sequence of weights as a weight to weighted modulate the time series of the historical training action semantic feature vector to obtain the sequence of modulated historical training action semantic feature vectors; wherein, in response to the historical training action semantic feature association strength prompt value being greater than or equal to the historical training action semantic feature association strength gate threshold, the value obtained by dividing the historical training action semantic feature association strength prompt value by the historical training action semantic feature association strength gate threshold is used as the weight; in response to the historical training action semantic feature association strength prompt value being less than the historical training action semantic feature association strength gate threshold, the weight is reset to zero.

[0013] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the historical training action semantic feature aggregation reasoning subunit is used to: input the sequence of the modulated historical training action semantic feature vectors into the feature dynamic propagation module based on the forward LTSM model to obtain the training action semantic temporal aggregation reasoning representation vector.

[0014] In the above-mentioned immersive wind power generation equipment virtual reality simulation training system, the training operation judgment module is used to: determine whether there is a training operation error based on the comparison between the semantic consistency measure value of the training action and a predetermined threshold; in response to the existence of a training operation error, generate an early warning prompt and generate a correct operation video of the wind power generation equipment.

[0015] Compared with the prior art, the present application provides an immersive wind power generation equipment virtual reality simulation training system, which collects the training operation action video of the trainee through the simulation training monitoring camera deployed on the immersive VR helmet worn by the trainee, and adopts the video image analysis and processing technology based on artificial intelligence to perform key frame sampling on the training operation action video, and extracts the semantic features of the practical training action and the semantic features of each historical training action, and then performs prompt gated dynamic aggregation on each historical training action semantic feature, so as to automatically judge whether there is a training operation error according to the semantic metric value between the semantic temporal aggregation feature of the training action and the semantic feature of the practical training action, and generate corresponding warnings and correct videos. In this way, the trainee's actions can be analyzed in real time, feedback can be generated instantly, errors in the operation can be pointed out, and correct operation demonstration videos can be provided, which can help trainees correct errors in time, improve training effects, and enhance interactivity and fun. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 It is a block diagram of an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application.

[0018] Figure 2 Schematic diagram of data flow of an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application.

[0019] Figure 3 It is a block diagram of a training action semantic feature extraction module in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application.

[0020] Figure 4 It is a block diagram of a training action dynamic aggregation module in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application.

[0021] Figure 5 The present invention is a block diagram of a training action semantic reasoning unit in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be noted that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where the application is located and with the authorization of the owner of the corresponding device.

[0024] As an important renewable energy technology, wind power generation plays an important role in promoting sustainable resource utilization and environmental protection. In recent years, with the development of the market, the wind power industry has grown rapidly and the number of wind farms has continued to increase. Despite the rapid development of the wind power industry, the shortage of wind power technical talents has gradually emerged. This not only restricts the further development of the industry, but also brings challenges to the daily operations of enterprises.

[0025] At present, the training methods for wind power workers generally have the following problems: First, the training content is too theoretical and lacks practical links combined with actual operations, which makes it difficult for trainees to effectively apply the knowledge they have learned to their work. Secondly, the training format is single, mostly using classroom lectures, which lack interactivity and fun, and cannot fully mobilize the learning enthusiasm of trainees. In addition, the lack of a systematic assessment mechanism makes it impossible to fully evaluate the trainees' actual operating capabilities and technical specifications. The existing training methods often rely on subjective evaluations and lack objective and quantitative evaluation standards, making it difficult to accurately reflect the trainees' true level. This not only affects the inspection of the training effect, but also makes it difficult to timely discover and correct problems and deficiencies in the training, which in turn affects the improvement of trainees' operating skills and safety awareness.

[0026] It should be understood that virtual reality is a computer technology that creates a virtual environment that is similar to or completely different from the real world by simulating multiple sensory experiences such as vision, hearing, and touch. Users can enter and interact in this virtual environment through special equipment (such as head-mounted displays, handles, etc.) to get an immersive feeling. Virtual reality technology is widely used in games, entertainment, education, medical care, military and other fields. Virtual reality is widely used in wind power equipment training. Through three-dimensional modeling technology, realistic wind power equipment and its working environment are created. These virtual scenes can include various components of wind turbines, control rooms, maintenance platforms, etc., making trainees feel as if they are in a real wind farm. This highly simulated environment helps trainees better understand and master the structure and working principle of the equipment.

[0027] Based on this, the present application proposes an immersive wind power generation equipment virtual reality simulation training system, which collects the training operation action video of the trainee through a simulation training monitoring camera deployed on an immersive VR helmet worn by the trainee, and adopts video image analysis and processing technology based on artificial intelligence to perform key frame sampling on the training operation action video, and extracts the semantic features of practical training actions and the semantic features of each historical training action, and then performs prompt gated dynamic aggregation on each historical training action semantic feature, so as to automatically judge whether there is a training operation error according to the semantic metric value between the semantic temporal aggregation feature of the training action and the semantic feature of the practical training action, and generate corresponding warnings and correct videos. In this way, the trainee's actions can be analyzed in real time, feedback can be generated instantly, errors in operation can be pointed out, and correct operation demonstration videos can be provided, which can help trainees correct errors in time, improve training effects, and enhance interactivity and fun.

[0028] Figure 1 It is a block diagram of an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the immersive wind power generation equipment virtual reality simulation training system 100 includes: a camera deployment module 110, which is used to deploy a simulation training monitoring camera on the immersive VR helmet worn by the trainee; a training operation video acquisition module 120, which is used to acquire the training operation action video of the trainee collected by the simulation training monitoring camera; a training action semantic feature extraction module 130, which is used to perform key frame sampling and semantic feature extraction on the training operation action video to obtain a time series of training action semantic features; a practical operation history training action extraction module 140, which is used to extract practical operation training action videos from the time series of the training action semantic features. The time series of the semantic features of the training action and the semantic features of the historical training action; a training action dynamic aggregation module 150, which is used to perform feature prompt gating dynamic aggregation on the time series of the semantic features of the historical training action to obtain the semantic temporal aggregation reasoning representation of the training action; a semantic measurement module 160, which is used to calculate the semantic similarity between the semantic temporal aggregation reasoning representation of the training action and the semantic features of the actual training action to obtain the semantic consistency measurement value of the training action; a training operation judgment module 170, which is used to determine whether there is a training operation error based on the training action semantic consistency measurement value, and determine whether to generate a warning prompt and a correct operation video of the wind power generation equipment.

[0029] In the embodiment of the present application, the camera deployment module 110 is used to deploy a simulation training monitoring camera on the immersive VR helmet worn by the trainee. It should be understood that the camera can record every operation detail of the trainee in the virtual environment in real time, including gestures, head movements, etc., and through real-time monitoring, the system can instantly detect whether the trainee's operation is correct and immediately give feedback to help the trainee correct the error in time.

[0030] In the embodiment of the present application, the training operation video acquisition module 120 is used to acquire the training operation action video of the trainee captured by the simulation training monitoring camera. It should be understood that the training operation action video of the trainee refers to all operations of the trainee in the virtual environment, including gestures, head movements, body postures, etc. Based on this, in order to be able to comprehensively and meticulously record and analyze the trainee's operations in wind power generation equipment training so as to provide real-time feedback and personalized guidance, in the technical solution of the present application, the training operation action video of the trainee captured by the simulation training monitoring camera is acquired.

[0031] In the embodiment of the present application, the training action semantic feature extraction module 130 is used to perform key frame sampling and semantic feature extraction on the training operation action video to obtain a time series of semantic features of the training action. Figure 3 FIG. 1 is a block diagram of a training action semantic feature extraction module in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. Specifically, Figure 3As shown, the training action semantic feature extraction module 130 includes: a training action key frame sampling unit 131, which is used to perform key frame sampling on the training operation action video to obtain a time sequence of training action key frames; a training action semantic feature extraction unit 132, which is used to pass each training action key frame in the time sequence of the training action key frames through a training action semantic feature extractor based on a hole convolutional neural network model to obtain a time sequence of training action semantic feature vectors as the time sequence of the training action semantic features. It should be understood that considering that the original training operation action video data usually contains a large number of frames, directly analyzing it may lead to excessive consumption of computing resources, and it also contains key and important student operation action information. Therefore, in the technical solution of the present application, the training operation action video is key frame sampled to obtain a time sequence of training action key frames. It should be understood that key frames refer to those frames that can represent important changes or events in the entire video clip, and usually contain key turning points or state changes of the action. By sampling the key frames of the video, it can be ensured that the important information that best reflects the characteristics and processes of the trainees in the wind power generation equipment operation training is retained, which is crucial for subsequent action analysis and evaluation. In particular, in a specific embodiment of the present application, the training operation action video is key frame sampled at a predetermined sampling frequency to obtain a time sequence of training action key frames, wherein the last training action key frame in the time sequence of the training action key frames is the image of the last frame in the entire video, so that it can be used to show whether the trainee's final operation state is correct. For example, in the simulation training of wind power generation equipment, the last frame can show whether the equipment is successfully started or shut down, which is crucial for evaluating the trainee's operation results. Accordingly, considering that each training action key frame in the time sequence of the training action key frames contains rich training action semantic information at different time points, such as the shape of the gesture, the speed of the operation, etc., it also contains some subtle movements in the trainee's operation. Based on this, in the technical solution of the present application, each training action key frame in the time sequence of the training action key frames is respectively passed through a training action semantic feature extractor based on a void convolutional neural network model to obtain a time sequence of training action semantic feature vectors. That is, the hole convolution can expand the receptive field without increasing the number of parameters by inserting holes in the convolution kernel (i.e. skipping some pixels), which means that the model can capture a wider range of training action context information to better represent the fine semantic features.

[0032] In an embodiment of the present application, the practical historical training action extraction module 140 is used to extract the time series of practical training action semantic features and historical training action semantic features from the time series of the training action semantic features. Specifically, in an embodiment of the present application, the practical historical training action extraction module is used to: extract the training action semantic feature vector at the last position in the time series of the training action semantic feature vector to obtain the practical training action semantic feature vector as the practical training action semantic feature; use the other training action semantic feature vectors in the time series of the training action semantic feature vector except the practical training action semantic feature vector as the historical training action semantic feature vector to obtain the time series of the historical training action semantic feature vector as the time series of the historical training action semantic feature. It should be understood that in order to more accurately evaluate the accuracy and standardization of the trainee's final operation, in the technical solution of the present application, the training action semantic feature vector at the last position in the time series of the training action semantic feature vector is extracted to obtain the practical training action semantic feature vector. At the same time, the other training action semantic feature vectors in the time series of the training action semantic feature vector except the practical training action semantic feature vector are used as historical training action semantic feature vectors to obtain the time series of historical training action semantic feature vectors. That is, the practical training action semantic feature vector represents the operational semantic information of the image of the last frame, and the time series of the historical training action semantic feature vector records in detail each state and action of the trainee during the entire operation process. By extracting the features of the last position and history, the historical dynamic behavior of the trainee's operation can be better understood and analyzed, providing data support for subsequent historical training action semantic reasoning and semantic measurement.

[0033] In an embodiment of the present application, the training action dynamic aggregation module 150 is used to perform dynamic aggregation of feature prompt gating on the time series of the semantic features of the historical training actions to obtain a semantic temporal aggregation reasoning representation of the training actions. Accordingly, considering that the historical training actions conform to the correct operating steps and specifications, under the correct training operation, the trainees will perform the established training actions. In this process, the training actions at the latter time point will be affected by the historical training actions at the previous moment. By combining the semantics of the historical actions at the previous moment, the subsequent correct operations can be inferred to determine whether the actions at the latter time point are consistent with the actions of the last frame of the actual operation. Therefore, in order to be able to infer based on historical training actions to infer the correct actions at the latter time point, so as to more accurately determine whether there is an operation error, in the technical solution of the present application, the time series of the semantic features of the historical training actions is dynamically aggregated with feature prompt gating to obtain a semantic temporal aggregation reasoning representation of the training actions.

[0034] Specifically, first, the current training action semantic feature vector is extracted from the time series of the historical training action semantic feature vector. That is, in the sequence, each historical training action semantic feature vector may have unique attributes or features, and these feature vectors symbolize the data of the element itself. Obtaining the current training action semantic feature vector is equivalent to determining a new reference point, and using this reference point to measure the relative importance of other vectors. Then, using the current training action semantic feature vector as the feature prompt gate, the feature association strength prompt value of each historical training action semantic feature vector relative to the current training action semantic feature vector is calculated to evaluate the strength of the relationship between the nodes, and based on this, it is determined which features need to be more widely propagated, and a sequence of historical training action semantic feature association strength prompt values ​​is obtained. In particular, this is actually an application example of the attention mechanism, in which the attention score is determined based on the similarity of the features between the nodes, so that the subsequent model can dynamically adjust the attention to other data according to the state of the current node, and give them greater weight in the subsequent feature propagation process. Then, the median of the sequence of historical training action semantic feature association strength prompt values ​​is extracted as the historical training action semantic feature association strength gate threshold. It can be understood that using the median as a judgment criterion can improve the model's sensitivity to the strength of feature associations at most time points, and effectively identify features with strong associations. Compared with the average, the median is less susceptible to interference from extreme values ​​or outliers, which helps prevent misjudgments caused by individual abnormal data points and more accurately reflects the central trend of the data. Afterwards, the historical training action semantic feature association strength gate threshold is used as the switch threshold of the gating unit, and the time series of the historical training action semantic feature vectors and the sequence of the historical training action semantic feature association strength prompt values ​​are input into the gating unit to manage the transmission of features by adjusting the weights, ensuring that those features with strong associations are more likely to be retained and sent to the next layer of the network, and obtaining a sequence of modulated historical training action semantic feature vectors. The gating mechanism enables the network to selectively allow or prevent the flow of specific feature information, and determine which features are worth passing to the next stage based on the previously set strength indicator value and threshold. Finally, the sequence of modulated historical training action semantic feature vectors is input into the feature dynamic propagation module based on the forward LSTM model to generate a training action semantic temporal aggregation reasoning representation vector containing the action semantic features of each time point by using LSTM to identify the interdependencies across time points or across adjacent nodes in the feature propagation process.

[0035] Figure 4 FIG. 1 is a block diagram of a training action dynamic aggregation module in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. Specifically, Figure 4As shown, the training action dynamic aggregation module 150 includes: a current training action semantic feature extraction unit 151, which is used to extract the current training action semantic feature vector from the time series of the historical training action semantic feature vectors; a training action semantic reasoning unit 152, which is used to use the current training action semantic feature vector as a feature prompt gate to perform gated temporal dynamic aggregation reasoning on the time series of the historical training action semantic feature vectors to obtain the training action semantic temporal aggregation reasoning representation.

[0036] Figure 5 : is a block diagram of a training action semantic reasoning unit in an immersive wind power generation equipment virtual reality simulation training system according to an embodiment of the present application. Specifically, Figure 5 As shown, the training action semantic reasoning unit 152 includes: a historical training action strength prompt value calculation subunit 1521, which is used to use the current training action semantic feature vector as a feature prompt gate to calculate the feature association strength prompt value of each historical training action semantic feature vector in the time series of the historical training action semantic feature vector relative to the current training action semantic feature vector to obtain a sequence of historical training action semantic feature association strength prompt values; a historical training action association strength gate threshold determination subunit 1522, which is used to extract the median of the sequence of historical training action semantic feature association strength prompt values ​​as the historical training action semantic feature association strength prompt value. degree gate threshold; a historical training action semantic feature modulation subunit 1523, which is used to use the historical training action semantic feature association strength gate threshold as the switch threshold of the gating unit, and input the time series of the historical training action semantic feature vector and the sequence of the historical training action semantic feature association strength prompt values ​​into the gating unit to obtain a sequence of modulated historical training action semantic feature vectors; a historical training action semantic feature aggregation reasoning subunit 1524, which is used to perform feature aggregation reasoning on the sequence of modulated historical training action semantic feature vectors to obtain a training action semantic temporal aggregation reasoning representation vector as the training action semantic temporal aggregation reasoning representation.

[0037] Specifically, in an embodiment of the present application, the historical training action intensity prompt value calculation subunit is used to: calculate the position point division between the historical training action semantic feature vector and the current training action semantic feature vector to obtain the historical training action semantic association intensity vector; calculate the logarithmic function value with base two of the absolute value of each eigenvalue of the historical training action semantic association intensity vector to obtain the logarithmic vector of the historical training action semantic association intensity; calculate the position point multiplication between the historical training action semantic feature vector and the logarithmic vector of the historical training action semantic association intensity to obtain the dot product vector, and calculate the position point of each eigenvalue in the dot product vector to obtain the historical training action semantic association intensity value; calculate the natural exponential function value with the natural constant e as the base and the historical training action semantic association intensity value as the exponential power to obtain the historical training action semantic feature association intensity prompt value.

[0038] Specifically, in an embodiment of the present application, the historical training action semantic feature modulation subunit is used to: compare each historical training action semantic feature association strength prompt value in the sequence of the historical training action semantic feature association strength prompt value with the historical training action semantic feature association strength gate threshold to obtain a sequence of weights; use each value in the sequence of weights as a weight to perform weighted modulation on the time series of the historical training action semantic feature vector to obtain the sequence of modulated historical training action semantic feature vectors; wherein, in response to the historical training action semantic feature association strength prompt value being greater than or equal to the historical training action semantic feature association strength gate threshold, the value obtained by dividing the historical training action semantic feature association strength prompt value by the historical training action semantic feature association strength gate threshold is used as the weight; in response to the historical training action semantic feature association strength prompt value being less than the historical training action semantic feature association strength gate threshold, the weight is reset to zero.

[0039] Specifically, in an embodiment of the present application, the historical training action semantic feature aggregation reasoning subunit is used to: input the sequence of the modulated historical training action semantic feature vectors into a feature dynamic propagation module based on a forward LSTM model to obtain the training action semantic temporal aggregation reasoning representation vector.

[0040] In the embodiment of the present application, specifically, the dynamic aggregation of feature prompt gating is performed on the time series of the historical training action semantic feature vector, which can be expressed by the formula: ; ; ; ; ; ; in, is the time queue of the historical training action semantic feature vector, , , and are the first, second, and third in the time queue of the historical training action semantic feature vector. and Historical training action semantic feature vectors, It is The feature value of each position in the semantic feature vector of the historical training action, It is The feature value of each position in the semantic feature vector of the historical training action, represents the logarithmic function value with base 2, is the number of eigenvalues ​​in each historical training action semantic feature vector, represents the exponential function value with the natural constant e as the base, yes The corresponding historical training action semantic feature association strength prompt value, , and , are the first, second and third in the sequence of the historical training action semantic feature association strength prompt values. The historical training action semantic feature association strength prompt value, To get the median of a sequence, is the historical training action semantic feature association strength gate threshold, For gated mask operation, for The modulated historical training action semantic feature vector after weighted modulation, , and are the first, second and third in the sequence of the modulated historical training action semantic feature vectors. modulated historical training action semantic feature vector, is the sequence of the modulated historical training action semantic feature vectors, Encode for LTSM, A semantic temporal aggregation reasoning representation vector is generated for the training action.

[0041] In an embodiment of the present application, the semantic measurement module 160 is used to calculate the semantic similarity between the training action semantic temporal aggregation reasoning representation and the practical training action semantic feature to obtain the training action semantic consistency measurement value. It should be understood that, considering that the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector respectively express the features obtained by the historical training action semantic reasoning and the action semantic information in the practical operation process, and in order to be able to evaluate the correctness and standardization of the trainee's operation in the wind power generation equipment training, in the technical solution of the present application, the semantic similarity between the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector is calculated to obtain the training action semantic consistency measurement value, so that the degree of matching between the trainee's operation and the standard operation can be quantified, and the result is converted into a specific numerical value, that is, the training action semantic consistency measurement value, so as to objectively evaluate the correctness and standardization of the trainee's operation. In particular, in a specific embodiment of the present application, the cosine distance between the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector is calculated as the training action semantic consistency measurement value.

[0042] In the embodiment of the present application, the training operation judgment module 170 is used to determine whether there is a training operation error based on the training action semantic consistency metric value, and determine whether to generate a warning prompt and a correct operation video of the wind power generation equipment. Specifically, in the embodiment of the present application, the training operation judgment module is used to: determine whether there is a training operation error based on the comparison between the training action semantic consistency metric value and the predetermined threshold; in response to the existence of a training operation error, generate a warning prompt and generate a correct operation video of the wind power generation equipment. That is, the training action semantic consistency metric value obtained by performing semantic measurement between the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector is compared with the predetermined threshold to automatically determine whether there is a training operation error, and generate a corresponding warning and correct video. In this way, errors in the trainee's operation can be detected in real time. If the consistency metric value is lower than the predetermined threshold, it indicates that the trainee's operation may be wrong or irregular. When an operation error is detected, the system will generate a warning prompt to remind the trainee to pay attention to the problem in the operation. The warning prompt can be in the form of text, voice or graphics, depending on the configuration of the system. The system will also generate a video of the correct operation of wind power equipment, showing the correct operating steps and specifications, which can help trainees intuitively understand the correct operation methods.

[0043] It should be understood that here, when the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector respectively represent the image semantic features of historical frames and current frames in the training operation action video, when calculating the semantic similarity, the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector will also have an imbalance in the semantic similarity space domain regression understanding representation relative to the feature distribution field domain selective state space of the local image semantic space domain, thereby deviating from the semantic similarity space domain mapping understanding of the source image semantic feature temporal distribution representation, affecting the calculation accuracy of the training action semantic consistency measure.

[0044] Preferably, calculating the semantic similarity between the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector to obtain a training action semantic consistency measure value includes: cascading the training action semantic temporal aggregation reasoning representation vector and the practical training action semantic feature vector into a training action-practical training action semantic joint representation vector; determining the number of zero eigenvalues ​​in the training action-practical training action semantic joint representation vector, and respectively calculating the reciprocal of the logarithm of the number of zero eigenvalues ​​with base two and the exponential value of the reciprocal of the number of zero eigenvalues ​​with base natural constants to obtain a first training action-practical training action semantic joint representation value and a second training action-practical training action semantic joint representation value; calculating the power function of each eigenvalue of the training action-practical training action semantic joint representation vector with the reciprocal of the number of zero eigenvalues ​​as the exponent to obtain a training action-practical training action semantic joint structural state vector; comparing the training action-practical training action semantic joint structural state vector with the first training action-practical training action The action semantics joint representation value is point multiplied to obtain the training action-practical training action semantics joint information representation vector; the autocorrelation matrix of the training action-practical training action semantics joint structural state vector is point multiplied with the second training action-practical training action semantics joint representation value and the weight hyperparameter respectively to obtain the training action-practical training action semantics joint regression understanding matrix; the training action-practical training action semantics joint information representation vector is matrix multiplied with the training action-practical training action semantics joint regression understanding matrix to obtain the optimized training action-practical training action semantics joint representation vector; the optimized training action-practical training action semantics joint representation vector is split into an optimized training action semantics temporal aggregation reasoning representation vector and an optimized practical training action semantics feature vector; the semantic similarity between the optimized training action semantics temporal aggregation reasoning representation vector and the optimized practical training action semantics feature vector, which is obtained by splitting the optimized training action-practical training action semantics joint representation vector, is calculated to obtain the training action semantics consistency measure value.

[0045] That is, the first training action-practical training action semantic joint representation value and the second training action-practical training action semantic joint representation value are expressed as: ; ; in, is the number of zero eigenvalues ​​in the training action-practical training action semantic joint representation vector, represents the logarithmic function with base 2, represents the exponential function value with the natural constant e as the base, is the semantic joint representation value of the first training action and the practical training action, The semantic joint representation value of the second training action-practical training action.

[0046] And the optimized training action-practical training action semantic joint representation vector is expressed as: ; ; ; in, is the semantic joint representation value of the first training action and the practical training action, The training action-practical training action semantic joint representation vector, and is a row vector, It means point multiplication by position. is the number of zero eigenvalues ​​in the training action-practical training action semantic joint representation vector, To calculate each eigenvalue of the training action-practical training action semantic joint representation vector, a power function with the inverse of the number of zero eigenvalues ​​as the exponent, is the training action-practical training action semantic joint information representation vector, represents the weight hyperparameter, is the semantic joint representation value of the second training action-practical training action, for The transposed vector of is the training action-practical training action semantic joint regression understanding matrix, represents matrix multiplication, It is the optimized training action-practical training action semantic joint representation vector.

[0047] Based on this, the present application realizes the effective long-range modeling of the training action semantics temporal aggregation reasoning representation vector and the practical training action semantics feature vector under the field linear complexity by modeling the vector field zero dimension of the training action semantics temporal aggregation reasoning representation vector and the practical training action semantics feature vector obtained by cascading the training action semantics temporal aggregation reasoning representation vector and the practical training action semantics feature vector under the field linear complexity, so that in the feature mapping regression process of the training action-practical training action semantics joint representation vector, the structured correlation self-distillation between information representation and regression understanding is performed under the selective state space based on the feature distribution, thereby maintaining the overall information complexity of the feature distribution of the training action-practical training action semantics joint representation vector, while promoting the remote dynamic regression understanding balance of its feature convergence, so as to improve the understanding consistency between the matching degree measurement mapping regression and feature extraction process, and improve the calculation accuracy of the training action semantics consistency measure. In this way, the trainee's actions can be analyzed in real time, feedback can be generated instantly, errors in the operation can be pointed out, and correct operation demonstration videos can be provided, which can help trainees correct errors in time, improve training effects, and enhance interactivity and fun.

[0048] In summary, the immersive wind power generation equipment virtual reality simulation training system 100 based on the embodiment of the present application is explained, which collects the training operation action video of the trainee through the simulation training monitoring camera deployed on the immersive VR helmet worn by the trainee, and adopts the video image analysis and processing technology based on artificial intelligence to perform key frame sampling on the training operation action video, and extracts the semantic features of the practical training action and the semantic features of each historical training action, and then performs prompt gated dynamic aggregation on each historical training action semantic feature, so as to automatically judge whether there is a training operation error according to the semantic metric value between the semantic temporal aggregation feature of the training action and the semantic feature of the practical training action, and generate corresponding warnings and correct videos. In this way, the trainee's actions can be analyzed in real time, feedback can be generated instantly, errors in the operation can be pointed out, and correct operation demonstration videos can be provided, which can help trainees correct errors in time, improve training effects, and enhance interactivity and fun.

[0049] As described above, the immersive wind power generation equipment virtual reality simulation training system 100 according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an immersive wind power generation equipment virtual reality simulation training algorithm. In a possible implementation, the immersive wind power generation equipment virtual reality simulation training system 100 according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the immersive wind power generation equipment virtual reality simulation training system 100 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the immersive wind power generation equipment virtual reality simulation training system 100 can also be one of the many hardware modules of the wireless terminal.

[0050] Alternatively, in another example, the immersive wind turbine virtual reality simulation training system 100 and the wireless terminal may also be separate devices, and the immersive wind turbine virtual reality simulation training system 100 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0051] The above descriptions of various implementations of the present disclosure are exemplary and not exhaustive. They are not limited to the disclosed implementations, and many modifications and variations are obvious to those skilled in the art without departing from the scope and spirit of the described implementations.

Claims

1. An immersive wind power generation equipment virtual reality simulation training system, characterized in that: include: The camera deployment module is used to deploy the simulation training monitoring camera on the immersive VR helmet worn by the trainees; A training operation video acquisition module is used to acquire the training operation action video of the trainee collected by the simulation training monitoring camera; A training action semantic feature extraction module is used to perform key frame sampling and semantic feature extraction on the training operation action video to obtain a time series of training action semantic features; A practical historical training action extraction module is used to extract the time series of practical training action semantic features and historical training action semantic features from the time series of the training action semantic features; A training action dynamic aggregation module, used for dynamically aggregating the time series of the semantic features of the historical training actions with feature prompt gating to obtain a semantic temporal aggregation reasoning representation of the training actions; A semantic measurement module, used to calculate the semantic similarity between the semantic temporal aggregation reasoning representation of the training action and the semantic features of the practical training action to obtain a semantic consistency measurement value of the training action; The training operation judgment module is used to determine whether there is a training operation error based on the semantic consistency measurement value of the training action, and determine whether to generate an early warning prompt and a correct operation video of the wind power generation equipment.

2. The immersive wind power generation equipment virtual reality simulation training system according to claim 1 is characterized in that: The training action semantic feature extraction module includes: A training action key frame sampling unit, used for performing key frame sampling on the training operation action video to obtain a time sequence of training action key frames; The training action semantic feature extraction unit is used to pass each training action key frame in the time sequence of the training action key frames through a training action semantic feature extractor based on a hole convolutional neural network model to obtain a time sequence of training action semantic feature vectors as the time sequence of the training action semantic features.

3. The immersive wind power generation equipment virtual reality simulation training system according to claim 2 is characterized in that: The practical historical training action extraction module is used to: Extracting the last position of the training action semantic feature vector in the time series of the training action semantic feature vector to obtain a practical training action semantic feature vector as the practical training action semantic feature; The other training action semantic feature vectors in the time series of the training action semantic feature vectors except the practical training action semantic feature vector are used as historical training action semantic feature vectors to obtain a time series of historical training action semantic feature vectors as the time series of historical training action semantic features.

4. The immersive wind power generation equipment virtual reality simulation training system according to claim 3 is characterized in that: The training action dynamic aggregation module includes: A current training action semantic feature extraction unit, used to extract a current training action semantic feature vector from the time series of the historical training action semantic feature vectors; The training action semantic reasoning unit is used to use the current training action semantic feature vector as a feature prompt gate to perform gated temporal dynamic aggregation reasoning on the time series of the historical training action semantic feature vectors to obtain the training action semantic temporal aggregation reasoning representation.

5. The immersive wind power generation equipment virtual reality simulation training system according to claim 4 is characterized in that: The training action semantic reasoning unit comprises: A historical training action strength prompt value calculation subunit is used to use the current training action semantic feature vector as a feature prompt gate to calculate the feature association strength prompt value of each historical training action semantic feature vector in the time series of the historical training action semantic feature vector relative to the current training action semantic feature vector to obtain a sequence of historical training action semantic feature association strength prompt values; A historical training action association strength gate threshold determination subunit is used to extract the median of the sequence of historical training action semantic feature association strength prompt values ​​as the historical training action semantic feature association strength gate threshold; A historical training action semantic feature modulation subunit is used to use the historical training action semantic feature association strength gate threshold as the switch threshold of the gate control unit, and input the time series of the historical training action semantic feature vectors and the sequence of the historical training action semantic feature association strength prompt values ​​into the gate control unit to obtain a sequence of modulated historical training action semantic feature vectors; The historical training action semantic feature aggregation reasoning subunit is used to perform feature aggregation reasoning on the sequence of the modulated historical training action semantic feature vectors to obtain a training action semantic temporal aggregation reasoning representation vector as the training action semantic temporal aggregation reasoning representation.

6. The immersive wind power generation equipment virtual reality simulation training system according to claim 5 is characterized in that: The historical training action intensity prompt value calculation subunit is used to: Calculate the historical training action semantic feature vector and the current training action semantic feature vector by dividing them by the position points to obtain a historical training action semantic association strength vector; Calculating the logarithmic function value with base 2 of the absolute value of each eigenvalue of the historical training action semantic association strength vector to obtain a logarithmic vector of the historical training action semantic association strength; Calculate the position-by-position point multiplication between the historical training action semantic feature vector and the historical training action semantic association strength logarithmic vector to obtain a point product vector, and calculate the position-by-position point of each feature value in the point product vector to obtain the historical training action semantic association strength value; The natural exponential function value with the natural constant e as the base and the historical training action semantic association strength value as the exponential power is calculated to obtain the historical training action semantic feature association strength prompt value.

7. The immersive wind power generation equipment virtual reality simulation training system according to claim 6 is characterized in that: The historical training action semantic feature modulation subunit is used to: Comparing each historical training action semantic feature association strength prompt value in the sequence of the historical training action semantic feature association strength prompt values ​​with the historical training action semantic feature association strength gate threshold to obtain a sequence of weights; Using each value in the weight sequence as a weight to perform weighted modulation on the time sequence of the historical training action semantic feature vector to obtain the sequence of the modulated historical training action semantic feature vector; Wherein, in response to the historical training action semantic feature association strength prompt value being greater than or equal to the historical training action semantic feature association strength gate threshold, the value obtained by dividing the historical training action semantic feature association strength prompt value by the historical training action semantic feature association strength gate threshold is used as the weight; In response to the historical training action semantic feature association strength prompt value being less than the historical training action semantic feature association strength gate threshold, the weight is reset to zero.

8. The immersive wind power generation equipment virtual reality simulation training system according to claim 7, characterized in that: The historical training action semantic feature aggregation reasoning subunit is used to: The sequence of the modulated historical training action semantic feature vectors is input into a feature dynamic propagation module based on a forward LTSM model to obtain the training action semantic temporal aggregation reasoning representation vector.

9. The immersive wind power generation equipment virtual reality simulation training system according to claim 8, characterized in that: The training operation judgment module is used to: Determining whether there is a training operation error based on a comparison between the training action semantic consistency measure value and a predetermined threshold; In response to the existence of a training operation error, an early warning prompt is generated and a video of the correct operation of the wind power generation equipment is generated.

Images