Wind-farm power equipment training method and apparatus based on virtual reality technology

Abstract

The present application relates to the technical field of virtual reality (VR). Disclosed are a wind-farm power equipment training method and apparatus based on VR technology. The method comprises: on the basis of a specific scenario of a wind-farm power equipment facility, constructing a three-dimensional digital scenario; on the basis of a VR device combined with electroencephalogram signal collection, collecting electroencephalogram signal data of when a trainee operates in the three-dimensional digital scenario; performing filtering and feature extraction on the collected electroencephalogram signal data, and constructing a time-frequency feature map; on the basis of the time-frequency feature map, training a deep learning model; and inputting real-time electroencephalogram signal data of the trainee into the trained deep learning model, identifying an attention state of the trainee, and outputting in real time the attention state of the trainee. In the present application, VR technology is used to simulate an actual operation scenario of a wind farm, thereby providing a real and comprehensive training experience; furthermore, an electroencephalogram signal of a trainee is collected and an attention situation thereof is analyzed and diagnosed, thereby facilitating the monitoring of the state of the trainee and improving the training effect.

Description

Training Method and Device for Wind Farm Power Equipment Based on Virtual Reality Technology

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410996054.1, filed on July 23, 2024, entitled “Training Method and Apparatus for Wind Farm Power Equipment Based on Virtual Reality Technology”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of virtual reality technology, specifically to a training method and device for wind farm power equipment based on virtual reality technology. Background Technology

[0004] Virtual Reality (VR) is a technology that garnered significant attention from the scientific and engineering communities in the 1990s. Its rise opened up new research areas for the development of human-computer interfaces; provided new interface tools for intelligent engineering applications; and offered new descriptive methods for large-scale data visualization in various engineering projects. The key feature of this technology is that computers generate a human-made virtual environment. This virtual environment is created through three-dimensional space constructed using computer graphics, or by incorporating other real-world environments into the computer to produce a realistic "virtual environment," thus giving users a visually immersive experience.

[0005] In today's energy sector, wind farms, as a crucial component of green and renewable energy, are increasingly in demand for technical personnel for their construction and operation. However, traditional wind farm power equipment training methods are often limited by site, equipment, and the safety of actual operation, making it difficult to provide a realistic and comprehensive training experience. Furthermore, the trainees' level of concentration during the training process significantly impacts the training effectiveness.

[0006] Summary of the Invention

[0007] In view of this, this application provides a wind farm power equipment training method and apparatus based on virtual reality technology to solve the problem of how to provide trainees with a realistic and comprehensive training experience while identifying the trainees' attention during the training process.

[0008] Firstly, this application provides a training method for wind farm power equipment based on virtual reality technology, the method comprising:

[0009] A three-dimensional digital scene is constructed based on the specific scenario of wind farm power equipment and facilities, and the three-dimensional digital scene is updated in real time based on the operation of the trainees;

[0010] The training personnel's electroencephalogram (EEG) signal data is collected using a VR device that combines EEG signal acquisition.

[0011] The collected EEG signal data is filtered and features are extracted, and a time-frequency feature map is constructed;

[0012] A deep learning model is trained based on the aforementioned time-frequency feature map;

[0013] The trainee's real-time EEG signal data is input into the trained deep learning model to identify the trainee's attention state and output the trainee's attention state in real time.

[0014] Beneficial effects: By constructing a 3D digital scene based on the specific scenario of wind farm power equipment, trainees can operate the power equipment in the 3D digital scene, such as rotating, splicing and installing the unit's fine equipment; after the trainees operate, the 3D digital scene will be rendered in real time according to the operation, so that the 3D digital scene can be updated in real time.

[0015] When trainees operate in a 3D digital scene, their EEG signal data is collected in real time by VR devices that collect EEG signals. The EEG signal data is then transmitted to a server that is wirelessly connected to the VR devices that collect EEG signals. The server filters and extracts features from the collected EEG signal data and then constructs a time-frequency feature map based on the extracted features.

[0016] The deep learning model is trained based on the constructed time-frequency feature map. After training, trainees' electroencephalogram (EEG) data is collected in real time while they perform actual operations in a 3D digital scene. This EEG data is then input into the trained deep learning model. The model calculates and identifies the trainees' attention state, outputting it in real time. By leveraging the immersive, interactive, and imaginative characteristics of virtual reality technology, the actual operation of a wind farm can be simulated in a virtual environment. This allows trainees to learn about the operation, maintenance, and management of wind farm power equipment without time and space constraints. Furthermore, by collecting and analyzing the trainees' EEG signals, their attention status can be analyzed and diagnosed, aiding in monitoring their condition and improving training effectiveness.

[0017] In one optional implementation, the acquisition of electroencephalogram (EEG) signal data of the trainee during operation in the three-dimensional digital scene includes:

[0018] Collect EEG signal data of the trainees when they are performing operations under attentive conditions and when they are performing operations under non-attentive conditions.

[0019] In one optional implementation, the step of filtering and extracting features from the acquired EEG signal data and constructing a time-frequency feature map includes:

[0020] A zero-phase bandpass filter is constructed at each target Hertz within a preset frequency domain range of the EEG signal data, and multiple sub-band data are obtained after filtering.

[0021] Each of the filtered EEG signal data is divided into non-overlapping segments within a preset time window to obtain multiple preset time data segments.

[0022] Perform Hilbert transform on the data segments of a preset time within each sub-band, and use the amplitude of the transformed sub-band as a feature to construct a time-frequency feature map.

[0023] In one optional implementation, the process of performing a Hilbert transform on the data segments at preset times within each sub-band includes:

[0024] The analytical signal of the EEG signal is obtained by the following formula:

[0025] The amplitude information of the EEG signal is calculated based on equation (2). The calculation formula is as follows:

[0026] Where z(t) is the analytic signal of the EEG signal; x(t) is the EEG signal; t and τ are time quantities; i∈{0,1,…}; φ(t) is the time window of the preset time; The Hilbert transform of the EEG signal x(t) is obtained by convolving it with the function h(t) = 1 / πt; pv is the Cauchy principal value; A(t) is the amplitude information of the EEG signal.

[0027] In one optional implementation, training the deep learning model based on the time-frequency feature map includes:

[0028] The amplitude features of all sub-bands in each preset time segment are concatenated to obtain the time-frequency matrix feature map of the EEG signal data;

[0029] The time-frequency matrix feature map is preprocessed;

[0030] An Inception deep learning model for attention recognition is built based on the Inception architecture, and a cross-subject attention model is established by combining the preprocessed time-frequency matrix feature map.

[0031] The time-frequency feature matrix in the preprocessed time-frequency matrix feature map is input into the cross-subject attention model to calculate the category of the EEG signal.

[0032] In one optional implementation, the preprocessing of the time-frequency matrix feature map includes:

[0033] The time-frequency feature matrix is ​​obtained based on the time-frequency matrix feature map;

[0034] The time-frequency feature matrix is ​​resampled in the time dimension to obtain the time-frequency feature matrix of each preset time segment.

[0035] In one optional implementation, the step of inputting the preprocessed time-frequency matrix feature map into the cross-subject attention model and calculating the category of the EEG signal includes:

[0036] Input the time-frequency feature matrix of the data segment for each preset time period into the cross-subject attention model;

[0037] Based on the convolutional and max pooling layers in the cross-subject attention model, feature information is extracted from the time-frequency feature matrix of the data segment at each preset time.

[0038] The extracted feature information is input into the fully connected layer;

[0039] The feature information output by the fully connected layer is calculated based on a multi-classification algorithm to obtain the category of the EEG signal.

[0040] In one optional implementation, the formula for calculating the category of the EEG signal is:

[0041] The conditional probability of an EEG signal belonging to one of its categories is calculated using the following formula:

[0042] After calculating the conditional probability of each class in the EEG signal, the class with the highest probability in the EEG signal is obtained and used as the final prediction output. The calculation formula is as follows:

[0043] Where p(y=k|x) is the conditional probability of one category of EEG signal; y is the output; k is one category of EEG signal; x is the input feature information of EEG signal; and X is all feature information of EEG signal. These are the weight coefficients in the fully connected layer; The category with the highest probability in the electroencephalogram (EEG) signal; represents all weight coefficients in the fully connected layer; K represents all categories of EEG signals; b represents the bias term.

[0044] Secondly, this application also provides a wind farm power equipment training device based on virtual reality technology, the device comprising:

[0045] The first construction module is used to construct a three-dimensional digital scene based on the specific scenario of wind farm power equipment and facilities. The three-dimensional digital scene is updated in real time based on the operation of the trainees.

[0046] The acquisition module is used to acquire the electroencephalogram (EEG) signal data of the trainee when operating in the three-dimensional digital scene based on a VR device that combines EEG signal acquisition;

[0047] The second construction module is used to filter and extract features from the collected EEG signal data, and to construct a time-frequency feature map;

[0048] The training module is used to train a deep learning model based on the time-frequency feature map;

[0049] The recognition module is used to input the real-time EEG signal data of the trainee into the trained deep learning model, recognize the trainee's attention state, and output the trainee's attention state in real time.

[0050] Thirdly, this application also provides an electronic device, including: a memory, a processor, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0051] The memory is used to store executable instructions that cause the processor to perform the above-described training method for wind farm power equipment based on virtual reality technology.

[0052] Fourthly, this application also provides a computer-readable storage medium, characterized in that the storage medium stores executable instructions, which, when executed on an electronic device or a wind farm power equipment training device based on virtual reality technology, cause the electronic device or the wind farm power equipment training device based on virtual reality technology to perform the operation of the wind farm power equipment training method based on virtual reality technology described above.

[0053] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 is a flowchart illustrating the wind farm power equipment training method based on virtual reality technology provided in an embodiment of this application;

[0056] Figure 2 is a schematic diagram of the structure of a VR device combining EEG signal acquisition provided in an embodiment of this application;

[0057] Figure 3 is a flowchart illustrating a wind farm power equipment training method based on virtual reality technology according to another embodiment of this application;

[0058] Figure 4 is a flowchart illustrating a wind farm power equipment training method based on virtual reality technology according to another embodiment of this application;

[0059] Figure 5 is a schematic diagram of the convolutional network structure of the deep learning model provided in the embodiments of this application;

[0060] Figure 6 is a schematic diagram of the structure of the wind farm power equipment training device based on virtual reality technology provided in an embodiment of this application;

[0061] Figure 7 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application.

[0062] Attached reference numerals: 1. VR device main body; 2. Glasses pad; 3. Headband; 4. Fixing plate; 5. Side straps; 6. Buffer adjustment pad; 7. Head pad; 8. Connecting rod; 9. EEG signal acquisition point. Detailed Implementation

[0063] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0064] This application provides a VR device that incorporates electroencephalogram (EEG) signal acquisition, as shown in Figure 2. It includes: a VR device main body 1, glasses cushion 2, a headband 3, a fixing plate 4, side straps 5, a cushioning adjustment pad 6, a head cushion 7, multiple connecting and fixing rods 8, and multiple EEG signal acquisition points 9. The position of the EEG signal acquisition points 9 can be adjusted according to the user's head size; the head cushion 7 can be adjusted and replaced according to usage needs. The data acquisition unit records the EEG signals measured by the sensors and transmits them to a server for processing and analysis.

[0065] After the trainee wears the VR device that collects EEG signals, it is necessary to ensure that the VR device fits snugly against the body surface to more accurately measure the weak biomagnetic field signals near the head. Once the kit is installed, the trainee's status parameters can be collected, such as EEG signals in the forehead and hemoglobin levels in the brain.

[0066] The following describes a specific embodiment of a wind farm power equipment training method based on virtual reality technology. Figure 1 is a flowchart illustrating a wind farm power equipment training method based on virtual reality technology provided in this embodiment. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-creative labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many possible execution orders and does not represent the only execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or figures, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in Figure 1, the method may include the following steps:

[0067] Step S100: Construct a three-dimensional digital scene based on the specific scenario of the wind farm's power equipment and facilities. The three-dimensional digital scene is updated in real time based on the operations of the trainees.

[0068] Step S200: Collect EEG signal data of trainees when they operate in a three-dimensional digital scene using VR devices that combine EEG signal acquisition;

[0069] Step S300: Filter and extract features from the collected EEG signal data, and construct a time-frequency feature map;

[0070] Step S400: Train a deep learning model based on time-frequency feature maps;

[0071] Step S500: Input the real-time EEG signal data of the trainees into the trained deep learning model, identify the trainees' attention state, and output the trainees' attention state in real time.

[0072] In this embodiment, a three-dimensional digital scene is constructed based on the specific scenario of the wind farm's power equipment. This allows trainees to operate the power equipment within the three-dimensional digital scene, such as rotating, assembling, and installing the unit's fine equipment. After the trainees operate the equipment, the three-dimensional digital scene is rendered in real time to update the scene in real time.

[0073] When trainees operate in a 3D digital scene, their EEG signal data is collected in real time by VR devices that collect EEG signals. The EEG signal data is then transmitted to a server that is wirelessly connected to the VR devices that collect EEG signals. The server filters and extracts features from the collected EEG signal data and then constructs a time-frequency feature map based on the extracted features.

[0074] The deep learning model is trained based on the constructed time-frequency feature map. After training, trainees' electroencephalogram (EEG) data is collected in real time while they perform actual operations in a 3D digital scene. This EEG data is then input into the trained deep learning model. The model calculates and identifies the trainees' attention state, outputting it in real time. By leveraging the immersive, interactive, and imaginative characteristics of virtual reality technology, the actual operation of a wind farm can be simulated in a virtual environment. This allows trainees to learn about the operation, maintenance, and management of wind farm power equipment without time and space constraints. Furthermore, by collecting and analyzing the trainees' EEG signals, their attention status can be analyzed and diagnosed, aiding in monitoring their condition and improving training effectiveness.

[0075] In step S100, a 3D digital scene of the wind farm's power equipment and facilities can be constructed using Unity3D modeling software. First, the VR device system pre-sets specific scenes within the wind farm and includes scene demonstrations and animations, such as the handling and unloading of turbine equipment, the rotation, assembly, and installation of delicate turbine components, and simulates different environmental conditions, such as onshore and offshore wind power environments, or high-altitude operations. Once the trainee wears the VR device and selects a training scene, the system loads the corresponding 3D digital scene and simulates and presents the object models, textures, and lighting within the 3D digital scene, integrating the rendered scene and river surface into the trainee's VR headset. During operation, the trainee interacts with the interface in the 3D digital scene using devices such as a mouse or controller, such as selecting equipment models and moving them. The system renders and updates the 3D digital scene in real-time based on the trainee's actions.

[0076] In one embodiment, step S200 includes:

[0077] Collect EEG signal data of trainees when they are performing operations under attentive conditions and when they are performing operations under non-attentive conditions.

[0078] In this embodiment, the deep learning model is trained by collecting electroencephalogram (EEG) signal data. Therefore, it is necessary to collect EEG signal data when the trainee is performing the operation in an attentive state (i.e., the operation requires the trainee to be highly focused); simultaneously, it is also necessary to collect EEG signal data when the trainee is performing the operation in a non-attentive state (i.e., the trainee needs to be relaxed and avoid concentrating as much as possible). By acquiring EEG signals in both states, the typical differences between attentive and non-attentive states can be clearly identified, providing a benchmark for subsequent analysis and judgment, and making the trained deep learning model more accurate.

[0079] In one embodiment, step S300 includes: constructing a zero-phase bandpass filter per target Hertz within a preset frequency domain range of the EEG signal data, and obtaining multiple sub-frequency band data after filtering; dividing each filtered EEG signal data into non-overlapping segments with a preset time window to obtain multiple preset time data segments; performing Hilbert transform on the preset time data segments in each sub-frequency band, and using the amplitude in the transformed sub-frequency band as a feature to construct a time-frequency feature map.

[0080] In this embodiment, the preset frequency range includes 1Hz-100Hz, the target Hertz is 2Hz, and the preset time is 10s. For the acquired EEG signal data within the 1Hz-100Hz frequency range, a zero-phase bandpass filter is constructed every 2Hz. Therefore, after filtering, 50 sub-bands of data are obtained, each containing EEG signal components within a specific frequency range. When the duration of each filtered EEG signal data is 60s, by dividing the 60s of EEG signal data into non-overlapping segments with 10s time windows, six 10s data segments are obtained for better data analysis and processing. After obtaining the six 10s data segments, amplitude detection is performed on each data segment, and data segments with amplitudes higher than 100μV are removed to reduce the influence of EMG signals and improve data quality and accuracy.

[0081] Hilbert transform is applied to the data segments within each sub-band to convert the EEG signal into an analytic signal, thereby obtaining the amplitude information of the EEG signal. This amplitude information is then used as a feature to construct a time-frequency feature map, providing information on the energy distribution and variations of the EEG signal data within a specific frequency band. Through fine decomposition, filtering, and feature extraction of the EEG signal, valuable information can be extracted from complex EEG signal data to better train deep learning models. The preset frequency domain range, target Hertz, and preset time are all exemplary and can be selected according to actual conditions without specific limitations.

[0082] In one embodiment, the process of performing Hilbert transform on the data segments of a preset time within each sub-band includes: obtaining the analytical signal of the EEG signal, calculated using the following formula:

[0083] The amplitude information of the EEG signal is calculated based on equation (2). The calculation formula is as follows:

[0084] Where z(t) is the analytic signal of the EEG signal; x(t) is the EEG signal; t and τ are time quantities; i∈{0,1,…}; φ(t) is the time window of the preset time; The Hilbert transform of the EEG signal x(t) is obtained by convolving it with the function h(t) = 1 / πt; pv is the Cauchy principal value; A(t) is the amplitude information of the EEG signal.

[0085] As shown in Figure 3, in one embodiment, step S400 includes:

[0086] Step S410: Connect the amplitude features of all sub-bands in the data segment of each preset time to obtain the time-frequency matrix feature map of the EEG signal data.

[0087] In step S410, the data segments at each preset time are filtered to obtain data from multiple sub-frequency bands, each with corresponding amplitude characteristics. All amplitude characteristics are then concatenated to form a time-frequency feature matrix, thus creating a time-frequency matrix feature map. Based on this time-frequency matrix feature map, the amplitude distribution of the EEG signal at different times and frequencies can be obtained, allowing us to understand the energy distribution of the EEG signal in different frequency bands and its changes over time.

[0088] Step S420: Preprocess the time-frequency matrix feature map.

[0089] Step S420 specifically includes: obtaining a time-frequency feature matrix based on the time-frequency matrix feature map; resampling the time-frequency feature matrix in the time dimension to obtain the time-frequency feature matrix of each preset time segment.

[0090] Specifically, since the 10-second data segment's length in the time dimension is much greater than the frequency dimension of 50 sub-bands, the obtained time-frequency feature matrix is ​​resampled in the time dimension to reduce network complexity. The time dimension in the time-frequency feature matrix is ​​compressed to reduce its length. The time-frequency feature matrix for each channel and each preset time segment obtained through resampling has a dimension of 50×100, where 50 represents the number of frequency bands and 100 represents the number of time points after resampling. Preprocessing the time-frequency matrix feature map reduces the dimensionality of the data segment while preserving frequency band information, thus reducing the number of network parameters and computational load, thereby reducing network complexity. Simultaneously, the resampled time-frequency feature matrix still reflects the amplitude distribution of the EEG signal at different times and frequencies.

[0091] Step S430: Build an Inception deep learning model for attention recognition based on the Inception structure, and combine it with the preprocessed time-frequency matrix feature map to establish a cross-subject attention model.

[0092] In step S430, the Inception structure is a neural network architecture component composed of multiple parallel branches, each using convolutional kernels or pooling operations of different sizes to extract features. By combining the outputs of these branches, feature information at different scales and levels can be captured. By designing a suitable network structure, the Inception structure is combined with convolutional layers and fully connected layers to form a complete Inception deep learning model. The preprocessed extracted time-frequency feature matrix is ​​then used as input data and fed into the constructed Inception deep learning model to build a cross-subject attention matrix model. By constructing a cross-subject attention model, attention patterns and differences between different subjects can be captured. By introducing an attention mechanism into the model, the model learns to pay attention to the features of different subjects, thereby improving the model's generalization ability and adaptability to individual differences.

[0093] Step S440: Input the time-frequency feature matrix from the preprocessed time-frequency matrix feature map into the cross-subject attention model to calculate the category of EEG signal.

[0094] As shown in Figures 4 and 5, step S440 specifically includes the following steps:

[0095] Step S441: Input the time-frequency feature matrix of the data segment for each preset time period into the cross-subject attention model;

[0096] Step S442: Based on the convolutional layer and max pooling layer in the cross-subject attention model, extract feature information from the time-frequency feature matrix of the data segment at each preset time.

[0097] Step S443: Input the extracted feature information into the fully connected layer;

[0098] Step S444: Calculate the feature information output by the fully connected layer based on the multi-classification algorithm to obtain the category of the EEG signal.

[0099] In this embodiment, the 50×100-dimensional time-frequency feature matrix of each preset time segment after preprocessing is used as the input to the cross-subject attention model. In the convolutional layer of the cross-subject attention model, 3×3 convolutions are used to capture local features in the time-frequency feature matrix and extract new information from it. The number of convolution kernels used in each layer of the convolutional layer is not exactly the same. After the input time-frequency feature matrix is ​​convolved with the convolution kernels, a new feature map is obtained, calculated as follows: Y p =f(Z) p )

[0100] Where p is the dimension of the pooling layer, with values ​​of 1, 2, ..., P; d has values ​​of 1, 2, ..., D; W p ∈R U×V×D This represents a three-dimensional convolution kernel with a depth D equal to the number of channels in the input feature X. d X 1 X 2 , ..., X D After performing convolution and summing the results, a bias term b is added to obtain the activation value Z of the convolutional layer. p Finally, after passing through the activation function, the output value Y of this layer is obtained. p For multiple convolution kernels, the final output Y is a mapping that matches the number of kernels. 1 Y 2 , ..., Y p .

[0101] The expression for the activation function is:

[0102] Where x is the input value.

[0103] Then, a max pooling layer is used to reduce the dimensionality of the feature information, thereby reducing information redundancy and preventing overfitting. The calculation formula is as follows:

[0104] in, The size of each pooling region; i takes the total number of elements within the region; x i The value of the node within the region. The output value is the result of dimensionality reduction; m and n represent the height and width of the input, respectively.

[0105] The max-pooled feature information is input into the fully connected layer, and all feature maps X∈R from the max-pooling layer output are mapped. M×N×D Tiling and unfolding yields a one-dimensional vector x = [x1; x2; ...; x...]. D Then, transform it using the following formula:

[0106] Where z is the output, d takes the value [1, 2, ..., D], and W T This represents the weights in the fully connected layer, with values ​​w = [w1; w2; ...; w...]. D ]∈R, x d 'b' is the input value, and 'b' is the bias term. The output of a fully connected layer is typically connected to activation layers, eventually yielding the neuron's output.

[0107] The multi-classification algorithm is used to calculate the feature information output by the fully connected layer to predict the multi-classification result, thereby obtaining the category of the EEG signal; specifically, the Softmax function can be used for calculation, and the calculation process includes:

[0108] The conditional probability of an EEG signal belonging to one of its categories is calculated using the following formula:

[0109] After calculating the conditional probability of each class in the EEG signal, the class with the highest probability in the EEG signal is obtained and used as the final prediction output. The calculation formula is as follows:

[0110] Where p(y=k|x) is the conditional probability of one category of EEG signal; y is the output; k is one category of EEG signal; x is the input feature information of EEG signal; and X is all feature information of EEG signal. These are the weight coefficients in the fully connected layer; The category with the highest probability in the electroencephalogram (EEG) signal; represents all weight coefficients in the fully connected layer; K represents all categories of EEG signals; b represents the bias term.

[0111] In other implementations, during the training of a deep learning model, the cross-entropy loss function can be used to optimize the deep learning model and learn its parameters, as expressed in:

[0112] Where n takes the value {0, 1, ..., N}; This represents the actual output of the sample. This represents the posterior probability of each class in the output of sample X(n). The magnitude of cross-entropy reflects how well the deep learning model fits the true distribution; the smaller the value, the closer the model prediction is to the true value. During model training, the number of data samples in each batch is set to 64, and the Adam (Adaptive Moment Estimation) optimizer is used to optimize the model.

[0113] In step S500, based on the established cross-subject attention model, when the trainee operates in the three-dimensional digital scene, the EEG brain signal status of the trainee is collected in real time. The cross-subject attention model is used to detect the attention state in real time, and the real-time output of attention is used as feedback. The node output value corresponding to the attention state is used as an indicator of the current attention level and displayed as a curve on the computer screen to provide feedback to the trainee. At the same time, the raw EEG brain signal data and attention output value at that moment are collected to facilitate further data analysis and calculation.

[0114] This application also provides a wind farm power equipment training device based on virtual reality technology. Figure 6 is a structural schematic diagram of an embodiment of the wind farm power equipment training device based on virtual reality technology provided in this application. As shown in Figure 6, the device includes:

[0115] The first construction module 100 is used to construct a three-dimensional digital scene based on the specific scenario of wind farm power equipment and facilities. The three-dimensional digital scene is updated in real time based on the operation of the trainees.

[0116] The acquisition module 200 is used to acquire brainwave signal data of trainees when they operate in a three-dimensional digital scene based on a VR device that combines brainwave signal acquisition.

[0117] The second construction module 300 is used to filter and extract features from the acquired EEG signal data and construct a time-frequency feature map;

[0118] Training module 400 is used to train deep learning models based on time-frequency feature maps;

[0119] The recognition module 500 is used to input the real-time EEG signal data of the trainees into the trained deep learning model, recognize the trainees' attention state, and output the trainees' attention state in real time.

[0120] In one embodiment, the acquisition module 200 includes:

[0121] The data acquisition unit is used to collect EEG signal data when trainees are performing operations under attentive conditions and when they are performing operations under non-attentive conditions.

[0122] In one embodiment, the second building module 300 includes:

[0123] The first building unit is used to build a zero-phase bandpass filter per target Hertz within a preset frequency domain range of EEG signal data, and obtain multiple sub-frequency band data after filtering.

[0124] The processing unit is used to divide each EEG signal data after filtering into non-overlapping segments within a preset time window to obtain multiple preset time data segments.

[0125] The transformation unit is used to perform Hilbert transformation on the data segments of a preset time in each sub-band, and to construct a time-frequency feature map using the amplitude of the transformed sub-band as a feature.

[0126] In one embodiment, the process of performing a Hilbert transform on the data segments at preset times within each sub-band includes:

[0127] The formula for obtaining the analytical signal of an electroencephalogram (EEG) signal is as follows:

[0128] The amplitude information of the EEG signal is calculated based on equation (2). The calculation formula is as follows:

[0129] Where z(t) is the analytic signal of the EEG signal; x(t) is the EEG signal; t and τ are time quantities; i∈{0,1,…}; φ(t) is the time window of the preset time; The Hilbert transform of the EEG signal x(t) is obtained by convolving it with the function h(t) = 1 / πt; pv is the Cauchy principal value; A(t) is the amplitude information of the EEG signal.

[0130] In one embodiment, the training module 400 includes:

[0131] The connection unit is used to connect the amplitude features of all sub-bands in each preset time data segment to obtain the time-frequency matrix feature map of the EEG signal data.

[0132] The preprocessing unit is used to preprocess the feature map of the time-frequency matrix;

[0133] The second building unit is used to build an Inception deep learning model for attention recognition based on the Inception structure, and to establish a cross-subject attention model by combining the preprocessed time-frequency matrix feature map.

[0134] The computational unit is used to input the time-frequency feature matrix from the preprocessed time-frequency matrix feature map into the cross-subject attention model to calculate the category of the EEG signal.

[0135] In one embodiment, the preprocessing unit includes:

[0136] Obtain sub-units to obtain the time-frequency feature matrix based on the time-frequency matrix feature map;

[0137] The resampling subunit is used to resample the time-frequency feature matrix in the time dimension to obtain the time-frequency feature matrix of each preset time segment.

[0138] In one embodiment, the computing unit includes:

[0139] The first input subunit is used to input the time-frequency feature matrix of each preset time segment into the cross-subject attention model;

[0140] Extraction sub-units are used to extract feature information from the time-frequency feature matrix of each preset time segment based on the convolutional layer and max pooling layer in the cross-subject attention model;

[0141] The second input subunit is used to input the extracted feature information into the fully connected layer;

[0142] The computational subunit is used to calculate the feature information output by the fully connected layer based on a multi-classification algorithm to obtain the category of the EEG signal. The formula for calculating the category of the EEG signal is:

[0143] The conditional probability of an EEG signal belonging to one of its categories is calculated using the following formula:

[0144] After calculating the conditional probability of each class in the EEG signal, the class with the highest probability in the EEG signal is obtained and used as the final prediction output. The calculation formula is as follows:

[0145] Where p(y=k|x) is the conditional probability of one category of EEG signal; y is the output; k is one category of EEG signal; x is the input feature information of EEG signal; and X is all feature information of EEG signal. These are the weight coefficients in the fully connected layer; The category with the highest probability in the electroencephalogram (EEG) signal; represents all weight coefficients in the fully connected layer; K represents all categories of EEG signals; b represents the bias term.

[0146] The apparatus and method embodiments in this application are based on the same application concept.

[0147] Figure 7 shows a schematic diagram of the structure of an embodiment of the electronic device provided in this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0148] As shown in Figure 7, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0149] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements, such as clients or other servers. Processor 502 executes program 510, specifically performing the relevant steps described in the embodiment of the training method for wind farm power equipment based on virtual reality technology.

[0150] Specifically, program 510 may include program code, which includes computer-executable instructions.

[0151] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0152] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0153] Specifically, program 510 can be called by processor 502 to cause electronic devices to execute the relevant steps in the above embodiment of the training method for wind farm power equipment based on virtual reality technology.

[0154] Those skilled in the art will understand that the structure shown in FIG7 is merely illustrative and does not limit the structure of the device described above. For example, the electronic device may include more or fewer components than shown in FIG7, or have a different configuration than that shown in FIG7.

[0155] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed on an electronic device or a wind farm power equipment training device based on virtual reality technology, the electronic device or the wind farm power equipment training device based on virtual reality technology performs the wind farm power equipment training method based on virtual reality technology in any of the above method embodiments.

[0156] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments in this application are not directed to any particular programming language.

[0157] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. Similarly, for the purpose of simplification and aiding understanding of one or more aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0158] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0159] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A wind farm power equipment training method based on virtual reality technology, characterized in that, The method comprises: constructing a three-dimensional digital scene based on the specific scene of the wind farm power equipment facility, the three-dimensional digital scene being updated in real time based on the operation of the training personnel; acquiring electroencephalogram data of the training personnel when operating in the three-dimensional digital scene based on a VR device combined with electroencephalogram signal acquisition; filtering and extracting features of the acquired electroencephalogram data, and constructing a time-frequency feature map; training a deep learning model based on the time-frequency feature map; inputting the real-time electroencephalogram data of the training personnel into the trained deep learning model, identifying the attention state of the training personnel, and outputting the attention state of the training personnel in real time.

2. The wind farm power plant training method based on virtual reality technology according to claim 1, characterized in that, The method comprises: acquiring electroencephalogram data of the training personnel when operating in the three-dimensional digital scene based on a VR device combined with electroencephalogram signal acquisition; 3.The wind farm power equipment training method based on virtual reality technology according to claim 1, characterized in that, acquiring electroencephalogram data of the training personnel when operating in the three-dimensional digital scene based on a VR device combined with electroencephalogram signal acquisition. The method comprises: filtering and extracting features of the acquired electroencephalogram data, and constructing a time-frequency feature map; constructing a zero-phase band-pass filter in each target hertz in a preset frequency domain range of the electroencephalogram data, and obtaining a plurality of sub-band data after filtering processing; 4. The wind farm power equipment training method based on virtual reality technology according to claim 3, characterized in that, non-overlappingly dividing each of the electroencephalogram data after filtering processing into a time window of a preset time to obtain a plurality of data segments of the preset time; An analytical signal of the electroencephalogram signal is obtained, and the calculation formula is: The amplitude information of the electroencephalogram signal is calculated based on formula (2), and the calculation formula is: Wherein, z(t) is an analytic signal of the electroencephalogram signal; x(t) is the electroencephalogram signal; t, τ are time quantities; i∈{0, 1, …}; φ(t) is a preset time window of time; performing Hilbert transform on each data segment of the preset time in each sub-band, and constructing a time-frequency feature map by taking the amplitude of the transformed sub-band as a feature.

5. The wind farm power equipment training method based on virtual reality technology according to claim 3, characterized in that, The calculation process of performing Hilbert transform on each data segment of the preset time in each sub-band comprises: the Hilbert transform of the electroencephalogram signal x(t) is obtained by convolution with the function h(t) = 1 / πt; p.v. is the Cauchy principal value; A(t) is the amplitude information of the electroencephalogram signal. The method comprises: concatenating the amplitude features of all sub-bands in each data segment of the preset time to obtain a time-frequency matrix feature map of the electroencephalogram data; preprocessing the time-frequency matrix feature map; building an Inception deep learning model for attention recognition based on the Inception structure, and establishing a cross-subject attention model combined with the preprocessed time-frequency matrix feature map; 6. The wind farm power equipment training method based on virtual reality technology according to claim 5, characterized in that, inputting the time-frequency feature matrix in the preprocessed time-frequency matrix feature map into the cross-subject attention model to calculate the category of the electroencephalogram signal. The method comprises: obtaining a time-frequency feature matrix based on the time-frequency matrix feature map; 7. The wind farm power plant training method based on virtual reality technology according to claim 6, characterized in that, resampling the time-frequency feature matrix in the time dimension to obtain a time-frequency feature matrix of each data segment of the preset time. The method comprises: inputting the time-frequency feature matrix of each data segment of the preset time into the cross-subject attention model; extracting feature information of the time-frequency feature matrix of each data segment of the preset time based on the convolution layer and the maximum pooling layer in the cross-subject attention model; inputting the extracted feature information into a fully connected layer; The feature information output by the full connection layer is calculated based on a multi-classification algorithm to obtain a class of the electroencephalogram signal. 8.The wind farm power equipment training method based on virtual reality technology according to claim 5, characterized in that, A calculation formula for calculating the class of the electroencephalogram signal is: The conditional probability belonging to one category of the brain electrical signal is calculated, and the calculation formula is: After the conditional probability of each class of the brain electrical signal is calculated, the class with the maximum probability in the brain electrical signal is obtained and used as the final prediction output, and the calculation formula is: wherein p(y=k|x) is a conditional probability of the electroencephalogram signal; y is an output; k is one of the electroencephalogram signal categories; x is input feature information of the electroencephalogram signal; and X is all feature information of the electroencephalogram signal. for the weight coefficients in the fully connected layer; the class of maximum probability in the electroencephalogram signal; is an all weight coefficient in the full connection layer; K is all classes of the electroencephalogram signal; and b is a bias term.

9. A wind farm power equipment training device based on virtual reality technology, characterized in that, The device comprises: A first construction module configured to construct a three-dimensional digital scene based on a specific scene of a wind farm power equipment facility, the three-dimensional digital scene being updated in real time based on an operation of a training personnel; A collection module configured to collect electroencephalogram data of the training personnel when operating in the three-dimensional digital scene based on a VR device combined with electroencephalogram signal collection; A second construction module configured to filter and extract features of the collected electroencephalogram data and construct a time-frequency feature map; A training module configured to train a deep learning model based on the time-frequency feature map; An identification module configured to input real-time electroencephalogram data of the training personnel into the trained deep learning model, identify an attention state of the training personnel, and output the attention state of the training personnel in real time.

10. A computer-readable storage medium, characterized in that, The storage medium stores executable instructions, and when the executable instructions are executed on an electronic device or a wind farm power equipment training device based on virtual reality technology, the electronic device or the wind farm power equipment training device based on virtual reality technology performs the operations of the wind farm power equipment training method based on virtual reality technology as claimed in any one of claims 1-8.

Images