Collaboration and knowledge transfer method and system based on wind power model

Abstract

The invention discloses a cooperation and knowledge transmission method and system based on a wind power model, and the method comprises the steps: obtaining a multi-modal data set in the field of wind power, carrying out the data preprocessing, obtaining a preprocessed multi-modal data set, carrying out the pre-training of a cloud multi-modal large model, obtaining a trained cloud multi-modal large model, carrying out the consistency evaluation, and obtaining the consistency of the cloud multi-modal large model. A fine tuning layer is adopted to perform fine tuning on the trained cloud multi-modal large model to obtain a fine-tuned cloud multi-modal large model, and after knowledge transmission is performed by using bidirectional knowledge distillation, a data updating mechanism and an online learning algorithm are executed to complete collaboration and knowledge transmission of the wind power model; through large and small model end cloud co-evolution, the defects of a cloud side centralized learning normal form in the aspects of expandability, real-time performance, individuation, load cost, privacy security and the like can be effectively overcome, the performance of a small model is improved, and it is ensured that a large model can adapt to a changing working condition environment in real time; therefore, the monitoring, maintenance and management efficiency of the wind power equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a wind power model-based collaboration and knowledge transfer method and system. Background Art

[0002] As the global demand for renewable energy grows, the wind power industry has entered a period of rapid development. China has actively promoted the development of the wind power industry through the formulation of national strategies and has achieved remarkable results. However, the efficient and reliable operation of wind power equipment still faces multiple challenges. Due to the complex structure of wind power equipment and the changeable operating environment, equipment failures occur frequently. Traditional detection methods are difficult to accurately identify potential defects, and traditional life prediction methods are also limited in accuracy and reliability. The rapid development of the wind power industry has put forward higher standards for equipment monitoring, maintenance and management. Traditional single models are difficult to cope with the needs of multiple tasks and multiple scenarios. Therefore, the large and small model collaborative strategy of hybrid expert models has become a possible solution. Under this strategy, the large model provides model capabilities to edge devices and terminals, and the small model is responsible for the actual reasoning and execution tasks. At the same time, the small model will feedback the effect of algorithm execution to the large model to form a closed-loop optimization mechanism.

[0003] In practical applications, due to the difference in computing power requirements, a cloud-edge architecture model is usually used to deploy large and small models, that is, deploying a general large model on the cloud and multiple dedicated small models on the edge. Although the collaboration of large and small models based on the cloud-edge architecture shows great potential, there are still a series of technical challenges in practice.

[0004] Cooperative knowledge transfer method of large and small models based on federated learning In the cloud-edge architecture, the edge side contains multiple edge nodes, and the dedicated small models deployed on each node are divided into parameter providing clients and parameter receiving clients. The parameter providing client uses its own computing power to use the edge side data for local training, obtains the updated model and parameters, and then uploads them to the cloud aggregation server; the parameter receiving client requests model parameters from the cloud aggregation server. The cloud aggregation server collects the updated models and parameters of each edge node, aggregates them to generate global model parameters, and then sends them back to the edge node for small model update. In order to improve the training efficiency of federated learning, the scheme also introduces a reward mechanism. However, this method is prone to parameter data loss and data version problems, and does not solve the problem of unbalanced resource load between the cloud and edge nodes. There is a lack of effective data synchronization mechanisms, such as incremental synchronization, timestamp synchronization, etc., which leads to untimely or inconsistent data updates. The large and small model co-evolution method based on weakly supervised learning and its terminal first use the existing large model to perform preliminary detection of the task, and then confirm the quality of the detection results through a one-time judgment. If the quality of the test result is high, a supervision model (i.e., small model) is constructed based on the result to replace the first judgment for automatic detection of subsequent similar tasks; if the quality of the result is low, a second annotation is introduced to correct the error, and then the supervision model is constructed. The entire process only requires two manual interventions (first judgment and second annotation), and the subsequent task detection is automatically completed by the collaboration of large and small models without the need for further manual intervention, which ensures the quality of detection while effectively reducing manual intervention. However, this method relies on manual intervention in the early stage, and only realizes one-way collaboration from large models to small models, ignoring the data synchronization mechanism between large and small models. In particular, with the changes in time and environment, the real-time data stream for executing small model reasoning on edge devices will change, while the data based on which the large model training in the cloud is based cannot be updated in time, which results in the cloud model being unable to adapt to changes in edge data in a timely manner, thereby affecting the accuracy of large model reasoning.

[0005] To sum up, although the existing technical solutions have their own advantages, they also expose problems such as data synchronization, uneven resource allocation, and lack of adaptability. More complete solutions are urgently needed to overcome these challenges. Summary of the invention

[0006] The purpose of this application is to overcome the existing technical defects and provide a method and system for collaboration and knowledge transfer based on a wind power model. Through the collaborative evolution of large and small models on the cloud side, the shortcomings of the centralized learning paradigm on the cloud side in terms of scalability, real-time performance, personalization, load cost, privacy and security can be effectively overcome.

[0007] The purpose of this application is achieved through the following technical solutions:

[0008] In a first aspect, the present application proposes a collaboration and knowledge transfer method based on a wind power model, the method comprising:

[0009] Acquire a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set;

[0010] The preprocessed multimodal data set is used to pre-train the cloud-based multimodal large model to obtain a trained cloud-based multimodal large model;

[0011] Pre-train the local data set corresponding to the task through the deep learning model to obtain multiple scenario-based professional small models;

[0012] Based on the same input, the output results of the trained cloud-based multimodal large model and the scenario-based professional small model are respectively evaluated for consistency. When the difference in the consistency evaluation is greater than a preset threshold, the trained cloud-based multimodal large model is fine-tuned using a fine-tuning layer to obtain a fine-tuned cloud-based multimodal large model.

[0013] After using two-way knowledge distillation to transfer knowledge between the fine-tuned cloud-based multimodal large model and the scenario-based professional small model, the data update mechanism and online learning algorithm are executed to complete the coordination and knowledge transfer of the wind power model.

[0014] In a possible implementation manner, the multimodal data set includes a fault text data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set includes:

[0015] Perform text cleaning on the fault text dataset, and then perform word segmentation to obtain the fault text dataset after word segmentation;

[0016] The Bert model is used to extract features from the fault text dataset after word segmentation to obtain the fault text feature vector.

[0017] In a possible implementation, the multimodal data set includes a vibration data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set further includes:

[0018] Denoising and normalizing the vibration data set to obtain a denoised and normalized vibration data set;

[0019] The denoised and normalized vibration data set is subjected to feature extraction to obtain time domain features and frequency domain features respectively;

[0020] The time domain features and frequency domain features are combined into a vibration feature vector.

[0021] In a possible implementation, the multimodal dataset includes a voiceprint dataset, and the step of performing data preprocessing on the multimodal dataset to obtain a preprocessed multimodal dataset further includes:

[0022] The voiceprint dataset is subjected to denoising by Wiener filtering, and then subjected to time normalization and amplitude normalization to obtain a normalized voiceprint dataset.

[0023] The Mel-frequency cepstral coefficients, formant and linear prediction coding are extracted from the normalized voiceprint dataset to form a voiceprint feature vector.

[0024] In a possible implementation, the multimodal dataset includes an image dataset, and the step of performing data preprocessing on the multimodal dataset to obtain a preprocessed multimodal dataset further includes:

[0025] The image enhancement technology is used to perform image enhancement on the image data set to obtain an image enhanced image data set;

[0026] The convolutional neural network is used to extract the features of the image dataset after image enhancement to obtain geometric features, shape features, color features and HOG features to form the image feature vector.

[0027] In a possible implementation, the step of using the preprocessed multimodal data set to pre-train the cloud multimodal large model to obtain the trained cloud multimodal large model includes:

[0028] The preprocessed multimodal data set is input into the constructed shared feature extractor to extract common features and obtain common features;

[0029] Use the intra-modal self-attention mechanism and transformer to perform feature interaction on common features to form a multi-modal representation of common features;

[0030] Based on the multimodal representation of common features and the defined loss function, the preprocessed multimodal dataset is input into the cloud-based multimodal big model for pre-training to obtain the trained cloud-based multimodal big model.

[0031] In a possible implementation, the fine-tuning layer is arranged after the output layer of the large model and adopts a parameter-efficient fine-tuning technique, and the fine-tuning strategy of the fine-tuning layer includes a learning rate, training rounds, and a batch size.

[0032] In one possible implementation, the data update mechanism includes periodic data update, data consistency assessment, and model parameter adjustment;

[0033] In data consistency assessment, the sum of squared differences between the cloud and edge data states is calculated. When the sum of squared differences is 0, it means the data is consistent. When the sum of squared differences is not 0, it means there is a difference in the data. The calculation formula for the sum of squared differences is: Data cloud,i It is the cloud data status. edge,i is the edge data state, and n is the total number of data items.

[0034] In a possible implementation, the online learning algorithm uses a trained multimodal large model as the initial model, continuously receives real-time data from the edge, calculates gradients based on data points, and updates model weights in real time based on the gradients. The calculation formula for the model weights is: ω t is the weight model at time t, η is the learning rate, L is the loss function, y t is the true label, is the model’s prediction at time t, ω t+1 is the weight model at time t+1, To represent ω t The gradient at time t.

[0035] In the second aspect, the present application proposes a wind power model-based collaboration and knowledge transfer system, the system comprising:

[0036] A preprocessing module, used to obtain a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set;

[0037] A first pre-training module is used to pre-train the cloud-based multimodal large model using the pre-processed multimodal data set to obtain a trained cloud-based multimodal large model;

[0038] The second pre-training module is used to pre-train the local data set corresponding to the task through a deep learning model to obtain multiple scene-based professional small models;

[0039] The consistency evaluation module is used to perform consistency evaluation on the output results of the trained cloud-based multimodal large model and the scenario-based professional small model based on the same input. When the difference in the consistency evaluation is greater than a preset threshold, the fine-tuning layer is used to fine-tune the trained cloud-based multimodal large model to obtain a fine-tuned cloud-based multimodal large model.

[0040] The knowledge distillation module is used to transfer knowledge between the fine-tuned cloud-based multimodal large model and the professional small model based on specific scenarios using two-way knowledge distillation, and then executes the data update mechanism and online learning algorithm to complete the coordination and knowledge transfer of the wind power model.

[0041] The above-mentioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0042] The present application discloses a method and system for collaboration and knowledge transfer based on a wind power model, which obtains a multimodal data set in the wind power field and performs data preprocessing to obtain a preprocessed multimodal data set, pretrains a large cloud-based multimodal model to obtain a trained large cloud-based multimodal model, and then performs a consistency evaluation. A fine-tuning layer is used to fine-tune the trained large cloud-based multimodal model to obtain a fine-tuned large cloud-based multimodal model. After using two-way knowledge distillation for knowledge transfer, a data update mechanism and an online learning algorithm are executed to complete the collaboration and knowledge transfer of the wind power model. The end-cloud collaborative evolution of large and small models can effectively overcome the deficiencies of the centralized learning paradigm on the cloud side in terms of scalability, real-time performance, personalization, load cost, privacy and security, etc., which not only improves the performance of the small model, but also ensures that the large model can adapt to the changing working environment in real time, thereby improving the monitoring, maintenance and management efficiency of wind power equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of a wind power model-based collaboration and knowledge transfer method proposed in an embodiment of the present application is shown.

[0045] Figure 2 A schematic diagram of a shared feature extractor proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0048] In the existing technology, due to problems in data synchronization and resource load, for example, due to multiple edge nodes transmitting parameter data to the cloud, data loss and version inconsistency are prone to occur, and there is a lack of effective data synchronization mechanism, resulting in untimely or inconsistent data updates, or manual intervention is required, only one-way collaboration is achieved, and cloud data cannot be updated in time to adapt to edge data changes, resulting in reduced model reasoning accuracy. These problems seriously affect the overall performance and reliability of the system.

[0049] Therefore, in order to solve the above technical problems, the embodiment of the present application proposes a method and system for collaboration and knowledge transfer based on a wind power model, and its application scenarios include defect detection, fault diagnosis, life prediction and intelligent scheduling, etc., and solves the deficiencies of traditional cloud-side centralized learning in scalability, real-time, personalization, load cost, privacy and security through the collaborative evolution of the end cloud. By rationally utilizing the distributed computing resources on the cloud and the terminal, and jointly optimizing with the communication and storage resources, the efficient operation of the system is ensured. By designing and training a shared feature extractor, the consistency of feature extraction between the large model and the small model when processing the same input data is achieved. The feature task fine-tuning method of multi-task collaboration is studied, and the shared feature extractor is fine-tuned in a targeted manner to improve the performance of the model on specific tasks. Through the knowledge distillation technology, the small model can learn key knowledge from the large model, improve its ability to process tasks locally, and design a collaborative workflow between large and small models to ensure the effective transfer and utilization of data and knowledge, and ultimately improve the monitoring, maintenance and management efficiency of wind power equipment, and realize smarter and more efficient wind power system operation, which will be described in detail below.

[0050] Please refer to Figure 1 , Figure 1 A schematic flow chart of a wind power model-based collaboration and knowledge transfer method proposed in an embodiment of the present application is shown, and the method includes the following steps:

[0051] Step S1: Acquire a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set.

[0052] First, based on the fault knowledge base data accumulated by the wind power company in the early stage, a wind power fault text dataset with i samples is obtained, which is represented as D = {D1, D2, ..., D i}, and obtain a vibration data set A of m samples by connecting to the vibration sensor. 2, ...,A m}、Connect the microphone built into the tower to obtain a voiceprint dataset of n samples B={B1,B2,...,B n} and the image dataset C = {C1, C2, ..., C o}, then integrate the vibration dataset, voiceprint dataset, image dataset and text dataset into a multimodal dataset, perform data preprocessing on the multimodal dataset to obtain a preprocessed multimodal dataset. The data preprocessing steps may include but are not limited to data cleaning, format conversion, feature extraction, normalization, etc. to ensure the quality and consistency of the data, and then perform model training.

[0053] The multimodal data set includes a fault text data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set includes:

[0054] Perform text cleaning on the fault text dataset, and then perform word segmentation to obtain the fault text dataset after word segmentation;

[0055] The Bert model is used to extract features from the fault text dataset after word segmentation to obtain the fault text feature vector.

[0056] Data preprocessing is performed on the multimodal dataset. First, text cleaning is performed to remove special characters and punctuation marks, and non-alphanumeric characters in the text, such as punctuation marks, line breaks, etc. Then, the identical text records in the dataset are deleted to avoid redundancy, and information irrelevant to the fault or low-quality text records are removed to ensure the purity of the dataset.

[0057] After that, the word segmentation is performed to divide the cleaned text data into words or phrases, and the sample size of the text dataset may be reduced. The Bert model is used to extract features from the segmented fault text dataset to obtain a vector representation of each text sample. The Bert model captures the semantic information of the text through a deep neural network and generates a high-dimensional feature vector. The preprocessed and feature-extracted fault text dataset is integrated with other modal datasets (such as vibration datasets, voiceprint datasets, and image datasets) to form the final preprocessed multimodal dataset.

[0058] The multimodal data set includes a vibration data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set also includes:

[0059] Denoising and normalizing the vibration data set to obtain a denoised and normalized vibration data set;

[0060] The denoised and normalized vibration data set is subjected to feature extraction to obtain time domain features and frequency domain features respectively;

[0061] The time domain features and frequency domain features are combined into a vibration feature vector.

[0062] The multimodal data set vibration data set is preprocessed. First, the vibration data set is denoised to remove the noise in the signal. The denoised vibration data is normalized to make the data have a uniform scale. It can be normalized by subtracting the mean and dividing by the standard deviation. Then, the average value of the vibration signal, the standard deviation of the vibration signal, the maximum value of the vibration signal, the kurtosis of the vibration signal, the skewness of the vibration signal, the power spectral density of the frequency domain signal, and the spectral entropy of the frequency domain signal are calculated. The time domain signal is converted into a frequency domain signal. The extracted time domain features and frequency domain features are combined into a feature vector to represent the multidimensional features of each vibration sample. Finally, the vibration data set that has been preprocessed and feature extracted is integrated with other modal data sets (text data set, voiceprint data set, image data set) to form the final preprocessed multimodal data set.

[0063] The multimodal dataset includes a voiceprint dataset, and the step of preprocessing the multimodal dataset to obtain a preprocessed multimodal dataset also includes:

[0064] The voiceprint dataset is subjected to denoising by Wiener filtering, and then subjected to time normalization and amplitude normalization to obtain a normalized voiceprint dataset.

[0065] The Mel-frequency cepstral coefficients, formant and linear prediction coding are extracted from the normalized voiceprint dataset to form a voiceprint feature vector.

[0066] For multimodal data sets, data preprocessing is performed. First, the Wiener filter technology is used to remove noise from the voiceprint data, improve the clarity of the signal, and standardize the time length of the voiceprint data to make it have a unified time scale. This can be achieved by truncating or padding the data, standardizing the amplitude value of the voiceprint data to make it within a certain range (between 0 and 1). The noise of the voiceprint data set is effectively removed, and both time and amplitude are standardized.

[0067] Then, Mel-frequency cepstral coefficients (MFCC), formants, and linear predictive coding (LPC) are extracted, and the extracted MFCC, formant, and LPC features are combined into a feature vector to represent the multi-dimensional features of each voiceprint sample. Finally, the preprocessed and feature-extracted voiceprint dataset is integrated with other modal datasets (such as text datasets, vibration datasets, and image datasets) to form the final preprocessed multimodal dataset.

[0068] The multimodal dataset includes an image dataset, and the step of performing data preprocessing on the multimodal dataset to obtain a preprocessed multimodal dataset also includes:

[0069] The image enhancement technology is used to perform image enhancement on the image data set to obtain an image enhanced image data set;

[0070] The convolutional neural network is used to extract the features of the image dataset after image enhancement to obtain geometric features, shape features, color features and HOG features to form the image feature vector.

[0071] The image dataset of the multimodal dataset is preprocessed. The image dataset is first enhanced to increase the diversity and robustness of the data. Image enhancement techniques include rotation, translation, scaling, flipping, brightness adjustment, contrast adjustment, etc. After image enhancement, the image enhanced dataset is subjected to feature extraction using a convolutional neural network to extract geometric features, shape features, color features, HOG features, etc. The extracted geometric features, shape features, color features, and HOG features are combined into a feature vector to represent the multidimensional features of each image sample. The preprocessed and feature-extracted image dataset is integrated with other modal datasets (such as text datasets, vibration datasets, and voiceprint datasets) to form the final preprocessed multimodal dataset.

[0072] Step S2: pre-train the cloud-based multimodal large model using the pre-processed multimodal data set to obtain a trained cloud-based multimodal large model.

[0073] Through the preprocessed multimodal datasets (including vibration datasets, voiceprint datasets, image datasets, and text datasets), denoising, normalization, and feature extraction are first performed to ensure data quality and consistency. Then, a deep neural network based on the Transformer architecture is designed as a shared feature extractor. The preprocessed multimodal data is input into the extractor. Through the intra-modal self-attention mechanism and the Transformer encoder layer, the features of different modalities are extracted and fused to generate common feature representations. The maximum mean difference (MMD) is used as the loss function to measure the distribution differences of the common features obtained after different modal data pass through the shared feature extractor, and the feature representations of different modalities are made more similar by minimizing the MMD loss. Finally, the model parameters are adjusted through optimization algorithms such as back propagation and gradient descent. After multiple iterative training, a pre-trained cloud-based multimodal large model with good performance on multimodal data is finally obtained.

[0074] Step S2 includes:

[0075] The preprocessed multimodal data set is input into the constructed shared feature extractor to extract common features and obtain common features;

[0076] Use the intra-modal self-attention mechanism and transformer to perform feature interaction on common features to form a multi-modal representation of common features;

[0077] Based on the multimodal representation of common features and the defined loss function, the preprocessed multimodal dataset is input into the cloud-based multimodal big model for pre-training to obtain the trained cloud-based multimodal big model.

[0078] Figure 2 A schematic diagram of a shared feature extractor proposed in an embodiment of the present application is shown, which extracts shared features from multiple modal data such as text, vibration, voiceprint, and vision. The data of these modalities are first converted into vector representations through their respective encoders (such as Word2Vec, Vibration2Vec, Voiceprint2Vec, and Vision2Vec), and then processed through a self-attention mechanism, so that the model can pay more attention to the important parts of the input sequence. The processed features will pass through multiple Transformer layers containing multi-head self-attention and feedforward neural networks to capture complex patterns and relationships. Ultimately, the features of different modalities are fused into a shared feature representation after being processed by multiple Transformer layers.

[0079] A deep neural network based on the Transformer architecture is designed as a shared feature extractor. The network can process preprocessed multimodal data and extract common features. It receives preprocessed multimodal data, including vibration datasets, voiceprint datasets, image datasets, and text datasets. The data of each modality is first captured through the intra-modal self-attention mechanism to capture the feature interactions within each modality. Through multiple Transformer encoder layers, the features between different modalities are further extracted and fused to generate common feature representations.

[0080] Feature interaction and multimodal representation use the intra-modal self-attention mechanism and Transformer to interact with common features and form a multimodal representation of common features. The data of each modality captures the feature dependencies within the modality through the self-attention mechanism. Through the Transformer encoder layer, the features of different modalities interact to generate a comprehensive multimodal feature representation.

[0081] Define the loss function to measure the similarity of the common features of different modalities obtained by the shared feature extractor of the source domain data and the target domain data. Use the Maximum Mean Discrepancy (MMD) to measure the distribution difference of the common features obtained by the shared feature extractor of different modal data. By minimizing MMD, the feature representations of different modalities can be made more similar, thereby improving the performance of the multimodal model. The specific formula of the loss function is:

[0082] Known preprocessed vibration data set The corresponding distribution is K;

[0083] Known preprocessed voiceprint dataset Its corresponding distribution is O;

[0084] Known preprocessed image dataset The corresponding distribution is P;

[0085] Known preprocessed text dataset The corresponding distribution is Q;

[0086] in Represents the mapping to the Reproducing Kernel Hilbert Space (RKHS), H represents the Reproducing Kernel Hilbert Space. m represents the vibration data set after preprocessing The number of; n represents the preprocessed voiceprint dataset The number of; O represents the preprocessed image dataset The number of; q represents the preprocessed text dataset The number of . Represents the preprocessed vibration dataset The i-th data; Represents the preprocessed voiceprint dataset The jth data of Represents the preprocessed image dataset The oth data; Represents the preprocessed text dataset The qth data of .

[0087] Based on the defined modal loss function and common feature extractor, the preprocessed multimodal dataset is used for pre-training to obtain a pre-trained cloud-based multimodal large model. The preprocessed vibration dataset, voiceprint dataset, image dataset and text dataset are first input into the shared feature extractor, and the common features are extracted through the shared feature extractor. The intra-modal self-attention mechanism and Transformer are used to perform feature interaction on the common features to form a multimodal representation of the common features. The MMD loss is calculated to measure the distribution differences of features of different modalities. By minimizing the MMD loss, the model parameters are optimized to make the feature representations of different modalities more similar, thereby improving the performance of the multimodal model. After multiple iterative training, the pre-trained cloud-based multimodal large model is finally obtained.

[0088] Step S3: Pre-train the local data set corresponding to the task through a deep learning model to obtain multiple scene-based professional small models.

[0089] In order to target specific tasks in different scenarios (such as defect detection, fault diagnosis, life prediction, and intelligent scheduling), local data sets corresponding to the tasks are obtained and pre-trained through deep learning models to obtain multiple small professional models based on specific scenarios.

[0090] First, collect and organize the local data sets required for each specific task, such as defect detection data sets, fault diagnosis data sets, life prediction data sets, and intelligent scheduling data sets. Preprocess the acquired local data sets, including cleaning, normalization, and feature extraction, to ensure the quality and consistency of the data. Select a suitable deep learning model for each specific task, such as convolutional neural networks for defect detection of image data, recurrent neural networks or long short-term memory networks for fault diagnosis and life prediction of time series data, and graph neural networks for intelligent scheduling. Input the preprocessed local data sets into the selected deep learning model for pre-training, and adjust the model parameters through optimization algorithms such as back propagation and gradient descent to make it perform well on specific tasks. During the pre-training process, regularly evaluate the performance of the model on the validation set, optimize the model based on the evaluation results, ensure the accuracy and robustness of the model on specific tasks, and after multiple iterations of training and optimization, finally obtain multiple professional small models based on specific scenarios, which have high performance and reliability on their respective specific tasks.

[0091] Step S4: Based on the same input, the output results of the trained cloud-based multimodal large model and the scenario-based professional small model are respectively evaluated for consistency. When the difference in the consistency evaluation is greater than a preset threshold, the trained cloud-based multimodal large model is fine-tuned using a fine-tuning layer to obtain a fine-tuned cloud-based multimodal large model.

[0092] Consistency evaluation and fine-tuning process In order to ensure the consistency of the output results of the cloud-based multimodal large model and the professional small model based on specific scenarios under the same input, and to fine-tune the large model when necessary, the multimodal large model and several professional small models based on specific scenarios are first trained using the same input data. The multimodal large model trains and verifies the shared feature extractor based on the multimodal representation of the common features that have been formed. The output results of the multimodal large model and the professional small model on the same task are compared and analyzed and consistency evaluated. The consistency between the models is evaluated by calculating the differences between the output results (such as mean square error, accuracy and other indicators).

[0093] If the consistency evaluation difference of the output results is greater than the preset threshold, the fine-tuning stage is entered; otherwise, the model has met the requirements and no further fine-tuning is required. When the consistency evaluation difference is large, the multimodal large model is fine-tuned using the parameter efficient fine-tuning method. According to different business scenarios (such as defect detection, fault diagnosis, life prediction and intelligent scheduling), a task-specific large model fine-tuning layer is designed. A new classification layer is added after the output layer of the large model to guide the large model to respond to specific tasks.

[0094] The fine-tuning layer is set after the output layer of the large model and adopts parameter-efficient fine-tuning technology. The fine-tuning strategies of the fine-tuning layer include learning rate, training rounds, and batch size.

[0095] Set fine-tuning strategies, including hyperparameters such as learning rate, training rounds (epochs), batch size, etc., and use parameter-efficient fine-tuning techniques to adapt to the needs of specific tasks by updating only some parameters in the model. Select appropriate learning rates, training rounds, and batch sizes, train the fine-tuning layer using task-specific datasets, and only update the parameters of the fine-tuning layer, keeping most of the parameters of the large model unchanged. During the training process, regularly evaluate the performance of the model on the validation set, and make necessary adjustments and optimizations based on the evaluation results.

[0096] Step S4 can ensure that the output results of the cloud-based multimodal large model on specific tasks are consistent with those of the professional small model. At the same time, by introducing the fine-tuning layer, the performance and adaptability of the large model on specific tasks can be improved.

[0097] Step S5: After the fine-tuned cloud-based multimodal large model and the scenario-based professional small model are transferred using two-way knowledge distillation, the data update mechanism and online learning algorithm are executed to complete the coordination and knowledge transfer of the wind power model.

[0098] In order to achieve the collaboration and knowledge transfer between the fine-tuned cloud multimodal large model and the professional small model based on specific scenarios, and ensure that the model can adapt to the dynamically changing data environment. First, two-way knowledge distillation is performed, and the large model knowledge distillation method is used to migrate the common knowledge in the large model to the small model. The output of the large model is used as a soft label to guide the training of the small model, so that the small model can learn the advanced features and decision logic of the large model. Through knowledge distillation, the domain adaptability and decision accuracy of the small model are improved, ensuring that the performance of the small model on specific tasks is close to that of the large model.

[0099] The data update mechanism includes periodic data update, data consistency assessment and model parameter adjustment;

[0100] In data consistency assessment, the sum of squared differences between the cloud and edge data states is calculated. When the sum of squared differences is 0, it means the data is consistent. When the sum of squared differences is not 0, it means there is a difference in the data. The calculation formula for the sum of squared differences is: Data cloud,i It is the cloud data status. edge,i is the edge data state, and n is the total number of data items.

[0101] Establish a periodic data update mechanism for the large model and the small model, regularly adjust the parameters of the large model according to the new data, ensure that the model can adapt to the changing working environment and task requirements, regularly collect new data from the edge, update the training data set of the cloud large model, calculate the sum of squares of the difference between the cloud and edge data states, and evaluate data consistency. According to the data consistency evaluation results, adjust the parameters of the large model to ensure that the model can adapt to the new data environment.

[0102] The online learning algorithm uses a trained multimodal large model as the initial model, continuously receives real-time data from the edge, calculates the gradient based on the data points, and updates the model weight in real time based on the gradient. The calculation formula for the model weight is: ω t is the weight model at time t, η is the learning rate, L is the loss function, y t is the true label, is the model’s prediction at time t, ω t+1 is the weight model at time t+1, To represent ω t The gradient at time t.

[0103] By adopting online learning algorithms, the multimodal large model can update itself according to the real-time data stream. First, the trained multimodal large model is used as the initial model, and real-time data from the edge is continuously received. The gradient is calculated according to each data point, and the model weights are updated in real time according to the gradient. Finally, the fine-tuned cloud-based multimodal large model and the professional small model based on specific scenarios can achieve collaboration and knowledge transfer, ensuring that the model can adapt to the dynamically changing data environment and improve the robustness and real-time performance of the model.

[0104] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0105] First, by embedding professional knowledge of different business scenarios in the big model and using the knowledge distillation method of the big model to transfer this knowledge to the small model, the performance and accuracy of the results of the professional small model deployed on the edge in business scenarios such as defect detection, fault diagnosis, life prediction and intelligent scheduling are significantly improved.

[0106] Second, through the periodic data update and parameter adjustment mechanism, the model can regularly adjust parameters according to the requirements of new data and new tasks, ensuring that the model can adapt to changing environments and task requirements, thereby improving the adaptability and robustness of the model.

[0107] Third, the online learning algorithm enables the model to update itself according to the real-time data stream without retraining the entire model, which greatly reduces the time cost of model updating and improves the real-time performance and response speed of the system.

[0108] Fourth, through cloud-edge collaboration, effective interaction and knowledge sharing are achieved between large models and small models. The large model performs complex calculations and knowledge generation in the cloud, and the small model performs efficient reasoning and execution on the edge, which optimizes the utilization of computing resources and reduces transmission delays and bandwidth requirements.

[0109] Fifth, through the shared feature extractor, multimodal data can be processed and common features can be extracted. The task-specific fine-tuning layer is designed according to different business scenarios and fine-tuned on specific data sets to ensure that the large model can accurately capture and understand the key information in the multimodal data, thereby improving the model's decision-making ability and prediction accuracy.

[0110] Sixth, the input-output consistency evaluation method of large and small models can ensure the consistency, effectiveness and accuracy of the output of large and small models on the same task, avoid misjudgment and errors caused by differences between models, and improve the reliability and stability of the system.

[0111] Seventh, use data fusion technology and modal alignment methods to adapt and fine-tune the large wind power model to ensure that the large model can accurately capture and understand the key information in the multimodal data, thereby improving the model's ability to process complex multimodal data.

[0112] Eighth, by designing information transmission and collaborative workflow between large and small models, a method for key component defect detection, fault diagnosis, life prediction and intelligent scheduling for collaborative large and small models was proposed, which achieved rapid response and optimization in different business scenarios and improved business efficiency and quality.

[0113] A possible implementation of a coordination and knowledge transfer system based on a wind power model is given below, which is used to execute the various execution steps and corresponding technical effects of the coordination and knowledge transfer method shown in the above embodiments and possible implementations. The system includes:

[0114] A preprocessing module, used to obtain a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set;

[0115] A first pre-training module is used to pre-train the cloud-based multimodal large model using the pre-processed multimodal data set to obtain a trained cloud-based multimodal large model;

[0116] The second pre-training module is used to pre-train the local data set corresponding to the task through a deep learning model to obtain multiple professional small models based on specific scenarios;

[0117] The consistency evaluation module is used to perform consistency evaluation on the output results of the trained cloud-based multimodal large model and the professional small model based on a specific scenario based on the same input. When the difference in the consistency evaluation is greater than a preset threshold, the fine-tuning layer is used to fine-tune the trained cloud-based multimodal large model to obtain a fine-tuned cloud-based multimodal large model.

[0118] The knowledge distillation module is used to transfer knowledge between the fine-tuned cloud-based multimodal large model and the professional small model based on specific scenarios using two-way knowledge distillation, and then executes the data update mechanism and online learning algorithm to complete the coordination and knowledge transfer of the wind power model.

[0119] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A wind power model-based collaboration and knowledge transfer method, characterized in that: The method comprises: Acquire a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set; The preprocessed multimodal data set is used to pre-train the cloud-based multimodal large model to obtain a trained cloud-based multimodal large model; Pre-train the local data set corresponding to the task through the deep learning model to obtain multiple scenario-based professional small models; Based on the same input, the output results of the trained cloud-based multimodal large model and the scenario-based professional small model are respectively evaluated for consistency. When the difference in the consistency evaluation is greater than a preset threshold, the trained cloud-based multimodal large model is fine-tuned using a fine-tuning layer to obtain a fine-tuned cloud-based multimodal large model. After using two-way knowledge distillation to transfer knowledge between the fine-tuned cloud-based multimodal large model and the scenario-based professional small model, the data update mechanism and online learning algorithm are executed to complete the coordination and knowledge transfer of the wind power model.

2. The collaboration and knowledge transfer method according to claim 1, characterized in that: The multimodal data set includes a fault text data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set includes: Perform text cleaning on the fault text dataset, and then perform word segmentation to obtain the fault text dataset after word segmentation; The Bert model is used to extract features from the fault text dataset after word segmentation to obtain the fault text feature vector.

3. The collaboration and knowledge transfer method as claimed in claim 2, characterized in that: The multimodal data set includes a vibration data set, and the step of performing data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set also includes: Denoising and normalizing the vibration data set to obtain a denoised and normalized vibration data set; The denoised and normalized vibration data set is subjected to feature extraction to obtain time domain features and frequency domain features respectively; The time domain features and frequency domain features are combined into a vibration feature vector.

4. The collaboration and knowledge transfer method as claimed in claim 3, characterized in that: The multimodal dataset includes a voiceprint dataset, and the step of preprocessing the multimodal dataset to obtain a preprocessed multimodal dataset also includes: The voiceprint dataset is subjected to denoising by Wiener filtering, and then subjected to time normalization and amplitude normalization to obtain a normalized voiceprint dataset. The Mel-frequency cepstral coefficients, formant and linear prediction coding are extracted from the normalized voiceprint dataset to form a voiceprint feature vector.

5. The collaboration and knowledge transfer method according to claim 4, characterized in that: The multimodal dataset includes an image dataset, and the step of performing data preprocessing on the multimodal dataset to obtain a preprocessed multimodal dataset also includes: The image enhancement technology is used to perform image enhancement on the image data set to obtain an image enhanced image data set; The convolutional neural network is used to extract the features of the image dataset after image enhancement to obtain geometric features, shape features, color features and HOG features to form the image feature vector.

6. The collaboration and knowledge transfer method according to claim 1, characterized in that: The steps of pre-training the cloud multimodal large model using the pre-processed multimodal data set to obtain the trained cloud multimodal large model include: The preprocessed multimodal data set is input into the constructed shared feature extractor to extract common features and obtain common features; Use the intra-modal self-attention mechanism and transformer to perform feature interaction on common features to form a multi-modal representation of common features; Based on the multimodal representation of common features and the defined loss function, the preprocessed multimodal dataset is input into the cloud-based multimodal big model for pre-training to obtain the trained cloud-based multimodal big model.

7. The collaboration and knowledge transfer method according to claim 1, characterized in that: The fine-tuning layer is set after the output layer of the large model and adopts parameter-efficient fine-tuning technology. The fine-tuning strategies of the fine-tuning layer include learning rate, training rounds, and batch size.

8. The collaboration and knowledge transfer method according to claim 1, characterized in that: The data update mechanism includes periodic data update, data consistency assessment and model parameter adjustment; In data consistency assessment, the sum of squared differences between the cloud and edge data states is calculated. When the sum of squared differences is 0, it means the data is consistent. When the sum of squared differences is not 0, it means there is a difference in the data. The calculation formula for the sum of squared differences is: Data cloud,i It is the cloud data status. edge,i is the edge data state, and n is the total number of data items.

9. The collaboration and knowledge transfer method according to claim 1, characterized in that: The online learning algorithm uses a trained multimodal large model as the initial model, continuously receives real-time data from the edge, calculates the gradient based on the data points, and updates the model weight in real time based on the gradient. The calculation formula for the model weight is: ω t is the weight model at time t, η is the learning rate, L is the loss function, y t is the true label, is the model’s prediction at time t, ω t+1 is the weight model at time t+1, To represent ω t The gradient at time t.

10. A wind power model-based collaboration and knowledge transfer system, characterized in that: The system comprises: A preprocessing module, used to obtain a multimodal data set in the field of wind power and perform data preprocessing on the multimodal data set to obtain a preprocessed multimodal data set; A first pre-training module is used to pre-train the cloud-based multimodal large model using the pre-processed multimodal data set to obtain a trained cloud-based multimodal large model; The second pre-training module is used to pre-train the local data set corresponding to the task through a deep learning model to obtain multiple scene-based professional small models; The consistency evaluation module is used to perform consistency evaluation on the output results of the trained cloud-based multimodal large model and the scenario-based professional small model based on the same input. When the difference in the consistency evaluation is greater than a preset threshold, the fine-tuning layer is used to fine-tune the trained cloud-based multimodal large model to obtain a fine-tuned cloud-based multimodal large model. The knowledge distillation module is used to transfer knowledge between the fine-tuned cloud-based multimodal large model and the scenario-based professional small model using two-way knowledge distillation, and then executes the data update mechanism and online learning algorithm to complete the coordination and knowledge transfer of the wind power model.

Images