Intelligent prediction model construction method based on multi-modal data fusion

Abstract

The invention provides an intelligent prediction model construction method based on multi-modal data fusion, and the method comprises the following steps: collecting multi-modal data in different electric power scenes, and carrying out the preprocessing of the multi-modal data, the electric power scenes including electric power inspection, electric power transaction and electric power equipment operation and maintenance; performing feature extraction, semantic association and space-time alignment on the preprocessed multi-modal data through a cross-modal attention mechanism and association analysis; constructing an intelligent prediction model, and performing knowledge injection on the intelligent prediction model through a retrieval enhancement generation technology and a lightweight adapter; a multi-task learning framework is introduced into the intelligent prediction model, exclusive output heads are set for different power scenes, and training is carried out through multi-modal data; after training is completed, lightweight compression is carried out on the intelligent prediction model through a knowledge distillation technology, a multi-modal data fusion technology system suitable for the power industry is formed, and key technology support is provided for deepening application of the intelligent prediction model of the power system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent prediction, in particular to an intelligent prediction model construction method based on multi-modal data fusion. BACKGROUND

[0002] With the deepening development of the new round of scientific and technological revolution and industrial change, artificial intelligence has become an important engine driving new productivity. At present, the power industry is in a key stage of new power system construction and a strategic window period of artificial intelligence industry development, and has extensive strategic high-value application scenario needs. It is urgent to integrate artificial intelligence technology in power operation, equipment management, power transaction, customer service, management and other key businesses to support the solution of problems such as reliable power supply and high-quality development of enterprises in the process of energy transformation.

[0003] Under the trend of digitalization and upgrading of the power industry, based on the above background, it is urgent to support power enterprises to carry out breakthroughs and frontier researches on key technologies and basic core technologies of artificial intelligence, especially multi-modal data fusion research. The power industry has massive data, and various business professionals are constantly generating multi-modal and multi-dimensional power-related data such as power time series data, environmental monitoring data, business data, and customer data. These data have very high data value to be mined, and are used for identifying key factors, predicting business trends, preventing potential risks, and guiding production operations. However, traditional data analysis methods are often limited by data volume and data dimension complexity, making it difficult to efficiently mine data value and carry out correlation analysis. SUMMARY

[0004] Therefore, the present application proposes an intelligent prediction model construction method based on multi-modal data fusion, which mines the potential value of multi-modal data and constructs an intelligent prediction model to improve the prediction accuracy of the intelligent prediction model.

[0005] The technical solution of the present application is as follows: The intelligent prediction model construction method based on multi-modal data fusion comprises the following steps: Step S1, collecting multi-modal data under different power scenarios, and preprocessing the multi-modal data, wherein the power scenarios include power inspection, power transaction, and power equipment operation and maintenance; Step S2, performing feature extraction, semantic correlation, and space-time alignment on the preprocessed multi-modal data through a cross-modal attention mechanism and correlation analysis; Step S3, constructing an intelligent prediction model, and injecting knowledge into the intelligent prediction model through retrieval enhancement generation technology and a lightweight adapter; Step S4, the intelligent prediction model introduces a multi-task learning framework, sets a dedicated output head for different power scenarios, and is trained through multi-modal data; Step S5, after training, the intelligent prediction model is compressed through the knowledge distillation technology.

[0006] Preferably, the multi-modal data in different power scenarios includes device images and defect report texts in the power inspection scenario, policy files, transaction detail texts and price time series data in the power transaction scenario, and sensor records, maintenance logs and device status images in the power equipment operation and maintenance scenario. The preprocessing includes: dividing the multi-modal data into text data, image data and time series data; cleaning and denoising the text data, entity labeling and power professional term standardization; Optimizing the illumination and resolution of the image data using enhancement techniques, and labeling the device components and defect areas; Denoising, filtering, and time series alignment of time series data.

[0007] Preferably, the specific steps of step S2 include: Step S21, a feature extraction network based on cross-modal attention is constructed, the feature extraction network includes a text sub-network, an image sub-network and a time series sub-network, the text sub-network adopts a bidirectional Transformer structure to capture semantic association, the image sub-network combines a convolutional neural network and a Transformer to extract visual features, and the time series sub-network mines dynamic change features through a time series convolution network. Through cross-modal attention, the different modal features interact and focus, and the associated data pairs in the same scene are obtained; Step S22, the associated data pairs in the same scene are taken as positive samples, and the unassociated data is taken as negative samples, the similarity loss between the samples is optimized, and each modal feature is converted into a high-dimensional vector. A semantic association rule is constructed by combining a power domain knowledge graph to guide the mapping direction of the vector space, so that the multi-modal data realizes semantic alignment in a unified representation space; Step S23, the correlation between the multi-modal data in the time and space dimensions is captured, and the alignment error is optimized through iteration.

[0008] Preferably, the specific steps of step S23 are: Topological data analysis is used to extract the spatio-temporal topological invariants of the multi-modal data, and a symbol pattern-association strength mapping table generated by symbolic dynamics is used to mine initial spatio-temporal association rules; Based on the initial spatio-temporal association rule, the key events in the multi-modal data are encoded into time-place containing pulse features by using the pulse neural network, and the heterogeneous space data in the multi-modal data is aligned to the shared manifold by using the Riemannian manifold learning, so as to complete the spatio-temporal feature coding; Taking the pulse features as nodes, the event trigger graph is constructed according to the initial association rule, the time association weight is learned by using the causal graph convolution, the extended optical flow field is calculated for the motion data and the spatial association strength is quantified, and the spatio-temporal association model is constructed; Based on the spatio-temporal association model, the quantum annealing is used to quickly reduce the error, the meta-learning is used to fine-tune the parameters, and finally the GAN adversarial network is used to add adversarial disturbance, so as to complete the iterative correction.

[0009] Preferably, the specific steps of step S3 of injecting knowledge into the intelligent prediction model by searching and enhancing generation technology and lightweight adapter include: Step S31, constructing a power professional multi-modal knowledge base for different power scenarios, and injecting the power regulations and typical cases in the power professional multi-modal knowledge base into the intelligent prediction model in real time by searching and enhancing generation technology; Step S32, constructing a lightweight adapter for different power scenarios, and encoding the core knowledge required for the power scenario into learnable parameters by the lightweight adapter and injecting them into the intelligent prediction model.

[0010] Preferably, the specific steps of step S32 include: Mining cross-scene general core knowledge from multi-modal data to generate an abstract concept lattice as a knowledge carrier; Encoding the abstract concept lattice into a low-rank knowledge factor chain by tensor train decomposition, and designing an adapter framework based on a neural state machine to use the knowledge factor chain as a state transition rule; Embedding the adapter into the residual connection path of the intelligent prediction model, and converting the knowledge factor chain into a dynamic bias term of the intelligent prediction model through a knowledge reparameterization mechanism; Freezing the main parameters of the intelligent prediction model, and only training the adapter to dynamically activate relevant knowledge factors according to the input data, and setting the weights of irrelevant knowledge factors to zero; Removing the knowledge factor branches with low activation frequency in the adapter, and reducing the parameter amount to obtain a lightweight adapter.

[0011] Preferably, the specific steps of step S4 include: Step S41, the intelligent prediction model introduces a multi-task learning framework, and respectively allocates defect prediction tasks, price fluctuation tasks and fault early warning tasks for power inspection, power transaction and power equipment operation and maintenance; Step S42, a defect prediction output head, a price fluctuation output head and a fault early warning output head are respectively set for the defect prediction task, the price fluctuation task and the fault early warning task. Step S43, the intelligent prediction model is trained through the multi-modal data, and the training weights of the tasks are adjusted through the dynamic loss function.

[0012] Preferably, the specific steps of the step S43 are: The shared feature layer of the intelligent prediction model is frozen, and the defect prediction output head, the price fluctuation output head and the fault early warning output head are trained respectively by using the data of the corresponding modal; The shared feature layer of the intelligent prediction model is unfrozen, and the multi-modal data is simultaneously inputted for joint training, and the parameters of the intelligent prediction model are updated by using small batch gradient descent; The training weights of the tasks are dynamically adjusted based on the sparsity of the different modal data through the dynamic loss function.

[0013] Preferably, the specific steps of the step S5 include: Step S51, the trained intelligent prediction model is recorded as a teacher model, a student model with a simplified structure is constructed, and a forward channel of the teacher model-student model is built; Step S52, the inference logic, feature extraction ability and prediction knowledge of the teacher model are migrated to the student model through a distillation loss function, and light-weight compression is realized.

[0014] Preferably, the post-migration of the step S52 includes the following steps: Sample data with a frequency lower than 0.1% is searched from the multi-modal data to form an independent validation set; A reverse channel of the student model-teacher model is added, the backbone parameters of the student model are frozen, and the light-weight features of the student model are inputted into the temporary learning layer of the teacher model; The temporary learning layer of the teacher model is trained by using the independent validation set to obtain reinforced features for the low-frequency data, and the parameters of the distillation loss function are adjusted based on the reinforced features.

[0015] Compared with the prior art, the present application has the following advantages: The intelligent prediction model construction method based on multi-modal data fusion provided by the application can classify and collect multi-modal data in different power scenarios, and after preprocessing, introduce a cross-modal attention mechanism and correlation analysis, perform feature extraction, semantic correlation and space-time alignment on the multi-modal data, verify the consistency of the feature correlation strategy, and provide a complete data basis for constructing an intelligent prediction model. In the intelligent prediction model framework, knowledge injection is performed through retrieval enhancement generation technology and a lightweight adapter, so that the intelligent prediction model can deeply master basic knowledge of the power industry and have a cognitive basis for performing various prediction tasks. At the same time, a multi-task learning framework is introduced into the intelligent prediction model, and after setting output heads for different tasks, training is performed, which can be applied to different power scenarios. Finally, the intelligent prediction model is compressed by a knowledge distillation technology to ensure efficient reasoning in the actual deployment of the power system. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only preferred embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 The flowchart of the intelligent prediction model construction method based on multi-modal data fusion of the application; Figure 2 The flowchart of step S2 of the intelligent prediction model construction method based on multi-modal data fusion of the application; Figure 3 The flowchart of step S3 of the intelligent prediction model construction method based on multi-modal data fusion of the application; Figure 4 The flowchart of step S4 of the intelligent prediction model construction method based on multi-modal data fusion of the application; Figure 5 The flowchart of step S5 of the intelligent prediction model construction method based on multi-modal data fusion of the application. DETAILED DESCRIPTION

[0018] In order to better understand the technical content of the application, a specific embodiment is provided below, and the application is further described in combination with the drawings.

[0019] Referring to Figures 1 to 5 The intelligent prediction model construction method based on multi-modal data fusion provided by the application includes the following steps: Step S1, collect multi-modal data under different power scenarios, and pre-process the multi-modal data, the power scenarios including power inspection, power transaction and power equipment operation and maintenance; Step S2, feature extraction, semantic association and space-time alignment are performed on the pre-processed multi-modal data through cross-modal attention mechanism and correlation analysis; Step S3, an intelligent prediction model is constructed, and knowledge injection is performed on the intelligent prediction model through retrieval enhancement generation technology and a lightweight adapter; Step S4, the intelligent prediction model introduces a multi-task learning framework, sets a dedicated output head for different power scenarios, and is trained through multi-modal data; Step S5, after training, the intelligent prediction model is compressed through knowledge distillation technology.

[0020] The intelligent prediction model construction method based on multi-modal data fusion of the application is used to construct a multi-modal intelligent prediction model suitable for power inspection, power transaction and power equipment operation and maintenance scenarios. By interfacing with power business scenarios, multi-modal data under different power scenarios is collected, and the multi-modal data is pre-processed to remove corresponding noise, fill in missing values and standardize, ensuring the accuracy and uniformity of the data. Then, cross-modal attention mechanism and correlation analysis are introduced to extract features, associate semantics and align space-time of the multi-modal data. Feature extraction is performed on different modal data, then cross-modal attention is used to realize interactive attention of different modal data, and semantic association is performed. Finally, space-time alignment is realized through semantic mapping, so that multi-modal data realizes semantic alignment in a unified representation space. At this time, the pre-processing process of the multi-modal data is completed, and an intelligent prediction model framework is constructed. Before training the intelligent prediction model, the intelligent prediction model needs to deeply master power professional knowledge, therefore, retrieval enhancement generation technology and a lightweight adapter are introduced. The corresponding professional knowledge is retrieved through retrieval enhancement generation technology and injected into the intelligent prediction model. At the same time, the lightweight adapter can encode the corresponding knowledge and inject it into the intelligent prediction model. Through multi-channel knowledge injection, the intelligent prediction model can master the power business-related knowledge necessary for power target intelligent prediction.

[0021] After the intelligent prediction model framework is built and knowledge is injected, a multi-task learning framework is introduced into the intelligent prediction model, corresponding exclusive output heads are set for power inspection, power transaction and power equipment operation and maintenance, then the corresponding parts of the multi-modal data are trained, after the training is completed, the intelligent prediction model can be used for multifunctional prediction of the power system, including inspection equipment defect prediction based on power inspection, price fluctuation prediction based on power transaction and device fault warning prediction based on power equipment operation and maintenance, and after the training is completed, the intelligent prediction model is compressed by the knowledge distillation technology, under the premise of retaining the core prediction performance, the parameter size and the calculation resource consumption of the intelligent prediction model are greatly reduced, ensuring that the inference response of the intelligent prediction model in different hardware environments such as power inspection terminals, transaction decision systems and operation and maintenance monitoring platforms can be realized with low delay and high stability, and the intelligent prediction technology is promoted to be deeply landed and value-transformed in the core business scenarios of the power industry.

[0022] Preferably, the multi-modal data in different power scenarios includes device images and defect report texts in the power inspection scenario, policy files, transaction details texts and price time series data in the power transaction scenario, and sensor records, maintenance logs and device state images in the power equipment operation and maintenance scenario, and the preprocessing includes: dividing the multi-modal data into text data, image data and time series data; cleaning and denoising the text data, entity labeling and power professional term standardization; optimizing the illumination and resolution of the image data by enhancement technology, and labeling the device components and defect areas; denoising, filtering, anomaly correction and time series alignment of the time series data.

[0023] The data content and data types required for prediction in different power scenarios are different, so a classification collection mechanism is established to collect multi-modal data in different power scenarios, and the multi-modal data can be divided into text data, image data and time series data containing sensor records according to the different types. The preprocessing methods of different types of modal data are different, the text data is mainly cleaned and denoised, entity labeled and power professional term standardized to remove redundant information and unify the format, for the image data, the illumination and resolution are optimized by enhancement technology, and the key features are labeled, and for the time series data such as sensor records, the unstructured information is converted into semi-structured format after corresponding processing, through targeted preprocessing, the integrity and consistency of various types of modal data are ensured, laying a foundation for the subsequent training of the intelligent prediction model.

[0024] Preferably, the specific steps of step S2 include: Step S21, a feature extraction network based on cross-modal attention is constructed, the feature extraction network includes a text subnetwork, an image subnetwork, and a time sequence subnetwork, the text subnetwork adopts a bidirectional Transformer structure to capture semantic association, the image subnetwork combines a convolutional neural network and a Transformer to extract visual features, the time sequence subnetwork mines dynamic change features through a time sequence convolution network, cross-modal attention is used to realize interactive attention of different modal features, and a pair of associated data in the same scene is obtained; Step S22, a pair of associated data in the same scene is taken as a positive sample, and non-associated data is taken as a negative sample, similarity loss between samples is optimized, meanwhile, each modal feature is converted into a high-dimensional vector, a semantic association rule is constructed by combining a power field knowledge graph, a mapping direction of a vector space is guided, and semantic alignment of multi-modal data is realized in a unified representation space; Step S23, a time-space dimension association relationship of multi-modal data is captured, and alignment error is iteratively optimized.

[0025] After multi-modal data preprocessing, a cross-modal attention mechanism and a contrastive learning are taken as a core framework, feature extraction, semantic association, and space-time alignment are highlighted, a feature extraction network based on cross-modal attention is first constructed, exclusive subnetworks are designed for different modal data features, including a text subnetwork, an image subnetwork, and a time sequence subnetwork, the text subnetwork captures semantic association, the image subnetwork extracts visual features, and the time sequence subnetwork mines dynamic change features, then cross-modal attention layers are used to realize interactive attention of different modal features, and weight distribution of key features is strengthened.

[0026] Meanwhile, a contrastive learning framework is introduced, a cross-modal contrastive loss function is designed, a pair of associated data in the same scene (such as an inspection image and corresponding defect text description) and non-associated data are taken as negative samples, similarity loss between samples is optimized, semantic consistency between modes is strengthened, on this basis, a semantic mapping method is developed, each modal feature is converted into a high-dimensional vector through an embedding layer, a semantic association rule is constructed by combining a power field knowledge graph, a mapping direction of a vector space is guided, multi-modal data is aligned in semantics, then a dynamic association is designed for a cross-modal alignment mechanism, a time-space dimension association relationship of multi-modal data (such as time sequence association of equipment running state and sensing data, spatial association of equipment image and location text) is captured in real time, alignment error is iteratively optimized, and precise association of multi-modal data from bottom features to high-level semantics is realized.

[0027] Preferably, the specific steps of step S23 are as follows: Topological data analysis is used to extract space-time topological invariants of multi-modal data, a symbol pattern-association strength mapping table generated by symbolic dynamics is combined, and initial space-time association rules are mined; Based on the initial spatio-temporal association rule, the key events in the multi-modal data are encoded into time-place containing pulse features by using the pulse neural network, and the heterogeneous space data in the multi-modal data is aligned to the shared manifold by using the Riemannian manifold learning, to complete the spatio-temporal feature coding; Taking the pulse features as nodes, the event trigger graph is constructed according to the initial association rule, the time association weight is learned by using the causal graph convolution, the extended optical flow field is calculated for the motion data and the spatial association strength is quantified, and the spatio-temporal association model is constructed; Based on the spatio-temporal association model, the quantum annealing is used to quickly reduce the error, the meta-learning is used to fine-tune the parameters, and finally the GAN adversarial network is added to add adversarial disturbance, to complete the iterative correction.

[0028] The time axis of the multi-modal data is divided into sliding windows, and the space is divided according to the grid. The multi-modal data points in each window form a topological node. The topological invariants of different modal nodes are calculated by topological data analysis, such as Betti numbers and Euler characteristics, to identify the topological structure shared by cross-modal, as the initial rule of spatio-temporal association. At the same time, the continuous multi-modal data is symbolically processed by symbolic dynamics, and the causal dependence of different modal symbol sequences is calculated to generate a symbol pattern-association strength mapping table, to construct the initial spatio-temporal association rule.

[0029] Then, based on the initial spatio-temporal association rule, the multi-modal data is processed by using the pulse neural network and the Riemannian manifold learning, to complete the spatio-temporal feature coding. Then, taking the pulse features as nodes, the event trigger graph is constructed according to the initial association rule, the time association weight is learned by using the causal graph convolution, and for the motion data in the multi-modal data, such as unmanned aerial vehicle video and mobile sensor trajectory, the spatial association is calculated by using the extended optical flow field. For the video modal, the pixel motion vector of adjacent frames is calculated by the optical flow algorithm to represent the change of spatial position with time. For the sensor modal, the trajectory data of the mobile sensor is converted into virtual optical flow and matched with the video optical flow field, to construct the spatio-temporal association model. Finally, the quantum annealing is introduced to reduce the error, and the meta-learning is used to fine-tune the parameters. Then, the GAN adversarial network is added to add adversarial disturbance to optimize the robustness, to complete the iterative correction.

[0030] Preferably, the specific steps of step S3 of knowledge injection of the intelligent prediction model through retrieval enhancement generation technology and lightweight adapter include: Step S31, constructing a power professional multi-modal knowledge base for different power scenarios, and injecting the power procedures and typical cases in the power professional multi-modal knowledge base into the intelligent prediction model in real time through retrieval enhancement generation technology; Step S32, constructing a lightweight adapter for different power scenarios, and encoding the core knowledge required for the power scenario into learnable parameters through the lightweight adapter and injecting the learnable parameters into the intelligent prediction model.

[0031] The knowledge injection stage adopts a double-path strategy to enable the intelligent prediction model to master the power business-related knowledge necessary for achieving intelligent prediction of power targets. On the one hand, through retrieval enhancement generation technology, the power regulations, typical cases, etc. in the constructed power professional multi-modal knowledge base are retrieved in real time and then fused into the intelligent prediction model reasoning process to improve the response accuracy of the intelligent prediction model to professional problems. By studying and constructing a power knowledge graph, knowledge extraction is performed on the knowledge related to a specific power business scenario to help the intelligent prediction model sort out the logical relationships between different knowledge entities and reduce the possibility of hallucination of the intelligent prediction model. The retrieval enhancement technology can guide the intelligent prediction model to follow the knowledge logic stored in the graph to perform reasoning and prediction.

[0032] On the other hand, a lightweight adapter concept is introduced into the network structure of the intelligent prediction model. Independent adapter modules are constructed for different business fields such as power inspection, power transaction, and power equipment operation and maintenance, the core knowledge required for implementing the prediction tasks in the above business fields is coded as learnable parameters to realize deep injection of professional knowledge into the model. At the same time, a dynamic routing mechanism is introduced to ensure that the corresponding adapter can be automatically selected according to the input scene type during reasoning, realizing flexible adaptation of one model to multiple scenes.

[0033] Preferably, the specific steps of the step S32 include: mining cross-scene general core knowledge from multi-modal data to generate an abstract concept lattice as a knowledge carrier; encoding the abstract concept lattice into a low-rank knowledge factor chain through tensor train decomposition, and designing an adapter framework based on a neural state machine to use the knowledge factor chain as a state transition rule; embedding the adapter into the residual connection path of the intelligent prediction model, and converting the knowledge factor chain into a dynamic bias term of the intelligent prediction model through a knowledge reparameterization mechanism; freezing the main parameters of the intelligent prediction model and only training the adapter to dynamically activate relevant knowledge factors according to the input data, with the weights of irrelevant knowledge factors being set to zero; removing knowledge factor branches with low activation frequency in the adapter and reducing the parameter amount to obtain a lightweight adapter.

[0034] By form concept analysis, cross-scene general core can be mined from multi-modal data, such as device state-environment factor-exception probability association, and an abstract concept lattice is generated as a knowledge carrier, then tensor train decomposition is introduced, the concept lattice is encoded into a low-rank knowledge factor chain, the knowledge parameter scale is compressed to 1 / 20 of the traditional method, then an adapter framework is designed, taking the knowledge factor chain as the state transition rule, the core scale parameter is controlled within 1000, then the adapter is embedded into the residual connection path of the intelligent prediction model, through the knowledge reparameterization mechanism, the knowledge factor chain is converted into the dynamic bias term of the intermediate layer of the intelligent prediction model, during training, the backbone parameters of the intelligent prediction model are frozen, only the adapter is trained, the adapter activates the relevant knowledge factors according to the input data, for example, the abnormal data preferentially activates the fault knowledge, and the irrelevant knowledge factor weight is zero, during the training process, the knowledge factor branch with an activation frequency <1% is removed, further reducing the final parameter quantity, and ensuring lightweight deployment.

[0035] During the training process, a parameter fine-tuning process is also designed, by freezing the main parameters of the intelligent prediction model, only fine-tuning the adapter module parameters and attention weights for supervised fine-tuning, so that the intelligent prediction model can further master the relevant knowledge reserves in the specific power business field, improve the model accuracy, and adopt a joint loss function to carry out fine-tuning training, while optimizing the retrieval accuracy of the retrieval enhancement generation technology, the adapter knowledge encoding ability and the loss function of the final prediction task, so that the joint loss function converges, in order to ensure the continuous effectiveness of the intelligent prediction model, a small sample incremental learning strategy is introduced, and the intelligent prediction model knowledge base and the adapter parameters are updated through incremental training.

[0036] Preferably, the specific steps of step S4 are as follows: Step S41, the intelligent prediction model introduces a multi-task learning framework, and respectively allocates defect prediction tasks, price fluctuation tasks and fault early warning tasks for power inspection, power transaction and power equipment operation and maintenance; Step S42, a defect prediction output head, a price fluctuation output head and a fault early warning output head are respectively set for the defect prediction task, the price fluctuation task and the fault early warning task; Step S43, the intelligent prediction model is trained by multi-modal data, and the training weights of each task are adjusted by a dynamic loss function.

[0037] The defect prediction output head focuses on the inference of defect types and positions, integrates image features and defect history data; the price fluctuation output head focuses on the fusion analysis of market supply and demand, policy text and price time series features; the fault early warning output head strengthens the time sequence correlation mining of equipment operation sensing data and maintenance records. During the training process, based on the power knowledge representation layer shared by a certain power large model, multi-task feature sharing is realized, and the labeled data of the three power scenarios are input synchronously for joint training. At the same time, a domain adaptive training method is adopted to optimize the attention mechanism according to the business characteristics of different scenarios, ensuring that the model can accurately capture key features in each scenario and form an intelligent prediction model adapted to multiple businesses.

[0038] Preferably, the specific steps of step S43 are: Freeze the shared feature layer of the intelligent prediction model, and train the defect prediction output head, the price fluctuation output head and the fault early warning output head respectively using the data of the corresponding modal; Unfreeze the shared feature layer of the intelligent prediction model, and input the multi-modal data simultaneously for joint training, and update the parameters of the intelligent prediction model by small batch gradient descent; Adjust the training weights of each task based on the sparsity of different modal data through a dynamic loss function.

[0039] A phased initialization training strategy is adopted. First, freeze the shared feature layer of the intelligent prediction model, train each task output head with single-scene data respectively, so that each output head is preliminarily adapted to the task characteristics. Then, unfreeze the shared feature layer, input multi-modal data simultaneously for joint training, and update the parameters by small batch gradient descent. Finally, adjust the training weights of each task through a dynamic loss function, increase the weight in the data scarce scene to strengthen the learning effect, and appropriately reduce the weight in the data sufficient scene to balance the model resource allocation.

[0040] Preferably, the specific steps of step S5 include: Step S51, record the trained intelligent prediction model as a teacher model, and construct a structure-reduced student model to build a forward channel of the teacher model-student model; Step S52, migrate the inference logic, feature extraction capability and prediction knowledge of the teacher model to the student model through a distillation loss function to realize lightweight compression.

[0041] To ensure efficient deployment of intelligent prediction models in the actual environment of the power system, the knowledge distillation technology is adopted to realize lightweight compression of the model, the completed intelligent prediction model is taken as a teacher model, a student model with a simplified structure is constructed, the reasoning logic, feature extraction capability and prediction knowledge of the teacher model are migrated to the student model through a distillation loss function, the model parameter size and the calculation resource consumption are greatly reduced on the premise of retaining the core prediction performance, the compressed intelligent prediction model is tested for deployment environment adaptation, and it is ensured that the intelligent prediction model can realize low-delay and high-stability inference response in different hardware environments such as power inspection terminals, transaction decision systems and operation and maintenance monitoring platforms, and the intelligent prediction technology is promoted to be deeply landed and value-transformed in the core business scenarios of the power system.

[0042] Preferably, the post-migration stage S52 includes the following steps: Find sample data with a frequency lower than 0.1% from the multi-modal data to form an independent validation set; Add a reverse channel of the student model-teacher model, freeze the backbone parameters of the student model, and input the lightweight features of the student model into the temporary learning layer of the teacher model; Train the temporary learning layer of the teacher model using the independent validation set to obtain reinforced features of low-frequency data, and adjust the parameters of the distillation loss function based on the reinforced features.

[0043] Set a post-migration trigger threshold, and after the trigger threshold is reached, perform knowledge distillation back propagation, wherein the low-frequency data in the multi-modal data is found out and an independent validation set is formed, a reverse channel is added based on the original forward channel of the teacher model-student model, the backbone parameters of the student model are frozen, the lightweight features of the student model are input into the temporary learning layer of the teacher model, only the temporary learning layer is trained, the reconstruction error of the teacher model to the lightweight features of the student model is reduced, and then the reinforced features of the low-frequency data obtained by the teacher model are used to adjust the parameters of the distillation loss function, inhibit the forgetting of the student model to low-frequency but key power knowledge, and improve the knowledge retention rate.

[0044] The above only describes preferred embodiments of the present application and should not be used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for constructing an intelligent prediction model based on multimodal data fusion, characterized in that, Includes the following steps: Step S1: Collect multimodal data under different power scenarios and preprocess the multimodal data. The power scenarios include power inspection, power trading and power equipment operation and maintenance. Step S2: Perform feature extraction, semantic association, and spatiotemporal alignment on the preprocessed multimodal data through cross-modal attention mechanism and association analysis; Step S3: Construct an intelligent prediction model and inject knowledge into the intelligent prediction model through retrieval-enhanced generation technology and a lightweight adapter; Step S4: The intelligent prediction model introduces a multi-task learning framework, sets up dedicated output heads for different power scenarios, and trains through multimodal data. Step S5: After training is completed, the intelligent prediction model is lightweighted and compressed using knowledge distillation technology.

2. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 1, characterized in that, The multimodal data under different power scenarios includes: equipment images and defect report texts for power inspection scenarios; policy documents, transaction details, and price time series data for power trading scenarios; and sensor records, maintenance logs, and equipment status images for power equipment operation and maintenance scenarios. The preprocessing includes: Multimodal data is divided into text data, image data, and time-series data; Text data is cleaned and denoised, entity labeled, and power industry terminology is standardized. Enhancement techniques are used to optimize lighting and resolution in image data, and equipment components and defect areas are labeled. Noise reduction filtering, outlier correction, and time series alignment are performed on time series data.

3. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 1, characterized in that, The specific steps of step S2 include: Step S21: Construct a feature extraction network based on cross-modal attention. The feature extraction network includes a text sub-network, an image sub-network, and a temporal sub-network. The text sub-network uses a bidirectional Transformer structure to capture semantic associations. The image sub-network combines a convolutional neural network and a Transformer to extract visual features. The temporal sub-network mines dynamically changing features through a temporal convolutional network. Cross-modal attention is used to achieve interactive attention to features of different modalities and to obtain associated data pairs in the same scene. Step S22 uses associated data pairs in the same scenario as positive samples and unassociated data as negative samples to optimize the similarity loss between samples. At the same time, it transforms each modal feature into a high-dimensional vector and constructs semantic association rules in conjunction with the knowledge graph of the power field to guide the mapping direction of the vector space, so that multimodal data can achieve semantic alignment in a unified representation space. Step S23: Capture the correlation between multimodal data in time and space dimensions, and optimize the alignment error through iterative optimization.

4. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 3, characterized in that, The specific steps of step S23 are as follows: We use topological data analysis to extract spatiotemporal topological invariants from multimodal data, and combine them with the symbol pattern-association strength mapping table generated by symbol dynamics to mine initial spatiotemporal association rules. Based on the initial spatiotemporal association rules, a spiking neural network is used to encode key events in multimodal data into time-location spiking features. Then, through Riemannian manifold learning, heterogeneous spatial data in multimodal data are aligned to a shared manifold to complete the spatiotemporal feature encoding. Using pulse features as nodes, an event triggering graph is constructed based on the initial association rules. Causal graph convolution is used to learn the temporal association weights. At the same time, the extended optical flow field is calculated on the motion data and the spatial association strength is quantified to construct a spatiotemporal association model. Based on the spatiotemporal correlation model, quantum annealing is used to quickly reduce the error, then meta-learning is used to fine-tune the parameters, and finally adversarial perturbations are added through GAN adversarial network to complete the iterative correction.

5. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 1, characterized in that, The specific steps of step S3, which involves knowledge injection into the intelligent prediction model through retrieval-enhanced generation technology and a lightweight adapter, include: Step S31: Construct a multimodal knowledge base for the power industry for different power scenarios. Through retrieval enhancement generation technology, retrieve power regulations and typical cases from the multimodal knowledge base for the power industry in real time and inject them into the intelligent prediction model. Step S32: For different power scenarios, construct lightweight adapters, encode the core knowledge required to realize the power scenario into learnable parameters through lightweight adapters, and inject them into the intelligent prediction model.

6. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 5, characterized in that, The specific steps of step S32 include: Mining cross-scenario common core knowledge from multimodal data and generating abstract concept lattices as knowledge carriers; Abstract concepts are encoded into low-rank knowledge factor chains using Tensor Train Decomposition, and an adapter framework based on neural state machine is designed, with the knowledge factor chains serving as state transition rules. The adapter is embedded in the residual connection path of the intelligent prediction model, and the knowledge factor chain is transformed into the dynamic bias term of the intelligent prediction model through the knowledge reparameterization mechanism. Freeze the backbone parameters of the intelligent prediction model, train only the adapter, and let the adapter dynamically activate relevant knowledge factors based on the input data, while resetting the weights of irrelevant knowledge factors to zero. Remove knowledge factor branches with low activation frequency from the adapter and reduce the number of parameters to obtain a lightweight adapter.

7. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 1, characterized in that, The specific steps of step S4 are as follows: Step S41: The intelligent prediction model introduces a multi-task learning framework, assigning defect prediction tasks, price fluctuation tasks, and fault early warning tasks to power inspection, power trading, and power equipment operation and maintenance, respectively. Step S42: Set up the defect prediction output head, price fluctuation output head, and fault early warning output head for the defect prediction task, price fluctuation task, and fault early warning task, respectively. Step S43: Train the intelligent prediction model using multimodal data and adjust the training weights of each task using a dynamic loss function.

8. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 7, characterized in that, The specific steps of step S43 are as follows: Freeze the shared feature layer of the intelligent prediction model, and use the data of the corresponding modalities to train the defect prediction output head, price fluctuation output head and fault warning output head respectively; Unfreeze the shared feature layer of the intelligent prediction model, input multimodal data simultaneously for joint training, and use mini-batch gradient descent to update the parameters of the intelligent prediction model. The training weights for each task are dynamically adjusted based on the sparsity of data from different modalities using a dynamic loss function.

9. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 1, characterized in that, The specific steps of step S5 include: Step S51: Record the trained intelligent prediction model as the teacher model, and construct a simplified student model to build a positive channel between the teacher model and the student model. Step S52: Transfer the reasoning logic, feature extraction capability, and prediction knowledge of the teacher model to the student model through the distillation loss function to achieve lightweight compression.

10. The method for constructing an intelligent prediction model based on multimodal data fusion according to claim 9, characterized in that, The later stage of migration in step S52 includes the following steps: Find samples that occur less than 0.1% of the multimodal data to form an independent validation set; A new reverse channel between the student model and the teacher model is added. The backbone parameters of the student model are frozen, and its lightweight features are input into the temporary learning layer of the teacher model. The temporary learning layer of the teacher model is trained using an independent validation set to obtain reinforcement features for low-frequency data, and the parameters of the distillation loss function are adjusted based on these reinforcement features.

Images