Power plant production data real-time question answering and early warning method based on multi-modal large model

Abstract

The invention discloses a power plant production data real-time question answering and early warning method based on a multi-modal large model, and the method comprises the following steps: collecting image, time sequence and text data in the operation process of a power plant, fusing multi-modal information through a Perciver IO model, and generating time-synchronized semantic vector representation for dynamically monitoring state changes and constructing task query. Historical annotation events are modeled in combination with the Hawkes process, the time sequence induction relation between the events is captured, the future risk intensity is calculated, and the query generation and decoding process is optimized in combination with initial output, so that an enhanced question and answer and early warning result is generated. Meanwhile, an abnormal causal chain is constructed through event path contrastive analysis, and closed-loop feedback and optimization of system response are achieved. According to the invention, dynamic identification and intelligent response to complex working conditions are realized, and the intelligent level of operation and maintenance of a power plant is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a power plant production data real-time question answering and early warning method based on a multi-modal large model. BACKGROUND

[0002] With the deep application of artificial intelligence in industrial scenarios, a large amount of heterogeneous data such as image monitoring, time series sensors and text records has been accumulated in the production and operation process of power plants. Based on this, intelligent question answering and abnormal early warning have become a key direction to improve the safety and operation efficiency of power equipment. At present, multi-modal intelligent analysis systems usually rely on specific modal fusion structures to construct unified representations, and use threshold rules or static classifiers for state monitoring and risk prediction.

[0003] The existing technology has obvious limitations in multi-modal expression ability, time series causal modeling and system response mechanism: on the one hand, traditional multi-modal neural network structures lack unified modeling mechanisms when processing input modal types and task output forms, making it difficult to flexibly adapt to the fusion expression of image, time series and text data, and lacking context alignment ability and task generalization ability; on the other hand, existing abnormal early warning methods generally ignore the time sequence induction characteristics of device events, and cannot construct a dynamic risk perception model based on the causal paths between events, resulting in early warning lag and insufficient generalization; in addition, current question answering systems mostly rely on fixed decoding structures, making it difficult to actively interface key abnormal content in combination with context evolution, and lacking dynamic response ability and causal explanation ability.

[0004] Therefore, how to provide a power plant production data real-time question answering and early warning method based on a multi-modal large model is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide a power plant production data real-time question answering and early warning method based on a multi-modal large model. The present application combines Perceiver IO model and Hawkes process modeling mechanism to construct a multi-modal semantic embedding representation system that uniformly processes images, time series and text, and combines time series induction modeling and dynamic task query decoding to realize real-time identification, question answering generation and risk prediction of abnormal states, with the advantages of unified semantic expression, timely early warning response and clear causal chain tracking.

[0006] The power plant production data real-time question answering and early warning method based on a multi-modal large model according to an embodiment of the present application includes the following steps: Collecting image, time series and text data in the operation process of a power plant, preprocessing and generating a multi-modal data segment sequence; Inputting the multi-modal data segment sequence into a multi-modal large model to obtain a multi-modal semantic embedding representation of the multi-modal data segment sequence; a model, fusing the information of each modality and outputting a set of synchronous semantic embedding vectors; Based on the set of synchronous semantic embedding vectors, monitor the state change, identify the abnormal evolution trend, generate a task query vector, and combine the synchronous semantic embedding vector to generate a preliminary question and answer and warning candidate output; Extract the time series of the labeled event from the set of synchronous semantic embedding vectors, input it into the Hawkes process model, calculate the event induction relationship and output the risk intensity function; Construct a weighted query vector by combining the risk intensity function with the preliminary question and answer and warning candidate output; Input the weighted query vector and the set of synchronous semantic embedding vectors to jointly execute the decoding operation, generate the reinforced question and answer and warning result, and record the corresponding modality path and time positioning information; Compare and analyze the key event nodes in the reinforced question and answer and warning result with the induction path of the Hawkes process model, build an abnormal causal chain, and feed back to the query generation mechanism to optimize the next round of response.

[0007] Optionally, the preprocessing includes image format resampling, normalization, semantic vector mapping, unified timestamp generation, sliding window segmentation and time segment alignment packaging.

[0008] Optionally, the output of the set of synchronous semantic embedding vectors includes: Perform channel normalization, resolution standardization and feature extraction processing on the image modality input data to generate an image modality representation sequence; Perform sliding window segmentation, amplitude normalization and time series encoding processing on the time series modality input data to generate a time series modality representation sequence; Perform text segmentation, word embedding mapping and time position encoding processing on the text modality input data to generate a text modality representation sequence; Concatenate the image modality representation sequence, the time series modality representation sequence and the text modality representation sequence in a unified time sequence to construct a unified input representation; Input the unified input representation into Model for fusion encoding operation to generate a fusion semantic representation aligned with the input modality; Output the fusion semantic representation in time segment order to form a set of synchronous semantic embedding vectors.

[0009] Optionally, the The model includes the following processing steps: Add time index information and modality identification information to the image modality representation sequence, time series modality representation sequence and text modality representation sequence contained in the unified input representation to generate a multi-modal input embedding; The modality correlation analysis is performed on the multi-modal input embedding, and a modality attention weight is constructed under each time slice, wherein the modality attention weight is calculated by the following formula: ; Wherein, represents the embedding representation of the m-th modality at the time slice t, represents the embedding representation of the m-th modality at the time slice t, represents the query vector at the time slice, represents the modality attention weight, represents the embedding representation of the m-th modality at the time slice t, represents the embedding representation of the m-th modality at the time slice t; The modality attention weight is applied to the multi-modal input embedding to generate a modality dynamic weighted input representation; The first round of the set of learnable query vectors is initialized, the modality dynamic weighted input representation is fused and interacted with the corresponding set of query vectors to generate a preliminary fusion representation; Based on the semantic activation features in the preliminary fusion representation, the set of query vectors is updated in reverse, so that the updated set of query vectors encodes the semantic feedback information in the fusion representation; The updated set of query vectors is again interacted with the modality dynamic weighted input representation to perform at least one round of fusion enhancement operation based on semantic feedback modulation to form a final fusion representation, forming a bidirectional interaction mechanism between the fusion representation and the query representation; The final fusion representation is input into the context modeling structure for time series modeling to output a context fusion representation consistent with the time slice sequence, forming a set of synchronous semantic embedding vectors.

[0010] Optionally, the generation of the preliminary question and answer and the warning candidate output comprises: The Euclidean distance calculation is performed on the embedding vectors corresponding to adjacent time slices in the set of synchronous semantic embedding vectors to generate a state change metric sequence; The state change metric sequence is compared with a dynamically set state sensitive threshold to identify a set of time slices whose continuous change amplitude exceeds the threshold; The synchronous semantic embedding vectors corresponding to the time slices are extracted from the potential abnormal state window, and an event candidate segment sequence is generated according to the modality source label thereof to construct an event candidate trajectory containing time index and modality path information; The image modality representation, time series modality representation and text modality representation contained in the event candidate trajectory are aligned and spliced in time sequence to generate an event focused state vector; ​​An event-aware task query vector is constructed based on the event focus state vector, and a semantic matching operation is performed in combination with the synchronous semantic embedding vector in the corresponding time segment to generate a query decoding input vector; The query decoding input vector is subjected to query decoding to obtain a preliminary question and answer output and a pre-warning candidate result, and the corresponding time index and modal path are labeled; The preliminary question and answer output and the pre-warning candidate result form a structured preliminary question and answer and pre-warning candidate output set.

[0011] Optionally, the output risk intensity function includes: The synchronous semantic embedding vector containing the event label is filtered from the synchronous semantic embedding vector set, and the timestamp information, event type information and modal source path information are extracted to generate a time series of labeled events; The time series of labeled events is classified by event type to construct a multi-class event set, and the occurrence time of each event type and the embedding source of the corresponding modal representation are recorded; Based on the multi-class event set, a self-excited point process modeling structure is initialized, including a basic excitation rate parameter, an event-induced weight parameter and a time decay kernel function structure; For each event type, the historical event trajectory within the specified time observation window is traversed to establish an induced structure graph; The maximum likelihood-based parameter estimation method is used to optimize the basic excitation rate parameter and the induced weight parameter, and the propagation structure and response delay characteristics between events are updated; According to the fitted self-excited point process modeling structure, the risk occurrence intensity of each event type in the future time period is calculated, and the risk intensity function corresponding to the event type and the time index is output.

[0012] Optionally, the construction of the weighted query vector includes: extracting the task query vector generated in the preliminary question and answer and pre-warning candidate result and the corresponding time index and modal path information, and simultaneously extracting the risk intensity value of the corresponding time segment from the risk intensity function, aligning the task query vector and the risk intensity value in the dimension, and forming a joint representation vector through splicing operation; In the joint representation vector construction process, a mask mechanism based on time index alignment is introduced to maintain semantic consistency, and the task label is used to guide the alignment relationship of the joint representation, and the spliced joint representation vector is used as the weighted query vector.

[0013] Optionally, the generating the reinforced question answering and early warning result comprises: inputting the weighted query vector as a query, inputting the set of synchronous semantic embedding vectors as context, introducing a cross-attention mechanism to establish dynamic connection between the query and the multi-modal semantic representation, introducing an event type label guided selective attention structure to activate only the context area consistent with the query task semantics, performing cross-modal attention interaction and residual fusion operation in each layer of the decoder, layer-by-layer matching the query vector with the context representation at the time segment level, and generating a decoding response vector with time alignment characteristics, selecting different generation heads at the output end according to the task identification, and type mapping the response vector to generate structured question answering content and risk classification output.

[0014] Optionally, the constructing the abnormal causal chain comprises: extracting the labeled key event nodes from the reinforced question answering and early warning result, and obtaining the time index information and the modality source path information corresponding to each key event node to construct a key event set; According to the time index information in the key event set, the event type of the corresponding time segment and the event induced path generated in the self-activation point process modeling structure are retrieved from the risk intensity function, an event induced path graph is constructed, and the trigger direction and response delay of each node are recorded; Based on the modality source path information, the original modality input relied on by the key event nodes is located in the set of synchronous semantic embedding vectors, and the modality combination path participating in the question answering generation is reconstructed to form a question answering generation path graph, and the modality label and time segment index are retained; Comparing and analyzing the event induced path graph and the question answering generation path graph at the node level structure, comparing the time sequence dependency, modality consistency and trigger relationship between nodes in turn, identifying the information missing area and redundant cross path in the causal structure, and generating a causal consistency difference graph; Task query is performed on the causal consistency difference graph to adjust the modality attention weight and position encoding strategy in the task query vector construction process; The updated modality attention weight and the adjusted position encoding strategy are used in the next round of task query vector generation.

[0015] The beneficial effects of the present application are: (1) The Perceiver IO model is introduced to construct a unified multi-modal semantic embedding vector set, which breaks through the semantic barrier between image, time series and text data, has the advantages of strong structure generalization ability and high modality adaptability, and significantly improves the fusion expression and downstream task adaptation ability of multi-modal data under complex working conditions of power plants.

[0016] (2) The application realizes dynamic modeling of the annotated event time sequence by constructing an event-induced path and a risk intensity function based on a Hawkes process, can capture the time sequence dependence characteristics in abnormal evolution, effectively improves the accuracy and forward-looking of risk early warning, and breaks through the lagging early warning limitation of traditional static feature-based judgment.

[0017] (3) The application realizes efficient matching between multi-modal semantic embedding and dynamic task response by constructing an event-aware task query vector and combining a weighted query mechanism and a bidirectional interactive query decoding strategy, significantly enhances the alignment ability and causal chain tracing ability of the question and answer result to abnormal events, and effectively supports real-time inquiry and reasoning analysis in complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, which together with the embodiments of the application are used to explain the application, and do not constitute a limitation of the application. In the drawings:

[0019] Fig. 1 The flowchart of the power plant production data real-time question and answer and early warning method based on the multi-modal large model proposed by the application; Fig. 2 The structural schematic diagram of the synchronous semantic embedding vector set construction process in the application; Fig. 3 The modeling and reasoning flowchart of the risk intensity function generated based on the Hawkes process in the application. DETAILED DESCRIPTION

[0020] The application will now be described in further detail in conjunction with the accompanying drawings. These drawings are all simplified schematic diagrams, which only schematically illustrate the basic structure of the application, and therefore only show the components related to the application.

[0021] REFERENCE Figs. 1-3 The power plant production data real-time question and answer and early warning method based on the multi-modal large model, comprising the following steps: Collecting image, time sequence and text data during power plant operation, preprocessing, and generating multi-modal data segment sequence; Inputting the multi-modal data segment sequence into The model fuses the information of each mode and outputs a synchronous semantic embedding vector set; Based on the synchronous semantic embedding vector set, monitor the state change, identify the abnormal evolution trend, generate a task query vector, and combine the synchronous semantic embedding vector to generate a preliminary question and answer and early warning candidate output; Extract the time series of labeled events from the synchronous semantic embedding vector set, input them into the Hawkes process model, calculate the event induced relationship and output the risk intensity function; A weighted query vector is constructed by combining the risk intensity function with the preliminary question answer and early warning candidate outputs. The input weighted query vector and the synchronous semantic embedding vector set are jointly decoded to generate enhanced question-answering and warning results, and the corresponding modal path and time location information are recorded. By comparing and analyzing the key event nodes in the enhanced question-and-answer and early warning results with the triggering paths of the Hawkes process model, an abnormal causal chain is constructed and fed back to the query generation mechanism to optimize the next round of response.

[0022] In this embodiment, the preprocessing includes image format resampling, normalization, semantic vector mapping, unified timestamp generation, sliding window segmentation, and time segment alignment and encapsulation.

[0023] In this embodiment, the set of output synchronization semantic embedding vectors includes: Perform channel normalization, resolution standardization, and feature extraction on the image modal input data to generate an image modal representation sequence; The time series modal input data is processed by sliding window segmentation, amplitude normalization and time series encoding to generate a time series modal representation sequence; The text modal input data is processed by text segmentation, word embedding mapping and time position encoding to generate a text modal representation sequence. Image modal representation sequences, time series modal representation sequences, and text modal representation sequences are concatenated in a unified time order to construct a unified input representation; Input the unified input representation to The model performs a fusion encoding operation to generate a fusion semantic representation aligned with the input modality; The fused semantic representations are output sequentially according to time segments to form a set of synchronous semantic embedding vectors; The process of performing fusion encoding on the unified input representation includes receiving a unified input representation consisting of an image modality representation sequence, a time series modality representation sequence, and a text modality representation sequence; establishing an intermodal mapping relationship based on the time index information and modality identification information in the unified input representation; performing modality semantic alignment processing on a unified time axis; performing feature dimension fusion calculation on different modal features under the same time slice; and generating a fusion expression result containing multimodal semantic aggregation information, wherein the fusion expression result maintains a one-to-one correspondence with the original multimodal data segment sequence in time.

[0024] In this embodiment, the The model includes the following processing steps: Add time index information and modality identification information to the image modality representation sequence, time series modality representation sequence and text modality representation sequence contained in the unified input representation to generate multimodal input embedding; Modal correlation analysis is performed on the multimodal input embedding, and modal attention weights are constructed at each time slice, where the modal attention weights are calculated by the following formula: ; in, Indicates the first Each modality in time slice Embedded representation below, This represents the query vector within that time slice. Indicate modal attention weights, Indicates the first Each modality in time slice Embedded representation below; The modal attention weight calculation formula used in this application originates from the exponential normalization function commonly found in mathematics. This function was first used in machine learning for multi-class classification probability calculation and has been introduced into deep learning as a core component of the attention mechanism. This application makes structural improvements based on this: the Softmax function, traditionally used for inter-task similarity calculation, is extended to dynamic weighted modeling between different modal embeddings, proposing a modal-level attention scheduling mechanism. Specifically, in each time slice... Next, utilize task-related query vectors. Embedded with each modal input The dot product represents the correlation, and the modal attention weights are obtained by normalization using the Softmax function. The dot product result is a dimensionless real number, and after exponentiation and normalization, the weights are... The value of is in the range (0,1) and satisfies the normalization property (the sum of all modal weights is 1). The dimensions of both sides are dimensionless pure numbers, satisfying mathematical and physical common sense. Based on the classic formula, this formula introduces a task-related query vector and modal structure alignment mechanism to achieve the ability to dynamically adjust the modal fusion strategy under different tasks or scenarios.

[0025] Modal attention weights are applied to the multimodal input embedding to generate a dynamically weighted modal input representation; Initialize the first set of learnable query vectors, and fuse the modal dynamic weighted input representation with the corresponding query vector set to generate a preliminary fused representation; Based on the semantic activation features in the preliminary fusion representation, the query vector set is updated in reverse so that the updated query vector set encodes the semantic feedback information in the fusion representation; The updated query vector set is interacted with the modal dynamic weighted input representation again to perform at least one round of fusion enhancement operation based on semantic feedback modulation, forming the final fusion representation and constituting a bidirectional interaction mechanism between the fusion representation and the query representation; The final fused representation is input into the context modeling structure for temporal modeling, and the output is a context fused representation consistent with the time segment order, forming a set of synchronous semantic embedding vectors.

[0026] In this embodiment, generating preliminary question-and-answer and early warning candidate outputs includes: Perform Euclidean distance calculation on the embedding vectors corresponding to adjacent time segments in the synchronous semantic embedding vector set to generate a state change metric sequence; The sequence of state change measurements is compared with a dynamically set state-sensitive threshold to identify a set of time segments in which continuous changes exceed the threshold. Extract the synchronization semantic embedding vector corresponding to the time segment from the potential abnormal state window, and generate the event candidate segment sequence according to its modal source label to construct the event candidate trajectory containing time index and modal path information; The image modal representation, time series modal representation, and text modal representation contained in the event candidate trajectory are modally aligned and concatenated in chronological order to generate an event focus state vector; An event-aware task query vector is constructed based on the event-focused state vector, and semantic matching is performed by combining the synchronous semantic embedding vector in the corresponding time segment to generate the query decoding input vector. The query decoding input vector is used to perform query decoding to obtain preliminary question-answer output and early warning candidate results, and their corresponding time index and modal path are labeled. The preliminary question and answer outputs and the early warning candidate results are combined to form a structured set of preliminary question and answer and early warning candidate outputs; The dynamically set state sensitivity threshold is dynamically generated based on the statistical distribution of the state change amplitude of the synchronous semantic embedding vector set under normal operating conditions. The judgment conditions include: first, statistically analyzing the Euclidean distance between synchronous semantic embedding vectors between adjacent time slices within a preset time period to form a baseline state change sample set; then, calculating the mean and standard deviation of the change amplitude based on this sample set, and setting the sensitivity threshold in the form of "mean-weighted standard deviation", wherein the weight coefficient is adaptively adjusted according to the power plant operation stage, environmental factors, or model evaluation signals; when any subsequent state change amplitude exceeds the dynamic threshold corresponding to the current period, it is judged as a potential state anomaly and enters the event candidate extraction process. The semantic matching operation is performed by calculating the semantic similarity between the event-aware task query vector and the synchronous semantic embedding vector in the corresponding time segment to filter the most relevant information segments. The execution process includes the following steps: First, the task query vector and the synchronous semantic embedding vector are uniformly normalized to ensure spatial consistency; then, a similarity measurement method in the vector space is used to compare the task query vector with the synchronous semantic embedding vector of each candidate time segment one by one; vector pairs with similarity scores higher than a preset semantic relevance threshold are selected according to similarity scores and sorted from high to low; finally, the top-ranked embedding vectors are selected as the semantic alignment targets of the query vector. The query decoding process, based on semantic matching, utilizes the task query vector and its matched synchronous semantic embedding vector to generate conditions. Its execution includes the following steps: The task query vector is used as the conditional control input and concatenated with the matched synchronous semantic embedding vector to form a decoding input sequence; this decoding input sequence is input into a multi-layer decoding structure, which guides attention distribution based on query semantics, focusing on task-related feature dimensions and time segments in the embedding vector; in each decoding layer, the generation state is recursively updated based on the semantic consistency between the generated output and the remaining embedding content until the output termination condition is met; finally, a structured question-and-answer response text or warning label information is output, along with its associated time location and modality source path.

[0027] In this embodiment, the output risk intensity function includes: Select synchronous semantic embedding vectors containing event labels from the synchronous semantic embedding vector set, extract their timestamp information, event type information and modality source path information, and generate time series of labeled events; The time series of labeled events are classified according to event type, and multiple event sets are constructed. The occurrence time of each type of event and the embedding source of the corresponding modal representation are recorded. The modeling structure for self-excited point processes is initialized based on various event sets, including basic excitation rate parameters, event-induced weight parameters, and time decay kernel function structure. For each event type, iterate through its historical event trajectory within the specified time observation window to build a trigger structure graph, which is used to represent the time dependency path and triggering direction between events. The basic excitation rate parameters and induced weight parameters are optimized using a maximum likelihood-based parameter estimation method, and the propagation structure and response delay characteristics between events are updated. Based on the fitted self-excited point process modeling structure, calculate the risk intensity of each event type in the future time period, and output the risk intensity function corresponding to the event type and time index.

[0028] In this embodiment, constructing the weighted query vector includes: extracting the task query vector generated from the preliminary question-and-answer and early warning candidate results, as well as its corresponding time index and modal path information; simultaneously extracting the risk intensity value of the corresponding time segment from the risk intensity function; aligning the task query vector and the risk intensity value in terms of dimension; and forming a joint representation vector through concatenation. During the construction of the joint representation vector, a masking mechanism based on time index alignment is introduced to maintain semantic consistency, and task labels are used to guide the alignment relationship of the joint representation. The concatenated joint representation vector is then used as the weighted query vector.

[0029] In this embodiment, generating enhanced question-and-answer and warning results includes: using a weighted query vector as query input, a joint synchronous semantic embedding vector set as context input, introducing a cross-attention mechanism to establish a dynamic connection between the query and multimodal semantic representation, introducing a selective attention structure guided by event type labels to activate only context regions consistent with the query task semantics, thereby reducing redundant information interference, performing cross-modal attention interaction and residual fusion operations in each decoder layer, matching the query vector with the context representation at the time segment level layer by layer, and generating a decoded response vector with time alignment characteristics, selecting different generation heads according to the task identifier at the output end, performing type mapping on the response vector, and generating structured question-and-answer content and risk classification output.

[0030] In this embodiment, constructing the abnormal causal chain includes: Extract the labeled key event nodes from the enhanced question-and-answer and early warning results, and obtain the time index information and modality source path information corresponding to each key event node to construct a set of key events; Based on the time index information in the key event set, retrieve the event type of the corresponding time segment from the risk intensity function and the event triggering path generated in the self-excited point process modeling structure, construct the event triggering path diagram, and record the triggering direction and response delay of each node; Based on modality source path information, the original modality inputs on which key event nodes depend are located in the synchronous semantic embedding vector set, and the modality combination path participating in question-answer generation is reconstructed to form a question-answer generation path graph, while retaining modality labels and time segment indices; By comparing the event-triggered path graph and the question-and-answer generation path graph at the node level, the temporal dependence, modal consistency and triggering relationship between nodes are compared in turn to identify information missing areas and redundant cross paths in the causal structure and generate a causal consistency difference map. The causal consistency difference map is used for task querying to adjust the modal attention weights and position encoding strategies in the task query vector construction process. Specifically, this includes: enhancing the attention intensity of missing path modal segments and compressing the redundant modal channel span of offset segments. The updated modal attention weights and the adjusted position encoding strategy are synchronized and used in the next round of task query vector generation.

[0031] Example 1: To verify the feasibility of this invention in practice, it was applied to a large coal-fired power plant in the eastern coastal region, where a real-time question-and-answer and early warning system based on a multimodal large model for production data was deployed. This system was used for intelligent monitoring and anomaly early warning tasks in the boiler operating area.

[0032] The operational data sources in this scenario include image data from industrial cameras installed inside the main boiler body and in the fuel delivery channel, boiler operation sequence data (such as temperature, pressure, and fuel flow) collected from the main control system, and structured and unstructured text data such as operation logs and records. After normalization by the preprocessing module, various data types are generated into image modal representation sequences, time series modal representation sequences, and text modal representation sequences. These are then windowed and aligned according to a unified time axis to construct a unified input representation.

[0033] The unified input representation described above is fed into the improved Perceiver IO model, where the representations of each modality are fused to form a time-sequential set of synchronous semantic embedding vectors. Internally, the model enhances the semantic consistency and modal alignment capabilities of multimodal collaboration at the semantic level through a multi-round query update mechanism and a modal attention mechanism.

[0034] During operation, the system continuously monitors the state changes of the synchronous semantic embedding vector set. It calculates the state change measurement sequence based on the Euclidean distance between embedding vectors in continuous time segments and identifies potential anomaly windows based on dynamic state sensitivity thresholds formed by operational condition statistics. Upon detecting a decrease in image brightness in the boiler's main combustion chamber area, a sudden drop in flow sensor output, or the appearance of the keyword combination "combustion anomaly" in the maintenance log, the system automatically focuses on the data within that time period, constructs an event-aware task query vector, performs semantic matching, executes preliminary question-and-answer generation, and outputs the question: "Is there a trend of incomplete combustion in the main combustion chamber?" and a preliminary warning: "Decreased combustion efficiency, suspected coal pulverizer nozzle blockage."

[0035] Subsequently, the system extracts event time series based on the previously labeled historical boiler anomaly event set, and establishes an induced relationship network between events using the Hawkes self-excited point process model. The model calculates the current risk intensity function based on historical event chains such as "coal pulverized nozzle blockage" inducing "unstable boiler combustion" and further evolving into "abnormal rise in flue gas temperature in the main combustion chamber," generating intensity estimation curves for multiple types of events within the prediction window.

[0036] The system merges the initially generated task query vector with the aforementioned risk intensity function to construct a weighted query vector. It then decodes the vector using a cross-attention mechanism and context embedding to output an enhanced warning result: "It is expected that the flue gas temperature in the main combustion chamber will suddenly rise within the next 15 minutes, triggering the secondary cooling mechanism."

[0037] The system further extracts key event nodes from the enhanced early warning results and traces their modal source paths and time index information. Combined with the event triggering paths in the Hawkes model, it performs structural comparison, identifies the missing coal powder wear anomaly factors in the current inference path, and feeds them back to the task query generation module to adjust the modal attention configuration of subsequent queries and improve the completeness of subsequent responses.

[0038] The table below shows a comparison of the system's performance with that of a manual inspection system in terms of anomaly early warning within three weeks of actual deployment: Table 1. Comparison of the effectiveness of system and manual early warning systems.

[0039] As shown in Table 1, the real-time question-and-answer and early warning method for power plant production data based on a multimodal large model demonstrates superior real-time performance and accuracy compared to traditional manual methods in several key anomaly detection scenarios. Taking "coal pulverizer pipeline blockage" as an example, manual early warnings are delayed by up to 24 minutes, while the system only requires 8 minutes, achieving an early warning rate of 66.7%. This indicates that the system can capture abnormal combustion signals in advance through joint monitoring of images and time series data. For events such as "abnormal fan vibration" and "increased flue gas temperature," the system completes early warnings in 10 minutes and 5 minutes respectively, shortening the time by 20 minutes and 13 minutes compared to manual methods, with early warning rates exceeding 60%. This reflects the system's excellent response performance to multimodal data fusion monitoring in mechanical vibration and thermal energy management. The early warning rates for events such as "decreased heat exchanger efficiency" and "local oxygen deficiency in the furnace" also reach 55.0% and 60.7% respectively, indicating that the system also possesses a certain ability to perceive and identify changes in operating efficiency and oxygen distribution. It is worth noting that the system maintained a false alarm rate below 5% throughout the entire process, with a maximum of only 5.0% and a minimum of 3.6%. This indicates that the present invention effectively suppressed redundant warnings and unrelated false alarms by combining Hawkes process modeling with semantic question answering mechanisms, balancing sensitivity and accuracy. This ability to integrate perception, evolutionary prediction, and structured response has been fully validated in the actual deployment of multiple operating subsystems.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A real-time question-and-answer and early warning method for power plant production data based on a multimodal large model, characterized in that, include: Collect images, time series, and text data during the operation of the power plant, perform preprocessing, and generate multimodal data fragment sequences; Input the multimodal data segment sequence to The model fuses information from various modalities and outputs a set of synchronous semantic embedding vectors. Based on the synchronous semantic embedding vector set, state changes are monitored, abnormal evolution trends are identified, task query vectors are generated, and preliminary question answering and early warning candidate outputs are generated by combining synchronous semantic embedding vectors. Extract the time series of labeled events from the synchronous semantic embedding vector set, input them into the Hawkes process model, calculate the event induced relationship and output the risk intensity function; A weighted query vector is constructed by combining the risk intensity function with the preliminary question answer and early warning candidate outputs. The input weighted query vector and the synchronous semantic embedding vector set are jointly decoded to generate enhanced question-answering and warning results, and the corresponding modal path and time location information are recorded. By comparing and analyzing the key event nodes in the enhanced question-and-answer and early warning results with the triggering paths of the Hawkes process model, an abnormal causal chain is constructed and fed back to the query generation mechanism to optimize the next round of response.

2. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 1, characterized in that, The preprocessing includes image format resampling, normalization, semantic vector mapping, unified timestamp generation, sliding window segmentation, and time segment alignment and encapsulation.

3. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 1, characterized in that, The set of output synchronization semantic embedding vectors includes: Perform channel normalization, resolution standardization, and feature extraction on the image modal input data to generate an image modal representation sequence; The time series modal input data is processed by sliding window segmentation, amplitude normalization and time series encoding to generate a time series modal representation sequence; The text modal input data is processed by text segmentation, word embedding mapping and time position encoding to generate a text modal representation sequence. Image modal representation sequences, time series modal representation sequences, and text modal representation sequences are concatenated in a unified time order to construct a unified input representation; Input the unified input representation to The model performs a fusion encoding operation to generate a fusion semantic representation aligned with the input modality; The fused semantic representations are output sequentially according to time segments to form a set of synchronous semantic embedding vectors.

4. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 3, characterized in that, The The model includes the following processing steps: Add time index information and modality identification information to the image modality representation sequence, time series modality representation sequence and text modality representation sequence contained in the unified input representation to generate multimodal input embedding; Modal correlation analysis is performed on the multimodal input embedding, and modal attention weights are constructed at each time slice, where the modal attention weights are calculated by the following formula: ； in, Indicates the first Each modality in time slice Embedded representation below, This represents the query vector within that time slice. Indicate modal attention weights, Indicates the first Each modality in time slice Embedded representation below; Modal attention weights are applied to the multimodal input embedding to generate a dynamically weighted modal input representation; Initialize the first set of learnable query vectors, and fuse the modal dynamic weighted input representation with the corresponding query vector set to generate a preliminary fused representation; Based on the semantic activation features in the preliminary fusion representation, the query vector set is updated in reverse so that the updated query vector set encodes the semantic feedback information in the fusion representation; The updated query vector set is interacted with the modal dynamic weighted input representation again to perform at least one round of fusion enhancement operation based on semantic feedback modulation, forming the final fusion representation and constituting a bidirectional interaction mechanism between the fusion representation and the query representation; The final fused representation is input into the context modeling structure for temporal modeling, and the output is a context fused representation consistent with the time segment order, forming a set of synchronous semantic embedding vectors.

5. The method for real-time question answering and early warning of power plant production data based on a multimodal large model according to claim 1, characterized in that, The generation of preliminary question-and-answer and early warning candidate outputs includes: Perform Euclidean distance calculation on the embedding vectors corresponding to adjacent time segments in the synchronous semantic embedding vector set to generate a state change metric sequence; The sequence of state change measurements is compared with a dynamically set state-sensitive threshold to identify a set of time segments in which continuous changes exceed the threshold. Extract the synchronization semantic embedding vector corresponding to the time segment from the potential abnormal state window, and generate the event candidate segment sequence according to its modal source label to construct the event candidate trajectory containing time index and modal path information; The image modal representation, time series modal representation, and text modal representation contained in the event candidate trajectory are modally aligned and concatenated in chronological order to generate an event focus state vector; An event-aware task query vector is constructed based on the event-focused state vector, and semantic matching is performed by combining the synchronous semantic embedding vector in the corresponding time segment to generate the query decoding input vector. The query decoding input vector is used to perform query decoding to obtain preliminary question answer output and early warning candidate results, and the corresponding time index and modal path are labeled. The preliminary question and answer outputs and the early warning candidate results are combined to form a structured set of preliminary question and answer and early warning candidate outputs.

6. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 1, characterized in that, The output risk intensity function includes: Select synchronous semantic embedding vectors containing event labels from the synchronous semantic embedding vector set, extract their timestamp information, event type information and modality source path information, and generate time series of labeled events; The time series of labeled events are classified according to event type, and multiple event sets are constructed. The occurrence time of each type of event and the embedding source of the corresponding modal representation are recorded. The modeling structure for self-excited point processes is initialized based on various event sets, including basic excitation rate parameters, event-induced weight parameters, and time decay kernel function structure. For each event type, iterate through its historical event trajectory within the specified time observation window to build a trigger structure graph; The basic excitation rate parameters and induced weight parameters are optimized using a maximum likelihood-based parameter estimation method, and the propagation structure and response delay characteristics between events are updated. Based on the fitted self-excited point process modeling structure, calculate the risk intensity of each event type in the future time period, and output the risk intensity function corresponding to the event type and time index.

7. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 1, characterized in that, The construction of the weighted query vector includes: extracting the task query vector generated from the preliminary question-answering and early warning candidate results, as well as its corresponding time index and modal path information; extracting the risk intensity value of the corresponding time segment from the risk intensity function; aligning the task query vector and the risk intensity value in terms of dimension; and forming a joint representation vector through concatenation. In the process of constructing the joint representation vector, a masking mechanism based on time index alignment is introduced to maintain semantic consistency, and the task label is used to guide the alignment relationship of the joint representation. The concatenated joint representation vector is used as the weighted query vector.

8. The method for real-time question answering and early warning of power plant production data based on a multimodal large model according to claim 1, characterized in that, The generation of enhanced question-and-answer and early warning results includes: using a weighted query vector as the query input, a joint synchronous semantic embedding vector set as the context input, introducing a cross-attention mechanism to establish a dynamic connection between the query and the multimodal semantic representation, introducing a selective attention structure guided by event type labels to activate only the context region consistent with the query task semantics, performing cross-modal attention interaction and residual fusion operations in each decoder layer, matching the query vector with the context representation at the time segment level layer by layer, and generating a decoded response vector with time alignment characteristics. At the output end, different generation heads are selected according to the task identifier, and the response vector is typified and mapped to generate structured question-and-answer content and risk classification output.

9. The real-time question-and-answer and early warning method for power plant production data based on a multimodal large model according to claim 1, characterized in that, The construction of the abnormal causal chain includes: Extract the labeled key event nodes from the enhanced question-and-answer and early warning results, and obtain the time index information and modality source path information corresponding to each key event node to construct a set of key events; Based on the time index information in the key event set, retrieve the event type of the corresponding time segment from the risk intensity function and the event triggering path generated in the self-excited point process modeling structure, construct the event triggering path diagram, and record the triggering direction and response delay of each node; Based on modality source path information, the original modality inputs on which key event nodes depend are located in the synchronous semantic embedding vector set, and the modality combination path participating in question-answer generation is reconstructed to form a question-answer generation path graph, while retaining modality labels and time segment indices; By comparing the event-triggered path graph and the question-and-answer generation path graph at the node level, the temporal dependence, modal consistency and triggering relationship between nodes are compared in turn to identify information missing areas and redundant cross paths in the causal structure and generate a causal consistency difference map. The causal consistency difference map is used for task querying to adjust the modal attention weights and position encoding strategies in the task query vector construction process; The updated modal attention weights and the adjusted position encoding strategy are synchronized and used in the next round of task query vector generation.

Images