Signal text data generation and bimodal fusion continuous learning method and system under long-tail distribution

Abstract

The invention belongs to the technical field of petrochemical equipment, and discloses a signal text data generation and bimodal fusion continuous learning method under long-tail distribution, and the method comprises the steps: constructing a signal generation diffusion model based on a Transform architecture and a Longform encoder, then evaluating the semantic similarity between the generated signal and a real text, and constructing a text generation method based on a semantic screening mechanism and a large model; firstly, a continuous learning model of signal text bimodal fusion is constructed, then a fault diagnosis verification system is constructed, and the continuous learning model is deployed to the verification system for verification. The method provides a new research thought for petrochemical large unit fault diagnosis sample expansion, and promotes the research of data expansion in a long-tail distribution scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to but is not limited to the technical field of petrochemical equipment, and particularly relates to a long-tail distribution signal text data generation and bimodal fusion continuous learning method and system. BACKGROUND

[0002] The petrochemical industry is an important support for national energy security and industrial chain stability. Under the guidance of the national strategy, Guangdong is accelerating the deep integration of green petrochemicals and intelligent manufacturing. Large petrochemical equipment (such as compressors, steam turbines, and fans) is the core unit of production devices, with complex operating conditions, high load, and high risk. Once a fault occurs, it may not only cause the device to shut down for a long time and cause hundreds of thousands of yuan in losses, but also may cause structural damage and even safety accidents. Therefore, improving the intelligent diagnosis capability of petrochemical large units has become a key requirement for high-quality development of the industry. Currently, multiple vibration sensors have been widely deployed in the field, which can collect rich time series signals and provide basic data for intelligent diagnosis. However, existing diagnosis methods still have obvious deficiencies in dynamic adaptability and multi-modal fusion capability.

[0003] Existing research mainly focuses on fault diagnosis methods based on signal processing, expert knowledge, and deep learning. Methods based on signal processing can extract features through time domain, frequency domain, and time-frequency domain analysis, but rely heavily on human experience; methods based on expert knowledge use rule bases, fuzzy logic, Bayesian networks, etc. to achieve diagnosis, but have limited generalization; deep learning methods have strong feature learning capabilities, but usually rely on single modal data and are easily affected by operating condition changes and long-tail distribution of samples, resulting in unstable diagnosis results. At the same time, multi-modal learning has made progress in the fields of images, texts, and medicine, but its application in petrochemical equipment fault diagnosis is still limited. In particular, cross-modal alignment and fusion methods for signals and texts are not mature, data long-tail problems are prominent, and models lack continuous learning and optimization mechanisms, making it difficult to cope with complex operating scenarios throughout the life cycle of industrial equipment.

[0004] (I) The closest prior art patent

[0005] CN115112400A discloses a diagnosis method and system for concurrent faults of petrochemical large units. The method relies on simulation to generate fault condition samples and real scene monitoring data, trains convolutional neural network and feedforward neural network models, and fuses the outputs of the two models through D-S evidence theory to improve fault diagnosis accuracy.

[0006] (II) Main technical problems of the prior art

[0007] Dependence on simulation and single-modal sensing: This method mainly generates multiple concurrent fault conditions through simulation software to construct the training set, and the actual available real fault samples are still scarce, which belongs to typical single-modal vibration or detection signal processing, lacks the integration strategy of multi-modal data such as text logs and maintenance records, and is difficult to form a comprehensive diagnosis basis.

[0008] Model generalization and long-tail sample processing are not enough: When the amount of real fault samples is insufficient, especially when the rare fault conditions are limited, the model trained based on traditional deep learning models (such as CNN+feedforward network) is prone to perform poorly on unseen faults, and lacks mechanisms for automatically enhancing long-tail samples, generating and continuously optimizing semantic consistency. SUMMARY

[0009] In view of the problems existing in the prior art, the present application provides a signal-text data generation and double-modal fusion continuous learning method under long-tail distribution.

[0010] The present application is implemented in the following way: a signal-text data generation and double-modal fusion continuous learning method under long-tail distribution, which comprises:

[0011] S1: Construct a signal generation diffusion model based on the Transformer architecture and the Longformer encoder, and then evaluate the semantic similarity of the generated signal and the real text, and construct a text generation method based on the semantic screening mechanism and the large model;

[0012] S2: First, construct a continuous learning model of signal-text double-modal fusion, then construct a fault diagnosis verification system, and deploy the continuous learning model to the verification system for verification.

[0013] Further, in the S1, the signal generation diffusion model based on the Transformer architecture and the Longformer encoder is constructed, specifically including:

[0014] First, a long-period sequence modeling encoder based on Longformer is constructed to capture long-time dependency information; then, trend, period and residual information are fused to improve modeling capability; finally, a multi-objective loss function and a sampling method are designed to realize the step-by-step generation of high-quality signals; the specific technical route of each module is as follows:

[0015] 1) Long-period sequence modeling encoder based on Longformer

[0016] For the super-long time sequence dependence characteristics existing in the evolution of large unit faults, the length of large unit fault signal data is generally more than one thousand steps. In this model, Longformer with sparse local-global hybrid attention structure is used to replace the original Transformer Encoder as the encoder module. For long-period signal input data x0∈R B×T , where B is the batch size and T is the time step number.

[0017]

[0018] The input data with position information is obtained , where E pos is the position embedding, which encodes the position of each time step. The context feature h output by the encoder is represented as:

[0019]

[0020] 2) Trend extraction and frequency domain synthesis method in three-branch decoding structure

[0021] The decoder uses a three-branch structure of trend, period, and residual to model the output. The trend branch uses a polynomial regression module to extract the smooth and slowly changing long-term trend, and the extraction formula is as follows:

[0022]

[0023] , where is the average value of the output of the i-th decoding block, is the network parameter related to trend extraction, C is the polynomial basis, and p is the order of the polynomial; the period term is represented in the frequency domain by Fourier transform, and the high-energy frequency components are selected for reconstruction in the Fourier synthesis layer:

[0024]

[0025] , where F represents the Fourier transform, argTopK is used to obtain the top K largest amplitudes, and K is a hyperparameter. , respectively, are the phase and amplitude of the k-th frequency of the input signal after discrete Fourier transform, , represents the Fourier frequency corresponding to the index , and TC represents the time coordinate, , represents the corresponding item of the corresponding conjugate component, and the final output formula is as follows:

[0026]

[0027] , where R is the residual term.

[0028] 3) Diffusion model training strategy of multi-objective loss and guided sampling

[0029] The training stage of the proposed diffusion model, the model optimization target is to directly reconstruct the original signal, combined with time domain and frequency domain loss, the total loss function of the training stage is as follows:

[0030]

[0031] Where λ1, λ2 are hyperparameters, used to balance the importance of time domain and frequency domain, the time domain and frequency domain loss is as follows:

[0032]

[0033] In the sampling process of the proposed diffusion model, the model is gradually back-sampled and combined with the conditional guidance error term to adjust the direction:

[0034]

[0035] Where is the mean part of the reverse sampling of DDPM, η controls the guidance strength, is the gradient of the conditional error, guiding the sampling to converge towards a specific working condition.

[0036] Further, in the S1, the semantic screening mechanism specifically comprises:

[0037] First, calculate the semantic similarity between each signal graph and all expert texts, and screen out the graphs without significant semantic association according to the set threshold; if the similarity of a graph to all expert texts is lower than the threshold, it is determined that the corresponding signal data is unqualified data;

[0038] The calculation of semantic similarity is based on the pre-trained CLIP model; specifically, all expert texts are encoded into vector form and uniformly saved in a vector list; then, the signal graph to be screened is encoded to obtain the corresponding graph vector, and the cosine similarity is calculated with the expert text vector in turn; during the calculation, once a similarity exceeds the set threshold, it is considered that the graph has semantic association and does not need to be compared with subsequent expert texts; the similarity calculation formula is as follows:

[0039]

[0040] Where is the normalized graph vector, and is the normalized text vector.

[0041] Further, in the S1, the text generation method of the large model specifically comprises:

[0042] For multiple signal graphs under the same fault category, first, the visual encoder ViT-L / 14 of the pre-trained CLIP model is used to extract the deep semantic features of the signal graph, specifically as follows:

[0043]

[0044] Subsequently, the extracted image features are dimensionally transformed by a linear projection layer to be consistent with the text feature dimensions in the large model, represented as:

[0045]

[0046] where W is the weight parameter of the projection layer, the obtained image features are spliced with the text features from the real expert diagnosis cases, and the spliced fusion features are input into the large model to obtain the generated text case data.

[0047] Further, the S2 specifically includes:

[0048] First, the dual-modal diagnosis model is deployed to the petrochemical large unit fault diagnosis verification system; once the model identifies a potential fault, the system will immediately send the diagnosis result to the enterprise; the enterprise technical personnel will confirm the fault condition by checking the unit operation log or on-site investigation according to the model prompt; when the enterprise confirms that the unit has indeed failed, its corresponding multi-modal data sample, including the signal sequence and the diagnosis text, will be sorted and stored in the abnormal sample database for subsequent training, and the collection process is represented as follows:

[0049]

[0050] where D new is the newly collected sample data, is the i-th signal sequence, is the text description corresponding to the i-th signal sequence, y i is the manually confirmed fault type label of the i-th signal;

[0051] In order to continuously improve the fault diagnosis capability of the model during the operation of the large unit, the platform will periodically use the accumulated abnormal data to continuously train the dual-modal model; since these new data generally have long-tail distribution characteristics, i.e. most belong to a small number of classes or novel faults, direct training may cause the model to overfit on some faults, therefore, the dual-modal generation model is needed to preprocess the newly collected data, represented as follows:

[0052]

[0053] where D train is the new data for training, D new is the normally distributed data, is the tail data after expansion using the generation model, G θ is the generation model, D tail is the tail data;

[0054] Using the preprocessed new data D train , a joint loss function composed of distillation regular term and self-adjusting weight cross-entropy is adopted, and incremental training is carried out on the basis of the original model M old , and the joint loss function is as follows:

[0055] L total =α·L distill +β·L ce (17)

[0056] Wherein, L distill is the knowledge distillation loss, L ce is the cross-entropy loss, and alpha and beta are hyperparameters; the role of distillation loss is to distill the knowledge of the old model, the teacher model, to the new model, the student model, thereby effectively preserving historical knowledge and mitigating catastrophic forgetting, and the loss function is represented as follows:

[0057]

[0058] Wherein z new is the output feature of the new model, z old is the output feature of the old model, represents the square of L2 norm; the role of the cross-entropy loss function L ce is to enable the new model to learn from new data, and by minimizing the loss function, the prediction of the new model is as consistent as possible with the true label of the new data, and the loss function is represented as follows:

[0059] L ce =-log(p y ) (19)

[0060] Wherein p y represents the probability that the model predicts the true class y;

[0061] By minimizing the above joint loss, the model M new can adapt to new data and effectively resist catastrophic forgetting and forget the knowledge of the old model; after training, the updated model M new will replace the old model on the line, and the model will be used as a new teacher model for distillation training in the next training.

[0062] In combination with the above technical solutions and the technical problems solved, the technical solutions protected by the present application have the following advantages and positive effects:

[0063] The application is aimed at the above-mentioned pain points, and proposes a signal-textual dual-modal alignment fusion enhancement method and a continuous learning model suitable for fault diagnosis of petrochemical large units. On the one hand, the dual-modal data generation method under the condition of long-tail distribution is studied, the diffusion model is combined with the Transformer and Longformer architecture to generate high-quality long-period signals, and the semantic screening mechanism and multi-modal large model are used to generate fault texts, realizing the enhancement of dual-modal tail samples. On the other hand, a continuous learning model based on knowledge distillation loss and cross-entropy loss is constructed, so that the model can adapt to new data while avoiding forgetting old knowledge, thereby maintaining long-term diagnostic effectiveness. Finally, the method is deployed to the "Guangyou-Shen Gu Yun" collaborative diagnosis platform, and based on the operation data of more than 1500 petrochemical large units in the country, the verification is carried out, forming an intelligent diagnosis system that takes into account accuracy, robustness and generalizability, systematically solving the two core problems of long-tail distribution sample shortage and continuous learning optimization.

[0064] (1) A new method of signal-textual dual-modal sample generation and enhancement based on diffusion model and semantic screening mechanism under long-tail distribution is proposed. The method first constructs a signal generation diffusion model based on Transformer architecture and Longformer encoder, then designs a semantic screening mechanism to further obtain high-quality generated signal data, and uses a large model to generate the corresponding text of the signal data.

[0065] The method provides a new research idea for sample expansion of petrochemical large unit fault diagnosis, and promotes the research of data expansion under long-tail distribution

[0066] (2) A signal-textual dual-modal fusion diagnosis continuous learning model for petrochemical large unit operation is proposed. The method first constructs a dual-modal fusion fault diagnosis continuous learning model, and then constructs a petrochemical large unit fault diagnosis verification platform. The method improves the continuous learning and verification system of petrochemical large unit fault diagnosis, and improves the stability and learning ability of the model in actual deployment.

[0067] Secondly, the technical scheme of the present application has significant economic benefits and commercial value. By improving the identification accuracy of long-tail rare faults, early warning can be realized in large equipment in petrochemical, energy and other industries, and each unit of the machine can save hundreds of thousands to millions of yuan in a single fault prevention, directly reducing the economic losses caused by accidents. At the same time, the generation enhancement and continuous learning technology significantly reduces the cost of manual annotation and sample collection, especially solving the pain point of long-term difficulty in obtaining tail samples. In terms of business, the present application relies on the "Guangyou-Shen Guyun" platform and has landed on more than 1500 units. Once popularized to the power, metallurgy and other related industries, a nationwide intelligent diagnosis network can be quickly formed to promote industrial application. In addition, the derived fault knowledge base service, cloud diagnosis SaaS and incremental learning engine authorization will also support enterprises to form core competitiveness in the cross-modal large model and industrial intelligent diagnosis market.

[0068] The present application fills an important technical gap in the international field. Existing methods are mostly limited to single-modal enhancement (such as signal resampling or GAN generation), lacking the semantic consistency guarantee of "signal-text" dual modalities; and the research of continuous learning is mainly concentrated in the field of image or natural language, with almost no mature solution for industrial long-tail fault diagnosis. The present application realizes the consistency of time domain and frequency domain under ultra-long sequence by proposing a three-branch Longformer+diffusion generation structure; realizes the consistent augmentation of signal atlas and diagnosis sentence by CLIP cross-modal screening and large model text generation; and further establishes an online closed-loop dual-modality continuous learning system, which is the first to realize the integration of "generation-screening-fusion-evolution" in the industrial field, truly filling the technical gap at home and abroad.

[0069] The present application successfully solves the key problems that have long been desired but not achieved in the industry. Firstly, in the scenario of extremely scarce data under long-tail distribution, the present application forms a systematic supplementary means through generation, screening and text matching mechanism; secondly, in the efficient modeling of ultra-long sequence, the previous models either have too high computational complexity or ignore long-range dependencies, while the present application introduces Longformer sparse attention and three-branch decoding, realizing controllable generation on a thousand-step time sequence for the first time; thirdly, in the cross-modal alignment and semantic reliability problem, the previous enhanced samples often have "correct waveform but wrong semantics", while the present application realizes the consistency of semantics and physics through CLIP screening and large model text generation; fourthly, in the problem of anti-forgetting in continuous learning, the present application combines knowledge distillation and generation enhancement to establish an anti-forgetting mechanism that can be stably improved, providing a guarantee for the long-term effective operation of industrial diagnosis models.

[0070] The application also overcomes the long-standing technical bias in the industry, embodying significant creativity. In view of the bias of only paying attention to the head class and ignoring the tail class, the application realizes accurate diagnosis of rare classes through tail generation and enhancement; in view of the bias that single-modal enhancement can solve the problem, the application models signals and texts uniformly and gives full play to the cross-modal collaborative value; in view of the inherent thinking that continuous learning must rely on complete historical data, the application replaces the original data through knowledge distillation and generation enhancement, breaking through the storage and privacy constraints; in view of the doubt that generated samples are not suitable for high-risk industrial scenarios, the application verifies the reliability and diagnostic value of generated data through time-frequency joint constraint and CLIP semantic verification, proving that generated samples can be fully applied to industrial intelligent diagnosis, thereby breaking the industry's inherent technical prejudice. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a signal generation diffusion model technical route based on the Transformer architecture and the Longformer encoder provided by the embodiment of the application;

[0072] Figure 2 a text generation technical route based on a semantic screening mechanism and a large model;

[0073] Figure 3 a signal-text dual-modal fusion continuous learning model technical route. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0075] In the traditional fault diagnosis scene, the data problem of long-tail distribution always restricts the application value of the model in the actual industrial scene. In the process of running the large unit, the common fault samples are relatively rich, but the rare or novel faults only occur in a small number of working conditions, which leads to excessive fitting of the model in the training stage to the mainstream class, and lack of effective representation to the tail class. This uneven distribution of samples directly causes misjudgment and omission of the diagnosis model after deployment, especially in the petrochemical, power and other industries, which may cause the delay response of the running state and bring safety and economic risks. Therefore, the key to solving the long-tail distribution problem lies in constructing a technical path that can generate, expand and screen rare class samples, and combining continuous learning to maintain the stability of the model in the long-term running.

[0076] For long time series signal modeling, existing models based on standard Transformer architecture have defects such as excessive attention calculation overhead and difficulty in capturing global dependencies in ultra-long sequences. The vibration signals of large units and electrical monitoring data often exceed the level of thousands of steps, and traditional convolution or short sequence recursive methods cannot completely capture the correlation across cycles. This method uses the Longformer sparse attention mechanism to realize hybrid modeling of local windows and global selected points, so that the integrity and controllability of features can be maintained in long time series. Through the branch decoding structure of trend items, periodic items and residual items, smooth changing trends, dominant frequency domain components and fine-grained disturbance features can be extracted respectively, so as to realize multi-scale reconstruction of complex industrial signals. This design provides physical rationality for generating realistic signals under rare working conditions.

[0077] After the signal generation is completed, how to ensure its semantic consistency with the actual fault text becomes a new technical challenge. Traditional data enhancement often stays at the statistical feature level and lacks cross-modal semantic constraints, which can easily generate redundant samples that are not consistent with the diagnostic corpus. This method introduces a semantic screening mechanism based on CLIP to perform deep semantic comparison between the generated signal atlas and the expert case text. The cosine similarity calculation is used to determine whether the sample has diagnostic significance. By threshold discrimination, irrelevant generated data is removed to ensure that the data entering the training has interpretability and actual engineering value, thereby avoiding misleading cases in industrial verification.

[0078] In the text generation link, traditional small-scale text generation models lack correspondence with signal features, resulting in limited diagnostic value of generated text. This method uses a visual encoder to extract deep semantic features of the signal atlas, which are projected linearly and then input into a large-scale language model with real diagnostic text to generate diagnostic cases strongly related to specific working conditions. This mechanism not only supplements the text description of tail samples, but also improves the semantic alignment between signals and text, forming multi-modal augmented data that conforms to the industrial diagnostic context. This cross-modal generation method breaks through the limitations of previous single-modal augmentation, enabling the model to capture knowledge mapping at both signal and language levels during the learning process.

[0079] In the continuous learning link, existing methods generally have the problem of catastrophic forgetting. Newly collected data often covers historical knowledge when the model is updated, leading to fluctuations and instability in diagnostic ability. This method introduces a distillation regularization term in incremental training, which fixes historical knowledge in the new model structure through feature transfer from the old model; at the same time, it combines a self-adjusting weight cross-entropy loss, enabling the new model to efficiently learn new fault samples under a long-tail distribution. This joint optimization mechanism not only ensures the long-term stability of diagnostic performance, but also alleviates the uncontrollable risk of model updating in industrial scenarios.

[0080] Ultimately, at the industrial application level, this method is embedded in the real-time diagnostic system of large petrochemical units. When the system detects a potential fault and triggers a diagnostic result, on-site technicians can verify and store the actual label along with the generated sample into the database. With accumulated operating time, the system continuously improves itself through cycles of generating and filtering tail samples, text enhancement, and continuous learning and updating, achieving a transformation from scarce data to reliable diagnosis. This working principle not only enhances the adaptability of the intelligent diagnostic system to long-tailed distribution problems but also provides long-term, sustainable, and intelligent protection for the safe operation of complex industrial equipment.

[0081] This invention provides a method for continuous learning of signal text data generation and dual-modal fusion under long-tailed distribution, the method comprising:

[0082] S1: Construct a signal generation and diffusion model based on the Transformer architecture and Longformer encoder, then evaluate the semantic similarity between the generated signal and the real text, and construct a text generation method based on semantic filtering mechanism and large model;

[0083] S2: First, construct a continuous learning model that fuses signal and text in a dual-modal manner. Then, construct a fault diagnosis verification system and deploy the continuous learning model to the verification system for verification.

[0084] like Figure 1 As shown, in S1, the signal generation and diffusion model based on the Transformer architecture and Longformer encoder is constructed, specifically including:

[0085] First, a long-period sequence modeling encoder based on Longformer is constructed to capture long-term dependency information. Then, trend, periodic, and residual information are fused to improve modeling capabilities. Finally, a multi-objective loss function and sampling method are designed to achieve the stepwise generation of high-quality signals. The specific technical roadmap for each module is as follows:

[0086] 1) Long-period sequence modeling encoder based on Longformer

[0087] To address the extremely long time-series dependencies in the fault evolution of large generating units, where fault signal data typically exceeds one thousand steps, this model replaces the original Transformer Encoder with a Longformer model featuring a sparse local-global hybrid attention structure. For long-period signal input data x0∈R... B×T Where B is the batch size and T is the number of time steps;

[0088]

[0089] Obtaining input data with position information where E pos is the position embedding for encoding the position of each time step, the context feature h of the encoder output is represented as:

[0090]

[0091] 2) Trend extraction and frequency domain synthesis method in three-branch decoding structure

[0092] The decoder uses a three-branch structure of trend, period and residual to model the output, where the trend branch uses a polynomial regression module to extract the smooth, slowly varying long-term trend, and the extraction formula is as follows:

[0093]

[0094] represents the average value of the i-th layer decoding block output, is the network parameter related to trend extraction, C is the polynomial basis, and p is the order of the polynomial; the period term is represented in the frequency domain by Fourier transform, and high-energy frequency components are selected in the Fourier synthesis layer for reconstruction:

[0095]

[0096] where F represents the Fourier transform, argTopK is used to obtain the top K largest amplitudes, and K is a hyperparameter; respectively, the phase and amplitude of the k-th frequency of the input signal after discrete Fourier transform, represents the Fourier frequency corresponding to the index TC represents the time coordinate, represents the corresponding item of the corresponding conjugate component, and the final output formula is as follows:

[0097]

[0098] where R is the residual term;

[0099] 3) Diffusion model training strategy of multi-objective loss and guided sampling

[0100] In the training stage of the proposed diffusion model, the optimization goal of the model is to directly reconstruct the original signal, and the time domain and frequency domain losses are combined, and the total loss function of the training stage is as follows:

[0101]

[0102] where λ1, λ2 are hyperparameters for balancing the importance of time domain and frequency domain, and the time domain and frequency domain losses are as follows:

[0103]

[0104] In the sampling process of the proposed diffusion model, the model is gradually back-sampled and combined with the conditional guidance error term to adjust the direction:

[0105]

[0106] wherein is the reverse sampling mean part of the DDPM, η controls the guidance intensity, is the gradient of the conditional error, guiding the sampling to converge towards a specific working condition.

[0107] As Figure 2 shown, in the S1, the semantic screening mechanism specifically includes:

[0108] First, the semantic similarity between each signal graph and all expert texts is calculated, and the graphs with no significant semantic association are screened out according to the set threshold; if the similarity between a graph and all expert texts is lower than the threshold, the corresponding signal data is determined as unqualified data;

[0109] The calculation of semantic similarity is based on the pre-trained CLIP model; specifically, all expert texts are encoded into vector form and saved uniformly in a vector list; then, the signal graph to be screened is encoded to obtain the corresponding graph vector, and the cosine similarity is calculated with the expert text vector one by one; during the calculation, once a similarity exceeds the set threshold, it is considered that the graph has semantic association and does not need to be compared with subsequent expert texts; the calculation formula of the similarity is as follows:

[0110]

[0111] wherein, is the normalized graph vector, and is the normalized text vector.

[0112] In the S1, the text generation method of the large model specifically includes:

[0113] For multiple signal graphs under the same fault category, first, the visual encoder ViT-L / 14 of the pre-trained CLIP model is used to extract the deep semantic features of the signal graph, specifically as follows:

[0114]

[0115] Then, the image features extracted are dimensionally transformed through a linear projection layer to make them consistent with the text feature dimensions in the large model, represented as:

[0116]

[0117] wherein is the weight parameter of the projection layer, the obtained image features are spliced with the text features from the real expert diagnosis cases, and the spliced fusion features are input into to obtain generated text case data.

[0118] As shown in the figure, S2 specifically includes: Figure 3

[0119] First, the dual-modal diagnosis model is deployed to a petrochemical large unit fault diagnosis verification system; once the model identifies a potential fault, the system will immediately send the diagnosis result to the enterprise; the enterprise technical personnel confirm the fault condition by checking the unit operation log or on-site investigation according to the model prompt; when the enterprise confirms that the unit has indeed failed, the corresponding multi-modal data sample of the unit, including the signal sequence and the diagnosis text, will be sorted and stored in the abnormal sample database for subsequent training, and the collection process is as follows:

[0120]

[0121] wherein D new is the newly collected sample data, is the i-th signal sequence, is the text description corresponding to the i-th signal sequence, y i is the fault type label of the i-th signal confirmed by the human being;

[0122] In order to continuously improve the fault diagnosis capability of the model during the operation of the large unit, the platform will periodically use the accumulated abnormal data to continuously train the dual-modal model; since these new data generally have the long-tail distribution characteristic, i.e., most of them belong to a small number of classes or novel faults, direct training may cause the model to overfit on some faults, so it is necessary to use the dual-modal generation model to pre-process the newly collected data, which is represented as follows:

[0123]

[0124] wherein D train is the new data used for training, D new is the normally distributed data, is the tail data expanded by using the generation model, G θ is the generation model, D tail is the tail data;

[0125] Using the pre-processed new data D tarain , a joint loss function composed of a distillation regularization term and a self-adjusting weight cross-entropy is used to perform incremental training on the original model M old , and the joint loss function is as follows:

[0126] L total = α·L​distill + beta * L ce (17)

[0127] wherein L distill is the knowledge distillation loss, L ce is the cross-entropy loss, and alpha and beta are hyperparameters; the role of the distillation loss is to distill the knowledge of the old model, the teacher model, to the new model, the student model, thereby effectively preserving historical knowledge and mitigating catastrophic forgetting; the loss function is represented as follows:

[0128]

[0129] wherein z new is the output feature of the new model, z old is the output feature of the old model, represents the square of the L2 norm; the role of the cross-entropy loss function L ce is to enable the new model to learn from new data; by minimizing this loss function, the prediction of the new model is made as consistent as possible with the true label of the new data; the loss function is represented as follows:

[0130] L ce = -log(p y ) (19)

[0131] wherein p y represents the probability that the model predicts the true class y;

[0132] By minimizing the above-mentioned joint loss, the model M new can both adapt to new data and effectively resist catastrophic forgetting and forget the knowledge of the old model; after training, the updated model M new will replace the old model online, and the model will be used as a new teacher model for distillation training in the next training.

[0133] The application field of the present application is very wide, and it is particularly suitable for large petrochemical and energy equipment (such as centrifugal compressors, turbine machinery, pump equipment, etc.), which are usually in a long-period running and fault sample extremely scarce working condition environment, and the conventional diagnosis method is difficult to effectively cover rare abnormalities. The present application is also suitable for key units in the power and metallurgical industries (such as gas turbines, steam turbines, generator sets), and can be used for online monitoring and intelligent diagnosis. In addition, in the device diagnosis module of the industrial internet platform (such as “Guangyou-Shen Gu cloud”), the present application can directly access the vibration signal and operation and maintenance log, and become one of the core functions of intelligent operation and maintenance. At the same time, its cross-modal intelligent diagnosis framework can be popularized to complex systems such as rail transit and aerospace, to realize the fusion modeling of time series signals and expert texts.

[0134] Around the product form, the present application derives multiple industrialized intelligent products. First, based on the method of the present application, a "signal-text dual-modal diagnostic software" can be constructed, which can be independently deployed on a factory local area network or in the cloud as an intelligent fault diagnosis system directly put into application; second, for equipment manufacturers and third-party service providers, the present application can be packaged as an API / SDK toolkit for tail sample generation, supporting data enhancement needs; third, the continuous learning diagnosis engine can be used as a kernel module of an industrial big data platform to realize model "online evolution"; finally, by combining with enterprise operation and maintenance platforms, an industry-level SaaS product can also be formed to output long-term diagnosis capabilities to end customers in the form of subscription or service.

[0135] At the theoretical level, the rationality and effectiveness of the generated samples of the present application are supported in multiple aspects. By introducing the Longformer sparse attention mechanism, the computational complexity is reduced from quadratic to linear, and in the case of a thousand-step long sequence, it can still capture global dependencies and local disturbances, ensuring that the generated signals have physical rationality. The trend / period / residual three-branch decoding mechanism decomposes the vibration signal into deterministic trend, main spectral component and random disturbance term, which conforms to the physical characteristics of rotating machinery. Combined with the time and frequency domain joint loss and the conditional guided sampling mechanism, it can ensure that the generated samples not only conform to the form of the real data, but also highly match the frequency distribution and mechanism level, theoretically avoiding the interference of "false signals".

[0136] In terms of dual-modal consistency, the present application introduces the cross-modal semantic screening capability of CLIP, treats the signal atlas as an image input, and projects it into a unified semantic space together with the diagnostic text to screen out high-similarity "signal-text" sample pairs. Then, through a multi-modal large model, the text is generated, so that the diagnostic sentence is not only constrained by the signal features, but also conforms to the expression of expert experience, theoretically avoiding semantic emptiness or deviation. At the same time, the present application significantly alleviates the data imbalance problem by synthesizing the tail samples, forming a more reasonable class distribution; compared with single-modal enhancement, "signal + text" paired expansion can provide more stable feature representation for the model, which helps to significantly improve the tail class recognition rate.

[0137] Finally, the continuous learning and engineering application effect of the present application also has solid evidence. At the theoretical level, the present application maintains the consistency of the feature space of the new and old models through knowledge distillation, compensates for the lack of tail data through generative enhancement, and under the balance mechanism of cross-entropy and distillation joint loss, realizes the performance improvement of "learning new and not forgetting old"; at the engineering level, the present application has covered more than 1500 units on the "Guangzhou Oil-Shenhu Drum Cloud" platform, and real industrial feedback shows that it can effectively warn of rare faults and continuously optimize model performance under the sample backflow mechanism. This shows that the present application not only has a breakthrough advantage in theory, but also exhibits replicable and generalizable technical value in industrial practical applications.

[0138] It should be noted that embodiments of the present application can be realized by hardware, software, or a combination of software and hardware. The hardware portion can be realized by a special logic; the software portion can be stored in a memory and executed by a proper instruction execution system, such as a microprocessor or a specially designed hardware. A person of ordinary skill in the art can understand that the above-mentioned apparatus and method can be realized by computer executable instructions and / or included in processor control codes, such as a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The apparatus of the present application and its modules can be realized by a hardware circuit, such as a very large scale integrated circuit or a gate array, a semiconductor, such as a logic chip, a transistor, or a programmable hardware device, such as a field programmable gate array, a programmable logic device, or the like, by software executed by various types of processors, or by a combination of the above-mentioned hardware circuit and software, such as firmware.

[0139] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent replacement, and improvement within the technical range disclosed by the present application, and within the spirit and principle of the present application, should be covered within the protection scope of the present application.

Claims

1. A method for signal text data generation and dual-modal fusion continual learning under long-tailed distribution, characterized in that, The application relates to a signal generation diffusion model based on a Transformer architecture and a Longformer encoder, semantic similarity evaluation of generated signals, and text generation combined with a semantic screening mechanism and a large model. The application also relates to a signal-text dual-modal fusion continuous learning model, and deployment of the continuous learning model into a fault diagnosis verification system for verification. The signal generation diffusion model in S1 comprises:

2. The method of claim 1, wherein, a Longformer-based long-period sequence modeling encoder for capturing long-time sequence dependency information; a trend, period and residual three-branch decoding structure for modeling output; and a multi-objective loss function combining time domain loss and frequency domain loss for optimization, and a condition-guided sampling method for signal generation. The trend branch uses polynomial regression to extract long-term smooth trends, the period branch uses Fourier transform to select the first K high-energy frequency components for reconstruction, and the residual branch compensates for the remaining details to finally realize signal generation output.

3. The method of claim 2, wherein, The semantic screening mechanism in S1 comprises:

4. The method of claim 1, wherein, using a pre-trained CLIP model to vectorize representations of signal graphs and expert texts, and calculating the cosine similarity between them; when the similarity between the signal graph and all expert texts is lower than a set threshold, the signal data is determined to be unqualified data and is rejected. The large model text generation method in S1 comprises:

5. The method of claim 1, wherein, using a CLIP model visual encoder to extract semantic features of multiple signal graphs under the same fault category; mapping the semantic features to the same dimension as the text features through a linear projection layer; concatenating the mapped signal graph features and the expert diagnosis text features and inputting them into the large model to generate enhanced diagnosis text data. The continuous learning process in S2 comprises:

6. The method of claim 1, wherein, when the model detects potential faults in the verification system, outputting diagnosis results and confirming them by technical personnel; storing the confirmed signal sequence and the corresponding text description in an abnormal sample database as a continuous training data source. The continuous learning adopts a data enhancement preprocessing method:

7. The method of claim 6, wherein, using a dual-modal generation model to expand new collected samples with long-tail distribution to obtain a mixed training set of normally distributed data and tail distributed data. The continuous learning training adopts a joint loss function, comprising:

8. The method of claim 6, wherein, a distillation loss for transferring old model knowledge to a new model to avoid catastrophic forgetting; a cross-entropy loss for enabling the new model to learn the true labels of new collected data; and model updating by minimizing the joint loss. The distillation loss is the square difference between the output features of the new model and the output features of the old model, and the cross-entropy loss is the difference between the model prediction probability and the real class label.

9. The method of claim 8, wherein, The updated dual-modal model replaces the online old model and participates in the distillation training as a teacher model in the next training, thereby forming an iterative updating continuous learning mechanism.

10. The method of claim 1, wherein, ​

Images