Fault diagnosis method and system based on time sequence imaging and visual large model

Abstract

The invention belongs to the technical field of industrial intelligent operation and maintenance and predictive maintenance, and particularly discloses a fault diagnosis method and system based on time sequence imaging and a visual large model, and the method comprises the steps: obtaining a two-dimensional image through various time sequence imaging codes based on the time sequence data of industrial equipment; on the basis of the equipment information, the fault type knowledge, the reference image retrieval result, the diagnosis requirement and the two-dimensional image, a cue word is constructed; the reference image retrieval result is obtained by retrieving a matched reference image from a reference sample library; indicating visual large model reasoning through cue words to obtain a diagnosis result; and based on the fault confidence threshold and the fault confidence in the diagnosis result, judging whether the industrial equipment has a fault, and if the fault is determined to occur, generating a fault record and pushing fault early warning information. By means of the fault diagnosis method and device, fault diagnosis with high generalization ability and high interpretability can be achieved under the condition that a large number of fault samples are lacked.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of industrial intelligent operation and maintenance and predictive maintenance, and more particularly relates to a fault diagnosis method and system based on time series image and visual large model, which is especially suitable for early weak fault detection of complex industrial equipment such as electrochemical energy storage batteries and rotating machinery. BACKGROUND

[0002] In the technical field of industrial intelligent operation and maintenance and predictive maintenance, the development of fault diagnosis technology has experienced evolution from traditional statistical analysis, machine learning to deep learning. Traditional methods such as moving average, exponential smoothing or rule judgment based on threshold have simple calculation but are difficult to cope with nonlinear and non-stationary complex fault patterns, and usually have hysteresis. Machine learning methods represented by support vector machine (SVM) and random forest have certain pattern recognition ability, but they need to rely on manual feature engineering and have high requirements for the balance of training data.

[0003] Deep learning methods represented by recurrent neural network (RNN) and long short-term memory network (LSTM) can automatically extract time series features, but have the following significant defects.

[0004] (1) Strong data dependence: training a high-precision model usually requires a large amount of labeled fault samples. However, in actual industrial scenarios (such as energy storage stations), fault samples (especially early faults before thermal runaway) are extremely scarce, making it difficult to train the model or overfitting.

[0005] (2) Weak generalization ability: special models trained for specific equipment or specific working conditions are often difficult to migrate directly to other models of equipment, facing high deployment costs of “one machine one model”.

[0006] (3) Poor interpretability: deep learning models are usually “black boxes”, which can only output fault probability and cannot point out diagnostic basis like human experts, such as “because the voltage curve jumped at a certain time”, making it difficult for operation and maintenance personnel to trust the model results.

[0007] It can be seen that how to use artificial intelligence technology to realize fault diagnosis with strong generalization ability and high interpretability in the absence of a large number of fault samples is a technical problem to be solved in the field. SUMMARY

[0008] In view of the defects of the prior art, the purpose of the present application is to use artificial intelligence technology to realize fault diagnosis with strong generalization ability and high interpretability in the absence of a large number of fault samples.

[0009] To achieve the above object, in a first aspect, the application provides a fault diagnosis method based on time series imaging and visual large model, comprising: Based on the time series data of the sensors of the industrial equipment, two-dimensional images are obtained through multiple time series imaging encoding methods. Based on the equipment information (which can include equipment working condition information), fault type knowledge, reference image retrieval results, diagnosis requirements and two-dimensional images, prompt words are constructed. The reference image retrieval results are obtained by retrieving reference images (normal state reference images and / or typical fault state reference images) matching the current working condition of the industrial equipment from a pre-set reference sample library, and the reference image retrieval results are embedded into the prompt words as visual context information. The visual large model is instructed to infer through the prompt words, and the industrial equipment diagnosis results output by the visual large model are obtained. Based on the fault confidence threshold and the fault confidence in the industrial equipment diagnosis results, it is determined whether the industrial equipment has failed. If it is determined that the fault has occurred, a fault record is generated and fault warning information is pushed. The fault record is generated based on the fault type and diagnosis basis in the industrial equipment diagnosis results, and the fault warning information is generated based on the fault type and maintenance suggestion in the industrial equipment diagnosis results. Among them, the prompt word is used to instruct the visual large model to infer based on the equipment information, fault type knowledge, reference image retrieval results, diagnosis requirements and two-dimensional images, output industrial equipment diagnosis results, and instruct the visual large model to compare the differences between the two-dimensional images and the reference image retrieval results in the inference process. The industrial equipment diagnosis results include fault type, fault confidence, diagnosis basis and maintenance suggestion.

[0010] It can be understood that by imaging the time series and using the multi-modal fusion capability of the general visual large model, the text data and image data are analyzed cooperatively, and in the absence of a large number of fault samples, strong generalization diagnosis of diversified industrial scenes is realized. At the same time, based on the constraint mechanism given by the diagnosis requirements in the prompt words, the diagnosis results contain traceable diagnosis basis, which significantly improves the explainability of the diagnosis process.

[0011] Exemplarily, the two-dimensional images corresponding to various time series imaging encoding methods can be directly filled into the prompt words and then provided to the visual large model. The two-dimensional images corresponding to various time series imaging encoding methods can also be fused into a multi-channel two-dimensional image, and the fused multi-channel two-dimensional image can be filled into the prompt words and then provided to the visual large model.

[0012] In one possible implementation, based on the time series data of the sensors of the industrial equipment, two-dimensional images are obtained through multiple time series imaging encoding methods, comprising: Based on the time series data of the sensor of the industrial equipment, through data cleaning, interpolation completion and sliding window segmentation, time series data segments are obtained; The time series data segments are normalized; Based on the normalized time series data segments, various time series image coding corresponding two-dimensional images are generated through various time series image coding methods; Based on the two-dimensional images corresponding to various time series image coding, size unification and multi-channel fusion processing are performed on the images to obtain fused multi-channel two-dimensional images.

[0013] Specifically, based on the time series data collected by the sensor of the industrial equipment, time series data segments are obtained through preprocessing operations such as data cleaning, interpolation completion and sliding window segmentation. Further, the time series data segments are normalized. Then, based on the normalized time series data segments, various time series image coding corresponding two-dimensional images are generated through various time series image coding methods (such as GAF coding, MTF coding, RP coding, etc.). Further, based on the two-dimensional images corresponding to various time series image coding, size unification (for example, unified to 256x256 pixels) and multi-channel fusion processing are performed on the images to obtain fused multi-channel two-dimensional images.

[0014] Here, the above data cleaning is exemplarily illustrated: based on the time series data collected by the sensor of the industrial equipment, abnormal data is identified and removed through outlier detection (for example, based on the 3σ principle, data points exceeding μ±3σ range are removed) to ensure the quality and reliability of the input data.

[0015] Here, the interpolation completion is exemplarily illustrated: for the data missing points that may be generated after data cleaning, interpolation completion technology is used to fill the missing values to form continuous and complete time series data.

[0016] Sliding window segmentation: sliding window technology is used to segment the time series data after interpolation completion. According to the specific application scenario, the window size is set to 256 to 1024 data points, and the step size is set to 1 / 4 to 1 / 2 of the window size, thereby generating a series of time series data segments with overlapping or continuous relationship, providing input for subsequent time series image coding.

[0017] In one possible implementation, the various time series image coding methods include at least two of the following coding methods: Gramian Angular Field (GAF) coding, Markov Transition Field (MTF) coding and Recurrence Plot (RP) coding.

[0018] Specifically, RP coding is used to characterize the periodicity and chaotic properties of time series in phase space and to identify battery micro-short circuits or nonlinear abrupt faults.

[0019] Battery micro-short circuit: A tiny short circuit occurs between cells within the battery pack or within a single cell.

[0020] RP encoding is particularly well-suited for transforming subtle signal variations (such as a small voltage drop caused by a battery short circuit) into significant texture breaks or noise distributions in an image, which can then be captured by large visual models.

[0021] GAF encoding is used to preserve time dependence and numerical magnitude information, and to identify capacity decay or internal resistance anomalies.

[0022] MTF encoding is used to characterize the state transition probability of a time series in different quantile buckets (or quantile intervals / bins) and to identify voltage jumps or poor contact faults.

[0023] In one possible implementation, the prompt words are constructed, including: Get the prompt word template. The prompt word template is configured with role description, fault diagnosis task description, visual thinking chain instruction and multiple placeholder strings. Based on equipment information, the current operating condition of the industrial equipment is determined. Based on the current operating condition, N most similar (Top-N) normal state reference images are retrieved from the reference sample library as positive sample anchors, and M most similar (Top-M) typical fault state reference images are retrieved as negative sample anchors, where N and M are positive integers. The reference image retrieval results include positive sample anchors and negative sample anchors. Using equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, the corresponding placeholder strings in the prompt word template are replaced to generate prompt words containing visual context information.

[0024] The role description item refers to the definition item in the structured prompts used to set the role of the visual large model in the fault diagnosis task.

[0025] The fault diagnosis task description item refers to the definition item in the structured prompt words used to explain the industrial equipment fault diagnosis task to be performed to the visual large model.

[0026] The Visual Retrieval Enhancement (Visual RAG) mechanism employed in this application is illustrated herein. The system determines the current operating condition of the industrial equipment (e.g., equipment operating parameters) based on equipment information. Then, using this operating condition as the query condition, it retrieves data from a pre-built reference sample library. This library contains two types of reference images: normal state reference images and typical fault state reference images, with each reference image having corresponding operating condition information as an index. The retrieval process calculates the similarity between the current operating condition and the corresponding operating condition of the sample (reference image) in the library (e.g., a distance metric based on the operating condition parameter vector), identifying the N (Top-N) most similar normal state reference images as positive sample anchors and the M (Top-M) most similar typical fault state reference images as negative sample anchors. These retrieved positive and negative sample anchors together constitute the reference image retrieval results, providing crucial references for subsequently generating prompts containing visual context information, thus achieving visual retrieval enhancement.

[0027] In one possible implementation, the Visual Chain of Thought instruction item is used to instruct the large visual model to perform the following reasoning steps: Extracting geometric features from two-dimensional images; Based on the geometric features of two-dimensional images, the geometric features of positive sample anchors, and the geometric features of negative sample anchors, the differences between two-dimensional images and positive sample anchors, as well as the similarities between two-dimensional images and negative sample anchors, are analyzed. Based on equipment information, fault type knowledge, diagnostic requirements, differences and similarities, generate diagnostic results for industrial equipment.

[0028] Understandably, the aforementioned visual thought chain instructions can direct the large model to output diagnostic results according to the reasoning steps of "geometric feature observation - difference comparison analysis - logical inductive judgment," and the aforementioned diagnostic requirements require the large model to explicitly point out visual evidence for the fault (such as "non-periodic chaotic patches appear in the lower left corner of the image"). It is evident that by forcing the large model to perform step-by-step reasoning and output traceable visual evidence, the interpretability of the diagnostic process is significantly enhanced, making the decision-making basis transparent.

[0029] For example, the geometric features of positive sample anchors and negative sample anchors can be obtained in advance and stored in a reference sample library. Then, when using the large visual model for inference, the geometric features of positive sample anchors and negative sample anchors can be extracted from the reference sample library and provided to the large visual model.

[0030] Regarding the difference between two-dimensional images and positive sample anchor points, generally the greater the difference, the higher the probability of industrial equipment failure.

[0031] Regarding the similarity between two-dimensional images and negative sample anchor points, generally, the higher the similarity, the higher the probability of industrial equipment failure.

[0032] In one possible implementation, the diagnostic requirements include the following limitations: The first limitation is that the diagnostic results of industrial equipment must be in the form of JSON structured data; The second limitation is that the diagnostic basis in the diagnostic results of industrial equipment includes information on abnormal texture features identified by the visual large model from the two-dimensional image. The information on abnormal texture features includes the coordinate region of the abnormal texture in the image and / or the morphological description of the abnormal texture.

[0033] Specifically, the first constraint mentioned above requires the large model to output diagnostic results in JSON structured data format to ensure the consistency of the data format of the output content.

[0034] By explicitly requiring the inclusion of specific information regarding anomalous texture features identified by the visual large model from two-dimensional images (i.e., images generated from time series using encoding techniques such as GAF, MTF, and RP), such as the coordinate regions of the anomalous textures in the image or their morphological descriptions, the diagnostic criteria are required to be structured. This mandatory output of structured diagnostic criteria significantly enhances the interpretability of the diagnostic process, making decision-making criteria (such as "a non-periodic chaotic patch appears in the lower left corner of the image") transparent and traceable, thus meeting the requirements of industrial applications for clear and verifiable diagnostic results.

[0035] One possible implementation also includes the following false alarm feedback mechanism steps: Obtain user feedback on false alarms for fault warning information. False alarm feedback is used to indicate that the fault warning information given by reasoning based on two-dimensional images is a false alarm and that no fault has actually occurred in the industrial equipment. In response to false alarm feedback, the two-dimensional image is stored in the reference sample library as a reference image for normal state, and the relevant fault records are deleted.

[0036] Understandably, when the system infers and outputs fault warning information based on two-dimensional images (i.e., images generated from time series data using encoding techniques such as GAF, MTF, and RP), if a user (such as maintenance personnel) confirms that the warning is a false alarm (i.e., the industrial equipment has not actually malfunctioned), the system receives this false alarm feedback. Subsequently, in response to this feedback, the system stores the two-dimensional image that triggered the false alarm as a new normal state reference image in the system's reference sample library and deletes the relevant fault records generated by this false alarm. The beneficial effect of this mechanism is that by transforming false alarm samples into normal reference data, it achieves dynamic expansion and self-learning optimization of the reference sample library. This helps the large visual model more accurately distinguish between normal and abnormal states in subsequent diagnosis, thereby continuously improving the universality and accuracy of diagnosis and effectively reducing future false alarm rates.

[0037] Secondly, this application provides a fault diagnosis system based on time-series image processing and large visual models, including: The encoding module is used to acquire two-dimensional images based on time-series data from sensors in industrial equipment through various time-series image encoding methods; The prompt word construction module is used to construct prompt words based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images. The reference image retrieval results are obtained by retrieving reference images from a preset reference sample library that match the current operating condition of the industrial equipment. The reference image retrieval results are embedded in the prompt words as visual context information. The visual big model calling module is used to instruct the visual big model to perform inference through prompt words and obtain the industrial equipment diagnostic results output by the visual big model. The diagnostic result processing module is used to determine whether the industrial equipment has malfunctioned based on the fault confidence threshold and the fault confidence in the industrial equipment diagnostic results. If a fault is determined to have occurred, a fault record is generated and a fault warning message is pushed. The fault record is generated based on the fault type and diagnostic basis in the industrial equipment diagnostic results, and the fault warning message is generated based on the fault type and maintenance suggestions in the industrial equipment diagnostic results. The prompt words are used to instruct the visual big model to perform reasoning based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, and output industrial equipment diagnostic results. They also instruct the visual big model to compare the differences between the two-dimensional images and the reference image retrieval results during the reasoning process. The industrial equipment diagnostic results include fault type, fault confidence, diagnostic basis, and maintenance suggestions.

[0038] Thirdly, this application provides an electronic device, including: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; the one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0039] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect or any possible implementation thereof.

[0040] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0041] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) By visualizing time series data and utilizing the multimodal fusion capability of a general visual model, text data and image data are analyzed collaboratively, thus achieving strong generalization diagnosis of diverse industrial scenarios in the absence of a large number of fault samples.

[0042] (2) Based on the constraint mechanism given in the diagnostic requirements in the prompt words, the generated diagnostic results contain traceable diagnostic evidence, which significantly improves the interpretability of the diagnostic process. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the fault diagnosis method based on time-series imaging and large visual model provided in the embodiments of this application. Figure 2 This is a schematic diagram illustrating the retrieval of reference images from a reference sample library provided in an embodiment of this application; Figure 3 This is a schematic diagram of the construction prompts provided in the embodiments of this application; Figure 4 This is a schematic diagram of the reasoning steps of the visual thinking chain provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of a fault diagnosis system based on time-series imaging and large visual model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In this application, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A existing alone, A and B existing simultaneously, and B existing alone. In this application, the symbol " / " indicates that the related objects are in an "or" relationship, for example, A / B means A or B.

[0046] In this application, the terms "first" and "second," etc., are used to distinguish different objects, not to describe a specific order of objects. For example, "first qualifier" and "second qualifier," etc., are used to distinguish different qualifiers, not to describe a specific order of qualifiers.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0049] The embodiments of this application are described below with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart illustrating the fault diagnosis method based on time-series imaging and large visual models provided in this application. Figure 1 As shown, the method includes the following steps S101 to S104.

[0051] Step S101: Based on the time series data of the sensors of industrial equipment, a two-dimensional image is obtained through various time series image encoding methods; Step S102: Based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, construct prompt words; the reference image retrieval results are obtained by retrieving reference images (normal state reference images and / or typical fault state reference images) that match the current operating conditions of industrial equipment from a preset reference sample library, and the reference image retrieval results are embedded in the prompt words as visual context information; Step S103: The Visual Large Model (VLM) is instructed to perform inference by prompting words, and the industrial equipment diagnostic results output by the Visual Large Model are obtained. Step S104: Based on the fault confidence threshold and the fault confidence in the industrial equipment diagnosis results, determine whether the industrial equipment has failed. If a fault is determined to have occurred, generate a fault record and push a fault warning message. The fault record is generated based on the fault type and diagnosis basis in the industrial equipment diagnosis results, and the fault warning message is generated based on the fault type and maintenance suggestions in the industrial equipment diagnosis results. The prompt words are used to instruct the visual big model to perform reasoning based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, and output industrial equipment diagnostic results. They also instruct the visual big model to compare the differences between the two-dimensional images and the reference image retrieval results during the reasoning process. The industrial equipment diagnostic results include fault type, fault confidence, diagnostic basis, and maintenance suggestions.

[0052] Specifically, based on time-series data from industrial equipment sensors, two-dimensional images reflecting different dimensions of temporal characteristics are obtained through various time-series image encoding methods (such as GAF, MTF, and RP encoding). Structured prompts are constructed using textual data (as prior knowledge or expert experience) such as equipment information, fault type knowledge, and diagnostic requirements, combined with image data such as two-dimensional images and retrieved reference images. These structured prompts instruct a pre-trained general-purpose visual model to perform inference on multimodal data (textual and image data). During inference, the model's image understanding and few-shot learning capabilities are utilized to compare and analyze the differences between the retrieved results of two-dimensional images and reference images (as a small sample), thereby obtaining the industrial equipment diagnostic results output by the visual model (containing fault type, fault confidence, diagnostic basis, and maintenance suggestions). Finally, the fault confidence in the diagnostic results is judged based on a preset fault confidence threshold. If a fault is determined to have occurred, a fault record containing the fault type and diagnostic basis (which may also include time-series data and two-dimensional images) is generated based on the diagnostic results, and a fault warning message containing the fault type and maintenance suggestions is pushed out.

[0053] This method visualizes time series data and leverages the multimodal fusion capabilities of a general visual model to collaboratively analyze textual and image data. This enables highly generalized diagnosis of diverse industrial scenarios even in the absence of a large number of fault samples. Furthermore, based on the constraint mechanism provided by the diagnostic requirements in the prompts, the generated diagnostic results contain traceable diagnostic evidence, significantly improving the interpretability of the diagnostic process.

[0054] This section provides an illustrative example of the time-series data from the sensors of the aforementioned industrial equipment. This time-series data refers to serialized data reflecting the operating status and physical parameter changes of the equipment, continuously collected from various sensors (such as voltage, current, temperature, and vibration sensors) of industrial equipment (e.g., electrochemical energy storage batteries, rotating machinery, or power systems) at a specific sampling frequency (e.g., 100Hz to 10kHz). This data serves as the raw input for fault diagnosis using the method provided in this application. After processing such as data cleaning, interpolation completion, sliding window segmentation, and normalization, it is used as input for various time-series image encoding methods (e.g., GAF, MTF, RP encoding) to generate corresponding two-dimensional images.

[0055] For example, fault type knowledge is used to describe the characteristics and diagnostic criteria of various fault types. For instance, diagnostic criteria describe the features of a two-dimensional image when a specified fault type occurs in industrial equipment. Diagnostic requirements describe the requirements that the visual large model should meet for fault diagnosis-related inference. For example, diagnostic requirements include output format specifications, which could specify that the diagnostic results for industrial equipment are in JSON format.

[0056] For example, in the diagnostic results of industrial equipment, the diagnostic basis is used to describe the basis related to the diagnostic results in order to improve the interpretability of fault diagnosis.

[0057] The above step S101 will be described by example below.

[0058] In step S101, taking an electrochemical energy storage battery as an example, time-series data including cell voltage, current, and temperature are acquired from BMS (Battery Management System) sensors at a sampling frequency of 1Hz. The raw data is cleaned, missing values ​​are filled using Lagrange interpolation, and time-series data segments are extracted using a sliding window method. Then, Z-score normalization is performed on the data segments. Subsequently, a multi-encoding collaborative approach is used to transform the preprocessed time-series signal into three types of two-dimensional images: an image reflecting the system's periodicity and chaos is generated through recursive graph (RP) encoding to capture features such as diagonal structure breaks caused by small voltage drops; an image retaining time dependence and numerical magnitude information is generated through Gram angle field (GAF) encoding (including GASF and GADF); and an image analyzing state transition probabilities is generated through Markov transition field (MTF) encoding, which is sensitive to faults such as voltage drops. The generated RP, GAF, and MTF encoded images can be uniformly adjusted to a 256×256 pixel size for subsequent large-scale visual model processing.

[0059] The above step S102 will be described by example below.

[0060] The reference sample library is pre-built, such as Figure 2 As shown, the reference sample library contains two types of reference images: normal state reference images and typical fault state reference images. Each reference image has corresponding operating condition information as an index. The retrieval process calculates the similarity between the current operating condition and the operating condition corresponding to the sample (reference image) in the library (e.g., a distance metric based on the operating condition parameter vector), and identifies the N (Top-N) most similar normal state reference images as positive sample anchors, and the M (Top-M) most similar typical fault state reference images as negative sample anchors.

[0061] like Figure 3 As shown, the corresponding placeholder strings in the prompt word template are replaced with equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images to generate prompt words containing visual context information.

[0062] like Figure 4 As shown, the Visual Chain of Thought instruction item is used to instruct the visual large model to perform the following reasoning steps: extract the geometric features of the two-dimensional image; based on the geometric features of the two-dimensional image, the geometric features of the positive sample anchor point, and the geometric features of the negative sample anchor point, analyze the differences between the two-dimensional image and the positive sample anchor point, and analyze the similarities between the two-dimensional image and the negative sample anchor point; generate industrial equipment diagnostic results based on equipment information, fault type knowledge, diagnostic requirements, differences, and similarities.

[0063] In step S102, visual retrieval enhancement prompts can be constructed based on the device information, fault type knowledge, diagnostic requirements, and the two-dimensional image generated in step S101 of the electrochemical energy storage battery.

[0064] The reference sample library stores confirmed normal battery RP images (gold samples) and typical internal short-circuit fault battery RP images (black samples) from historical data, and stores the corresponding operating condition information as an index.

[0065] Based on the current operating conditions of the battery to be diagnosed (such as SOC range and current rate), the most similar normal image is retrieved from the reference sample library as "Reference Image A (Anchor Positive)," and the most similar fault image is retrieved as "Reference Image B (Anchor Negative)." Then, prompt words are assembled to construct a structured prompt word system containing "role setting + task description + multi-image context + thought chain instructions."

[0066] For example, the prompts specifically include: the role setting is "You are a senior battery safety diagnostic expert."; the multi-image context is "There are three images in the attachment: the first image is a two-dimensional image of the battery under test; the second image is a normal reference image under the same operating conditions; the third image is a typical internal short-circuit fault reference image."; the thought chain instruction is "Please think step by step: First, compare the first and second images and observe whether there is a difference in the continuity of the main diagonal; second, compare the first and second images and determine whether there are similar chaotic textures; third, comprehensively judge whether the first image is abnormal."

[0067] The above step S103 will be described by example below.

[0068] In step S103, the structured prompts constructed in step S102 instruct the large visual model (such as the Qwen2.5-VL series, DeepSeek-VL series, Claude Opus 4.5, or a finely tuned CLIP model) to perform inference. The large visual model not only utilizes its pre-trained general visual capabilities to recognize features of 2D images such as RP, GAF, and MTF, but also performs comparative inference based on the multi-image context provided in the prompts (including the 2D image of the current battery under test, the normal reference image under the same operating conditions, and the typical internal short-circuit fault reference image) to "find differences" and "find similarities".

[0069] The visual big data model follows a thought chain instruction, analyzing image features step by step, and ultimately outputting structured diagnostic results for industrial equipment. These results are output in JSON format and include: fault type, fault confidence level, diagnostic basis, and maintenance recommendations.

[0070] For example, the fault type is: early internal short circuit. The fault confidence level is 0.92. The diagnostic criteria are: obvious nonlinear noise accumulation appears in the lower left corner of the image under test (coordinate range 0-50), and the texture of the main diagonal line is broken. This is highly consistent with the internal short circuit characteristics of the third image (a typical internal short circuit fault reference image), and significantly different from the smooth texture of the second image (a normal reference image under the same operating condition). The maintenance recommendation is: it is recommended to immediately stop the charging and discharging of this battery cluster and arrange on-site testing.

[0071] The above step S104 will be described by example below.

[0072] In step S104, the diagnostic results output in step S103 are judged based on a preset fault confidence threshold (e.g., 0.85). If the fault confidence in the diagnostic results is greater than or equal to the threshold, a fault is determined to have occurred. The system will automatically generate a fault record containing the fault type, confidence level, diagnostic basis, and maintenance suggestions, and push a fault warning message to prompt maintenance personnel to take action.

[0073] In addition, the system integrates a false alarm feedback closed-loop mechanism: if maintenance personnel confirm on the platform that a warning is a false alarm, the system will automatically mark the corresponding two-dimensional image (RP, GAF, MTF, etc.) as a "normal state reference image" and store it in the reference sample library. This mechanism uses feedback data to continuously optimize the reference sample library, thereby continuously improving the robustness and accuracy of the diagnosis.

[0074] The fault diagnosis system based on time-series imaging and large visual model provided in this application is described below. The fault diagnosis system based on time-series imaging and large visual model described below can be referred to in correspondence with the fault diagnosis method based on time-series imaging and large visual model described above.

[0075] Figure 5 This is a schematic diagram of the structure of the fault diagnosis system based on time-series imaging and large visual model provided in the embodiments of this application, such as... Figure 5 As shown, the system includes: an encoding module 10, a prompt word construction module 20, a visual large model calling module 30, and a diagnostic result processing module 40. Among them: Encoding module 10 is used to acquire two-dimensional images based on time-series data from sensors in industrial equipment through various time-series image encoding methods; The prompt word construction module 20 is used to construct prompt words based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images. The reference image retrieval results are obtained by retrieving reference images from a preset reference sample library that match the current operating condition of the industrial equipment. The reference image retrieval results are embedded in the prompt words as visual context information. The visual big model calling module 30 is used to instruct the visual big model to perform inference through prompt words and obtain the industrial equipment diagnostic results output by the visual big model. The diagnostic result processing module 40 is used to determine whether the industrial equipment has failed based on the fault confidence threshold and the fault confidence in the industrial equipment diagnostic results. If a fault is determined to have occurred, a fault record is generated and a fault warning information is pushed. The fault record is generated based on the fault type and diagnostic basis in the industrial equipment diagnostic results, and the fault warning information is generated based on the fault type and maintenance suggestions in the industrial equipment diagnostic results. The prompt words are used to instruct the visual big model to perform reasoning based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, and output industrial equipment diagnostic results. They also instruct the visual big model to compare the differences between the two-dimensional images and the reference image retrieval results during the reasoning process. The industrial equipment diagnostic results include fault type, fault confidence, diagnostic basis, and maintenance suggestions.

[0076] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0077] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0078] Based on the methods in the above embodiments, this application provides an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0079] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0080] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0081] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0082] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0083] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0084] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0085] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0086] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A fault diagnosis method based on time-series image processing and large visual model, characterized in that, include: Two-dimensional images are obtained from time-series data of sensors in industrial equipment through various time-series image encoding methods; Based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, prompt words are constructed. The reference image retrieval results are obtained by retrieving reference images from a preset reference sample library that match the current operating conditions of the industrial equipment. The reference image retrieval results are embedded in the prompt words as visual context information. The visual big data model is guided to perform inference by prompting words, and the diagnostic results of industrial equipment output by the visual big data model are obtained. Based on the fault confidence threshold and the fault confidence in the industrial equipment diagnosis results, it is determined whether the industrial equipment has failed. If a fault is determined to have occurred, a fault record is generated and a fault warning message is pushed. The fault record is generated based on the fault type and diagnosis basis in the industrial equipment diagnosis results, and the fault warning message is generated based on the fault type and maintenance suggestions in the industrial equipment diagnosis results. The prompt words are used to instruct the visual big model to perform reasoning based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, and output industrial equipment diagnostic results. They also instruct the visual big model to compare the differences between the two-dimensional images and the reference image retrieval results during the reasoning process. The industrial equipment diagnostic results include fault type, fault confidence, diagnostic basis, and maintenance suggestions.

2. The fault diagnosis method based on time-series image processing and large visual model according to claim 1, characterized in that, The time-series data from the sensors based on industrial equipment is used to obtain two-dimensional images through various time-series image encoding methods, including: Based on time-series data from sensors in industrial equipment, time-series data segments are obtained through data cleaning, interpolation completion, and sliding window segmentation. Normalize the time series data segments; Based on the normalized time series data segments, two-dimensional images corresponding to various time series image encoding methods are generated. Based on the two-dimensional images corresponding to various time-series image encodings, the images are subjected to size unification and multi-channel fusion processing to obtain the fused multi-channel two-dimensional images.

3. The fault diagnosis method based on time-series image processing and large visual model according to claim 2, characterized in that, Various time series image encoding methods include at least two of the following encoding methods: Gram-angle field (GAF) coding, Markov transition field (MTF) coding, and recursive graph (RP) coding.

4. The fault diagnosis method based on time-series image processing and large visual model according to claim 1, characterized in that, The constructed prompt words include: Get the prompt word template. The prompt word template is configured with role description, fault diagnosis task description, visual thinking chain instruction and multiple placeholder strings. Based on equipment information, the current operating condition of the industrial equipment is determined, and based on the current operating condition, N most similar normal state reference images are retrieved from the reference sample library as positive sample anchors, and M most similar typical fault state reference images are retrieved as negative sample anchors, where N and M are positive integers; the reference image retrieval results include positive sample anchors and negative sample anchors. Using equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, the corresponding placeholder strings in the prompt word template are replaced to generate prompt words containing visual context information.

5. The fault diagnosis method based on time-series imaging and large visual model according to claim 4, characterized in that, Visual thought chain instructions are used to instruct the large visual model to perform the following reasoning steps: Extracting geometric features from two-dimensional images; Based on the geometric features of two-dimensional images, the geometric features of positive sample anchors, and the geometric features of negative sample anchors, the differences between two-dimensional images and positive sample anchors, as well as the similarities between two-dimensional images and negative sample anchors, are analyzed. Based on equipment information, fault type knowledge, diagnostic requirements, differences and similarities, generate diagnostic results for industrial equipment.

6. The fault diagnosis method based on time-series image processing and large visual model according to claim 1, characterized in that, Diagnostic requirements include the following limitations: The first limitation is that the diagnostic results of industrial equipment must be in the form of JSON structured data; The second limitation is that the diagnostic basis in the diagnostic results of industrial equipment includes information on abnormal texture features identified by the visual large model from the two-dimensional image. The information on abnormal texture features includes the coordinate region of the abnormal texture in the image and / or the morphological description of the abnormal texture.

7. The fault diagnosis method based on time-series image processing and large visual model according to claim 1, characterized in that, It also includes the following false alarm feedback mechanism steps: Obtain user feedback on false alarms for fault warning information. False alarm feedback is used to indicate that the fault warning information given by reasoning based on two-dimensional images is a false alarm and that no fault has actually occurred in the industrial equipment. In response to false alarm feedback, the two-dimensional image is stored in the reference sample library as a reference image for normal state, and the relevant fault records are deleted.

8. A fault diagnosis system based on time-series image processing and large visual model, characterized in that, include: The encoding module is used to acquire two-dimensional images based on time-series data from sensors in industrial equipment through various time-series image encoding methods; The prompt word construction module is used to construct prompt words based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images. The reference image retrieval results are obtained by retrieving reference images from a preset reference sample library that match the current operating conditions of the industrial equipment. The reference image retrieval results are embedded in the prompt words as visual context information. The visual big model calling module is used to instruct the visual big model to perform inference through prompt words and obtain the industrial equipment diagnostic results output by the visual big model. The diagnostic result processing module is used to determine whether the industrial equipment has malfunctioned based on the fault confidence threshold and the fault confidence in the industrial equipment diagnostic results. If a fault is determined to have occurred, a fault record is generated and a fault warning message is pushed. The fault record is generated based on the fault type and diagnostic basis in the industrial equipment diagnostic results, and the fault warning message is generated based on the fault type and maintenance suggestions in the industrial equipment diagnostic results. The prompt words are used to instruct the visual big model to perform reasoning based on equipment information, fault type knowledge, reference image retrieval results, diagnostic requirements, and two-dimensional images, and output industrial equipment diagnostic results. They also instruct the visual big model to compare the differences between the two-dimensional images and the reference image retrieval results during the reasoning process. The industrial equipment diagnostic results include fault type, fault confidence, diagnostic basis, and maintenance suggestions.

9. An electronic device, characterized in that, include: Memory and one or more processors; The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions; The one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

Images