A dynamic weighted multi-modal pharmaceutical equipment fault diagnosis reasoning method and system

Abstract

This invention discloses a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method and system, belonging to the interdisciplinary field of artificial intelligence and intelligent pharmaceutical manufacturing. Addressing the technical bottlenecks in existing pharmaceutical equipment fault diagnosis methods, such as insufficient robustness of multimodal fusion, weak long-term fault feature mining capabilities, and susceptibility to illusions in large-model reasoning, which fail to meet GMP compliance requirements, this invention proposes a layered collaborative diagnostic scheme: at the method level, accurate diagnosis is achieved through synchronous acquisition of multimodal data, robust feature extraction, dynamic weighted fusion using DWFF, structured cue word construction, and GraphRAG-enhanced large-model causal reasoning; at the system level, a bottom-up four-layer collaborative architecture is adopted, which adaptively allocates fusion weights based on the quality of each modality's data, combines knowledge graph retrieval to suppress illusions in output, and finally outputs an interpretable compliance diagnostic report. This invention effectively improves the accuracy, stability, and interpretability of fault diagnosis, ensuring efficient operation of pharmaceutical equipment and adapting to the actual production needs of the pharmaceutical industry.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and intelligent pharmaceutical manufacturing, specifically to a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method and system. Background Technology

[0002] In modern pharmaceutical manufacturing processes, continuous, stable, and reliable equipment operation is a core prerequisite for ensuring controllable drug quality and achieving compliance with Good Manufacturing Practice (GMP) standards. Key pharmaceutical equipment such as tablet compressors, mixers, granulators, and integrated packaging and sealing machines have complex structures and operate under harsh conditions. Their failures are generally characterized by high concealment, long evolution cycles, and the coupling of multiple factors. In actual fault diagnosis, it is usually necessary to integrate heterogeneous information from multiple sources, including operator descriptions, on-site fault images, equipment operation videos, and process parameters, to achieve accurate identification and location.

[0003] Currently, there are still many technical bottlenecks in the field of pharmaceutical equipment fault diagnosis, mainly reflected in the following aspects: (1) Multimodal fusion lacks robustness and is susceptible to feature distortion due to environmental interference. The pharmaceutical manufacturing site presents complex conditions such as light fluctuations, powder dust obstruction, and vibration interference, leading to unstable image and video data quality. Existing multimodal fusion methods mostly adopt fixed weight strategies, which cannot dynamically adjust the contribution based on the data quality of each modality. When the data of a certain modality is damaged or has strong noise, it will directly introduce interference information, significantly reducing diagnostic accuracy and easily causing misjudgment and missed judgment.

[0004] (2) The ability to mine long-term dynamic features is weak and the correlation of fault root causes is unclear. Faults in pharmaceutical equipment often evolve gradually over time, with weak early signs and short duration. Traditional diagnostic models are unable to effectively extract and correlate the temporal evolution features in long video sequences, and cannot accurately capture subtle changes in the operating status of equipment. This results in a blurred mapping relationship between the "visual appearance" and the "essential root cause" of the fault, making it difficult to achieve early warning and in-depth localization.

[0005] (3) The model has poor interpretability and lacks compliance-oriented causal reasoning ability. Existing general-purpose fault diagnosis models are mostly "black box" structures, only able to output fault types and unable to conduct logical, traceable causal reasoning by combining pharmaceutical process knowledge and GMP standards. Furthermore, when directly using large models for diagnosis, the lack of domain knowledge constraints and retrieval enhancement mechanisms easily leads to problems such as content illusion and sloppy reasoning logic, making it difficult for diagnostic results to meet the stringent requirements of compliance, traceability, and interpretability in the pharmaceutical industry.

[0006] In summary, given the GMP compliance requirements and complex operating conditions of pharmaceutical manufacturing, existing technologies cannot simultaneously achieve robust fusion of multimodal data, accurate extraction of long-term fault features, and interpretable causal reasoning. Therefore, there is an urgent need for a pharmaceutical equipment fault diagnosis reasoning scheme with dynamic weight adaptive adjustment and the integration of multimodal large models and knowledge graphs to improve the accuracy, stability, and interpretability of fault diagnosis, ensuring the safe and efficient operation of pharmaceutical equipment. Summary of the Invention

[0007] To address the technical problems existing in the prior art and overcome its shortcomings such as low diagnostic accuracy, poor robustness, and insufficient interpretability, this paper provides a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method and system to meet the GMP compliance requirements of pharmaceutical production and achieve accurate fault location, root cause tracing, and compliant handling.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: The present invention provides a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method, comprising the following steps: Step S1: Synchronous acquisition of heterogeneous multimodal data; collect fault description text, on-site fault images and equipment operation video sequences of the pharmaceutical equipment to be inspected. The text contains information such as fault phenomena, location of occurrence, and abnormal status, while the images and videos are used to reflect the spatial and temporal characteristics of the fault.

[0009] Specifically, the system simultaneously acquires fault description text of the pharmaceutical equipment under inspection, which includes semantic information such as fault phenomenon, location, and abnormal state; simultaneously acquires on-site fault images of the pharmaceutical equipment under inspection to reflect the spatial details of the fault; simultaneously acquires video sequences of the equipment operation of the pharmaceutical equipment under inspection to capture the dynamic temporal evolution characteristics of the fault; performs cleaning operations on the acquired defect images to remove environmental noise caused by dust and light fluctuations at the pharmaceutical site; and performs keyframe extraction operations on the acquired maintenance videos to compress redundant temporal data and reduce the computational overhead of subsequent processing.

[0010] Step S2: Robust feature extraction; a multimodal encoder is used to extract features from each modality of data: for video data, a 3D-CNN combined with a Transformer architecture is used to extract temporal evolution features; for images, a residual network is used to extract spatial features; for text, a pre-trained language model is used to extract semantic features, resulting in a robust multimodal feature vector.

[0011] Specifically, a text encoder combining BERT and NER models is used to semantically encode the fault description text, extracting fault entities, relationships, and deep semantic features to obtain text feature vectors. A visual encoder combining CLIP-ViT and residual networks is used to visually encode the cleaned defect images, enhancing feature robustness and extracting fault spatial detail features to obtain image feature vectors. A temporal encoder combining TCN, 3D-CNN, and Transformer architectures is used to temporally encode the maintenance video after keyframe extraction, extracting fault dynamic evolution features in long-term video and capturing subtle early fault signs to obtain video temporal feature vectors. The above three types of feature vectors are unified to the same dimension to obtain a unified-dimensional multimodal independent feature vector.

[0012] Step S3: Dynamically allocate weights based on the DWFF algorithm; evaluate the quality of each modal data using the modal quality detection operator to obtain the corresponding modal confidence; and adaptively weight each modal feature using the Dynamic Weighted Feature Fusion (DWFF) algorithm to reduce the interference of noisy data and low-quality modalities on the diagnostic results.

[0013] Specifically, the modal quality detection operator is used to evaluate the data quality of each modality, and the data quality score s of the i-th modality is calculated. i The quality confidence score for the i-th mode is calculated as follows: q i Where σ(·) is the Sigmoid normalization function, α is the quality adjustment coefficient, and s i Let be the data quality score for the i-th modality.

[0014] The confidence scores of each mode are normalized to obtain the dynamic weights of the i-th mode: Among them, w i Let be the final weight of the i-th mode, and n be the total number of modes participating in the fusion.

[0015] The features are weighted and fused according to the dynamic weights of each modality to obtain joint semantic features: Among them, F i Let F be the feature vector extracted from the i-th mode. fusion The multimodal joint features are dynamically weighted and fused; the fused multimodal joint features are used as input for subsequent inference.

[0016] Step S4: Constructing prompt words for large models; convert the fused multimodal joint features into structured text vectors through semantic mapping, and generate structured instructions to guide the reasoning of large models based on the preset prompt word template of role, background, evidence and constraint.

[0017] Step S5: Collaborate with the knowledge graph to perform large-scale model reasoning; input structured prompts into the multimodal large-scale model fine-tuned by pharmaceutical fault domain data, introduce the GraphRAG (Graph Structure Retrieval Enhancement Generation) mechanism to retrieve the pharmaceutical equipment fault domain knowledge graph, and obtain the fault causal evidence chain; the large-scale model performs logical reasoning based on the retrieval results and multimodal features, and outputs a diagnostic report containing fault location, root cause analysis, and GMP compliance handling suggestions.

[0018] This invention provides a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning system, which, from bottom to top, includes a multi-source heterogeneous data layer, a knowledge encoding and fusion layer, a source tracing reasoning layer, and an intelligent decision-making layer. The output of the multi-source heterogeneous data layer is connected to the input of the knowledge encoding and fusion layer, the output of the knowledge encoding and fusion layer is connected to the input of the source tracing reasoning layer, and the output of the source tracing reasoning layer is connected to the input of the intelligent decision-making layer. Data flow between the layers is achieved through standardized data interfaces, forming a complete fault diagnosis closed loop.

[0019] Specifically, the multi-source heterogeneous data layer is the system's underlying data support layer, used to provide the raw multi-source heterogeneous data required for fault diagnosis. This includes three data sources: equipment documents, defect images, and maintenance videos. The equipment documents consist of textual data such as historical maintenance records, fault description documents, and standard operating procedures for pharmaceutical equipment. The defect images are on-site photos of the faulty parts of the equipment, and the maintenance videos are long-term video data documenting equipment operation and fault evolution. The multi-source heterogeneous data layer also performs preprocessing operations on the raw data, cleansing the defect images to remove environmental noise and interference, and extracting keyframes from the maintenance videos to compress redundant time-series data. The preprocessed data is synchronously transmitted upwards to the knowledge encoding and fusion layer, providing standardized input for subsequent knowledge encoding.

[0020] Specifically, the knowledge encoding and fusion layer is the core knowledge processing layer of the system, used to perform feature encoding and knowledge fusion on heterogeneous data from the multi-source heterogeneous data layer to construct a unified multimodal knowledge index. The knowledge encoding and fusion layer includes a text encoder, a visual encoder, a temporal encoder, a structured knowledge graph, a multimodal feature vector library, and a cross-modal alignment unit. The text encoder uses a BERT combined with a NER model to perform semantic encoding on device documents from the multi-source heterogeneous data layer, extracting fault entities, relationships, and deep semantic features from the text, providing support for the construction of structured knowledge. The visual encoder uses a CLIP-ViT model to perform visual feature encoding on cleaned defect images, extracting fault spatial detail features from the images, and realizing the vectorized representation of image data.

[0021] Specifically, the temporal encoder employs a TCN temporal convolutional network to encode the temporal features of the maintenance video after keyframe extraction, extracting the dynamic evolution features of the fault and capturing subtle fault signs in long-term video. The structured knowledge graph is constructed using the Neo4j graph database to store the structured fault knowledge processed by the text encoder, including equipment entities, fault phenomena, root causes, maintenance measures, and GMP compliance requirements, forming a searchable causal relationship network.

[0022] Specifically, the multimodal feature vector library is constructed using the VectorDB vector database to store multimodal feature vectors processed by the visual encoder and temporal encoder, achieving efficient storage and fast retrieval of high-dimensional features. The cross-modal alignment unit is used to perform cross-modal alignment processing on text, visual, and temporal features, eliminating semantic gaps between heterogeneous modalities. In a further embodiment of the invention, the cross-modal alignment unit uses the DWFF dynamic weighted fusion algorithm to perform fusion processing on multimodal features. The unit first evaluates the confidence level of each modality data through a modality quality detection operator, calculates the dynamic weight allocation of each modality based on the confidence level, and then performs weighted fusion on the multimodal features according to the weights, effectively eliminating interference from low-quality modalities, improving the robustness of the fused features, and finally constructing a unified multimodal knowledge index to support subsequent retrieval and reasoning.

[0023] Specifically, the source tracing reasoning layer is the core reasoning layer of the system, used to receive user input and perform retrieval-enhanced fault reasoning. It includes a user input unit, a multimodal retrieval unit, and a multimodal large model. The user input unit receives diagnostic data input by maintenance personnel, including fault description text, on-site fault images, and equipment operation videos, providing target data for fault diagnosis.

[0024] Specifically, the multimodal retrieval system employs a GraphRAG architecture to perform joint retrieval on the multimodal knowledge index constructed by the knowledge encoding and fusion layer based on the user-input diagnostic data. It matches corresponding causal chains of failures from the structured knowledge graph and similar historical failure features from the multimodal feature vector library, outputting retrieval results containing causal evidence and similar cases. This provides reliable knowledge support for large model reasoning and effectively suppresses the large model illusion problem. The multimodal large model (MLLM) receives the user-input diagnostic data and the retrieval results output by the multimodal retrieval system. Based on the multimodal large model fine-tuned with data from the pharmaceutical equipment failure domain, it performs logical reasoning to complete fault type identification, root cause localization, and processing solution generation, transmitting the reasoning results upwards to the intelligent decision-making layer.

[0025] Specifically, the intelligent decision-making layer is the top output layer of the system, used to receive the reasoning results from the source tracing reasoning layer and generate standardized and interpretable fault diagnosis reports. The diagnosis reports include fault type identification results, fault root cause analysis, targeted maintenance suggestions, and GMP compliance guidance information, realizing the visualization and traceable storage of diagnosis results, meeting the compliance management needs of the pharmaceutical industry. At the same time, the diagnosis reports can provide incremental data for the knowledge graph, enabling continuous iterative optimization of the system.

[0026] Compared with the prior art, the present invention has the following beneficial effects: This invention overcomes the shortcomings of existing pharmaceutical equipment fault diagnosis, such as multimodal feature distortion, weak correlation of long-term features, poor interpretability of diagnosis, large model susceptibility to hallucinations, and inability to meet GMP compliance traceability, by combining multimodal synchronous acquisition, DWFF dynamic weighted fusion, structured prompt word construction, fault knowledge graph retrieval, and multimodal large model causal reasoning. It significantly improves the accuracy, stability and traceability of fault diagnosis.

[0027] The present invention employs the DWFF dynamic weighted fusion module, which can calculate the confidence level in real time and adaptively allocate weights based on the data quality of each modality of text, image, and video. This effectively suppresses environmental noise such as unstable lighting, powder dust, and vibration interference at the pharmaceutical site, eliminates low-quality modalities and redundant frame information, significantly improves the robustness of multimodal fusion, and reduces the rate of false positives and false negatives.

[0028] This invention employs the aforementioned multimodal feature extraction module, which extracts dynamic evolution features of long-term videos through a combination of 3DCNN and Transformer architecture, extracts spatial detail features of fault images through residual networks, and extracts textual semantic features through a pre-trained language model. This enables the accurate capture of early, subtle signs of equipment failure, strengthens the mapping relationship between visual manifestations of failures and their underlying causes, and improves the ability to identify early failures.

[0029] This invention employs the structured prompt word construction module, the fault knowledge graph, and the GraphRAG module. It guides the large model's reasoning through a four-in-one prompt word encompassing role, background, evidence, and constraints, and achieves enhanced retrieval generation based on the causal evidence chain of the knowledge graph. This fundamentally avoids the large model's hallucination output, enabling the diagnostic process to have a clear logical chain and strong interpretability.

[0030] This invention employs the multimodal large model inference module and the GMP compliance diagnostic output module, which can directly output a complete diagnostic report containing fault location, root cause analysis, maintenance measures, compliance recommendations, and traceability information. It fully complies with the GMP requirements of the pharmaceutical industry, meets the needs of on-site compliance review, fault tracing, and standardized handling, and is more suitable for the actual working conditions of intelligent manufacturing in pharmaceuticals. Attached Figure Description

[0031] Figure 1 This is a flowchart of a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to the present invention; Figure 2 This is an architecture diagram of a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning system according to the present invention; Figure 3 A schematic diagram of the fault knowledge graph constructed for this invention; Figure 4 This is a schematic diagram illustrating the content retrieved from a knowledge graph based on fault phenomena according to the present invention. Figure 5 This is a schematic diagram of the diagnostic report output of the present invention; Figure 6 This is a demonstration image of the integrated opening, packaging, and palletizing machine of the present invention. Detailed Implementation

[0032] The following is in conjunction with the appendix Figure 1 To be continued Figure 6 The technical solution of the present invention will be further described in detail below with specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiment.

[0033] Example 1 This embodiment provides a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method and system, as shown in the attached figure. Figure 1 As shown, attached Figure 1 This is a flowchart illustrating a dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method provided in an embodiment of the present invention. The method includes five core steps: synchronous acquisition of heterogeneous multimodal data, robust feature extraction, DWFF dynamic weighted fusion, structured prompt word construction, and collaborative knowledge graph reasoning. See the attached diagram for a corresponding explanation. Figure 2 The diagram shown is an overall hierarchical architecture diagram of the system described in this invention. The system adopts a bottom-up four-layer collaborative architecture, which corresponds one-to-one with the method steps. It can realize collaborative deployment between the edge side and the cloud, effectively solving the technical problems of poor robustness of multimodal fusion, weak long-term feature mining, and insufficient interpretability of reasoning in the prior art.

[0034] A dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method includes the following steps: Step S1: Synchronous acquisition of heterogeneous multimodal data; collect fault description text, on-site fault images and equipment operation video sequences of the pharmaceutical equipment to be inspected. The text contains information such as fault phenomena, location of occurrence, and abnormal status, while the images and videos are used to reflect the spatial and temporal characteristics of the fault.

[0035] Step S2: Robust feature extraction; a multimodal encoder is used to extract features from each modality of data: for video data, a 3D-CNN combined with a Transformer architecture is used to extract temporal evolution features; for images, a residual network is used to extract spatial features; for text, a pre-trained language model is used to extract semantic features, resulting in a robust multimodal feature vector.

[0036] Step S3: Dynamically allocate weights based on the DWFF algorithm; evaluate the quality of each modal data using the modal quality detection operator to obtain the corresponding modal confidence; and adaptively weight each modal feature using the Dynamic Weighted Feature Fusion (DWFF) algorithm to reduce the interference of noisy data and low-quality modalities on the diagnostic results.

[0037] The quality confidence score for the i-th mode is calculated as follows: q i Where σ(·) is the Sigmoid normalization function, α is the quality adjustment coefficient, and s i Let be the data quality score for the i-th modality.

[0038] The confidence scores of each mode are normalized to obtain the dynamic weights of the i-th mode: Among them, w i Let be the final weight of the i-th mode, and n be the total number of modes participating in the fusion.

[0039] The features are weighted and fused according to the dynamic weights of each modality to obtain joint semantic features: Among them, F i Let F be the feature vector extracted from the i-th mode. fusionThis refers to the multimodal joint features after dynamic weighted fusion.

[0040] Step S4: Constructing prompt words for large models; convert the fused multimodal joint features into structured text vectors through semantic mapping, and generate structured instructions to guide the reasoning of large models based on the preset prompt word template of role, background, evidence and constraint.

[0041] Step S5: Collaborate with the knowledge graph to perform large-scale model reasoning; input structured prompts into the multimodal large-scale model fine-tuned by pharmaceutical fault domain data, introduce the GraphRAG (Graph Structure Retrieval Enhancement Generation) mechanism to retrieve the pharmaceutical equipment fault domain knowledge graph, and obtain the fault causal evidence chain; the large-scale model performs logical reasoning based on the retrieval results and multimodal features, and outputs a diagnostic report containing fault location, root cause analysis, and GMP compliance handling suggestions.

[0042] Specifically, see the attached document. Figure 2 As shown, the bottom layer of this system is a multi-source heterogeneous data layer. This layer is the system's underlying data support layer, used to provide the raw multi-source heterogeneous data required for fault diagnosis. Specifically, it includes three types of data sources: equipment documents, defect images, and maintenance videos, which are completely consistent with the annotations in the diagram. This layer corresponds to step S1 of the method, where the front-end edge acquisition terminal synchronously collects the three types of data to be diagnosed: first, the fault description text input by the maintenance personnel, which includes semantic information such as fault phenomena, location of occurrence, and abnormal state; second, the on-site defect image of the equipment fault location, used to reflect the spatial details of the fault; and third, the long-term maintenance video of the equipment operation, used to capture the dynamic evolution characteristics of the fault. After the collection is completed, the layer performs preprocessing operations on the raw data: for defect images, a cleaning operation is performed to remove environmental noise caused by dust and light fluctuations at the pharmaceutical site; for maintenance videos, a keyframe extraction operation is performed to compress redundant time-series data and reduce the computational overhead of subsequent processing, which completely corresponds to the annotations "cleaning and keyframe extraction" in the diagram; the preprocessed data of all types is synchronously transmitted upwards to the knowledge encoding and fusion layer to provide standardized input for subsequent feature extraction.

[0043] Specifically, see the attached document. Figure 2 As shown, the knowledge encoding and fusion layer is located above the multi-source heterogeneous data layer. This layer is the core knowledge processing layer of the system, used to perform feature encoding and knowledge fusion on heterogeneous data to construct a unified multimodal knowledge index, which corresponds completely to the "multimodal knowledge index" module in the diagram. This layer sequentially completes steps S2 and S3 of the method, and the specific implementation process is as follows: (1) Robust feature extraction Reference Appendix Figure 2 As shown, the layer contains three types of dedicated encoders, which are completely consistent with the annotations in the figure: Text Encoder (BERT / NER): Used to perform semantic encoding on device documents and fault description text, extract fault entities, relationships and deep semantic features from the text, and realize the vectorized expression of text data; Visual Encoder (CLIP-ViT): Used to encode visual features of cleaned defect images, combined with residual networks to enhance feature robustness and extract spatial details of faults in the images; Temporal Encoder (TCN): Used to encode temporal features in maintenance videos after keyframe extraction. Combining 3D-CNN and Transformer architectures, it extracts the dynamic evolution features of faults in long-term videos, capturing subtle early signs of faults. Through the above encoding, three independent feature vectors of a unified dimension are finally obtained, providing a foundation for subsequent fusion.

[0044] (2) DWFF dynamic weighted fusion After feature extraction is completed, the cross-modal alignment unit of the layer adopts the DWFF dynamic weighted fusion algorithm, which adaptively allocates weights according to the data quality of each modality. Its mathematical logic is as follows: First, calculate the quality confidence score for the i-th mode: q i Where σ(·) is the Sigmoid normalization function, α is the quality adjustment coefficient, and s i Let be the data quality score for the i-th modality.

[0045] The confidence scores of each mode are normalized to obtain the dynamic weights of the i-th mode: Among them, w i Let be the final weight of the i-th mode, and n be the total number of modes participating in the fusion.

[0046] The features are weighted and fused according to the dynamic weights of each modality to obtain joint semantic features: Among them, F i Let F be the feature vector extracted from the i-th mode. fusion This refers to the multimodal joint features after dynamic weighted fusion.

[0047] Reference Appendix Figure 2As shown, after fusion, the structured fault knowledge is stored in the Neo4j structured knowledge graph, and the multimodal features are stored in the VectorDB vector library. Cross-modal alignment is performed between the two types of storage to eliminate the semantic gap between heterogeneous modalities, which is completely consistent with the "cross-modal alignment" annotation in the figure. Finally, a unified multimodal knowledge index is constructed to provide support for subsequent reasoning.

[0048] Reference Appendix Figure 2 As shown, above the knowledge encoding and fusion layer is the source reasoning layer, which is the core reasoning layer of the system. This layer contains three core modules: user input unit, multimodal retrieval unit (GraphRAG), and multimodal large model (MLLM), consistent with the diagram annotations. This layer sequentially completes steps S4 and S5 of the method, with the specific implementation process as follows: (1) Construction of structured prompt words Reference Appendix Figure 2 As shown, the user input unit first receives the diagnostic data input by the maintenance personnel, including three types of data: text, images, and videos, which are completely consistent with the label "User Input (Text / Image / Video)" in the diagram. The system maps the fused joint features into natural language descriptions, and based on the preset "role, background, evidence, and constraint" four-in-one template, it assembles and generates standardized reasoning guidance instructions, clarifies the expert identity, reasoning task, and constraints of the large model, and guides the large model to perform accurate diagnosis.

[0049] (2) GraphRAG causal reasoning Reference Appendix Figure 2 As shown, after receiving user input, the multimodal retrieval tool (GraphRAG) performs a joint search on the multimodal knowledge index below, which is completely consistent with the annotation in the figure; see attached document. Figure 3 , 4 As shown, the system pre-constructs a domain knowledge graph for pharmaceutical equipment failures, storing causal relationships across the entire industry. The retrieval tool can quickly extract matching causal chains based on failure keywords, providing reliable domain knowledge support for the large model. Subsequently, the Multimodal Large Model (MLLM) receives guidance instructions and the retrieved causal evidence chains, performs logical reasoning based on the domain-fine-tuned MLLM, and completes accurate causal tracing by comparing on-site features with standard failure features, perfectly corresponding to the function of the MLLM shown in the diagram.

[0050] Reference Appendix Figure 2 As shown, above the source tracing and reasoning layer is the intelligent decision-making layer. This layer is the system's top-level output layer, used to output the final interpretable fault diagnosis report, which is completely consistent with the label "interpretable fault diagnosis report" in the diagram. (See attached reference.) Figure 5As shown, the diagnostic report includes fault type identification results, fault root cause analysis, targeted maintenance suggestions, and GMP compliance guidance information, enabling the visualization and traceable storage of diagnostic results to meet the compliance management needs of the pharmaceutical industry; at the same time, the report can provide incremental data for the knowledge graph, enabling continuous iterative optimization of the system.

[0051] Example 2 The following describes the complete implementation process of this invention in detail using the fault diagnosis of an integrated packaging and palletizing machine as an example. A practical demonstration diagram of the integrated packaging and palletizing machine is attached. Figure 6 As shown.

[0052] In this fault diagnosis, the user-input multimodal data is as follows: the text modality is "wavy wrinkles appear on the sealing side, and the edges feel rough," and the video modality's temporal features show a 0.08-second delay in the sealing and cutting blade closure and a 1.2° offset in the blade engagement angle. The system processes the data according to the above architecture and process: the multi-source heterogeneous data layer completes data acquisition and preprocessing, the knowledge encoding and fusion layer completes feature extraction, and DWFF calculation assigns a weight of 0.72 to the higher-quality video modality. After fusion, the GraphRAG of the source inference layer retrieves the causal chain of "sealing and cutting blade offset - sealing wrinkles." The large model combines the prompt words to perform inference, and finally, the intelligent decision layer outputs a diagnostic report, determining that the root cause of the fault is the offset of the sealing and cutting blade position, and the repair solution is to calibrate the blade angle. On-site verification shows that the diagnostic results are completely accurate and effectively solve the on-site fault problem.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method, characterized in that, Includes the following steps: Step S1: Synchronous acquisition of heterogeneous multimodal data; collection of fault description text, on-site fault images, and equipment operation video sequences of the pharmaceutical equipment under inspection; Step S2: Robust feature extraction; a multimodal encoder is used to extract features from each modality of the data to obtain multimodal independent feature vectors of a unified dimension; Step S3: Dynamically allocate weights based on the DWFF algorithm; evaluate the quality of each modality data through the modality quality detection operator to obtain the confidence level of each modality quality; calculate the dynamic weight of each modality based on the confidence level; perform weighted fusion of multimodal features according to the dynamic weight to obtain multimodal joint semantic features; Step S4: Constructing prompt words for large models; mapping the joint semantic features into structured text, and generating structured instructions to guide the reasoning of large models based on a preset four-in-one template; Step S5: Collaborate with the knowledge graph to perform large-scale model reasoning; retrieve the knowledge graph of pharmaceutical equipment failure domain through the GraphRAG mechanism to obtain the causal evidence chain of the failure, and perform reasoning based on the structured instructions and causal evidence chain of the multimodal large model to output a GMP-compliant failure diagnosis report.

2. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to claim 1, characterized in that, In step S1, heterogeneous multimodal data is acquired synchronously, including the following steps: S1.1: Synchronously collect the fault description text of the pharmaceutical equipment under inspection. The text includes the fault phenomenon, the location of occurrence, and semantic information of the abnormal state. S1.2: Synchronously acquire on-site fault images of the pharmaceutical equipment under inspection to reflect the spatial details of the fault; S1.3: Synchronously acquire video sequences of the equipment operation of the pharmaceutical equipment under inspection to capture the dynamic temporal evolution characteristics of the fault; S1.4: Perform a cleaning operation on the acquired defect images to remove environmental noise caused by dust and light fluctuations at the pharmaceutical site; S1.5: Perform keyframe extraction on the acquired maintenance video, compress redundant time-series data, and reduce the computational overhead of subsequent processing.

3. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to claim 1, characterized in that, In step S2, robust feature extraction includes the following steps: S2.1: A text encoder combining BERT and NER models is used to perform semantic encoding on the fault description text, extract fault entities, relationships and deep semantic features from the text, and obtain text feature vectors; S2.2: A visual encoder combining CLIP-ViT and residual network is used to encode visual features of the cleaned defect image, enhance feature robustness, extract fault space detail features in the image, and obtain image feature vectors. S2.3: A time-series encoder combining TCN, 3D-CNN and Transformer architecture is used to encode the time-series features of the maintenance video after extracting keyframes, extract the dynamic evolution features of the fault in the long time-series video, capture weak early fault signs, and obtain the video time-series feature vector. S2.4: Unify the above three types of feature vectors to the same dimension to obtain a multimodal independent feature vector with a unified dimension.

4. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to claim 1, characterized in that, In step S3, dynamic weight allocation is performed based on the DWFF algorithm; this includes the following steps: S3.1: Evaluate the data quality of each modality using the modal quality detection operator, and calculate the data quality score si for the i-th modality; S3.2: Calculate the quality confidence score for the i-th mode: ,in The Sigmoid normalization function, This is the quality adjustment coefficient; S3.3: Normalize the confidence scores of each mode to obtain the dynamic weights of the i-th mode: Among them, w i Here, n represents the final weight of the i-th mode, and n is the total number of modes participating in the fusion. S3.4: Weighted fusion of features based on the dynamic weights of each modality yields joint semantic features: Among them, F i Let F be the feature vector extracted from the i-th mode. fusion The multimodal joint features are obtained after dynamic weighted fusion; S3.5: Use the fused multimodal joint features as input for subsequent inference.

5. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to claim 1, characterized in that, In S4, prompt words are constructed for large models; Includes the following steps: S4.1, Combine the fused multimodal features It is converted into a structured text vector through semantic mapping; S 4.2 Based on the preset four-in-one prompt word template of role, background, evidence and constraint, assemble and generate structured instructions to guide the large model to perform reasoning.

6. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning method according to claim 1, characterized in that, In step S5, large-scale model reasoning is performed using collaborative knowledge graphs; this includes the following steps: S5.1: Input structured prompts into the multimodal large model fine-tuned with pharmaceutical fault domain data; S5.2: Introduces a graph-based retrieval enhancement generation (GraphRAG) mechanism to retrieve knowledge graphs of pharmaceutical equipment faults based on user-input fault keywords; S5.3: Extract matching fault causal evidence chains from the knowledge graph, including fault phenomena, root causes, maintenance measures, and GMP compliance requirements; S5.4: The multimodal large model performs logical reasoning based on the retrieved causal evidence chain and multimodal joint features; S5.5: By comparing on-site characteristics with standard fault characteristics, fault type identification and root cause location are completed; S5.6: Output a diagnostic report that includes fault location, root cause analysis, and GMP compliance handling recommendations.

7. A dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning system, characterized in that, From bottom to top, it includes: a multi-source heterogeneous data layer, a knowledge encoding and fusion layer, a source tracing and reasoning layer, and an intelligent decision-making layer; data flow between each layer is achieved through standardized data interfaces, forming a closed loop for fault diagnosis.

8. The dynamic weighted multimodal pharmaceutical equipment fault diagnosis reasoning system according to claim 7, characterized in that, The multi-source heterogeneous data layer includes the following sub-modules and operations: Equipment documentation data sources include historical maintenance records, fault description documents, and standard operating procedures; Defect image data source: contains on-site real-time images of the equipment malfunctioning parts, and performs cleaning operations to remove environmental noise; Maintenance video data source: contains long time-series videos of equipment operation and fault evolution, and performs key frame extraction to compress redundant data; The preprocessed data is synchronously transmitted to the knowledge encoding and fusion layer.

Images