Method for causal inference and quality prediction and early warning of large model facing multi-modal space-time data in intelligent manufacturing

Abstract

The application provides a method for large model assisted causal inference and quality prediction and early warning of multi-modal spatio-temporal data in intelligent manufacturing, aiming to solve the problems of difficulty in human-machine-object interaction data fusion, insufficient causal relationship mining and low accuracy of quality prediction and early warning in the whole process of intelligent manufacturing. The method firstly constructs a multi-modal spatio-temporal knowledge base by fusing static process knowledge and dynamic production data through offline initialization and online incremental updating; then adopts a hierarchical multi-agent architecture: the decomposition agent disassembles the task, the plug-in heterogeneous retrieval agent (vector, graph, time series) mines information in parallel, and the decision agent constructs a timeline, causal inference and evidence pruning, refines multi-source information into a quality causal chain, and generates an early warning result. The application realizes the deep fusion and causal correlation mining of multi-modal spatio-temporal data, significantly improves the accuracy, real-time performance and reliability of quality prediction and early warning in the whole process of intelligent manufacturing, and has good scalability.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and artificial intelligence, specifically to a large-model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing. Background Technology

[0002] Intelligent manufacturing, as a product of the deep integration of new-generation information technology and manufacturing, is driving industrial production towards intelligence, networking, and automation. In the entire intelligent manufacturing process—"production-monitoring-testing" scenario—high-frequency and complex interactions between humans, machines, and materials generate massive amounts of multi-source, heterogeneous spatiotemporal data. This includes static unstructured text and image data in the form of equipment manuals, process specifications, and quality standards, as well as dynamic time-series data, image data, and semi-structured text data generated in real time by equipment sensors, monitoring systems, and operational logs. How to efficiently extract causal relationships, accurately predict product quality, and promptly warn of anomalies from this vast amount of multimodal spatiotemporal data has become a core challenge in the field of intelligent manufacturing.

[0003] In recent years, large-model-assisted data mining and retrieval-enhanced generation technologies, as a new paradigm combining information retrieval and large language model generation capabilities, have shown great potential in knowledge-intensive industrial analysis tasks. However, existing methods have revealed many shortcomings when applied to complex intelligent manufacturing quality prediction and early warning scenarios. First, most methods primarily target single-modal data, making it difficult to effectively integrate multimodal information such as images and time-series sensor data commonly found in intelligent manufacturing scenarios, resulting in insufficient causal relationship mining. Second, existing methods typically construct static knowledge bases in a one-time manner, lacking real-time perception and incremental update mechanisms for high-frequency dynamic production data, failing to meet the stringent timeliness requirements of end-to-end quality monitoring. Furthermore, existing architectures are often singular and rigid; when faced with quality analysis tasks involving multiple data sources and complex causal couplings, their analysis strategies lack flexibility and specificity, making it difficult to guarantee the efficiency of causal inference and the accuracy of quality prediction.

[0004] Therefore, there is an urgent need for a novel large-scale model-assisted method for quality prediction and early warning. This method should be able to overcome the limitations of single-modality models, achieving deep fusion and causal mining of multimodal spatiotemporal data from the entire intelligent manufacturing process; it needs to possess dynamic learning capabilities, seamlessly integrating real-time production data streams into a knowledge system and establishing spatiotemporal causal relationships between data; simultaneously, its system architecture needs to be sufficiently flexible and intelligent, capable of adaptively scheduling different processing strategies according to the complexity of the quality analysis task, thereby providing accurate, reliable, and interpretable quality prediction, early warning, and root cause analysis for the entire intelligent manufacturing process. Summary of the Invention

[0005] Technical issues:

[0006] 1. Existing technologies struggle to effectively integrate and mine causal relationships among multimodal spatiotemporal data such as text, images, and time-series sensor data in smart manufacturing scenarios. This results in a one-sided identification of quality influencing factors, an inability to establish complete causal relationships, and an impact on the accuracy of quality prediction.

[0007] 2. Existing technologies lack the ability to process high-frequency dynamic production data and cannot effectively correlate real-time human-machine-object interaction data with key metadata such as equipment, workstations, and time, making it difficult to meet the requirements of full-process quality monitoring for data timeliness and spatiotemporal causal correlation.

[0008] 3. Existing analytical architectures are typically rigid. When faced with complex, multi-factor coupled quality problems, they cannot intelligently decompose analytical tasks and selectively mine causal information from different types of data sources (such as vector libraries, knowledge graphs, and time-series databases), resulting in low efficiency of causal inference and incomplete coverage of quality warnings.

[0009] Technical Solution: This invention proposes a large-model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing. Its core lies in constructing a dynamically evolving multimodal spatiotemporal knowledge base and utilizing a hierarchical collaborative intelligent agent system to intelligently handle complex quality analysis and early warning tasks. In the initial stage of method execution, the system constructs an offline knowledge base using industrial static data such as equipment manuals and process specifications. A document parser decomposes the text and image content into multimodal units containing original text, associated images, and metadata. Subsequently, a large multimodal model is used to generate image descriptions to enhance the text content, and the vectorized enhanced text is stored in a multimodal vector database. Simultaneously, extracted entities and relationships are used to construct a manufacturing knowledge graph. Based on this, the system continuously performs online incremental updates to dynamic real-time streams from industrial sites, such as equipment logs, monitoring images, and sensor data. Differentiated time windows and feature extraction strategies are adopted for text, high-frequency images, and high-frequency time-series data, and the processed information is synchronously updated to the vector database and knowledge graph to ensure the timeliness of the knowledge base. When the system receives a complex quality analysis task or a quality anomaly warning signal, the method employs a hierarchical multi-agent collaborative framework: First, the decomposition agent in the decomposition layer uses a large language model to analyze the task intent and intelligently breaks it down into multiple logical sub-tasks pointing to specific data modalities. Then, in the retrieval layer, a pluggable heterogeneous retrieval agent module containing vector-based, graph-based, and time-series data-based agents is launched in parallel according to the characteristics of the sub-questions. These agents employ strategies such as spatiotemporal aligned retrieval, weighted multi-hop graph traversal, and online time-series analysis to efficiently retrieve relevant information fragments, subgraph structures, and analysis data from multimodal vector databases, manufactured knowledge graphs, and time-series databases. Second, in the decision and generation layer, the decision agent performs a refined evidence chain construction process on the retrieved information: it first sorts all fragments according to timestamps to construct a coherent event timeline; then, it uses a cross-modal cross-attention model to perform secondary relevance evaluation and rearrangement of each piece of evidence on the timeline, filtering out irrelevant information; finally, it combines the filtered and sorted evidence into a structured final context. Finally, based on this context, the large language model generates quality prediction and early warning results and root cause analysis reports, and simultaneously generates a structured causal source list. This list contains all the metadata needed to access evidence such as original images and time-series data, as well as their roles in the causal chain, so as to enable visualization and closed-loop feedback control in the manufacturing execution system.

[0010] The specific plan is as follows:

[0011] A method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, comprising the following steps:

[0012] a) Steps for building and dynamically updating the knowledge base:

[0013] i. Offline initialization is performed using static data from the field of intelligent manufacturing. The static data includes equipment manuals, process specifications, quality inspection standards, and historical quality defect cases. Multimodal data units containing text, images, and metadata are generated through parsing. An initial multimodal vector database and manufacturing knowledge graph are constructed based on the multimodal data units.

[0014] ii. Utilize dynamic real-time data from the entire intelligent manufacturing production-monitoring-testing process to perform online incremental updates to the multimodal vector database and manufacturing knowledge graph. The dynamic real-time data includes production operation logs, production line monitoring images, equipment sensor time-series data, and quality inspection data, and adopt differentiated processing strategies for different types and frequencies of data.

[0015] b) Hierarchical multi-agent collaborative problem-solving steps:

[0016] i. Decomposition layer: Receives complex quality analysis tasks or early warning signals from the input. The decomposition agent analyzes the intent of the task and determines whether decomposition is necessary. If so, the original task is decomposed into multiple sub-tasks that are logically related and point to specific data modes.

[0017] ii. Retrieval Layer: Based on the subtasks decomposed by the decomposition layer, the corresponding heterogeneous retrieval agents are invoked to mine multi-source information in parallel and return the corresponding heterogeneous information fragments;

[0018] iii. Decision and Generation Layer: The decision-making agent aggregates all heterogeneous information fragments returned by the retrieval agents, performs information verification, causal link inference, and evidence chain integration, and finally generates a coherent, accurate, and interpretable quality prediction and early warning result and root cause analysis report.

[0019] Furthermore, the offline initialization specifically includes: First, using a document parser to split static data into image-text units, each unit containing original text paragraphs, associated images, and metadata; then, using a multimodal large model to generate detailed text descriptions for each associated image, and concatenating these descriptions with the original text paragraphs to form enhanced text; subsequently, using a pre-trained language model to vectorize the enhanced text, and storing it along with the corresponding metadata in a multimodal vector database; finally, using an entity and relation extraction model to extract industrial entities, attributes, and relation triples from the enhanced text to construct a manufacturing knowledge graph, wherein image entity nodes contain attributes pointing to their original storage paths, and the relationships between entities include process causal relationships and quality influence relationships.

[0020] Furthermore, the differentiated processing strategy for the online incremental update is as follows: For text data: production operation logs and quality inspection records, entity recognition is directly performed, and the device ID, workstation ID, timestamp, and key metadata of quality indicators are extracted, vectorized, and stored in a vector database; For high-frequency image data: a time window is set, and for continuous image sequences within the window, a multimodal large model is used to generate not only single-frame descriptions but also a temporal change summary describing the dynamic changes of image content within the time window. This summary is then vectorized and stored in the vector database, with metadata including the device ID and time window; For high-frequency sensor data: a time window is also set, and statistical features are extracted from the original time-series data within the window to obtain mean, variance, peak value, and trend features. These structured features are then vectorized and stored in the vector database; The newly extracted entities and relationships from the above three types of dynamic data are updated in the manufacturing knowledge graph.

[0021] Furthermore, the process of the decomposition agent judging and breaking down the quality analysis task specifically includes: First, using a large language model to perform a binary classification prompt to determine whether the quality analysis task is a "single factor" or a "multi-factor coupling"; if it is determined to be a "multi-factor coupling", then using the large language model to perform a structured decomposition prompt to decompose the task into 2 to N logically progressive or parallel sub-tasks. Each sub-task retains the core keywords of the original task while specifying the data type it needs to query (such as "query the operating status description of equipment A at time T", "query the causal relationship between equipment A and process B", "query the vibration trend and quality index changes of equipment A from time window T to T+1").

[0022] Furthermore, the heterogeneous retrieval agents in the retrieval layer operate as follows: a) A vector-based fine-grained information retrieval agent: employing a two-stage retrieval strategy with spatiotemporal alignment. In the first stage, based on the device ID and time window metadata in the sub-question, a fast filter is performed in the vector database to narrow down the candidate vector subset. In the second stage, the sub-question is vectorized, semantic similarity is calculated within the candidate subset, and the K most relevant information fragments are returned. b) A graph-based relational information retrieval agent: identifying seed entities from the sub-question, performing weighted multi-hop traversal in the knowledge graph, where the edge weights are dynamically adjusted according to the relation type and relevance to the question, and finally retrieving and returning the subgraph structure most relevant to the question. c) A time-series data-based causal analysis retrieval agent: retrieving raw data from the time-series database based on the device ID and time window in the sub-task, and performing online causal analysis, including anomaly detection, trend analysis, Fourier transform, and Granger causality test, returning the raw data, causal association strength assessment, and analysis conclusions.

[0023] Furthermore, the decision-making agent in the decision-making and generation layer operates as follows: a) Multi-source information aggregation: Heterogeneous information fragments from different retrieval agents, including text snippets, knowledge subgraphs, time-series data, and their analysis conclusions, are aggregated; b) Quality event timeline construction: Based on the metadata timestamps associated with each information fragment, all fragments are time-aligned and sorted to construct a quality event timeline that reflects the chronological order and causal propagation path of quality anomaly events; c) Relevance rearrangement and pruning: Based on the event timeline, a cross-modal cross-attention model is used to perform a secondary scoring of the precise relevance of each information fragment to the original user query, and redundant or noisy information with relevance scores below a preset threshold is filtered out; d) Structured evidence context synthesis: The sorted and filtered highly relevant information fragments are integrated into a clear and logically coherent structured text block as the final context; e) Traceable early warning result generation: Based on the final context, the large language model is driven to generate quality prediction early warning results and root cause analysis reports, and a structured causal traceability list containing all adopted evidence is generated simultaneously, where each traceability item contains metadata for accessing the original data and its role in the causal chain.

[0024] Beneficial effects:

[0025] 1. High-fidelity multimodal spatiotemporal information fusion and causal understanding have been achieved. This invention creates context-rich enhanced data units by converting image content into text descriptions and combining them with the original text. This approach not only preserves independent modal information but also establishes strong correlations between images and text at the source of data processing. This enables the model to perform causal inference based on more comprehensive information, significantly improving the depth of identification and prediction accuracy of factors affecting the quality of the entire intelligent manufacturing process.

[0026] 2. Possesses powerful dynamic data processing and spatiotemporal correlation capabilities. This invention designs differentiated processing and update strategies for dynamic data (text, images, sensor data) at different frequencies, and uses metadata to imprint device and time information on all information. This constructs a unified spatiotemporal reference system, enabling the system to handle queries involving specific points in time or time periods, taking the latest on-site conditions into account, and greatly enhancing the timeliness of the method and its practical value in the quality monitoring of the entire intelligent manufacturing process.

[0027] 3. A flexible, efficient, and scalable intelligent agent collaboration framework was constructed. The hierarchical multi-agent architecture adopted in this invention decomposes complex problems into smaller, manageable parts, and enables targeted parallel retrieval of multiple database types through pluggable heterogeneous retrieval agents. This architecture not only significantly improves the efficiency of problem handling and the comprehensiveness of information retrieval, but its modular design also makes the system easily expandable, allowing for convenient access to new data sources or more advanced analysis models, and supports encapsulation as microservice components for integration into the intelligent manufacturing end-to-end collaborative optimization platform.

[0028] 4. The invention significantly improves the reliability and logical consistency of quality early warnings through the construction of a refined causal evidence chain. It uniquely introduces a mechanism for constructing a timeline of quality events and rearranging causal correlations. This goes beyond simply piling up information; instead, it reconstructs discrete, heterogeneous data points into a logically rigorous causal chain for quality through time-series alignment and precise causal correlation calculations. This effectively eliminates noise interference, ensuring the causal coherence and root cause analysis accuracy of the final quality prediction and early warning results. Attached Figure Description

[0029] Figure 1 This is an offline initialization flowchart showcasing a large-scale model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing.

[0030] Figure 2 It is an online incremental update flowchart demonstrating a large-scale model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing.

[0031] Figure 3 This is a schematic diagram of the main principle of the method of the present invention. Detailed Implementation

[0032] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0033] like Figure 3 As shown, this invention provides a large-model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing. The specific implementation of its core process and key technical modules is as follows:

[0034] a) Knowledge base construction and dynamic update steps

[0035] The knowledge base of this invention is a multimodal knowledge center that can integrate static historical data and dynamic real-time data and evolve over time. Its construction and updating are divided into two stages: offline initialization and online incremental update.

[0036] Offline initialization:

[0037] like Figure 1 As shown, this phase aims to establish a comprehensive and structured foundation of industrial domain knowledge for the system.

[0038] 1. Data Acquisition and Preprocessing: First, collect static data from the smart manufacturing field, such as PDF operation manuals for specific equipment models, Word-format process specification documents, quality inspection standards, and historical quality defect case libraries containing text and images. Use a document parser, such as a model combining OCR technology and layout analysis, to break down these unstructured documents into multiple independent "text and image units." Each unit is logically cohesive; for example, a chapter in a manual, an operating procedure, or a fault case contains the original text paragraph, directly related images (such as equipment structure diagrams and circuit diagrams), and its metadata (such as document source, chapter title, equipment model, etc.).

[0039] 2. Image Content Textification: To enable the language model to understand image information, this invention utilizes a pre-trained Visual Language Model (VLM), such as the BLIP-2 framework, to generate detailed text descriptions for the images in each image-text unit. For example, for a "main bearing lubrication system diagram," the VLM-generated description is not just "This is a bearing," but rather "A schematic diagram of the main bearing lubrication system, showing the oil pump, filter, pressure gauge, and oil pipeline leading to the bearing, wherein the oil pump model is XZ-300."

[0040] 3. Enhanced Text Construction and Vectorization: The image text description generated by VLM is concatenated with the original text paragraphs in the image-text unit to form an "enhanced text" with higher information density and richer context. Subsequently, a powerful pre-trained language model (such as BGE-M3) is used as a text encoder to convert these "enhanced texts" into high-dimensional semantic vectors. These vectors, along with their corresponding metadata (document source, device ID, associated image path, etc.), are stored in a multimodal vector database (such as Milvus or FAISS).

[0041] 4. Manufacturing Knowledge Graph Construction: While generating enhanced text, an entity and relation extraction model based on a Large Language Model (LLM) (such as the strategy adopted by LightRAG) is used to automatically identify and extract industrial entities (such as 'Equipment-A', 'Sensor-S1', 'Fault-F01'), entity attributes (such as 'Equipment-A's rated power is 15kW'), and relational triples between entities (such as 'Equipment-A' - 'Contains' -> 'Sensor-S1'; 'Fault-F01' - 'Possible Cause' -> 'Insufficient Lubricating Oil'). These triples are stored in a graph database (such as Neo4j) to construct an initial manufacturing knowledge graph. In particular, for entities identified from images (such as "main bearing"), their nodes in the graph will contain a special attribute that points to the storage path of their original image, thereby achieving cross-modal integration of the knowledge graph and visual information.

[0042] Online incremental updates:

[0043] like Figure 2 As shown, this stage ensures that the knowledge base can perceive the dynamic changes in the industrial site in real time.

[0044] 1. Text Data Update: For low-frequency or semi-structured text data, such as daily operation logs generated by the equipment, the system monitors the log files in real time. Once a new entry is generated, entity recognition is immediately performed to extract key metadata such as device ID, timestamp, operator, and alarm code. Then, the content is vectorized and stored in a vector database.

[0045] 2. High-Frequency Image Data Update: For high-frequency image sequences from production line monitoring cameras, a time window-based summarization strategy is employed. For example, a one-minute time window is set, and a multimodal large model (MLLM) is used to analyze the continuous image sequence within this one-minute period (e.g., 600 frames). This not only generates descriptions of individual frames but, more importantly, generates a "time-series change summary," such as: "In the past minute, the conveyor belt operated smoothly, and robotic arm A completed 10 grasp-place cycles without any abnormal jitter or lag." This summary is vectorized and stored in a vector database, with its metadata explicitly annotating the corresponding device ID and the one-minute time window range (start and end timestamps).

[0046] 3. High-Frequency Sensor Data Update: For high-frequency time-series data from equipment vibration, temperature, and pressure sensors, a time window strategy is also employed. For example, a 5-second time window is set, and online statistical feature extraction is performed on the raw time-series data (e.g., 5000 data points) within the window to calculate a series of structured features such as mean, variance, kurtosis, trend slope, and the dominant frequency after Fourier transform. These features constitute a feature vector, which is then vectorized and stored in a vector database. This processing method avoids storing massive amounts of raw data points, instead storing compact features that reflect the core state of the data.

[0047] 4. Knowledge Graph Synchronous Update: In the above three dynamic data processing processes, newly extracted entities with persistent value (such as a new alarm event) and relationships (such as the association between the alarm and the status of a specific device) will be synchronously updated into the knowledge graph, realizing the dynamic evolution of the graph.

[0048] b) Hierarchical multi-agent collaborative problem-solving steps

[0049] When the system detects a quality anomaly warning signal or receives a quality analysis task (e.g., "Analyze the root cause of the abnormal surface roughness of the product processed by machine tool A from 2 pm to 4 pm yesterday, and correlate it with relevant vibration data, maintenance records and process procedures"), the system will process it collaboratively through a three-layer intelligent agent architecture.

[0050] i. Decomposition layer:

[0051] The decomposition agent is responsible for this. This agent first determines the complexity of the question. It invokes a large language model (LLM, such as GPT-4 or Llama3) and uses a pre-defined binary classification prompt: "Please determine whether the following question is a 'single intent' or a 'multiple intent' question. Single intent questions typically query only one type of information, while multi-intent questions require combining different types of information to answer. Question: [User-input question]".

[0052] If the determination is "multiple intents", the decomposition agent will continue to call LLM, using a structured decomposition prompt: "Please decompose the following complex industrial problem into 2 to 5 logically clear and independently executable subproblems. Each subproblem should, as far as possible, point to a specific data type (text description, relation, or time series trend) and retain the core keywords of the original problem (such as device ID, timestamp). The list of decomposed subproblems should be as follows:\n1. [Subproblem 1]\n2. [Subproblem 2]\n...".

[0053] For the example above, the decomposition result might be:

[0054] 1. Query the time-series data of machine tool A for vibration, temperature, etc., from 2 PM to 4 PM yesterday, and analyze its abnormal patterns and causal relationship with quality indicators. (Refer to time-series data)

[0055] 2. Retrieve quality inspection records, fault alarm logs, or operation records of machine tool A within a similar time period. (Referring to text in the vector database)

[0056] 3. Locate historical quality defect cases and standard operating procedures (SOPs) related to the abnormal surface roughness of machine tool A. (Refer to vector databases and knowledge graphs)

[0057] 4. Inquire about the causal relationships between machine tool A and other equipment and processes on the production line that may affect processing quality. (Refer to the knowledge graph)

[0058] ii. Retrieval layer:

[0059] It is executed by a pluggable, heterogeneous multi-source retrieval agent module. This module acts as a task scheduling center, launching one or more specialized retrieval agents in parallel based on the characteristics of the decomposed sub-problems.

[0060] • Vector-based fine-grained information retrieval agent: This agent handles sub-questions pointing to textual descriptions or features (such as sub-questions 2 and 3 above). It employs a spatiotemporally aligned two-stage retrieval strategy: In the first stage, it parses the metadata in the sub-questions, such as "machine tool A" and "yesterday afternoon 2 PM to 4 PM," and uses metadata filtering in the vector database to quickly narrow the search space to a subset of candidate vectors. In the second stage, it vectorizes the sub-questions themselves and then performs efficient semantic similarity calculations (such as cosine similarity) within this subset, returning the top-K most relevant information fragments (such as relevant log entries and SOP paragraphs).

[0061] • Graph-based Retrieval Agent: Responsible for handling sub-problems that explore complex relationships between entities (such as sub-problem 4 above). It first identifies the core "seed entity" (e.g., "machine A, bed T") from the sub-problems. Then, in the knowledge graph, it performs a weighted multi-hop graph traversal starting from this seed entity. The edge weights can be dynamically adjusted based on the relationship type (e.g., "physical connection" has a higher weight than "located in the same workshop") and its relevance to the problem. The traversal results in a subgraph structure containing the entities most relevant to the problem and their relationships.

[0062] • Temporal Causal Analysis Agent: This is a unique agent in this invention designed for intelligent manufacturing quality prediction and early warning scenarios. It is responsible for handling causal analysis sub-tasks (such as sub-task 1 above). Based on the device ID and time window in the sub-task, it directly retrieves the raw high-frequency sensor data within that time period from a time-series database (such as InfluxDB). It not only returns the data but, more importantly, performs online causal analysis. For example, it calls a pre-defined anomaly detection algorithm (such as Isolation Forest) to mark abnormal data points, performs Fourier transform analysis of the dominant vibration frequency, and uses Granger causality tests to assess the strength of the causal relationship between changes in equipment parameters and abnormal quality indicators. Finally, it returns a structured information package containing a visualization of the raw data, key statistical values, causal relationship assessment, and analysis conclusions.

[0063] iii. Decision and Generation Layer:

[0064] The decision agent is responsible for the final synthesis, refinement, and generation of all retrieved heterogeneous information. Its highly structured workflow ensures the quality of the final answer.

[0065] 1. Multi-source information aggregation: First, it aggregates all the results returned from the three retrieval agents, including text fragments, knowledge subgraphs, time-series data and their analysis conclusions.

[0066] 2. Event Timeline Construction: The decision-making agent traverses all information fragments and, based on their precise metadata timestamps, precisely aligns and sorts all fragments along a timeline. This constructs an "event timeline" that clearly reflects the sequence and causal chain of industrial events. For example, the timeline might show "14:05 Vibration sensor S1 reading begins to show abnormal spikes (causal confidence 0.87)" -> "14:08 Equipment operation log records 'spindle overload' alarm" -> "14:15 Quality inspection record shows workpiece surface roughness exceeds standard," forming a causal propagation chain of "abnormal vibration → spindle overload → quality defect."

[0067] 3. Relevance Reordering and Pruning: Based on the event timeline, to ensure that every piece of adopted evidence is highly relevant to the original user question, the decision-making agent employs a cross-modal cross-attention model. This model calculates the precise relevance score between each information fragment on the timeline (whether text, graph nodes, or time-series analysis conclusions) and the original complex user query. Subsequently, it filters out all redundant or noisy information with relevance scores below a preset threshold (e.g., 0.5) and reorders the highly relevant information.

[0068] 4. Structured Evidence Context Synthesis: Highly relevant information fragments, after being sorted and filtered, are integrated into a well-organized and logically coherent structured text block, serving as the "Final Context" that drives the generation of the final answer. This context is not merely a collection of information, but rather organized in a narrative manner with causal logic.

[0069] 5. Traceable Early Warning Result Generation: Finally, the decision-making agent submits this structured final context, along with the original quality analysis task, to a powerful generative large language model (such as GPT-4o). The LLM is required to generate the final quality prediction early warning results and root cause analysis report based solely on the provided context. Simultaneously, the system generates a structured causal attribution list. Each key conclusion in the early warning report is accompanied by one or more attribution tags, which link to specific entries in the causal attribution list and indicate the role of the evidence in the causal chain (root cause, intermediate transmission factor, or direct representation). Each attribution entry contains complete metadata, enabling engineers to trace the original data source with a single click. For example, clicking an attribution tag immediately displays the corresponding original sensor data graph or quality inspection record, achieving full interpretability and verifiability of the early warning results and supporting the feedback of analysis results to the manufacturing execution system to form a closed-loop quality control.

[0070] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A large-model-assisted causal inference and quality prediction and early warning method for multimodal spatiotemporal data in intelligent manufacturing, characterized by the following steps: as follows ： a) Steps for building and dynamically updating the knowledge base: i. Offline initialization is performed using static data from the field of intelligent manufacturing. The static data includes equipment manuals, process specifications, quality inspection standards, and historical quality defect cases. Multimodal data units containing text, images, and metadata are generated through parsing. An initial multimodal vector database and manufacturing knowledge graph are constructed based on the multimodal data units. ii. Utilize dynamic real-time data from the entire intelligent manufacturing production-monitoring-testing process to perform online incremental updates to the multimodal vector database and manufacturing knowledge graph. The dynamic real-time data includes production operation logs, production line monitoring images, equipment sensor time-series data, and quality inspection data, and adopt differentiated processing strategies for different types and frequencies of data. b) Hierarchical multi-agent collaborative problem-solving steps: i. Decomposition layer: Receives complex quality analysis tasks or early warning signals from the input. The decomposition agent analyzes the intent of the task and determines whether decomposition is necessary. If so, the original task is decomposed into multiple sub-tasks that are logically related and point to specific data modes. ii. Retrieval Layer: Based on the subtasks decomposed by the decomposition layer, the corresponding heterogeneous retrieval agents are invoked to mine multi-source information in parallel and return the corresponding heterogeneous information fragments; iii. Decision and Generation Layer: The decision-making agent aggregates all heterogeneous information fragments returned by the retrieval agents, performs information verification, causal link inference, and evidence chain integration, and finally generates a coherent, accurate, and interpretable quality prediction and early warning result and root cause analysis report.

2. The method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, as described in claim 1, is characterized by: The offline initialization specifically includes: First, using a document parser to split static data into image-text units, each unit containing original text paragraphs, associated images, and metadata; then, using a multimodal large model to generate detailed text descriptions for each associated image, and concatenating these descriptions with the original text paragraphs to form enhanced text; subsequently, using a pre-trained language model to vectorize the enhanced text, and storing it along with the corresponding metadata in a multimodal vector database; finally, using an entity and relation extraction model to extract industrial entities, attributes, and relation triples from the enhanced text to construct a manufacturing knowledge graph, wherein image entity nodes contain attributes pointing to their original storage paths, and the relationships between entities include process causal relationships and quality influence relationships.

3. The method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, as described in claim 2, is characterized by: The differentiated processing strategy for the online incremental update is as follows: For text data: production operation logs and quality inspection records, entity recognition is performed directly, and the device ID, workstation ID, timestamp, and key metadata of quality indicators are extracted, vectorized, and stored in a vector database. For high-frequency image data: a time window is set, and for continuous image sequences within the window, a multimodal large model is used to generate not only single-frame descriptions but also a temporal change summary describing the dynamic changes of image content within the time window. This summary is then vectorized and stored in the vector database, with metadata including the device ID and time window. For high-frequency sensor data: a time window is also set, and statistical features are extracted from the original time-series data within the window to obtain mean, variance, peak value, and trend features. These structured features are then vectorized and stored in the vector database. The newly extracted entities and relationships from the above three types of dynamic data are updated in the manufacturing knowledge graph.

4. The method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, as described in claim 3, is characterized by: The process of the decomposition agent judging and deconstructing the quality analysis task specifically includes: First, using a large language model to perform a binary classification prompt to determine whether the quality analysis task is a "single factor" or a "multi-factor coupling"; if it is determined to be a "multi-factor coupling", then using a large language model to perform a structured decomposition prompt to decompose the task into 2 to N logically progressive or parallel sub-tasks. Each sub-task retains the core keywords of the original task while clarifying the data type it needs to query.

5. A method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, as described in claim 4, is characterized by: The heterogeneous retrieval agent in the retrieval layer works as follows: a) Vector-based fine-grained information retrieval agent: adopts a two-stage retrieval strategy with spatiotemporal alignment. In the first stage, based on the device ID and time window metadata in the sub-question, it performs fast filtering in the vector database to identify a subset of candidate vectors. In the second stage, the sub-problems are vectorized, semantic similarity is calculated within the candidate subset, and the K most relevant information fragments are returned; b) Graph-based relational information retrieval agent: Seed entities are identified from the sub-problems, and weighted multi-hop traversal is performed in the knowledge graph, where the weight of the edges is dynamically adjusted according to the relation type and relevance to the problem, and finally the subgraph structure most relevant to the problem is retrieved and returned; c) Causal analysis retrieval agent based on time series data: Based on the device ID and time window in the sub-task, the original data is retrieved from the time series database, and online causal analysis is performed, including anomaly detection, trend analysis, Fourier transform and Granger causality test, and the original data, causal association strength assessment and analysis conclusions are returned.

6. The method for large-scale model-assisted causal inference and quality prediction and early warning based on multimodal spatiotemporal data in intelligent manufacturing, as described in claim 5, is characterized in that: The decision-making agent in the decision-making and generation layer operates as follows: a) Multi-source information aggregation: Collecting heterogeneous information fragments such as text snippets, knowledge subgraphs, time-series data, and their analysis conclusions from different retrieval agents; b) Quality event timeline construction: Based on the metadata timestamps associated with each information fragment, performing time-series alignment and sorting on all fragments to construct a quality event timeline that reflects the chronological order and causal propagation path of quality anomaly events; c) Relevance rearrangement and pruning: Based on the event timeline, a cross-modal cross-attention model is used to perform a secondary scoring of the precise relevance of each information fragment to the original user query, and filtering out redundant or noisy information with relevance scores below a preset threshold; d) Structured evidence context synthesis: Integrating the sorted and filtered highly relevant information fragments into a clear, logically coherent structured text block as the final context. e) Generation of traceable early warning results: Based on the final context, the large language model is driven to generate quality prediction early warning results and root cause analysis reports, and a structured causal traceability list containing all adopted evidence is generated simultaneously, where each traceability item contains metadata for accessing the original data and its role in the causal chain.

Images