Method for semantic alignment of industrial multi-modal data based on vector space and topological constraints

Abstract

This invention belongs to the field of information processing and discloses a semantic alignment method for industrial multimodal data based on vector space and topological constraints. It aims to solve the problem of semantic fragmentation and difficulty in unifying and associating multimodal data in industrial scenarios. By extracting features and performing structured parsing on each modality of data, and mapping them to a unified semantic vector space through a projection layer, the method introduces topological constraints derived from process drawings, etc., to verify neighbor relationships of candidate entities based on vector similarity matching. Furthermore, it combines a large model for contextual logic verification, thereby significantly improving the accuracy of cross-source entity alignment. An efficient vector database index is established to support hybrid retrieval, and a large model is used to assist in reasoning and rule base verification to achieve incremental updates of the knowledge graph. This invention effectively integrates vector semantics with the structural dependencies unique to industrial systems, achieving high-precision, evolvable cross-modal semantic alignment and intelligent association analysis.

Description

Technical Field

[0001] This invention belongs to the field of information processing, specifically relating to a semantic alignment method for industrial multimodal data based on vector space and topological constraints. Background Technology

[0002] With the deepening development of industrial intelligence and digital transformation, industries such as manufacturing, energy, chemicals, and equipment manufacturing are continuously generating massive amounts of multimodal data, covering various forms such as process documents, engineering drawings, on-site images, equipment timing signals, and system logs. This heterogeneous data contains rich information on equipment status, operational logic, and fault clues, but it differs significantly in semantic expression, structural form, and spatiotemporal scale, leading to severe information fragmentation and making it difficult to form a unified knowledge view. Especially in complex industrial scenarios, the same physical entity, such as a pump or valve, may be presented in different systems with different names, formats, or modalities, making it impossible to effectively correlate cross-source data and hindering the improvement of intelligent diagnosis, knowledge reuse, and autonomous decision-making capabilities.

[0003] Among these approaches, multimodal fusion methods based on vector space have received widespread attention in recent years. Existing solutions typically employ independent encoders to process text, images, or time-series data separately, and achieve single-modal retrieval through vector databases. However, such methods lack the ability to model the uniqueness and structural dependencies of entities in industrial contexts. On the one hand, the embedding spaces of each modality are isolated from each other, making it impossible to support cross-modal semantic interactions such as searching for images by text or searching for drawings by time sequence. On the other hand, relying solely on vector similarity for entity matching can easily lead to misalignment of similar devices with similar functions but different physical locations, such as misclassifying Pump-A and #1 pressurization pump as the same object while ignoring their topological differences in the process flow. Furthermore, traditional knowledge graph construction relies on manual rules and regularization templates, making it difficult to handle unstructured drawings and dynamic IoT events, and it cannot automatically evolve with new data.

[0004] Therefore, there is an urgent need for a multimodal alignment mechanism that can integrate vector semantics and industrial topological constraints to achieve accurate mapping of cross-modal data, precise entity identification, and continuous knowledge evolution in a unified semantic space, thereby supporting highly reliable industrial intelligent analysis and reasoning. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic alignment method for industrial multimodal data based on vector space and topological constraints, which can effectively solve the problems in the background art mentioned above.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A semantic alignment method for industrial multimodal data based on vector space and topological constraints includes the following specific steps:

[0008] Step 1: Preprocess multimodal industrial data, performing structured analysis and feature extraction on text, images, drawings, and time-series signals respectively;

[0009] Step 2: Construct a unified semantic vector space, map different modal data to the same dimension of the embedding space, and transform the embedding vectors through a learnable projection layer to make the final output a unified multidimensional vector.

[0010] Step 3: Perform semantic alignment of industrial entities based on topological constraints. Calculate the cosine similarity between the pairs of entities to be matched in a unified vector space. When the similarity is greater than the preset similarity, it is included in the candidate entity set. Obtain the topological neighbor set of the two candidate entities in the process system. Construct the multimodal context description of the candidate entities into a reasoning prompt template. Input it into the large model for logical judgment. Verify whether the functions, parameters and operating environment are consistent. After confirmation, merge the corresponding knowledge entries.

[0011] Step 4: Establish a semantic index for the vector database and support efficient hybrid retrieval. Store all the unified multidimensional vectors into the vector database, organize the data using a hierarchical navigable small-world (HNSW) index structure, set index parameters and search parameters, specify vector similarity thresholds and structured filtering conditions, and return the Top-K most relevant results.

[0012] Step 5: Trigger the large model-assisted reasoning and knowledge graph incremental update mechanism, integrate the search results to form an evidence package, input it into the large model for causal chain analysis, and verify the logical rationality through a preset rule base.

[0013] Furthermore, the method also includes:

[0014] For text documents, a row and column label inference algorithm is used to identify the table structure and the data is divided into blocks according to semantic paragraphs;

[0015] For image data, an optical character recognition model is used to extract text information, and a multi-dimensional visual feature vector is generated through a deep convolutional neural network.

[0016] For drawing element detection, target detection model is used to identify pump, valve and process equipment nodes, combined with skeleton extraction algorithm to track pipeline connection relationship, construct topology graph structure containing nodes and edges, and retain the spatial coordinate code of each node;

[0017] For time-series data collected by the device's sensors, the sliding window length and step size are set, and the data in each time window is input into the time-series feature encoder to extract its dynamic evolution mode.

[0018] In step 2, a large language model is used to encode the text data block, outputting a 768-dimensional text embedding vector. A visual transformer model is used to jointly encode the image and its OCR results, outputting a 512-dimensional image embedding vector. The topology map corresponding to the drawing is input into a graph neural network, using node features and adjacency matrix as input. Node-level embeddings are generated through a message passing mechanism, and a 512-dimensional global embedding vector of the drawing is obtained through global pooling. The time window features of the time series data are encoded using a time series transformer model, outputting a 128-dimensional time series embedding vector.

[0019] In step 5, for newly accessed data, a 512-dimensional vector is generated and compared with the existing entity database. If the similarity is less than 0.75, a new entity record is created. If it is greater than 0.9, it is automatically merged and the version number is updated. If it is between 0.75 and 0.9, it is added to the manual review queue.

[0020] Furthermore, the method also includes:

[0021] In step 1, during the text semantic segmentation process, document paragraphs containing technical parameter tables or fault logs are given priority to retain complete field alignment, and are given priority to be broken at punctuation marks to ensure semantic integrity.

[0022] Furthermore, the method also includes:

[0023] In the drawing processing stage of step 1, the target detection model adopts an improved YOLOv8 architecture and designs a special anchor frame size.

[0024] The skeleton extraction algorithm employs refined morphological operations combined with a direction tracking strategy to ensure the correct reconstruction of the connectivity of complex intersecting pipelines, with an error not exceeding the width of a single pixel.

[0025] Furthermore, the method also includes:

[0026] In step 2, the projection layer adopts a fully connected network structure, which includes a weight matrix W and a bias vector b. It is trained under supervision by an alignment loss function to minimize the Euclidean distance of the mapping vectors of the same entity in different modalities.

[0027] Furthermore, the method also includes:

[0028] In step 3, when calculating the Jaccard similarity coefficient, the topological neighbor set is expanded to the second-level neighborhood range, that is, it includes the neighbors of the neighbors.

[0029] Furthermore, the method also includes:

[0030] In step 3, the large model used for context consistency verification is fine-tuned by industrial manuals, maintenance reports, and process specification corpora to understand the physical meaning of professional terms and their manifestation under specific working conditions.

[0031] Furthermore, the method also includes:

[0032] In step 4, the HNSW index is constructed using a batch insertion strategy, which first performs clustering preprocessing on all vectors, and then groups them to build a hierarchical connection graph.

[0033] Furthermore, the method also includes:

[0034] In step 4, the structured conditions for hybrid retrieval support Boolean combination filtering based on multiple dimensions such as time range, equipment type, factory area number, and responsible person, and generate a comprehensive relevance score by using a weighted fusion strategy with vector similarity scores.

[0035] Furthermore, the method also includes:

[0036] The evidence package in step 5 consists of associated text descriptions, historical alarm records, screenshots of real-time trend curves, partial views of relevant equipment drawings, and upstream and downstream operating condition data. All elements are converted into vector form to participate in context injection.

[0037] In step 5, the rule base verification module has more than 100,000 built-in industrial logic rules, covering equipment operation boundary conditions, interlocking protection logic and temporal causal constraints; the version number management adopts an incrementing integer identifier, and a unique version snapshot is generated for each entity update.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention constructs a unified vector semantic space, enabling unified encoding and representation of various industrial data modalities such as text, images, drawings, and time-series signals, fundamentally solving the problem of semantic fragmentation between heterogeneous data. It introduces topological constraints based on the connection relationship between drawings and processes, superimposing structural dependency verification unique to industrial systems on top of vector similarity, significantly improving entity alignment accuracy when similar equipment has inconsistent naming in different systems, effectively avoiding the problem of mismatching homonymous and heteronymous objects. Employing a dual reasoning mechanism combining a large model and a rule base, it can automatically mine potential causal relationships from massive amounts of related information, generating diagnostic suggestions that conform to engineering logic. This approach enhances the interpretability and credibility of intelligent systems. A three-level incremental update strategy based on similarity intervals is designed to support the continuous evolution of the knowledge graph with new data, significantly reducing manual maintenance workload. Actual testing shows a reduction of over 80% in knowledge management costs. The overall method supports millisecond-level cross-modal hybrid retrieval, and combined with an efficient HNSW index structure, it improves efficiency by 5 to 20 times compared to traditional layer-by-layer filtering methods, meeting the real-time analysis needs of industrial sites. The system possesses excellent scalability; the core model, database, and alignment algorithm can all be replaced with other equivalent components, adapting to diverse deployment environments. Its effectiveness and robustness have been verified in complex industrial scenarios such as large-scale petrochemical and power industries. Attached Figure Description

[0040] Figure 1 The flowchart illustrates a semantic alignment method for industrial multimodal data based on vector space and topological constraints, as claimed in an embodiment of the present invention. Detailed Implementation

[0041] Currently, industries such as manufacturing, energy, chemicals, and equipment manufacturing continuously generate massive amounts of multimodal data, encompassing various forms including process documents, engineering drawings, on-site images, equipment timing signals, and system logs. This heterogeneous data contains rich information on equipment status, operational logic, and fault clues, but significant differences exist in semantic expression, structural form, and spatiotemporal scale, leading to severe information fragmentation and making it difficult to form a unified knowledge view. Especially in complex industrial scenarios, the same physical entity may be presented in different systems with different names, formats, or modalities, making it impossible to effectively correlate cross-source data and hindering the improvement of intelligent diagnosis, knowledge reuse, and autonomous decision-making capabilities. To address these technical problems, this invention proposes a semantic alignment method for industrial multimodal data based on vector space and topological constraints. This method effectively solves the problems mentioned in the background and is applied to a semantic alignment method for industrial multimodal data based on vector space and topological constraints.

[0042] Reference Appendix Figure 1 The overall technical solution architecture diagram of this invention illustrates the complete process from multimodal data input to incremental update of the knowledge graph.

[0043] A semantic alignment method for industrial multimodal data based on vector space and topological constraints includes the following specific steps:

[0044] Step 1: Preprocess multimodal industrial data, performing structured analysis and feature extraction on text, images, drawings, and time-series signals respectively;

[0045] Step 2: Construct a unified semantic vector space, map different modal data to the same dimension of the embedding space, and transform the embedding vectors through a learnable projection layer to make the final output a unified multidimensional vector.

[0046] Step 3: Perform semantic alignment of industrial entities based on topological constraints. Calculate the cosine similarity between the pairs of entities to be matched in a unified vector space. When the similarity is greater than the preset similarity, it is included in the candidate entity set. Obtain the topological neighbor set of the two candidate entities in the process system. Construct the multimodal context description of the candidate entities into a reasoning prompt template. Input it into the large model for logical judgment. Verify whether the functions, parameters and operating environment are consistent. After confirmation, merge the corresponding knowledge entries.

[0047] Step 4: Establish a semantic index for the vector database and support efficient hybrid retrieval. Store all the unified multidimensional vectors into the vector database, organize the data using a hierarchical navigable small-world (HNSW) index structure, set index parameters and search parameters, specify vector similarity thresholds and structured filtering conditions, and return the Top-K most relevant results.

[0048] Step 5: Trigger the large model-assisted reasoning and knowledge graph incremental update mechanism, integrate the search results to form an evidence package, input it into the large model for causal chain analysis, and verify the logical rationality through a preset rule base.

[0049] Furthermore, the method also includes:

[0050] For text documents, a row and column label inference algorithm is used to identify the table structure and the data is divided into blocks according to semantic paragraphs;

[0051] For image data, an optical character recognition model is used to extract text information, and a multi-dimensional visual feature vector is generated through a deep convolutional neural network.

[0052] For drawing element detection, target detection model is used to identify pump, valve and process equipment nodes, combined with skeleton extraction algorithm to track pipeline connection relationship, construct topology graph structure containing nodes and edges, and retain the spatial coordinate code of each node;

[0053] For time-series data collected by the device's sensors, the sliding window length and step size are set, and the data in each time window is input into the time-series feature encoder to extract its dynamic evolution mode.

[0054] In step 2, a large language model is used to encode the text data block, outputting a 768-dimensional text embedding vector. A visual transformer model is used to jointly encode the image and its OCR results, outputting a 512-dimensional image embedding vector. The topology map corresponding to the drawing is input into a graph neural network, using node features and adjacency matrix as input. Node-level embeddings are generated through a message passing mechanism, and a 512-dimensional global embedding vector of the drawing is obtained through global pooling. The time window features of the time series data are encoded using a time series transformer model, outputting a 128-dimensional time series embedding vector.

[0055] In step 5, for newly accessed data, a 512-dimensional vector is generated and compared with the existing entity database. If the similarity is less than 0.75, a new entity record is created. If it is greater than 0.9, it is automatically merged and the version number is updated. If it is between 0.75 and 0.9, it is added to the manual review queue.

[0056] Furthermore, the method also includes:

[0057] In step 1, during the text semantic segmentation process, document paragraphs containing technical parameter tables or fault logs are given priority to retain complete field alignment, and are given priority to be broken at punctuation marks to ensure semantic integrity.

[0058] Furthermore, the method also includes:

[0059] In the drawing processing stage of step 1, the target detection model adopts an improved YOLOv8 architecture and designs a special anchor frame size.

[0060] The skeleton extraction algorithm employs refined morphological operations combined with a direction tracking strategy to ensure the correct reconstruction of the connectivity of complex intersecting pipelines, with an error not exceeding the width of a single pixel.

[0061] Furthermore, the method also includes:

[0062] In step 2, the projection layer adopts a fully connected network structure, which includes a weight matrix W and a bias vector b. It is trained under supervision by an alignment loss function to minimize the Euclidean distance of the mapping vectors of the same entity in different modalities.

[0063] Furthermore, the method also includes:

[0064] In step 3, when calculating the Jaccard similarity coefficient, the topological neighbor set is expanded to the second-level neighborhood range, that is, it includes the neighbors of the neighbors.

[0065] Furthermore, the method also includes:

[0066] In step 3, the large model used for context consistency verification is fine-tuned by industrial manuals, maintenance reports, and process specification corpora to understand the physical meaning of professional terms and their manifestation under specific working conditions.

[0067] Furthermore, the method also includes:

[0068] In step 4, the HNSW index is constructed using a batch insertion strategy, which first performs clustering preprocessing on all vectors, and then groups them to build a hierarchical connection graph.

[0069] Furthermore, the method also includes:

[0070] In step 4, the structured conditions for hybrid retrieval support Boolean combination filtering based on multiple dimensions such as time range, equipment type, factory area number, and responsible person, and generate a comprehensive relevance score by using a weighted fusion strategy with vector similarity scores.

[0071] Furthermore, the method also includes:

[0072] The evidence package in step 5 consists of associated text descriptions, historical alarm records, screenshots of real-time trend curves, partial views of relevant equipment drawings, and upstream and downstream operating condition data. All elements are converted into vector form to participate in context injection.

[0073] In step 5, the rule base verification module has more than 100,000 built-in industrial logic rules, covering equipment operation boundary conditions, interlocking protection logic and temporal causal constraints; the version number management adopts an incrementing integer identifier, and a unique version snapshot is generated for each entity update.

[0074] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 1 involves preprocessing multimodal industrial data, performing structured parsing and feature extraction on text, images, drawings, and time-series signals, respectively. Specifically, step 1 includes four parallel and independent data processing sub-processes, each performing in-depth parsing on different modal data sources, laying the foundation for subsequent unified semantic encoding.

[0075] For text document processing, the original content is first obtained through optical scanning or direct reading of electronic files. Then, a row and column label inference algorithm is used to identify potential table structures. This algorithm constructs a two-dimensional grid model through joint analysis of character positions, font styles, and delimiter patterns, mapping the unstructured text stream to a structured form with clear row and column relationships. Based on this, the system segments the document into data blocks of 150 to 300 tokens in length according to semantic paragraphs. To ensure the integrity of critical information, when a paragraph contains a technical parameter table or fault log, the system prioritizes preserving complete field alignment to avoid incorrect separation of critical fields such as the pressure limit and 1.2 MPa due to line breaks. Furthermore, a sentence boundary detection mechanism is introduced, prioritizing breaks at punctuation marks such as periods and semicolons to ensure semantic coherence and logical consistency within each data block. Each generated data block is assigned a unique document source identifier, page number offset, and timestamp metadata to facilitate subsequent source tracing and context reconstruction.

[0076] For image data processing, the system first utilizes a high-precision optical character recognition (OCR) model to extract embedded text information from the image. This OCR model, trained on a large number of industrial field images, can effectively handle complex situations such as low light, blur, tilt, and partial occlusion. The extracted text information, along with the original image pixel data, is fed into a deep convolutional neural network. This network uses ResNet-152 as its backbone architecture, pre-trained on the ImageNet dataset, and then fine-tuned on a private dataset containing millions of industrial equipment images. The network's final output layer is replaced with a global average pooling layer, generating a 2048-dimensional visual feature vector. This vector not only encodes the overall appearance information of the image but also integrates the semantic content of the readable text, forming a primary representation of image-text joint representation.

[0077] For drawing processing, the system performs specialized primitive detection and connectivity tracking. First, an improved target detection model is used to identify standard process equipment nodes such as pumps, valves, instruments, and heat exchangers in the drawings. This model is customized based on the YOLOv8 architecture, and the anchor frame size set has been redesigned to better match the actual pixel ratio of small symbols such as shut-off valves and safety valves, thus improving the mAP index for small target detection by 17 percentage points. After node localization, the system initiates a skeleton extraction algorithm, which combines thinning morphological operations with a direction tracking strategy. The thinning operation gradually erodes pipe lines to a single pixel width, while direction tracking records the connection direction along the skeleton path, ensuring correct connectivity even in areas with dense pipe intersections, with geometric errors strictly controlled within a single pixel width. Finally, all identified nodes and their spatial coordinate codes, along with the edge connections derived from the skeleton algorithm, constitute a topological graph structure in the form of a directed acyclic graph. Each node object includes attributes such as its type label, center coordinates, bounding box size, and associated text annotations.

[0078] For time-series data acquired by the device's sensors, the system employs a dynamic sliding window strategy. Under default conditions, the window length is 60 seconds with a step size of 10 seconds, dividing the continuous time series into a series of overlapping time windows. Multi-channel sensor data within each time window, such as temperature, pressure, flow rate, and vibration, are organized into a two-dimensional matrix with dimensions of [number of channels, number of time points], and input into a dedicated time-series feature encoder. This encoder, built on an LSTM or Transformer architecture, can capture the dynamic evolution patterns, periodic regularities, and abrupt events within the data. It is worth noting that the sliding window strategy is not fixed. The system incorporates a device status monitoring module that analyzes the variance, mean, and first derivative of the time-series data in real time. When the device is detected to be in steady-state operation (e.g., variance below a preset threshold with no significant trend), the window length automatically extends to 120 seconds to capture longer-term performance drift trends; conversely, when a step change, alarm event, or absolute value of the derivative exceeds the threshold occurs, the window length shortens to 5 seconds to improve the response sensitivity to transient anomalies. Each time window is ultimately encoded into a high-dimensional feature vector, representing the overall operating status of the device within that time period.

[0079] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 2 involves constructing a unified semantic vector space to map different modal data to an embedding space of the same dimension. Specifically, the core of step 2 lies in establishing a cross-modal consistent 512-dimensional vector representation space, enabling data from different sources to be compared and manipulated within the same mathematical framework.

[0080] For the text data block generated in step 1, the system uses a large language model to encode it. This large language model has billions of parameters and has been fully pre-trained on general corpora and industrial technical documents. After inputting the text data block, the model outputs its corresponding 768-dimensional text embedding vector, which deeply encodes the semantic connotation, technical terms, and contextual logic of the text.

[0081] For image data, the system employs a visual transformer model to jointly encode the original image and the text extracted by OCR. The visual transformer first divides the image into fixed-size blocks, each of which is linearly embedded and added to its positional encoding. Simultaneously, the OCR text, after word segmentation, is also transformed through a word embedding layer. The two embeddings begin interacting and fusing in early layers of the model, undergoing multi-layered self-attention and cross-attention mechanisms, ultimately outputting a 512-dimensional image embedding vector. This vector achieves a deep fusion of visual content and textual semantics.

[0082] For the topology graph structure corresponding to the drawing, the system inputs it into a graph neural network. This graph neural network adopts a graph attention network (GAT) structure, whose core advantage lies in allowing different neighboring nodes to contribute differentiated weights based on their importance. During message passing, information from high-influence nodes such as key control valves and measuring instruments is assigned higher attention weights, thus being strengthened in the aggregation phase. After multiple rounds of message passing, each node obtains an embedded representation containing its local topological context. Subsequently, global pooling operations such as summation or average pooling are used to fuse all node embeddings, generating a 512-dimensional global embedding vector for the drawing. This vector fully characterizes the structural layout and equipment configuration of the entire process system.

[0083] For time-series data, the system employs a time-series transformer model to encode its time window features. This model utilizes a self-attention mechanism to capture long-range dependencies between time points, effectively modeling complex dynamic behaviors. The encoder ultimately outputs a 128-dimensional time-series embedding vector.

[0084] To bridge the differences in the original embedding dimensions across modalities, all the aforementioned embedding vectors—768-dimensional text vectors, 512-dimensional image vectors, 512-dimensional drawing vectors, and 128-dimensional temporal vectors—are fed into a learnable projection layer. This projection layer employs a fully connected network structure, containing a weight matrix W adapted to the input dimensions and a bias vector b. Training of this projection layer is supervised by a specially designed alignment loss function, aiming to minimize the Euclidean distance between the mapped vectors of the same physical entity across different modalities. For example, text paragraphs describing Pump-101, on-site photographs containing Pump-101 tags, Pump-101 nodes in drawings, and runtime temporal data of Pump-101, while having different original embedding dimensions, should have their 512-dimensional output vectors spatially close to each other after passing through the projection layer. Through this supervised learning, the system successfully constructs a cross-modal consistent 512-dimensional vector representation space.

[0085] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 3 performs semantic alignment of industrial entities based on topological constraints to achieve high-precision cross-source entity matching. Specifically, step 3 employs a three-level verification mechanism, sequentially performing coarse alignment, fine alignment, and contextual consistency verification to ensure the accuracy of the matching results.

[0086] The first stage is preliminary coarse alignment. In a unified 512-dimensional vector space, the system calculates the cosine similarity between the embedding vectors of two entities to be matched, such as a device mention from a text log and a device node from a drawing. When this similarity is greater than a preset threshold of 0.82, both are included in the candidate entity set. This stage utilizes an efficient approximate nearest neighbor search algorithm to generate the candidate set in sub-seconds from a database of millions of vectors, significantly reducing the computational load of the subsequent verification stage.

[0087] The second level is fine alignment, the core of which is the introduction of industrial topology constraints. The system obtains the topological neighbor sets of two candidate entities in the process system. The construction of this set is strictly based on drawings or system connection diagrams, determining the upstream and downstream directly connected device nodes of each entity. To further enhance robustness, the topological neighbor set not only includes directly adjacent device nodes but also extends to the second-level neighborhood, i.e., the neighbors of neighbors, to capture a broader range of process context dependencies. Subsequently, the system calculates the Jaccard similarity coefficient between these two expanded neighbor sets. The Jaccard coefficient is defined as the ratio of the intersection size to the union size of the two sets. If the coefficient is greater than 0.6, it is determined that the two are consistent in industrial topology, strongly supporting that they are the same physical entity. This step effectively solves the problem of mismatch caused by functional similarity but different locations.

[0088] The third level is contextual consistency verification. Even if the first two levels are passed, there may still be cases where the semantics are similar but the physical entities are different. To address this, the system constructs a structured reasoning prompt template from the multimodal contextual descriptions of candidate entities, including their textual descriptions, associated images, partial views of drawings, and historical time-series features. This template is input into a domain-fine-tuned large model. The training corpus of this large model covers a large number of industrial manuals, maintenance reports, and process specifications, enabling it to accurately understand the physical meaning of technical terms such as bearing wear and sudden pressure increases, and their manifestations under specific operating conditions. The task of the large model is to logically judge the functions, technical parameters, operating environment, and historical events of two candidate entities to verify whether they match. If the judgment result is contradictory or highly inconsistent, the matching pair is excluded; otherwise, the matching is confirmed as successful, and the knowledge entry merging operation is triggered.

[0089] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 4 establishes a semantic index for the vector database and supports efficient hybrid retrieval. Specifically, step 4 aims to provide efficient storage and query capabilities for massive 512-dimensional vectors.

[0090] The system stores all uniformly encoded 512-dimensional vectors into a high-performance vector database. This database uses a hierarchical, navigable small-world (HNSW) index structure to organize the data. To handle industrial data storage scenarios with hundreds of millions of records, the system employs a batch insertion strategy during index construction: first, all vectors are preprocessed using K-means clustering to divide the data into several clusters; then, a hierarchical connection graph is built for each group. This method effectively reduces peak memory consumption during index construction, enabling large-scale index construction with limited hardware resources. The key parameters of the index are set as M=48 (maximum number of connections per node) and efSearch=128 (dynamic candidate set size during search), achieving an optimal balance between recall and query latency.

[0091] This index supports powerful combined query patterns. Users can specify not only a query vector for semantic similarity searching but also attach multiple structured filtering conditions. These conditions support Boolean combinations of AND / OR / NOT filtering based on multiple dimensions such as time range, equipment type, plant area number, and responsible person. During retrieval, the system first uses the HNSW index to quickly find a candidate set that meets the vector similarity requirements, and then applies structured filters to this candidate set. The final relevance ranking uses a weighted fusion strategy, taking into account both the vector similarity score and the matching degree of the structured conditions to generate a comprehensive relevance score and return the Top-K most relevant results. For example, a user can retrieve all data fragments that are semantically similar to temperature anomalies, expressed by a query vector, with a timestamp on the current day, and whose equipment type is centrifugal pump.

[0092] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 5 triggers a large-model-assisted reasoning and knowledge graph incremental update mechanism. Specifically, step 5 completes a closed loop from data retrieval to intelligent decision-making and then to knowledge evolution.

[0093] Once a search is complete, the system integrates all matching results to form a comprehensive evidence package. This evidence package is extremely rich in content, including but not limited to related text descriptions, historical alarm records, real-time trend curve screenshots, partial views of relevant equipment drawings, and upstream and downstream operating data. All these heterogeneous elements have been converted into vector form before being input into the large model and participate in inference through context injection, greatly improving the information completeness and accuracy of the large model's reasoning. Based on this evidence package, the large model performs causal chain analysis, attempting to infer potential root causes of failures or provide optimized operational suggestions.

[0094] However, to prevent the large model from generating illusory conclusions that contradict common engineering sense, the system introduces a pre-defined rule base for logical rationality verification. This rule base verification module contains over 100,000 industrial logic rules, covering equipment operating boundary conditions such as the motor bearing temperature not exceeding 95°C, interlocking protection logic such as the standby pump being prohibited from starting if the main pump is not running, and temporal causal constraints such as the cooling water flow rate not returning to zero before the motor starts. Any reasoning output that violates these fundamental engineering principles will be directly rejected by the rule base, ensuring the reliability and safety of the final output.

[0095] Meanwhile, the system continuously processes newly received data streams. For each new data unit, the system generates a corresponding 512-dimensional vector representation and compares it with all entity vectors in the existing entity database. Based on the similarity of the comparison results, the system executes a three-level incremental update strategy:

[0096] A. If the similarity between the old and new vectors is less than 0.75, it is considered a completely new and previously unseen entity, and the system will create a new independent entity record for it.

[0097] B. If the similarity is higher than 0.9, it is highly certain that the new data is the same object as an existing entity. The system will automatically merge the new data into the existing entity and update its entity version number. Version number management uses an incrementing integer identifier. Each update generates a unique version snapshot, supporting historical state retrospection and change difference comparison, which facilitates audit tracking and knowledge evolution path analysis.

[0098] C. If the similarity is between 0.75 and 0.9, the system cannot make a high-confidence automatic judgment. Therefore, the matching task is added to the manual review queue, awaiting final confirmation from domain experts. This human-machine collaborative mechanism ensures the accuracy and controllability of the knowledge system during its automated evolution.

[0099] As a specific application example, the method of this invention has been deployed on an intelligent operation and maintenance platform for a large-scale petrochemical plant. This platform needs to handle an additional 10TB of multi-source data streams daily, including DCS system logs, equipment inspection reports, high-definition monitoring videos, updated drawings, and real-time data from tens of thousands of sensors. On this platform, the response time for a single cross-modal retrieval, such as searching for relevant time-series segments, equipment drawings, and historical maintenance records based on a text describing outlet pressure fluctuations, is less than 200 milliseconds. After six months of continuous operation testing, the system achieved an accuracy of 96.8% in entity merging tasks, a 39 percentage point improvement over the 57.8% accuracy of traditional pure vector matching methods. Furthermore, this method significantly reduces manual maintenance workload through automated alignment and updates, and in practice, it can reduce knowledge management costs by more than 80%.

[0100] Based on the aforementioned first embodiment, the present invention can also be applied to intelligent substation inspection scenarios in the power industry. In this scenario, the multimodal data sources include: infrared thermal images and visible light images of the equipment captured by the inspection robot, voltage and current time-series data collected by the SCADA system, OCR recognition results of the equipment nameplates, and a structured text database of the equipment ledger.

[0101] In the preprocessing stage of step 1, the infrared image is processed separately, generating a thermal feature map using a temperature matrix extraction algorithm, and then fused with the visible light image. The drawing is replaced with an electrical primary wiring diagram in this scenario, and its element detection model is retrained to identify electrical equipment such as circuit breakers, disconnect switches, and transformers. The sliding window strategy for the time-series data is adjusted according to the grid load cycle; for example, a shorter window is used during peak electricity consumption periods to capture instantaneous overloads.

[0102] In the topology constraint alignment in step 3, the topology relationships provided by the electrical primary wiring diagram, such as bus connections and switch open / closed states, become the key verification criteria. For example, two circuit breaker entities from thermal imaging and SCADA data, respectively, even if their vector similarity is high, will have a very low Jaccard coefficient for their topological neighbor set if one is located on a 110kV bus and the other on a 10kV bus, thus correctly identifying them as different entities.

[0103] In the rule base of step 5, a large number of safety regulations specific to the power industry are integrated, such as prohibiting the opening of high-voltage room doors if the grounding switch is not closed. The domain fine-tuning corpus of the large model is also replaced with power safety regulations, equipment maintenance procedures, and accident analysis reports. In this way, the method of the present invention also achieves high-precision cross-modal semantic alignment and intelligent reasoning in the power scenario, verifying the universality and transferability of its technical solution.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0107] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A semantic alignment method for industrial multimodal data based on vector space and topological constraints, characterized in that, The specific steps include the following: Step 1: Preprocess multimodal industrial data, performing structured analysis and feature extraction on text, images, drawings, and time-series signals respectively; Step 2: Construct a unified semantic vector space, map different modal data to the same dimension of the embedding space, and transform the embedding vectors through a learnable projection layer to make the final output a unified multidimensional vector. Step 3: Perform semantic alignment of industrial entities based on topological constraints. Calculate the cosine similarity between the pairs of entities to be matched in a unified vector space. When the similarity is greater than the preset similarity, it is included in the candidate entity set. Obtain the topological neighbor set of the two candidate entities in the process system. Construct the multimodal context description of the candidate entities into a reasoning prompt template. Input it into the large model for logical judgment. Verify whether the functions, parameters and operating environment are consistent. After confirmation, merge the corresponding knowledge entries. Step 4: Establish a semantic index for the vector database and support efficient hybrid retrieval. Store all the unified multidimensional vectors into the vector database, organize the data using a hierarchical navigable small-world (HNSW) index structure, set index parameters and search parameters, specify vector similarity thresholds and structured filtering conditions, and return the Top-K most relevant results. Step 5: Trigger the large model-assisted reasoning and knowledge graph incremental update mechanism, integrate the search results to form an evidence package, input it into the large model for causal chain analysis, and verify the logical rationality through the preset rule base; The method also includes: For text documents, a row and column label inference algorithm is used to identify the table structure and the data is divided into blocks according to semantic paragraphs; For image data, an optical character recognition model is used to extract text information, and a multi-dimensional visual feature vector is generated through a deep convolutional neural network. For drawing element detection, target detection model is used to identify pump, valve and process equipment nodes, combined with skeleton extraction algorithm to track pipeline connection relationship, construct topology graph structure containing nodes and edges, and retain the spatial coordinate code of each node; For time-series data collected by the device's sensors, the sliding window length and step size are set, and the data in each time window is input into the time-series feature encoder to extract its dynamic evolution mode. In step 2, a large language model is used to encode the text data block, outputting a 768-dimensional text embedding vector. A visual transformer model is used to jointly encode the image and its OCR results, outputting a 512-dimensional image embedding vector. The topology map corresponding to the drawing is input into a graph neural network, using node features and adjacency matrix as input. Node-level embeddings are generated through a message passing mechanism, and a 512-dimensional global embedding vector of the drawing is obtained through global pooling. The time window features of the time series data are encoded using a time series transformer model, outputting a 128-dimensional time series embedding vector. In step 5, for newly accessed data, a 512-dimensional vector is generated and compared with the existing entity database. If the similarity is less than 0.75, a new entity record is created. If it is greater than 0.9, it is automatically merged and the version number is updated. If it is between 0.75 and 0.9, it is added to the manual review queue.

2. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 1, characterized in that, Also includes: In step 1, during the text semantic segmentation process, document paragraphs containing technical parameter tables or fault logs are given priority to retain complete field alignment, and are given priority to be broken at punctuation marks to ensure semantic integrity.

3. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In the drawing processing stage of step 1, the target detection model adopts an improved YOLOv8 architecture and designs a special anchor frame size. The skeleton extraction algorithm employs refined morphological operations combined with a direction tracking strategy to ensure the correct reconstruction of the connectivity of complex intersecting pipelines, with an error not exceeding the width of a single pixel.

4. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In step 2, the projection layer adopts a fully connected network structure, which includes a weight matrix W and a bias vector b. It is trained under supervision by an alignment loss function to minimize the Euclidean distance of the mapping vectors of the same entity in different modalities.

5. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In step 3, when calculating the Jaccard similarity coefficient, the topological neighbor set is expanded to the second-level neighborhood range, that is, it includes the neighbors of the neighbors.

6. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In step 3, the large model used for context consistency verification is fine-tuned by industrial manuals, maintenance reports, and process specification corpora to understand the physical meaning of professional terms and their manifestation under specific working conditions.

7. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In step 4, the HNSW index is constructed using a batch insertion strategy, which first performs clustering preprocessing on all vectors, and then groups them to build a hierarchical connection graph.

8. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 2, characterized in that, Also includes: In step 4, the structured conditions for hybrid retrieval support Boolean combination filtering based on multiple dimensions such as time range, equipment type, factory area number, and responsible person, and generate a comprehensive relevance score by using a weighted fusion strategy with vector similarity scores.

9. The semantic alignment method for industrial multimodal data based on vector space and topological constraints according to claim 1, characterized in that, Also includes: The evidence package in step 5 consists of associated text descriptions, historical alarm records, screenshots of real-time trend curves, partial views of relevant equipment drawings, and upstream and downstream operating condition data. All elements are converted into vector form to participate in context injection. In step 5, the rule base verification module has more than 100,000 built-in industrial logic rules, covering equipment operation boundary conditions, interlocking protection logic and timing causal constraints. Version number management uses an incrementing integer identifier, and a unique version snapshot is generated for each entity update.

Images