Industrial multi-modal data semantic alignment method based on vector space and topological constraint

Abstract

The invention belongs to the field of information processing, discloses an industrial multi-modal data semantic alignment method based on vector space and topological constraints, and aims to solve the problem that multi-modal data semantic segmentation in an industrial scene is difficult to unify and associate. The method comprises the following steps: performing feature extraction and structured analysis on modal data, and mapping the modal data to a unified semantic vector space through a projection layer; on the basis of vector similarity matching, topological structure constraints derived from process drawings and the like are introduced, neighbor relation verification is carried out on candidate entities, context logic verification is carried out in combination with a large model, and therefore the accuracy of cross-source entity alignment is remarkably improved; mixed retrieval is supported by establishing an efficient vector database index, and incremental updating of the knowledge graph is achieved through large-model-assisted reasoning and rule base verification. Vector semantics and specific structural dependency of an industrial system are effectively fused, and high-precision and evolvable cross-modal semantic alignment and intelligent association analysis are achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of information processing, and particularly relates to an industrial multi-modal data semantic alignment method based on a vector space and topological constraints. BACKGROUND

[0002] With the in-depth development of industrial intelligentization and digitalization transformation, the manufacturing, energy, chemical industry and equipment industries continue to generate massive multi-modal data, covering process documents, engineering drawings, field videos, device time series signals and system logs and other forms. These heterogeneous data contain rich device status, operation logic and fault clues, but there are significant differences in semantic expression, structural form and space-time scale, resulting in serious information fragmentation and difficulty in forming a unified knowledge view. Especially in complex industrial scenarios, the same physical entity such as a pump or valve may be presented in different naming, format or modality in different systems, making it difficult to effectively associate cross-source data and restricting the improvement of intelligent diagnosis, knowledge reuse and autonomous decision-making capabilities.

[0003] Among them, the multi-modal fusion method based on vector space has received widespread attention in recent years. Existing schemes usually use independent encoders to process text, images or time series data, and implement single-modal retrieval through vector databases. However, such methods lack the modeling capability of entity uniqueness and structural dependence in the industrial context. On the one hand, the embedding spaces of each modality are isolated from each other, which cannot support cross-modal semantic interaction such as searching images by text or searching drawings by time series; on the other hand, relying only on vector similarity for entity matching can easily misalign similar devices with different physical locations, such as misjudging Pump-A and 1# pressurizing pump as the same object, while ignoring their topological differences in the process flow. In addition, traditional knowledge graph construction relies on manual rules and regular templates, which is difficult to handle unstructured drawings and dynamic IoT events, and cannot evolve automatically with new data.

[0004] Therefore, there is an urgent need for a multi-modal alignment mechanism that can integrate vector semantics and industrial topological constraints to achieve accurate mapping of cross-modal data, accurate entity recognition and continuous evolution of knowledge in a unified semantic space, thereby supporting high-reliability industrial intelligent analysis and reasoning. SUMMARY

[0005] The purpose of the present application is to provide an industrial multi-modal data semantic alignment method based on vector space and topological constraints, which can effectively solve the problems in the background art.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: An industrial multi-modal data semantic alignment method based on vector space and topological constraints, comprising the following specific steps: Step 1: Pre-process multi-modal industrial data, structure and extract features from text, images, drawings and time series signals respectively; Step 2: Construct a unified semantic vector space, map different modal data to the same dimension embedding space to obtain embedding vectors, and perform linear transformation on the embedding vectors through a learnable projection layer to unify the final output as a unified multi-dimensional vector; Step 3: Perform topological constraint-based industrial entity semantic alignment, calculate the cosine similarity between the entity pairs to be matched in the unified vector space, when the similarity is greater than the preset similarity, include it in the candidate entity set, obtain the topological neighbor set of the two candidate entities in the process system, construct the multi-modal context description of the candidate entities into a reasoning prompt template, input the large model for logical judgment, verify whether the function, parameter and running environment are consistent, and confirm the corresponding knowledge entries after merging; Step 4: Establish a vector database semantic index and support efficient hybrid retrieval, store all the unified multi-dimensional vectors in the vector database, use a hierarchical navigable small world HNSW index structure to organize the data, set index parameters and search parameters, specify vector similarity threshold and structured filtering conditions, and return the top-K most relevant results; Step 5: Trigger large model assisted reasoning and knowledge graph incremental update mechanism, integrate the retrieval hit results to form an evidence package, input the large model for causal chain analysis, and perform logical reasonableness verification through a preset rule base.

[0007] Further, the method further comprises: For text documents, use row and column label inference algorithms to identify table structures, and divide them into data blocks according to semantic paragraphs; For image data, use an optical character recognition model to extract text information, and generate a multi-dimensional visual feature vector through a deep convolutional neural network; For drawings, perform graph element detection, use a target detection model to identify pump and valve process equipment nodes, track the pipeline connection relationship using a skeleton extraction algorithm, construct a topological graph structure containing nodes and edges, and retain the spatial coordinate encoding of each node; For time series data collected by equipment sensors, set the sliding window length and step, input the data in each time window into a time series feature encoder, and extract its dynamic evolution pattern; The step 2 adopts a large language model to encode the text data block, outputs a 768-dimensional text embedding vector, adopts a visual transformer model to jointly encode the image and its OCR result, outputs a 512-dimensional image embedding vector, inputs the topological graph corresponding to the drawing into a graph neural network, takes the node features and the adjacency matrix as inputs, generates node-level embedding through a message passing mechanism, and then obtains a 512-dimensional global embedding vector of the drawing through a global pooling operation; and encodes the time window features of the time series data by using a time series transformer model, and outputs a 128-dimensional time series embedding vector. In the step 5, for the newly accessed data, a 512-dimensional vector is generated and compared with the existing entity library, if the similarity is lower than 0.75, a new entity record is created, if the similarity is higher than 0.9, the new data is automatically merged and the version number is updated, and if the similarity is between 0.75 and 0.9, the new data is added to the artificial review queue.

[0008] Further, the method further comprises: In the text semantic segmentation process of the step 1, the document paragraphs containing technical parameter tables or fault logs are preferentially retained in the complete field alignment, and preferentially disconnected at the punctuation marks to ensure the semantic integrity.

[0009] Further, the method further comprises: In the drawing processing stage of the step 1, the target detection model adopts an improved YOLOv8 architecture and designs a special anchor box size. The skeleton extraction algorithm adopts a thinning morphological operation combined with a direction tracking strategy to ensure the correct restoration of the connectivity of complex cross-pipeline, and the error is not more than a single-pixel width.

[0010] Further, the method further comprises: In the step 2, the projection layer adopts a fully connected network structure, includes a weight matrix W and a bias vector b, and is supervised and trained through an alignment loss function to minimize the Euclidean distance of the mapping vectors of the same entity under different modalities.

[0011] Further, the method further comprises: In the step 3, when calculating the Jaccard similarity coefficient, the topological neighbor set is expanded to a two-level neighborhood range, i.e. the neighbors of the neighbors.

[0012] Further, the method further comprises: In the step 3, the large model used for context consistency verification is fine-tuned by industrial manuals, maintenance reports and process procedure corpus to understand the physical meaning of professional terms and their forms under specific working conditions.

[0013] Further, the method further comprises: In the step 4, the HNSW index adopts a batch insertion strategy during construction, first performs clustering preprocessing on all vectors, and then establishes a hierarchical connection graph in groups.

[0014] Further, the method further comprises: The structured condition support time range, device type, plant number, and multiple dimensions of the person responsible for the screening in step 4 are combined by a Boolean combination, and a weighted fusion strategy is used to generate a comprehensive correlation score with the vector similarity score.

[0015] Further, the method further comprises: The composition of the evidence package in step 5 includes associated text description, historical alarm record, real-time trend curve screenshot, relevant device drawing partial view, and upstream and downstream working condition data, all of which are converted into vector form to participate in context injection; The rule library verification module in step 5 has more than 100,000 industrial logic rules built-in, covering device operation boundary conditions, interlock protection logic and time sequence causal constraints; version number management uses an incremental integer identifier, and a unique version snapshot is generated each time the entity is updated.

[0016] Compared with the prior art, the present application has the following beneficial effects: The present application realizes the unified coding and expression of various industrial data modalities such as text, image, drawing and time sequence signal by constructing a unified vector semantic space, fundamentally solving the semantic fragmentation problem between heterogeneous data; the topological structure constraint based on the connection relationship between drawings and processes is introduced, and the structural dependency verification specific to the industrial system is superimposed on the basis of vector similarity, which significantly improves the entity alignment accuracy of similar devices in different systems with inconsistent names, effectively avoiding the mis-matching problems of same name different things and different name same things; the dual reasoning mechanism of large model combined with rule base can automatically mine potential causal relationships in massive associated information, generate diagnosis suggestions conforming to engineering logic, and enhance the explainability and credibility of the intelligent system; a three-level incremental update strategy based on similarity interval is designed, which supports the continuous evolution of the knowledge graph with new data, greatly reduces the manual maintenance workload, and can reduce the knowledge management cost by more than 80%; the overall method supports millisecond-level cross-modal hybrid retrieval, combined with the efficient HNSW index structure, the efficiency is improved by 5 to 20 times compared with the traditional layer-by-layer filtering method, meeting the real-time analysis requirements of industrial sites; the system has good scalability, the core model, database and alignment algorithm can be replaced by other equivalent components, and is suitable for various deployment environments, and has been verified in large petrochemical, power and other complex industrial scenes. The effectiveness and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The workflow diagram of an industrial multi-modal data semantic alignment method based on vector space and topological constraint claimed by the embodiments of the present application. DETAILED DESCRIPTION

[0018] Currently, massive multi-modal data is continuously generated in manufacturing, energy, chemical industry and equipment industries, covering process documents, engineering drawings, field videos, equipment time series signals and system logs and other forms. These heterogeneous data contain rich device status, operation logic and fault clues, but there are significant differences in semantic expression, structural form and space-time scale, resulting in serious information fragmentation and difficulty in forming a unified knowledge view. Especially in complex industrial scenarios, the same physical entity may be presented in different naming, format or modal in different systems, making it difficult to effectively associate cross-source data, and restricting the improvement of intelligent diagnosis, knowledge reuse and autonomous decision-making capability. In view of the above technical problems, the present application proposes a multi-modal data semantic alignment method based on vector space and topological constraint, which can effectively solve the problems in the above background technology and be applied to the multi-modal data semantic alignment method based on vector space and topological constraint.

[0019] Reference is made to the accompanying drawings Figure 1 The overall technical scheme architecture diagram of the present application shows the complete process from multi-modal data input to knowledge graph incremental update.

[0020] A multi-modal data semantic alignment method based on vector space and topological constraint, comprising the following specific steps: Step 1: Preprocess multi-modal industrial data, and perform structural analysis and feature extraction on text, image, drawing and time series signal respectively; Step 2: Construct a unified semantic vector space, map different modal data to the same dimension embedding space, and obtain embedding vectors by linear transformation through a learnable projection layer, so that the final output is a unified multi-dimensional vector; Step 3: Perform topological constraint-based industrial entity semantic alignment, calculate the cosine similarity between the matched entity pairs in the unified vector space, when the similarity is greater than the preset similarity, include the candidate entity set, obtain the topological neighbor set of the two candidate entities in the process system, construct the multi-modal context description of the candidate entities into a reasoning prompt template, input the large model for logical judgment, verify whether the function, parameter and running environment are consistent, and confirm the corresponding knowledge entry after merging; Step 4: Establish a vector database semantic index and support efficient hybrid retrieval, store all the unified multi-dimensional vectors into the vector database, use a hierarchical navigable small world HNSW index structure to organize the data, set the index parameters and search parameters, specify the vector similarity threshold and the structured filtering conditions, and return the top-K most relevant results; Step 5: Trigger the large model assisted reasoning and knowledge graph incremental update mechanism, integrate the retrieval hit results to form an evidence package, input the large model for causal chain analysis, and perform logical rationality verification through a preset rule base.

[0021] Further, the method further comprises: For text documents, a row-column label inference algorithm is used to identify table structures, and semantic paragraphs are segmented into data blocks; For image data, an optical character recognition model is used to extract text information, and a deep convolutional neural network is used to generate a multi-dimensional visual feature vector; For drawings, a graph element detection is performed, a target detection model is used to identify pumps, valves, and process equipment nodes, a skeleton extraction algorithm is used to track pipeline connections, a topological graph structure containing nodes and edges is constructed, and the spatial coordinates of each node are preserved; For time series data collected by device sensors, a sliding window length and step size are set, and the data in each time window is input into a time series feature encoder to extract its dynamic evolution pattern; In step 2, a large language model is used to encode the text data block, outputting a 768-dimensional text embedding vector, a visual transformer model is used to jointly encode the image and its OCR results, outputting a 512-dimensional image embedding vector, the topological graph corresponding to the drawing is input into a graph neural network, the node features and adjacency matrix are used as input, and the node-level embedding is generated through a message passing mechanism., and a global pooling operation is performed to obtain a 512-dimensional global embedding vector of the drawing, and a time series transformer model is used to encode the time window features of the time series data, outputting a 128-dimensional time series embedding vector; In step 5, for newly accessed data, a 512-dimensional vector is generated and compared with the existing entity library, if the similarity is less than 0.75, a new entity record is created, if the similarity is greater than 0.9, the new entity is automatically merged and the version number is updated, and if the similarity is between 0.75 and 0.9, the new entity is added to the manual review queue.

[0022] Further, the method further comprises: In the text semantic segmentation process of step 1, the document paragraphs containing technical parameter tables or fault logs are preferentially kept complete field alignment, and preferentially disconnected at punctuation marks to ensure semantic integrity.

[0023] Further, the method further comprises: In the drawing processing stage of step 1, the target detection model uses an improved YOLOv8 architecture and designs a special anchor box size; The skeleton extraction algorithm uses a thinning morphological operation combined with a direction tracking strategy to ensure the correct restoration of the connectivity of complex cross-pipeline, with an error of not more than a single-pixel width.

[0024] Further, the method further comprises: In step 2, the projection layer uses a fully connected network structure, including a weight matrix W and a bias vector b, and is supervised by an alignment loss function to minimize the Euclidean distance between the mapping vectors of the same entity under different modalities.

[0025] Further, the method further comprises: In the step 3, the Jaccard similarity coefficient calculation is extended to the second neighborhood range, i.e. the neighbors of the neighbors.

[0026] Further, the method further comprises: In the step 3, the large model used for context consistency verification is fine-tuned by industrial manuals, maintenance reports and process procedure corpus to understand the physical meaning of professional terms and their manifestations under specific working conditions.

[0027] Further, the method further comprises: In the step 4, the HNSW index adopts a batch insertion strategy during construction, first performs clustering preprocessing on all vectors, and then establishes hierarchical connection graphs in groups.

[0028] Further, the method further comprises: In the step 4, the structured conditions of the mixed retrieval support Boolean combination filtering of multiple dimensions such as time range, device type, plant number, and responsible person, and use a weighted fusion strategy to generate a comprehensive relevance score with the vector similarity score.

[0029] Further, the method further comprises: In the step 5, the composition of the evidence package includes associated text description, historical alarm records, real-time trend curve screenshots, relevant device drawing partial views, and upstream and downstream working condition data, all of which are converted into vector form for context injection. In the step 5, the rule library verification module has more than 100,000 industrial logic rules built-in, covering device running boundary conditions, interlocking protection logic and time sequence causal constraints; version number management uses incremental integer identification, and each entity update generates a unique version snapshot.

[0030] In the above method of semantic alignment of industrial multi-modal data based on vector space and topological constraints, the step 1, preprocessing multi-modal industrial data, structurally analyzes and extracts features from text, images, drawings and time series signals. Specifically, the step 1 contains four parallel and independent data processing sub-processes, which respectively analyze different modal data sources in depth to lay the foundation for subsequent unified semantic coding.

[0031] For text documents, the raw content is first obtained through optical scanning or direct reading of electronic files, and then the potential table structure is identified using a row-column label inference algorithm. This algorithm constructs a two-dimensional grid model by jointly analyzing character positions, font styles, and separator patterns, mapping unstructured text streams to structured forms with clear row-column relationships. On this basis, the system divides the document into data blocks with a length of 150 to 300 word units according to semantic paragraphs. To ensure the integrity of key information, when a paragraph is detected to contain a technical parameter table or a fault log, the system prioritizes the preservation of complete field alignment to avoid the separation of key fields such as the pressure upper limit and 1.2 MPa due to cross-line cutting. In addition, a sentence boundary detection mechanism is introduced to preferentially disconnect at punctuation marks such as periods and semicolons, ensuring that each data block has coherent semantics and logically consistent content. Each generated data block is assigned unique document source identification, page offset, and timestamp metadata to facilitate subsequent tracing and context reconstruction.

[0032] For image data, the system first extracts the embedded text information in the image using a high-precision optical character recognition model. This OCR model is trained on a large number of industrial field images and can effectively handle complex situations such as low light, blur, tilt, and partial occlusion. The extracted text information is input together with the original image pixel data into a deep convolutional neural network. The network uses ResNet-152 as the backbone architecture, which is pre-trained on the ImageNet dataset and then fine-tuned on a private dataset containing millions of industrial equipment images. The final output layer of the network is replaced with a global average pooling layer, generating a 2048-dimensional visual feature vector. This vector not only encodes the overall appearance information of the image but also integrates the semantic content of the readable text, forming a primary representation of the image-text combination.

[0033] For the processing of drawings, the system performs special graph element detection and connection relationship tracking. First, an improved object detection model is used to identify standard process equipment nodes such as pumps, valves, instruments, and heat exchangers in the drawings. This model is based on the YOLOv8 architecture and is customized to address the size distribution characteristics of industrial graph elements. The anchor box size set is redesigned to better match the actual pixel ratio of small symbols such as stop valves and safety valves, thereby improving the mAP index of small target detection by 17 percentage points. After completing node positioning, the system starts the skeleton extraction algorithm, which uses thinning morphological operations combined with direction tracking strategies. The thinning operation gradually erodes the pipeline lines to a single-pixel width, and the direction tracking records the connection direction along the skeleton path, ensuring that the connectivity can be correctly restored even in densely intersected areas, with a geometric error strictly controlled within a single-pixel width. Finally, all identified nodes and their spatial coordinates are encoded, along with the edge connection relationships derived from the skeleton algorithm, to form a topological graph structure in the form of a directed acyclic graph. Each node object contains its type label, center coordinates, bounding box size, and associated text annotations, etc.

[0034] For time series data collected by device sensors, the system sets a dynamic sliding window strategy. In the default working condition, the window length is 60 seconds and the step is 10 seconds, dividing the continuous time series into a series of overlapping time windows. The multi-channel sensor data in each time window, such as temperature, pressure, flow, vibration, etc., is organized into a two-dimensional matrix with dimensions [channel number, time point number], and input into a dedicated time series feature encoder. This encoder is based on LSTM or Transformer architecture and can capture dynamic evolution patterns, periodic patterns, and sudden events within the data. It is worth noting that the sliding window strategy is not fixed. The system has a built-in device state monitoring module that analyzes the variance, mean, and first derivative of the time series data in real time. When the device is in a steady state, such as when the variance is below a pre-set threshold and there is no significant trend, the window length is automatically extended to 120 seconds to capture longer-term performance drift trends; conversely, when there are step changes, alarm events, or derivative absolute values exceeding the threshold, the window length is shortened to 5 seconds to improve the response sensitivity to transient anomalies. Each time window is finally encoded into a high-dimensional feature vector representing the comprehensive operating state of the device during that period.

[0035] In the above method of semantic alignment of industrial multi-modal data based on vector space and topological constraints, step 2, a unified semantic vector space is constructed to map different modal data to the same dimension embedding space. Specifically, the core of step 2 is to establish a 512-dimensional vector representation space that is consistent across modalities, so that data from different sources can be compared and operated in the same mathematical framework.

[0036] For the text data block generated in step 1, the system encodes it using a large language model. This large language model has a parameter size of tens of billions and has been fully pre-trained on general corpus and industrial technical documents. After inputting the text data block, the model outputs its corresponding 768-dimensional text embedding vector, which deeply encodes the semantic connotation, technical terms, and contextual logic of the text.

[0037] For image data, the system uses a visual transformer model to jointly encode the original image and the OCR extracted text results. The visual transformer first divides the image into fixed-size image blocks, and each block is linearly embedded and added to the position encoding. At the same time, the OCR text is also converted through the word embedding layer after tokenization. The two embeddings start to interact and fuse in the early layers of the model, and after multiple layers of self-attention and cross-attention mechanisms, a 512-dimensional image embedding vector is finally output. This vector realizes the deep fusion of visual content and text semantics.

[0038] For the topology graph structure corresponding to the drawing, the system inputs it into a graph neural network. This graph neural network uses the graph attention network (GAT) structure, which has the core advantage of allowing different neighbor nodes to contribute different weights according to their importance. During message passing, the information of high-influence nodes such as key control valves and measuring instruments will be given higher attention weights, so as to be strengthened in the aggregation stage. After multiple rounds of message passing, each node obtains an embedding representation containing its local topological context. Then, through global pooling operations such as summation or average pooling, all node embeddings are fused to generate a 512-dimensional global embedding vector of the drawing, which fully describes the structure and layout of the entire process system.

[0039] For time series data, the system uses a time series transformer model to encode its time window features. This model uses self-attention mechanisms to capture long-range dependencies between time points, and can effectively model complex dynamic behavior. The encoder finally outputs a 128-dimensional time series embedding vector.

[0040] To bridge the gap between the original embedding dimensions of different modalities, all the aforementioned embedding vectors, 768-dimensional text vector, 512-dimensional image vector, 512-dimensional drawing vector, and 128-dimensional time series vector, are fed into a learnable projection layer. The projection layer adopts a fully connected network structure, containing a weight matrix W with an input dimension adaptation and a bias vector b. The training of the projection layer is supervised by a specially designed alignment loss function, which aims to minimize the Euclidean distance between the mapping vectors of the same physical entity in different modalities. For example, the text paragraph describing Pump-101, the on-site photo containing the Pump-101 label, the Pump-101 node in the drawing, and the runtime data of Pump-101, although their original embeddings are different in dimension, their 512-dimensional output vectors should be close to each other in space after passing through the projection layer. Through this supervised learning, the system successfully constructs a 512-dimensional vector representation space that is consistent across modalities.

[0041] In the above-mentioned industrial multi-modal data semantic alignment method based on vector space and topological constraints, step 3, performing industrial entity semantic alignment based on topological constraints, realizes high-precision cross-source entity matching. Specifically, step 3 adopts a three-level verification mechanism to perform coarse alignment, fine alignment, and context consistency verification in sequence to ensure the accuracy of the matching results.

[0042] The first level is preliminary coarse alignment. The system calculates the cosine similarity between the embedding vectors of two entities to be matched, for example, a device mention from the text log and a device node from the drawing, in the unified 512-dimensional vector space. When the similarity is greater than 0.82, a preset threshold, they are included in the candidate entity set. This stage uses an efficient approximate nearest neighbor search algorithm to complete candidate set generation in a million-level vector library at sub-second speed, significantly reducing the computational load of the subsequent verification stage.

[0043] The second level is fine alignment, the core of which is to introduce industrial topological constraints. The system obtains the topological neighbor set of the two candidate entities in the process system. This set is strictly based on the drawing or system connection graph to determine the upstream and downstream directly connected device nodes of each entity. To further enhance robustness, the topological neighbor set not only includes directly adjacent device nodes, but also extends to the second neighborhood range, i.e., the neighbors of the neighbors, to capture more extensive process context dependency relationships. Subsequently, the system calculates the Jaccard similarity coefficient between the two extended neighbor sets. The Jaccard coefficient is defined as the ratio of the size of the intersection to the size of the union of the two sets. If the coefficient is greater than 0.6, it is determined that they have consistency in the industrial topological structure, strongly supporting that they are the same physical entity. This step effectively solves the problem of false matching caused by similar functions but different locations.

[0044] The third level is the context consistency check. Even if the previous two barriers are passed, there may still be cases of semantic approximation but different physical entities. To this end, the system constructs a structured reasoning prompt template from the multi-modal context description of the candidate entities, including their text description, associated images, partial views of drawings, historical temporal features, etc. This template is input into a large model that is fine-tuned in the field. The training corpus of this large model covers a large number of industrial manuals, maintenance reports and process procedures, enabling it to accurately understand the physical meaning of terms such as bearing wear and pressure surge, as well as their manifestations in specific working conditions. The task of the large model is to logically judge the functions, technical parameters, operating environment and historical events of the two candidate entities, and verify whether they are consistent. If the judgment result is contradictory or highly inconsistent, the matching pair is excluded; otherwise, the matching is confirmed successful and the knowledge item merging operation is triggered.

[0045] In the above industrial multi-modal data semantic alignment method based on vector space and topological constraints, step 4, the vector database semantic index is established and efficient hybrid retrieval is supported. Specifically, step 4 aims to provide efficient storage and query capabilities for massive 512-dimensional vectors.

[0046] The system stores all uniformly encoded 512-dimensional vectors in a high-performance vector database. This database uses a hierarchical navigable small-world HNSW index structure to organize data. When building the index, to cope with the scenario of storing billions of industrial data, the system uses a batch insertion strategy: first, perform K-means clustering preprocessing on all vectors to divide the data into several clusters, then establish hierarchical connection graphs in groups. This method effectively reduces the peak memory consumption during index construction, enabling large-scale index construction to be completed under limited hardware resources. The key parameters of the index are set to M=48, the maximum number of connections per node, and efSearch=128, the dynamic candidate set size during search, achieving the best balance between recall rate and query delay.

[0047] This index supports powerful combined query mode. Users can not only specify a query vector for semantic similarity search, but also can attach multiple structured filter conditions at the same time. These conditions support Boolean combination filtering AND / OR / NOT in multiple dimensions such as time range, device type, plant number, and responsible person. During retrieval, the system first uses the HNSW index to quickly find the candidate set that meets the vector similarity requirements, and then applies the structured filter on the candidate set. The final relevance ranking uses a weighted fusion strategy to consider both the vector similarity score and the matching degree of the structured condition, generating a comprehensive relevance score and returning the Top-K most relevant results. For example, users can retrieve all data segments that are semantically similar to temperature anomalies expressed by the query vector and have timestamps on the current day and device type centrifugal pump.

[0048] In the aforementioned semantic alignment method for industrial multimodal data based on vector space and topological constraints, step 5 triggers a large-model-assisted reasoning and knowledge graph incremental update mechanism. Specifically, step 5 completes a closed loop from data retrieval to intelligent decision-making and then to knowledge evolution.

[0049] Once a search is complete, the system integrates all matching results to form a comprehensive evidence package. This evidence package is extremely rich in content, including but not limited to related text descriptions, historical alarm records, real-time trend curve screenshots, partial views of relevant equipment drawings, and upstream and downstream operating data. All these heterogeneous elements have been converted into vector form before being input into the large model and participate in inference through context injection, greatly improving the information completeness and accuracy of the large model's reasoning. Based on this evidence package, the large model performs causal chain analysis, attempting to infer potential root causes of failures or provide optimized operational suggestions.

[0050] However, to prevent the large model from generating illusory conclusions that contradict common engineering sense, the system introduces a pre-defined rule base for logical rationality verification. This rule base verification module contains over 100,000 industrial logic rules, covering equipment operating boundary conditions such as the motor bearing temperature not exceeding 95°C, interlocking protection logic such as the standby pump being prohibited from starting if the main pump is not running, and temporal causal constraints such as the cooling water flow rate not returning to zero before the motor starts. Any reasoning output that violates these fundamental engineering principles will be directly rejected by the rule base, ensuring the reliability and safety of the final output.

[0051] Meanwhile, the system continuously processes newly received data streams. For each new data unit, the system generates a corresponding 512-dimensional vector representation and compares it with all entity vectors in the existing entity database. Based on the similarity of the comparison results, the system executes a three-level incremental update strategy: A. If the similarity between the old and new vectors is less than 0.75, it is considered a completely new and previously unseen entity, and the system will create a new independent entity record for it. B. If the similarity is higher than 0.9, it is highly certain that the new data is the same object as an existing entity. The system will automatically merge the new data into the existing entity and update its entity version number. Version number management uses an incrementing integer identifier. Each update generates a unique version snapshot, supporting historical state retrospection and change difference comparison, which facilitates audit tracking and knowledge evolution path analysis. C. If the similarity is between 0.75 and 0.9, the system cannot make a high-confidence automatic judgment. Therefore, the matching task is added to the manual review queue, awaiting final confirmation from domain experts. This human-machine collaborative mechanism ensures the accuracy and controllability of the knowledge system during its automated evolution.

[0052] As a specific application example, the method of this invention has been deployed on an intelligent operation and maintenance platform for a large-scale petrochemical plant. This platform needs to handle an additional 10TB of multi-source data streams daily, including DCS system logs, equipment inspection reports, high-definition monitoring videos, updated drawings, and real-time data from tens of thousands of sensors. On this platform, the response time for a single cross-modal retrieval, such as searching for relevant time-series segments, equipment drawings, and historical maintenance records based on a text describing outlet pressure fluctuations, is less than 200 milliseconds. After six months of continuous operation testing, the system achieved an accuracy of 96.8% in entity merging tasks, a 39 percentage point improvement over the 57.8% accuracy of traditional pure vector matching methods. Furthermore, this method significantly reduces manual maintenance workload through automated alignment and updates, and in practice, it can reduce knowledge management costs by more than 80%.

[0053] Based on the aforementioned first embodiment, the present invention can also be applied to intelligent substation inspection scenarios in the power industry. In this scenario, the multimodal data sources include: infrared thermal images and visible light images of the equipment captured by the inspection robot, voltage and current time-series data collected by the SCADA system, OCR recognition results of the equipment nameplates, and a structured text database of the equipment ledger.

[0054] In the preprocessing stage of step 1, the infrared image is processed separately, generating a thermal feature map using a temperature matrix extraction algorithm, and then fused with the visible light image. The drawing is replaced with an electrical primary wiring diagram in this scenario, and its element detection model is retrained to identify electrical equipment such as circuit breakers, disconnect switches, and transformers. The sliding window strategy for the time-series data is adjusted according to the grid load cycle; for example, a shorter window is used during peak electricity consumption periods to capture instantaneous overloads.

[0055] In the topology constraint alignment in step 3, the topology relationships provided by the electrical primary wiring diagram, such as bus connections and switch open / closed states, become the key verification criteria. For example, two circuit breaker entities from thermal imaging and SCADA data, respectively, even if their vector similarity is high, will have a very low Jaccard coefficient for their topological neighbor set if one is located on a 110kV bus and the other on a 10kV bus, thus correctly identifying them as different entities.

[0056] In the rule base of step 5, a large number of safety regulations specific to the power industry are integrated, such as prohibiting the opening of high-voltage room doors if the grounding switch is not closed. The domain fine-tuning corpus of the large model is also replaced with power safety regulations, equipment maintenance procedures, and accident analysis reports. In this way, the method of the present invention also achieves high-precision cross-modal semantic alignment and intelligent reasoning in the power scenario, verifying the universality and transferability of its technical solution.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

[0058] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0059] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0060] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for industrial multi-modal data semantic alignment based on vector space and topological constraints, characterized in that, Comprise the following specific steps: Step 1: Pre-process multi-modal industrial data, structure analysis and feature extraction for text, image, drawing and time series signal respectively; Step 2: Construct a unified semantic vector space, map different modal data to the same dimension embedding space, and get embedding vectors through linear transformation of the learnable projection layer, so that the final output is unified as a unified multi-dimensional vector; Step 3: Perform topological constraint-based industrial entity semantic alignment, calculate the cosine similarity between the matched entity pairs in the unified vector space, when the similarity is greater than the preset similarity, it is included in the candidate entity set, get the topological neighbor set of the two candidate entities in the process system, construct the multi-modal context description of the candidate entity into a reasoning prompt template, input the large model for logical judgment, verify whether the function, parameter and running environment are consistent, and confirm the corresponding knowledge entry after merging; Step 4: Establish vector database semantic index and support efficient hybrid retrieval, store all the unified multi-dimensional vectors into the vector database, use hierarchical navigable small world HNSW index structure to organize data, set index parameters and search parameters, specify vector similarity threshold and structured filtering conditions, and return the top-K most relevant results; Step 5: Trigger large model assisted reasoning and knowledge graph incremental update mechanism, integrate the retrieval hit results to form an evidence package, input the large model for causal chain analysis, and perform logical rationality verification through the preset rule base.

2. The method of claim 1, wherein, Also includes: For text documents, use row and column label inference algorithm to identify table structure, and divide into data blocks according to semantic paragraphs; For image data, use optical character recognition model to extract text information, and generate multi-dimensional visual feature vector through deep convolutional neural network; For drawings, perform graph element detection, use target detection model to identify pump, valve process equipment nodes, track pipeline connection relationship combined with skeleton extraction algorithm, construct topological graph structure containing nodes and edges, and retain the spatial coordinate code of each node; For time series data collected by device sensors, set the sliding window length and step, input the data in each time window into the time series feature encoder to extract its dynamic evolution mode; In step 2, use large language model to encode text data blocks, output 768-dimensional text embedding vectors, use visual transformer model to jointly encode images and their OCR results, output 512-dimensional image embedding vectors, input the topological graph corresponding to the drawing into the graph neural network, use node features and adjacency matrix as input, generate node-level embedding through message passing mechanism, and obtain 512-dimensional global embedding vector of the drawing through global pooling operation, use time series transformer model to encode the time window features of time series data, and output 128-dimensional time series embedding vectors; In step 5, for newly accessed data, generate 512-dimensional vectors and compare them with existing entity library, if the similarity is less than 0.75, create a new entity record, if it is higher than 0.9, automatically merge and update the version number, if it is between 0.75 and 0.9, add it to the artificial audit queue.

3. The method of claim 2, wherein, Also includes: In the step 1 Chinese text semantic segmentation process, the document paragraph containing the technical parameter table or fault log is preferentially retained with complete field alignment, and preferentially disconnected at the punctuation mark to ensure semantic integrity.

4. The method of claim 2, wherein, Also includes: In the step 1 drawing processing stage, the target detection model adopts an improved YOLOv8 architecture and designs a special anchor box size. The skeleton extraction algorithm adopts a refinement morphological operation combined with a direction tracking strategy to ensure the correct restoration of the connectivity of complex cross-pipeline with an error of not more than a single pixel width.

5. The method of claim 2, wherein, Also includes: The projection layer in step 2 adopts a fully connected network structure, including a weight matrix W and a bias vector b, and is supervised by an alignment loss function for training, minimizing the Euclidean distance of the mapping vectors of the same entity under different modalities.

6. The method of claim 2, wherein, Also includes: In step 3, when calculating the Jaccard similarity coefficient, the topological neighbor set is expanded to a two-level neighborhood range, i.e., including the neighbors of the neighbors.

7. The method of claim 2, wherein, Also includes: In step 3, the large model used for context consistency verification is fine-tuned on industrial manuals, maintenance reports, and process procedure corpus to understand the physical meaning of professional terms and their manifestations under specific working conditions.

8. The method of claim 2, wherein, Also includes: In step 4, the HNSW index adopts a batch insertion strategy during construction, first clustering all vectors for preprocessing, and then grouping to establish a hierarchical connection graph.

9. The method of claim 2, wherein, Also includes: In step 4, the structured conditions of mixed retrieval support Boolean combination filtering in multiple dimensions such as time range, device type, plant number, and responsible person, and use a weighted fusion strategy to generate a comprehensive relevance score with vector similarity score.

10. The method of claim 1, wherein, Also includes: In step 5, the composition of the evidence package includes associated text description, historical alarm record, real-time trend curve screenshot, relevant device drawing local view, and upstream and downstream working condition data, all elements are converted into vector form for context injection; The rule library verification module in step 5 has more than 100,000 industrial logic rules built-in, covering device running boundary conditions, interlocking protection logic, and time sequence causal constraints; Version number management uses an incremental integer identifier, and each entity update generates a unique version snapshot.

Images