Industrial cognitive base system based on multi-modal contrast learning and execution method

Abstract

The application provides an industrial cognitive base system based on multi-modal contrast learning and an execution method, wherein the industrial cognitive base system based on multi-modal contrast learning constructs a system architecture comprising a knowledge graph management module and a cross-modal encoding fusion module, dynamically injects industrial field knowledge in the form of a structured sub-graph into a multi-modal feature learning process, and combines a contrast learning mechanism with the introduction of industrial semantic constraints, so that the model obtained through final training can not only realize semantic alignment of multi-modal data, but also ensure that the unified semantic representation and reasoning results generated by the model strictly comply with pre-defined industrial logic rules, thereby effectively overcoming the defects that the simple data-driven method in the prior art may produce results contrary to industrial common sense, and significantly improving the output reliability, decision confidence and practical application value of the industrial cognitive system in key tasks such as fault diagnosis and state monitoring.

Description

Technical Field

[0001] This invention relates to the field of industrial artificial intelligence technology, and in particular to an industrial cognitive foundation system and execution method based on multimodal contrastive learning. Background Technology

[0002] With the continuous improvement of industrial intelligence, a large number of heterogeneous data acquisition devices have been deployed in industrial sites, generating multimodal industrial data, including visual images, text records, time-series data, and structured knowledge. In existing technologies, industrial artificial intelligence systems mainly adopt single-modal or simple multimodal fusion methods, extracting features of each modality through deep learning models and then jointly training them.

[0003] However, these methods have significant limitations: when processing multimodal industrial data, various deep learning models often focus on data-driven feature extraction and pattern recognition, lacking an effective mechanism to systematically integrate prior industrial knowledge such as domain expert knowledge, equipment operating principles, and process specifications into the model learning process. This results in the model's inference results in real-world industrial scenarios potentially violating basic industrial logic and physical laws, making it difficult to meet the stringent safety and accuracy requirements of industrial applications in terms of reliability and credibility. This deficiency severely restricts the in-depth application and promotion of artificial intelligence technology in key industrial fields. Summary of the Invention

[0004] In view of this, the present invention provides an industrial cognitive foundation system based on multimodal contrastive learning. One or more embodiments of this specification also relate to a method for an industrial cognitive foundation based on multimodal contrastive learning, in order to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the present invention, an industrial cognitive foundation system based on multimodal contrastive learning is provided, comprising:

[0006] The data acquisition and preprocessing module is used to synchronously acquire text modality, visual modality, temporal modality and structured knowledge modality data from multi-source systems in the industrial field, and generate standardized sample triples after alignment. The sample triples include an image, a text description and a set of candidate identifiers pointing to the core entities in the preset target knowledge graph. The set of candidate identifiers is obtained from the aligned text description through named entity recognition and preliminary entity linking.

[0007] The knowledge graph management module is used to store and maintain the target knowledge graph, and to perform semantic parsing and entity linking based on the text description and identifier set in the sample triples. It extracts and constructs a semantically rich sub-graph from the stored target knowledge graph that is related to the current context.

[0008] The cross-modal coding and fusion module is used to extract visual features, text features and knowledge features from the associated sub-graph and fuse them to generate knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space along with the visual features.

[0009] The contrastive learning training module is used to calculate the contrastive loss and industrial semantic constraint loss using vector representation, and optimize the model parameters based on the composite loss function to obtain the optimized industrial cognitive base model.

[0010] The online continuous learning module is used to monitor the new sample data stream and detect semantic drift. When the semantic drift exceeds the threshold, it triggers incremental model updates. The model service module is used to encapsulate the optimized industrial cognitive base model and provide interfaces for embedding computation, similarity query, and model update.

[0011] In some implementations, the data acquisition and preprocessing module is specifically used to: align visual frames, time-series segments, and corresponding text descriptions based on timestamps and device identifiers; and link industrial terms in the text to knowledge graph nodes through named entity recognition and entity linking technologies.

[0012] In some implementations, the target knowledge graph stored in the knowledge graph management module includes equipment knowledge graphs, process parameter libraries, and historical fault case libraries, and supports SPARQL queries and subgraph extraction.

[0013] In some implementations, the cross-modal coding and fusion module includes a visual encoder, a text encoder, and a knowledge injection module; the visual encoder uses a Vision Transformer network to extract image features; the text encoder uses a Transformer encoder to extract text features; the knowledge injection module uses a graph attention network to encode the sub-graph to obtain knowledge features, and fuses the knowledge features with the text features to generate knowledge-enhanced text features.

[0014] In some implementations, the composite loss function used by the contrastive learning training module includes standard contrastive loss and industrial semantic constraint loss, which penalizes vector representations for violating predefined industrial logic rules based on a rule consistency discriminator.

[0015] In some implementations, the industrial semantic constraint loss is calculated based on a rule violation score, and the formula for calculating the rule violation score includes:

[0016]

[0017] in, It is the rule violation score, a non-negative scalar value used to quantify the degree to which the i-th sample violates the industry rules; the larger the value, the more serious the violation. It is a visual feature vector, which comes from the visual encoder output of the cross-modal coding and fusion module, and represents the semantic embedding of the image in the i-th sample; is the knowledge-enhanced text feature vector, which comes from the knowledge injection module output of the cross-modal coding and fusion module, representing the semantic embedding of the text in the i-th sample after knowledge enhancement; K is the total number of predefined industrial logic rules, which comes from the rule base maintained in the knowledge graph management module; is the violation penalty weight of the k-th rule, a non-negative scalar parameter used to adjust the importance of the rule in the total violation score; This is the smoothed violation strength function for the k-th rule, used to calculate the violation strength of the k-th rule at a given distance value. Its output value smoothly increases from 0. The calculation formulas include:

[0018]

[0019] in, is the tolerance threshold for the k-th rule, a non-negative scalar representing the maximum value of the squared distance between the visual and text projection features allowed by the rule. Exceeding this value indicates that the degree of violation begins to increase significantly. It is the sensitivity coefficient of the k-th rule, a positive scalar that controls the rate at which the violation intensity increases as the distance exceeds the threshold;

[0020] It is a distance function, defined as the square of the Euclidean distance, i.e. , used to calculate the difference between two vectors; It is the visual projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project visual features onto the semantic subspace of interest for the k-th rule; It is the text projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project knowledge-enhanced text features onto the same rule-based semantic subspace. This represents the dimension of the original feature vector. This represents the dimension of the projected feature vector.

[0021] In some implementations, the projection matrix is ​​initialized. and The steps include:

[0022] The embedding representations of the entities and relations involved in the k-th rule are extracted from the knowledge graph management module, and then mapped to D through a linear transformation layer. r 3D space, to initialize the projection matrices respectively. and ;

[0023] Optimize the projection matrix and The steps include:

[0024] During the training process of the contrastive learning training module, the projection matrix is... and As a learnable parameter, a regularization term is added to the composite loss function. The calculation formula for the regularization term includes:

[0025]

[0026] in, It is the projection matrix regularization loss term corresponding to the k-th industrial logic rule. and The matrix obtained from the initialization step, The Frobenius norm of the matrix is ​​used to constrain the deviation between the optimized projection matrix and the initialization matrix.

[0027] In some implementations, the penalty weight Calculated through an adaptive weight adjustment mechanism, its formula includes:

[0028]

[0029] in, It is the global penalty scaling factor, which is a positive scalar hyperparameter preset by the trainer and used to adjust the strength of all rule penalty weights as a whole; It is a hybrid coefficient of static and dynamic weights. This parameter is an adjustable hyperparameter in the range [0,1], used to balance the ratio between the static preset importance of the rule and the dynamic violation intensity learned during model training. It is the static importance weight of the k-th rule. This parameter is a real number and comes from the rule importance score pre-set by domain experts in the knowledge graph management module. It is an S-shaped function that maps the input to the (0,1) interval, which is used to map the static importance of expert ratings to a stable numerical range; It is the hyperbolic tangent function, which maps the input to the interval (-1, 1); for non-negative inputs... Under the condition, the actual output range is [0,1), which is used to map the scaled dynamic violation intensity to a bounded range; It is a scaling factor for the dynamic violation intensity. This parameter is a positive scalar hyperparameter that is preset by the trainer to adjust the range of dynamic violation intensity values ​​and control the sensitivity of the dynamic part to the final weights. This represents the smooth dynamic violation intensity of the k-th rule during the current training phase. This parameter is a non-negative real number, calculated using momentum updates, and reflects the average violation intensity of the k-th rule in the current training batch, while avoiding batch fluctuations. The calculation formulas include;

[0030]

[0031] in, The smoothing coefficient has a value in the range [0.9,1) and is used to control the degree to which historical violation intensity information is retained in the smoothing calculation; Batch size; This represents the smooth dynamic violation strength value at the t-th training iteration.

[0032] In some implementations, the model service module is encapsulated as a microservice and provides services through multiple application programming interfaces (APIs), including:

[0033] The visual embedding interface is used to receive input images and return the corresponding visual embedding vectors;

[0034] The knowledge-enhanced text embedding interface is used to receive input text and return the corresponding knowledge-enhanced text embedding vector.

[0035] A similarity calculation interface for calculating and returning the semantic similarity between an image and text;

[0036] The model update interface is used to receive new samples and trigger the online continuous learning module to perform incremental updates.

[0037] According to a second aspect of the present invention, an execution method for an industrial multimodal cognitive base is provided, applied to the aforementioned industrial cognitive base system based on multimodal contrastive learning, the method comprising:

[0038] An industrial multimodal heterogeneous training dataset is constructed by synchronously collecting text modality, visual modality, temporal modality, and structured knowledge modality data from multiple sources in the industrial field, and aligning them to generate standardized sample triples. Each sample triple includes an image, a text description, and a set of candidate identifiers pointing to core entities in a pre-defined target knowledge graph. The set of candidate identifiers is obtained from the aligned text description through named entity recognition and preliminary entity linking.

[0039] An industrial-enhanced cross-modal encoder is constructed, including a visual encoder, a text encoder, and a knowledge injection module, which is used to extract visual features, text features, and knowledge features from the associated sub-graph and fuse them to generate knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space together with the visual features.

[0040] Design a composite contrastive loss function for industrial semantic constraints, including a standard contrastive loss and an industrial semantic constraint loss calculated based on rule violation scores, whereby the rule violation scores are used to quantify the degree to which the model output violates industrial rules;

[0041] Perform end-to-end contrastive learning training, optimize model parameters using the vector representation and composite contrastive loss function, and output the optimized industrial cognitive foundation model;

[0042] Deploy an online continuous learning and model update mechanism to monitor the new sample data stream, detect semantic drift, and trigger incremental model updates when semantic drift exceeds a threshold.

[0043] At least one embodiment of this invention constructs a system architecture that includes a knowledge graph management module and a cross-modal coding fusion module. This dynamically injects industrial domain knowledge into the multimodal feature learning process in the form of structured subgraphs. Combined with a contrastive learning mechanism that introduces industrial semantic constraints, the model trained in this invention not only achieves semantic alignment of multimodal data but also ensures that the unified semantic representation and inference results it generates strictly conform to predefined industrial logic rules. This effectively overcomes the shortcomings of existing purely data-driven methods that may produce results that violate common industrial sense. It significantly improves the output reliability, decision credibility, and practical application value of industrial cognitive systems in key tasks such as fault diagnosis and condition monitoring. Attached Figure Description

[0044] Figure 1 This is a simplified structural diagram of an industrial cognitive foundation system based on multimodal contrastive learning provided by the present invention;

[0045] Figure 2 This is a flowchart of an industrial cognitive foundation method based on multimodal contrastive learning provided by the present invention. Detailed Implementation

[0046] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0047] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The modifications “a” and “a plurality” as used in this disclosure are illustrative and not restrictive, and those skilled in the art will understand that they should be understood as “one or more” unless the context clearly indicates otherwise.

[0048] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0049] See Figure 1 , Figure 1 This diagram illustrates a simplified structural schematic of an industrial cognitive foundation system based on multimodal contrastive learning, according to some embodiments of this specification, specifically including:

[0050] According to a first aspect of the present invention, an industrial cognitive foundation system based on multimodal contrastive learning is provided, comprising:

[0051] The data acquisition and preprocessing module is used to synchronously acquire text modality, visual modality, temporal modality and structured knowledge modality data from multi-source systems in the industrial field, and generate standardized sample triples after alignment. The sample triples include an image, a text description and a set of candidate identifiers pointing to the core entities in the preset target knowledge graph. The set of candidate identifiers is obtained from the aligned text description through named entity recognition and preliminary entity linking.

[0052] The knowledge graph management module is used to store and maintain the target knowledge graph, and to perform semantic parsing and entity linking based on the text description and identifier set in the sample triples. It extracts and constructs a semantically rich sub-graph from the stored target knowledge graph that is related to the current context.

[0053] The cross-modal coding and fusion module is used to extract visual features, text features and knowledge features from the associated sub-graph and fuse them to generate knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space along with the visual features.

[0054] The contrastive learning training module is used to calculate the contrastive loss and industrial semantic constraint loss using vector representation, and optimize the model parameters based on the composite loss function to obtain the optimized industrial cognitive base model.

[0055] The online continuous learning module is used to monitor the new sample data stream and detect semantic drift. When the semantic drift exceeds the threshold, it triggers incremental model updates. The model service module is used to encapsulate the optimized industrial cognitive base model and provide interfaces for embedding computation, similarity query, and model update.

[0056] The data acquisition and preprocessing module refers to the component responsible for collecting and initially organizing data from various sources in the industrial field. For example, this module, through an interface program deployed in the industrial network, synchronously acquires images, text logs, sensor time-series signals, and structured data from a database based on a unified clock source's timestamp and the device's unique identifier. This data is used to generate standardized and aligned multimodal data samples. Text modality refers to industrial data existing in natural language form. For example, this modality data originates from unstructured text recorded in equipment maintenance logs, operation manuals, or on-site worker notes, used to describe equipment status, process steps, or abnormal events. Visual modality refers to industrial data existing in image or video form. For example, this modality data originates from two-dimensional pixel arrays captured by monitoring cameras or industrial cameras deployed on the production line, used to record equipment appearance, instrument readings, and the production environment. Time-series modality refers to numerical sequence-type industrial data arranged in chronological order. For example, this modality data originates from one-dimensional signals generated by sensors such as vibration, temperature, and pressure sampling at fixed frequencies, used to reflect the dynamic operating parameters of the equipment. Structured knowledge modalities refer to industrial domain knowledge organized and stored using predefined models. For example, this modal data might originate from equipment knowledge graphs, relational databases, or expert systems. The data exists in the form of entities, attributes, and relationships, providing computer-readable representations of domain concepts, facts, and rules. Multi-source systems refer to multiple independent data generation and management systems in an industrial setting, such as manufacturing execution systems, monitoring and data acquisition systems, programmable logic controllers, and asset performance management systems, providing complementary and heterogeneous raw data streams. Synchronous acquisition refers to the process of acquiring data from different sources in a time-aligned manner, such as through publish-subscribe messaging middleware or embedding high-precision timestamps in data packets, ensuring consistency of different modal data in the temporal dimension and establishing instantaneous associations between cross-modal data. Sample triples refer to a standard data structure containing three elements, such as an industrial image, text describing the image, and a list of identifiers pointing to relevant entities in a knowledge graph, used as basic data units for model training. Text descriptions can refer to natural language elements in the sample triples that describe the content of an image, such as a brief textual description of the device's status, behavior, or scene in the image, used as input for a text encoder. The target identifier set can refer to a list of unique identifiers in the sample triples used to locate one or more nodes in a knowledge graph. For example, this set might contain Uniform Resource Identifiers (URIs) representing device models, components, and failure modes, used to accurately retrieve relevant semantic knowledge from the knowledge graph. Aligned information can refer to data used to establish correspondences between data elements of different modalities, such as uniform timestamps, spatial coordinates, or logical event identifiers, used to ensure that each element in the sample triples semantically points to the same industrial entity or event.A target knowledge graph can refer to a pre-built semantic network that covers knowledge in a specific industrial field. For example, this graph might use "equipment-component-fault-process" as its core schema, containing entities, attributes, and relationships, serving as an authoritative knowledge source for the system's semantic parsing and reasoning. Core entities can refer to nodes in the knowledge graph that represent key industrial concepts or objects, such as specific equipment instances, component types, or fault modes, serving as anchor points for knowledge retrieval and association.

[0057] A knowledge graph management module can refer to a software component responsible for storing, updating, and querying knowledge graphs in an industrial field. For example, this module stores entities and relationships through a graph database and supports the SPARQL query language to dynamically construct semantically rich sub-graphs based on a set of identifiers. Semantic parsing refers to the process of extracting structured semantic information from text descriptions and understanding their internal logical relationships. For example, performing syntactic analysis and deep understanding on the text "Pump A outlet pressure fluctuates, accompanied by abnormal sounds," extracting "Pump A" as the subject, "outlet pressure fluctuation" as the state, and "abnormal sounds" as the accompanying phenomenon, and organizing them into a machine-processable predicate logic form to provide a precise semantic framework for subsequent entity linking and knowledge graph retrieval. Entity linking refers to the process of associating entity references from semantic parsing results with corresponding unique nodes in the target knowledge graph. For example, the parsed entities "pump A" and "abnormal sound" are linked to specific nodes in the knowledge graph by calculating their semantic similarity with candidate entity nodes (such as the equipment entity "centrifugal pump-A-01" and the fault mode entity "cavitation noise") and using context disambiguation. This achieves accurate and unambiguous connection between unstructured text and structured knowledge graphs. A related subgraph refers to a subset of the target knowledge graph that is relevant to the current context. For example, it can be a local graph formed by extracting directly connected entities and relationships from a core entity through multi-hop traversal, providing condensed and relevant background knowledge for the current task.

[0058] A cross-modal coding and fusion module can refer to a software component that converts and merges data from different modalities into vectors. For example, this module might include a visual encoder, a text encoder, and a graph neural network, processing images, text, and sub-graphs respectively to generate vector representations in a unified semantic space. Visual features can refer to abstract numerical vector representations extracted from industrial images, such as feature tensors obtained by encoding image patch sequences using a Vision Transformer model, used to capture deep semantic information of the images. Text features can refer to abstract numerical vector representations extracted from text descriptions, such as feature vectors obtained by contextually encoding word sequences using a Transformer encoder, used to capture semantic information of the text. Knowledge features can refer to abstract numerical vector representations extracted from associated sub-graphs, such as graph embedding vectors obtained by aggregating information from nodes and edges in a graph using a graph attention network, used to encode structured knowledge into numerical form. Knowledge-enhanced text features can refer to text feature vectors that incorporate knowledge features, such as new feature vectors obtained by fusing knowledge features output from a graph attention network with text features output from a text encoder through gating mechanisms or cross-attention, used to enrich text representations with external domain knowledge. A unified semantic space can refer to a common vector space shared by feature vectors from all modalities. For example, a linear projection layer maps visual features and knowledge-enhanced text features to a Euclidean space of the same dimension, which is used for cross-modal similarity calculation and contrastive learning. Vector representation can refer to the mathematical representation of data points in the unified semantic space, such as a fixed-length array of floating-point numbers, used to encapsulate the semantic content of the input data in the form of dense vectors.

[0059] A contrastive learning training module can refer to a software component that optimizes model parameters by comparing positive and negative sample pairs. For example, this module might use the InfoNCE loss function to narrow the vector distance between positive sample pairs while widening the vector distance between negative sample pairs, thus training the model to achieve good feature representation capabilities. Industrial semantic constraint loss can refer to an additional term introduced into the loss function to penalize behaviors that violate industrial rules. For example, a predefined logic rule calculator can perform consistency checks on the vector representation of the model output and generate penalty scores to ensure that the model output conforms to common sense and logic in the industrial domain. A composite loss function can refer to an overall optimization objective composed of a weighted combination of multiple loss components, such as a linear weighted sum of standard contrastive loss and industrial semantic constraint loss, used to jointly guide the model in learning data similarity and domain rule consistency. Model parameters can refer to all adjustable weights and biases in the model, such as the convolutional kernel weights in a visual encoder, the attention parameters in a Transformer, and the weight matrix of the projection layer, used for iterative updates during training using the gradient descent algorithm. A trained model can refer to a machine learning model that has undergone optimization and has fixed parameters, such as a set of neural network parameters saved in a specific file format, used for prediction and feature extraction of new data during the inference phase. An industrial cognitive foundation model can refer to an artificial intelligence model trained through multimodal contrastive learning and possessing knowledge of the industrial domain, such as a set of neural network parameters encapsulating a visual encoder, a text encoder, and a knowledge injection module, used for semantic understanding and feature extraction of input multimodal data.

[0060] Online continuous learning modules refer to software components that enable a system to adapt to new data without forgetting old knowledge. For example, this module monitors changes in the distribution of input data and calculates drift metrics, initiating an incremental learning process when a threshold is exceeded to maintain model performance in changing environments. New sample data streams refer to the continuous flow of new data received during online system operation, such as sequences of untrained industrial images and text descriptions from real-time data interfaces, used to trigger incremental model updates. Semantic drift refers to changes in the statistical properties of model input data or its feature representations over time. For example, due to equipment aging or process improvements, the distribution of newly acquired images and text in the feature space differs significantly from the training set, serving as a key indicator for triggering model updates. Incremental model updates refer to the process of locally adjusting model parameters using only new data without retraining the entire model. For example, elastic weight consolidation algorithms add regularization penalties to important old parameters during optimization, allowing the model to adapt to new knowledge while retaining existing knowledge.

[0061] A model service module can refer to a software component that encapsulates model capabilities into callable services. For example, this module might provide a well-defined set of application programming interfaces (APIs) through a microservice architecture based on HTTP or gRPC, offering model inference functionality to external applications. Embedded computation can refer to a service that uses a model to convert input data into vector representations. For instance, by calling a visual encoder or text encoder, it can convert user-provided images or text into feature vectors in a unified semantic space for subsequent similarity matching or retrieval tasks. Similarity query can refer to a service that calculates and returns the degree of similarity between two data points. For example, it might calculate the cosine similarity or Euclidean distance between two vectors and return a scalar similarity score, used to evaluate the semantic association strength between different modalities of data such as images and text. A model update interface can refer to a programming interface that allows external systems to trigger model retraining. For example, it might be a RESTful API that receives new sample data and returns the training task status, used for integration into automated machine learning pipelines.

[0062] The present invention will be further described below through a detailed embodiment:

[0063] In a refinery pump status monitoring scenario, the data acquisition and preprocessing module obtains real-time vibration time-series data of pump A from the distributed control system via the OPC UA protocol, and simultaneously captures an image of the equipment's exterior from a high-definition camera deployed in the pump room via the plant's Ethernet. The Manufacturing Execution System provides a corresponding work order text description: "Pump A outlet pressure fluctuation, accompanied by abnormal noise." The module aligns these three types of data based on a unified time stamp and links "Pump A" and "abnormal noise" in the text to the corresponding entities in the equipment knowledge graph through entity links, generating a sample triplet containing an image, text, and entity identifier set. After receiving this identifier set, the knowledge graph management module executes a SPARQL query to extract a related sub-graph from the plant-wide equipment knowledge graph stored in the Neo4j graph database, centered on pump A and including its model, associated pipeline, historical fault records, and related maintenance procedures. The cross-modal encoding and fusion module then commences: the Vision Transformer extracts image features, the Transformer encoder extracts text features, and the graph attention network encodes associated subgraphs to obtain knowledge features. Subsequently, the knowledge features are injected into the text features through a cross-modal attention mechanism, generating knowledge-enhanced text features. Both are ultimately projected into a unified 512-dimensional semantic space. The contrastive learning training module uses the vector representation of this batch of data, combined with a predefined rule (e.g., "high vibration signals should not correspond to normal appearance images"), to calculate the industrial semantic constraint loss, which, together with the standard contrastive loss, optimizes the model. The online continuous learning module monitors for drift in the feature distribution of a new batch of pump machine data exceeding a preset Mahalanobis distance threshold, triggering incremental learning through the model update interface. Finally, the RESTful API provided by the model service module can be called by the upper-layer diagnostic system for embedding calculations and similarity queries, assisting engineers in decision-making.

[0064] The beneficial effects of one of the embodiments in this specification include at least the following: by constructing a system architecture that includes a knowledge graph management module and a cross-modal coding fusion module, industrial domain knowledge is dynamically injected into the multimodal feature learning process in the form of structured subgraphs. Combined with a contrastive learning mechanism that introduces industrial semantic constraints, the model finally trained can not only achieve semantic alignment of multimodal data, but also ensure that the unified semantic representation and reasoning results generated strictly conform to predefined industrial logic rules. This effectively overcomes the defects of the purely data-driven method in the prior art, which may produce results that violate industrial common sense. It significantly improves the output reliability, decision credibility and practical application value of the industrial cognitive system in key tasks such as fault diagnosis and condition monitoring.

[0065] In some implementations, the data acquisition and preprocessing module is specifically used to: align visual frames, time-series segments, and corresponding text descriptions based on timestamps and device identifiers; and link industrial terms in the text to knowledge graph nodes through named entity recognition and entity linking technologies.

[0066] A visual frame can refer to a single still image captured by an industrial camera or webcam at a specific moment, such as a two-dimensional pixel array extracted from a continuous video stream at fixed time intervals or triggered by an event. It serves as the basic visual unit for time-series analysis or alignment with text descriptions. A time-series segment can refer to a finite-length block of data extracted from a continuous time-series signal, such as a one-dimensional numerical sequence obtained by sliding across a fixed time window from raw signals acquired by a vibration sensor. It is used to analyze the dynamic behavior characteristics of equipment within a specific time period. Corresponding text descriptions can refer to natural language descriptions that are temporally or logically associated with specific visual frames and time-series segments. For example, a log entry recorded by an operator in a monitoring system for a specific equipment anomaly event, used to semantically describe the visual and temporal phenomena occurring concurrently. A timestamp can refer to precise time information recording the moment data is generated or acquired, such as time encoding in Coordinated Universal Time (UTC) format accurate to milliseconds. It serves as a key index for aligning and matching cross-modal data in the time dimension. Equipment identifiers can be unique codes that identify a physical device or logical unit in an industrial setting. For example, equipment identifiers conforming to international standards are used to accurately associate different modalities of data from the same data source in a distributed system. Alignment refers to the process of establishing and ensuring correct correspondences between different data elements. For example, using a hash index table based on timestamps and equipment identifiers, images, data, and text records generated by the same device at the same time can be quickly matched to construct semantically consistent multimodal training samples. Named entity recognition (NER) refers to the technology of automatically identifying and classifying specific types of entity phrases from unstructured text. For example, using a Transformer-based sequence labeling model, equipment names, parameter names, fault phenomena, and other industrial entities can be extracted from work order text to provide structured input for subsequent entity linking. Entity linking technology refers to the process of associating entity mentions identified in text with specific nodes in a knowledge graph. For example, by calculating the semantic similarity between entity mentions and candidate entities in the knowledge graph vector space and selecting the most relevant entities for linking, semantic bridging between text and structured knowledge can be achieved. Industrial terminology refers to words or phrases with specialized meanings within a specific industrial field. Examples include terms like "axial runout," "cavitation margin," and "inverter overload," which are specialized terms used in equipment maintenance and process control to precisely describe states, behaviors, and parameters in industrial scenarios. Knowledge graph nodes refer to the basic units in a knowledge graph that represent a specific concept, entity, or object. For instance, in an equipment knowledge graph, a node can represent a specific pump, a bearing component, or a failure mode, serving as a fundamental element for storing and expressing domain knowledge.

[0067] As a concrete example: In a wind turbine generator condition monitoring scenario, the data acquisition and preprocessing modules operate synchronously. When the SCADA system records a vibration exceeding limit event of a wind turbine generator (equipment identifier WTG-07) at a specific time (timestamp 2023-11-07T10:05:32.123Z), the module simultaneously triggers the following actions: acquiring a visual frame of the gearbox appearance from an industrial camera installed in the nacelle; extracting 10 seconds of raw waveform data (5 seconds before and after the event time) from the continuous data stream of the vibration sensor as a time sequence segment; and automatically retrieving the corresponding text description recorded by the operator from the maintenance log: "WTG-07 gearbox area vibration value is high, accompanied by periodic knocking sound." The module then performs an alignment operation, binding these three types of data based on the timestamp and equipment identifier WTG-07. Next, the module initiates a named entity recognition process, identifying industrial terms such as "WTG-07," "gearbox," "high vibration value," and "periodic knocking sound" from the text description. Finally, through entity linking technology, these identified entities are matched and associated with corresponding nodes in the wind farm equipment knowledge graph, thereby completing the construction of a high-quality, semantically aligned multimodal sample, providing a reliable data foundation for subsequent model training.

[0068] By explicitly defining a data alignment mechanism based on timestamps and device identifiers, and combining named entity recognition and entity linking technologies, this invention ensures the consistency of multimodal industrial data in terms of time and semantics, providing high-quality, strongly correlated training samples for subsequent cross-modal comparative learning, and fundamentally improving the accuracy and reliability of the model's understanding of complex industrial scenarios.

[0069] In some implementations, the target knowledge graph stored in the knowledge graph management module includes equipment knowledge graphs, process parameter libraries, and historical fault case libraries, and supports SPARQL queries and subgraph extraction.

[0070] Equipment knowledge graphs can refer to domain-specific semantic networks built around industrial equipment. For example, such graphs are modeled and stored based on equipment models, component composition, performance parameters, and interrelationships, providing structured knowledge related to the entire equipment lifecycle. Process parameter libraries can refer to collections storing various process standards and operating parameters in industrial production, such as setpoints, allowable fluctuation ranges, and interrelationships for key process indicators like temperature, pressure, flow rate, and speed, used to define and constrain normal industrial production states. Historical failure case libraries can refer to collections of structured records of past equipment failures or abnormal events. For example, each case includes information such as failure symptoms, root causes, handling measures, and replaced components, and is associated with specific equipment and operating conditions, providing empirical evidence for failure diagnosis and prediction. SPARQL queries can refer to a standardized query language for retrieving and manipulating RDF graph data. For example, by writing SPARQL SELECT or CONSTRUCT queries, entities, attributes, and paths that meet specific conditions can be extracted from the graph database, enabling precise extraction of desired information fragments from large knowledge graphs. Subgraph extraction can refer to the operation of separating a smaller and complete local graph from a large-scale knowledge graph. For example, starting with a set of specified core entities, relevant nodes and edges are collected by traversing the directly connected entities and relationships, or by performing multi-hop queries, in order to construct a subset of knowledge that is highly relevant to the current context.

[0071] As a concrete example: In a fault diagnosis system of a semiconductor wafer manufacturing plant, the knowledge graph management module integrates three core knowledge sources: the equipment knowledge graph, which details the component trees and dependencies of key equipment such as lithography machines and etching machines; the process parameter library, which strictly defines the standard values ​​of hundreds of parameters such as temperature, humidity, and gas flow rate for each manufacturing process (such as exposure and development); and the historical fault case library, which records hundreds of abnormal events that have occurred over the past few years, such as "lens thermal drift causing linewidth deviation." When the system processes a new alarm for a lithography machine, the knowledge graph management module receives the identifier of that equipment. Subsequently, the module executes a predefined SPARQL query, which simultaneously retrieves the component information of the lithography machine from the equipment knowledge graph, searches the process parameter library for the standard parameters of the currently executed process step, and matches past cases with similar symptoms from the historical fault case library. Finally, through subgraph extraction, these knowledge nodes and relationships that are scattered in different libraries but are semantically closely related are merged to dynamically construct a comprehensive relational subgraph for this lithography machine alarm event, providing rich domain context for subsequent cross-modal analysis.

[0072] By integrating heterogeneous knowledge from multiple sources, such as equipment knowledge graphs, process parameter databases, and historical fault case databases, and using SPARQL query and subgraph extraction techniques for dynamic association and fusion, this invention can construct highly context-relevant and knowledge-dense related subgraphs for specific industrial scenarios. This provides a solid and rich domain knowledge foundation for subsequent cross-modal understanding and reasoning, significantly enhancing the accuracy and interpretability of cognitive system decisions.

[0073] In some implementations, the cross-modal coding and fusion module includes a visual encoder, a text encoder, and a knowledge injection module; the visual encoder uses a Vision Transformer network to extract image features; the text encoder uses a Transformer encoder to extract text features; the knowledge injection module uses a graph attention network to encode the sub-graph to obtain knowledge features, and fuses the knowledge features with the text features to generate knowledge-enhanced text features.

[0074] A visual encoder can refer to a neural network model that converts an input image into a numerical feature vector. For example, it might employ a Vision Transformer (ViT) architecture, segmenting the input image into a sequence of fixed-size image patches and performing global context encoding through multiple layers of Transformer blocks to extract deep semantic features. A Vision Transformer network can also refer to a computer vision model based on a pure Transformer architecture. For instance, it might reshape the input image into a series of flat image patches, add positional embeddings, and then perform sequence encoding through a multi-head self-attention mechanism and a feedforward neural network to achieve image feature extraction that does not rely on convolutional operations.

[0075] A text encoder can refer to a neural network model that converts an input text sequence into a numerical feature vector. For example, a standard Transformer encoder architecture uses multi-head self-attention and a feedforward neural network to perform contextual encoding on the input word sequence, extracting a deep semantic representation of the text. A Transformer encoder can also refer to a neural network structure composed of multiple identical encoding layers stacked together. Each encoding layer contains a multi-head self-attention sublayer and a feedforward neural network sublayer, employing residual connections and layer normalization to perform deep bidirectional feature encoding on the input sequence.

[0076] A knowledge injection module can refer to a specific component that integrates external knowledge features into text features. For example, it can use a graph attention network to encode the associated subgraph to obtain a graph representation, and then combine it with text features through cross-modal attention or feature concatenation to generate an enhanced text representation rich in domain knowledge. A graph attention network can refer to a neural network that operates on graph-structured data. For example, it can use an attention mechanism to calculate the weighted correlation between each node in the graph and its neighboring nodes, and iteratively aggregate neighbor information to update the node representation, effectively encoding structural information and semantic relationships in the graph. Knowledge features can refer to numerical vector representations extracted from graph-structured data. For example, graph-level embedding vectors obtained by propagating and aggregating information from all nodes in the associated subgraph using a graph neural network are used to encapsulate the semantic content of structured knowledge in the form of dense vectors.

[0077] As a concrete example, in boiler tube wall condition monitoring at a thermal power plant, cross-modal coding and fusion modules work collaboratively. When the system receives a sample triple containing an infrared image of the boiler furnace, the inspection record text "Local overheating of the water-cooled wall on the southeast side of boiler #3," and relevant equipment identifiers, the visual encoder is activated first. Its VisionTransformer network segments the infrared image into a 16x16 pixel block sequence and, through multi-layer self-attention computation, outputs a visual feature vector representing the global semantics of the image. Simultaneously, the Transformer encoder in the text encoder performs lexical processing on the input inspection text and, through multi-layer Transformer block encoding, outputs a text feature vector. The knowledge injection module runs concurrently: based on the equipment identifiers, it extracts a related sub-graph from the knowledge graph containing information about the material, design temperature limit, and historical leakage records of the water-cooled wall of boiler #3. Then, it uses a graph attention network to encode this sub-graph, obtaining condensed knowledge features. Finally, the knowledge injection module injects these knowledge features as contextual information into the text feature generation process through a cross-attention layer, producing the final knowledge-enhanced text features. Visual features and knowledge-enhanced text features are then projected into the same semantic space for comparative learning.

[0078] By employing the Vision Transformer network to extract image features and the Transformer encoder to extract text features, and innovatively introducing a knowledge injection module based on graph attention network to deeply integrate structured knowledge features with text features, the knowledge-enhanced text features generated by this invention not only contain the semantics of the original text, but also incorporate rich domain knowledge, enabling subsequent cross-modal comparative learning to achieve deeper semantic alignment and understanding that is more in line with domain logic in a unified semantic space.

[0079] In some implementations, the composite loss function used by the contrastive learning training module includes standard contrastive loss and industrial semantic constraint loss, which penalizes vector representations for violating predefined industrial logic rules based on a rule consistency discriminator.

[0080] A composite loss function can refer to an overall optimization objective composed of a weighted combination of multiple loss components. For example, it could be a linear weighted sum of standard contrastive loss and industry semantic constraint loss, used to jointly guide the model in learning data similarity and domain rule consistency. Standard contrastive loss can refer to the basic loss function used in multimodal contrastive learning to bring positive sample pairs closer together and push negative sample pairs further apart. For instance, it could employ the InfoNCE loss form based on noise contrastive estimation, calculating the ratio of the similarity of positive sample pairs to the similarity of all sample pairs and taking the negative logarithm, to encourage the model to learn discriminative feature representations. Industry semantic constraint loss can refer to an additional term introduced into the loss function to penalize behaviors that violate industry rules. For example, it could use a predefined logic rule calculator to perform consistency checks on the vector representation of the model output and generate a penalty score, ensuring that the model output conforms to common sense and logic in the industry domain. A rule consistency discriminator can refer to a component used to determine whether the model output violates predefined industry rules. For example, it could be a discriminant module based on logic rules or a small neural network, receiving the model's vector representation as input and outputting a binary decision or violation probability, used to quantify the degree of consistency between the model's behavior and domain knowledge. Predefined industrial logic rules refer to logical constraints pre-set based on industrial knowledge. Examples include propositions expressing causal or correlational relationships, such as "a high-temperature alarm state should not correspond to a normal equipment image" or "vibration amplitude within a certain speed range must not exceed a specific threshold." These constraints are used to constrain the model to learn within a physically reasonable space. Penalties refer to applying negative numerical incentives to behaviors that do not meet expectations during the optimization process. For instance, when a rule consistency discriminator determines that a sample pair violates a rule, a penalty term proportional to the degree of violation is added to the overall loss function. This is used to guide the model parameters to update in a direction that conforms to the rule through gradient descent.

[0081] As a concrete example: In a pump group health management system at a chemical plant, the contrastive learning training module uses a composite loss function to perform end-to-end optimization of the model. For a training batch, the module first calculates the standard contrastive loss, which brings the feature vector distance between the text describing "bearing wear on pump A" and its actual vibration spectrum image closer, while simultaneously increasing its distance from the feature vector distance of the image showing "pump B operating normally." Simultaneously, a rule consistency discriminator is activated, loading a set of predefined industrial logic rules, such as "the severity of bearing wear should be proportional to vibration energy." The discriminator examines the feature vectors generated by the current model for the text "slight wear" and the image of "high vibration energy," determining whether their relationship conforms to this physical rule. If the model brings the feature distance between "slight wear" and "high vibration energy" too close, violating the proportional relationship in the rule, the rule consistency discriminator outputs a high violation score. Subsequently, the industrial semantic constraint loss calculates a penalty term based on this score and is weighted and summed with the standard contrastive loss to form the final composite loss. Through backpropagation, the model parameters are optimized synchronously, learning to match the correct image-text pairs while ensuring that the matching behavior does not violate basic industrial physical logic.

[0082] By constructing a composite loss function that includes standard contrast loss and industrial semantic constraint loss, and introducing a rule consistency discriminator to check and penalize the industrial logic compliance of the model output, this invention enables the model to strictly follow the physical laws and business logic defined by domain knowledge while learning data-driven features, thereby producing inference results that are not only statistically reasonable but also physically reliable, which greatly improves the reliability of the system in safety-critical industrial applications.

[0083] In some implementations, the industrial semantic constraint loss is calculated based on the rule violation score, and the formula for calculating the rule violation score includes:

[0084]

[0085] in, It is the rule violation score, a non-negative scalar value used to quantify the degree to which the i-th sample violates the industry rules; the larger the value, the more serious the violation. It is a visual feature vector, which comes from the visual encoder output of the cross-modal coding and fusion module, representing the semantic embedding of the image in the i-th sample; is the knowledge-enhanced text feature vector, which comes from the knowledge injection module output of the cross-modal coding and fusion module, representing the semantic embedding of the text in the i-th sample after knowledge enhancement; K is the total number of predefined industrial logic rules, which comes from the rule base maintained in the knowledge graph management module; is the violation penalty weight of the k-th rule, a non-negative scalar parameter used to adjust the importance of the rule in the total violation score; This is the smoothed violation strength function for the k-th rule, used to calculate the violation strength of the k-th rule at a given distance value. Its output value smoothly increases from 0. The calculation formulas include:

[0086]

[0087] in, is the tolerance threshold for the k-th rule, a non-negative scalar representing the maximum value of the squared distance between the visual and text projection features allowed by the rule. Exceeding this value indicates that the degree of violation begins to increase significantly. It is the sensitivity coefficient of the k-th rule, a positive scalar that controls the rate at which the violation intensity increases as the distance exceeds the threshold;

[0088] It is a distance function, defined as the square of the Euclidean distance, i.e. , used to calculate the difference between two vectors; It is the visual projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project visual features onto the semantic subspace of interest for the k-th rule; It is the text projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project knowledge-enhanced text features onto the same rule-based semantic subspace. This represents the dimension of the original feature vector. This represents the dimension of the projected feature vector.

[0089] A rule violation score can be a quantified numerical value used to measure the extent to which the vector representation generated from a specific image-text sample pair violates all predefined industry logic rules. For example, it can be a non-negative scalar value obtained by weighted summation of multiple rule violation intensities, used to calculate the penalty in the loss function. A non-negative scalar value can also refer to a single numerical value greater than or equal to 0. For instance, when calculating the rule violation score, the output is designed to always be no less than 0 so that it can be directly used as a loss penalty term. The degree of violation of industry rules can refer to the magnitude of the deviation between the model's cross-modal semantic understanding of a sample pair and the hard or soft logic defined by domain experts. A higher score indicates that the model's understanding is less in line with industry common sense. A visual feature vector can refer to a fixed-dimensional numerical array representing high-level semantic information of the input image after deep processing by a visual encoder, such as a D-dimensional vector v extracted and output by a Vision Transformer network. iIt captures key visual patterns in an image. Semantic embedding refers to mapping unstructured data (such as images and text) to a point in a high-dimensional vector space through a deep learning model. The position of this point represents the semantic information of the data. For example, the visual feature vector v_i is the semantic embedding representation of an image in a unified semantic space. Knowledge-enhanced text feature vectors refer to text semantic representation vectors that incorporate structured knowledge graph information. For example, text features are injected with sub-graph knowledge features encoded by a graph attention network, forming a more domain-deep D-dimensional vector. Violation penalty weights refer to the weight coefficients assigned to each specific rule violation when calculating the total rule violation score. For example, a non-negative scalar parameter λ_k is used to adjust the relative importance of different rules in the total penalty. A non-negative scalar parameter can refer to an adjustable constant in the model or calculation formula whose value is restricted to be greater than or equal to 0. For example, the violation penalty weight λ k It is a non-negative scalar parameter that needs to be preset or learned. A smooth violation intensity function can refer to a function that maps the distance between visual and text projection features to a smooth, continuous, and differentiable violation intensity value, which facilitates model optimization.

[0090] Violation intensity at a distance value refers to the degree of rule violation calculated when the difference between visual and text projection features reaches a specific distance value for a particular rule. The tolerance threshold refers to the maximum allowable difference between visual and text projection features when judging whether a rule has been violated. Visual and text projection features refer to the new representations of the original visual features and knowledge-enhanced text features in the rule's semantic subspace after being transformed by a rule-specific projection matrix. The maximum value of the squared distance refers to the square of the maximum allowable difference between two objects under a certain metric. The sensitivity coefficient refers to a parameter that controls how quickly the output value of a function changes with the input. A positive scalar can refer to a real number that is strictly greater than 0, such as the sensitivity coefficient β. k Defined as a positive scalar to ensure that the violation strength monotonically increases with distance. A distance function can refer to a mathematical function used to measure the difference or similarity between two vectors. The square of the Euclidean distance can refer to the square of the Euclidean distance metric, i.e., the sum of the squares of the differences in each dimension of the two vectors. A projection matrix can refer to a linear transformation matrix used to map vectors from their original high-dimensional space to a new low-dimensional or specific subspace. A visual projection matrix can refer to a matrix specifically learned for the k-th industrial logic rule, used to project the visual feature vector v... iThe matrix mapping to the semantic subspace related to the rule. The semantic subspace of interest to the rule can refer to a lower-dimensional or more targeted feature space implicitly defined by a certain industrial logic rule. The text projection matrix can refer to a matrix specifically learned for the k-th industrial logic rule, used to map the knowledge-enhanced text feature vectors to the semantic subspace related to that rule. The dimension of the projected feature vector can refer to the number of numerical elements in the feature vector in the new semantic subspace after the transformation by the projection matrix.

[0091] As a concrete example: When training a cognitive foundation model for safety monitoring in a chemical plant, the system sets an industrial logic rule: the text description of "Reaction vessel pressure too high alarm" should be semantically highly correlated with an image showing the pressure gauge pointer in the red warning zone, while semantically distant from an image showing normal pressure. This rule is numbered k=1, assigned a high static importance of 2.0 by domain experts, and set a tolerance threshold of 0.1 and a sensitivity coefficient of 5.0. During training, for a mismatched sample pair containing the text "Reaction vessel pressure too high alarm" and an image showing normal pressure, its visual feature v i The knowledge-enhanced text features are extracted. The system uses the initialized projection matrix to project both features onto the semantic subspace of Rule 1, obtaining projected features and calculating their squared Euclidean distance. Since this sample pair does not conform to the rule logic, the distance d is expected to be relatively large (e.g., d=2.0). Substituting this d value into the smoothing violation intensity function, since d>tolerance threshold, a high violation intensity value (e.g., 0.8) is obtained. Simultaneously, the system calculates the smoothing dynamic violation intensity based on the current batch of data. Finally, combining static importance and dynamic intensity, the penalty weight λ1 is calculated through an adaptive weight adjustment mechanism. The rule violation score S is λ1*0.8, which will be incorporated into the industrial semantic constraint loss, thereby adjusting the model parameters during backpropagation and forcing the model to learn to distinguish such sample pairs that do not conform to industrial logic.

[0092] By transforming abstract industrial domain knowledge into differentiable and optimizable mathematical constraints and incorporating them into a contrastive learning framework, the cross-modal semantic representations learned by the model not only depend on the statistical correlation of the data itself, but are also required to conform to the logical rules defined by domain experts. This significantly improves the reliability, security, and interpretability of the model's reasoning in industrial scenarios, and reduces absurd or dangerous semantic associations caused by data bias or annotation noise.

[0093] In some implementations, the projection matrix is ​​initialized. and The steps include:

[0094] The embedding representations of entities and relations involved in the k-th rule are extracted from the knowledge graph management module, and then mapped to D through a linear transformation layer. r 3D space, to initialize the projection matrix respectively. and ;

[0095] Optimize projection matrix and The steps include:

[0096] During the training process of the comparative learning training module, the projection matrix is... and As a learnable parameter, a regularization term is added to the composite loss function. The formula for calculating the regularization term includes:

[0097]

[0098] in, It is the projection matrix regularization loss term corresponding to the k-th industrial logic rule. and The matrix obtained from the initialization step, The Frobenius norm of the matrix is ​​used to constrain the deviation between the optimized projection matrix and the initialization matrix.

[0099] The embedding representation of entities and relations can refer to the pre-trained or randomly initialized vectors corresponding to nodes (entities) and edges (relationships) related to the content of a given industrial logic rule in a knowledge graph. Examples include low-dimensional vectors obtained through knowledge representation learning techniques such as TransE or RotatE. A linear transformation layer can refer to a fully connected layer in a neural network that contains only weight matrices and bias vectors. It is used to perform affine transformations on the input vector to change its dimension or feature distribution. For example, it can map the embedding dimension of a knowledge graph to D. r A learnable layer. Learnable parameters refer to variables whose values ​​are automatically adjusted during model training using optimization algorithms such as gradient descent to minimize the loss function. The Frobenius norm of a matrix refers to the square root of the sum of the squares of all its elements. It is a commonly used metric to measure the overall size of a matrix or the difference between two matrices, for example, in regularization terms to calculate the difference between a projected matrix and its initialized version.

[0100] As a concrete example, let's continue with the pressure alarm rule in a chemical plant. Before model training begins, the system first extracts pre-trained embedding vectors from the knowledge graph management module for the entities (such as "reaction vessel", "pressure sensor", and "alarm status") and relationships (such as "hasStatus" and "exceedsThreshold") directly associated with the rule. These embeddings are concatenated or averaged and then fed into a dedicated linear transformation layer, whose weight matrix is ​​designed with an output dimension of D_r. This generates initialized versions of the visual and text projection matrices, respectively. During training, the visual and text projection matrices are set as learnable parameters, and optimization begins from these meaningful initial values. When calculating the total loss, in addition to the standard contrastive loss and the industrial semantic constraint loss composed of rule violation scores, an additional regularization term is added. This regularization term acts as a "flexible constraint," allowing the projection matrices to be fine-tuned during training to adapt to specific visual and text encoding features, while simultaneously penalizing them for excessive deviations from the direction initialized by the knowledge graph semantics. This ensures that the optimization process of the rule projection matrix never deviates from its initial domain semantic foundation.

[0101] By utilizing the semantic information contained in the knowledge graph to initialize the rule projection matrix and adding regularization constraints based on the initialization matrix to the optimization process, it is ensured that these key rule mapping parameters can learn subtle adjustments from the data during training without completely deviating from the original semantic direction defined by the domain knowledge. This enhances the stability and interpretability of the rule injection process, making the representation of the "rules" themselves in the model more solid and reliable.

[0102] In some implementations, penalty weights Calculated through an adaptive weight adjustment mechanism, its formula includes:

[0103]

[0104] in, It is the global penalty scaling factor, which is a positive scalar hyperparameter preset by the trainer and used to adjust the strength of all rule penalty weights as a whole; It is a hybrid coefficient of static and dynamic weights. This parameter is an adjustable hyperparameter in the range [0,1], used to balance the ratio between the static preset importance of the rule and the dynamic violation intensity learned during model training. It is the static importance weight of the kth rule. This parameter is a real number and comes from the rule importance score pre-set by domain experts in the knowledge graph management module. It is an S-shaped function that maps the input to the (0,1) interval, which is used to map the static importance of expert ratings to a stable numerical range; It is the hyperbolic tangent function, which maps the input to the interval (-1, 1); for non-negative inputs... Under the condition, the actual output range is [0,1), which is used to map the scaled dynamic violation intensity to a bounded range; It is a scaling factor for the dynamic violation intensity. This parameter is a positive scalar hyperparameter that is preset by the trainer to adjust the range of dynamic violation intensity values ​​and control the sensitivity of the dynamic part to the final weights. This represents the smooth dynamic violation intensity of the k-th rule during the current training phase. This parameter is a non-negative real number, calculated using momentum updates, and reflects the average violation intensity of the k-th rule in the current training batch, while avoiding batch fluctuations. The calculation formulas include;

[0105]

[0106] in, The smoothing coefficient has a value in the range [0.9,1) and is used to control the degree to which historical violation intensity information is retained in the smoothing calculation; Batch size; This represents the smooth dynamic violation strength value at the t-th training iteration.

[0107] An adaptive weight adjustment mechanism can refer to a method for dynamically adjusting the calculation of penalty weights for each rule based on real-time feedback during model training (such as the frequency and intensity of rule violations). For example, it can calculate the final λ by combining statically preset importance and dynamically learned violation intensity. kThe global penalty scaling factor can refer to a common multiplier applied to the final penalty weights of all rules, used to control the relative strength of the industrial semantic constraint loss in the total loss. For example, it can be a positive scalar hyperparameter ω preset by the trainer. The hybrid coefficient of static and dynamic weights can refer to a hyperparameter used to balance the proportions of the static preset importance and dynamic violation strength in the final penalty weights. For example, it can be an adjustable coefficient α in the [0,1] interval, where a larger α indicates greater reliance on expert priors. The static importance weight of a rule can refer to the importance score pre-assigned to each rule by domain experts, reflecting the rule's criticality in the business logic. The sigmoid function can refer to a non-linear activation function that maps real input numbers to the (0,1) interval, with an S-shaped curve. The scaled dynamic violation strength can refer to the value obtained by multiplying the smoothed dynamic violation strength f^k by a scaling factor η, used to adjust the order of magnitude of the dynamic part to match the static weights processed by the sigmoid function. The scaling factor for dynamic violation intensity can refer to a hyperparameter used to adjust the range of the smoothed dynamic violation intensity f~_k, such as a positive scalar η, controlling the sensitivity of the dynamic part to the final weights. Smoothing dynamic violation intensity can refer to a smoothed value calculated using momentum updates, reflecting the average violation intensity of the rule in recent training batches. Momentum updates can refer to a calculation method that uses historical values ​​to smooth the current value, commonly used in optimization algorithms or moving average calculations. The smoothing coefficient can refer to the weighting coefficient assigned to historical values ​​in momentum update calculations, determining the degree to which historical information is preserved, such as a γ value ranging from [0.9, 1), where a larger value results in smoother updates.

[0108] As a concrete example: When training a cognitive model for a chemical plant, the system presets a global scaling factor ω=1.0, a mixing coefficient α=0.7 (more biased towards expert priors), and a dynamic scaling factor η=2.0. For the "high pressure alarm" rule (k=1), the expert sets its static importance to 2.0. In the early stages of training, the model has not yet learned well, and this rule is frequently violated. Assuming its smooth dynamic violation strength is high at 1.5, the dynamic part is tanh(2.0*1.5)=tanh(3.0)≈0.995. The static part is sigmoid(2.0)≈0.88. Substituting into the formula, we get λ1=1.0*[0.7*0.88+0.3*0.995]≈0.91. As training progresses, the model gradually learns to obey the rule, and the smooth dynamic violation strength decreases to 0.2. At this point, the dynamic part becomes tanh(2.0*0.2)=tanh(0.4)≈0.38. The static part remains unchanged. The final value is λ1 = 1.0 * [0.7 * 0.88 + 0.3 * 0.38] ≈ 0.73. The penalty weight is automatically reduced, decreasing the rule's contribution to the overall loss, allowing optimization to focus on other rules that are still severely violated. Simultaneously, the smoothing coefficient γ (e.g., 0.95) ensures that updates to the dynamic violation intensity are smoothed without drastic fluctuations caused by a single anomalous batch.

[0109] By introducing an adaptive weight adjustment mechanism that integrates static expert priors and dynamic training feedback, the penalty weights of industrial rules are no longer fixed values, but can be intelligently adjusted according to the model's current learning state. This guides the optimization process to prioritize rules that are both important (expert-defined) and not yet mastered by the model (high dynamic violation intensity), achieving more efficient and intelligent allocation of training resources and improving model convergence efficiency and final performance.

[0110] In some implementations, the model service module is encapsulated as a microservice and provides services through multiple application programming interfaces (APIs). These APIs include: a visual embedding interface for receiving input images and returning corresponding visual embedding vectors; a knowledge-enhanced text embedding interface for receiving input text and returning corresponding knowledge-enhanced text embedding vectors; a similarity calculation interface for calculating and returning the semantic similarity between the image and the text; and a model update interface for receiving new samples and triggering the online continuous learning module to perform incremental updates.

[0111] Microservices can refer to an architectural style that divides a single application into a set of small, loosely coupled services. For example, each service runs an independent process, interacts through lightweight communication mechanisms, and is built around business capabilities to achieve high maintainability, scalability, and technological heterogeneity. Application Programming Interfaces (APIs) can refer to a set of well-defined rules and tools for interaction between software components. For example, an HTTP interface using a RESTful design style specifies request methods, resource paths, parameters, and response formats to enable standardized data exchange and function calls between different software systems. Visual embedding interfaces can refer to service endpoints that specifically process image input and return its vector representation. For example, it could be a POST request interface that accepts multipart / form-data format image file uploads, internally calls a visual encoder for processing, and returns a JSON-formatted embedding vector to provide image semantic feature extraction services to clients. Visual embedding vectors can refer to the numerical representation obtained after feature extraction of the input image by a visual encoder. For example, it could be a 512-dimensional floating-point array that encapsulates the deep semantic information of the input image, used for similarity calculation and semantic retrieval in vector space. A knowledge-enhanced text embedding interface can refer to a server endpoint that specifically processes text input and returns its knowledge-enhanced vector representation. For example, it might be a POST request interface that receives text content in application / json format, internally performing text encoding, knowledge retrieval, and fusion processes to return a text feature vector incorporating domain knowledge. A knowledge-enhanced text embedding vector can refer to a numerical representation obtained after enhancing the input text through a knowledge injection module. For example, it could be a 512-dimensional floating-point array that contains both the semantic information of the text itself and incorporates relevant domain knowledge, providing a more in-depth text feature representation. A similarity calculation interface can refer to a server endpoint that specifically calculates the similarity between two embedding vectors. For example, it might be a POST request interface that receives two embedding vectors as input parameters, internally calculating metrics such as cosine similarity or Euclidean distance to quantify the association strength between two semantic representations. Semantic similarity between an image and text can refer to the closeness between a visual embedding vector and a knowledge-enhanced text embedding vector in a unified semantic space. For example, it could be a scalar value between -1 and +1 obtained by calculating the cosine of the angle between the two vectors, used to evaluate the semantic matching degree between image content and text description. A model update interface can refer to a service endpoint that allows external systems to trigger model retraining or incremental updates. For example, it can be a POST request interface that receives new training sample sets and training configuration parameters, and internally calls an online continuous learning module to start the training process, thereby enabling continuous optimization and adaptation of model capabilities.Incremental update can refer to the learning process of adjusting local parameters using new samples based on the existing model. For example, the elastic weight consolidation algorithm adds regularization penalties to important old parameters during optimization, so that the model can adapt to new knowledge while retaining existing knowledge.

[0112] As a concrete example: In the central monitoring platform of a smart factory, the model service module is deployed as a set of Docker containerized microservices. Maintenance personnel initiate a troubleshooting request through the HMI (Human-Machine Interface): uploading an image of a suspected faulty motor. The front-end application first calls the visual embedding interface, a RESTful API with endpoint ` / api / v1 / visual-embedding`. This interface receives the JPEG input image via POST and returns a 512-dimensional visual embedding vector. Simultaneously, the engineer enters the observed phenomenon, "The motor makes abnormal noises and the casing is overheating," into the text box. The front-end application calls the knowledge-augmented text embedding interface (endpoint ` / api / v1 / knowledge-augmented-text-embedding`), submitting this input text. The interface returns the corresponding knowledge-augmented text embedding vector. Subsequently, the front-end calls the similarity calculation interface (endpoint ` / api / v1 / semantic-similarity`), taking the two resulting embedding vectors as input. Internally, the interface calculates their cosine similarity and returns a score of 0.87, indicating a high correlation between the image and the text. Based on this result, the system recommends similar historical failure cases. Furthermore, once a new batch of qualified samples has been accumulated, the administrator can upload these new samples through the model update interface (endpoint ` / api / v1 / model-incremental-update`) to safely trigger an incremental update of the model without downtime.

[0113] By encapsulating the trained model into a set of well-defined microservices and providing standardized application programming interfaces including visual embedding, knowledge-enhanced text embedding, similarity calculation, and model updates, this invention enables complex multimodal cognitive capabilities to serve upper-level industrial applications in an easily integrated and on-demand manner, while ensuring the system itself can continuously evolve and effectively support the construction and operation of intelligent industrial systems.

[0114] Corresponding to the above system embodiments, this specification also provides embodiments of the execution method of an industrial cognitive foundation system based on multimodal contrastive learning. Figure 2 A flowchart illustrating an industrial cognitive foundation method based on multimodal contrastive learning, as provided in some embodiments of this specification, is shown. Figure 2 As shown, the specific steps include:

[0115] An industrial multimodal heterogeneous training dataset is constructed by synchronously collecting textual, visual, temporal, and structured knowledge modal data from multiple sources in the industrial field. Standardized sample triples are generated through alignment, where each sample triple includes an image, a text description, and a set of candidate identifiers pointing to core entities in a predefined target knowledge graph. The candidate identifier set is obtained from the aligned text description through named entity recognition and preliminary entity linking. An industrial-enhanced cross-modal encoder is constructed, including a visual encoder, a text encoder, and a knowledge injection module, used to extract visual features, textual features, and knowledge features from the associated sub-graphes. The system integrates and generates knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space along with visual features. A composite contrastive loss function for industrial semantic constraints is designed, comprising a standard contrastive loss and an industrial semantic constraint loss calculated based on rule violation scores, whereby the rule violation scores quantify the degree to which the model output violates industrial rules. End-to-end contrastive learning training is performed, using the vector representations and the composite contrastive loss function to optimize model parameters, outputting an optimized industrial cognitive foundation model. An online continuous learning and model update mechanism is deployed to monitor the new sample data stream, detect semantic drift, and trigger incremental model updates when semantic drift exceeds a threshold.

[0116] Standardized sample triples refer to data units that, after unified processing, contain images, text descriptions, and associated subgraphs. For example, timestamp alignment and entity linking techniques ensure semantic consistency among elements within the triples, serving as basic data input for model training. Industrial-enhanced cross-modal encoders refer to cross-modal feature extraction components that integrate industry domain knowledge. For instance, a visual encoder, text encoder, and knowledge injection module work collaboratively to map image, text, and subgraph features to a unified semantic space, generating knowledge-enhanced cross-modal representations. Vector representations in a unified semantic space refer to the numerical encoding of different modal data in a shared vector space. For example, a linear projection layer maps visual features and knowledge-enhanced text features to floating-point arrays in a Euclidean space of the same dimension, used for cross-modal similarity calculations within this space. A composite contrastive loss function with industrial semantic constraints refers to an optimization objective that combines standard contrastive loss and industry rule violation penalties. For example, a weighted summation of the standard InfoNCE loss and an industrial semantic constraint loss calculated based on rule violation scores forms the overall loss, jointly guiding the model to learn data similarity and domain rule consistency. End-to-end contrastive learning training can refer to the contrastive learning process that optimizes the entire process from raw data input to model output. For example, it uses composite loss functions and optimization algorithms to iteratively update encoder parameters until convergence, which is used to achieve automatic feature learning and alignment of the model on multimodal data.

[0117] As an example:

[0118] Step 1: Construct an industrial multimodal heterogeneous training dataset:

[0119] Data Acquisition: The following four types of data are collected synchronously from multi-source systems in the industrial field:

[0120] Text modality: Equipment operation manual, process procedure documents, fault repair logs, safety regulations;

[0121] Visual modalities: high-definition images of equipment appearance, production line monitoring video frames, and screenshots of AR operation interfaces;

[0122] Temporal modes: sampling sequences from sensors such as temperature, pressure, vibration, and current (sampling frequency ≥ 1Hz);

[0123] Structured knowledge modalities: Equipment knowledge graphs (e.g., <Pump A, Rated Power, 55kW>) stored in the form of triples, process parameter databases, and historical fault case databases.

[0124] Data alignment and annotation:

[0125] Based on timestamps and device IDs, visual frames, time-series segments, and corresponding text descriptions are aligned.

[0126] By using Named Entity Recognition (NER) and entity linking technology, industrial terms in the text (such as "centrifugal pump" and "reflux ratio") are linked to knowledge graph nodes.

[0127] Construct training sample triples:

[0128]

[0129] in For image / video frames, For text description, For associated sub-maps.

[0130] Step 2: Construct an industrial-enhanced cross-modal encoder:

[0131] This step builds three core coding modules:

[0132] 1. Vision Encoder:

[0133] Vision Transformer (ViT) is used as the backbone network;

[0134] Input: Image ;

[0135] Output: Visual feature vector .

[0136] 2. Text Encoder:

[0137] Employs a Transformer encoder;

[0138] Input: Text sequence (Through word segmentation and positional encoding);

[0139] Output: Original text features .

[0140] 3. Knowledge Injection Module:

[0141] enter: Sub-graph (number of nodes ≤ 50);

[0142] The sub-graph is encoded using a graph attention network (GAT):

[0143]

[0144] Integrating knowledge embedding with text features:

[0145]

[0146] Output: Knowledge-enhanced text features .

[0147] Step 3: Design a composite contrastive loss function for industrial semantic constraints:

[0148] Based on the standard contrast loss, an industrial rule consistency discriminant is introduced to construct a composite loss:

[0149] 1. Standard Contrast Loss (InfoNCE):

[0150]

[0151] in For cosine similarity, This is the temperature coefficient.

[0152] 2. Industrial semantic constraint loss:

[0153] A set of predefined industrial logic rules (such as "the pump cannot be operated in a liquid-free state") are formalized into predicate logic;

[0154] Build a rule consistency discriminator Input feature pairs Output whether the rule is violated (0 / 1);

[0155] Penalize samples that violate the rules:

[0156]

[0157] 3. Total loss function:

[0158]

[0159] Step 4: Perform end-to-end comparative learning training:

[0160] Dataset Input encoder, in loss Training was conducted under guidance;

[0161] The AdamW optimizer is used with a learning rate of 1. Batch size 256;

[0162] Training stops when the semantic alignment error rate on the validation set is less than 3%.

[0163] Output: Industrial cognitive foundation model, including three encoders: ViT, Transformer, and GAT, and fusion parameters.

[0164] Step 5: Deploy an online continuous learning and model update mechanism:

[0165] 1. New Data Monitoring: The system continuously receives new samples. ;

[0166] 2. Semantic drift detection: Calculate the average semantic distance between the new sample and the existing knowledge base. ;

[0167] 3. Incremental update triggered: If (threshold) If the value is 0.35, then fine-tuning is initiated:

[0168] Freeze the underlying parameters of ViT and Transformer;

[0169] Only minor adjustments were made to the higher-level adapter layer and GAT module;

[0170] Fine-tuning period ≤ 24 hours;

[0171] 4. Model version management: Uses semantic version numbers and supports rollback and A / B testing.

[0172] By deeply embedding structured prior knowledge such as industrial knowledge graphs, equipment parameter databases, and safety standards into a cross-modal contrastive learning framework, and designing a composite loss function with industrial semantic constraints, the problem of high semantic alignment error in general models in industrial scenarios is fundamentally solved. Because the model explicitly follows physical and technological rules such as "pumps must not run dry" and "reactor temperature must not exceed limits" during training, its output semantic vectors are not only aligned in mathematical space but also reliable in engineering logic, thus significantly reducing the semantic alignment error rate and greatly improving the accuracy and reliability of downstream tasks (such as fault diagnosis, intelligent question answering, and anomaly warning).

[0173] The above is an illustrative scheme of an execution method for an industrial cognitive base system based on multimodal contrastive learning, as described in this embodiment. It should be noted that the technical solution of this execution method for an industrial cognitive base system based on multimodal contrastive learning belongs to the same concept as the technical solution of the aforementioned industrial cognitive base system based on multimodal contrastive learning. Details not described in detail in the technical solution of the execution method for an industrial cognitive base system based on multimodal contrastive learning can be found in the description of the technical solution of the aforementioned system.

[0174] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0175] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this invention. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An industrial cognitive foundation system based on multimodal contrastive learning, characterized in that, include: The data acquisition and preprocessing module is used to synchronously acquire text modality, visual modality, temporal modality and structured knowledge modality data from multi-source systems in the industrial field, and generate standardized sample triples after alignment. The sample triples include an image, a text description and a set of candidate identifiers pointing to core entities in a preset target knowledge graph. The set of candidate identifiers is obtained from the aligned text description through named entity recognition and preliminary entity linking. The knowledge graph management module is used to store and maintain the target knowledge graph, and to perform semantic parsing and entity linking based on the text description in the sample triples and the identifier set, and to extract and construct a semantically rich association subgraph related to the current context from the stored target knowledge graph. The cross-modal coding and fusion module is used to extract visual features, text features and knowledge features from the associated sub-graph and fuse them to generate knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space together with the visual features. The contrastive learning training module is used to calculate the contrastive loss and industrial semantic constraint loss using the vector representation, and optimize the model parameters based on the composite loss function to obtain the optimized industrial cognitive foundation model. The composite loss function used by the contrastive learning training module includes the standard contrastive loss and the industrial semantic constraint loss. The industrial semantic constraint loss is based on a rule consistency discriminator that penalizes whether the vector representation violates predefined industrial logic rules. The industrial semantic constraint loss is calculated based on a rule violation score, and the formula for calculating the rule violation score includes: in, It is the rule violation score, a non-negative scalar value used to quantify the degree to which the i-th sample violates the industry rules; the larger the value, the more serious the violation. It is a visual feature vector, which comes from the visual encoder output of the cross-modal coding and fusion module, and represents the semantic embedding of the image in the i-th sample; is the knowledge-enhanced text feature vector, which comes from the knowledge injection module output of the cross-modal coding and fusion module, representing the semantic embedding of the text in the i-th sample after knowledge enhancement; K is the total number of predefined industrial logic rules, which comes from the rule base maintained in the knowledge graph management module; is the violation penalty weight of the k-th rule, a non-negative scalar parameter used to adjust the importance of the rule in the total violation score; It is the smooth violation strength function of the k-th rule, used to calculate the violation strength of the k-th rule at a given distance value, and its output value increases smoothly from 0; It is a distance function, defined as the square of the Euclidean distance, i.e. , used to calculate the difference between two vectors; It is the visual projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project visual features onto the semantic subspace of interest for the k-th rule; It is the text projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project knowledge-enhanced text features onto the same rule-based semantic subspace. This represents the dimension of the original feature vector. Indicates the dimension of the projected feature vector; The online continuous learning module is used to monitor the new sample data stream, detect semantic drift, and trigger incremental model updates when semantic drift exceeds a threshold. The model service module is used to encapsulate the optimized industrial cognitive foundation model and provide interfaces for embedding computation, similarity query and model update.

2. The system according to claim 1, characterized in that, The data acquisition and preprocessing module is specifically used to: align visual frames, time-series segments, and corresponding text descriptions based on timestamps and device identifiers; and link industrial terms in the text to knowledge graph nodes through named entity recognition and entity linking technology.

3. The system according to claim 1, characterized in that, The knowledge graph management module stores target knowledge graphs including equipment knowledge graphs, process parameter libraries, and historical fault case libraries, and supports SPARQL queries and subgraph extraction.

4. The system according to claim 1, characterized in that, The cross-modal coding and fusion module includes a visual encoder, a text encoder, and a knowledge injection module. The visual encoder uses a Vision Transformer network to extract image features. The text encoder uses a Transformer encoder to extract text features. The knowledge injection module uses a graph attention network to encode sub-graphs to obtain knowledge features, and then fuses the knowledge features with the text features to generate knowledge-enhanced text features.

5. The system according to claim 1, characterized in that, calculate The calculation formulas include: in, is the tolerance threshold for the k-th rule, a non-negative scalar representing the maximum value of the squared distance between the visual and text projection features allowed by the rule. Exceeding this value indicates that the degree of violation begins to increase significantly. It is the sensitivity coefficient of the k-th rule, a positive scalar that controls the rate at which the violation intensity increases as the distance exceeds the threshold.

6. The system according to claim 5, characterized in that, Initialize the projection matrix and The steps include: The embedding representations of the entities and relations involved in the k-th rule are extracted from the knowledge graph management module, and then mapped to D through a linear transformation layer. r 3D space, to initialize the projection matrices respectively. and ; Optimize the projection matrix and The steps include: During the training process of the contrastive learning training module, the projection matrix is... and As a learnable parameter, a regularization term is added to the composite loss function. The calculation formula for the regularization term includes: in, It is the projection matrix regularization loss term corresponding to the k-th industrial logic rule. and The matrix obtained from the initialization step, The Frobenius norm of the matrix is ​​used to constrain the deviation between the optimized projection matrix and the initialization matrix.

7. The system according to claim 5, characterized in that, The penalty weight Calculated through an adaptive weight adjustment mechanism, its formula includes: in, It is the global penalty scaling factor, which is a positive scalar hyperparameter preset by the trainer and used to adjust the strength of all rule penalty weights as a whole; It is a hybrid coefficient of static and dynamic weights. This parameter is an adjustable hyperparameter in the range [0,1], used to balance the ratio between the static preset importance of the rule and the dynamic violation intensity learned during model training. It is the static importance weight of the k-th rule. This parameter is a real number and comes from the rule importance score pre-set by domain experts in the knowledge graph management module. It is an S-shaped function that maps the input to the (0,1) interval, which is used to map the static importance of expert ratings to a stable numerical range; It is the hyperbolic tangent function, which maps the input to the interval (-1, 1); for non-negative inputs... Under the condition, the actual output range is [0,1), which is used to map the scaled dynamic violation intensity to a bounded range; It is a scaling factor for the dynamic violation intensity. This parameter is a positive scalar hyperparameter that is preset by the trainer to adjust the range of dynamic violation intensity values ​​and control the sensitivity of the dynamic part to the final weights. This represents the smooth dynamic violation intensity of the k-th rule during the current training phase. This parameter is a non-negative real number, calculated using momentum updates, and reflects the average violation intensity of the k-th rule in the current training batch, while avoiding batch fluctuations. The calculation formulas include; in, The smoothing coefficient has a value in the range [0.9,1) and is used to control the degree to which historical violation intensity information is retained in the smoothing calculation; Batch size; This represents the smooth dynamic violation strength value at the t-th training iteration.

8. The system according to claim 1, characterized in that, The model service module is encapsulated as a microservice and provides services through multiple application programming interfaces (APIs), including: The visual embedding interface is used to receive input images and return the corresponding visual embedding vectors; The knowledge-enhanced text embedding interface is used to receive input text and return the corresponding knowledge-enhanced text embedding vector. A similarity calculation interface for calculating and returning the semantic similarity between an image and text; The model update interface is used to receive new samples and trigger the online continuous learning module to perform incremental updates.

9. An execution method for an industrial multimodal cognitive base, characterized in that, The method is applied to the industrial cognitive foundation system based on multimodal contrastive learning as described in any one of claims 1 to 8, and the method includes: An industrial multimodal heterogeneous training dataset is constructed by synchronously collecting data from multiple sources in the industrial field, including text modality, visual modality, temporal modality, and structured knowledge modality, and aligning them to generate standardized sample triples. The sample triples include an image, a text description, and a set of candidate identifiers pointing to core entities in a preset target knowledge graph. The set of candidate identifiers is obtained from the aligned text description through named entity recognition and preliminary entity linking. An industrial-enhanced cross-modal encoder is constructed, including a visual encoder, a text encoder, and a knowledge injection module, which is used to extract visual features, text features, and knowledge features from the associated sub-graph and fuse them to generate knowledge-enhanced text features, which are then mapped to vector representations in a unified semantic space together with the visual features. Design a composite contrastive loss function for industrial semantic constraints, including a standard contrastive loss and an industrial semantic constraint loss calculated based on rule violation scores, whereby the rule violation scores are used to quantify the degree to which the model output violates industrial rules; An end-to-end contrastive learning training is performed, using the vector representation and composite contrastive loss function to optimize model parameters, outputting an optimized industrial cognitive foundation model. The composite loss function used in the contrastive learning training module includes standard contrastive loss and industrial semantic constraint loss. The industrial semantic constraint loss penalizes whether the vector representation violates predefined industrial logic rules based on a rule consistency discriminator. The industrial semantic constraint loss is calculated based on a rule violation score, and the formula for calculating the rule violation score includes: in, It is the rule violation score, a non-negative scalar value used to quantify the degree to which the i-th sample violates the industry rules; the larger the value, the more serious the violation. It is a visual feature vector, which comes from the visual encoder output of the cross-modal coding and fusion module, and represents the semantic embedding of the image in the i-th sample; is the knowledge-enhanced text feature vector, which comes from the knowledge injection module output of the cross-modal coding and fusion module, representing the semantic embedding of the text in the i-th sample after knowledge enhancement; K is the total number of predefined industrial logic rules, which comes from the rule base maintained in the knowledge graph management module; is the violation penalty weight of the k-th rule, a non-negative scalar parameter used to adjust the importance of the rule in the total violation score; It is the smooth violation strength function of the k-th rule, used to calculate the violation strength of the k-th rule at a given distance value, and its output value increases smoothly from 0; It is a distance function, defined as the square of the Euclidean distance, i.e. , used to calculate the difference between two vectors; It is the visual projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project visual features onto the semantic subspace of interest for the k-th rule; It is the text projection matrix corresponding to the k-th rule, which is a A matrix is ​​used to project knowledge-enhanced text features onto the same rule-based semantic subspace. This represents the dimension of the original feature vector. Indicates the dimension of the projected feature vector; Deploy an online continuous learning and model update mechanism to monitor the new sample data stream, detect semantic drift, and trigger incremental model updates when semantic drift exceeds a threshold.

Images