Supervision private knowledge base construction method and system based on multi-modal knowledge graph

Abstract

The invention discloses a supervision privatized knowledge base construction method and system based on a multi-modal knowledge graph, and the method comprises the steps: collecting images, videos, texts and sensor multi-modal data in a supervision scene, and carrying out the local ETL cleaning and time sequence alignment processing; a multi-mode embedding algorithm is adopted to convert data features, and a supervision domain entity is extracted in combination with an entity extraction algorithm; constructing an entity-attribute-relationship-containing supervision knowledge graph according to the entities and the associated information, and storing the entity-attribute-relationship-containing supervision knowledge graph into a private knowledge base; and designing a knowledge retrieval agent, receiving semantic query, calling a graph reasoning algorithm for operation, obtaining matched structured evidence from the private knowledge base, and returning the matched structured evidence in an interpretable path. According to the system, multi-modal knowledge structured integration is achieved through multi-step cooperation, semantic retrieval accuracy and interpretability are improved, data security is guaranteed in a private environment, supervision scene decision and agent reasoning are effectively supported, and the problems of data dispersion, low retrieval efficiency and the like in traditional knowledge management are solved.

Description

Technical Field

[0001] This invention relates to the field of gate knowledge base construction technology, and in particular to a method and system for constructing a private knowledge base for supervision based on multimodal knowledge graphs. Background Technology

[0002] In the context of deepening digital transformation, knowledge data generated across various fields exhibits significant multimodal characteristics, including images, videos, text, and sensor data. This data carries core business information and generally involves confidentiality requirements. Enterprises' needs for knowledge management have evolved from traditional storage to intelligent decision support. Most enterprises prioritize data security and controllability as the primary criterion for system selection, while significantly increasing their demands for knowledge integration efficiency and business adaptability. Under this trend, how to effectively integrate and intelligently apply multimodal knowledge in a private environment that ensures data security and compliance has become a key direction for knowledge management upgrades across various fields. There is an urgent need for knowledge management solutions that can adapt to the characteristics of multimodal data and meet the requirements of private deployment.

[0003] Existing technologies suffer from two major drawbacks. First, they lack the ability to integrate and structure multimodal knowledge. Various types of data are stored in a fragmented manner and lack a unified association mechanism, failing to form an effective structured knowledge system. Data processing often remains at a single-modal level, making it difficult to uncover the connections between different types of data, and hindering the efficient transfer and reuse of knowledge assets. Second, their semantic retrieval and reasoning capabilities are weak. The retrieval process relies heavily on surface-level word matching rather than deep semantic understanding, making it difficult to accurately respond to query needs in specialized scenarios. Furthermore, the retrieval results lack clear association paths and supporting evidence, failing to provide traceable evidence for subsequent decision-making or agent reasoning. They also struggle to meet the data security and efficient retrieval requirements of private environments. Summary of the Invention

[0004] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a method and system for constructing a private knowledge base for supervision based on multimodal knowledge graphs.

[0005] The technical solution utilized in this invention is a method for constructing a private knowledge base for supervision based on a multimodal knowledge graph, comprising the following steps: S1, collecting multimodal data in the supervision scenario, wherein the multimodal data includes image data, video data, text data, and sensor data; S2, performing local ETL operations on the collected multimodal data to remove redundant information and abnormal data, and performing time-series alignment processing on different types of multimodal data to ensure consistency of each type of data in the time dimension; S3, using a multimodal embedding algorithm to perform feature transformation on the time-series aligned multimodal data to obtain feature vectors corresponding to different modal data, and simultaneously using an entity extraction algorithm to extract features from text data and processed image and video data. S4. Extract entities related to the supervision domain; S5. Based on the extracted entities and the association information between them, construct a supervision knowledge graph including entities, attributes, and relationship structures, and store the constructed supervision knowledge graph in a privately deployed knowledge base, which has data isolation storage capabilities; S6. Design a knowledge retrieval agent, which receives semantic query requests input by the user and calls a preset graph reasoning algorithm to perform reasoning operations on the supervision knowledge graph; S7. Based on the reasoning operation results, the knowledge retrieval agent obtains structured evidence matching the semantic query request from the private knowledge base, and returns the structured evidence to the user interaction terminal in the form of an interpretable path according to preset interpretable path generation rules.

[0006] Furthermore, the multimodal embedding algorithm used in S3 satisfies the following model formula: V com =α·MLP(V img )+β·Transformer(V txt )+γ·LSTM(V sen ), where V com V represents the comprehensive feature vector after multimodal data fusion, where α, β, and γ represent the weight coefficients of the image data feature vector, text data feature vector, and sensor data feature vector in the fusion process, respectively. MLP(·) represents the multilayer perceptron function used to process the image data feature vector V. img Performing nonlinear transformations, Transformer(·) represents the Transformer model function, used to transform the text data feature vector V. txt Feature extraction and transformation are performed. LSTM(·) represents the Long Short-Term Memory network function, used to process the sensor data feature vector V. sen Perform time-series feature processing.

[0007] Furthermore, the entity extraction algorithm used in S3 satisfies the following model formula: Wherein, P(e i |D) represents extracting the i-th entity e from the data D. iThe probability, W e Represents entity e i The corresponding weight matrix, F D b represents the feature matrix obtained after preprocessing data D. e Represents entity e i The corresponding bias term, n represents the total number of entities that may exist in the data D, W j Let b represent the weight matrix corresponding to the j-th potential entity. j This represents the bias term corresponding to the j-th potential entity.

[0008] Furthermore, when constructing the supervision knowledge graph in S4, the calculation of the relationship weights between entities satisfies the following model formula: Among them, R ij S represents the relationship weight between the i-th entity and the j-th entity, m represents the total number of entity feature dimensions, and S represents the relationship weight between the i-th entity and the j-th entity. ik S represents the value of the i-th entity in the k-th feature dimension. jk T represents the value of the j-th entity in the k-th feature dimension. ij This represents the correlation coefficient between the i-th entity and the j-th entity in the historical interaction data.

[0009] Furthermore, the preset graph inference algorithm in S5 satisfies the following model formula: Where Inf(q) represents the inference result value for semantic query request q, p represents the total number of entity pairs involved in the inference process, R represents the set of all relation types in the knowledge graph, and Rel(h) represents the total number of relation types in the knowledge graph. t ,r,t t ) represents the head entity h in the t-th entity pair. t With tail entity t t Establish the confidence level of the association through relation r, Att(h) t ,t t ,q) represents the relevance weight of the t-th entity pair to the semantic query request q.

[0010] Furthermore, the interpretable path generation rule in S6 satisfies the following model formula: Where Path(q) represents the interpretable path generated for the semantic query request q, K represents the set of all candidate paths, and o represents the total number of structured evidences included in the k-th candidate path. This represents the l-th structured evidence in the k-th candidate path. Similarity to semantic query request q, This represents the l-th structured evidence in the k-th candidate path. Importance coefficient.

[0011] Further, step S3 includes the following sub-steps: S31, separating image data from the time-aligned multimodal data, extracting features from the image data, extracting local and global features of the image using convolution operations, combining the local and global features to form an initial image feature set, and then removing redundant features from the initial image feature set through a feature filtering mechanism to obtain the feature vector corresponding to the image data; S32, performing word segmentation and part-of-speech tagging on the time-aligned text data, removing meaningless characters and stop words from the text, converting the processed text into a word vector sequence, and using a pre-trained language model to process the word vector sequence. Deep feature mining generates feature vectors corresponding to text data; S33, the time-aligned sensor data is sorted according to the time sequence of data acquisition, and the sensor data is segmented using sliding window technology. The statistical and trend features of each segment are calculated, and the statistical and trend features are fused to obtain the feature vectors corresponding to the sensor data; S34, the feature vectors corresponding to image data, text data, and sensor data are integrated to form a unified feature representation of multimodal data. At the same time, the entities in the supervision field in different modal data are identified. Through entity matching and entity disambiguation operations, the finally extracted entities related to the supervision field are determined.

[0012] Further, S4 includes the following sub-steps: S41, defining attributes for the extracted entities related to the supervision domain, determining the attribute type and value range for each entity according to the supervision business requirements, establishing a mapping relationship between entities and attributes, and forming an entity-attribute correspondence table; S42, analyzing the interaction behavior and associated scenarios of each entity in the supervision scenario, determining the possible relationship types between entities, obtaining supporting evidence of relationships between entities from historical supervision data through association rule mining algorithms, and establishing an entity-relationship correspondence table; S43, based on the entity-attribute correspondence table and the entity-relationship correspondence table, using graph structure construction technology, using entities as nodes, entity attributes as node attributes, and relationships between entities as edges, constructing an initial supervision knowledge graph; S44, performing integrity checks and consistency verification on the initial supervision knowledge graph, supplementing missing entity attributes and entity relationships, correcting conflicting entity association information, obtaining the final supervision knowledge graph, and storing it in a privately deployed knowledge base according to a preset data storage format.

[0013] Further, S5 includes the following sub-steps: S51, designing an interactive interface module for the knowledge retrieval agent. This module receives semantic query requests input by the user through a terminal, performs format conversion and encoding processing on the semantic query requests, and converts them into query instructions that the knowledge retrieval agent can recognize; S52, constructing a graph reasoning module in the knowledge retrieval agent. This module loads a preset graph reasoning algorithm, obtains the structural data and node information of the supervision knowledge graph, parses the query instructions into reasoning tasks, and determines the entity nodes and relationship paths required for the reasoning process; S53, the graph reasoning module traverses and calculates the entity nodes and relationship paths in the supervision knowledge graph according to the requirements of the reasoning task, generates intermediate reasoning results, performs a rationality judgment on the intermediate reasoning results, and filters out illogical intermediate results; S54, integrating the filtered intermediate reasoning results to form reasoning operation results, and transmitting the reasoning operation results to the data matching module of the knowledge retrieval agent to provide a basis for subsequent structured evidence acquisition.

[0014] This system, based on a multimodal knowledge graph, constructs a private knowledge base for supervision. It utilizes a multimodal knowledge graph-based method for building such a knowledge base and includes: a multimodal data acquisition and processing unit, a supervision knowledge graph construction and storage unit, a semantic query receiving and parsing unit, a graph inference and computation unit, a structured evidence matching and extraction unit, and a retrieval result generation and feedback unit. The multimodal data acquisition and processing unit collects image, video, text, and sensor data from the supervision scenario. It performs local ETL operations and time-series alignment on the collected data and transmits the processed data to the supervision knowledge graph construction and storage unit. The supervision knowledge graph construction and storage unit receives the data from the multimodal data acquisition and processing unit, processes the data using multimodal embedding and entity extraction algorithms, constructs the supervision knowledge graph, stores it in a private storage module, and establishes a data interaction channel with the graph inference and computation unit, providing it with knowledge graph data. The semantic query receiving and parsing unit... Upon receiving a semantic query request from the user, the system encodes and parses the request, generates a query instruction, and sends it to the graph inference unit. The graph inference unit receives the query instruction from the semantic query receiving and parsing unit, invokes a preset inference algorithm, and performs inference operations using the knowledge graph data provided by the supervision knowledge graph construction and storage unit. The inference results are then transmitted to the structured evidence matching and extraction unit. The structured evidence matching and extraction unit receives the inference results from the graph inference unit, matches and extracts structured evidence from the private storage module of the supervision knowledge graph construction and storage unit based on the results, and transmits the structured evidence to the retrieval result generation and feedback unit. The retrieval result generation and feedback unit receives the structured evidence transmitted by the structured evidence matching and extraction unit, generates an interpretable path according to the interpretable path generation rules, and feeds back the retrieval results, including the interpretable path, to the user interaction terminal. Different units transmit data and collaborate using a preset communication protocol.

[0015] Beneficial Effects: This invention proposes a method and system for constructing a private knowledge base for supervision based on a multimodal knowledge graph. This system enables efficient integration and secure management of multimodal knowledge in supervision scenarios, improves the accuracy and interpretability of knowledge retrieval, provides reliable support for intelligent decision-making and reasoning, and meets data compliance requirements in a private environment. Addressing the issue of insufficient multimodal knowledge integration and structured processing capabilities, the system collects multimodal data covering various data types, performs local ETL cleaning and time-series alignment to ensure data quality and consistency, and then uses multimodal embedding and entity extraction to form structured entity information. Finally, it constructs a knowledge graph containing entities, attributes, and relationships and stores it privately, achieving effective association and structured integration of different types of data and promoting efficient knowledge flow and reuse. Addressing the weak semantic retrieval and reasoning support capabilities, the system's designed knowledge retrieval agent can receive semantic queries, call graph reasoning algorithms for computation, match and extract structured evidence from the private knowledge base, and return results in an interpretable path format, improving the semantic accuracy and traceability of the retrieval, while also adapting to the data security and efficient access requirements in a private environment. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method steps of the present invention;

[0017] Figure 2 This is a diagram showing the system unit composition of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 As shown, the method for constructing a private knowledge base for supervision based on a multimodal knowledge graph includes the following steps:

[0020] S1, collect multimodal data in the supervision scenario, including image data, video data, text data and sensor data;

[0021] Specifically, step S1 involves multimodal data acquisition in the supervision scenario. This requires coverage of four main data types: image data, video data, text data, and sensor data. Image data acquisition uses an industrial camera with a resolution of at least 3840×2160 pixels, a frame rate of 25 frames per second, and covers key areas such as equipment status and environmental conditions at the supervision site. A single continuous acquisition session should last at least 12 hours. Video data acquisition uses a camera supporting H.265 encoding, with a bitrate controlled between 8-12 Mbps. Recording time is synchronized with image acquisition to ensure data integrity in the time dimension. Text data acquisition includes supervision logs, contract documents, and inspection reports, supporting formats such as DOC, PDF, and TXT. It is obtained through a combination of document scanning and manual input, with an accuracy rate of at least 99.5%. Sensor data acquisition targets parameters such as temperature, humidity, vibration, and pressure, using industrial-grade sensors with a sampling frequency of 1Hz-10Hz. Data transmission latency is controlled within 50ms, and the acquisition cycle is consistent with that of image and video data. This step involves collaborative data collection using multiple devices to ensure the acquisition of comprehensive and high-quality supervision data. This provides a foundation for subsequent data processing and knowledge graph construction, preventing data loss or insufficient quality from affecting the effectiveness of later stages.

[0022] S2 performs local ETL operations on the collected multimodal data to remove redundant information and abnormal data, and performs time-series alignment processing on different types of multimodal data to ensure that the data of each type maintains consistency in the time dimension.

[0023] Specifically, step S2 performs local ETL operations and time-series alignment processing. The local ETL operation is divided into three stages: extraction, transformation, and loading. The extraction stage uses incremental extraction, acquiring only newly added and changed data, with an extraction frequency of once per hour to reduce redundant data transmission. The transformation stage removes redundant and abnormal data using data cleaning algorithms. Redundant information includes duplicate records and blank fields, while abnormal data is judged based on deviations from the normal range by three standard deviations. After cleaning, data integrity must remain above 98%. The loading stage stores the transformed data in a local temporary database using a partitioned storage strategy, partitioning by data type and acquisition time to improve subsequent data access efficiency. Time-series alignment processing uses the sensor data acquisition time as the baseline timestamp. Image and video data are matched with the baseline timestamp by extracting the shooting time information, with time deviation controlled within 1 second. Text data is aligned with the baseline timestamp through document creation time or time information within the content. If the text lacks explicit time information, the data acquisition time is used as the alignment basis. This step eliminates data format differences and time deviations through standardized data processing, providing a unified data foundation for multimodal data fusion and feature extraction, and ensuring the accuracy of the results of subsequent processing steps.

[0024] S3 utilizes a multimodal embedding algorithm to perform feature transformation on the time-aligned multimodal data, obtaining feature vectors corresponding to different modal data. At the same time, an entity extraction algorithm is used to extract entities related to the supervision field from text data and processed image and video data.

[0025] Specifically, step S3 implements multimodal embedding and entity extraction. During multimodal embedding, differentiated processing methods are adopted for different data types. Image data is first resized to 224×224 pixels, and then a feature extraction network is used to extract a 1024-dimensional feature vector. The feature extraction network iteration is set to 50 rounds, the initial learning rate is 0.001, and it decays to 0.5 times the previous round every 10 rounds. Text data is first segmented and stop word removed, and then converted into a 768-dimensional feature vector through a pre-trained language model. The batch size is set to 32 during model training, and the training iteration cycle is 30 rounds. Sensor data is converted into a 512-dimensional feature vector through a temporal feature extraction method. The sliding window size is set to 60 sampling points, and the step size is 10 sampling points. For text data, entity extraction employs named entity recognition (NAME) technology to extract entities from the supervision domain, including equipment names, inspection indicators, and personnel information, with an entity recognition accuracy rate exceeding 92%. For image and video data, object detection algorithms are first used to identify entities in the images, followed by optical character recognition (OCR) technology to extract relevant text information and determine entity categories and attributes, with an object detection recall rate of no less than 88%. This step transforms unstructured data into structured features and entity information through multimodal feature conversion and entity extraction, providing core data elements for knowledge graph construction and supporting subsequent entity association and relationship establishment.

[0026] S4. Based on the extracted entities and the relationship information between them, construct a supervision knowledge graph including entities, attributes, and relationship structures, and store the constructed supervision knowledge graph in a privately deployed knowledge base, which has data isolation storage function.

[0027] Specifically, step S4 involves constructing and storing the supervision knowledge graph. In the entity-attribute-relationship structure design, entity types are divided into eight categories, including equipment, personnel, documents, and inspection. Each entity category includes 10-15 attributes, such as model, manufacturing date, and maintenance cycle. Attribute values ​​must conform to supervision industry standards. Entity relationships are divided into six types, including association, dependency, and inspection relationships. Relationship definitions are based on supervision business processes and data association rules; for example, "inspection personnel—execution—inspection task" is a typical association relationship. The knowledge graph is constructed using a graph database for storage. Nodes store entity information, and edges store relationships between entities. The initial number of nodes is no less than 1000, and the number of edges is no less than 2000. The database supports at least 500 query operations per second. The private storage deployment employs a local server cluster architecture, consisting of at least three application servers and at least two database servers. The servers utilize RAID 5 disk arrays for data storage. The data backup strategy includes daily full backups and hourly incremental backups, with a 30-day data retention period. Access control lists are used to set different access permissions for different roles, ensuring that only authorized personnel can perform data read and write operations. This step, through the construction and secure storage of a structured knowledge graph, achieves systematic management of supervisory knowledge, ensuring data privacy and security, and providing reliable data support for knowledge retrieval and reasoning.

[0028] S5, Design a knowledge retrieval agent. The knowledge retrieval agent receives the semantic query request input by the user and calls the preset graph reasoning algorithm to perform reasoning operations on the supervision knowledge graph.

[0029] Specifically, step S5 involves designing a knowledge retrieval agent and performing inference operations. The knowledge retrieval agent includes an interaction module, an inference module, and a data interface module. The interaction module supports natural language input, with the input text length limited to 10-500 characters, and the text parsing response time not exceeding 1 second. The inference module loads a graph inference algorithm, which is based on the entity and relationship information of the knowledge graph. It obtains associated data through relational path traversal and logical inference, with the inference depth set to 3-5 layers. That is, starting from the query entity, it traverses 3-5 layers of associated entities and relationships. During the inference process, the confidence score of each path is calculated, and the confidence score threshold is set to 0.7. Paths below the threshold are considered invalid paths. The data interface module adopts a RESTful API design to establish a connection with the private knowledge base. Data transmission uses the HTTPS encryption protocol, and the interface call response time is controlled within 3 seconds. Upon receiving a semantic query request, the interaction module first performs word segmentation and intent recognition on the query text to determine the core query requirement, such as "query the detection data of a certain device for the past week." Based on the intent recognition results, the inference module determines the starting entity and target relationship for inference, calls a graph inference algorithm to traverse the knowledge graph, calculates the confidence score of each path, and filters out valid inference paths. Finally, the entity and relationship information corresponding to the inference paths are integrated to form the inference result, providing direction for subsequent evidence acquisition. This step, through agent design and inference operations, transforms fuzzy semantic queries into precise data retrieval requirements, improving the targeting and efficiency of knowledge retrieval.

[0030] S6: Based on the reasoning results, the knowledge retrieval agent obtains structured evidence matching the semantic query request from the private knowledge base, and returns the structured evidence to the user interaction terminal in the form of an interpretable path according to the preset interpretable path generation rules.

[0031] Specifically, in step S6, structured evidence is acquired and an interpretable path is generated. When acquiring structured evidence, the knowledge retrieval agent matches entity attribute data, relationship data, and associated documents related to the query requirements from the private knowledge base based on the reasoning results. The evidence selection criteria are the degree of matching with the reasoning path. The matching degree is calculated based on entity similarity and relationship relevance, and the matching degree threshold is set to 0.85. Only evidence data with a matching degree higher than the threshold is retained. The acquired structured evidence must include information such as entity name, attribute value, relationship, and data source. The completeness of the evidence data must reach more than 95% to avoid missing key information. The explainable path generation adopts a hierarchical display approach. Starting from the query originating entity, it sequentially displays related entities, relationship types, and evidence data according to the reasoning path. The path display hierarchy is consistent with the reasoning depth, ranging from 3 to 5 levels. Each node in the path is labeled with key entity attribute information, and each edge is labeled with the relationship definition and confidence level. A link to the source of the evidence data is also attached at the end of the path, allowing users to view the complete evidence document. After path generation, it is formatted and presented using a combination of tables and text. The tables display entity and relationship information, while the text describes the reasoning logic, ensuring the path is clear and easy to understand. This step, through accurate evidence matching and visualized path generation, presents the search results in a traceable and easily understandable form, providing clear evidence for user decision-making while ensuring the reliability and usability of the search results.

[0032] Preferably, the multimodal embedding algorithm used in S3 satisfies the following model formula: V com =α·MLP(V img )+β·Transformer(V txt )+γ·LSTM(V sen ), where V com V represents the comprehensive feature vector after multimodal data fusion, where α, β, and γ represent the weight coefficients of the image data feature vector, text data feature vector, and sensor data feature vector in the fusion process, respectively. MLP(·) represents the multilayer perceptron function used to process the image data feature vector V. img Performing nonlinear transformations, Transformer(·) represents the Transformer model function, used to transform the text data feature vector V. txt Feature extraction and transformation are performed. LSTM(·) represents the Long Short-Term Memory network function, used to process the sensor data feature vector V. sen Perform time-series feature processing.

[0033] Specifically, the multimodal embedding algorithm in step S3 is used to convert different types of multimodal supervision data into a unified-dimensional comprehensive feature vector to achieve effective fusion of multimodal data. The weight coefficients for image data feature vectors range from 0.3 to 0.4, text data feature vectors range from 0.35 to 0.45, and sensor data feature vectors range from 0.2 to 0.3. The sum of these three weight coefficients is 1. This weight allocation allows adjustment of the contribution of each modality in the fusion process based on the differences in the importance of the supervision data. The multilayer perceptron function is used to perform nonlinear transformations on image data feature vectors. In implementation, three hidden layers are set, with 512, 256, and 1024 neurons per layer, respectively. An activation function is used to deeply extract image features, enhancing the representation of key information in the image. The Transformer model function, when processing text data feature vectors, employs a six-layer encoder structure with eight attention heads, capturing semantic relationships between entities in the text through a self-attention mechanism. The Long Short-Term Memory (LSTM) network function, when processing sensor data feature vectors, uses two hidden layers with 256 neurons. A gating mechanism effectively preserves the temporal characteristics of the sensor data, avoiding long-term dependency issues. In the algorithm implementation, feature transformations are first performed on each modality of data separately, then linearly fused according to weight coefficients to obtain a comprehensive feature vector. This provides high-quality feature support for subsequent entity extraction, improving the accuracy of entity recognition.

[0034] Preferably, the entity extraction algorithm used in S3 satisfies the following model formula: Wherein, P(e i |D) represents extracting the i-th entity e from the data D. i The probability, W e Represents entity e i The corresponding weight matrix, F D b represents the feature matrix obtained after preprocessing data D. e Represents entity e i The corresponding bias term, n represents the total number of entities that may exist in the data D, W j Let b represent the weight matrix corresponding to the j-th potential entity. j This represents the bias term corresponding to the j-th potential entity.

[0035] Specifically, the entity extraction algorithm in step S3 is used to accurately extract domain-related entities from the multimodal data of the supervision system. Its core is to calculate the probability of extracting a specific entity from the data to filter out entities with high confidence. Regarding parameter settings, the dimension of the entity weight matrix is ​​determined based on the dimension of the data feature matrix. If the data feature matrix is ​​m rows and n columns (m is the number of samples, n is the feature dimension), then the entity weight matrix is ​​n rows and k columns (k is the number of entity categories). The initial values ​​of the weight matrix are generated through a random normal distribution with a mean of 0 and a standard deviation of 0.01. The bias term is a k-dimensional vector with an initial value of 0.1. During model training, the weight matrix and bias term are continuously adjusted using the gradient descent algorithm to minimize the error between the predicted probability and the true label. When constructing the feature matrix after data preprocessing, the features of text, image, and sensor data need to be standardized to ensure that the values ​​of each feature dimension are uniformly within the [0,1] interval, eliminating the influence of dimensional differences on the entity extraction results. The total number of entities is determined based on the business needs of the supervision field, including eight major categories of entities such as equipment, personnel, documents, and testing indicators. Each category includes several subcategories, with the total number of entities controlled between 50 and 80. During algorithm implementation, the preprocessed feature matrix is ​​first input into the model. The raw score is calculated through matrix multiplication and bias term addition. Then, the raw score is converted into a probability value using the Softmax function. The entity with the highest probability value exceeding a preset threshold (usually 0.5) is selected as the extraction result. This process ensures accurate identification of key entities in the supervision field from multimodal data, providing core nodes for knowledge graph construction.

[0036] Preferably, when constructing the supervision knowledge graph in S4, the calculation of the relationship weights between entities satisfies the following model formula: Among them, R ij S represents the relationship weight between the i-th entity and the j-th entity, m represents the total number of entity feature dimensions, and S represents the relationship weight between the i-th entity and the j-th entity. ik S represents the value of the i-th entity in the k-th feature dimension. jk T represents the value of the j-th entity in the k-th feature dimension. ij This represents the correlation coefficient between the i-th entity and the j-th entity in the historical interaction data.

[0037] Specifically, step S4 involves calculating the relational weights between entities in the supervisory knowledge graph. This calculation quantifies the strength of associations between entities, providing a basis for constructing knowledge graph edges and ensuring the rationality and accuracy of the graph structure. Regarding parameters, the total number of entity feature dimensions is determined based on the number of entity attributes. Each entity includes 10-15 attributes, so the total number of feature dimensions is set to 15, including basic attributes, state attributes, and association attributes. The values ​​of entities on each feature dimension are standardized, converting the original attribute values ​​into values ​​within the [0,1] range to avoid affecting the relational weight calculation results due to differences in the magnitude of attribute values. The association coefficient between entities in historical interaction data is determined based on the frequency and depth of interaction within the past 6 months. Higher interaction frequency and greater depth result in a larger association coefficient, ranging from 0.1 to 0.9. If two entities have no historical interaction records, the association coefficient is set to 0.05. The algorithm first calculates the sum of the products of the values ​​of the two entities across each feature dimension, then calculates the magnitude of the feature vectors of the two entities, and uses the cosine similarity formula to calculate the basic correlation between the entities. Finally, it multiplies the basic correlation by the correlation coefficient to obtain the final relationship weight. The relationship weight ranges from 0 to 1. Entity pairs with a weight value greater than 0.5 are considered to have a strong relationship and require edge connections to be built in the knowledge graph. Entity pairs with a weight value less than 0.2 are considered to have a weak relationship and do not require edge connections to be built temporarily. This process optimizes the knowledge graph structure by quantifying the relationship strength, thereby improving the efficiency of subsequent retrieval and reasoning.

[0038] Preferably, the preset graph inference algorithm in S5 satisfies the following model formula: Where Inf(q) represents the inference result value for semantic query request q, p represents the total number of entity pairs involved in the inference process, R represents the set of all relation types in the knowledge graph, and Rel(h) represents the total number of relation types in the knowledge graph. t ,r,t t ) represents the head entity h in the t-th entity pair. t With tail entity t t Establish the confidence level of the association through relation r, Att(h) t ,t t ,q) represents the relevance weight of the t-th entity pair to the semantic query request q.

[0039] Specifically, the graph reasoning algorithm in step S5 is used to deduce related information from the supervision knowledge graph based on the semantic query request, generate reasoning results, and provide direction for obtaining structured evidence. Regarding parameter settings, the total number of entity pairs involved in the reasoning process is determined according to the complexity of the query request. Simple queries (such as single entity attribute queries) involve 5-10 entity pairs, while complex queries (such as multi-entity association queries) involve 15-20 entity pairs. The set of all relationship types in the knowledge graph includes six types: association, dependency, detection, etc., each with a unique identifier and confidence benchmark value. The confidence level for establishing a relationship between the head and tail entities in an entity pair is determined based on the relationship type, entity attribute matching degree, and historical data verification results. The confidence level ranges from 0 to 1. If the relationship type is a core business relationship (such as a detection relationship), the confidence level benchmark is set to 0.6; if it is a non-core business relationship, the confidence level benchmark is set to 0.4. The relevance weight between the entity pair and the semantic query request is determined based on the degree of matching between the entities in the entity pair and the query keywords. The relevance weight for a complete match is 1.0, the relevance weight for a partial match is 0.5-0.8, and the relevance weight for a no-match is 0.1. During algorithm implementation, the entity pairs related to the query request in the knowledge graph are traversed first. The association confidence and relevance weight of each entity pair are calculated, and the two are multiplied to obtain the contribution value of the entity pair. Then, the contribution values ​​of all entity pairs are multiplied to obtain the final inference result value. Inference result values ​​exceeding 0.6 are considered valid inference results and used for subsequent evidence matching. This process improves the accuracy and reliability of the inference results by quantifying the inference basis in multiple dimensions.

[0040] Preferably, the interpretable path generation rule in S6 satisfies the following model formula: Where Path(q) represents the interpretable path generated for the semantic query request q, K represents the set of all candidate paths, and o represents the total number of structured evidences included in the k-th candidate path. This represents the l-th structured evidence in the k-th candidate path. Similarity to semantic query request q, This represents the l-th structured evidence in the k-th candidate path. Importance coefficient.

[0041] Specifically, the interpretable path generation rule in step S6 is used to filter out the path that best matches the semantic query request from the candidate paths, presenting the search results in a visual way to improve the understandability of the results. Regarding parameters, the set of candidate paths is generated based on the inference operation results. Each candidate path includes 3-5 entity nodes and corresponding relation edges, with the number of candidate paths controlled at 10-15 to avoid excessive paths leading to low filtering efficiency. The total number of structured evidences included in each candidate path is determined based on the path length: a 3-layer path includes 5-8 pieces of evidence, and a 5-layer path includes 10-12 pieces of evidence. Evidence includes entity attribute data, relation data, document data, etc. The similarity between structured evidence and semantic query requests is calculated using a text similarity algorithm. If the evidence contains content completely identical to the query keywords, the similarity value is 0.9-1.0; if it contains partially identical content, the similarity value is 0.6-0.8; and if it contains no identical content, the similarity value is 0.1-0.3. The importance coefficient of the evidence is determined based on the evidence type: core evidence (such as test report data) has an importance coefficient of 0.8-0.9, and auxiliary evidence (such as basic equipment information) has an importance coefficient of 0.4-0.6. During algorithm implementation, the sum of the products of the similarity and importance coefficients of all evidence in each candidate path is first calculated to obtain the comprehensive score for that path. Then, the path with the highest comprehensive score is selected as the interpretable path. If multiple paths have similar scores (difference less than 0.05), the shorter path is selected as the final result based on the path length priority principle. This process ensures that the generated interpretable paths not only meet the query requirements but are also concise and easy to understand, providing a clear basis for user decision-making.

[0042] Preferably, step S3 includes the following sub-steps: S31, separating image data from the time-aligned multimodal data, extracting features from the image data, extracting local and global features of the image using convolution operations, combining the local and global features to form an initial feature set of the image, and then removing redundant features from the initial feature set of the image through a feature filtering mechanism to obtain the feature vector corresponding to the image data; S32, performing word segmentation and part-of-speech tagging on the time-aligned text data, removing meaningless characters and stop words from the text, converting the processed text into a word vector sequence, and using a pre-trained language model to perform deep processing on the word vector sequence. S33: For time-aligned sensor data, sort the data according to the time sequence of data acquisition, use sliding window technology to segment the sensor data, calculate the statistical and trend features of each segment, and fuse the statistical and trend features to obtain the feature vector corresponding to the sensor data; S34: Integrate the feature vectors corresponding to image data, text data, and sensor data to form a unified feature representation of multimodal data, and identify the entities in the supervision field in different modal data. Through entity matching and entity disambiguation operations, determine the finally extracted entities related to the supervision field.

[0043] Specifically, step S3 involves multimodal data processing and entity extraction, with each step working together to achieve multimodal feature transformation and accurate extraction of supervisory entities. In S31, when processing image data, a 3×3 convolution kernel is first used for convolution, with a stride of 1 and padding mode set to Same, extracting 64 local feature maps and 256 global feature maps. Then, a non-maximum suppression algorithm is used to filter redundant features, with a filtering threshold set to 0.6, ultimately yielding an image feature vector of dimension 1024. This process enhances the feature representation of key supervisory targets in the image. In S32, when processing text data, a word segmentation tool is used to split the text by semantic units, removing high-frequency meaningless words (word frequency threshold set to 500). The text is then converted into a 300-dimensional word vector sequence using the Word2Vec model, and then input into a BERT pre-trained model (12 layers, 768 hidden layer dimensions) to mine deep features, generating a 768-dimensional text feature vector. Effectively captures semantic relationships in text; when processing sensor data, S33 sets the sliding window size to 60 seconds (corresponding to 60 sampling points at a sampling frequency of 1Hz) and the step size to 10 seconds. It calculates four statistical features (mean, variance, maximum, minimum) and two trend features (linear trend and periodicity) within the window, and merges them into a 512-dimensional sensor feature vector, preserving the temporal characteristics of the data; when integrating multimodal feature vectors, S34 uses feature concatenation to form a 2304-dimensional unified feature representation. In the entity recognition stage, entities are identified using a CRF model (50 iterations, learning rate 0.001), and then entity matching and disambiguation are performed using Jaccard similarity (threshold 0.8) to ensure the uniqueness and accuracy of extracted entities, providing high-quality entity data for knowledge graph construction.

[0044] Preferably, step S4 includes the following sub-steps: S41, defining attributes for the extracted entities related to the supervision field, determining the attribute type and value range for each entity according to the supervision business requirements, establishing a mapping relationship between entities and attributes, and forming an entity-attribute correspondence table; S42, analyzing the interaction behavior and associated scenarios of each entity in the supervision scenario, determining the possible relationship types between entities, obtaining supporting evidence of relationships between entities from historical supervision data through association rule mining algorithms, and establishing an entity-relationship correspondence table; S43, based on the entity-attribute correspondence table and the entity-relationship correspondence table, using graph structure construction technology, using entities as nodes, entity attributes as node attributes, and relationships between entities as edges, constructing an initial supervision knowledge graph; S44, performing integrity checks and consistency verification on the initial supervision knowledge graph, supplementing missing entity attributes and entity relationships, correcting conflicting entity association information, obtaining the final supervision knowledge graph, and storing it in a privately deployed knowledge base according to a preset data storage format.

[0045] Specifically, step S4, the construction and storage of the supervision knowledge graph, is divided into steps. A structured process is used to achieve the complete construction and secure storage of the knowledge graph. In S41, when defining entity attributes, 12 attributes such as model, manufacturing date, and rated parameters are set for equipment entities according to supervision industry standards. Ten attributes such as position, qualification level, and years of service are set for personnel entities. The attribute value range is calibrated using industry standard documents (e.g., the allowable error range for equipment rated parameters is ±5%). A mapping table between entities and attributes is established to ensure the standardization of attribute definitions. In S42, when analyzing entity relationships, historical supervision data is mined using the Apriori association rule algorithm (minimum support 0.2, minimum confidence 0.6) to identify six core relationships, such as "equipment—inspection—indicator" and "personnel—responsible—project." Three to five historical data points are matched for each relationship as supporting evidence, forming an entity-relationship correspondence table to ensure the relationships are secure. The rationality of the definition; S43 When constructing the initial graph, the Neo4j graph database is used, with entities as nodes (node ​​labels set to entity categories), entity attributes as node attributes (data types set to strings or numbers), and entity relationships as edges (edge ​​types set to relationship categories). The initial number of nodes is no less than 1000 and the number of edges is no less than 2000, realizing structured knowledge storage; S44 When optimizing the graph, a graph integrity detection tool (coverage threshold 0.95) is used to supplement the missing 3%-5% of entity attributes and relationships, and a consistency check algorithm (conflict rate threshold 0.02) is used to correct conflicting data. The final graph is stored in a private database in CSV format (storage partitions are divided according to entity categories) to ensure graph quality and data security.

[0046] Preferably, step S5 includes the following sub-steps: S51, designing an interactive interface module for the knowledge retrieval agent. This module receives semantic query requests input by the user through a terminal, performs format conversion and encoding processing on the semantic query requests, and converts them into query instructions that the knowledge retrieval agent can recognize; S52, constructing a graph reasoning module in the knowledge retrieval agent. This module loads a preset graph reasoning algorithm, obtains the structural data and node information of the supervision knowledge graph, parses the query instructions into reasoning tasks, and determines the entity nodes and relationship paths required for the reasoning process; S53, the graph reasoning module traverses and calculates the entity nodes and relationship paths in the supervision knowledge graph according to the requirements of the reasoning task, generates intermediate reasoning results, performs a rationality judgment on the intermediate reasoning results, and filters out illogical intermediate results; S54, integrating the filtered intermediate reasoning results to form a reasoning operation result, and transmitting the reasoning operation result to the data matching module of the knowledge retrieval agent to provide a basis for subsequent structured evidence acquisition.

[0047] Specifically, step S5 involves the design and reasoning of the knowledge retrieval agent, which is divided into steps. Precise semantic query reasoning is achieved through modular design. In S51, when designing the interaction interface module, a RESTful API architecture is adopted, supporting JSON-formatted semantic query requests. The input text length is limited to 10-500 characters. Format conversion is achieved through the FastAPI framework (response time ≤ 1 second), and the query request is encoded into a UTF-8 format query command to ensure interface compatibility and processing efficiency. In S52, when building the graph reasoning module, a rule-based reasoning algorithm is loaded (the rule base includes 50-80 reasoning rules for the supervision domain). Node and edge data of the knowledge graph are obtained through the graph data interface (data transmission is encrypted using HTTPS). The query command is parsed into a reasoning task of "entity + relation + target," and the core entity nodes (1-3) and relationships required for reasoning are determined. The system identifies 2-4 relationship paths to clarify the reasoning direction. During S53, a depth-first traversal algorithm is used to traverse the relationship paths (traversing 3-5 layers). At each layer, the reasoning confidence is calculated (based on relationship weights and rule matching). An illogical intermediate result is filtered out using a confidence threshold (0.7) (filtering ratio approximately 10%-15%), retaining only valid intermediate results. During S54, when integrating the reasoning results, a result aggregation algorithm (grouped by entity category) is used to merge valid intermediate results into a structured reasoning result (including reasoning paths, confidence, and associated entities). This result is then transmitted to the data matching module via an internal data interface (transmission delay ≤200ms), providing precise guidance for subsequent evidence acquisition and improving retrieval accuracy.

[0048] like Figure 2As shown, a system for constructing a private knowledge base for supervision based on a multimodal knowledge graph is described. This system is applied to a method for constructing a private knowledge base for supervision based on a multimodal knowledge graph. It includes: a multimodal data acquisition and processing unit, a supervision knowledge graph construction and storage unit, a semantic query receiving and parsing unit, a graph inference and operation unit, a structured evidence matching and extraction unit, and a retrieval result generation and feedback unit. The multimodal data acquisition and processing unit collects image data, video data, text data, and sensor data from the supervision scenario. It performs local ETL operations and time-series alignment on the collected data and transmits the processed data to the supervision knowledge graph construction and storage unit. The supervision knowledge graph construction and storage unit receives the data transmitted by the multimodal data acquisition and processing unit, processes the data using multimodal embedding algorithms and entity extraction algorithms, constructs the supervision knowledge graph, stores the supervision knowledge graph in a private storage module, and simultaneously establishes a data interaction channel with the graph inference and operation unit, providing it with knowledge graph data. The semantic query receiving and parsing unit... The system receives semantic query requests from users, encodes and parses these requests, generates query instructions, and sends the instructions to the graph inference unit. The graph inference unit receives the query instructions from the semantic query receiving and parsing unit, calls a preset inference algorithm, and performs inference operations using the knowledge graph data provided by the supervision knowledge graph construction and storage unit. The inference results are then transmitted to the structured evidence matching and extraction unit. The structured evidence matching and extraction unit receives the inference results from the graph inference unit, matches and extracts structured evidence from the private storage module of the supervision knowledge graph construction and storage unit based on the results, and transmits the structured evidence to the retrieval result generation and feedback unit. The retrieval result generation and feedback unit receives the structured evidence transmitted by the structured evidence matching and extraction unit, generates interpretable paths according to interpretable path generation rules, and feeds back the retrieval results, including the interpretable paths, to the user interaction terminal. Different units transmit data and collaborate using a preset communication protocol.

[0049] The method and system for constructing a private knowledge base for supervision based on multimodal knowledge graphs can comprehensively collect and deeply process multimodal data in supervision scenarios. By constructing a knowledge graph containing entities, attributes, and relational structures, it achieves structured integration and orderly management of multimodal knowledge, breaking down data fragmentation barriers. At the same time, relying on knowledge retrieval intelligent agents and graph reasoning mechanisms, it significantly improves the semantic accuracy of knowledge retrieval and returns results with interpretable paths, providing clear basis for decision-making and reasoning. In addition, the private deployment mode can strictly guarantee data security, meet compliance requirements, and fully adapt to the knowledge management needs of the supervision field.

[0050] This invention addresses the problem of insufficient multimodal knowledge integration and structured processing capabilities. The system collects multimodal data covering image, video, text, and sensor data, removes redundant and abnormal data through local ETL cleaning, and ensures data consistency over time through time alignment. It then obtains structured entity information through multimodal embedding and entity extraction, ultimately constructing a knowledge graph and storing it privately, achieving data association, integration, and efficient flow and reuse. Addressing the weakness in semantic retrieval and reasoning support, the system's knowledge retrieval agent can receive semantic queries, invoke graph reasoning algorithms, and extract structured evidence from the private knowledge base, presenting results with interpretable paths to improve retrieval accuracy and traceability, adapting to the data security and access requirements of private environments.

[0051] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for constructing a private knowledge base for supervision based on multimodal knowledge graphs, characterized in that, Includes the following steps: S1. Collect multimodal data in the supervision scenario, including image data, video data, text data, and sensor data; S2. Perform local ETL operations on the collected multimodal data to remove redundant and abnormal information, and perform time-series alignment processing on different types of multimodal data to ensure consistency across time dimensions; S3. Use a multimodal embedding algorithm to perform feature transformation on the time-series aligned multimodal data to obtain feature vectors corresponding to different modalities, and simultaneously use an entity extraction algorithm to extract supervision-related entities from the text data and processed image and video data; S4. Based on the extracted entities... S5. Construct a supervision knowledge graph including entities, attributes, and relationship structures, and store the constructed supervision knowledge graph in a privately deployed knowledge base with data isolation storage capabilities; S6. Design a knowledge retrieval agent that receives semantic query requests input by the user and calls a preset graph reasoning algorithm to perform reasoning operations on the supervision knowledge graph; S7. Based on the reasoning operation results, the knowledge retrieval agent obtains structured evidence matching the semantic query request from the private knowledge base and returns the structured evidence to the user interaction terminal in the form of an interpretable path according to preset interpretable path generation rules.

2. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, The multimodal embedding algorithm used in S3 satisfies the following model formula: V com =α·MLP(V img )+β·Transformer(V txt )+γ·LSTM(V sen ), where V com V represents the comprehensive feature vector after multimodal data fusion, where α, β, and γ represent the weight coefficients of the image data feature vector, text data feature vector, and sensor data feature vector in the fusion process, respectively. MLP(·) represents the multilayer perceptron function used to process the image data feature vector V. img Performing nonlinear transformations, Transformer(·) represents the Transformer model function, used to transform the text data feature vector V. txt Feature extraction and transformation are performed. LSTM(·) represents the Long Short-Term Memory network function, used to process the sensor data feature vector V. sen Perform time-series feature processing.

3. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, The entity extraction algorithm used in S3 satisfies the following model formula: Wherein, P(e i |D) represents extracting the i-th entity e from the data D. i The probability, W e Represents entity e i The corresponding weight matrix, F D b represents the feature matrix obtained after preprocessing data D. e Represents entity e i The corresponding bias term, n represents the total number of entities that may exist in the data D, W j Let b represent the weight matrix corresponding to the j-th potential entity. j This represents the bias term corresponding to the j-th potential entity.

4. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, When constructing the supervision knowledge graph in S4, the calculation of the relationship weights between entities satisfies the following model formula: Among them, R ij S represents the relationship weight between the i-th entity and the j-th entity, m represents the total number of entity feature dimensions, and S represents the relationship weight between the i-th entity and the j-th entity. ik S represents the value of the i-th entity in the k-th feature dimension. jk T represents the value of the j-th entity in the k-th feature dimension. ij This represents the correlation coefficient between the i-th entity and the j-th entity in the historical interaction data.

5. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, The preset graph inference algorithm in S5 satisfies the following model formula: Where Inf(q) represents the inference result value for semantic query request q, p represents the total number of entity pairs involved in the inference process, R represents the set of all relation types in the knowledge graph, and Rel(h) represents the total number of relation types in the knowledge graph. t ,r,t t ) represents the head entity h in the t-th entity pair. t With tail entity t t Establish the confidence level of the association through relation r, Att(h) t ,t t ,q) represents the relevance weight of the t-th entity pair to the semantic query request q.

6. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, The interpretable path generation rule in S6 satisfies the following model formula: Where Path(q) represents the interpretable path generated for the semantic query request q, K represents the set of all candidate paths, and o represents the total number of structured evidences included in the k-th candidate path. This represents the l-th structured evidence in the k-th candidate path. Similarity to semantic query request q, This represents the l-th structured evidence in the k-th candidate path. Importance coefficient.

7. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, S3 includes the following steps: S31, separating image data from the time-aligned multimodal data, extracting features from the image data, using convolution operations to extract local and global features of the image, combining the local and global features to form an initial image feature set, and then removing redundant features from the initial image feature set through a feature filtering mechanism to obtain the feature vector corresponding to the image data; S32, performing word segmentation and part-of-speech tagging on the time-aligned text data, removing meaningless characters and stop words from the text, converting the processed text into a word vector sequence, and using a pre-trained language model to perform deep feature extraction on the word vector sequence. S33: Generating feature vectors corresponding to text data; S44: Sort the time-aligned sensor data according to the time sequence of data acquisition, segment the sensor data using sliding window technology, calculate the statistical and trend features of each segment, and fuse the statistical and trend features to obtain the feature vectors corresponding to the sensor data; S55: Integrate the feature vectors corresponding to image data, text data, and sensor data to form a unified feature representation of multimodal data, and identify the entities in the supervision field in different modal data. Through entity matching and entity disambiguation operations, determine the finally extracted entities related to the supervision field.

8. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, The S4 includes the following sub-steps: S41, defining attributes for the extracted entities related to the supervision field, determining the attribute type and attribute value range for each entity according to the supervision business requirements, establishing a mapping relationship between entities and attributes, and forming an entity-attribute correspondence table; S42, Analyze the interaction behavior and associated scenarios of each entity in the supervision scenario, determine the possible relationship types between entities, and obtain supporting evidence of the relationships between entities from historical supervision data through association rule mining algorithms to establish an entity-relationship correspondence table; S43, Based on the entity-attribute correspondence table and the entity-relationship correspondence table, use graph structure construction technology to construct an initial supervision knowledge graph, with entities as nodes, entity attributes as node attributes, and relationships between entities as edges; S44, Perform integrity checks and consistency verification on the initial supervision knowledge graph, supplement missing entity attributes and entity relationships, correct conflicting entity association information, obtain the final supervision knowledge graph, and store it in a privately deployed knowledge base according to a preset data storage format.

9. The method for constructing a private knowledge base for supervision based on a multimodal knowledge graph according to claim 1, characterized in that, S5 includes the following sub-steps: S51, designing an interactive interface module for the knowledge retrieval agent. This module receives semantic query requests input by the user through the terminal, performs format conversion and encoding processing on the semantic query requests, and converts them into query instructions that the knowledge retrieval agent can recognize; S52, constructing a graph reasoning module in the knowledge retrieval agent. This module loads a preset graph reasoning algorithm, obtains the structural data and node information of the supervision knowledge graph, parses the query instructions into reasoning tasks, and determines the entity nodes and relationship paths required for the reasoning process. S53, the graph reasoning module traverses and calculates the entity nodes and relationship paths in the supervision knowledge graph according to the reasoning task requirements, generates intermediate reasoning results, judges the rationality of the intermediate reasoning results, and filters out intermediate results that do not conform to logic; S54, the filtered intermediate reasoning results are integrated to form the reasoning operation results, and the reasoning operation results are transmitted to the data matching module of the knowledge retrieval agent to provide a basis for subsequent structured evidence acquisition.

10. A system for constructing a private knowledge base for supervision based on a multimodal knowledge graph, characterized in that: This system is applied to the method for constructing a private knowledge base for supervision based on a multimodal knowledge graph as described in claim 1, comprising: a multimodal data acquisition and processing unit, a supervision knowledge graph construction and storage unit, a semantic query receiving and parsing unit, a graph inference and operation unit, a structured evidence matching and extraction unit, and a retrieval result generation and feedback unit; the multimodal data acquisition and processing unit is used to acquire image data, video data, text data, and sensor data in the supervision scenario, perform local ETL operations and time-series alignment processing on the acquired data, and transmit the processed data to the supervision knowledge graph construction and storage unit; the supervision knowledge graph construction and storage unit receives the data transmitted by the multimodal data acquisition and processing unit, processes the data using multimodal embedding algorithms and entity extraction algorithms, constructs a supervision knowledge graph, stores the supervision knowledge graph in a private storage module, and simultaneously establishes a data interaction channel with the graph inference and operation unit, providing it with knowledge graph data; the semantic query receiving and parsing unit is used to receive user input. The semantic query request is encoded and parsed to generate a query instruction, which is then sent to the graph inference unit. The graph inference unit receives the query instruction from the semantic query receiving and parsing unit, invokes a preset inference algorithm, and performs inference operations using the knowledge graph data provided by the supervision knowledge graph construction and storage unit. The inference results are then transmitted to the structured evidence matching and extraction unit. The structured evidence matching and extraction unit receives the inference results from the graph inference unit, matches and extracts structured evidence from the private storage module of the supervision knowledge graph construction and storage unit based on the inference results, and transmits the structured evidence to the retrieval result generation and feedback unit. The retrieval result generation and feedback unit receives the structured evidence transmitted by the structured evidence matching and extraction unit, generates an interpretable path according to the interpretable path generation rules, and feeds back the retrieval results, including the interpretable path, to the user interaction terminal. Different units transmit data and collaborate through a preset communication protocol.

Images