Method for constructing and dynamically evolving multi-modal knowledge base for industrial embodied intelligence

Abstract

The present application relates to the technical field of industrial intelligent knowledge processing, in particular to a multi-modal knowledge base construction and dynamic evolution method for industrial embodied intelligence, which comprises the following steps: constructing an embodied multi-modal data unit sequence and an embodied knowledge triple set, constructing an initial multi-modal knowledge base containing a knowledge graph structure, a multi-modal feature index table and a data source trace chain through knowledge fusion under the constraint of an ontology, and obtaining an optimized knowledge base through quality evaluation, cleaning and completion. A production line task-related knowledge sub-graph is extracted, an incremental knowledge candidate set is generated through online incremental learning in combination with real-time data, valid incremental knowledge is screened through confidence calculation and conflict detection, an evolved knowledge base is generated after fusion and update, and the index table and the trace chain are updated synchronously. The present method can realize the structured construction and dynamic iterative adaptation of an industrial embodied intelligent multi-modal knowledge base.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent knowledge processing technology, and in particular to a method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence. Background Technology

[0002] Current multimodal knowledge construction in the field of industrial embodied intelligence mostly adopts conventional knowledge graph construction methods, relying solely on the integration of single-modal data. The knowledge fusion process lacks ontological constraints, and the constructed knowledge base only possesses a basic graph structure, without synchronously supporting multimodal feature index tables and data source traceability chains. Current industrial knowledge base updates mostly employ offline batch update modes, which cannot adapt to the dynamic changes in real-time production line data. Incremental knowledge generation does not incorporate knowledge subgraph extraction for production line tasks, and incremental knowledge is only simply merged without confidence calculations or conflict detection.

[0003] Such technical solutions result in chaotic multimodal knowledge relationships in the knowledge base, a lack of data traceability, and difficulty in guaranteeing the completeness, consistency, and accuracy of knowledge. Offline update modes cannot match the real-time requirements of industrial production lines, the validity of incremental knowledge cannot be verified, knowledge conflicts occur frequently, and the knowledge base is difficult to adapt to the dynamic operation scenarios of industrial embodied intelligent agents.

[0004] An initial multimodal knowledge base needs to be constructed, which combines a knowledge graph structure, a multimodal feature index table, and a data source traceability chain. Incremental knowledge needs to be generated by combining the production line task knowledge subgraph with real-time data. After performing confidence calculation and conflict detection on the incremental knowledge, the knowledge base is integrated and updated, and the associated index and traceability chain are updated synchronously. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence, comprising:

[0007] Construct a sequence of embodied multimodal data units and a set of embodied knowledge triples for an industrial embodied intelligent agent;

[0008] Based on the aforementioned embodied knowledge triplet set, an initial multimodal knowledge base for industrial embodied intelligence is constructed through knowledge fusion under ontological constraints. The initial multimodal knowledge base includes a knowledge graph structure, a multimodal feature index table, and a data source traceability chain.

[0009] The initial multimodal knowledge base is subjected to knowledge quality assessment, generating a quality assessment report containing completeness score, consistency score and accuracy score. Based on the quality assessment report, the initial multimodal knowledge base is cleaned and supplemented to obtain an optimized multimodal knowledge base.

[0010] The optimized multimodal knowledge base extracts knowledge subgraphs related to the current production line task, and combines them with real-time data from the embodied multimodal data unit sequence to generate an incremental knowledge candidate set through an online incremental learning algorithm;

[0011] The incremental knowledge candidate set is subjected to confidence calculation and conflict detection to select the effective incremental knowledge set with a confidence level higher than a preset threshold and no knowledge conflict.

[0012] The effective incremental knowledge set is merged and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base, and the multimodal feature index table and the data source tracing chain are updated.

[0013] As a further aspect of the present invention, the construction of the embodied multimodal data unit sequence and embodied knowledge triple set of the industrial embodied intelligent agent includes:

[0014] Collect visual sensing data streams, force sensing data streams, acoustic sensing data streams, and operational behavior command streams generated synchronously by industrial embodied intelligent agents during production line operations to form a multi-source heterogeneous multimodal data set of industrial sites.

[0015] The industrial site multimodal data set is spatiotemporally aligned and associated to generate an embodied multimodal data unit sequence containing visual features, force features, acoustic features and behavioral semantic tags;

[0016] Knowledge extraction is performed on the embodied multimodal data unit sequence to obtain an embodied knowledge triplet set consisting of entity nodes, attribute edges, and relation edges, wherein the entity nodes include equipment parts, tools, materials, and operating actions, and the relation edges include assembly relations, action relations, temporal relations, and causal relationships.

[0017] As a further aspect of the present invention, the industrial site multimodal data set is spatiotemporally aligned and associated to generate an embodied multimodal data unit sequence containing visual features, force features, acoustic features, and behavioral semantic tags, including:

[0018] The visual sensing data stream is subjected to image frame extraction and preprocessing to generate a standardized visual image sequence, and a visual feature vector containing shape features, texture features and color features is extracted from the visual image sequence.

[0019] The force sensing data stream is filtered, denoised, and normalized to generate a standardized force signal sequence, and a force feature vector containing the torque magnitude, torque direction, and point of force application is extracted from the force signal sequence.

[0020] The acoustic sensing data stream is subjected to spectral analysis and endpoint detection to generate a standardized acoustic signal sequence, and an acoustic feature vector containing sound frequency, sound intensity and duration features is extracted from the acoustic signal sequence.

[0021] The operation behavior instruction stream is semantically parsed and decomposed into actions to generate a standardized operation behavior sequence, and behavior semantic tags containing action type, action target and action parameters are extracted from the operation behavior sequence.

[0022] Based on the acquisition timestamps of the visual sensing data stream, force sensing data stream, acoustic sensing data stream, and operation behavior command stream, the visual feature vector, force feature vector, acoustic feature vector, and behavior semantic label are timestamped and arranged in the aligned time order to generate the embodied multimodal data unit sequence.

[0023] As a further aspect of the present invention, knowledge extraction is performed on the embodied multimodal data unit sequence to obtain a set of embodied knowledge triples composed of entity nodes, attribute edges, and relation edges, including:

[0024] Target detection and recognition are performed on the visual feature vectors in the embodied multimodal data unit sequence to determine the entity categories of equipment parts, tools and materials, and generate equipment part entity nodes, tool entity nodes and material entity nodes;

[0025] Mechanical analysis is performed on the force feature vectors in the embodied multimodal data unit sequence to determine the assembly relationship and interaction relationship between each entity, and to generate assembly relationship edges and interaction relationship edges, wherein the assembly relationship edge represents the fixed connection relationship between two entities, and the interaction relationship edge represents the mechanical action of one entity on another entity;

[0026] Anomaly detection is performed on the acoustic feature vectors in the embodied multimodal data unit sequence to identify abnormal events during production line operation. Combined with the behavioral semantic tags, the causal relationship between abnormal events and operation actions is determined, and causal relationship edges are generated.

[0027] Temporal analysis is performed on the behavioral semantic tags in the embodied multimodal data unit sequence to determine the sequential execution order of each operation action and generate temporal relationship edges, wherein the temporal relationship edges represent the sequential execution relationship between two operation actions;

[0028] The generated equipment component entity nodes, tool entity nodes, material entity nodes, assembly relationship edges, action relationship edges, causal relationship edges, and temporal relationship edges are combined to form the embodied knowledge triple set, wherein each embodied knowledge triple consists of a head entity, a relationship edge, and a tail entity.

[0029] As a further aspect of the present invention, based on the aforementioned embodied knowledge triplet set, an initial multimodal knowledge base for industrial embodied intelligence is constructed through knowledge fusion under ontological constraints, including:

[0030] Based on expertise in the field of industrial embodied intelligence, a domain ontology is defined, comprising equipment, tools, materials, operations, attributes, and relationships. The domain ontology specifies the hierarchical structure and relationship types between various entities.

[0031] Ontology mapping is performed on the entity nodes in the embodied knowledge triple set, and the extracted entity nodes are mapped to the corresponding class or subclass of the domain ontology to generate a set of entity nodes with ontology classification labels.

[0032] Ontology verification is performed on the relation edges in the set of embodied knowledge triples to check whether the type of the relation edge conforms to the relation type defined in the domain ontology. Relation edges that do not conform to the domain ontology are converted or deleted to obtain a set of relation edges that conform to the ontology constraints.

[0033] The set of entity nodes with ontology classification labels and the set of relation edges that conform to ontology constraints are merged to generate a structured knowledge graph. Entity nodes with the same semantics are merged and duplicate relation edges are deduplicated.

[0034] Establish a mapping relationship between each entity node in the structured knowledge graph and its corresponding multimodal feature vector, generate the multimodal feature index table, record the data source of each entity node, generate the data source traceability chain, integrate the structured knowledge graph, the multimodal feature index table and the data source traceability chain to construct the initial multimodal knowledge base.

[0035] As a further aspect of the present invention, a knowledge quality assessment is performed on the initial multimodal knowledge base to generate a quality assessment report including a completeness score, a consistency score, and an accuracy score, including:

[0036] For each entity node in the initial multimodal knowledge base, check whether it contains all the necessary attribute information, count the number of entity nodes with missing attributes and the types of missing attributes, and calculate the integrity score according to the preset integrity evaluation criteria.

[0037] For the relation edges in the initial multimodal knowledge base, check whether there are contradictory relation assertions, identify conflicting relation edge pairs, count the number and severity of conflicts, and calculate the consistency score according to the preset consistency evaluation criteria;

[0038] The entity nodes and relation edges in the initial multimodal knowledge base are compared and verified with real industrial field data. The number of erroneous entity nodes and erroneous relation edges is counted, and the accuracy score is calculated according to the preset accuracy evaluation standard.

[0039] The integrity score, consistency score, and accuracy score are summarized to generate the quality assessment report, which includes the numerical values ​​of each score indicator and the score level classification.

[0040] As a further aspect of the present invention, a knowledge subgraph related to the current production line task is extracted from the optimized multimodal knowledge base, and combined with real-time data in the embodied multimodal data unit sequence, an incremental knowledge candidate set is generated through an online incremental learning algorithm, including:

[0041] Based on the task description of the current production line task, retrieve the entity nodes and relation edges related to the task from the knowledge graph of the optimized multimodal knowledge base, and extract the knowledge subgraph containing the task objective, required tools, operation steps and related equipment;

[0042] The entity nodes in the knowledge subgraph are matched with the real-time data in the embodied multimodal data unit sequence to identify the actual entity status and real-time operation status in the current production line environment.

[0043] By comparing the relation edges in the knowledge subgraph with the real-time data in a time series, new operation patterns or new relationships between entities not included in the existing knowledge subgraph are discovered, and a set of potential new relation edges is generated.

[0044] The potential set of new relation edges is associated with the feature vectors in the real-time data to generate the incremental knowledge candidate set containing the new relation edges and their supporting evidence, wherein the supporting evidence includes the feature vectors of the real-time data and the corresponding timestamps.

[0045] As a further aspect of the present invention, the incremental knowledge candidate set is subjected to confidence calculation and conflict detection to select an effective incremental knowledge set with a confidence level higher than a preset threshold and no knowledge conflict, including:

[0046] For each incremental knowledge in the incremental knowledge candidate set, the frequency of its occurrence in the real-time data of the embodied multimodal data unit sequence and the number of pieces of evidence supporting the incremental knowledge are counted, and the confidence level is calculated based on the frequency and the number of pieces of evidence.

[0047] Each incremental knowledge in the incremental knowledge candidate set is compared with the existing knowledge in the optimized multimodal knowledge base to check for semantic conflicts or logical contradictions, and to identify the incremental knowledge set that conflicts with the existing knowledge.

[0048] Set a confidence threshold and a conflict tolerance threshold, remove incremental knowledge with a confidence level lower than the confidence threshold and incremental knowledge belonging to the conflicting incremental knowledge set, and retain incremental knowledge with a confidence level higher than the confidence threshold and without conflict to generate the effective incremental knowledge set.

[0049] As a further aspect of the present invention, the effective incremental knowledge set is fused and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base, and the multimodal feature index table and the data source tracing chain are updated, including:

[0050] The entity nodes in the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base. If the added entity node has the same semantics as the existing entity node, the node is merged; otherwise, a new entity node is created.

[0051] Add the relation edges from the effective incremental knowledge set to the knowledge graph of the optimized multimodal knowledge base, establish connections between the newly added relation edges and the corresponding entity nodes, and adjust and optimize the structure of the knowledge graph.

[0052] Based on the feature vectors of the real-time data corresponding to the incremental knowledge in the effective incremental knowledge set, update the multimodal feature index table, add new feature vector mapping relationships or modify existing feature vector mapping relationships;

[0053] Record the data source, addition time, and update reason of each incremental knowledge in the effective incremental knowledge set, and append the recorded information to the data source tracing chain to complete the construction of the evolved multimodal knowledge base.

[0054] As a further aspect of the present invention, the method further includes:

[0055] The relevant knowledge from the evolved multimodal knowledge base is pushed to the decision control unit of the industrial embodied intelligence agent to guide it in executing production line tasks, specifically:

[0056] Based on the current production line task execution stage, retrieve the operation action entity nodes and corresponding relationship edges related to the execution stage from the knowledge graph of the evolved multimodal knowledge base, and extract task execution knowledge containing operation steps, required tools and operation parameters;

[0057] The task execution knowledge is matched with the current state information of the industrial embodied intelligent agent to determine the appropriate sequence of operation actions and tool selection scheme for the current state;

[0058] The operation sequence and tool selection scheme are converted into an instruction format that can be understood by an industrial embodied intelligent agent, generating a set of job task instructions that includes action instructions, parameter settings and time arrangements;

[0059] The set of work task instructions is sent to the decision control unit of the industrial embodied intelligent agent. The decision control unit controls the robotic arm, end effector and other actuators to complete the production line work tasks according to the set of work task instructions.

[0060] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0061] Under ontological constraints, knowledge fusion is carried out to construct an initial multimodal knowledge base that integrates a knowledge graph structure, a multimodal feature index table, and a data source traceability chain. Knowledge quality assessment is conducted, and knowledge cleaning and knowledge completion operations are performed. The multimodal knowledge relationships within the knowledge base are consistent with the ontological definition of industrial embodied intelligence. Multimodal data can be quickly matched and retrieved through the index table, and the data generation and integration path can be completely traced back through the traceability chain. The completeness, consistency, and accuracy of knowledge are optimized for adaptability, and the structure and multimodal adaptability of the knowledge base are in line with the application requirements of industrial embodied intelligent agents.

[0062] The knowledge subgraph corresponding to the current production line task is extracted. Combined with real-time data from the embodied multimodal data unit sequence, an online incremental learning algorithm is used to generate an incremental knowledge candidate set. Confidence calculation and conflict detection are performed on the incremental knowledge candidate set. After screening the effective incremental knowledge set, it is merged and updated with the optimized knowledge base. The multimodal feature index table and data source traceability chain are updated synchronously. The generation of incremental knowledge is directly related to the actual task requirements of the production line. The online incremental learning method is adapted to the real-time data iteration rhythm of the industrial production line. Confidence calculation can eliminate incremental knowledge with low reliability. Conflict detection can avoid logical contradictions between new knowledge and existing knowledge. The merged and updated knowledge base can adapt to the dynamic operation scenarios of industrial embodied intelligent agents. The synchronous update of the index table and traceability chain maintains the structural consistency and traceability integrity of the knowledge base throughout its entire life cycle. Attached Figure Description

[0063] Figure 1 This is a flowchart of the method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence as described in this invention;

[0064] Figure 2 A flowchart for constructing a sequence of embodied multimodal data units and a set of embodied knowledge triples;

[0065] Figure 3 A flowchart for generating a set of embodied knowledge triples;

[0066] Figure 4 A line graph showing the evolution quality score of a multimodal knowledge base;

[0067] Figure 5 This is a trend chart of the multiple rounds of knowledge base updates. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0070] See Figure 1 This study constructs an embodied multimodal data unit sequence and an embodied knowledge triple set for industrial embodied intelligence. Based on the embodied knowledge triple set, an initial multimodal knowledge base for industrial embodied intelligence is built through knowledge fusion under ontological constraints. This initial multimodal knowledge base includes a knowledge graph structure, a multimodal feature index table, and a data source traceability chain. The initial multimodal knowledge base undergoes a knowledge quality assessment, generating a quality assessment report containing completeness, consistency, and accuracy scores. Based on the quality assessment report, the initial multimodal knowledge base is cleaned and supplemented to obtain an optimized multimodal knowledge base. In the dynamic evolution phase, knowledge subgraphs related to the current production line task are extracted from the optimized multimodal knowledge base. Combined with real-time data from the embodied multimodal data unit sequence, an incremental knowledge candidate set is generated using an online incremental learning algorithm. The incremental knowledge candidate set is then subjected to confidence calculation and conflict detection, selecting valid incremental knowledge sets with confidence scores higher than a preset threshold and no knowledge conflicts. The effective incremental knowledge set is merged and updated with the optimized multimodal knowledge base to generate the evolved multimodal knowledge base, and the multimodal feature index table and the data source traceability chain are updated.

[0071] In one embodiment of the present invention, see [reference] Figure 2This study collects visual, force, and acoustic sensor data streams, as well as operational command streams, synchronously generated by industrial embodied agents during production line operations, forming a multi-source, heterogeneous, multimodal data set from the industrial field. The multimodal data set is then spatiotemporally aligned and correlated to generate a sequence of embodied multimodal data units containing visual, force, acoustic, and behavioral semantic tags. Image frames are extracted and preprocessed from the visual sensor data stream to generate standardized visual image sequences, from which visual feature vectors containing shape, texture, and color features are extracted. The force sensor data stream is filtered, denoised, and normalized to generate standardized force signal sequences, from which force feature vectors containing torque magnitude, direction, and point of application are extracted. The acoustic sensor data stream undergoes spectral analysis and endpoint detection to generate standardized acoustic signal sequences, from which acoustic feature vectors containing sound frequency, intensity, and duration are extracted. Semantic parsing and action decomposition are performed on the operation behavior command stream to generate a standardized operation behavior sequence. Behavioral semantic labels containing action type, action target, and action parameters are extracted from the operation behavior sequence. Based on the acquisition timestamps of the visual sensing data stream, force sensing data stream, acoustic sensing data stream, and operation behavior command stream, the visual feature vectors, force feature vectors, acoustic feature vectors, and behavioral semantic labels are timestamped and arranged in the aligned chronological order to generate an embodied multimodal data unit sequence.

[0072] In practical implementation, spatiotemporal alignment and data association are performed on the multimodal data set from the industrial site to generate an embodied multimodal data unit sequence containing visual features, force features, acoustic features, and behavioral semantic labels. Image frames are extracted from the visual sensing data stream at a rate of 30 frames per second, and each frame undergoes size normalization and illumination normalization preprocessing to generate a standardized visual image sequence. From each standardized visual image, a 1024-dimensional feature vector is extracted using a pre-trained convolutional neural network. This visual feature vector encodes the shape features of the capacitor elements, the texture features of the feeder tape, and the color features of different elements in the image. A moving average filter is applied to the force sensing data stream for noise reduction, and the original voltage values ​​are converted into force and torque values ​​in standard physical units according to the sensor range. Normalization is then performed to generate a standardized force signal sequence. From this standardized force signal sequence, a 6-dimensional force feature vector is extracted at the same 30Hz frequency. This force feature vector contains the magnitude and direction of the torque in three-dimensional spatial coordinates, as well as the coordinates of the point of force application. Short-time Fourier transform (SFT) is performed on the acoustic sensor data stream to perform spectral analysis. Effective sound segments are then segmented using energy-based endpoint detection to generate standardized acoustic signal sequences. From each standardized acoustic signal segment, Mel-frequency cepstral coefficients are extracted as a 128-dimensional acoustic feature vector containing sound frequency, sound intensity, and duration features. The operation command stream is semantically parsed using regular expressions, resolving "MoveTo(Feeder1,XYZ)" into action type "move", action target "feeder1", and action parameters "coordinates (X,Y,Z)". Action decomposition is then performed to generate standardized operation sequence. From the standardized operation sequence, the combination of "action type - action target" is extracted as the action semantic label. Based on the millisecond-precise timestamps uniformly assigned by the system during the acquisition of visual, force, acoustic, and operational command streams, visual feature vectors, force feature vectors, acoustic feature vectors, and behavioral semantic labels within the same time moment or time window are paired and aligned with the timestamps. These aligned data are then arranged strictly according to their chronological order, generating a time-indexed sequence of embodied multimodal data units. Each data unit contains a timestamp, a visual feature vector, a force feature vector, an acoustic feature vector, and a behavioral semantic label. In some embodiments, the timestamp alignment operation is implemented by solving an optimization problem. The goal is to achieve the best temporal match between feature points from different data streams. An objective function is defined to minimize the overall time deviation after alignment.

[0073]

[0074] Wherein: T align This represents the calculated set of optimal alignment times. This represents minimizing the variable T, ∑ i ∑ j≠i Indicates double summation, t v t f t a t c These represent the original timestamps of the visual feature vector, force feature vector, acoustic feature vector, and behavioral semantic label, respectively. The superscripts (i) and (j) represent the indices from different data streams, and ω... ij These are alignment weight coefficients between different data streams, preset by the sampling rate and importance of the data streams. This optimization process ensures that at the instant the robotic arm performs the "alignment and pressure" action, the corresponding force feature vector reflects the sudden increase in pressure, the acoustic feature vector captures the "click" sound of the component contacting the pad, and the behavioral semantic label is precisely "Place." The visual feature vector shows the alignment status of the capacitor pin and the pad hole, thus forming an embodied multimodal data unit that is strictly synchronized in time and semantically interconnected. In some embodiments, when there is a fixed delay in the original timestamps from different sensors, a pre-calibrated time offset is applied for compensation before alignment. Optionally, the dimension of the force feature vector can be expanded to a higher-dimensional vector that includes torque and contact surface estimation, depending on the sensor type. Optionally, wavelet transform can also be used to obtain the time-frequency domain feature representation for acoustic feature extraction. It can be understood that the process of generating the sequence of embodied multimodal data units is the basis for subsequent knowledge extraction. The precise temporal alignment of the multimodal data ensures that the subsequently extracted entities and relationships have accurate spatiotemporal context. Understandably, without spatiotemporal alignment, the "capacitance" entity detected from the visual stream and the "clamping force" detected from the kinetic stream may be incorrectly associated due to temporal asynchrony, resulting in the generation of incorrect knowledge triples.

[0075] In one embodiment of the present invention, knowledge extraction is performed on the embodied multimodal data unit sequence to obtain a set of embodied knowledge triples composed of entity nodes, attribute edges, and relation edges. See also... Figure 3The system performs target detection and recognition on visual feature vectors in the embodied multimodal data unit sequence to determine the entity categories of equipment parts, tools, and materials, generating equipment part entity nodes, tool entity nodes, and material entity nodes. It then performs mechanical analysis on force feature vectors to determine the assembly and interaction relationships between entities, generating assembly relationship edges and interaction relationship edges. Finally, it performs anomaly detection on acoustic feature vectors to identify abnormal events during production line operation, and combines these with behavioral semantic tags to determine the causal relationship between abnormal events and operational actions, generating causal relationship edges. The system also performs temporal analysis on behavioral semantic tags to determine the execution order of various operational actions, generating temporal relationship edges. Finally, it combines the generated equipment part entity nodes, tool entity nodes, material entity nodes, assembly relationship edges, interaction relationship edges, causal relationship edges, and temporal relationship edges to form a set of embodied knowledge triples.

[0076] In practical implementation, based on the embodied knowledge triple set, an initial multimodal knowledge base for industrial embodied intelligence is constructed through knowledge fusion under ontological constraints. According to professional knowledge in the industrial assembly field, a domain ontology is predefined, including "equipment," "tool," "material," "operation," "attribute," and "relationship" categories. The "equipment" category can be further subdivided into subcategories such as "power equipment" and "transmission equipment." The "tool" category can be subdivided into subcategories such as "hand tools" and "electric tools." The "relationship" category explicitly defines relationship types such as "assembly relationship," "action relationship," "temporal relationship," and "causal relationship," along with their domain and value range constraints. Ontology mapping is performed on the entity nodes in the embodied knowledge triple set. The extracted "servo motor" entity node is mapped to the "power equipment" subcategory under the "equipment" category in the domain ontology, and the "hex wrench" entity node is mapped to the "hand tool" subcategory under the "tool" category. This adds an ontology classification label to each entity node, generating a set of entity nodes with ontology classification labels. Ontology validation is performed on the relation edges in the embodied knowledge triple set. This checks whether the type of each relation edge conforms to the relation type and constraints defined in the domain ontology. For example, it checks whether the relation edge "apply_torque_to" exists in the definition of "relation class" and whether its head entity is "tool class" or "equipment class" and its tail entity is "material class" or "equipment class". For relation edges that do not conform to the ontology definition or constraints, such as a vague "connection" relation edge, it is converted into a more precise "is_fixed_to" assembly relation edge or directly deleted, resulting in a set of relation edges that conform to the ontology constraints. Knowledge merging is performed on the set of entity nodes with ontology classification labels and the set of relation edges conforming to ontology constraints. Entity nodes with the same semantics are merged. For example, the entity node "Hex wrench" extracted from sequences of different time periods, despite slight differences in their corresponding visual feature vectors under different lighting conditions, is determined to be the same entity through semantic similarity calculation. Therefore, it is merged into a single entity node, and all its associated relation edges are integrated. Duplicate relation edges are removed to generate a structured knowledge graph. Each entity and relation in the knowledge graph is identified by a Uniform Resource Identifier (URI). A mapping relationship is established between each entity node in the structured knowledge graph and its corresponding multimodal feature vector, generating a multimodal feature index table. The multimodal feature index table uses the entity's unique identifier as the primary key and associates the storage address pointers of the entity's feature vectors in different modalities in the feature database. For example, the entity "bolt" is associated with its feature vector ID in the visual feature database and its force feature vector ID representing its clamping position in the force feature database.Record the original data source for each entity node and relation edge, generating a data source traceability chain. The data source traceability chain records information including the timestamp range of the original embodied multimodal data unit sequence upon which the triplet for generating this knowledge is based, the sensor number, and the algorithm version number for processing the data. In some embodiments, when merging entity nodes with the same semantics, the similarity is determined by calculating the weighted similarity of the multimodal feature vectors corresponding to the two entity nodes. The similarity function is defined as:

[0077] S entity =α·F cosine (P v Q v )+β·F euclidean (P f Q f )+γ·F semantic (P l Q l )

[0078] Wherein: S entity P represents the combined similarity score between two entity nodes. v and Q v F represents the visual feature vectors of two entity nodes respectively. cosine It is the cosine similarity calculation function, P f and Q f F represents the force-feeling feature vector. euclidean It is the normalized Euclidean distance reciprocal calculation function, P l and Q l F represents the semantic embedding vector of the entity name parsed from the behavioral semantic label. semantic This is a semantic similarity calculation function, where α, β, and γ are the weight coefficients of each modality and satisfy α+β+γ=1. In some embodiments, the domain ontology can be formally represented using a web ontology language and stored in a separate ontology file. Optionally, the multimodal feature index table can be stored using extended attributes of a graph database or an external relational database. Optionally, the data source traceability chain can be stored in a blockchain structure to ensure immutability. It can be understood that knowledge fusion under ontological constraints ensures that potentially ambiguous or loose knowledge extracted from the original data is standardized into a unified and consistent conceptual system, providing a structured foundation for subsequent knowledge quality assessment and dynamic evolution. It can be understood that without ontology mapping and verification, the "robotic arm" entity extracted from different production lines or tasks may be considered different objects from entities such as "robot arm" and "robotic hand," leading to redundancy and chaos in the knowledge base. By mapping to the "operating equipment" subclass under the "equipment class," these synonymous entities can be merged.

[0079] In one embodiment of the present invention, a knowledge quality assessment is performed on the initial multimodal knowledge base to generate a quality assessment report containing completeness, consistency, and accuracy scores. For each entity node in the initial multimodal knowledge base, it is checked whether it contains all necessary attribute information, the number of entity nodes with missing attributes and the types of missing attributes are counted, and a completeness score is calculated according to a preset completeness evaluation standard. For the relation edges in the initial multimodal knowledge base, contradictory relation assertions are checked, conflicting relation edge pairs are identified, the number and severity of conflicts are counted, and a consistency score is calculated according to a preset consistency evaluation standard. The entity nodes and relation edges in the initial multimodal knowledge base are compared and verified with real industrial field data, the number of erroneous entity nodes and erroneous relation edges are counted, and an accuracy score is calculated according to a preset accuracy evaluation standard. The completeness, consistency, and accuracy scores are summarized to generate a quality assessment report. A knowledge subgraph related to the current production line task is extracted from the optimized multimodal knowledge base, and combined with real-time data from the embodied multimodal data unit sequence, an incremental knowledge candidate set is generated through an online incremental learning algorithm. Based on the task description of the current production line task, entity nodes and relational edges related to the task are retrieved from the knowledge graph of the optimized multimodal knowledge base, and a knowledge subgraph containing the task objective, required tools, operation steps, and related equipment is extracted. The entity nodes in the knowledge subgraph are matched with real-time data in the embodied multimodal data unit sequence to identify the actual entity state and real-time operation state in the current production line environment. The relational edges in the knowledge subgraph are compared with the real-time data over time to discover new operation patterns or new relationships between entities not included in the existing knowledge subgraph, generating a set of potential new relational edges. This set of potential new relational edges is then associated with feature vectors in the real-time data to generate an incremental knowledge candidate set containing the new relational edges and their supporting evidence.

[0080] In practical implementation, a knowledge subgraph related to the current production line task is extracted from the optimized multimodal knowledge base. Combined with real-time data from the embodied multimodal data unit sequence, an incremental knowledge candidate set is generated using an online incremental learning algorithm. The current production line task is "replacing piston rings," and the task description is "removing the old piston rings, cleaning the ring grooves, and installing new piston rings." Based on the task description, the knowledge graph of the optimized multimodal knowledge base is traversed and retrieved starting from entity nodes such as "piston rings," "pistons," and "ring grooves." This extracts a knowledge subgraph containing the task objective "installing new piston rings," the required tool "piston ring expander pliers," the operation steps "removal," "cleaning," and "installation," and the related equipment "pistons." This knowledge subgraph is stored in memory as a subgraph structure. The entity nodes in the knowledge subgraph are matched with real-time data in the embodied multimodal data unit sequence. The vision sensor at the end of the robotic arm identifies the specific parts on the workbench in real time, and the force sensor senses the contact force between the tool and the part. The actual entity status in the current production line environment is identified as "old piston ring", "piston", and "piston ring expander". The real-time operation status is "attempting to disassemble the broken piston ring directly by hand". By comparing the relation edges in the knowledge subgraph with the real-time data, the standard operational flow defined in the knowledge subgraph is that "removing the old piston ring" precedes "cleaning the ring groove" before "installing the new piston ring." However, analysis of the behavioral semantic label sequence and force and acoustic features of the real-time data reveals that the operator added a new action, "applying a small amount of lubricant," after "cleaning the ring groove." This new action is temporally adjacent to the subsequent action of "installing the new piston ring," thus discovering a new operational pattern not included in the existing knowledge subgraph and generating a potential new relation edge, "applying a small amount of lubricant before installing the new piston ring." Simultaneously, real-time visual data identifies a "needle-nose pliers" tool used to remove broken piston ring fragments, a tool not included in the knowledge subgraph. This reveals a new "removed_by" relationship between the entities "piston ring" and "needle-nose pliers." Associating the set of potential new relation edges with the feature vectors in the real-time data generates an incremental knowledge candidate set, where each incremental knowledge record contains the new relation edge. In some embodiments, the consistency score calculation applies logical reasoning rules to detect more subtle conflicts. For example, if the knowledge base contains edge pairs that form a cycle, such as "Ais_part_ofB" and "Bis_part_ofA", they will be marked as logical conflicts even if there is no direct semantic contradiction. In some embodiments, when a new operation pattern is discovered from real-time data, the dynamic time-warped distance between the new action sequence and the standard sequence in the knowledge subgraph is calculated. If the distance exceeds a threshold, the detection of the new pattern is triggered. Optionally, the accuracy score calculation can adopt a function based on the mean absolute error, the expression of which is:

[0081]

[0082] Among them: MAE acc The mean absolute error component represents the accuracy score, N represents the total number of entity and relation samples used for comparison, and Predicted i Actual represents the predicted category or existence of the i-th entity or relation in the knowledge base. i This represents the actual category or existence of the i-th entity or relation obtained from real data. Optionally, when generating the incremental knowledge candidate set, an initial confidence weight can be calculated for each candidate knowledge, based on the quantity and consistency of supporting evidence. It is understood that the knowledge quality assessment report provides quantitative and targeted guidance for knowledge cleaning and completion. For example, if the report indicates that the "link" lacks the "length" attribute, then the knowledge cleaning phase will trigger an operation to query the value of that attribute from an external database or sensor. It is also understood that the new operation of "applying lubricant" discovered from the comparison between real-time data and the knowledge subgraph is a key input for the dynamic evolution of the knowledge base to adapt to actual process improvements; otherwise, the knowledge base will be unable to guide the robot to learn this optimized operation step.

[0083] In one embodiment of the present invention, confidence scores are calculated and conflict detection is performed on the incremental knowledge candidate set to filter out the effective incremental knowledge set with confidence scores higher than a preset threshold and no knowledge conflicts. For each incremental knowledge in the incremental knowledge candidate set, the frequency of its occurrence in the real-time data of the embodied multimodal data unit sequence and the number of pieces of evidence supporting the incremental knowledge are counted, and the confidence score is calculated based on the frequency and the number of pieces of evidence. Each incremental knowledge in the incremental knowledge candidate set is compared with the existing knowledge in the optimized multimodal knowledge base to check for semantic conflicts or logical contradictions, and the incremental knowledge set that conflicts with the existing knowledge is identified. A confidence score threshold and a conflict tolerance threshold are set, and incremental knowledge with confidence scores lower than the confidence score threshold and incremental knowledge belonging to conflicting incremental knowledge sets are removed, while incremental knowledge with confidence scores higher than the confidence score threshold and no conflicts is retained to generate an effective incremental knowledge set. The effective incremental knowledge set is fused and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base, and the multimodal feature index table and the data source tracing chain are updated. Entity nodes from the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base. If the added entity node has the same semantics as an existing entity node, the nodes are merged; otherwise, a new entity node is created. Relational edges from the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base, establishing connections between the new relational edges and their corresponding entity nodes, and the structure of the knowledge graph is adjusted and optimized. Based on the feature vectors of the real-time data corresponding to the incremental knowledge in the effective incremental knowledge set, the multimodal feature index table is updated, adding new feature vector mapping relationships or modifying existing feature vector mapping relationships. The data source, addition time, and update reason of each incremental knowledge in the effective incremental knowledge set are recorded, and this recorded information is appended to the data source traceability chain.

[0084] In practice, the effective incremental knowledge set is merged and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base. The multimodal feature index table and data source traceability chain are also updated. Taking the effective incremental knowledge set selected from the welding production line as an example, entity nodes from the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base. The effective incremental knowledge set contains a new "appearance inspection" operation entity node. Checking the graph of the optimized multimodal knowledge base reveals that a "quality inspection" type node already exists, but no specific "appearance inspection" node. Since "appearance inspection" is a specific subclass of "quality inspection" and their semantics are not entirely the same, a new "appearance inspection" entity node is created as its child node under the "quality inspection" node. Simultaneously, the "cooling fan" entity node mentioned in the effective incremental knowledge set already exists in the optimized multimodal knowledge base and has the same semantics; therefore, node merging is performed, associating the new evidence with the existing nodes. The relational edges from the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base, establishing connections between the new relational edges and their corresponding entity nodes. The effective incremental knowledge set contains the new relational edges "welding completed before visual inspection" and "visual inspection completed before ultrasonic testing." These two temporal relational edges are added to the knowledge graph, connecting the three operation entity nodes: "welding completed," "visual inspection," and "ultrasonic testing." The structure of the knowledge graph is adjusted and optimized. For example, due to the addition of the "visual inspection" node, the original edge directly connecting "welding completed" to "ultrasonic testing" is replaced by a new path and may be marked as redundant or downweighted. Based on the feature vectors of the real-time data corresponding to the incremental knowledge in the effective incremental knowledge set, the multimodal feature index table is updated, adding new feature vector mapping relationships or modifying existing ones. For the newly added "visual inspection" entity node, a new entry is created in the feature index table, and the typical visual feature vector and behavioral semantic label encoding corresponding to this operation are stored together. For the merged "cooling fan" node, the pointer to the newly observed acoustic feature vector is added to the existing feature vector list of that node. Record the data source, addition time, and update reason for each incremental knowledge in the effective incremental knowledge set. Append the record information to the data source traceability chain. For the newly added "Appearance Inspection" node and related relationship edges, create a record in the data source traceability chain. The record information includes: the data source is "Real-time monitoring data stream of production line A on May 10, 2026 (timestamp: 10:15:00-10:30:00)", the addition time is "2026-05-10 18:00:00", and the update reason is "New inspection step added for process optimization". The traceability chain is stored in a linked list structure to complete the construction of the evolved multimodal knowledge base. See Table 1.

[0085] Table 1: Evaluation Table of Incremental Knowledge Candidate Sets

[0086]

[0087] In some embodiments, confidence scores are calculated using the harmonic mean of the precision and coverage of supporting evidence, i.e., the F1 score, calculated as follows:

[0088]

[0089] Wherein: F conf Let P represent the confidence score, P represent the precision (calculated as the proportion of times the incremental knowledge pattern is identified as correct when it appears), and R represent the recall (calculated as the proportion of times the correct pattern is found in all relevant contexts). In some embodiments, conflict detection checks not only direct semantic conflicts but also implicit logical contradictions by describing the logic inference engine. For example, if existing knowledge states "all welding operations must use fixture A," while new knowledge implies "a certain welding operation uses fixture B," the inference engine will infer a conflict. Optionally, the node merging operation can be based on an entity alignment algorithm that simultaneously compares the semantic similarity of entity names, the overlap of attributes, and the similarity of association relationships. Optionally, the records of the data source traceability chain can include digital signatures to ensure log integrity. It is understood that the screening mechanism based on confidence and conflict detection ensures that only new knowledge with high reliability and compatibility with the existing knowledge system can be integrated into the knowledge base. For example, although "adjusting the pressure to 370N" performs well in local data, it is rejected because it conflicts with core process specifications, thus preventing the knowledge base from being contaminated by abnormal or unverified temporary changes. It is understandable that updating the multimodal feature index table and traceability chain during the fusion update ensures that the new knowledge maintains a traceable connection with the original sensor data source from which it was generated. This is crucial for subsequent knowledge verification, knowledge evolution analysis, and quality auditing.

[0090] See Figure 4 This is a line graph showing the quality score of a multimodal knowledge base evolution, visually illustrating the quality changes of the knowledge base in three key stages. All three quality indicators continuously improve with knowledge base iteration, reflecting the effectiveness of knowledge cleaning, completion, and incremental fusion. The initial knowledge base has the lowest consistency score, indicating that there are many logical contradictions or conflicts in the early knowledge, making it a key area for quality optimization. From the initial stage to the optimized stage, all three scores show significant improvement, validating the role of the knowledge quality assessment and cleaning process. After evolution, the knowledge base continues to improve slightly on the basis of optimization, indicating that the incremental knowledge screening and fusion mechanism can continuously consolidate the quality of the knowledge base without introducing significant quality loss. Through continuous iteration, the knowledge base ultimately achieves a completeness score of 92, a consistency score of 90, and an accuracy score of 91, meeting the high-reliability knowledge requirements of industrial embodied intelligence.

[0091] In one embodiment of the present invention, relevant knowledge from the evolved multimodal knowledge base is pushed to the decision control unit of the industrial embodied agent. Based on the current execution stage of the production line task, the entity nodes of the operation actions related to the execution stage and their corresponding relational edges are retrieved from the knowledge graph of the evolved multimodal knowledge base, extracting task execution knowledge containing operation steps, required tools, and operation parameters. The task execution knowledge is matched with the current state information of the industrial embodied agent to determine a suitable sequence of operation actions and a tool selection scheme for the current state. The sequence of operation actions and the tool selection scheme are converted into an instruction format understood by the industrial embodied agent, generating a work task instruction set containing action instructions, parameter settings, and timing. The work task instruction set is sent to the decision control unit of the industrial embodied agent, which then controls the robotic arm, end effector, and other actuators to complete the production line operation task according to the work task instruction set.

[0092] In practical implementation, relevant knowledge from the evolved multimodal knowledge base is pushed to the decision control unit of the industrial embodied agent to guide it in executing production line tasks. The current production line task is "completing circuit board assembly welding." The industrial embodied agent is a six-axis robotic arm system integrating a welding torch, a vision camera, and a force sensor. Based on the current stage of the production line task, the industrial embodied agent is in the "spot welding completed, entering quality inspection" stage. From the knowledge graph of the evolved multimodal knowledge base, a graph search is performed with the "circuit board welding" task node as the root. The search retrieves the operation action entity nodes and corresponding relationship edges related to the "quality inspection" execution stage. The task execution knowledge, including the operation steps "perform visual re-inspection" and "perform power-on test," the required tools "high-definition industrial camera" and "multimeter," and the operation parameters "camera resolution 2048x1536" and "inspection standard file 'IPC-A-610'," is extracted from the evolved multimodal knowledge base into the memory workspace in a subgraph structure. The task execution knowledge is matched with the current state information of the industrial embodied intelligent agent. The current state information of the industrial embodied intelligent agent includes the position of the robotic arm end effector "above solder joint P12", the status of the welding gun "idle", the sensor status "camera online, multimeter ready", and the preliminary result of "solder joint P12 has a solder bridge" identified from real-time vision. The state information is matched with the extracted task execution knowledge. The matching process calculates the similarity between the current state and the features of the knowledge nodes. The appropriate operation sequence for the current state is determined to be "adjust the camera focus to focus on solder joint P12 -> take a high-resolution image -> call the detection algorithm for analysis -> mark it if it is confirmed as a defect". The tool selection scheme is to use a "high-definition industrial camera" for re-inspection. In the matching, the preliminary result of "solder joint P12 has a solder bridge" triggers the handling process of "solder bridge defect" in the knowledge base. This process is associated with the operation node "high-precision visual re-inspection confirmation is required". The sequence of operational actions and tool selection schemes are converted into an instruction format that can be understood by the industrial embodied intelligent agent, generating a set of job task instructions that includes action instructions, parameter settings, and timing. The sequence of action instructions is converted into specific service calls and topic messages in the robot operating system. For example, an action instruction of "call vision service: capture_high_res_image" is generated, with parameter settings including target position "coordinates (x,y,z) of weld point P12", camera parameters "resolution=2048x1536, exposure time=20ms", and timing is arranged by the task scheduler to "execute within 100ms after receiving 'contact complete' feedback from the force sensor". The entire set of job task instructions is encapsulated into a structured JSON file or an equivalent robot instruction script.The task instruction set is sent to the decision control unit of the industrial embodied intelligence. The decision control unit is an industrial control computer running a real-time operating system. The decision control unit parses the received task instruction set, controls the robotic arm to move to the specified coordinates according to the instructions, controls the high-definition industrial camera on the end effector to take pictures, controls the communication interface to send an analysis request to the image processing unit, and controls the robotic arm to mark the defect location or proceed to the next process according to the returned re-inspection results.

[0093] In some embodiments, the process of matching the current state with task execution knowledge can employ a similarity-based decision logic. This involves calculating the similarity between the multimodal feature vector observed by the current sensor and the expected feature vectors associated with each operation action node stored in the evolved multimodal knowledge base, to select the most matching subsequent operation. The matching degree function is defined as follows:

[0094]

[0095] Where: M represents the overall matching score between the current state and a certain knowledge node, K is the number of feature modalities participating in the matching, and w i These are the weight coefficients of the i-th feature mode. It is the feature vector of the i-th mode extracted from the current state information. This refers to the expected feature vector of the i-th modality associated with the knowledge node in the evolved multimodal knowledge base. The function `sim` is a function that calculates the similarity between two vectors, such as cosine similarity. In some embodiments, the task instruction set can be generated by filling in predefined robot instruction templates. The templates reserve slots for variables such as action type, target coordinates, tool parameters, and conditional branches. The system extracts specific values ​​from the evolved multimodal knowledge base according to the matching results to fill these slots, thereby generating an executable instruction sequence. Optionally, when the decision control unit executes the task instruction set, it monitors the execution status in real time and compares it with the expected status in the evolved multimodal knowledge base. If a large deviation occurs, an exception handling process is triggered, and an attempt is made to retrieve a correction strategy from the knowledge base. Optionally, the task instruction set sent to the decision control unit can be represented in the form of a hierarchical task network, with the top layer being the task objective, the middle layer being the sub-task sequence, and the bottom layer being the specific control primitives. It is understandable that extracting and pushing task execution knowledge from the evolved multimodal knowledge base enables industrial embodied agents to directly utilize collective experience and optimized processes for operations. For example, the evolved multimodal knowledge base added a "post-spot welding visual re-inspection" step in its previous dynamic evolution, and this task accurately retrieved and applied this optimized operational process. It is also understandable that transforming abstract knowledge into a concrete set of task instructions with temporal and parameter constraints is a crucial link connecting symbolic knowledge representation with physical-layer robot control. The accuracy of the instruction set directly determines the success rate and quality of the industrial embodied agent's task execution.

[0096] See Figure 5 This is a trend chart showing the changes in the number of entities, relationships, and accuracy across multiple update rounds. Both the number of entities and relationships exhibit linear and stable growth, with the relationship growth rate consistently outpacing the entity growth rate, reflecting the continuous enrichment of the relationship logic between entities within the knowledge base. From round 1 to round 5, the number of entities increased by 250%, and the number of relationships increased by 353%, indicating a continuous improvement in the connection density of the knowledge graph. Accuracy steadily increased with each update round, rising from an initial 0.75 to a final 0.95, validating the effectiveness of the incremental knowledge filtering, conflict detection, and fusion mechanisms, achieving simultaneous progress in "scale expansion + quality improvement." The accuracy growth rate was faster in the early stages (rounds 1-3) and leveled off in the later stages (rounds 4-5), consistent with the quality convergence pattern after the knowledge system matures. The simultaneous positive growth of knowledge scale and accuracy indicates that the knowledge base update process did not introduce a large amount of erroneous knowledge, avoiding the typical problem of "data expansion but quality decline."

[0097] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence, characterized in that: The method includes: Construct a sequence of embodied multimodal data units and a set of embodied knowledge triples for an industrial embodied intelligent agent; Based on the aforementioned embodied knowledge triplet set, an initial multimodal knowledge base for industrial embodied intelligence is constructed through knowledge fusion under ontological constraints. The initial multimodal knowledge base includes a knowledge graph structure, a multimodal feature index table, and a data source traceability chain. The initial multimodal knowledge base is subjected to knowledge quality assessment, generating a quality assessment report containing completeness score, consistency score and accuracy score. Based on the quality assessment report, the initial multimodal knowledge base is cleaned and supplemented to obtain an optimized multimodal knowledge base. The optimized multimodal knowledge base extracts knowledge subgraphs related to the current production line task, and combines them with real-time data from the embodied multimodal data unit sequence to generate an incremental knowledge candidate set through an online incremental learning algorithm; The incremental knowledge candidate set is subjected to confidence calculation and conflict detection to select the effective incremental knowledge set with a confidence level higher than a preset threshold and no knowledge conflict. The effective incremental knowledge set is merged and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base, and the multimodal feature index table and the data source tracing chain are updated.

2. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, The constructed industrial embodied intelligent agent includes an embodied multimodal data unit sequence and an embodied knowledge triple set, comprising: Collect visual sensing data streams, force sensing data streams, acoustic sensing data streams, and operational behavior command streams generated synchronously by industrial embodied intelligent agents during production line operations to form a multi-source heterogeneous multimodal data set of industrial sites. The industrial site multimodal data set is spatiotemporally aligned and associated to generate an embodied multimodal data unit sequence containing visual features, force features, acoustic features and behavioral semantic tags; Knowledge extraction is performed on the embodied multimodal data unit sequence to obtain an embodied knowledge triplet set consisting of entity nodes, attribute edges, and relation edges. The entity nodes include equipment parts, tools, materials, and operational actions, and the relation edges include assembly relations, action relations, temporal relations, and causal relationships.

3. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 2, characterized in that, The industrial site multimodal dataset is spatiotemporally aligned and correlated to generate an embodied multimodal data unit sequence containing visual features, force features, acoustic features, and behavioral semantic tags, including: The visual sensing data stream is subjected to image frame extraction and preprocessing to generate a standardized visual image sequence, and a visual feature vector containing shape features, texture features and color features is extracted from the visual image sequence. The force sensing data stream is filtered, denoised, and normalized to generate a standardized force signal sequence, and a force feature vector containing the torque magnitude, torque direction, and point of force application is extracted from the force signal sequence. The acoustic sensing data stream is subjected to spectral analysis and endpoint detection to generate a standardized acoustic signal sequence, and an acoustic feature vector containing sound frequency, sound intensity and duration features is extracted from the acoustic signal sequence. The operation behavior instruction stream is semantically parsed and decomposed into actions to generate a standardized operation behavior sequence, and behavior semantic tags containing action type, action target and action parameters are extracted from the operation behavior sequence. Based on the acquisition timestamps of the visual sensing data stream, force sensing data stream, acoustic sensing data stream, and operation behavior command stream, the visual feature vector, force feature vector, acoustic feature vector, and behavior semantic label are timestamped and arranged in the aligned time order to generate the embodied multimodal data unit sequence.

4. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 2, characterized in that, Knowledge extraction is performed on the embodied multimodal data unit sequence to obtain a set of embodied knowledge triples consisting of entity nodes, attribute edges, and relation edges, including: Target detection and recognition are performed on the visual feature vectors in the embodied multimodal data unit sequence to determine the entity categories of equipment parts, tools and materials, and generate equipment part entity nodes, tool entity nodes and material entity nodes; Mechanical analysis is performed on the force feature vectors in the embodied multimodal data unit sequence to determine the assembly relationship and interaction relationship between each entity, and to generate assembly relationship edges and interaction relationship edges, wherein the assembly relationship edge represents the fixed connection relationship between two entities, and the interaction relationship edge represents the mechanical action of one entity on another entity; Anomaly detection is performed on the acoustic feature vectors in the embodied multimodal data unit sequence to identify abnormal events during production line operation. Combined with the behavioral semantic tags, the causal relationship between abnormal events and operation actions is determined, and causal relationship edges are generated. Temporal analysis is performed on the behavioral semantic tags in the embodied multimodal data unit sequence to determine the sequential execution order of each operation action and generate temporal relationship edges, wherein the temporal relationship edges represent the sequential execution relationship between two operation actions; The generated equipment component entity nodes, tool entity nodes, material entity nodes, assembly relationship edges, action relationship edges, causal relationship edges, and temporal relationship edges are combined to form the embodied knowledge triple set, wherein each embodied knowledge triple consists of a head entity, a relationship edge, and a tail entity.

5. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, Based on the aforementioned set of embodied knowledge triples, an initial multimodal knowledge base for industrial embodied intelligence is constructed through knowledge fusion under ontological constraints, including: Based on expertise in the field of industrial embodied intelligence, a domain ontology is defined, comprising equipment, tools, materials, operations, attributes, and relationships. The domain ontology specifies the hierarchical structure and relationship types between various entities. Ontology mapping is performed on the entity nodes in the embodied knowledge triple set, and the extracted entity nodes are mapped to the corresponding class or subclass of the domain ontology to generate a set of entity nodes with ontology classification labels. Ontology verification is performed on the relation edges in the set of embodied knowledge triples to check whether the type of the relation edge conforms to the relation type defined in the domain ontology. Relation edges that do not conform to the domain ontology are converted or deleted to obtain a set of relation edges that conform to the ontology constraints. The set of entity nodes with ontology classification labels and the set of relation edges that conform to ontology constraints are merged to generate a structured knowledge graph. Entity nodes with the same semantics are merged and duplicate relation edges are deduplicated. Establish a mapping relationship between each entity node in the structured knowledge graph and its corresponding multimodal feature vector, generate the multimodal feature index table, record the data source of each entity node, generate the data source traceability chain, integrate the structured knowledge graph, the multimodal feature index table and the data source traceability chain to construct the initial multimodal knowledge base.

6. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, A knowledge quality assessment is performed on the initial multimodal knowledge base to generate a quality assessment report containing completeness, consistency, and accuracy scores, including: For each entity node in the initial multimodal knowledge base, check whether it contains all the necessary attribute information, count the number of entity nodes with missing attributes and the types of missing attributes, and calculate the integrity score according to the preset integrity evaluation criteria. For the relation edges in the initial multimodal knowledge base, check whether there are contradictory relation assertions, identify conflicting relation edge pairs, count the number and severity of conflicts, and calculate the consistency score according to the preset consistency evaluation criteria; The entity nodes and relation edges in the initial multimodal knowledge base are compared and verified with real industrial field data. The number of erroneous entity nodes and erroneous relation edges is counted, and the accuracy score is calculated according to the preset accuracy evaluation standard. The integrity score, consistency score, and accuracy score are summarized to generate the quality assessment report, which includes the numerical values ​​of each score indicator and the score level classification.

7. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, From the optimized multimodal knowledge base, a knowledge subgraph related to the current production line task is extracted. Combined with real-time data from the embodied multimodal data unit sequence, an incremental knowledge candidate set is generated using an online incremental learning algorithm, including: Based on the task description of the current production line task, retrieve the entity nodes and relation edges related to the task from the knowledge graph of the optimized multimodal knowledge base, and extract the knowledge subgraph containing the task objective, required tools, operation steps and related equipment; The entity nodes in the knowledge subgraph are matched with the real-time data in the embodied multimodal data unit sequence to identify the actual entity status and real-time operation status in the current production line environment. By comparing the relation edges in the knowledge subgraph with the real-time data in a time series, new operation patterns or new relationships between entities not included in the existing knowledge subgraph are discovered, and a set of potential new relation edges is generated. The potential set of new relation edges is associated with the feature vectors in the real-time data to generate the incremental knowledge candidate set containing the new relation edges and their supporting evidence, wherein the supporting evidence includes the feature vectors of the real-time data and the corresponding timestamps.

8. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, The incremental knowledge candidate set is subjected to confidence calculation and conflict detection to select a valid incremental knowledge set with a confidence level higher than a preset threshold and no knowledge conflicts, including: For each incremental knowledge in the incremental knowledge candidate set, the frequency of its occurrence in the real-time data of the embodied multimodal data unit sequence and the number of pieces of evidence supporting the incremental knowledge are counted, and the confidence level is calculated based on the frequency and the number of pieces of evidence. Each incremental knowledge in the incremental knowledge candidate set is compared with the existing knowledge in the optimized multimodal knowledge base to check for semantic conflicts or logical contradictions, and to identify the incremental knowledge set that conflicts with the existing knowledge. Set a confidence threshold and a conflict tolerance threshold, remove incremental knowledge with a confidence level lower than the confidence threshold and incremental knowledge belonging to the conflicting incremental knowledge set, and retain incremental knowledge with a confidence level higher than the confidence threshold and without conflict to generate the effective incremental knowledge set.

9. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, The effective incremental knowledge set is fused and updated with the optimized multimodal knowledge base to generate an evolved multimodal knowledge base, and the multimodal feature index table and the data source tracing chain are updated, including: The entity nodes in the effective incremental knowledge set are added to the knowledge graph of the optimized multimodal knowledge base. If the added entity node has the same semantics as the existing entity node, the node is merged; otherwise, a new entity node is created. Add the relation edges from the effective incremental knowledge set to the knowledge graph of the optimized multimodal knowledge base, establish connections between the newly added relation edges and the corresponding entity nodes, and adjust and optimize the structure of the knowledge graph. Based on the feature vectors of the real-time data corresponding to the incremental knowledge in the effective incremental knowledge set, update the multimodal feature index table, add new feature vector mapping relationships or modify existing feature vector mapping relationships; Record the data source, addition time, and update reason of each incremental knowledge in the effective incremental knowledge set, and append the recorded information to the data source tracing chain to complete the construction of the evolved multimodal knowledge base.

10. The method for constructing and dynamically evolving a multimodal knowledge base for industrial embodied intelligence according to claim 1, characterized in that, The method further includes: The relevant knowledge from the evolved multimodal knowledge base is pushed to the decision control unit of the industrial embodied intelligence agent to guide it in executing production line tasks, specifically: Based on the current production line task execution stage, retrieve the operation action entity nodes and corresponding relationship edges related to the execution stage from the knowledge graph of the evolved multimodal knowledge base, and extract task execution knowledge containing operation steps, required tools and operation parameters; The task execution knowledge is matched with the current state information of the industrial embodied intelligent agent to determine the appropriate sequence of operation actions and tool selection scheme for the current state; The operation sequence and tool selection scheme are converted into an instruction format that can be understood by an industrial embodied intelligent agent, generating a set of job task instructions that includes action instructions, parameter settings and time arrangements; The set of work task instructions is sent to the decision control unit of the industrial embodied intelligent agent. The decision control unit controls the robotic arm, end effector and other actuators to complete the production line work tasks according to the set of work task instructions.

Images