A method and system for constructing and dynamically updating environmental knowledge in a smart earphone factory

Abstract

The application discloses a kind of wisdom earphone factory in environment knowledge construction and dynamic updating method and system, it is related to intelligent manufacturing technical field, and the present scheme is by constructing multilayer semantic mapping system, and combining ontology, template and the multistage expression framework of instance, the attribute of component part, operating equipment attribute, sensor monitoring mode, assembly skill and complete production process are converted into structured knowledge node and associated relationship.Visual atlas, auditory atlas and tactile atlas are fused into composite multimodal knowledge atlas;Through entity layer comparison, relationship layer comparison, attribute layer comparison and local relationship graph network inference, dynamic synchronization of knowledge is realized;Cross-rate time lag compensation is completed through asynchronous double-rate Kalman filter, and the execution control result of assembly equipment is output combined with reinforcement learning.The present scheme is conducive to improving wisdom earphone factory environment understanding depth, knowledge updating timeliness, flexible production adaptability and assembly safety.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, and specifically relates to a method and system for constructing and dynamically updating environmental knowledge in a smart headphone factory. Background Technology

[0002] As intelligent manufacturing continues to evolve towards precision, flexibility, and intelligence, headphone factories are gradually shifting from a fixed-cycle, single-equipment-dominated manufacturing model to a complex model characterized by multi-equipment collaboration, tightly coupled processes, and online quality control. Headphone assembly involves multiple objects, including shells, driver units, cables, fixtures, and testing equipment, and encompasses various operational behaviors such as grasping, alignment, insertion, pressing, inspection, and transfer. For robots and other assembly equipment to reliably complete precision assembly tasks in such scenarios, they need not only to acquire the spatial state of components, equipment operating status, and process execution status, but also to develop a calculable, callable, and updatable structured representation of object relationships, action relationships, process relationships, and task relationships within the production environment. While visual, auditory, and tactile perception methods can provide heterogeneous information such as pose, latching sound, and pressure feedback, different modal data exhibit significant differences in sampling frequency, physical dimension, and semantic granularity. Therefore, a hierarchical environmental knowledge representation and dynamic update mechanism adapted to the headphone manufacturing scenario is required. Existing technologies typically use visual, auditory, and tactile information separately for local detection, single-step judgment, or single-stage control, lacking a unified hierarchical semantic organization structure. This makes it difficult to form stable object, process, and task associations among multi-source information. When faced with dynamic conditions such as the introduction of new parts, adjustment of process parameters, or switching of production processes, existing solutions generally lack real-time comparison, relational reasoning, and incremental synchronization mechanisms for new knowledge entities and new knowledge triples, leading to mismatches between knowledge base content and on-site conditions. When there is a difference in output frequency between the multimodal fusion results and the assembly equipment control layer, without dedicated time-delay compensation methods, the control results are easily affected by delayed states, thereby reducing the quality of path planning and the accuracy of operation correction in precision assembly scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for constructing and dynamically updating environmental knowledge in a smart headphone factory, thereby solving the aforementioned technical problems.

[0004] On the one hand, a method for constructing and dynamically updating environmental knowledge in a smart headphone factory is proposed, including the following steps:

[0005] Step S1: Collect multimodal perception data in the headphone production environment. The multimodal perception data includes at least visual data, auditory data, and tactile data. Based on the multimodal perception data, establish a hierarchical semantic mapping system including an entity layer, an intelligent agent layer, a perception layer, an action layer, and a task layer.

[0006] Step S2: Generate a corresponding multimodal environment knowledge graph based on the multimodal perception data, and perform cross-modal alignment, association calculation and fusion representation on the multimodal environment knowledge graph to generate a composite multimodal knowledge graph;

[0007] Step S3: If a new knowledge entity is detected, a local relation graph network is constructed based on the comparison results with the hierarchical semantic mapping system, and the updated multimodal environment knowledge graph is obtained through relation reasoning and dynamic synchronization; if no new knowledge entity is detected, the current composite multimodal knowledge graph is directly output.

[0008] Step S4: Perform time delay compensation, knowledge reasoning and assembly decision control based on the updated multimodal environment knowledge graph, and output path planning results and precision operation correction results for assembly equipment execution control.

[0009] Preferably, the visual multimodal perception data includes at least the pose information of the headphone components;

[0010] The auditory multimodal perception data includes at least assembly and locking sound information;

[0011] The tactile multimodal sensing data includes at least assembly pressure feedback information;

[0012] The multimodal environmental knowledge graph includes a visual graph, an auditory graph, and a tactile graph.

[0013] Preferably, in the hierarchical semantic mapping system, the entity layer is used to represent the attributes of headphone components, the intelligent agent layer is used to represent the attributes of operating devices, the perception layer is used to represent the sensor monitoring mode, the action layer is used to represent assembly skills, and the task layer is used to represent production processes.

[0014] By combining a multi-level representation framework of ontology, templates, and instances, component attributes, operating equipment attributes, sensor monitoring modes, assembly skills, and production processes are transformed into structured knowledge nodes and relationships.

[0015] Preferably, the multi-level expression framework includes a domain ontology unit, a process template unit, and a real-time instance unit;

[0016] Among them, the domain ontology unit is used to define the object categories, logical rules and process associations in the headphone production scenario, the process template unit is used to establish a parameterized expression template based on the standard operating procedure, and the real-time instance unit is used to receive real-time sensing data and fill it into the parameterized expression template.

[0017] Preferably, step S2, which generates a composite multimodal knowledge graph, specifically includes the following steps:

[0018] Spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment are performed on the multimodal environment knowledge graph to obtain the aligned multimodal environment knowledge graph;

[0019] Based on the aligned multimodal environment knowledge graph, the interaction weights between entities of different modalities are calculated using a differentiated weighted cross-modal association network;

[0020] Based on the obtained interaction weights, the aligned multimodal environment knowledge graph is fused using a parallel guided multimodal knowledge fusion algorithm to generate the composite multimodal knowledge graph.

[0021] Preferably, when performing spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment on a multimodal environment knowledge graph, the following steps are also included:

[0022] Graph convolutional neural networks are used to extract topological information of multimodal environmental knowledge graphs;

[0023] A multi-head cross-attention mechanism is used to calculate the interaction weights between entities of different modalities;

[0024] A differentiated weight allocation matrix is ​​generated based on the current process status, and the weighted features are projected into the joint embedding latent space to perform parallel feature aggregation.

[0025] Preferably, step S3, which compares the new knowledge entity with the existing knowledge graph database, includes entity-level comparison, relation-level comparison, and attribute-level comparison.

[0026] Among them, entity layer comparison includes semantic distance calculation, relation layer comparison includes topological consistency verification, and attribute layer comparison includes numerical threshold comparison.

[0027] Preferably, the time delay compensation in step S4 specifically includes the following:

[0028] The asynchronous dual-rate Kalman filter uses the slow-rate state output by the composite multimodal knowledge graph as the quantization update term and the fast-rate state output by the assembly equipment as the prediction driving term.

[0029] Extrapolation prediction is performed using the state transition matrix to reconstruct the pose estimate of the headphone workpiece at the current physical moment.

[0030] Preferably, the knowledge reasoning and assembly decision control in step S4 specifically includes the following:

[0031] Knowledge retrieval and task reasoning are performed based on the updated multimodal environmental knowledge graph. Key subgraph retrieval is performed based on task target input. Combined with process constraint reasoning and knowledge-guided reinforcement learning, path planning results and precision operation correction results are output.

[0032] When the acoustic feature corresponding to auditory perception is characterized as structural damage abnormality, the corresponding abnormal information is input as a high-priority new knowledge entity into step S3.

[0033] On the other hand, a system for constructing and dynamically updating environmental knowledge in a smart headphone factory is proposed, including the following:

[0034] A multimodal sensing acquisition module is used to collect multimodal sensing data in the headphone production environment. The multimodal sensing data includes at least visual data, auditory data, and tactile data.

[0035] A hierarchical knowledge construction module is used to establish a hierarchical semantic mapping system based on the multimodal perception data, including an entity layer, an agent layer, a perception layer, an action layer, and a task layer.

[0036] The knowledge fusion module is used to generate a corresponding multimodal environment knowledge graph based on the multimodal perception data, and to perform cross-modal alignment, association calculation and fusion representation on the multimodal environment knowledge graph to generate a composite multimodal knowledge graph;

[0037] The dynamic update module is used to construct a local relation graph network based on the comparison results with the hierarchical semantic mapping system when a new knowledge entity is detected, and obtain the updated multimodal environment knowledge graph through relation reasoning and dynamic synchronization; if no new knowledge entity is detected, the current composite multimodal knowledge graph is directly output.

[0038] The decision control module is used to perform time delay compensation, knowledge reasoning and assembly decision control based on the updated multimodal environment knowledge graph, and output path planning results and precision operation correction results for assembly equipment execution control.

[0039] The beneficial effects of this invention are as follows: This solution constructs a five-layer semantic mapping system consisting of an entity layer, an intelligent agent layer, a perception layer, an action layer, and a task layer. Combined with a multi-level expression framework of ontology, templates, and instances, it transforms component attributes, operating equipment attributes, sensor monitoring modalities, assembly skills, and complete production processes into structured knowledge nodes and relationships. Through a spatiotemporal hybrid association alignment model, a differentiated weighted cross-modal association network, and a parallel-guided multimodal knowledge fusion algorithm, visual, auditory, and tactile graphs are fused into a composite multimodal knowledge graph. Dynamic knowledge synchronization is achieved through entity-layer comparison, relation-layer comparison, attribute-layer comparison, and local relation graph network reasoning. Cross-rate time delay compensation is completed using an asynchronous dual-rate Kalman filter, and the assembly equipment execution control results are output by combining knowledge retrieval, task reasoning, process constraint reasoning, and knowledge-guided reinforcement learning. This solution is beneficial for improving the depth of environmental understanding, the timeliness of knowledge updates, the adaptability of flexible production, and assembly safety in smart headphone factories. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the steps of a method for constructing and dynamically updating environmental knowledge in a smart headphone factory, provided by this invention;

[0042] Figure 2 The flowchart illustrates the steps involved in generating a composite multimodal knowledge graph using a method for constructing and dynamically updating environmental knowledge in a smart headphone factory, as provided by this invention.

[0043] Figure 3 This invention provides a structural diagram of an environmental knowledge construction and dynamic update system in a smart headphone factory. Detailed Implementation

[0044] The following disclosure provides many different embodiments or examples for implementing various embodiments of the invention. To simplify the disclosure, specific embodiments are described below. Of course, these are merely examples and are not intended to limit the scope of the invention.

[0045] The embodiments of the invention will now be described in detail with reference to the accompanying drawings.

[0046] like Figure 1 As shown, a method for constructing and dynamically updating environmental knowledge in a smart headphone factory includes the following steps:

[0047] Step S1: Collect multimodal perception data in the headphone production environment. The multimodal perception data includes at least visual data, auditory data, and tactile data. Based on the multimodal perception data, establish a hierarchical semantic mapping system including an entity layer, an intelligent agent layer, a perception layer, an action layer, and a task layer.

[0048] Step S2: Generate a corresponding multimodal environment knowledge graph based on the multimodal perception data, and perform cross-modal alignment, association calculation and fusion representation on the multimodal environment knowledge graph to generate a composite multimodal knowledge graph;

[0049] Step S3: If a new knowledge entity is detected, a local relation graph network is constructed based on the comparison results with the hierarchical semantic mapping system, and the updated multimodal environment knowledge graph is obtained through relation reasoning and dynamic synchronization; if no new knowledge entity is detected, the current composite multimodal knowledge graph is directly output.

[0050] Step S4: Perform time delay compensation, knowledge reasoning and assembly decision control based on the updated multimodal environment knowledge graph, and output path planning results and precision operation correction results for assembly equipment execution control.

[0051] In one implementation, for precision assembly processes in a smart headphone factory, such as shell fastening, cable insertion, driver unit pressing, and detection switching, a visual perception branch collects headphone component pose information, an auditory perception branch collects assembly fastening sound information, and a tactile perception branch collects assembly pressure feedback information. These three types of perception results are then input into a hierarchical multimodal environment knowledge construction module. This module establishes a five-layer semantic mapping system: entity layer, agent layer, perception layer, action layer, and task layer. Combined with a multi-level expression framework consisting of domain ontology, process templates, and real-time instances, it transforms headphone component attributes, operating equipment attributes, sensor monitoring modalities, assembly skills, and complete production processes into structured knowledge nodes and relationships, forming visual, auditory, and tactile graphs, respectively. Subsequently, spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment are performed. Cross-modal relationship association is executed through a differentiated weighted cross-modal association network, and a composite multimodal knowledge graph is obtained through a parallel-guided multimodal knowledge fusion algorithm. During production line operation, if a new knowledge entity or a new knowledge triple is detected, entity-level comparison, relation-level comparison, and attribute-level comparison are performed between the new knowledge and the existing knowledge graph. Based on the comparison results, a local relation graph network is constructed, and new relational edges are generated through relational reasoning and association establishment engines to achieve dynamic synchronization of the knowledge graph. In this scheme, the judgment of a new knowledge triple is based on the following three criteria: the appearance of a new instance, a change in logical rule matching, and exceeding the physical parameter threshold. In one embodiment, the appearance of a new instance is manifested as a part change, a change in logical rule matching is manifested as a process logic offset, and exceeding the physical parameter threshold is manifested as a sudden change in acoustic emission characteristics indicating structural damage. The three criteria jointly trigger the real-time update of the knowledge graph.

[0052] An asynchronous dual-rate Kalman filter is set at the output of the knowledge graph. The slow-rate state output from the composite multimodal knowledge graph is used as the quantization update term, and the fast-rate state output from the end joint sensor of the assembly equipment is used as the one-step prediction driver. Extrapolation prediction is performed through the state transition matrix to reconstruct the delay-free pose estimate of the headphone workpiece at the current real physical moment. The updated multimodal environment knowledge graph is input into the knowledge retrieval and task reasoning module, and combined with process constraint reasoning and the assembly equipment execution control module, it outputs obstacle avoidance and path planning results as well as precision operation correction results. The asynchronous dual-rate Kalman filter is used to estimate the current motion state of the headphone workpiece to be assembled. The motion state includes at least spatial position and attitude information. The slow-rate input consists of the workpiece state information output from the composite multimodal knowledge graph, and the fast-rate input consists of the joint state information and / or end effector state information collected periodically by the robot control system. The pose estimate of the current moment output by the filter is input to the path planning unit or the precision operation correction unit to generate robot execution control commands that match the current physical moment. This solution utilizes a five-layer semantic mapping to unify the organization of physical objects, perception results, assembly actions, and production tasks. It establishes semantic complementarity between heterogeneous perception data through cross-modal association and parallel fusion, achieves incremental knowledge evolution using a relational graph network, and compensates for the time lag between knowledge output and control execution using an asynchronous dual-rate Kalman filter. Compared to existing methods that use visual, auditory, and tactile results in a fragmented manner, this solution incorporates part pose, fastening sound characteristics, assembly pressure changes, and process flow status into a unified knowledge framework. It maintains synchronization between the knowledge graph and the on-site status in scenarios involving new part introduction, new process switching, or process parameter adjustments, and reduces control overshoot caused by cross-rate delays. This improves the accuracy of environmental understanding, flexible production changeover adaptability, and precision assembly safety of assembly equipment in a smart headphone factory.

[0053] More specifically, the visual multimodal perception data includes at least the pose information of the headphone components;

[0054] The auditory multimodal perception data includes at least assembly and locking sound information;

[0055] The tactile multimodal sensing data includes at least assembly pressure feedback information;

[0056] The multimodal environmental knowledge graph includes a visual graph, an auditory graph, and a tactile graph.

[0057] In one implementation, visual multimodal perception data selects the pose information of earphone parts, specifically corresponding to the spatial position and posture relationship of the earphone shell, driver unit, cable, fixture, and related equipment in the assembly station; auditory multimodal perception data selects assembly fastening sound information, specifically corresponding to the acoustic features formed during shell fastening or cable insertion; tactile multimodal perception data selects assembly pressure feedback information, specifically corresponding to the changes in contact force during grasping, alignment, insertion, and pressing. Visual, auditory, and tactile maps are constructed based on these three types of perception data, respectively. The visual map is used to characterize the pose relationship of parts, fixtures, and equipment; the auditory map is used to characterize the relationship between fastening state and acoustic events; and the tactile map is used to characterize the relationship between assembly pressure, contact state, and action response. In the shell fastening operation embodiment, the visual map provides the positional relationship between the shell and the fixture, the auditory map provides the assembly events corresponding to the fastening sound features, and the tactile map provides the pressure feedback changes during the pressing process. All three jointly characterize the fastening condition during the fusion stage. Different modalities represent the same headphone assembly event with different dimensions. Pose information emphasizes spatial geometric relationships, latching sound information emphasizes event occurrence characteristics, and assembly pressure feedback information emphasizes contact response characteristics. Separate mapping can preserve the original semantic boundaries of each modality and provide stable input for subsequent cross-modal alignment and fusion. Compared to single visual detection or single pressure monitoring methods, this implementation can establish a more complete working condition representation structure in the context of precision headphone assembly, reducing environmental perception bias caused by the lack of single-modal information.

[0058] More specifically, in the hierarchical semantic mapping system, the entity layer is used to represent the attributes of headphone components, the intelligent agent layer is used to represent the attributes of operating devices, the perception layer is used to represent the sensor monitoring modes, the action layer is used to represent assembly skills, and the task layer is used to represent production processes.

[0059] By combining a multi-level representation framework of ontology, templates, and instances, component attributes, operating equipment attributes, sensor monitoring modes, assembly skills, and production processes are transformed into structured knowledge nodes and relationships.

[0060] In one implementation, the entity layer of the five-layer semantic mapping system is used to represent the attributes of headphone components, including at least the category and attribute information of production objects such as shells, driver units, and ribbon cables; the intelligent agent layer is used to represent the attributes of operating devices, including at least the device attributes of execution subjects or auxiliary subjects such as robotic arms, fixtures, and detection devices; the perception layer is used to represent sensor monitoring modes, including at least the monitoring results corresponding to vision, hearing, and touch; the action layer is used to represent assembly skills, including at least assembly actions such as grasping, alignment, insertion, lifting, and pressing; and the task layer is used to represent the complete production process, including at least the headphone assembly task flow formed by the arrangement of multiple action logics. This five-layer semantic mapping system, combined with a multi-level expression framework of ontology, templates, and instances, transforms objects, parameters, states, actions, and tasks into structured knowledge nodes and relationships. In the ribbon cable insertion embodiment, the entity layer records the object information of the ribbon cable and its connection position; the intelligent agent layer records the information of the robotic arm and fixture performing the insertion action; the perception layer records the sound and pressure feedback during the insertion process; the action layer records the logical connection between the insertion action and the lifting action; and the task layer records the position of the insertion action in the overall assembly process. In the precision manufacturing of headphones, objects, perceptions, actions, and tasks naturally exhibit hierarchical differences. Directly organizing knowledge using a planar structure can easily lead to fragmented process logic. Adopting a five-layer semantic mapping system can map underlying physical information layer by layer to upper-layer process semantics, forming a unified and reasonable representation of environmental knowledge. Compared to existing solutions that only build static knowledge structures around equipment or parts, this implementation can more comprehensively describe multi-agent relationships, perception relationships, and process relationships in the headphone production environment, enhancing the knowledge graph's ability to express complex assembly tasks.

[0061] More specifically, the multi-level expression framework includes a domain ontology unit, a process template unit, and a real-time instance unit;

[0062] Among them, the domain ontology unit is used to define the object categories, logical rules and process associations in the headphone production scenario, the process template unit is used to establish a parameterized expression template based on the standard operating procedure, and the real-time instance unit is used to receive real-time sensing data and fill it into the parameterized expression template.

[0063] In one implementation, the domain ontology unit is used to construct a meta-model for the headphone production scenario, defining object categories, logical rules, and process relationships; the process template unit is used to establish parameterized expression templates based on standard operating procedures (SOPs), forming reusable standard logical paradigms for the production processes of different headphone models; the real-time instance unit is used to receive real-time data from the five-layer semantic mapping system, fill the corresponding parameters into the parameterized expression templates, and generate real-time knowledge nodes with specific physical temporal attributes. The parameterized expression templates adopt a slotted structure containing process IDs, target entities, skill actions, perceptual constraints, and SOP judgment thresholds. Through a semantic anchoring mechanism, real-time perceived physical features are mapped and filled into the corresponding slots, thereby generating dynamic knowledge instances with process compliance conclusions.

[0064] During shell engagement, the domain ontology unit defines the basic relationships between the shell, fixture, robotic arm, and engagement event. The process template unit provides a parameterized template for the engagement operation. The real-time instance unit receives the shell pose obtained from visual perception, the engagement sound characteristics obtained from auditory perception, and the pressure feedback obtained from tactile perception, and forms a knowledge instance for the current workstation. The real-time instance unit may include real-time monitoring instances for engagement energy characteristics to determine whether the working condition is in the normal frequency band or the structural damage frequency band. In this embodiment, the domain ontology is responsible for providing stable semantic boundaries, the process template is responsible for carrying standard operating logic, and the real-time instance is responsible for injecting dynamic on-site states. The combination of these three can bind prior process knowledge with real-time physical information. Compared to simply relying on a static rule base or only recording raw sensor values, this approach can balance the standardization of knowledge expression with on-site adaptability, improving the scalability and dynamic adaptation capabilities of the multimodal environment knowledge graph.

[0065] like Figure 2 As shown, more specifically, step S2, which generates a composite multimodal knowledge graph, includes the following steps:

[0066] Spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment are performed on the multimodal environment knowledge graph to obtain the aligned multimodal environment knowledge graph;

[0067] Based on the aligned multimodal environment knowledge graph, the interaction weights between entities of different modalities are calculated using a differentiated weighted cross-modal association network;

[0068] Based on the obtained interaction weights, the aligned multimodal environment knowledge graph is fused using a parallel guided multimodal knowledge fusion algorithm to generate the composite multimodal knowledge graph.

[0069] In one implementation, the driving unit pose entities in the visual atlas, the latching or contact acoustic events in the auditory atlas, and the assembly pressure changes in the tactile atlas are initially aligned using a spatiotemporal hybrid association alignment model. Then, interactive weights are established through a differentiated weighted cross-modal association network. Finally, a composite multimodal knowledge graph capable of simultaneously representing objects, actions, parameters, and states is generated in the parallel guided fusion stage. Since visual, auditory, and tactile observations of the same assembly process differ, the cross-modal relationship calculation results only possess technological significance after alignment is achieved at the spatial, temporal, and entity levels. Parallel guided fusion can compensate for single-modal deficiencies by utilizing complementary relationships between modalities within a unified semantic space. Compared to directly splicing multiple sensor results or simply superimposing them at the result layer, this implementation can more stably establish correspondences between multimodal entities, improving the knowledge graph's comprehensive expressive ability of the headphone assembly environment.

[0070] More specifically, when performing spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment on a multimodal environment knowledge graph, the following steps are also included:

[0071] Graph convolutional neural networks are used to extract topological information of multimodal environmental knowledge graphs;

[0072] A multi-head cross-attention mechanism is used to calculate the interaction weights between entities of different modalities;

[0073] A differentiated weight allocation matrix is ​​generated based on the current process status, and the weighted features are projected into the joint embedding latent space to perform parallel feature aggregation.

[0074] In one implementation, during cable insertion, a graph convolutional neural network extracts the structural relationships between the cable, fixture, and equipment from a visual graph, the acoustic event relationships during insertion from an auditory graph, and the assembly pressure changes and action response relationships from a tactile graph. A multi-head cross-attention mechanism performs interactive weight calculations on the three types of relationships. A differentiated weight allocation matrix weights different modal features according to the current process state, and then a joint embedding latent space completes parallel feature aggregation. In this implementation, the graph convolutional neural network is suitable for processing node relationships and topological relationships in graph structure data, the multi-head cross-attention mechanism is suitable for handling the importance differences between different modalities, the differentiated weight allocation can avoid the expression bias caused by fixed fusion weights under different process states, and the joint embedding latent space is conducive to completing the unified mapping of heterogeneous features.

[0075] More specifically, in step S3, the process of comparing the new knowledge entity with the existing knowledge graph database includes entity layer comparison, relation layer comparison and attribute layer comparison.

[0076] Among them, entity layer comparison includes semantic distance calculation, relation layer comparison includes topological consistency verification, and attribute layer comparison includes numerical threshold comparison.

[0077] In one implementation, when a new component is introduced, the system detects the introduction of a new part into the production line, firstly by comparing the entity layer to determine the semantic similarity between the part and existing part categories, then by comparing the relationship layer to determine what kind of topological relationship should be established between the part and fixtures, equipment, actions, and tasks, and finally by comparing the attribute layer to determine whether its key parameters conform to the established process boundaries. Based on this, local relationship graph network reasoning and synchronous updates are completed. Since new knowledge in dynamic scenarios does not always manifest as purely newly added nodes, but may also manifest as new relationships of known entities or new states of known attributes, relying solely on a single-level comparison is insufficient to accurately locate the knowledge change location. Three-level comparison can perform hierarchical parsing of new knowledge, and the local relationship graph network can complete incremental updates without restarting the system. Compared to whole-graph reconstruction or manual maintenance of the knowledge base, this implementation can improve the efficiency and consistency of knowledge updates in scenarios of high-frequency production changes and process adjustments in headphone factories. When using numerical threshold comparison for attribute layer comparison, the numerical threshold mainly originates from the standard process range defined by the standard operating procedure, and is dynamically calibrated with the statistical characteristics of historical process parameters.

[0078] More specifically, the time delay compensation in step S4 includes the following:

[0079] The asynchronous dual-rate Kalman filter uses the slow-rate state output by the composite multimodal knowledge graph as the quantization update term and the fast-rate state output by the assembly equipment as the prediction driving term.

[0080] Extrapolation prediction is performed using the state transition matrix to reconstruct the pose estimate of the headphone workpiece at the current physical moment.

[0081] In one implementation, during the press-fitting of the drive unit, the multimodal fusion result includes graph state information such as workpiece pose, pressure changes, and acoustic events. However, there is a cross-rate difference between the output frequency of the graph calculation result and the drive frequency of the assembly equipment control layer. During the period of computational delay in the generation of the fused graph, the asynchronous dual-rate Kalman filter uses the state transition matrix to perform extrapolation prediction on the workpiece state, enabling the control module to receive a pose estimation result that is more consistent with the current real physical moment. The pose estimation result is obtained through the assembly equipment joint sensors, which are derived from the built-in high-precision encoders of the robot servo motors. The reconstructed pose estimation value is input into the trajectory tracking feedback closed loop of the motion control module as a reference input for high-frequency real-time deviation compensation. In this implementation, the time scales of the multimodal knowledge graph calculation stage and the assembly equipment control stage are different. If the slow-rate graph state is directly sent to the fast-rate control loop, the control quantity will be affected by lag. The asynchronous dual-rate Kalman filter compensates for the cross-rate time delay by combining quantization updates and predictive driving, which can reduce the impact of delayed states on control accuracy.

[0082] More specifically, the knowledge reasoning and assembly decision control in step S4 includes the following:

[0083] Knowledge retrieval and task reasoning are performed based on the updated multimodal environmental knowledge graph. Key subgraph retrieval is performed based on task target input. Combined with process constraint reasoning and knowledge-guided reinforcement learning, path planning results and precision operation correction results are output.

[0084] When the acoustic feature corresponding to auditory perception is characterized as structural damage abnormality, the corresponding abnormal information is input as a high-priority new knowledge entity into step S3.

[0085] In one implementation, the knowledge reasoning and assembly equipment decision control in step S4 includes knowledge retrieval and task reasoning, key subgraph retrieval, process constraint reasoning, knowledge-guided reinforcement learning, obstacle avoidance and path planning, and precision operation correction. The reinforcement learning takes the environmental semantic state and task objective extracted from the composite knowledge graph as input, outputs the optimal precision operation decision instruction, and uses the SOP process specifications and physical parameters embedded in the graph as execution constraints.

[0086] After receiving the task target input, the system extracts key subgraphs related to parts, fixtures, equipment, actions, processes, events, attributes, parameters, and states from the real-time composite multimodal knowledge graph, and performs inference by combining process knowledge, state knowledge, object knowledge, and task knowledge. During the assembly equipment execution control phase, the obstacle avoidance and path planning module outputs a safe motion trajectory based on the knowledge inference results, and the precision operation correction module performs residual correction by combining the delay-free pose estimation data output by the asynchronous dual-rate Kalman filter with pose and pressure feedback. Furthermore, when the fastening energy characteristics collected by the acoustic emission sensor abruptly change from the normal frequency band to the preset structural damage frequency band, an "structural damage" abnormal node can be generated in the relationship graph network, and subsequent assembly actions can be forcibly intercepted through the local network inference mechanism, issuing a control command to stop the robotic arm pressing. During the shell fastening process, the system performs subgraph retrieval and process constraint inference based on the shell, fixture, pressure, detection, and pressing relationships in the knowledge graph, and stops pressing when a structural damage abnormality is identified. In this embodiment, the composite multimodal knowledge graph not only serves as an environmental representation tool but also supports task-level reasoning and action-level control. Key subgraph retrieval establishes a correspondence between task objectives and local operating conditions, process constraint reasoning introduces production rules into the control process, and the abnormal node mechanism transforms quality risks into control constraints. Compared to methods that do not directly apply knowledge reasoning results to assembly equipment control or rely solely on single threshold alarms, this approach balances assembly efficiency, path safety, and structural damage prevention capabilities.

[0087] like Figure 3 As shown, a system for constructing and dynamically updating environmental knowledge in a smart headphone factory includes the following:

[0088] A multimodal sensing acquisition module is used to collect multimodal sensing data in the headphone production environment. The multimodal sensing data includes at least visual data, auditory data, and tactile data.

[0089] A hierarchical knowledge construction module is used to establish a hierarchical semantic mapping system based on the multimodal perception data, including an entity layer, an agent layer, a perception layer, an action layer, and a task layer.

[0090] The knowledge fusion module is used to generate a corresponding multimodal environment knowledge graph based on the multimodal perception data, and to perform cross-modal alignment, association calculation and fusion representation on the multimodal environment knowledge graph to generate a composite multimodal knowledge graph;

[0091] The dynamic update module is used to construct a local relation graph network based on the comparison results with the hierarchical semantic mapping system when a new knowledge entity is detected, and obtain the updated multimodal environment knowledge graph through relation reasoning and dynamic synchronization; if no new knowledge entity is detected, the current composite multimodal knowledge graph is directly output.

[0092] The decision control module is used to perform time delay compensation, knowledge reasoning and assembly decision control based on the updated multimodal environment knowledge graph, and output path planning results and precision operation correction results for assembly equipment execution control.

[0093] The operation and effect of the intelligent headphone factory environmental knowledge construction and dynamic update system of the present invention are consistent with the above-mentioned intelligent headphone factory environmental knowledge construction and dynamic update method. Therefore, the description of the intelligent headphone factory environmental knowledge construction and dynamic update system will not be repeated here.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using necessary general-purpose hardware platforms, or it can be implemented through a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for constructing and dynamically updating environmental knowledge in a smart headphone factory, characterized in that, Includes the following steps: Step S1: Collect multimodal perception data from the headphone production environment. The multimodal perception data includes at least visual, auditory, and tactile data. Based on the multimodal perception data, establish a hierarchical semantic mapping system including an entity layer, an agent layer, a perception layer, an action layer, and a task layer. Step S2: Generate a corresponding multimodal environment knowledge graph based on the multimodal perception data. Perform cross-modal alignment, association calculation, and fusion representation on the multimodal environment knowledge graph to generate a composite multimodal knowledge graph. Step S3: If a new knowledge entity is detected, construct a local relational graph network based on the comparison results with the hierarchical semantic mapping system. Obtain the updated multimodal environment knowledge graph through relational reasoning and dynamic synchronization. If no new knowledge entity is detected, directly output the current composite multimodal knowledge graph. Step S4: Perform time delay compensation, knowledge reasoning, and assembly decision control based on the updated multimodal environment knowledge graph. Output path planning results and precision operation correction results for use in assembly equipment execution control.

2. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, The visual multimodal perception data includes at least the pose information of the headphone components; the auditory multimodal perception data includes at least the assembly and fastening sound information. The tactile multimodal perception data includes at least assembly pressure feedback information; the multimodal environmental knowledge graph includes a visual graph, an auditory graph, and a tactile graph.

3. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, In the hierarchical semantic mapping system, the entity layer is used to represent the attributes of headphone components, the intelligent agent layer is used to represent the attributes of operating devices, the perception layer is used to represent the sensor monitoring modes, the action layer is used to represent assembly skills, and the task layer is used to represent production processes. By combining a multi-level representation framework of ontology, templates, and instances, component attributes, operating equipment attributes, sensor monitoring modes, assembly skills, and production processes are transformed into structured knowledge nodes and relationships.

4. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 3, characterized in that, The multi-level expression framework includes a domain ontology unit, a process template unit, and a real-time instance unit. The domain ontology unit is used to define object categories, logical rules, and process associations in the headphone production scenario. The process template unit is used to establish a parameterized expression template based on standard operating procedures. The real-time instance unit is used to receive real-time sensing data and populate the parameterized expression template.

5. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, Step S2, which generates the composite multimodal knowledge graph, specifically includes the following steps: aligning the multimodal environment knowledge graph spatial coordinates, synchronizing temporal features, and performing preliminary entity alignment to obtain an aligned multimodal environment knowledge graph; based on the aligned multimodal environment knowledge graph, calculating the interaction weights between entities of different modalities using a differentiated weighted cross-modal association network; and according to the obtained interaction weights, fusing the aligned multimodal environment knowledge graph using a parallel guided multimodal knowledge fusion algorithm to generate the composite multimodal knowledge graph.

6. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 5, characterized in that, When performing spatial coordinate alignment, temporal feature synchronization, and preliminary entity alignment on a multimodal environment knowledge graph, the following steps are also included: extracting the topological structure information of the multimodal environment knowledge graph using a graph convolutional neural network; calculating the interaction weights between entities of different modalities using a multi-head cross-attention mechanism; generating a differentiated weight allocation matrix based on the current process state, and projecting the weighted features into the joint embedding latent space to perform parallel feature aggregation.

7. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, In step S3, the process of comparing the new knowledge entity with the existing knowledge graph database includes entity-level comparison, relation-level comparison, and attribute-level comparison. Entity-level comparison includes semantic distance calculation, relation-level comparison includes topological consistency verification, and attribute-level comparison includes numerical threshold comparison.

8. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, The time delay compensation in step S4 specifically includes the following: The asynchronous dual-rate Kalman filter uses the slow-rate state output from the composite multimodal knowledge graph as the quantization update term and the fast-rate state output from the assembly equipment as the prediction driving term; it performs extrapolation prediction through the state transition matrix to reconstruct the pose estimate of the headphone workpiece at the current physical moment.

9. The method for constructing and dynamically updating environmental knowledge in a smart headphone factory according to claim 1, characterized in that, The knowledge reasoning and assembly decision control in step S4 specifically includes the following: Knowledge retrieval and task reasoning are performed based on the updated multimodal environmental knowledge graph. Key subgraph retrieval is performed based on the task target input, and path planning results and precision operation correction results are output by combining process constraint reasoning and knowledge-guided reinforcement learning. When the acoustic feature representation corresponding to auditory perception is structural damage anomaly, the corresponding anomaly information is input as a high-priority new knowledge entity in step S3.

10. A system for constructing and dynamically updating environmental knowledge in a smart headphone factory, characterized in that, Includes the following: a multimodal perception acquisition module, used to acquire multimodal perception data in the headphone production environment, wherein the multimodal perception data includes at least visual data, auditory data, and tactile data; and a hierarchical knowledge construction module, used to establish a hierarchical semantic mapping system based on the multimodal perception data, including an entity layer, an intelligent agent layer, a perception layer, an action layer, and a task layer. The knowledge fusion module generates a corresponding multimodal environment knowledge graph based on the multimodal perception data, performs cross-modal alignment, association calculation, and fusion representation on the multimodal environment knowledge graph, and generates a composite multimodal knowledge graph. The dynamic update module constructs a local relation graph network based on the comparison results with the hierarchical semantic mapping system when a new knowledge entity is detected, and obtains the updated multimodal environment knowledge graph through relation reasoning and dynamic synchronization. If no new knowledge entity is detected, the current composite multimodal knowledge graph is directly output. The decision control module performs time delay compensation, knowledge reasoning, and assembly decision control based on the updated multimodal environment knowledge graph, and outputs path planning results and precision operation correction results for assembly equipment execution control.

Images