Natural semantic-based intelligent earphone factory multi-modal environment perception method and system
Patent Information
- Application Number
- CN202611116848.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-27
- Publication Date
- 2026-10-27
AI Technical Summary
现有的耳机工厂感知方案普遍停留在图像检测、声纹识别或压力阈值判断的单点处理层面，跨模态数据之间缺乏稳定的事件对齐机制，导致视觉结果、操作声音与扣合受力信息难以形成统一语义解释
[0047]本发明的有益效果是：本发明通过构建视觉、声学、力觉与生产文本协同驱动的多模态环境感知框架，将耳机装配现场的离散物理现象转换为具备工艺逻辑含义的结构化语义结果。以视觉关键事件作为跨模态对齐锚点，能够在同一装配动作下建立图像、声音和受力数据的样本对应关系，降低人工标注依赖程度。并且，通过耳机组装工艺知识图谱承载零部件属性、工序关系与质量参数约束，再结合统一语义空间映射和规程逻辑校验，实现环境实体、交互行为和任务状态的联合判定。不仅能提升精密装配场景的工艺合规性判断能力和异常识别精度，还能够达到在线监控和过程追溯的目的。
Smart Images
Figure CN122986871A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent manufacturing technology, and specifically relates to a multimodal environment perception method and system for a smart headphone factory based on natural semantics. Background Technology
[0002] As consumer electronics manufacturing processes continue to evolve towards miniaturization, precision, and flexibility, the headphone assembly process has gradually shifted from traditional segmented workstations to a data-driven intelligent manufacturing stage. Given the complex working conditions in headphone production, relying solely on a single sensor is insufficient to fully represent the actual processing status. Therefore, industrial sites are increasingly incorporating multi-source information, including visual, acoustic, pressure, tactile, and textual procedures, to enhance the comprehensive understanding of object status, operational behavior, and process progress. How to unify the representation, correlation, and semantic output of this real-time sensing data and process knowledge has become a crucial technical direction for improving the intelligent assembly monitoring capabilities, process tracking capabilities, and closed-loop process control capabilities of headphones. Existing headphone factory sensing solutions generally remain at the single-point processing level of image detection, voiceprint recognition, or pressure threshold judgment. The lack of a stable event alignment mechanism between cross-modal data makes it difficult to form a unified semantic interpretation of visual results, operational sounds, and fastening force information. Furthermore, manual annotation methods are insufficiently adaptable to assembly sites with frequent process changes and a wide variety of parts, resulting in high sample construction costs, inconsistent annotation scales, and limited model transfer capabilities. Furthermore, production procedures, quality standards, and design parameters often exist independently in the form of documents, making it difficult to establish a direct correspondence between the output of the perception model and the process logic. Although the system can identify local physical phenomena, it is difficult to determine whether the process is matched, whether the parameters are compliant, or whether the task is completed. The availability of subsequent quality traceability and anomaly warning is significantly limited. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention provides a method and system for multimodal environment perception in smart headphone factories based on natural language processing, in order to solve the aforementioned technical problems.
[0004] On the one hand, a multimodal environment perception method for smart headphone factories based on natural semantics is proposed, including the following steps:
[0005] Step S1: Collect visual data, multi-source physical signals, and production text data of the target workstation. The multi-source physical signals include at least operation sound signals and force signals.
[0006] Step S2: Extract features from the visual data and multi-source physical signals to obtain visual features, acoustic features and force features. Based on the visual data, identify key visual events in the headphone assembly process. Using the key visual events as logical anchors, perform synchronous segmentation, mutual labeling and cross-modal temporal alignment on the acoustic features and / or force features in the corresponding time intervals to obtain a cross-modal fusion feature vector.
[0007] Step S3: Based on the production text data, extract entity recognition results, process relationship features, and physical parameter features to construct a headphone assembly process knowledge graph. Perform vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. Map the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, perform knowledge anchor matching and cross-granularity semantic aggregation to obtain a set of perceptual semantic tags. The set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states.
[0008] Step S4: Based on the SOP procedure logic verifier, perform procedure conformity judgment on the set of perceptual semantic tags, and evaluate the confidence level of the judgment result; when the confidence level reaches the preset threshold, output a structured perception report containing environmental entities, interactive behaviors and task status.
[0009] Preferably, in step S1,
[0010] The visual data refers to single-view or multi-view image data of headphone components during the assembly process.
[0011] The multi-source physical signals are operation sound data, fastening force data and / or tactile data during the earphone assembly process;
[0012] The production text data includes headphone design specifications, standard operating procedure documents, operating logic, and procedure constraint relationship data.
[0013] Preferably, in step S2, the feature extraction is performed using a multimodal feature parallel extraction method, specifically including the following steps:
[0014] The pose information and three-dimensional geometric information of the headphone components are extracted using an object detection model combined with a point cloud feature extraction architecture, and used as the visual features.
[0015] The operation tone data is encoded using a masked audio autoencoder to extract the acoustic features;
[0016] The fastening force data and / or tactile data are encoded using a selective state-space model to extract force features.
[0017] Preferably, in step S2, when using visual key events as logical anchors to synchronously segment acoustic features and / or force features, the specific steps include:
[0018] Based on at least one event from visual data, such as component contact, fastening completion, dispensing start, or welding start, determine the timing of the visual key event.
[0019] Based on the time of occurrence of the visual key event, the acoustic features and force features are divided into time windows to form acoustic segments and force segments corresponding to the visual key event.
[0020] Based on the event labels corresponding to the visual key events, the acoustic segments and force segments are labeled and associated to establish cross-modal sample correspondence between the visual modality and the acoustic and force modalities.
[0021] Preferably, step S2, when performing cross-modal timing alignment, specifically includes the following steps:
[0022] Visual features, acoustic features, and force features belonging to the same visual key event are time-synchronized and semantically corresponded to constrain the sampling offset between different sensors, and the aligned modal features are mapped to a joint representation with consistent dimensions.
[0023] The cross-modal fusion feature vector is a joint vector representation obtained by fusing aligned visual features, acoustic features, and force features.
[0024] Preferably, the assembly process knowledge graph in step S3 is established through the following steps:
[0025] The entity recognition results include physical dimensions, component materials, and technical specifications extracted from the headphone design specifications.
[0026] The process relationship features include the operation steps, operation logic, and operation sequence relationships extracted from the standard operating procedure document;
[0027] The physical parameter features include quality parameters, qualified product judgment information, and defective product judgment information extracted from the quality judgment standard database;
[0028] An assembly process knowledge graph is established based on the extracted entity recognition results, process relationship features, and physical parameter features.
[0029] Preferably, the entity recognition result includes at least PCB nodes, drive nodes, and housing nodes; the process relationship features include at least welding nodes, dispensing nodes, and fastening nodes; and the physical parameter features include at least pressure nodes, wettability nodes, and curve feature nodes.
[0030] Establish associated edges between nodes that have process dependencies, structural fit relationships, or parameter constraints;
[0031] An assembly process knowledge graph is formed based on associated edges. The RotatE model is used to perform representation learning on the assembly process knowledge graph to obtain vector features. The node association weights are calculated using a joint graph attention network.
[0032] The guiding knowledge semantic vector is output based on the vector features and node association weights.
[0033] Preferably, step S3, which maps the cross-modal fusion feature vector and the guiding knowledge semantic vector to a unified semantic representation space, specifically includes the following steps:
[0034] The cross-modal fusion feature vector and the guided knowledge semantic vector are input into a multi-granularity semantic mapping model and mapped to a unified semantic representation space;
[0035] The semantic dimension in the unified semantic representation space is divided into three levels: environmental entity perception, interactive behavior modeling, and task state decomposition.
[0036] The semantic association between real-time perceived features and corresponding nodes or associated edges in the headphone assembly process knowledge graph is determined by knowledge anchor matching, and the semantic perception results corresponding to environmental entities, interactive behaviors and task states are output based on cross-granularity semantic aggregation, which serve as the set of perceived semantic tags.
[0037] During the mapping process, inconsistent semantics are corrected and feedback is provided based on process rule constraints.
[0038] Preferably, the procedure conformity determination in step S4 specifically includes the following steps:
[0039] The environmental entities, interactive behaviors, and task states in the set of perceptual semantic tags are compared with the step sequence constraints, object matching constraints, and parameter threshold constraints in the headphone assembly process specification to determine whether they match.
[0040] When the judgment result is inconsistent, the knowledge graph association relationship is updated, the perception parameters are adjusted, and the cross-modal labels are realigned.
[0041] When the judgment result is satisfactory, an assembly process prompt message is generated, and semantic synthesis is performed on the set of perceptual semantic tags based on the natural language big data model. The confidence level of the semantic synthesis result is then evaluated.
[0042] On the other hand, a multimodal environment perception system for a smart headphone factory based on natural semantics is proposed, including the following:
[0043] A multimodal data acquisition module is used to acquire visual data, multi-source physical signals, and production text data of the target workstation. The multi-source physical signals include at least operation sound signals and force signals.
[0044] The feature extraction and mutual labeling spatiotemporal correlation module is used to extract features from the visual data and multi-source physical signals to obtain visual features, acoustic features and force features. Based on the visual data, it identifies key visual events in the headphone assembly process, and uses the key visual events as logical anchors to perform synchronous segmentation, mutual labeling and cross-modal temporal alignment of the acoustic features and / or force features in the corresponding time intervals to obtain cross-modal fusion feature vectors.
[0045] The knowledge graph construction and semantic mapping module is used to extract entity recognition results, process relationship features, and physical parameter features from production text data to construct a headphone assembly process knowledge graph. It then performs vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. The module maps the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, performs knowledge anchor matching and cross-granularity semantic aggregation, and obtains a set of perceptual semantic tags. This set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states.
[0046] The semantic verification and result synthesis module performs a procedure conformity judgment on the set of perceived semantic tags based on the SOP procedure logic verifier, and evaluates the confidence level of the judgment result; when the confidence level reaches a preset threshold, it outputs a structured perception report containing environmental entities, interactive behaviors and task status.
[0047] The beneficial effects of this invention are as follows: By constructing a multimodal environmental perception framework driven by vision, acoustics, force perception, and production text, this invention transforms the discrete physical phenomena of the headphone assembly site into structured semantic results with process logic meaning. Using key visual events as cross-modal alignment anchors, it can establish sample correspondences between images, sounds, and force data under the same assembly action, reducing the reliance on manual annotation. Furthermore, by using a headphone assembly process knowledge graph to carry component attributes, process relationships, and quality parameter constraints, combined with unified semantic space mapping and procedural logic verification, it achieves joint judgment of environmental entities, interactive behaviors, and task states. This not only improves the ability to judge process compliance and the accuracy of anomaly identification in precision assembly scenarios but also achieves the purpose of online monitoring and process traceability. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 A flowchart illustrating the steps of the multimodal environment perception method for a smart headphone factory based on natural semantics provided by this invention;
[0050] Figure 2 A flowchart illustrating the steps involved in obtaining guided knowledge semantic vectors for the multimodal environment perception method for a smart headphone factory based on natural semantics provided by this invention.
[0051] Figure 3 This is a schematic diagram of the structure of the intelligent headphone factory multimodal environment perception system based on natural semantics provided by the present invention. Detailed Implementation
[0052] The following disclosure provides many different embodiments or examples for implementing various embodiments of the present invention. To simplify the disclosure, specific embodiments are described below. Of course, these are merely examples and are not intended to limit the scope of the invention.
[0053] The embodiments of the invention will now be described in detail with reference to the accompanying drawings.
[0054] like Figure 1 , Figure 2 As shown, the multimodal environment perception method for a smart headphone factory based on natural language processing includes the following steps:
[0055] Step S1: Collect visual data, multi-source physical signals, and production text data of the target workstation. The multi-source physical signals include at least operation sound signals and force signals.
[0056] Step S2: Extract features from the visual data and multi-source physical signals to obtain visual features, acoustic features and force features. Based on the visual data, identify key visual events in the headphone assembly process. Using the key visual events as logical anchors, perform synchronous segmentation, mutual labeling and cross-modal temporal alignment on the acoustic features and / or force features in the corresponding time intervals to obtain a cross-modal fusion feature vector.
[0057] Step S3: Based on the production text data, extract entity recognition results, process relationship features, and physical parameter features to construct a headphone assembly process knowledge graph. Perform vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. Map the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, perform knowledge anchor matching and cross-granularity semantic aggregation to obtain a set of perceptual semantic tags. The set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states.
[0058] Step S4: Based on the SOP procedure logic verifier, perform procedure conformity judgment on the set of perceptual semantic tags, and evaluate the confidence level of the judgment result; when the confidence level reaches the preset threshold, output a structured perception report containing environmental entities, interactive behaviors and task status.
[0059] In one specific embodiment, this invention is deployed at the precision assembly station of a smart headphone factory to perform multimodal semantic perception, process logic verification, and result output for the headphone processing environment. The original specification has been published. Multi-view visual sensors, array microphones, and tactile and pressure sensors are connected to the production site. Operation manuals and standard operating procedure documents serve as textual knowledge sources input into the system. The system first uses a multimodal mutual annotation self-supervised learning network to autonomously extract features from visual, acoustic, and pressure signals and perform spatiotemporal correlation matching. Then, it uses a headphone precision assembly process knowledge graph to complete environmental entity perception, interactive behavior modeling, and task state decomposition, finally outputting a structured perception report. This solution converts multi-source physical signals into a unified semantic representation and uses process knowledge to perform semantic anchoring of physical features. Furthermore, it uses procedural logic to implement consistency constraints on the mapping results, enabling the underlying perception results to not only identify physical phenomena but also correspond to the process meaning within the headphone manufacturing scenario. This embodiment can solve problems such as the small size of parts, heterogeneous physical signals, high cost of manual annotation, and the disconnect between physical phenomena and process logic in headphone precision machining, improving the semantic understanding capabilities of headphone factories in processing monitoring, quality traceability, anomaly warning, and flexible scheduling.
[0060] More specifically, in step S1,
[0061] The visual data refers to single-view or multi-view image data of headphone components during the assembly process.
[0062] The multi-source physical signals are operation sound data, fastening force data and / or tactile data during the earphone assembly process;
[0063] The production text data includes headphone design specifications, standard operating procedure documents, operating logic, and procedure constraint relationship data.
[0064] In one specific embodiment, the data acquisition in step S1 employs a multi-view visual sensor, an array microphone, a tactile sensor, and a pressure sensor. The multi-view visual sensor is used to acquire image information of the headphone components during the assembly process. Operational sound signals are used to characterize the acoustic feedback generated during processes such as fastening, contact, and welding triggering. Pressure and tactile signals are used to characterize the mechanical changes during pressing, fastening, or contact. Operating manuals, SOPs, and other production line text data are used to provide process rules and operational logic. This acquisition method covers spatial appearance information, transient acoustic information, contact force information, and process logic information at the input level, enabling different modalities to assume different semantic expression functions in subsequent processing stages. The visual path corresponds to the target pose and spatial relationships, the acoustic path corresponds to triggering behavior and transient changes, the pressure and tactile paths correspond to contact states and mechanical actions, and the text path corresponds to procedural constraints and domain semantic boundaries. This design enables the system to collaboratively characterize the headphone assembly state from the physical, action, and process layers. It maintains high perceptual stability under complex noise and partial occlusion conditions and forms a complete input suitable for subsequent mutual annotation, atlas construction, and procedure verification.
[0065] More specifically, in step S2, feature extraction is performed using a multimodal feature parallel extraction method, which specifically includes the following steps:
[0066] The pose information and three-dimensional geometric information of the headphone components are extracted using an object detection model combined with a point cloud feature extraction architecture, and used as the visual features.
[0067] The operation tone data is encoded using a masked audio autoencoder to extract the acoustic features;
[0068] The fastening force data and / or tactile data are encoded using a selective state-space model to extract force features.
[0069] In one specific embodiment, the feature extraction in step S2 employs a multimodal parallel feature extraction mechanism. The visual path uses the YOLOv13 real-time detection model to accurately locate the pose of the earphone's minute components, and combines this with the PointNeXt architecture to extract three-dimensional geometric features, forming visual features usable for assembly recognition. The acoustic path uses a masked audio autoencoder to extract deep acoustic signature features with high generalization ability to represent the acoustic responses of fastening sounds, contact pulse sounds, and other processes. The pressure path uses a selective state-space model to model the high-frequency pressure numerical flow, capturing the transient dynamic features of the precision pressing moment. These three paths process data with different physical properties in parallel, avoiding the problem that a single model cannot simultaneously account for spatial structure, acoustic pulses, and continuous mechanical changes. A unified judgment is then formed through subsequent spatiotemporal correlation matching and semantic mapping. The statistical characteristics and semantic contributions of the visual, acoustic, and pressure data differ; only targeted modeling can preserve the most discriminative patterns for each. This solution improves the ability to perceive the pose of minute components, recognize instantaneous process triggers, and express the quality of fastening / pressing, thereby enhancing the overall perception accuracy and robustness in precision earphone assembly scenarios.
[0070] More specifically, in step S2, when using visual key events as logical anchors to synchronously segment acoustic features and / or force features, the following steps are included:
[0071] Based on at least one event from visual data, such as component contact, fastening completion, dispensing start, or welding start, determine the timing of the visual key event.
[0072] In this embodiment, the visual key events are mainly set based on the physical mutation nodes defined in the headphone assembly SOP logic. The following are some examples of visual key events:
[0073] Component contact: Physical contact between the moving coil unit and the acoustic cavity base;
[0074] The earphone shell clips are fully locked in place.
[0075] Assembly complete: The battery is installed in the battery compartment.
[0076] Process start-up: The dispensing needle reaches the dispensing position, and the soldering head contacts the PCB solder joint;
[0077] Precision pressing completed: The pressing machine reaches the preset displacement depth or the pressure holding start point;
[0078] Test connection established: The headphone charging contacts make contact with the test fixture.
[0079] For example, in one implementation, the visual key event of engagement completion is determined in the following way:
[0080] (a) Select the target bounding box and displacement change as the judgment objects.
[0081] (b) The visual path detects that the overlap ratio of the target bounding boxes of the upper and lower shells of the earphone is greater than or equal to the set overlap threshold; the displacement change of the three-dimensional pose coordinates of the component center point between consecutive frames is less than or equal to the change threshold.
[0082] (c) The states of the two judgment objects mentioned above are maintained simultaneously. When the frame is above a certain value, it is determined that the snapping completion event has occurred.
[0083] This judgment logic eliminates false triggers caused by robotic arm vibration.
[0084] Based on the time of occurrence of the visual key event, the acoustic features and force features are divided into time windows to form acoustic segments and force segments corresponding to the visual key event.
[0085] Based on the event labels corresponding to the visual key events, the acoustic segments and force segments are labeled and associated to establish cross-modal sample correspondence between the visual modality and the acoustic and force modalities.
[0086] In one specific embodiment, during the synchronous segmentation and mutual labeling in step S2, key events reflecting the occurrence of process actions are first detected from the visual data. These key events are then used as logical anchors to segment the synchronously generated sound and pressure segments. A cross-modal sample correspondence is established through a mutual labeling enhancement loss function. Industrial sound and pressure data often lack clear semantic boundaries, while visual events are relatively easier to observe and locate. Using visual events as the segmentation starting point can reduce label contamination caused by heterogeneous signal misalignment. In this embodiment, the consistency of training samples is enhanced, enabling the online recognition stage to more accurately attribute the perception results to specific process actions, improving the stability of cross-modal fusion and the credibility of semantic expression.
[0087] More specifically, in step S2, cross-modal timing alignment includes the following steps:
[0088] Visual features, acoustic features, and force features belonging to the same visual key event are time-synchronized and semantically corresponded to constrain the sampling offset between different sensors, and the aligned modal features are mapped to a joint representation with consistent dimensions.
[0089] The cross-modal fusion feature vector is a joint vector representation obtained by fusing aligned visual features, acoustic features, and force features.
[0090] In one specific embodiment, the cross-modal temporal alignment in step S2 is completed by the spatiotemporal feature association matching module. After synchronous segmentation driven by mutual annotation, visual features, acoustic features, and pressure features are further segmented in this scheme by adaptive segmentation driven by the action semantics of visual path perception. The input spatiotemporal feature association matching module performs sampling delay correction and cross-modal consistency correction to obtain the cross-modal fused feature vector. Different sensors have different sampling frequencies, response delays, and physical observation dimensions. Without consistency alignment, the same process action may correspond to different time positions and different spatial meanings in different modalities, making it difficult to form a stable joint representation. Through this alignment mechanism, visual pose, acoustic pulse, and pressure changes can be more reliably mapped to the same process action, improving the accuracy of subsequent knowledge anchor matching, semantic aggregation, and procedure verification, and enhancing the robustness of the system under complex working conditions.
[0091] More specifically, the assembly process knowledge graph in step S3 is established through the following steps:
[0092] The entity recognition results include physical dimensions, component materials, and technical specifications extracted from the headphone design specifications.
[0093] The process relationship features include the operation steps, operation logic, and operation sequence relationships extracted from the standard operating procedure document;
[0094] The physical parameter features include quality parameters, qualified product judgment information, and defective product judgment information extracted from the quality judgment standard database;
[0095] An assembly process knowledge graph is established based on the extracted entity recognition results, process relationship features, and physical parameter features.
[0096] In one specific embodiment, the headphone assembly process knowledge graph in step S3 is established based on the design specifications, standard operating procedure documents, and a quality judgment standard database. The quality judgment standard database is established based on international electroacoustic standards, IPC welding quality specifications, and historical yield logs and failure mode libraries accumulated over many years in the smart factory. The data stored therein is calibrated through material limit stress tests for specific headphone models and standard operating instructions developed by process engineers to ensure the professionalism of quality judgment. The system extracts physical dimensions, part materials, and technical specifications from the headphone design specifications as entity features; extracts operation steps, operation logic, and operation sequence relationships from the standard operating procedure documents as process relationship features; and extracts quality parameters, qualified product judgment information, and defective product judgment information from the quality judgment standard database as physical parameter features. Based on entity features, process relationship features, and physical parameter features, a headphone assembly process knowledge graph is established. This design extracts knowledge originally scattered across design documents, process procedures, and quality databases into a unified graph structure, enabling the perception system to obtain stable domain prior semantic support when processing real-time physical signals. This embodiment enables the perception results to be directly mapped to component attributes, process sequence, and quality constraint relationships, thereby improving the process interpretability and anomaly tracing capabilities of semantic judgment.
[0097] More specifically, the entity recognition results include at least PCB nodes, drive nodes, and housing nodes; the process relationship features include at least welding nodes, dispensing nodes, and fastening nodes; and the physical parameter features include at least pressure nodes, wettability nodes, and curve feature nodes.
[0098] Establish associated edges between nodes that have process dependencies, structural fit relationships, or parameter constraints;
[0099] An assembly process knowledge graph is formed based on associated edges. The RotatE model is used to perform representation learning on the assembly process knowledge graph to obtain vector features. The node association weights are calculated using a joint graph attention network.
[0100] The guiding knowledge semantic vector is output based on the vector features and node association weights.
[0101] In one specific embodiment, the headphone assembly process knowledge graph encompasses component topology nodes, production process logic nodes, and physical parameter constraint nodes. Entity features include PCB nodes, driver nodes, and housing nodes; process relationship features include welding nodes, dispensing nodes, and fastening nodes; and physical parameter features include pressure nodes, wetting nodes, and curve feature nodes. Curve feature nodes are defined as dynamic pattern templates such as pressure-time evolution curves, audio energy envelopes, and pose trajectory offsets, used to characterize complex process features. During the verification phase, the system matches the similarity of real-time acquired waveforms with standard curve nodes as a basis for determining implicit quality states such as "fastening physical lock" or "dispensing consistency."
[0102] The system establishes association edges between nodes with process dependencies, structural fit relationships, or parameter constraints. Entities such as welding offset, insufficient adhesive dispensing, and abnormal fastening pressure under abnormal operating conditions are input into the RotatE knowledge representation learning model to obtain entity vector features. Subsequently, a joint graph attention network is used to remove redundant information and calculate semantic association weights between entities, generating guiding knowledge semantic vectors. When calculating semantic association weights, the node and relationship vectors generated by the RotatE model are used as input. A linear mapping matrix with relational bias and a Softmax function are used to calculate the scalar weights reflecting the strength of node associations. These weights represent the guiding contribution of domain knowledge entities to the current perception task, allowing the large model to prioritize core process constraints during semantic synthesis. This design enables the knowledge structure to describe not only assembly objects but also process logic and quality constraints, thereby improving the knowledge constraint strength when mapping multimodal physical features to process semantics and enhancing the ability to identify and locate abnormal paths.
[0103] More specifically, in step S3, mapping the cross-modal fusion feature vector and the guiding knowledge semantic vector to a unified semantic representation space specifically includes the following steps:
[0104] The cross-modal fusion feature vector and the guided knowledge semantic vector are input into a multi-granularity semantic mapping model and mapped to a unified semantic representation space;
[0105] The semantic dimension in the unified semantic representation space is divided into three levels: environmental entity perception, interactive behavior modeling, and task state decomposition.
[0106] The semantic association between real-time perceived features and corresponding nodes or associated edges in the headphone assembly process knowledge graph is determined by knowledge anchor matching, and the semantic perception results corresponding to environmental entities, interactive behaviors and task states are output based on cross-granularity semantic aggregation, which serve as the set of perceived semantic tags.
[0107] During the mapping process, inconsistent semantics are corrected and feedback is provided based on process rule constraints.
[0108] In one specific embodiment, the unified semantic mapping in step S3 is implemented through a multi-granularity semantic mapping model. Cross-modal fusion feature vectors and guiding knowledge semantic vectors are input into the multi-granularity mapping model, and a semantic space alignment operator is used to project the underlying numerical feature stream onto a preset high-dimensional industrial semantic space. Subsequently, through knowledge anchor matching logic, real-time physical features such as pressure peaks, pose offsets, or acoustic pulses are anchored to specific process entities or business logic nodes. The perception results are then output from three dimensions: environmental entity perception, interactive behavior modeling, and task state decomposition. This design establishes a computable intermediate layer between physical signals and business semantics, capable of transforming discrete images, sounds, and pressure signals into interpretable part entities, current actions, and process states, improving the completeness and logical consistency of environmental understanding.
[0109] Inconsistent semantics refers to real-time perceived behavioral sequences or physical parameters that violate the pre-defined SOP process logical dependency paths or quality constraint thresholds in the knowledge graph. It is used to characterize the logical conflict between the perceived state and prior process knowledge. During correction feedback, the initially perceived semantic labels and process specifications are used as inputs, and a logical consistency verification matrix is used as the correction medium to calculate the semantic deviation. Finally, a dynamic adjustment signal for the mapping weight is output. This correction feedback is used to backtrack and correct erroneous paths in the multi-granularity mapping model.
[0110] More specifically, the procedure conformity determination in step S4 includes the following steps:
[0111] The environmental entities, interactive behaviors, and task states in the set of perceptual semantic tags are compared with the step sequence constraints, object matching constraints, and parameter threshold constraints in the headphone assembly process specification to determine whether they match.
[0112] When the judgment result is inconsistent, the knowledge graph association relationship is updated, the perception parameters are adjusted, and the cross-modal labels are realigned.
[0113] When the judgment result is satisfactory, an assembly process prompt message is generated, and semantic synthesis is performed on the set of perceptual semantic tags based on the natural language big data model. The confidence level of the semantic synthesis result is then evaluated.
[0114] In one specific embodiment, the semantic verification and result synthesis in step S4 are performed according to the semantic verification and synthesis process of the perception result. After the system obtains the set of perception semantic tags, it first enters the domain knowledge logic verification stage, compares the consistency between the perception result and the process specification, and analyzes whether the current production status matches the prior logic in the headphone precision assembly process knowledge graph. When the semantic logic does not conform to the headphone assembly specification, the system executes the knowledge reconstruction and feedback logic, and converts the correction information into perception parameter feedback to guide the underlying mutual annotation to realign. When the semantic logic conforms to the headphone assembly specification, the system enters the assembly process prompt word construction stage, encapsulates the discrete tags into prompt instructions for the large model, generates an environmental perception description with coherent logic, and then performs a result confidence evaluation. When the confidence is greater than a preset threshold, it outputs the accurate perception result of the headphone processing environment and a structured perception report.
[0115] The evaluation process relies on the semantic consistency score between the natural language description generated by the computational model and the underlying multimodal feature flow and process logic to achieve cross-modal mutual verification. In one implementation, cosine similarity or semantic distance metrics are used to quantify the degree of matching between the synthesized semantics and the physical reality and business logic. The evaluation is deemed successful if the calculated confidence score exceeds a preset safety threshold, indicating that the synthesized result does not contain model illusions and highly matches the actual physical situation.
[0116] If the score is below the threshold, it is determined that there is a logical deviation or data conflict in the perception result, triggering the feedback correction process of the perception parameters.
[0117] The structured perception report covers the workstation environment background, part pose recognition results, process execution status, and task progress percentage, and provides consistency perception evaluation based on SOP logic. The report includes logical compliance comments for the current process, as well as detailed anomaly alerts, such as pressure threshold exceeding alarms, solder joint poor wettability warnings, or assembly order reversal alerts, aiming to provide accurate semantic decision support for production traceability.
[0118] This design adds process logic verification and semantic reconstruction mechanisms after the perception model output, which can suppress the direct amplification of local recognition errors into the final conclusion and improve the usability and credibility of the final report.
[0119] like Figure 3 As shown, the intelligent headphone factory multimodal environment perception system based on natural language processing includes the following:
[0120] A multimodal data acquisition module is used to acquire visual data, multi-source physical signals, and production text data of the target workstation. The multi-source physical signals include at least operation sound signals and force signals.
[0121] The feature extraction and mutual labeling spatiotemporal correlation module is used to extract features from the visual data and multi-source physical signals to obtain visual features, acoustic features and force features. Based on the visual data, it identifies key visual events in the headphone assembly process, and uses the key visual events as logical anchors to perform synchronous segmentation, mutual labeling and cross-modal temporal alignment of the acoustic features and / or force features in the corresponding time intervals to obtain cross-modal fusion feature vectors.
[0122] The knowledge graph construction and semantic mapping module is used to extract entity recognition results, process relationship features, and physical parameter features from production text data to construct a headphone assembly process knowledge graph. It then performs vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. The module maps the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, performs knowledge anchor matching and cross-granularity semantic aggregation, and obtains a set of perceptual semantic tags. This set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states.
[0123] The semantic verification and result synthesis module is used to perform procedure conformity judgment on the set of perceived semantic tags based on the SOP procedure logic verifier, and to evaluate the confidence level of the judgment result; when the confidence level reaches a preset threshold, it outputs a structured perception report containing environmental entities, interactive behaviors and task status.
[0124] The operation and effect of the intelligent headphone factory multimodal environment perception system based on natural semantics of the present invention are consistent with the above-mentioned intelligent headphone factory multimodal environment perception method based on natural semantics. Therefore, the description of the intelligent headphone factory multimodal environment perception system based on natural semantics will not be repeated here.
[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using necessary general-purpose hardware platforms, or it can be implemented through a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A multimodal environment perception method for a smart headphone factory based on natural language processing, characterized in that: The process includes the following steps: Step S1: Collect visual data, multi-source physical signals, and production text data of the target workstation. The multi-source physical signals include at least operation sound signals and force signals. Step S2: Extract features from the visual data and multi-source physical signals to obtain visual features, acoustic features, and force features. Based on the visual data, identify key visual events in the headphone assembly process. Using the key visual events as logical anchors, sequentially perform synchronous segmentation, mutual labeling, and cross-modal temporal alignment on the acoustic features and / or force features within the corresponding time interval to obtain a cross-modal fusion feature vector. Step S3: Based on the production text data, extract entity recognition results, process relationship features, and physical parameter features to construct a headphone assembly process knowledge graph. Perform vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. Map the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, perform knowledge anchor matching and cross-granularity semantic aggregation to obtain a set of perceptual semantic tags. The set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states. Step S4: Based on the SOP procedure logic checker, perform procedure conformity judgment on the set of perceptual semantic tags and evaluate the confidence level of the judgment results. When the confidence level reaches a preset threshold, output a structured perceptual report containing environmental entities, interactive behaviors, and task states.
2. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, In step S1, the visual data is single-view or multi-view image data of the headphone components during the assembly process; the multi-source physical signals are operation sound data, fastening force data and / or tactile data during the headphone assembly process; the production text data includes headphone design specifications, standard operating procedure documents, operating logic and procedure constraint relationship data.
3. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, In step S2, feature extraction is performed using a multimodal feature parallel extraction method, specifically including the following steps: using a target detection model combined with a point cloud feature extraction architecture to extract the pose information and three-dimensional geometric information of the headphone components as the visual features; using a masked audio autoencoder to encode the operation sound data to extract the acoustic features; and using a selective state space model to encode the fastening force data and / or tactile data to extract the force features.
4. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, In step S2, when using visual key events as logical anchors to synchronously segment acoustic features and / or force features, the specific steps include: determining the occurrence time of a visual key event based on at least one event in the visual data, such as component contact, fastening completion, dispensing start, or welding start; using the occurrence time of the visual key event as a benchmark, segmenting the acoustic features and force features into time windows to form acoustic segments and force segments corresponding to the visual key event; and using the event tags corresponding to the visual key event as a basis to associate the acoustic segments and force segments with tags to establish a cross-modal sample correspondence between the visual modality and the acoustic and force modalities.
5. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, In step S2, cross-modal temporal alignment specifically includes the following steps: visual features, acoustic features, and force features belonging to the same visual key event are time-synchronized and semantically corresponded, the sampling offset between different sensors is corrected, and the aligned modal features are mapped to a joint representation with consistent dimensions; the cross-modal fusion feature vector is a joint vector representation obtained by fusing the aligned visual features, acoustic features, and force features.
6. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, The assembly process knowledge graph in step S3 is established through the following steps: the entity recognition results include physical dimensions, part materials and technical specifications extracted from the headphone design specification; the process relationship features include operation steps, operation logic and operation sequence extracted from the standard operating procedure document; the physical parameter features include quality parameters, qualified product judgment information and defective product judgment information extracted from the quality judgment standard database; and the assembly process knowledge graph is established based on the extracted entity recognition results, process relationship features and physical parameter features.
7. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 6, characterized in that, The entity recognition results include at least PCB nodes, drive nodes, and housing nodes; the process relationship features include at least welding nodes, dispensing nodes, and fastening nodes; and the physical parameter features include at least pressure nodes, wettability nodes, and curve feature nodes. Association edges are established between nodes with process dependencies, structural fit relationships, or parameter constraint relationships. An assembly process knowledge graph is formed based on these association edges. The RotatE model is used to perform representation learning on the assembly process knowledge graph to obtain vector features, and a joint graph attention network is used to calculate node association weights. The guiding knowledge semantic vector is output based on the vector features and node association weights.
8. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, In step S3, mapping the cross-modal fusion feature vector and the guiding knowledge semantic vector to a unified semantic representation space specifically includes the following steps: inputting the cross-modal fusion feature vector and the guiding knowledge semantic vector into a multi-granularity semantic mapping model and mapping them to a unified semantic representation space; dividing the semantic dimensions in the unified semantic representation space into three levels: environmental entity perception, interactive behavior modeling, and task state decomposition; determining the semantic association between real-time perception features and corresponding nodes or associated edges in the headphone assembly process knowledge graph through knowledge anchor matching, and outputting semantic perception results corresponding to environmental entities, interactive behaviors, and task states based on cross-granularity semantic aggregation, as the set of perception semantic labels; and correcting and feeding back inconsistent semantics based on process rule constraints during the mapping process.
9. The method for multimodal environment perception in a smart headphone factory based on natural language processing as described in claim 1, characterized in that, The procedure conformity judgment in step S4 specifically includes the following steps: comparing the environmental entities, interactive behaviors, and task states in the perceptual semantic tag set with the step sequence constraints, object matching constraints, and parameter threshold constraints in the headphone assembly process procedure to determine whether they conform; when the judgment result is non-conforming, performing knowledge graph association update, perceptual parameter adjustment, and cross-modal tag realignment; when the judgment result is conforming, constructing assembly process prompt information, and performing semantic synthesis on the perceptual semantic tag set based on a natural language big data model, and evaluating the confidence level of the semantic synthesis result.
10. A multimodal environment perception system for a smart headphone factory based on natural language processing, characterized in that: The system includes the following components: a multimodal data acquisition module for acquiring visual data, multi-source physical signals, and production text data from the target workstation, wherein the multi-source physical signals include at least operation sound signals and force signals; and a feature extraction and cross-labeling spatiotemporal correlation module for extracting features from the visual data and multi-source physical signals to obtain visual features, acoustic features, and force features. Based on the visual data, the module identifies key visual events during the headphone assembly process and uses these key visual events as logical anchors to sequentially perform synchronous segmentation, cross-labeling, and cross-modal temporal alignment on the acoustic features and / or force features within the corresponding time intervals to obtain a cross-modal fusion feature vector. The knowledge graph construction and semantic mapping module is used to extract entity recognition results, process relationship features, and physical parameter features from production text data to construct a headphone assembly process knowledge graph. It then performs vector representation learning and node association weight calculation on the headphone assembly process knowledge graph to obtain guiding knowledge semantic vectors. The module maps the cross-modal fusion feature vectors and the guiding knowledge semantic vectors to a unified semantic representation space, performs knowledge anchor matching and cross-granularity semantic aggregation, and obtains a set of perceptual semantic tags. This set of perceptual semantic tags includes at least environmental entities, interactive behaviors, and task states. The semantic verification and result synthesis module is used to perform procedure conformity judgment on the set of perceived semantic tags based on the SOP procedure logic verifier, and to evaluate the confidence level of the judgment result; when the confidence level reaches a preset threshold, it outputs a structured perception report containing environmental entities, interactive behaviors and task status.