Multi-modal fusion ar intelligent interaction system and method based on functional modularization and interface specification, and storage medium

Abstract

The application relates to the technical field of augmented reality (AR) intelligent terminals, and discloses a multi-modal fusion AR intelligent interaction system and method based on functional modularization and interface specification, which solves the technical problems of high coupling, interface fragmentation and high end-side reasoning delay of existing AR systems. The system comprises: a functional module layer, which decouples an AR whole machine into 12 independent replaceable functional modules; an interface adaptation layer, which defines three types of physical interfaces, namely USB-C, FPC and BTB, and six types of logical protocols, namely MIPI, SPI and the like, and clearly defines the module-interface-protocol binding rules; and a multi-modal fusion processing layer, which is deployed on a main control SoC and an AI coprocessor and runs a lightweight Transform model with a parameter of less than or equal to 10 MB. According to the scheme, the hardware iteration cost is reduced by 35%, the end-side reasoning delay is less than or equal to 5 ms, the multi-modal time alignment accuracy is less than or equal to 0.5 ms, the transmission error rate is less than or equal to 1x10 ‑6 , and the scheme is suitable for lightweight consumer-grade AR glasses.

Description

Technical Field

[0001] This invention relates to the field of augmented reality (AR) smart terminal technology, specifically to a functional modular system architecture, standardized hardware interface specifications, and edge-side multimodal fusion processing scheme for lightweight consumer-grade AR glasses. Background Technology

[0002] As AR technology evolves towards the consumer level, the existing AR device design suffers from the following core bottlenecks, none of which have been effectively resolved by current technologies: 1. High system coupling: Existing AR devices are only vaguely divided into 3-5 major modules such as display, sensing, and computing, without clear definitions of subdivided functional modules. When iterating hardware, the entire circuit board needs to be redesigned, and the cost of replacing components in the supply chain increases by more than 40%. 2. Fragmented interface protocols: Different manufacturers use proprietary physical interfaces and communication protocols for their functional modules, without a unified module-protocol binding standard. This results in poor signal integrity, prominent electromagnetic interference (EMI) problems between modules, and an inability to adapt to different main control chips across platforms. 3. Low efficiency of edge deployment: The multimodal fusion algorithm is decoupled from the hardware architecture and is not optimized for specific interface bandwidth and computing resources. The inference latency of Transformer models is generally ≥20ms, which cannot meet the real-time requirements of AR interaction. Summary of the Invention

[0003] This invention aims to solve the technical problems of poor module compatibility, low interface communication efficiency, and high edge inference latency in existing AR systems, and provides a multimodal fusion AR intelligent interaction system, method, and storage medium based on functional modularity and interface specifications.

[0004] A multimodal fusion AR intelligent interaction system based on functional modularity and interface specifications is provided, comprising: The functional module layer decouples the AR system into 12 independent functional modules, specifically including: micro-nano LED display module, ToF depth sensing module, MEMS voice array module, IMU attitude sensing module, capacitive touch module, main control SoC module, AI coprocessor module, power management module, storage module, optical waveguide module module, structural component module and peripheral interface module. The interface adaptation layer is used to define the physical interface types and logical transmission protocols between the 12 functional modules. The physical interface types include USB-C connectors, FPC flexible circuit boards, and BTB board-to-board connectors. The logical transmission protocols include MIPI display serial interface, Serial Peripheral Interface (SPI), Integrated Circuit Bus (I2C), Peripheral Component Fast Interconnect (PCIe), Integrated Audio I2S bus, and Universal Asynchronous Receiver / Transmitter (UART). The multimodal fusion processing layer, deployed in the main control SoC module and the AI ​​coprocessor module, is configured to perform spatiotemporal alignment and semantic fusion of multimodal data.

[0005] In the system described in this invention, the correspondence between the physical interfaces of the interface adaptation layer and each functional module is as follows: The micro-nano LED display module, ToF depth sensing module, MEMS voice array module, IMU attitude sensing module, capacitive touch module, and AI coprocessor module are interconnected with the main control SoC module through FPC connectors. The power management module and the storage module are interconnected with the main control SoC module via BTB connectors; The peripheral interface module uses a USB-C connector and is exposed on the temple structure of the AR glasses.

[0006] In the system described in this invention, the binding relationship between the logical transmission protocol of the interface adaptation layer and each functional module is as follows: The micro-nano LED display module and the main control SoC module transmit video streams using the MIPI DSI protocol; The ToF depth sensing module, the IMU attitude sensing module and the main control SoC module use the SPI protocol to transmit sensor data. The MEMS voice array module and the main control SoC module transmit audio data using the I2S protocol. The AI ​​coprocessor module and the main control SoC module use the PCIe 3.0 protocol to transmit high-speed computing data; The power management module and the main control SoC module use the I2C protocol to transmit control commands.

[0007] In the system described in this invention, the physical interface satisfies the following electrical specifications: The USB-C connector supports the USB 3.2 Gen2 standard with a transmission rate of ≥10Gbps; The FPC connector has a differential impedance matching of 100Ω±10%, a MIPI DSI signal eye diagram height of ≥80%, and jitter of ≤0.1UI, conforming to the MIPI DSI v1.3 specification; The BTB connector has a pin pitch of ≤0.4mm and a single pin current carrying capacity of ≥2A, meeting the high current transmission requirements of the power management module.

[0008] In the system described in this invention, the logical transmission protocol adopts a custom compact binary encapsulation format: The data packet header contains a 1-byte synchronization header, a 2-byte module ID, a 4-byte timestamp, and a 2-byte data length field; The data packet tail includes a 2-byte CRC-16 check field, and the data transmission error rate is ≤1×10⁻⁶. -6 .

[0009] The system described in this invention includes an AI coprocessor module that deploys a lightweight Transformer fusion model that has undergone structured pruning and INT8 quantization. The model has 8–10 MB of parameters, 4–8 attention heads, and 128–256 hidden layer dimensions. The model has an edge inference latency of ≤5ms on the AI ​​coprocessor and a multimodal interaction recognition accuracy of ≥95%.

[0010] In the system described in this invention, the spatiotemporal alignment accuracy of the multimodal fusion processing layer satisfies: Using the 1kHz timestamp of the IMU attitude sensing module as a benchmark, the timestamps of ToF gesture data, MEMS voice data, and capacitive touch data are calibrated using a cross-correlation coefficient peak alignment algorithm, with a time alignment error ≤0.5ms. The multi-sensor coordinate system is registered to the IMU coordinate system using a pre-calibrated rigid transformation matrix, with a spatial registration error ≤2mm.

[0011] The system of the present invention further includes a dynamic power consumption optimization unit: When the system detects that the user has not interacted for 30 consecutive seconds, it automatically switches the ToF depth sensing module and the IMU attitude sensing module to a low-power sleep mode, reducing the power consumption of a single module from 120mW to below 15mW. By detecting ambient light intensity through a built-in CMOS light sensor, the brightness of the micro-nano LED display module is adjusted accordingly. When the light intensity is below 50 lux, the brightness drops from 500 nits to 200 nits, and the display power consumption is reduced by 40%.

[0012] The present invention also provides a multimodal fusion AR intelligent interaction method based on any of the foregoing systems, comprising the following steps: S1. Synchronously collect multi-source sensor data through the interface adapter layer; S2. The multimodal fusion processing layer performs spatiotemporal alignment and semantic fusion of multimodal data and outputs fusion interaction instructions; S3. Arbitrate multi-source instructions based on preset priority rules and execute the highest priority instruction.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the methods described above.

[0014] The present invention has the following beneficial effects: 1. Strong standardization and compatibility, significantly reducing hardware iteration costs: Through the decoupled design of 12 independent functional modules, combined with three physical interfaces (USB-C / FPC / BTB), six logical protocols (MIPI / SPI / PCIe, etc.), and the "module-interface-protocol" binding specification, complete decoupling of software and hardware is achieved. Module replacement does not require changes to the system architecture, shortening the hardware iteration cycle by 50% and reducing BOM costs by 35%; the standard interface design ensures a signal transmission bit error rate of ≤1×10⁻⁶. -6 It reduces EMI interference by 25%, can be adapted to different main control platforms, and improves mass production reusability by 60%.

[0015] 2. Real-time performance on the edge meets standards, and the efficiency of fusion processing is significantly improved: The main control SoC and the AI ​​coprocessor achieve computing power collaboration through high-speed interconnection via PCIe, which is suitable for deployment of lightweight Transformer models of ≤10MB, with edge inference latency of ≤5ms, multimodal time alignment accuracy of ≤0.5ms, interaction recognition accuracy of ≥95%, and error rate of ≤1%; the interface bandwidth and protocol matching design improve the efficiency of various data transmissions by 40%, and solves the problem of the disconnect between multimodal fusion and hardware architecture.

[0016] 3. Outstanding low power consumption and high feasibility for mass production: The dynamic power consumption optimization mechanism reduces the standby power consumption of the whole device by 30% and extends the battery life to more than 5.5 hours; all interface electrical parameters and data encapsulation formats have passed hardware compatibility verification, and industry-standard components can be directly reused without the need for customized special components. The mass production ramp-up cycle is shortened by 40%, which fully meets the lightweight and low power consumption mass production requirements of consumer-grade AR glasses. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a block diagram illustrating the architecture principle of a multimodal fusion AR intelligent interaction system based on functional modularity and interface specifications in an embodiment of the present invention. Figure 2 This is a flowchart of a multimodal fusion AR intelligent interaction method based on functional modularity and interface specifications in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0019] The system architecture is as follows: Figure 1 As shown, the system includes a functional module layer 10, an interface adaptation layer 20, and a multimodal fusion processing layer 30. The implementation of the functional module layer 10 involves breaking down all hardware functions of the AR device into 12 independent physical modules. Each module is designed as an independently replaceable component, including: The micro-nano LED display module 101, ToF depth sensing module 102, MEMS voice array module 103, IMU attitude sensing module 104, capacitive touch module 105, main control SoC module 106, AI coprocessor module 107, power management module 108, storage module 109, optical waveguide module 110, structural component module 111, and peripheral interface module 112 are all included. Each module only implements its own defined function. For example, the optical waveguide module 110 is only responsible for the optical path conduction of the optical display, and the structural component module 111 is only responsible for the physical support and fixation of the whole machine. There is no strong binding between the modules in terms of circuit or logic. When replacing any module, there is no need to modify the design of other modules.

[0020] The implementation of interface adaptation layer 20 includes: physical interfaces are uniformly adapted according to scenarios; all internal modules connect to the main control SoC using FPC flexible circuit boards; power management, storage, and other modules requiring high current or high pin counts use BTB board-to-board connectors; and exposed parts of peripheral interface modules use USB-C connectors. These three physical interfaces cover all interconnection requirements. The logic transmission protocol uniformly adopts six common standards: display-related transmission uses the MIPI display serial interface; sensor control uses the serial peripheral interface SPI or integrated circuit bus I2C; high-speed computing interconnection uses peripheral component fast interconnect PCIe; audio transmission uses the integrated audio I2S bus; and debugging uses Universal Asynchronous Receiver / Transmitter (UART). No modules may use proprietary communication protocols; communication rules are completely unified from the physical layer to the protocol layer.

[0021] The implementation of the multimodal fusion processing layer 30: The scheduling logic for multimodal fusion is deployed in the main control SoC module 106, which is responsible for receiving the raw data transmitted by each sensor through the interface adaptation layer and completing basic tasks such as time base unification and data format preprocessing; The core algorithm for fusion processing is deployed in the AI ​​coprocessor module 107, which is responsible for completing the spatiotemporal alignment of multimodal data - that is, calibrating multi-source data with a unified timestamp, registering the spatial position of multi-sensor with a unified coordinate system, and then performing semantic fusion to output interactive instructions; The two processing units interact with data at high speed through the PCIe protocol defined by the interface adaptation layer and work together to complete the entire fusion process.

[0022] This system adopts a three-layer architecture of "layered decoupling + standardization + computing power collaboration", which solves the problems of high coupling, fragmented interfaces, multimodal fusion and hardware decoupling in existing AR systems. The specific principle is described as follows: 1. Layered Principle of Overall Architecture The AR system is divided into three independent architectures: a functional module layer, an interface adaptation layer, and a multimodal fusion processing layer. Each layer has clear boundaries and its evolution does not affect the others. The functional module layer is the hardware entity foundation, the interface adaptation layer is the connection between layers / modules, and the multimodal fusion processing layer is the core of upper-level intelligent interaction. The three layers provide progressive support, avoiding the problem of the whole system being affected by the mixed design of various functions in the existing AR device.

[0023] 2. Decoupling principle of functional module layer Breaking away from the existing convention of AR systems being divided into only 3-5 large, vague modules, this project breaks down the entire system's functionality into 12 clearly defined, single-function independent modules. These modules cover the entire process, including optical display, sensing and acquisition, computing and processing, power management, storage, optical modules, physical structure, and peripheral connections. Each module is responsible for its own defined function, with no hardware coupling between modules. This adheres to the hardware design principle of "high cohesion and low coupling," fundamentally solving the problem of needing to redesign the entire circuit board for system iterations.

[0024] 3. The unified principle of the interface adaptation layer To address the interconnection requirements of the 12 modules, three general physical interfaces and six industry-standard logical transmission protocols are uniformly defined to replace the existing vendor-defined proprietary interfaces and protocols. The physical interfaces are categorized according to connection scenarios to cover all scenarios, including internal short-distance transmission, high-current connections, and exposed peripherals. All logical protocols are open standard protocols, unifying communication rules from the physical layer to the protocol layer and solving the problems of poor signal integrity, prominent electromagnetic interference, and inability to adapt to cross-platforms caused by proprietary protocols.

[0025] 4. Computational power collaboration principle of the multimodal fusion processing layer Multimodal fusion capabilities are deployed simultaneously in two hardware units: the main control SoC and the AI ​​coprocessor. The main control SoC is responsible for general scheduling and data preprocessing, while the AI ​​coprocessor is responsible for dedicated acceleration and core fusion algorithm computation. The two work together through a high-speed interface to ensure the computing power requirements for multimodal processing while controlling power consumption on the edge, thus solving the problems of existing fusion algorithms being disconnected from hardware architecture and low efficiency of edge deployment.

[0026] In a further embodiment, based on the compact structural constraints of AR smart glasses and the transmission requirements and pin characteristics of different functional modules, a suitable physical interface is matched for each type of module. This replaces the problem of arbitrary physical interface selection and mismatch with module characteristics in existing AR designs, further solidifying the implementation rules of modular design at the physical layer. The specific principle can be divided into three parts: Internal sensing and computing modules are adapted to FPC connectors: Micro-nano LED displays, ToF depth sensors, MEMS voice arrays, IMU attitude sensors, capacitive touch, and AI coprocessors are all internal modules with few pins, short transmission distances, and the need to adapt to the bending space inside the glasses. The flexible, bendable, and thin characteristics of FPC perfectly match the connection requirements of these modules. High-current, multi-pin modules are adapted to BTB connectors: Power management modules require high current transmission, and storage modules require high-speed transmission with multiple pins. The high pin density, strong connection stability, and strong current carrying capacity of BTB connectors match the needs of these modules. The peripheral interface is compatible with USB-C and exposed on the temple: The peripheral interface requires users to frequently plug and unplug it, and to balance charging and data transfer. USB-C is a universal standard interface, and its exposure on the temple conforms to the ergonomic design of AR glasses and makes it convenient for users to use.

[0027] Specific implementation methods include: The first type of module connection implementation: The micro-nano LED display module, ToF depth sensing module, MEMS voice array module, IMU attitude sensing module, capacitive touch module, and AI coprocessor module are physically interconnected with the main control SoC module through FPC flexible circuit boards. The FPC interface pin definitions of each module match the functions of the corresponding module. When replacing any module, only the corresponding FPC needs to be plugged in and unplugged, without modifying other circuits.

[0028] The second type of module connection is achieved by physically interconnecting the power management module and the storage module with the main control SoC module through BTB board-to-board connectors. The number of pins of the BTB connector matches the high current pin requirements of the power management module and the high-speed transmission pin requirements of the storage module. After connection, no additional fixing is required, which is suitable for the compact internal structure of AR glasses.

[0029] Peripheral interface implementation: The peripheral interface module directly adopts a USB-C connector, and the exposed part of the interface is fixed to the side of the temple of the AR glasses, ensuring that users can complete charging or peripheral connection operations without removing the glasses.

[0030] In a further embodiment, a dedicated standard logic protocol is matched for each type of module according to the data type, transmission rate requirements, and real-time requirements of each functional module. This replaces the compatibility issues caused by arbitrary protocol selection and inconsistent protocols for the same functional module in the existing AR design. The interface specifications are further solidified at the protocol layer to achieve a one-to-one correspondence between "module and protocol" and avoid problems such as low transmission efficiency and signal interference caused by the mixed use of private protocols.

[0031] The specific binding logic works as follows: The micro-nano LED display module transmits high-bandwidth video streams. MIPI DSI is an industry-standard protocol for mobile device displays, supporting multi-channel high-speed transmission, perfectly matching the bandwidth requirements of video streams; therefore, it is bound to the MIPI DSI protocol. The ToF depth sensing module and IMU attitude sensing module transmit medium-to-low speed, high real-time sensor data. The SPI protocol is a full-duplex synchronous serial bus with controllable clock and low transmission latency, suitable for real-time sensor data acquisition; therefore, it is bound to the SPI protocol. The MEMS voice array module transmits continuous, low-latency audio streams. I2S is a dedicated audio transmission protocol that supports continuous transmission of audio data at fixed sampling rates and bit depths, matching the characteristics of audio streams; therefore, it is bound to the I2S protocol. The AI ​​coprocessor module and the main control SoC module need to transmit high-performance fusion computing data. PCIe 3.0 is a high-speed serial interconnect standard, supporting high-bandwidth, low-latency point-to-point transmission, matching the high-performance data interaction requirements; therefore, it is bound to PCIe. The 3.0 protocol is used; the power management module transmits control commands with small data volumes, and I2C is a low-speed serial control bus that supports multiple slave devices and small data packet transmission, which matches the control requirements of power management. Therefore, the I2C protocol is used.

[0032] The logic protocol implementation between the micro-nano LED display module and the main control SoC module is as follows: The MIPI DSI protocol stack is configured in the display controller of the main control SoC and set to 4-channel output mode. The control chip of the micro-nano LED display module is configured as a MIPI DSI slave device. The two are interconnected through the physical interface defined by the interface adapter layer to transmit a video stream with a resolution of 1920×1080 and a frame rate of 60fps.

[0033] The logic protocol implementation of the ToF depth sensing module, IMU attitude sensing module and main control SoC module is as follows: Configure the SPI master controller in the main control SoC, set the clock frequency to 10MHz (ToF) and 8MHz (IMU), and adopt the time-division chip select method. The ToF module is assigned chip select CS0 and the IMU module is assigned chip select CS1. Both are SPI slave devices and transmit sensor data at a sampling rate of 100Hz (ToF) and 1kHz (IMU).

[0034] The logical protocol implementation between the MEMS voice array module and the main control SoC module is as follows: The main control SoC is configured with an I2S controller, set to 16-bit depth, 16kHz sampling rate, and dual-channel mode. The MEMS voice array module is configured as an I2S slave device to continuously transmit audio stream data.

[0035] The logical protocol implementation of the AI ​​coprocessor module and the main control SoC module: Both are configured with PCIe 3.0 controllers, set to x2 channel mode, with a transmission rate of 16Gbps. The main control SoC acts as the root complex device, and the AI ​​coprocessor acts as the endpoint device, transmitting the input data and inference results of the fusion model.

[0036] The logic protocol implementation between the power management module and the main control SoC module is as follows: an I2C master controller is configured in the main control SoC with a clock frequency of 400kHz, and the power management module is configured as an I2C slave device with an address of 0x36 to transmit control commands such as voltage regulation and sleep mode switching.

[0037] In a further embodiment, quantifiable parameter standards are formulated for the application scenarios and transmission requirements of each type of physical interface. This replaces the problems of unstable transmission and poor compatibility caused by the lack of unified standards for physical interface selection and clear criteria for signal quality in existing AR designs. It further solidifies the implementation rules of the interface adaptation layer from the physical layer. The principle of the specific standards is as follows: 1. The standard principle of USB-C connector: The peripheral interface module is responsible for the exposed connection and needs to support high-speed data transmission and peripheral charging at the same time. The USB 3.2 Gen2 standard can guarantee a transmission rate of ≥10Gbps, taking into account the needs of data export, peripheral expansion and high-power charging of 30W or more, and conforming to the general interface standard of consumer electronic products, reducing the threshold for users.

[0038] 2. FPC Connector Specification and Principle: FPC is used to connect six types of internal modules (micro-nano LED, ToF, MEMS voice, IMU, capacitive touch, and AI coprocessor). Among them, the micro-nano LED display module transmits MIPI high-speed video signals, while the other modules transmit sensor and computational interaction data. Differential impedance matching of 100Ω±10% is the core requirement to ensure the integrity of differential signals. The MIPI signal eye diagram conforms to the DSI v1.3 specification to ensure that the display and sensing signals are not distorted, avoiding signal reflection and eye diagram closure problems caused by proprietary FPC designs.

[0039] 3. BTB connector specifications and principles: BTB is used to connect power management and storage modules. Power management modules need to transmit large currents, so the current carrying capacity of a single pin is required to be ≥2A / pin to avoid overheating of the pins and excessive voltage drop. Storage modules require high-density pins to achieve high-speed transmission, so the pin pitch is required to be ≤0.4mm to fit the compact stacking space inside AR glasses and balance the pin density and space occupation requirements.

[0040] Preferably, the USB-C connector is configured as follows: a USB 3.2 Gen2 standard USB-C connector certified by USB-IF is selected, and its transmission rate is tested with an oscilloscope to be ≥12Gbps (meeting the ≥10Gbps requirement). It also supports the USB PD 3.0 fast charging protocol with a maximum charging power of 36W. The exposed part of the connector is fixed to the side of the right temple of the AR glasses for easy plugging and unplugging by the user.

[0041] Preferably, the FPC connectors implement the following: all FPC flexible circuit boards used for internal module connections are designed with differential lines using 100Ω impedance control, with tolerance controlled within ±10%, and the impedance deviation is verified to be ≤10% by time domain reflectometer (TDR) testing; the MIPI DSI signal of the micro-nano LED display module is tested with an oscilloscope and the eye diagram height is ≥80% and the jitter is ≤0.1UI, which meets the eye diagram requirements of the MIPI DSI v1.3 specification, and the differential signals of the remaining FPCs are also designed according to the same impedance standard.

[0042] Preferably, the BTB connector achieves the following: a BTB board-to-board connector with a pin pitch of 0.4mm is selected, and the temperature rise of a single pin is ≤10℃ under 2A current verified by DC voltage drop test, and the current carrying capacity meets the requirements; when the power management module is connected to the main control SoC, it is configured with 6 dedicated power pins with a total current carrying capacity ≥12A, matching the maximum output current requirement of the power management module; the storage module uses an 80-pin BTB connector with a pin pitch of 0.4mm to meet the high-speed transmission pin requirements of UFS3.1 storage.

[0043] In a further embodiment, addressing the issues of inconsistent encapsulation formats, high redundancy, lack of metadata required for fusion, and unreliable transmission in existing AR systems, a universal low-overhead encapsulation rule is designed for all logical transmission protocols. This minimizes bandwidth consumption while carrying key information required for multimodal fusion, and ensures transmission accuracy through a verification mechanism. The specific principles can be divided into three points: Low-overhead design principle: A binary structure of "fixed-length header + variable-length valid data + fixed-length trailer" is adopted, with no redundant text fields. Compared to common text encapsulation such as JSON and XML, this reduces transmission overhead by more than 70%, fully adapting to the limited interface bandwidth resources within AR glasses. Fusion adaptation principle: The header integrates two types of key metadata: module ID and timestamp. No additional transmission of auxiliary information such as data source and acquisition time is required, saving the subsequent steps of parsing and matching during multimodal fusion and reducing the computational overhead of fusion processing. Reliability assurance principle: A CRC-16 check field is added to the trailer to detect single-bit, double-bit, and multiple-bit errors during transmission, ensuring data integrity at the protocol layer and solving the problem of high error rates caused by the lack of verification in private encapsulation.

[0044] The data packet header has a fixed 9-byte structure: the first byte is a 1-byte synchronization header with a preset fixed value of 0xAA to identify the start of the data packet; the next 2 bytes are the module ID, uniquely corresponding to the sender among the 12 functional modules; the next 4 bytes are the timestamp in microseconds, recording the baseline time of data acquisition; and the last 2 bytes are the data length, indicating the number of bytes in the subsequent payload. The payload is the original transmission data of the corresponding module, with the same length as the data length field in the packet header. The payload content of different modules is defined according to their functions, but the outer encapsulation format is completely uniform. The data packet tail is a fixed 2-byte CRC-16 check field, which uses a 0x8005 polynomial to calculate the overall check value of the packet header and payload. The data is considered valid only if the receiver verifies the consistency. Through the above encapsulation rules, combined with the error control mechanisms of each logic protocol, the overall data transmission bit error rate is ≤1×10⁻⁶. -6 .

[0045] In a further embodiment, to address the issues of large parameter size (generally exceeding 100MB), high edge inference latency (generally ≥20ms), and high power consumption of existing Transformer models, a two-step lightweight solution of "structured pruning + INT8 quantization" is adopted to meet the requirements of low power consumption and high real-time performance of AR glasses in edge scenarios. At the same time, the core structural parameters and performance indicators of the model are limited to the range that is compatible with the edge computing power, replacing the high latency and short battery life caused by direct deployment of existing general large models. This matches the hardware capabilities of the AI ​​coprocessor and enables efficient edge operation of the multimodal fusion processing layer.

[0046] Specifically, structured pruning can directly remove redundant structures such as attention heads and feedforward channels from the model, reducing computation and adapting to the hardware acceleration architecture of AI coprocessors; INT8 quantization compresses 32-bit floating-point parameters into 8-bit integers, reducing memory usage and bandwidth requirements. The combination of these two methods can achieve model compression with minimal loss of accuracy.

[0047] Meanwhile, limiting the number of parameters to 8-10MB, the number of attention heads to 4-8, and the hidden layer dimension to 128-256 is the experimentally verified optimal range for "accuracy-latency-power consumption": a parameter size of less than 8MB will cause the accuracy to drop below 90%, while a parameter size of more than 10MB will cause the inference latency to exceed 5ms; fewer than 4 attention heads will result in the loss of multimodal feature associations, while more than 8 attention heads will exceed the parallel computing capabilities of the AI ​​coprocessor; similarly, the hidden layer dimension of 128-256 is the optimal range for adapting to the computing power of the edge NPU.

[0048] In addition, the inference latency of ≤5ms is lower than the human perception latency threshold of 10ms, which can avoid AR interaction lag; the recognition accuracy of ≥95% can ensure the availability of multimodal fusion, and the error rate is controlled within 1%, which matches the experience requirements of consumer-grade AR products.

[0049] Preferably, the baseline model is pre-trained: a general Transformer Encoder is selected as the base model and initialized with a structure of 12-layer Encoder, 8 attention heads, and 512-dimensional hidden layers. The baseline model is pre-trained using a public multimodal AR interaction dataset (containing 100,000 sets of labeled data for gestures, voice, touch, IMU and other modalities) to obtain the baseline model (with approximately 120MB of parameters, an inference latency of approximately 25ms, and a recognition accuracy of approximately 96%).

[0050] Preferably, structured pruning is implemented as follows: with the recognition accuracy not less than 95% as a constraint, a structured pruning method based on the weighted L1 norm is adopted to calculate the importance scores of the attention heads and feedforward layer channels layer by layer, and prune the 60% of redundant structures with the lowest scores. Finally, the model is compressed to 2 to 4 encoder layers, retaining 4 to 8 attention heads, and the hidden layer dimension is compressed to 128 to 256 dimensions. After pruning, the model parameter size is reduced to 8 to 10 MB.

[0051] Preferably, INT8 quantization is implemented as follows: 1000 sets of multimodal interaction data from real AR scenarios are used as calibration sets to perform offline quantization on the pruned model: the floating-point 32-bit weights and activation values ​​are uniformly converted into 8-bit integers. A layer-by-layer calibration strategy is adopted during the quantization process to minimize the accuracy loss caused by quantization. Finally, the accuracy of the model decreases by ≤0.5% after quantization.

[0052] Preferably, deployment and performance verification: the quantized model is deployed to the NPU unit of the AI ​​coprocessor module and communicates with the main control SoC module through the PCIe logic protocol. The measured single-round inference latency on the edge side is ≤5ms, and the multimodal interaction recognition accuracy on the independent test set is ≥95%, which meets the performance indicators of the additional feature constraints.

[0053] In a further embodiment, to address the pain points of asynchronous time and inconsistent spatial coordinate systems of multi-source data in existing AR multimodal fusion, which lead to feature misalignment and decreased fusion accuracy, a standardized spatiotemporal benchmark selection and high-precision alignment algorithm are used to achieve accurate matching of multimodal data, while limiting the quantitative accuracy index to adapt to the real-time and accuracy requirements of AR interaction.

[0054] The specific principle can be divided into three parts: 1. Time Alignment Reference Selection Principle: The 1kHz timestamp of the IMU attitude sensing module is selected as the global time reference because the IMU is the core sensor of AR head posture tracking. It has a high sampling rate and continuous output, and serves as the reference coordinate system carrier for spatial interaction. Its timestamp sequence has strong continuity and is suitable as an alignment reference. The cross-correlation coefficient peak alignment algorithm is adopted. By calculating the statistical correlation between different modal data sequences and the IMU reference sequence, the optimal time offset is found. This can offset the time error caused by clock offset, transmission delay, and sampling jitter, and improve the accuracy by several times compared with the simple timestamp method.

[0055] 2. Spatial Registration Principle: Given the fixed position of AR glasses sensors after installation, a rigid transformation involving only rotation and translation is used to complete the coordinate system transformation, eliminating the need for complex transformations such as scaling and deformation. Rigid transformation matrices from each sensor to the IMU coordinate system are generated through factory pre-calibration. During real-time processing, these matrices are directly called for coordinate transformation, eliminating the need for real-time calculations and balancing registration accuracy and real-time performance. All modes are unified to the IMU coordinate system, eliminating coordinate system differences and ensuring spatial feature alignment.

[0056] 3. Numerical Limitation Principle: Time alignment error ≤ 0.5ms, which is lower than the 1ms threshold for human perception of AR interaction delay, thus avoiding user perception of interaction lag; Spatial registration error ≤ 2mm, which is less than the 5mm accuracy threshold for gesture interaction recognition, and will not affect the accuracy of interaction intent recognition.

[0057] Specifically, the multimodal fusion processing layer is deployed on the main control SoC module and the AI ​​coprocessor module. The main control SoC is responsible for the acquisition and preprocessing of multi-source data, and the AI ​​coprocessor is responsible for high-precision spatiotemporal alignment calculation. Among them, the time alignment implementation includes: (1) the IMU attitude sensing module outputs attitude data with timestamps at a frequency of 1kHz to generate a continuous reference timestamp sequence; (2) the gesture data of the ToF depth sensing module, the voice data of the MEMS voice array module, and the touch data of the capacitive touch module are acquired, and the timestamp sequence of each modality data is extracted; (3) for each modality timestamp sequence, the cross-correlation coefficient is calculated with the IMU reference timestamp sequence, and the time offset between the modality and the reference is obtained by finding the peak position of the cross-correlation coefficient; (4) the time compensation of each modality data is performed according to the offset to complete the time alignment. The measured time alignment error of all modalities is ≤0.5ms.

[0058] Spatial registration includes: (1) Factory calibration stage: Fix the AR glasses to the calibration device, use the standard calibration board to collect the feature points in the coordinate system of the ToF depth sensing module, MEMS voice array module (sound source positioning coordinates), and capacitive touch module, and pair them with the corresponding feature points in the coordinate system of the IMU attitude sensing module. Solve the 3×3 rotation matrix and 3×1 translation vector of each sensor to the IMU coordinate system by the least squares method, combine them into a rigid transformation matrix, and store it in the non-volatile memory of the main control SoC; (2) Real-time processing stage: Read the pre-stored rigid transformation matrix, multiply the spatial coordinate data of each mode by the corresponding transformation matrix, and uniformly transform to the IMU coordinate system. The measured spatial registration error is ≤2mm.

[0059] In a further embodiment, to address the pain point of limited battery life in lightweight AR glasses, a combined strategy of "high-power sensor module in sleep mode without interaction + high-power display module that adjusts in conjunction with ambient light" is adopted to minimize system power consumption with almost no impact on user experience, thus replacing the battery life shortcoming caused by all modules being constantly on in existing AR systems.

[0060] The dynamic power consumption optimization unit's logic functions are deployed in the main control SoC module, which acts as the system scheduling core, responsible for monitoring the interaction status, reading light sensor data, and sending control commands. The main control SoC continuously monitors four types of interaction data: ToF gestures, MEMS voice, capacitive touch, and IMU head tracking. When no valid interaction command is detected for 30 consecutive seconds, it sends low-power sleep commands to the ToF depth sensing module and the IMU attitude sensing module respectively via the logic transmission protocol (SPI / I2C). Both modules enter sleep mode, and the measured power consumption of a single module drops from 120mW to the 12-15mW range, meeting the requirement of below 15mW. The main control SoC detects the ambient light intensity in real time through the built-in CMOS light sensor. When the light intensity is below 50 lux, it sends a brightness adjustment command to the micro-nano LED display module through the MIPI display serial interface, reducing the display brightness from 500 nits to 200 nits. The measured power consumption of the display module is reduced by 38%-42%, meeting the requirement of a 40% reduction.

[0061] The selection principle for non-interactive sleep mode is as follows: The ToF depth sensing module and IMU posture sensing module are sensing units with high system power consumption. When there is no effective interaction, there is no need to collect gesture and posture data. After sleep mode, the power consumption of a single module can be reduced from 120mW to below 15mW, a reduction of more than 87%. Choosing "30 seconds of continuous no interaction" as the trigger threshold is the optimal value to balance "frequent false triggering of sleep mode" and "improved battery life": if the threshold is too short, it will cause the user to trigger sleep mode after a short pause, requiring a re-wake-up and waiting; if the threshold is too long, it will not effectively reduce standby power consumption.

[0062] The principle of brightness linkage is as follows: the micro-nano LED display module is the unit with the highest power consumption of the whole machine (accounting for more than 40% of the total power consumption). Its brightness requirement changes dynamically with the ambient light intensity. In dim environments, a high brightness of 500 nits is not required to meet the viewing needs. The "50 lux" is selected as the brightness adjustment threshold, which is the dividing point between dim indoor environments (such as indoors at night or in cinemas) and bright environments. When it is below this threshold, the brightness is reduced from 500 nits to 200 nits. There is no obvious difference in viewing for the human eye, and the display power consumption can be reduced by 40%.

[0063] The design principle of the quantitative indicators is as follows: all power consumption, duration, and brightness values ​​are the optimal range verified by actual testing, which ensures the power consumption optimization effect without significantly affecting the interactive experience and display effect, and avoids insufficient optimization effect or experience degradation caused by setting the values ​​too wide.

[0064] In a further embodiment, such as Figure 2 As shown, for reference Figure 1This paper provides a multimodal fusion AR intelligent interaction method. The method relies on the AR intelligent interaction system described above for implementation. The three steps correspond one-to-one with the three-layer architecture of the system: S1 corresponds to the data acquisition function of the interface adaptation layer, S2 corresponds to the calculation function of the multimodal fusion processing layer, and S3 corresponds to the instruction execution logic of the system. The process is completely matched with the hardware capability division of the system and cannot be operated independently without the modular architecture, standard interface, and computing power collaborative design of this system.

[0065] Specifically, step S1 is implemented by synchronously collecting multi-source sensor data through any of the interface adaptation layers defined by the system (including its defined physical interface type, logical transmission protocol, encapsulation format, etc.). The sensor type, transmission method, timestamp marking rules, etc. of the collected data are completely consistent with the limitations of the system's functional module layer and interface adaptation layer, which will not be elaborated here.

[0066] Step S2: Call the multimodal fusion processing layer defined by the system (deployed in the main control SoC module and AI coprocessor module) to perform spatiotemporal alignment and semantic fusion of multimodal data, and output fusion interaction instructions and confidence scores.

[0067] Step S3 is implemented by: deduplicating and arbitrating multi-source instructions based on the priority rules preset by the system, and executing the instruction with the highest priority; wherein, the priority rules are preferably: capacitive touch > ToF gesture > MEMS voice > IMU head tracking, etc.

[0068] In a further embodiment, a computer-readable storage medium is provided, which corresponds to the storage module among the 12 independent modules defined by the functional module layer of this system. Its physical interface (BTB connector) and logical protocol (UFS protocol) are fully compatible with the interface adaptation layer specification of the system. It is a built-in component of this system, rather than a general-purpose external storage device, and is strongly bound to the system architecture from the hardware layer. The computer program stored in the storage medium is not a general-purpose multimodal fusion program, but a dedicated program specifically adapted to the modular architecture, standard interface, and dual-processor computing power division of this system. It includes driver code for calling the interface adaptation layer to collect data, code for scheduling the main control SoC and AI coprocessor to perform collaborative computing, and code for instruction arbitration execution. It must run on the processor of this system (main control SoC + AI coprocessor) to achieve the aforementioned method, and cannot achieve the same technical effect by running on general-purpose computing devices or AR devices not belonging to this system.

[0069] The storage medium hardware implementation uses a storage module from the functional module layer of this system, specifically a 128GB flash memory chip with UFS 3.1 protocol. It is interconnected with the main control SoC module through a 40-pin BTB board-to-board connector and uses the UFS logic protocol to transmit data, serving as the built-in non-volatile storage medium of this system.

[0070] The computer program storage implementation involves compiling the computer program code implementing the aforementioned method embodiments into an executable file adapted to the main control SoC (Qualcomm Snapdragon AR1 Gen1) and the AI ​​coprocessor NPU architecture. This executable file is then permanently written to the non-volatile storage area of ​​the storage module during system production. It contains only three types of code adapted to the system hardware: interface driver code, adapted to the three physical interfaces and six logical transmission protocols defined by the system, used to synchronously acquire multi-source sensor data through the interface adaptation layer; computing power scheduling code, used to call the main control SoC and the AI ​​coprocessor to collaboratively perform spatiotemporal alignment and semantic fusion of multimodal data; and arbitration execution code, used to implement instruction deduplication arbitration and execution with preset priorities. After the system powers on, the main control SoC automatically reads and loads the program from the storage module. During program execution, it calls various functional modules and interfaces of the system.

[0071] The foregoing solutions of the present invention will be described in detail below through several more specific embodiments.

[0072] Specific Implementation Example 1: System Architecture and Module Connection

[0073] The AR glasses in this embodiment are lightweight consumer-grade products, with a total weight of ≤65g. The specific implementation and connection relationship of the 12 functional modules are as follows:

[0074] Specific Implementation Example 2: Detailed Specification of Interface Adaptation Layer

[0075] Physical interface test data: The USB-C port has a measured transfer rate of 12Gbps and supports simultaneous charging (30W) and data transfer. The measured impedance of the FPC connector is 98Ω~102Ω, the eye diagram height of the MIPI DSI signal is 82%, and the jitter is 0.08UI, which conforms to the MIPI DSI v1.3 specification. The BTB connector has 40 pins, with a single pin current carrying capacity of 2.5A, and the power management module has a power supply efficiency of ≥92%.

[0076] Logical protocol encapsulation format: Taking the SPI data of the ToF depth sensing module as an example: Synchronization header: 0xAA (1 byte); Module ID: 0x01 (1 byte, representing the ToF module); Timestamp: 4 bytes (in μs, taken from IMU reference timestamp); Data length: 2 bytes (length of subsequent sensor data); Sensor data: N bytes (containing the three-dimensional coordinates of 26 hand bone points); CRC-16 checksum: 2 bytes (polynomial is 0x8005).

[0077] In actual testing, there were no error packets in 1000 transmissions, with a bit error rate of ≤1×10⁻⁶. -6 .

[0078] Specific Implementation Example 3: Deployment of AI Coprocessor Model The lightweight Transformer model built into the AI ​​coprocessor module is obtained through the following steps: Structured pruning: With recognition accuracy as a constraint, a structured pruning method based on the L1 norm is adopted to prune the pre-trained 12-layer Transformer Encoder model, removing 60% of redundant attention heads and feedforward layer channels, compressing the number of model layers to 2, retaining 4 attention heads, and retaining 128 dimensions in the hidden layer.

[0079] INT8 quantization: Converts pruned floating-point parameters into 8-bit integers. The quantization calibration set uses 1000 sets of real AR interaction scene data. The accuracy decreases by ≤0.5% after quantization.

[0080] Deployment testing: The final model has a parameter size of 8.2MB, an inference latency of 4.1ms under PCIe 3.0 x2 bandwidth, and a trimodal recognition accuracy of 95.7% for gesture, voice, and touch.

[0081] Specific Implementation Example 4: Spatiotemporal Alignment Accuracy Test Using the IMU's 1kHz timestamp (1ms period) as a benchmark, 1000 sets of multimodal data were collected to test alignment accuracy: The average timestamp offset of ToF gesture data is 0.32ms, with a maximum of 0.48ms; The average timestamp offset of MEMS voice data is 0.28ms, with a maximum of 0.45ms; The average timestamp offset of capacitive touch data is 0.15ms, with a maximum of 0.32ms; Alignment errors for all modes were ≤0.5ms. Spatial registration was performed using a checkerboard calibration board. The corresponding point sets of each sensor and the IMU coordinate system were pre-acquired, and the rigid transformation matrix was calculated using the least squares method. The measured spatial registration error was ≤1.8mm.

[0082] Specific Implementation Example 5: Dynamic Power Consumption Optimization Effect When a user has no touch, gesture, or voice interaction for 30 consecutive seconds: The ToF depth sensing module and the IMU attitude sensing module automatically enter sleep mode, with the current dropping from 120mA to 12mA and the power consumption of a single module dropping from 120mW to 15mW. When the CMOS photosensitive element detects an ambient light level of 30 lux (below the 50 lux threshold), the brightness of the micro-nano LED display module decreases from 500 nits to 200 nits, the display power consumption decreases from 400mW to 240mW, and the overall standby power consumption is reduced by 35%.

[0083] Specific Implementation Example 6: Interaction Method Implementation The method in this embodiment only involves the execution flow at the system level, and the specific algorithm includes: Step S1: Synchronously acquire ToF, MEMS, IMU, and touch data via FPC interface and SPI / I2S protocol; ToF sensor acquires 3D coordinates of 26 skeletal points of the hand; MEMS microphone acquires audio stream with a 16kHz sampling rate; IMU outputs quaternion posture data; touchpad outputs X / Y coordinates and pressure value; Step S2: The multimodal fusion processing layer calls the PCIe interface to transmit data to the AI ​​coprocessor, performs spatiotemporal alignment and Transformer inference, and outputs fusion instructions; using the 1kHz timestamp of the IMU as a reference, it calculates the peak value of the cross-correlation coefficient of other modal data, performs time offset compensation, and the time alignment error is ≤0.5ms; Step S3: Model Inference. The four types of features are mapped to 128-dimensional vectors respectively, concatenated and processed by two TransformerEncoder layers to output 64-dimensional fused features. These features are then classified and output as interactive commands (such as "Confirm", "Return", "Zoom") by a fully connected layer. Step S4: Arbitrate the priority order of “Capacitive Touch > ToF Gesture > MEMS Voice > IMU Head Tracking” and execute the highest priority command.

[0084] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal fusion AR intelligent interaction system based on functional modularity and interface specifications, characterized in that, include: The functional module layer decouples the AR system into 12 independent functional modules, specifically including: micro-nano LED display module, ToF depth sensing module, MEMS voice array module, IMU attitude sensing module, capacitive touch module, main control SoC module, AI coprocessor module, power management module, storage module, optical waveguide module module, structural component module and peripheral interface module. The interface adaptation layer is used to define the physical interface types and logical transmission protocols between the 12 functional modules. The physical interface types include USB-C connectors, FPC flexible circuit boards, and BTB board-to-board connectors. The logical transmission protocols include MIPI display serial interface, Serial Peripheral Interface (SPI), Integrated Circuit Bus (I2C), Peripheral Component Fast Interconnect (PCIe), Integrated Audio I2S bus, and Universal Asynchronous Receiver / Transmitter (UART). The multimodal fusion processing layer, deployed in the main control SoC module and the AI ​​coprocessor module, is configured to perform spatiotemporal alignment and semantic fusion of multimodal data.

2. The system according to claim 1, characterized in that, The correspondence between the physical interfaces of the interface adaptation layer and each functional module is as follows: The micro-nano LED display module, ToF depth sensing module, MEMS voice array module, IMU attitude sensing module, capacitive touch module, and AI coprocessor module are interconnected with the main control SoC module through FPC connectors. The power management module and the storage module are interconnected with the main control SoC module via BTB connectors; The peripheral interface module uses a USB-C connector and is exposed on the temple structure of the AR glasses.

3. The system according to claim 1, characterized in that, The binding relationship between the logical transmission protocol of the interface adaptation layer and each functional module is as follows: The micro-nano LED display module and the main control SoC module transmit video streams using the MIPI DSI protocol; The ToF depth sensing module, the IMU attitude sensing module and the main control SoC module use the SPI protocol to transmit sensor data. The MEMS voice array module and the main control SoC module transmit audio data using the I2S protocol. The AI ​​coprocessor module and the main control SoC module use the PCIe 3.0 protocol to transmit high-speed computing data; The power management module and the main control SoC module use the I2C protocol to transmit control commands.

4. The system according to claim 2, characterized in that, The physical interface meets the following electrical specifications: The USB-C connector supports the USB 3.2 Gen2 standard with a transmission rate of ≥10Gbps; The FPC connector has a differential impedance matching of 100Ω±10%, a MIPI DSI signal eye diagram height of ≥80%, and jitter of ≤0.1UI, conforming to the MIPI DSI v1.3 specification; The BTB connector has a pin pitch of ≤0.4mm and a single pin current carrying capacity of ≥2A, meeting the high current transmission requirements of the power management module.

5. The system according to claim 3, characterized in that, The logical transmission protocol uses a custom compact binary encapsulation format: The data packet header contains a 1-byte synchronization header, a 2-byte module ID, a 4-byte timestamp, and a 2-byte data length field; The data packet tail includes a 2-byte CRC-16 check field, and the data transmission error rate is ≤1×10⁻⁶. -6 .

6. The system according to claim 1, characterized in that, The AI ​​coprocessor module deploys a lightweight Transformer fusion model that has undergone structured pruning and INT8 quantization. The model has 8–10 MB of parameters, 4–8 attention heads, and 128–256 hidden layer dimensions. The model has an edge inference latency of ≤5ms on the AI ​​coprocessor and a multimodal interaction recognition accuracy of ≥95%.

7. The system according to claim 1, characterized in that, The spatiotemporal alignment accuracy of the multimodal fusion processing layer satisfies: Using the 1kHz timestamp of the IMU attitude sensing module as a benchmark, the timestamps of ToF gesture data, MEMS voice data, and capacitive touch data are calibrated using a cross-correlation coefficient peak alignment algorithm, with a time alignment error ≤0.5ms. The multi-sensor coordinate system is registered to the IMU coordinate system using a pre-calibrated rigid transformation matrix, with a spatial registration error ≤2mm.

8. The system according to claim 1, characterized in that, The system is also equipped with a dynamic power consumption optimization unit: When the system detects that the user has not interacted for 30 consecutive seconds, it automatically switches the ToF depth sensing module and the IMU attitude sensing module to a low-power sleep mode, reducing the power consumption of a single module from 120mW to below 15mW. By detecting ambient light intensity through a built-in CMOS light sensor, the brightness of the micro-nano LED display module is adjusted accordingly. When the light intensity is below 50 lux, the brightness drops from 500 nits to 200 nits, and the display power consumption is reduced by 40%.

9. A multimodal fusion AR intelligent interaction method based on the system described in any one of claims 1 to 8, characterized in that, Includes the following steps: S1. Synchronously collect multi-source sensor data through the interface adapter layer; S2. The multimodal fusion processing layer performs spatiotemporal alignment and semantic fusion of multimodal data and outputs fusion interaction instructions; S3. Arbitrate multi-source instructions based on preset priority rules and execute the highest priority instruction.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of claim 9.

Images