Autonomous mobile interaction and multi-mode control display system based on edge AI

Abstract

The invention discloses an autonomous mobile interaction and multi-mode control display system based on edge AI. The system comprises an all-terrain adaptive mobile module used for driving a display device to move on different types of terrains; the three-dimensional space interaction module is used for identifying the posture of a human body lying station and detecting to form environmental map information; the edge intelligent voice module is internally provided with an end-side large model platform or a mixed architecture voice engine and a special acceleration chip to realize an edge AI function and provide offline voice instruction recognition and continuous dialogue; the data sovereignty protection module carries out localization processing and encrypted storage on the user data; the multi-mode cooperative control module is used for fusing multi-source sensing data and executing a cooperative decision; the end-side large model module is used for providing localized AI reasoning capability; the ecological interconnection module provides a cross-platform protocol conversion gateway and an open API interface so as to establish application circulation and seamless projection with different systems of external equipment. According to the invention, autonomous movement, intelligent interaction, privacy protection and cross-platform seamless cooperation of equipment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of display devices, in particular to a display system based on edge AI autonomous mobile interaction and multi-modal control. BACKGROUND

[0002] The existing mobile smart screen display device has the following shortcomings: (1) lack of mobility: relying on manual push and pull, the resistance is obvious when moving on different terrains (such as carpets, uneven ground), single-person operation is more laborious, and the trajectory deviation is more than ±5cm when moving on flat ground, with low positioning accuracy. (2) Low space adaptation efficiency: when used in a lying state (such as a sofa or bed), the device position needs to be adjusted by standing up, and remote posture adaptation cannot be achieved; the screen angle and height adjustment lack intelligent memory function, and the best viewing angle needs to be repeatedly adjusted every time, with a debugging time ratio of more than 30%; (3) Strong network dependency: voice control function is completely disabled in offline state, and manual network configuration must be completed for the first use; voice interaction needs to be awakened every time, and continuous dialogue is not supported, with a lower operation fluency than smart phone assistants (continuous dialogue success rate less than 50%); (4) Privacy and security concerns: voice control needs to connect to external network to call third-party AI models, which may cause user behavior data leakage; there is no local privacy protection solution, and voice control cannot be realized in offline state (offline function coverage is 0%); (5) Ecological interconnection defects: lack of super desktop function, unable to directly project mobile phone application ecosystem; limited cross-device collaboration capability, with technical barriers in content flow between mobile terminals (screen projection delay is more than 300ms, and application flow success rate is less than 70%). SUMMARY

[0003] The present application aims to solve the problems of lack of mobility, low space adaptation efficiency, strong network dependency, privacy and security concerns, and ecological interconnection defects of existing mobile smart screen display devices, and provides a display system based on edge AI autonomous mobile interaction and multi-modal control, which realizes device autonomous movement, intelligent interaction, end-side large model and privacy protection, and seamless cross-platform collaboration. The technical solution adopted by the present application is:

[0004] The display system based on edge AI autonomous mobile interaction and multi-modal control comprises: a full-terrain self-adaptive mobile module for carrying a display device and moving the display device on different types of terrain; the full-terrain self-adaptive mobile module comprises a multi-modal driving chassis and a UWB positioning chip; the display device is detachably installed and configured on the upper end of the multi-modal driving chassis; the UWB positioning chip is integrated in the multi-modal driving chassis to obtain real-time positioning information of the multi-modal driving chassis.

[0005] Specifically, through the cooperation of the multi-modal driving chassis and the positioning module, centimeter-level trajectory control is achieved: the "driving mode adaptive switching algorithm" is adopted, and the real-time environment map is constructed through the UWB ultra-wideband signal and the SLAM algorithm, realizing centimeter-level trajectory control (trajectory accuracy ≤±2 cm) and automatic obstacle avoidance (obstacle avoidance success rate ≥99%).

[0006] The three-dimensional space interaction module includes a biological feature acquisition unit and an environment modeling unit. The biological feature acquisition unit acquires biological features around the display device to identify human lying and standing postures. The environment modeling unit detects the environment around the display device and models the environment to form environment map information for the all-terrain adaptive mobile module.

[0007] Specifically, an AI posture learning algorithm (based on an improved ResNet-50 network) is developed, a user-specific viewing parameter model is established (supporting multi-user identification, user identification accuracy ≥95%), and the screen elevation angle (0-60°) and height (50-150 cm) are dynamically adjusted to the best viewing position, with an adjustment response time ≤1s.

[0008] The edge intelligent voice module has a built-in end-side large model platform and a hybrid architecture voice engine, which realizes edge AI functions through a special acceleration chip (computing power ≥4TOPS), supports offline full-scene voice control and multi-round continuous dialogue.

[0009] Specifically: dual-mode inference architecture: hybrid architecture voice engine: lightweight CNN-LSTM model (parameter quantity ≤5M) is adopted, supporting offline identification of 1000+ basic instructions (such as "move to the study" and "increase brightness"), with an identification delay ≤100ms; end-side large model platform: 7B parameter voice semantic understanding model (inference speed ≥100tokens / s), realizing complex intent deep analysis (such as "continue playing the documentary that was not finished last night at the 32nd minute"); continuous dialogue technology breakthrough: supporting custom dialogue duration (1-10 minutes), without the need for repeated wake-up; maintaining dialogue coherence through a context memory engine (32K capacity), with an interaction delay ≤200ms and a success rate ≥90% (traditional solution ≤50%). Privacy and security design: voice data is processed in a closed loop within the module, raw audio is destroyed in real time, and only desensitized text instructions are retained; sensitive word library runs in TEE environment (such as "encrypt files" and "meeting records" triggering local data locking).

[0010] The data sovereignty protection module is used to provide a TEE trusted execution environment and an end-side federated learning framework to locally process and encrypt user data for privacy protection.

[0011] Specifically, user behavior data is encrypted by TEE (Trusted Execution Environment) technology, and external network interaction only transmits anonymized features (data desensitization rate 100%), avoiding leakage of original data; the end-side AI large model is locally trained based on a federated learning framework after speech data desensitization, supporting homomorphic encryption and secure multi-party computation (SMC).

[0012] A multi-modal collaborative control module is used to fuse multi-source perception data and perform collaborative decision-making; multi-source data space-time synchronization is achieved through hardware timestamp alignment; parallel processing pipeline is deployed to call algorithm resources of each module; a cross-modal attention mechanism is used to map composite instructions; sensitive operations are completed in a TEE environment for instruction desensitization.

[0013] An end-side large model module is used to provide localized AI inference capability; that is, to provide local inference capability for the edge intelligent voice module, ensuring smoothness of offline voice control; an improved ResNet-50 network of the three-dimensional space interaction module is driven to achieve millisecond-level pose recognition; in conjunction with the data sovereignty protection module, user portrait local training is achieved through a federated learning framework, so that sensitive data processing is completed in a TEE environment; the multi-modal collaborative control module is empowered with cross-scene recommendation function and composite instruction analysis based on 32K context understanding capability.

[0014] An ecological interconnection module provides a cross-platform protocol conversion gateway and an open API interface, and the display device establishes application flow conversion and seamless projection with different systems of external devices through the ecological interconnection module.

[0015] Specifically, the cross-platform protocol conversion gateway (WSP protocol) supports system-level interworking of HarmonyOS / iOS / Android, realizes three-mode flow conversion of APP image, projection and control; the open API interface is compatible with mainstream ecological seamless connection, with projection delay ≤50ms and application flow conversion success rate ≥95%.

[0016] Further, the multi-modal driving chassis adopts a composite structure of electric wheel type, tracked type and foot type; the all-terrain adaptive mobile module is equipped with an adaptive terrain switching algorithm to adapt to different types of terrain such as carpets, tiles, slopes and uneven ground.

[0017] Further, the edge intelligent voice module is in communication connection with the display device and the all-terrain adaptive mobile module respectively.

[0018] Further, the multi-modal driving chassis supports adaptive switching of at least 5 terrains and centimeter-level trajectory control.

[0019] Further, the system adopts a multi-modal control method to realize collaborative processing of voice, gesture and touch signals through a hierarchical control architecture, and dynamically adjusts the control strategy by using a hybrid inference engine, with a response time ≤200ms (traditional scheme 300-500ms).

[0020] Further, the voice data in the data sovereignty protection module is locally extracted for features, and the original audio is not stored; and the gesture data adopts differential privacy noise injection (ε = 1.0, δ = 1e-5).

[0021] Further, the three-dimensional space interaction module is equipped with an AI posture learning algorithm (based on an improved ResNet-50 network), establishes a user-specific viewing parameter model (supports multi-user identification, and the user identification accuracy is ≥ 95%), dynamically adjusts the screen elevation angle (0-60°) and height (50-150 cm) to the best viewing position through the control of the matching mechanical structure, and the adjustment response time is ≤ 1s; the three-dimensional space interaction module biological feature acquisition unit realizes millimeter-level tracking (error ≤ 5mm) of the human body lying posture and dynamic adjustment of the screen.

[0022] Further, the biological feature acquisition unit includes a 3D ToF camera (resolution 1080P, frame rate 30fps); and the environment modeling unit includes a 60GHz millimeter wave radar and an RGBD camera.

[0023] Further, the input end of the edge intelligent voice module is connected with a (4+1 channel) microphone array and a noise reduction module (signal-to-noise ratio ≥ 65dB), to realize 5-meter far-field sound pickup and voice noise reduction.

[0024] Further, the display device is a display screen; and the display device is configured with a magnetic attraction module, and the display device is detachably installed on an external device through the magnetic attraction module.

[0025] Specifically, the display device is conveniently installed and removed through the magnetic attraction module, is adapted to a desktop support, a wall-mounted component and other external devices, covers multiple scene requirements such as office (conference mode), learning (split screen mode) and entertainment (cinema mode), and the scene switching time is ≤ 3s.

[0026] Further, the multi-modal collaborative control module includes: a multi-source perception layer that aligns the space-time data (synchronization error ≤ 1ms) of the millimeter wave radar, the 3D ToF camera, the microphone array and the UWB module through a hardware timestamp; a dynamic arbitration mechanism that executes conflict instructions according to priorities (obstacle avoidance demand > privacy instruction > ecological interconnection task), and resolves composite instructions through a cross-modal attention mechanism (for example, voice "turn up the volume" + gesture up → trigger volume increase); and performance indicators: multi-modal response delay ≤ 200ms (traditional scheme 300-500ms), emergency obstacle avoidance response time ≤ 0.2s (improvement 150%), composite instruction execution accuracy rate ≥ 95% (traditional scheme 60-70%).

[0027] Specifically, the multi-modal collaborative control module realizes an offline control closed loop: voice wake-up → radar gesture detection → UWB target positioning → SLAM path planning → chassis movement → screen pre-adjustment → ToF posture verification.

[0028] Further, the end-side large model module comprises: computing power architecture: 16-20 TOPS NPU chip supports 7B parameter model local deployment (traditional scheme ≤1B parameter); inference acceleration: realizes ≥100 tokens / s generation speed (400% improvement), supports 32K context continuous dialogue understanding; federated learning framework: interacts with the data sovereignty protection module, and user data is 100% localized training (leakage risk ≤0.1%); scene intelligence: cross-scene recommendation accuracy based on user habits ≥85% (traditional scheme 60-70%), model switching time ≤200 ms (500% improvement).

[0029] Compared with the prior art, the above technical scheme has the following beneficial effects:

[0030] 1. Mobile performance improvement: terrain adaptive switching response time ≤1s, trajectory accuracy improved by 60% (±2cm vs ±5cm) compared with traditional devices, and single-person operation effort reduced by 80%;

[0031] 2. Interaction efficiency optimization: viewing parameter automatic adjustment saves users 80% of debugging time, continuous dialogue smoothness improved by 50% (success rate ≥90% vs 50%), and offline voice function coverage rate increased from 0% to 100%;

[0032] 3. Privacy and security enhancement: localized voice processing avoids data export, federated learning framework reduces privacy leakage risk by 90% (leakage probability ≤0.1%), and TEE environment ensures data encryption storage;

[0033] 4. Ecological compatibility expansion: cross-system screen projection delay reduced by 83% (50ms vs 300ms), and application stream conversion success rate improved to more than 95%;

[0034] 5. Intelligent decision-making breakthrough: end-side large model realizes ≥100 tokens / s generation speed (400% improvement), and 7B parameter scale supports complex intent understanding; multi-modal collaborative control improves composite instruction execution accuracy to ≥95% (traditional scheme 60-70%); emergency obstacle avoidance response time ≤0.2s (150% improvement), ensuring mobile safety.

[0035] The application realizes efficient fusion of multi-modal signals (response delay ≤200 ms) through a hierarchical control architecture, combines environmental adaptive algorithms with user behavior learning, and significantly improves interaction naturalness (WPM ≥40 words / minute). The privacy protection mechanism adopts triple protection of federated learning + TEE + differential privacy, ensures that user data does not leave the domain throughout, and builds a safe and reliable edge intelligent interaction system. BRIEF DESCRIPTION OF DRAWINGS

[0036] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0037] Figure 1 The structure diagram of the display system for autonomous mobile interaction and multi-modal control based on edge AI of the application;

[0038] Figure 2 The principle architecture diagram of the all-terrain adaptive mobile module of the application;

[0039] Figure 3 The principle architecture diagram of the three-dimensional space interaction module of the application;

[0040] Figure 4 The principle architecture diagram of the edge intelligent voice module of the application;

[0041] Figure 5 The principle architecture diagram of the data sovereignty protection module of the application;

[0042] Figure 6 The principle architecture diagram of the ecological interconnection module of the application;

[0043] Figure 7 The application architecture diagram of the display device magnetic adsorption of the application;

[0044] Figure 8 The principle architecture diagram of the end-side large model of the application;

[0045] Figure 9 The architecture diagram of the multi-modal collaborative control module of the application. DETAILED DESCRIPTION

[0046] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application.

[0047] As shown in one of Figures 1 to 9 The application discloses a display system for autonomous mobile interaction and multi-modal control based on edge AI, which comprises:

[0048] The all-terrain adaptive mobile module, as shown in Figure 2As shown, for carrying display devices and driving display devices to move on different types of terrain; all-terrain adaptive mobile module includes multi-modal driving chassis and UWB positioning chip; the display device is detachably mounted on the upper end of the multi-modal driving chassis; the UWB positioning chip is integrated in the multi-modal driving chassis to obtain the positioning information of the multi-modal driving chassis in real time;

[0049] Specifically, through the cooperation of the multi-modal driving chassis and the positioning module, centimeter-level trajectory control is achieved: a "driving mode adaptive switching algorithm" is adopted to construct a real-time environment map through UWB ultra-wideband signals and SLAM algorithm, realizing centimeter-level trajectory control (trajectory accuracy ≤±2 cm) and automatic obstacle avoidance (obstacle avoidance success rate ≥99%).

[0050] The three-dimensional space interaction module, as shown in Figure 3 includes a biological feature acquisition unit and an environment modeling unit, the biological feature acquisition unit acquires the biological features around the display device in a lying or standing posture; the environment modeling unit detects the environment around the display device and models the environment to form environment map information for the all-terrain adaptive mobile module;

[0051] Specifically, an AI posture learning algorithm (based on an improved ResNet-50 network) is developed, a user-specific viewing parameter model is established (supporting multi-user identification, user identification accuracy ≥95%), the screen elevation angle (0-60°) and height (50-150 cm) are dynamically adjusted to the best viewing position, and the adjustment response time is ≤1s.

[0052] The edge intelligent voice module, as shown in Figure 4 built-in end-side large model platform and hybrid architecture voice engine, through a special acceleration chip (computing power ≥4TOPS) to realize edge AI function, support offline full-scene voice control and multi-round continuous dialogue;

[0053] Specifically: dual-mode reasoning architecture: hybrid architecture voice engine: adopts lightweight CNN-LSTM model (parameter quantity ≤5M), supports offline recognition of 1000+ basic instructions (such as "move to the study" and "turn up the brightness"), and the recognition delay is ≤100ms; End-side large model platform: 7B parameter speech semantic understanding model (inference speed ≥100tokens / s), realizes complex intent deep analysis (such as "continue playing the documentary that was not finished last night at the 32nd minute"); Continuous dialogue technology breakthrough: supports custom dialogue duration (1-10 minutes), without the need to repeat the wake-up; Through the context memory engine (32K capacity) to maintain the coherence of the dialogue, the interaction delay is ≤200ms, and the success rate is ≥90% (traditional scheme ≤50%). Privacy and security design: voice data is processed in a closed loop within the module, and the original audio is destroyed in real time, leaving only desensitized text instructions; Sensitive word library runs in TEE environment (such as "encrypt files" and "meeting records" to trigger local data locking).

[0054] Data sovereignty protection module, such as Figure 5 shown, for providing a TEE trusted execution environment and an end-side federated learning framework to locally process and encrypt store user data;

[0055] Specifically, user behavior data is encrypted and stored through TEE (trusted execution environment) technology, and only anonymized features (data desensitization rate 100%) are transmitted for external network interaction, avoiding leakage of original data.

[0056] Multi-modal collaborative control module for fusing multi-source perception data and performing collaborative decision-making; Time synchronization of multi-source data is achieved through hardware timestamp alignment; Deploy parallel processing pipeline to call algorithm resources of each module; Cross-modal attention mechanism is used to map composite instructions; Sensitive operations are completed in TEE environment for instruction desensitization;

[0057] Specifically, as Figure 9As shown, the multi-modal collaborative control module includes: a multi-source perception layer: using integrated millimeter wave radar (micro-motion detection accuracy 0.1mm), 3D ToF camera, microphone array and UWB positioning module, and aligning the space-time data of millimeter wave radar, 3D ToF camera, microphone array and UWB module through hardware timestamp (synchronization error ≤1ms); as shown in Table 1, a dynamic arbitration mechanism: executing conflict instructions according to priority (obstacle avoidance demand > privacy instruction > ecological interconnection task), and resolving composite instructions through cross-modal attention mechanism (such as voice "turn up the volume" + gesture up → trigger volume increase); performance indicators: multi-modal response delay ≤200ms (traditional scheme 300-500ms), emergency obstacle avoidance response time ≤0.2s (improved by 150%), composite instruction execution accuracy rate ≥95% (traditional scheme 60-70%). Deploying parallel processing pipeline: as shown in Table 2, the visual pipeline calls the improved ResNet-50 network to identify human posture (delay ≤50ms), the voice pipeline resolves offline instructions (delay ≤100ms), and the motion pipeline plans the path through the SLAM algorithm (delay ≤30ms); offline control closed loop: voice wake-up → radar gesture detection → UWB positioning target → SLAM path planning → chassis movement → screen pre-adjustment → ToF posture verification. Among them, the data fusion adopts an adaptive weighting algorithm: Math W_{fusion}=0.4·W_{radar}+0.3·W_{ToF}+0.3·W_{audio}.

[0058] Table 1: Dynamic arbitration mechanism

[0059] Conflict scenarios Dominant modalities Execution rules Sudden obstacles in movement Visual radar Interrupt non-safe instructions and emergency obstacle avoidance Privacy-sensitive operations Local voice Disable camera data export

[0060] Table 2: Parallel processing pipeline

[0061] Pipeline types Processing models Resource allocation Visual pipeline Improved ResNet-50 8 TOPS Voice pipeline CNN-LSTM hybrid architecture 4.5 TOPS Motion pipeline SLAM + path planning algorithm 2 TOPS

[0062] End-side large model module, for providing localized AI inference capability; that is, providing local inference capability for edge intelligent voice module, guaranteeing the fluency of offline voice control; driving the improved ResNet-50 network of the three-dimensional space interaction module to realize millisecond-level posture recognition; in linkage with the data sovereignty protection module, realizing local training of user portrait through the federated learning framework to complete sensitive data processing in the TEE environment; enabling the cross-scene recommendation function of the multi-modal collaborative control module and resolving composite instructions based on 32K context understanding capability;

[0063] Specifically, as Figure 8As shown, the end-side AI large model (based on a federated learning framework) enables local training after voice data desensitization, supports homomorphic encryption and secure multi-party computation (SMC). As shown in Table 3, the computing power architecture of the end-side large model module: integrates 16-20 TOPS NPU chips, supports 3B / 7B parameter model deployment; inference performance: achieves a generation speed of ≥100 tokens / s, supports 32K context understanding, and supports long instruction parsing (such as "continue playing the documentary from the 32nd minute that was not finished last night"); privacy protection: through the federated learning framework and the data sovereignty module, 100% local closed-loop training of user data is realized, and privacy protection of "data not out of domain" training is realized; scene application: as shown in Table 4, the end-side AI large model is used to realize cross-scene intelligent recommendation (meeting / learning / entertainment) based on user habit analysis, with a recommendation accuracy of ≥85%.

[0064] Table 3: Technical advantages of end-side AI large model

[0065]

[0066] Table 4: Cross-scene recommendation application

[0067] Scenarios Functions Performance indicators Meeting scenarios Real-time generation of meeting minutes Accuracy ≥ 90% Learning scenarios Push personalized learning materials Matching degree ≥ 85% Entertainment scenarios Intelligent recommendation of film and television content Recall rate ≥ 95%

[0068] Ecological interconnection module, such as Figure 6 As shown, the cross-platform protocol conversion gateway and open API interface are provided, and the display device establishes application flow conversion and seamless projection with different systems of external devices through the ecological interconnection module.

[0069] Specifically, the cross-platform protocol conversion gateway (WSP protocol) supports system-level interconnection of HarmonyOS / iOS / Android, realizes APP image, projection, and control three-mode flow conversion. The open API interface is compatible with mainstream ecological seamless connection, with a projection delay of ≤50 ms and an application flow conversion success rate of ≥95%.

[0070] Further, the multi-modal driving chassis adopts an electric wheel type, a track type, and a foot type composite structure; the all-terrain adaptive mobile module is equipped with an adaptive terrain switching algorithm to adapt to different types of terrain such as carpets, tiles, slopes, and uneven ground.

[0071] Further, the edge intelligent voice module is in communication connection with the display device and the all-terrain adaptive mobile module, respectively.

[0072] Further, the multi-modal driving chassis supports adaptive switching of at least 5 terrains and centimeter-level trajectory control.

[0073] Further, the system adopts a multi-modal control method to realize the collaborative processing of voice, human lying posture, and touch signal through a hierarchical control architecture, and dynamically adjusts the control strategy using a hybrid inference engine, with a response time ≤ 300 ms. Specifically, the specific architecture of the multi-modal control method is as follows:

[0074] The perception layer: deploy an end-side large model platform (7B parameters) to realize localized modal recognition; a special voice ASIC chip (4.5TOPS) processes voice streams; an NPU accelerates visual processing units (8TOPS) to analyze human posture.

[0075] The decision layer: a hybrid inference engine, i.e., a rule engine + a lightweight LSTM network, is used.

[0076] Dynamic strategy adjustment: when the current scene is emergency obstacle avoidance, the voice instruction is prioritized; if the current scene is a private meeting, the visual data export is disabled.

[0077] Collaborative mechanism: align multi-modal signals (error ≤ 1 ms) through IEEE 1588 Precision Time Protocol (PTP), and prioritize voice triggering for emergency instructions (such as device emergency stop).

[0078] Feature extraction: MFCC features + attention mechanism are used for voice feature extraction, with a key frame recognition rate ≥ 98%; 3D skeleton key point detection (COCO pre-training) is used for human posture feature extraction, with an attitude error ≤ 3 mm; a pressure distribution heat map (16x16) with a resolution of 0.1 mm is used for touch feature extraction.

[0079] Conflict resolution: infer user intent through Bayesian network (confidence ≥ 0.95 to execute instructions). For example, voice "turn up brightness" + touch downslide → execute brightness + 20% (voice priority).

[0080] The following describes the control principle of a multi-modal control embodiment:

[0081] Basic scenario:

[0082] Voice instruction: "turn the screen to the living room sofa" (signal-to-noise ratio 25 dB).

[0083] Human posture: lying down (trunk inclination 25°).

[0084] System response: end-side large model extracts keywords "turn" and "sofa" (delay ≤ 50 ms); ToF camera verifies posture, calculates optimal height (65 cm) and elevation angle (40°); chassis rotates screen (error ≤ 1°) and synchronously adjusts posture;

[0085] Meeting scenario (end-side large model linkage):

[0086] Emergency obstacle avoidance (multi-modal coordination): millimeter wave radar detects moving objects within 3m (confidence ≥ 99%); ToF identifies child intrusion (delay ≤ 50ms).

[0087] End-side large model decision chain: interrupt current instruction → play alarm sound; chassis emergency braking (response ≤ 0.2s); retreat 1.5m and adjust screen orientation.

[0088] Further, the voice data in the data sovereignty protection module is locally extracted for features, and the original audio is not stored; and the gesture data uses differential privacy noise injection (ε = 1.0, δ = 1e-5).

[0089] Further, the three-dimensional space interaction module is equipped with an AI pose learning algorithm (based on an improved ResNet-50 network) to establish a user-specific viewing parameter model (supporting multi-user identification, user identification accuracy ≥ 95%), and dynamically adjusts the screen elevation angle (0-60°) and height (50-150cm) to the best viewing position through the control of the matching mechanical structure, with an adjustment response time ≤ 1s; the three-dimensional space interaction module biological feature acquisition unit realizes millimeter-level tracking of the human lying posture and dynamic adjustment of the screen.

[0090] Specifically, the three-dimensional space interaction module models the user operation sequence through an LSTM network to predict the next operation (accuracy ≥ 85%). User preferences such as voice command words and gesture sensitivity are stored in the local database (SQLite),

[0091] Specifically, the mechanical structure is a double-axis push rod; two independent electric push rods are integrated into the back of the screen of the display device, respectively controlling the adjustment of the elevation angle between 40 degrees and the depression angle between 30 degrees and the horizontal rotation (-90° to +90°); the double-axis push rod group is installed to the base of the display device, and the double-axis push rod group is connected to the spherical hinge and the screen support. The spherical hinge allows fine adjustment of the screen within a range of ±5°, compensating for mechanical errors, and the spherical hinge is made of aviation aluminum CNC machining, with a load capacity ≥ 12kg and an IP65 protection level. The parameters of the electric push rod are: stroke: 100mm (adapted to a 32-inch screen); thrust: 150N (safety factor 1.5); transmission mode: worm and gear mechanism, built-in magnetostrictive displacement sensor (resolution 0.01mm), which can be designed with different parameters according to specific sizes. The electric push rod uses a stepper motor (torque 0.8N·m) driven by a PWM signal, with a built-in limit switch to prevent overtravel.

[0092] Or the mechanical structure is or a steering engine group, two groups of industrial grade steering engines are integrated at the back of the screen of the display device, and the rotary motion is converted into linear thrust through a gear and rack mechanism; the parameters of the industrial grade steering engine are: torque: 20kg·cm (single group); gear modulus: 1.5mm, transmission ratio 3:1; feedback device: potentiometer (resolution 0.1°), which can be designed with different parameters according to the specific size. The steering engine is directly controlled by a PWM signal, and the potentiometer feedbacks the angle data in real time.

[0093] It should be noted that: through the cooperation of the electric push rod and the steering engine double drive scheme, or separately using the electric push rod or the steering engine drive scheme or using the servo machine plus the harmonic reducer force conversion structure are all within the protection scope of the present application.

[0094] Further, the biological feature acquisition unit includes a 3D ToF camera (resolution 1080P, frame rate 30fps); the environment modeling unit includes a 60GHz millimeter wave radar and an RGBD camera, realizing millimeter level tracking (pose recognition error ≤5mm) and device displacement detection in a lying posture.

[0095] Specifically, as a feasible implementation manner, a depth camera (ToF+RGB dual mode, 1080P@60fps) and a capacitive touch screen (10-point touch, resolution 4096×2160) are used to realize the sensing layer and touch.

[0096] Further, as shown in Figure 4 The input end of the edge intelligent voice module is connected with a (4+1 channel) microphone array (4-microphone ring layout, signal-to-noise ratio 65dB) and a noise reduction module, and voice acquisition and voice noise reduction are realized through the microphone array and the noise reduction module.

[0097] Specifically, based on the UWB positioning data, the beam direction of the microphone array is dynamically adjusted, and the signal-to-noise ratio is improved by 12dB. At the same time, the ToF camera automatically switches the infrared auxiliary lighting (starts when the light intensity is <50lux).

[0098] Further, the display device is a display screen; as shown in Figure 7 The display device is provided with a magnetic attraction module, which is convenient to install and remove through the magnetic attraction module, adapts to external devices such as desktop supports, wall hanging components, covers office (conference mode), learning (split screen mode), entertainment (cinema mode) and other multi-scene needs, and the scene switching time is ≤3s.

[0099] Compared with the prior art, the application has the following beneficial effects: 1. Improved mobility: terrain adaptive switching response time is less than or equal to 1s, trajectory accuracy is improved by 60% (±2cm vs ±5cm) compared with traditional devices, and the degree of labor intensity of single operation is reduced by 80%; 2. Optimized interaction efficiency: automatic adjustment of viewing parameters saves 80% of the user's debugging time, continuous dialogue fluency is improved by 50% (success rate is greater than or equal to 90% vs 50%), and offline voice function coverage is improved from 0% to 100%; 3. Strengthened privacy and security: localized voice processing avoids data transmission, a federated learning framework reduces the risk of privacy leakage by 90% (leakage probability is less than or equal to 0.1%), and a TEE environment ensures encrypted data storage; 4. Ecological compatibility expansion: cross-system screen projection delay is reduced by 83% (50ms vs 300ms), application flow conversion success rate is improved to more than 95%, and ecological barriers are broken; 5. Intelligent decision-making breakthrough: an end-side large model realizes a generation speed of greater than or equal to 100 tokens / s (increased by 400%), a 7B parameter size supports complex intent understanding; multi-modal collaborative control improves composite instruction execution accuracy to greater than or equal to 95% (traditional scheme 60-70%); emergency obstacle avoidance response time is less than or equal to 0.2s (increased by 150%), ensuring mobile safety; cross-scene recommendation accuracy is greater than or equal to 85% (increased by 40%), and model switching time is less than or equal to 200ms (increased by 500%).

[0100] The application realizes efficient fusion of multi-modal signals through a hierarchical control architecture, combines environment adaptive algorithm and user behavior learning, and significantly improves interaction naturalness (WPM is greater than or equal to 40 words / minute). A privacy protection mechanism adopts data desensitization and federated learning technology to ensure that user data is not out of the domain throughout. The application adopts an end-side multi-modal arbitration framework to realize hardware synchronization (error is less than or equal to 1ms) + dynamic weight fusion; a Bayesian network solves instruction conflicts (accuracy is greater than or equal to 95%); the application runs a 7B large model in a TEE environment to realize privacy-performance balance, and federated learning realizes data training without domain out; the application supports millimeter-level pose adjustment through a double-shaft push rod / steering gear double scheme, realizes centimeter-level movement through SLAM path planning+UWB positioning, and achieves mechanical-algorithm collaborative innovation.

[0101] Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

Claims

1. A display system for autonomous mobile interaction and multi-modal control based on edge AI, characterized in that: It comprises: An all-terrain adaptive mobile module for carrying a display device and moving the display device on different types of terrain; the all-terrain adaptive mobile module comprises a multi-modal driving chassis and a UWB positioning chip; the display device is detachably mounted on the upper end of the multi-modal driving chassis; the UWB positioning chip is integrated in the multi-modal driving chassis to obtain real-time positioning information of the multi-modal driving chassis; a real-time environment map is constructed by the ultra-wideband signal of the UWB positioning chip and the SLAM algorithm, and the multi-modal driving chassis realizes trajectory control; A three-dimensional space interaction module, which includes a biological feature acquisition unit and an environment modeling unit, the biological feature acquisition unit acquires biological features around the display device to identify human lying and standing postures; the environment modeling unit detects the environment around the display device and models the environment to form environment map information for use by the all-terrain adaptive mobile module; An edge intelligent voice module, which has an end-side large model platform or a hybrid architecture voice engine and a special-purpose acceleration chip to realize edge AI functions, and facilitates offline voice command recognition and continuous dialogue; A data sovereignty protection module for providing a TEE trusted execution environment and an end-side federated learning framework to process user data locally and store it encrypted to protect privacy; A multi-modal collaborative control module for fusing multi-source perception data and performing collaborative decision-making; Multi-source data space-time synchronization is achieved through hardware timestamp alignment; parallel processing pipelines are deployed to call algorithm resources of each module; a cross-modal attention mechanism is used to map composite instructions; sensitive operations are completed in the TEE environment to desensitize instructions; An end-side large model module for providing local AI inference capabilities; that is, providing local inference capabilities for the edge intelligent voice module to ensure the smoothness of offline voice control; An improved ResNet-50 network driving the three-dimensional space interaction module to achieve millisecond-level posture recognition; in conjunction with the data sovereignty protection module, user portrait local training is realized through the federated learning framework to complete sensitive data processing in the TEE environment; The multi-modal collaborative control module enables cross-scene recommendation functions and analyzes composite instructions based on 32K context understanding capabilities; An ecological interconnection module provides cross-platform protocol conversion gateways and open API interfaces, and the display device establishes application flow conversion and seamless projection with different systems of external devices through the ecological interconnection module.

2. The edge-AI based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: The multi-modal driving chassis adopts an electric wheel type, a tracked type, and a foot type composite structure; the all-terrain adaptive mobile module carries an adaptive terrain switching algorithm to adapt to different types of terrain.

3. The edge-AI based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: The edge intelligent voice module is in communication connection with the display device and the all-terrain adaptive mobile module, respectively.

4. The edge-AI-based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: A multi-modal control method is used to realize collaborative processing of voice, human lying and standing postures, and touch signals through a hierarchical control architecture, and a hybrid inference engine is used to dynamically adjust the control strategy with a response time ≤ 300 ms.

5. The edge-AI based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: The three-dimensional space interaction module is provided with an AI posture learning algorithm to establish a user-specific viewing parameter model, so as to dynamically adjust the screen elevation and height to an optimal viewing position according to the user-specific viewing parameters; the three-dimensional space interaction module realizes millimeter-level tracking of a human body lying posture and dynamic adjustment of the screen through the biological feature acquisition unit.

6. The edge-AI based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: The biological feature acquisition unit includes a 3D ToF camera; the environment modeling unit includes a 60GHz millimeter wave radar and an RGBD camera.

7. The edge-AI-based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: The input end of the edge intelligent voice module is connected with a microphone array and a noise reduction module, and voice collection and voice noise reduction are realized through the microphone array and the noise reduction module.

8. The edge-AI-based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: A magnetic attraction module is arranged on the display device, and the display device is detachably installed on an external device through the magnetic attraction module.

9. The edge-AI-based autonomous mobile interaction and multi-modal control display system of claim 1, wherein: In the data sovereignty protection module, feature extraction is performed on voice data locally, and original audio is not stored; and gesture data is injected with differential privacy noise.

Images