A distributed wearable multi-modal AI interaction system

Abstract

The application discloses a distributed wearable multi-modal AI interaction system, which relates to a distributed collaborative architecture of an AI earphone device, an AI glasses device and a mobile terminal device, three exclusive wireless communication links are built, the real-time synchronization of task state, execution progress and feedback state messages is realized between the AI earphone device and the AI glasses device through a third communication link, the mobile terminal device is provided with an algorithm scheduling module, the task type is determined after the voice and visual dual-modal feature fusion is graded, and the computing power is dynamically distributed to the wearable device or the mobile terminal device for execution, the application also supports the adaptive switching of a single working mode and a collaborative working mode, and a factory pre-binding mechanism realizes the automatic networking of multiple devices. Compared with the prior art, the application can realize voice-visual joint multi-modal AI interaction, is suitable for multiple use scenarios such as commuting, outdoor real scene recognition and travel explanation, and has the advantages of low time delay, low power consumption and distributed collaborative interaction.

Description

Technical Field

[0001] This invention relates to the fields of wearable smart devices, multimodal large-scale human-computer interaction distributed scheduling technology, and in particular a distributed wearable multimodal AI interaction system. Background Technology

[0002] Currently, existing multimodal wearable smart devices have the following problems: 1. Simple hardware architecture and lack of distributed dual-wear collaborative carrier; existing multimodal wearable smart devices generally only use a single pair of glasses / a single pair of headphones, which cannot simultaneously collect environmental visual images and user natural voice dual-channel multimodal information, resulting in missing human-computer interaction input dimensions and poor information integrity;

[0003] 2. The wireless communication link is single and relies on general Bluetooth, resulting in high latency and synchronization failure. Existing multimodal wearable smart devices communicate with mobile phones through a single Bluetooth link, and there is no direct connection channel between headphones and glasses. The general Bluetooth protocol does not have customized messages for lightweight task status synchronization of wearable devices, resulting in high latency in task progress and feedback status synchronization across devices, which can easily lead to problems such as asynchronous interaction feedback and device status conflicts.

[0004] 3. Multimodal large-scale model inference lacks a distributed hierarchical scheduling mechanism; existing multimodal wearable smart devices perform fusion inference entirely on the glasses / phone alone, without dynamically allocating execution hardware according to task type (voice, vision, fusion); lightweight voice tasks, image recognition tasks, and complex multimodal question answering tasks all occupy the mobile terminal's computing power, resulting in wasted computing resources and high interaction response latency;

[0005] 4. Fixed working mode, unable to adapt to individual / collaborative scenarios; existing multimodal wearable smart devices only support stand-alone independent operation, and there is no control logic for two devices to collaboratively execute the same AI task; scenarios such as users wearing headphones to listen to music alone, using glasses to take photos for recognition alone, and wearing both ears together for real-time multimodal question answering cannot be adaptively switched, limiting the applicable scenarios.

[0006] 5. The device networking is cumbersome and the pairing cost of multiple wearable devices is high; headphones and glasses require users to manually pair them with mobile phones and with each other. After changing devices or restarting, the network is disconnected and cannot be automatically restored. The user operation threshold is high.

[0007] Existing patents disclose a multimodal smart glasses interaction system that uses only a single pair of glasses as the interaction carrier, supports only image-based single-modal input, lacks a voice-visual dual-wearable device distributed collaborative architecture, and lacks a direct communication link between the headphones and glasses. It relies solely on local processing on the glasses or communication from a single terminal, does not have a three-wireless communication link network, cannot achieve real-time synchronization of status across wearable devices, does not have an independent / collaborative dual-mode adaptive switching mechanism, and multimodal fusion inference is executed only locally on the glasses. The computing power allocation is singular, and there is no distributed task scheduling strategy.

[0008] Existing patents also disclose a method for controlling smart glasses with infrared smart home based on voice recognition. The interaction carrier is limited to AI glasses, and there are no AI headphones as an independent voice acquisition entry point. There is only one-way voice control output, no large-scale inference model for visual + voice multimodal fusion, and wireless communication only uses general Bluetooth single-point connection. There is no dedicated communication protocol for direct connection between wearable devices, and the devices cannot synchronize their task execution status. It only supports single-device independent work and has no logic for collaborative task execution of multiple wearable devices.

[0009] Therefore, how to design a distributed wearable multimodal AI interaction system that can overcome the shortcomings of existing technologies is a technical problem that the industry urgently needs to solve. Summary of the Invention

[0010] To address the shortcomings of existing multimodal wearable smart devices, this invention proposes a distributed wearable multimodal AI interaction system. The system aims to build a three-link dedicated wireless communication network to achieve high-speed, low-latency direct communication between wearable devices and mobile terminal devices, as well as between wearable devices themselves.

[0011] The technical solution of this invention is to propose a distributed wearable multimodal AI interaction system, comprising:

[0012] The AI ​​headset device is used to collect user voice input and output audio. The AI ​​headset device has a built-in first wireless communication module for establishing a first communication link with a mobile terminal device.

[0013] AI glasses device, which is used to collect environmental visual images and video visual information and to provide feedback visual interaction information, has a second wireless communication module built into the AI ​​glasses device to establish a second communication link with the mobile terminal device.

[0014] And mobile terminal devices, which have a built-in scheduling algorithm module, can obtain user voice input collected by the AI ​​headset device and environmental visual images and video visual information collected by the AI ​​glasses device through the first communication link and the second communication link, and perform multimodal fusion tasks and allocate lightweight tasks to the AI ​​headset device and the AI ​​glasses device;

[0015] A third wireless communication module is also provided in the AI ​​headset device and the AI ​​glasses device, which is used to establish a third communication link for bidirectional transmission between the AI ​​headset device and the AI ​​glasses device. The AI ​​headset device and the AI ​​glasses device can exchange messages about the task status, execution progress and feedback status of the lightweight task through the third communication link.

[0016] Furthermore, the distributed wearable multimodal AI interaction system includes an adaptively switchable individual working mode and a collaborative working mode;

[0017] When the AI ​​headset and the AI ​​glasses work simultaneously, the third communication link is successfully established, and the distributed wearable multimodal AI interaction system works in collaborative mode.

[0018] When the AI ​​headset or the AI ​​glasses are working independently, and the establishment of the third communication link fails, the distributed wearable multimodal AI interaction system operates in independent working mode.

[0019] In the standalone working mode, if the connected wearable device is an AI headset device, a first communication link is established; if the connected wearable device is an AI glasses device, a second communication link is established.

[0020] Furthermore, the AI ​​glasses device also integrates a camera module, a feedback module, and a first local microprocessor;

[0021] The feedback module includes at least a micro vibration unit and an LED array optical prompting unit;

[0022] The camera module is used to capture visual images and complete the visual lightweighting task locally through the first local microprocessor.

[0023] The AI ​​glasses device can upload the image preprocessing results to the mobile terminal device via the second communication link, or synchronize the image preprocessing results to the AI ​​earphone device via the third communication link.

[0024] Furthermore, the AI ​​headset device also integrates a microphone module, an audio output module, and a second local microprocessor;

[0025] The microphone module is used to collect user voice input and complete the voice lightweighting task locally through the second local microprocessor;

[0026] The AI ​​headset device can upload the results of voice interaction to the mobile terminal device through the first communication link, or synchronize the results of voice interaction to the AI ​​glasses device through the third communication link.

[0027] Furthermore, the AI ​​headset device shown is the main interaction triggering device of the distributed wearable multimodal AI interaction system. After the AI ​​headset device completes the collection of user voice input, it triggers the mobile terminal device to perform multimodal fusion tasks and assign lightweight tasks to the AI ​​headset device and the AI ​​glasses device.

[0028] Furthermore, the algorithm execution steps of the mobile terminal device include:

[0029] The temporal features in the user's voice input collected by the AI ​​headset device and the image spatial features in the environmental visual images and video visual information collected by the AI ​​glasses device are obtained. The temporal features and the image spatial features are then normalized to generate a fused feature vector.

[0030] Based on the fusion feature vector, the current task type is identified and the task type is divided into speech lightweight task, vision lightweight task, and multimodal fusion task.

[0031] The voice lightweighting task is sent to the AI ​​headset device, the vision lightweighting task is sent to the AI ​​glasses device, and the multimodal fusion task is executed through the mobile terminal device;

[0032] The task status, execution progress, and feedback status of the voice lightweight task, vision lightweight task, and multimodal fusion task are acquired and updated through the first communication link, the second communication link, and the third communication link.

[0033] Furthermore, the third wireless communication module adopts a lightweight state synchronization communication protocol;

[0034] The lightweight state synchronization communication protocol adopts a message layered structure and includes task status messages, execution progress messages, and feedback control messages. Each message has a device ID frame, a status type frame, a data payload frame, and a check frame. The maximum length of a single message is 128 bytes.

[0035] Furthermore, the lightweight state synchronization communication protocol uses 10ms short-slot heartbeat packets, and only retransmits state synchronization messages when a message is lost.

[0036] When the AI ​​headset, the AI ​​glasses, and the mobile terminal communicate, the third communication link has a higher priority than the first and second communication links.

[0037] Furthermore, the AI ​​earphone device and the AI ​​glasses device are bound to a unique identification code;

[0038] When either the AI ​​headset device or the AI ​​glasses device is connected to the mobile terminal device, the mobile terminal device can read the unique identification code to establish the third communication link.

[0039] Compared with the prior art, the present invention has at least the following beneficial effects:

[0040] 1. This invention constructs a dedicated communication architecture of distributed dual wearable devices + three communication links, which can solve the communication defects of existing single devices and single links;

[0041] 2. This invention employs a distributed multimodal large model scheduling algorithm, which can classify tasks and dynamically allocate computing power, thereby saving computing power and reducing interactive response latency.

[0042] 3. This invention can automatically switch between standalone and collaborative working modes, adapt to different usage scenarios of single and dual wearable devices, and is equipped with complete switching caching logic, resulting in stronger interaction continuity.

[0043] 4. This invention features a factory-pre-bound automatic networking mechanism, which reduces the user's operational threshold and solves the problem of cumbersome pairing between multiple wearable devices;

[0044] 5. This invention can achieve simultaneous fusion interaction of voice and vision, resulting in more complete information dimensions, higher accuracy of reasoning structure, and superior interactive intelligence. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the hardware architecture of the distributed wearable multimodal AI interaction system in this invention;

[0047] Figure 2 This is a schematic diagram of the layered structure of the three-link dedicated communication protocol message in this invention;

[0048] Figure 3 This is a flowchart illustrating the execution of the multimodal fusion large-scale model distributed scheduling algorithm in this invention.

[0049] Figure 4 This is the overall flowchart of the overall control method of the present invention;

[0050] Figure 5 This is a timing diagram showing the switching logic between the individual working mode and the collaborative working mode in this invention. Detailed Implementation

[0051] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0052] Therefore, a feature pointed out in this specification is used to describe one feature of one embodiment of the invention, and does not imply that every embodiment of the invention must have the described feature. Furthermore, it should be noted that this specification describes many features. While certain features may be combined to illustrate possible system designs, these features may also be used in other combinations not explicitly stated. Therefore, unless otherwise stated, the described combinations are not intended to be limiting.

[0053] The principles and structure of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0054] Existing multimodal wearable smart devices have the following problems: 1. Simple hardware architecture, lacking a distributed dual-wear collaboration carrier; 2. Simple wireless communication link, relying on general Bluetooth, resulting in high latency and synchronization failure; 3. Lack of distributed hierarchical scheduling mechanism for multimodal large model inference; 4. Fixed working mode, unable to adapt to individual / collaborative scenarios; 5. Cumbersome device networking, high cost of pairing multiple wearable devices.

[0055] Please see the appendix Figure 1 To address the aforementioned issues, this invention proposes a distributed wearable multimodal AI interaction system, which includes an AI headset device, an AI glasses device, and a mobile terminal device.

[0056] The AI ​​headset device is used to collect user voice input in real time and output audio back. The AI ​​headset device has a built-in first wireless communication module, which is used to establish a first communication link with the mobile terminal device, serving as the main voice interaction entry point of the present invention.

[0057] The AI ​​glasses device is used to collect environmental visual images and video visual information, and to provide feedback visual interaction information. This visual interaction information can be either optical cues or vibration feedback. The AI ​​glasses device has a built-in second wireless communication module, which is used to establish a second communication link with the mobile terminal device, serving as the visual information collection entry point of this invention.

[0058] The mobile terminal device has a built-in multimodal fusion large model distributed scheduling algorithm module, also known as the scheduling algorithm module mentioned above. It is mainly used to receive two raw data sources, voice input and visual input, to complete multimodal feature fusion. It can dynamically allocate execution carriers according to the AI ​​task computing power requirements and task type, send lightweight tasks to AI headset devices and AI glasses devices for execution, and complete complex inference tasks of multimodal fusion by the local large model of the mobile terminal device.

[0059] A third wireless communication module is also set in the AI ​​headset and AI glasses device. It is used to establish a direct third communication link through a lightweight dedicated synchronous communication protocol to realize bidirectional information transmission between the AI ​​headset and AI glasses device. The AI ​​headset and AI glasses device can exchange messages of task status, execution progress and feedback status through the third communication link to realize real-time synchronization and updating of the status between wearable devices.

[0060] As can be seen from the above description, this invention simultaneously incorporates two wearable devices: an AI headset and AI glasses. It constructs a dual-wearable distributed hardware architecture, solving the problem of existing technologies having a single hardware architecture and lacking a distributed dual-wearable collaborative platform. Furthermore, the first and second communication links in this invention enable high-speed, low-latency direct communication between wearable devices (such as AI headsets and AI glasses) and mobile terminal devices, while the third communication link enables high-speed, low-latency direct communication between wearable devices. This solves the problems of existing technologies with single wireless communication links, reliance on general Bluetooth, and issues such as high latency and synchronization failure.

[0061] To adapt to different usage scenarios, the distributed wearable multimodal AI interaction system provided by this invention includes an adaptively switchable individual working mode and a collaborative working mode.

[0062] When the AI ​​headset and AI glasses work simultaneously, the third communication link is successfully established, and the distributed wearable multimodal AI interaction system works in collaborative mode.

[0063] When the AI ​​headset or AI glasses device is working alone, and the third communication link fails to be established, the distributed wearable multimodal AI interaction system operates in stand-alone mode.

[0064] In standalone working mode, if the connected wearable device is an AI headset, a first communication link is established; if the connected wearable device is an AI glasses device, a second communication link is established.

[0065] Specifically, this invention incorporates complete mode switching control logic:

[0066] The trigger condition for standalone working mode is: only the AI ​​headset device or only the AI ​​glasses device establishes the first or second communication link with the mobile terminal device, and the third communication link is disconnected at this time;

[0067] The corresponding control logic is as follows: Wearable devices independently collect single-modal data. If it is an AI headset device, it only collects user voice input through the AI ​​headset device. If it is an AI glasses device, it only collects environmental visual images and video visual information through the AI ​​glasses device. Interactive feedback is only sent to the currently online wearable device. If it is an AI headset device, it only supports pure voice interaction. If it is an AI glasses device, it only supports visual image recognition interaction.

[0068] The conditions for triggering the collaborative work mode are: both the AI ​​headset and the AI ​​glasses are online, and a direct connection channel is automatically established through the third communication link;

[0069] The corresponding control logic is as follows: the AI ​​headset and AI glasses device simultaneously collect two modal data, the mobile terminal device executes a multimodal fusion algorithm, the AI ​​headset and AI glasses device simultaneously receive interactive feedback, the AI ​​headset device outputs audio feedback, and the AI ​​glasses device outputs visual interactive information.

[0070] Automatic mode switching logic: Real-time detection of the connection status of the third communication link. If the third communication link is disconnected, switch to standalone working mode. If the third communication link is established, switch back to collaborative working mode. Unfinished tasks are cached during the switching process and task progress is restored after the switch.

[0071] Appendix Figure 5 The corresponding switching logic timing diagram shows that at time T0, both the AI ​​headset and the AI ​​glasses are online, the mobile terminal device executes the multimodal fusion algorithm, and the AI ​​headset and the AI ​​glasses synchronously receive interactive feedback and are in collaborative working mode.

[0072] At time T1, the AI ​​glasses device goes offline, the mobile terminal device executes the single-modal voice algorithm and caches the interaction task at the same time, and the second and third communication links are disconnected.

[0073] During the T2-T3 time period, the AI ​​headset device is online, collecting user voice input and receiving interactive feedback, while the AI ​​glasses device is offline, not receiving interactive feedback, and the mobile terminal device executes a single-modal voice algorithm, operating in a standalone mode (headset standalone mode).

[0074] During the T3-T4 time period, the mobile terminal device rebuilds the second communication link, synchronously caches tasks, and restores the connection. At T4, the AI ​​glasses device goes back online and completes the establishment of the second communication link and the third communication circuit. During the T4-T5 time period, the AI ​​headset device and the AI ​​glasses device synchronously receive interactive feedback and are in collaborative working mode.

[0075] Then the AI ​​headset device disconnects, the mobile terminal device executes the single-modal vision algorithm, and caches the interaction task. At this time, the first and third communication links are disconnected, the AI ​​glasses device is online, collects environmental visual images and video visual information, and receives interactive feedback, the AI ​​headset device is offline, does not receive interactive feedback, and is in stand-alone working mode (glasses stand-alone mode).

[0076] As can be seen from the above introduction, the present invention can automatically switch between standalone working mode and collaborative working mode, adapt to different usage scenarios of single wearable devices and dual wearable devices, and is equipped with complete switching caching logic, resulting in stronger interaction continuity. It solves the problem of fixed working modes in the prior art, which cannot adapt to standalone / collaborative scenarios.

[0077] Based on the above settings, the present invention can adapt to the situation of listening to music and asking questions during commuting, in which the user only wears the AI ​​headphone device and does not wear the AI ​​glasses device;

[0078] When the third communication link is disconnected, it enters a standalone working mode. The AI ​​earphone device collects the user's voice command "Check today's schedule" and only uploads the single-modal voice data to the mobile terminal device.

[0079] The algorithm identifies the task as a lightweight voice task, generates a text schedule, and outputs it via voice broadcast through an AI headset device. There is no visual interaction involved, making it suitable for scenarios such as running or commuting where it is inconvenient to wear glasses.

[0080] This invention can also adapt to outdoor image and text recognition scenarios, where the user only wears the AI ​​glasses device and does not wear the AI ​​earphone device, and the stand-alone working mode is activated;

[0081] The camera module of the AI ​​glasses takes pictures of the text on roadside products, and its microprocessor unit performs lightweight OCR text recognition (a lightweight visual task). The recognition result is indicated by short vibrations from the vibration unit, and simplified text prompts are displayed on the LED waveguide. No AI headset device or voice intervention is required, making it suitable for outdoor photo recognition and QR code scanning scenarios.

[0082] This invention can also adapt to real-time multimodal interactive scenarios. At this time, the user wears AI headphones and AI glasses at the same time, and all three wireless communication links are established, and the collaborative working mode is activated.

[0083] When a user inputs "Help me explain the history of this building," the AI ​​glasses simultaneously capture a real-world image of the building and upload it to the mobile device.

[0084] The mobile terminal device performs multimodal feature fusion, which is determined to be a complex fusion task. It then calls a distributed multimodal large model scheduling algorithm to combine voice questioning and real-world image joint reasoning.

[0085] The reasoning results are sent out simultaneously in two streams: the AI ​​headset plays a voice explanation of the building's history, and the AI ​​glasses simultaneously overlay text annotations and optical highlights on the screen to highlight key building structures.

[0086] Meanwhile, the third communication link synchronizes the task progress in real time, and the AI ​​headset and AI glasses devices provide synchronous feedback without delay or misalignment, making it suitable for scenarios such as real-life tourism explanation, real-life product consultation, and real-time teaching image recognition interaction.

[0087] This invention can also be adapted to scenarios that require switching modes, and can be used to demonstrate the collaborative working mode of wearing AI headphones and AI glasses at the same time when going out.

[0088] If you remove the AI ​​glasses and put them in your pocket, the third communication link will be disconnected, and the system will automatically switch to standalone working mode, with the interaction switching to pure voice Q&A.

[0089] After the AI ​​glasses are put back on, the third communication link automatically reconnects, switches back to collaborative working mode, and resumes synchronous voice and visual interaction. Unfinished question-and-answer tasks are cached and continue running without interruption.

[0090] Please see the appendix Figure 1 In this invention, the AI ​​glasses device also integrates a camera module, a feedback module, and a local microprocessor (i.e., a first local microprocessor).

[0091] The feedback module includes at least a micro vibration unit and an LED array optical prompting unit, which are used to provide feedback on visual interaction information. The visual interaction information can be provided through optical prompting by the LED array optical prompting unit or through vibration feedback by the micro vibration unit.

[0092] The camera module is used to capture visual images and complete the visual lightweight task locally through the first local microprocessor. Here, the lightweight task includes the visual lightweight task sent to the AI ​​glasses device and the voice lightweight task sent to the AI ​​headset device. The visual lightweight task can include: image text recognition, local object recognition, simple drawing and marking, etc., which can be completed locally on the AI ​​glasses device without occupying the mobile terminal device.

[0093] The AI ​​glasses device can upload the image preprocessing results to the mobile terminal device through the second communication link, or synchronize the image preprocessing results to the AI ​​headset device through the third communication link. Here, the second communication link is mainly used to complete the information interaction and feedback with the mobile terminal device, and the third communication link is used to complete the synchronous update of the task status, execution progress, and feedback status messages with the AI ​​headset device.

[0094] Please see the appendix Figure 1In this invention, the AI ​​earphone device also integrates a microphone module, an audio output module, and a local microprocessor (i.e., a second local microprocessor).

[0095] The microphone module is used to collect user voice input and complete lightweight voice tasks locally through a second local microprocessor. These lightweight voice tasks can include pure voice question and answer, voice broadcast, voice reminders, etc., which can be completed locally on the AI ​​headset device without occupying the mobile terminal device.

[0096] AI headset devices can upload the results of voice interaction to mobile terminal devices via a first communication link, or synchronize the results of voice interaction to AI glasses devices via a third communication link. Here, the first communication link is mainly used to complete information interaction and feedback with the mobile terminal device, and the third communication link is used to synchronize and update messages regarding the task status, execution progress, and feedback status with the AI ​​glasses device.

[0097] Please see the appendix Figure 1 In this invention, the mobile terminal device also integrates a computing power allocation unit and a state synchronization management unit. Here, the computing power allocation unit is used to coordinate with the multimodal fusion large model distributed scheduling algorithm module to dynamically allocate execution carriers according to the AI ​​task computing power requirements and task type after the above-mentioned multimodal fusion large model distributed scheduling algorithm module receives two raw data from voice input and visual input to complete multimodal feature fusion. The lightweight task is allocated to the AI ​​glasses device and the AI ​​earphone device, and the multimodal fusion task is used for local large model processing.

[0098] The status synchronization management unit is used to synchronize and update the task status, execution progress, and feedback status messages of the AI ​​glasses and AI earphone devices.

[0099] Based on the above-mentioned solution, this invention can classify tasks into multimodal fusion tasks and lightweight tasks, and dynamically allocate computing power. Lightweight tasks are sent to AI glasses and AI earphones for processing, while multimodal fusion tasks are processed locally. This can save computing power, reduce interaction response latency, and solve the problem in the prior art where lightweight voice tasks, image recognition tasks, and complex multimodal question-answering tasks all occupy the computing power of mobile terminals, resulting in wasted computing power resources and high interaction response latency.

[0100] Furthermore, in this invention, the AI ​​headset device is used as the main interaction triggering device of the distributed wearable multimodal AI interaction system. After the AI ​​headset device completes the collection of user voice input, it triggers the mobile terminal device to execute the multimodal fusion task and assigns lightweight tasks to the AI ​​headset device and the AI ​​glasses device.

[0101] As mentioned above, the mobile terminal device in this invention is used to execute the multimodal fusion algorithm (i.e., the distributed multimodal large model scheduling algorithm), and its specific execution steps are as follows:

[0102] The system acquires temporal features from user voice input collected by AI headset devices and image spatial features from environmental visual images and video visual information collected by AI glasses devices. It performs unified dimension normalization on temporal and image spatial features and completes voice-visual feature fusion through a cross-modal attention mechanism to generate a fused feature vector.

[0103] Based on the fusion feature vector, the current task type is identified and the task type is divided into speech lightweight task, vision lightweight task, and multimodal fusion task.

[0104] The lightweight voice tasks include: pure voice question and answer, voice broadcast, and voice reminder;

[0105] Lightweight visual tasks include: image text recognition, local object recognition, and simple image labeling;

[0106] Multimodal fusion tasks include: image-text joint question answering, real-world scene analysis, and multi-turn visual-voice interaction;

[0107] The voice lightweight task is assigned to the second local microprocessor in the AI ​​headset device for execution, and only the processing result of the second local microprocessor is sent back to the mobile terminal device.

[0108] The visual lightweight task is assigned to the first local microprocessor in the AI ​​glasses device for execution, and the image labeling and text recognition results are generated locally by the AI ​​glasses device.

[0109] Mobile terminal devices perform multimodal fusion tasks and generate complete interactive responses;

[0110] The task status, execution progress, and feedback status messages of the voice lightweight task, vision lightweight task, and multimodal fusion task are acquired and updated through the first communication link, the second communication link, and the third communication link, and synchronized to all wearable devices.

[0111] Appendix Figure 3 The specific steps for the aforementioned mobile terminal device to execute the multimodal fusion algorithm are as follows: it first collects the temporal features of the headphone voice and the visual spatial features of the glasses, that is, the temporal features of the user voice input collected by the AI ​​headphone device and the image spatial features of the environmental visual images and video visual information collected by the AI ​​glasses device mentioned above.

[0112] Then, multimodal feature uniform dimension normalization processing is performed, and fused feature vector is generated through cross-modal attention mechanism. That is, the temporal domain features and image spatial features are uniformly normalized as mentioned above, and speech-visual feature fusion is completed through cross-modal attention mechanism to generate fused feature vector.

[0113] At the same time, based on the fusion feature recognition, that is, based on the fusion feature vector recognition of the current task type mentioned above, the task type is divided into voice lightweight task, vision lightweight task, and multimodal fusion task.

[0114] Then, the task is classified and determined. The lightweight voice task (i.e., the voice lightweight task) is sent to the local microprocessor unit of the AI ​​headset for execution (i.e., the second local microprocessor). The lightweight vision task (i.e., the vision lightweight task) is sent to the local microprocessor unit of the AI ​​glasses (i.e., the first local microprocessor) for lightweight image recognition. The complex multimodal fusion task is completed by the multimodal large model (i.e., the multimodal fusion large model distributed scheduling algorithm module) running locally on the mobile terminal device to complete the complete fusion inference.

[0115] Then, the synchronization message is encapsulated and synchronized to the headphones and glasses through three wireless links. That is, the message obtained and updated through the first communication link, the second communication link, and the third communication link, the task status, execution progress, and feedback status of the voice lightweight task, the vision lightweight task, and the multimodal fusion task, and synchronized to all wearable devices.

[0116] At this point, the wearable device will output corresponding interactive feedback and determine whether to continue collecting user input. If yes, the above steps will be repeated; otherwise, the process will end.

[0117] Under the above-described scheme, this invention can classify tasks into multimodal fusion tasks and lightweight tasks, and dynamically allocate computing power. Lightweight tasks are distributed to AI glasses and AI earphones for processing, while multimodal fusion tasks are processed locally, saving computing power and reducing interaction response latency. In addition, under the above-described scheme, this invention can also achieve simultaneous fusion interaction of voice and vision, resulting in more complete information dimensions, higher accuracy of reasoning structure, and superior interactive intelligence. This solves the problem in existing technologies where lightweight voice tasks, image recognition tasks, and complex multimodal question-answering tasks all occupy mobile terminal computing power, leading to wasted computing resources and high interaction response latency.

[0118] Furthermore, the third wireless communication module in this invention adopts the lightweight state synchronization communication protocol customized in this invention. Unlike the standard Bluetooth RFCOMM general serial port protocol, the lightweight state synchronization communication protocol in this invention adopts a message layered structure, so that each message has a device ID frame, a state type frame, a data payload frame, and a check frame. The maximum length of a single message is 128 bytes, which is suitable for wearable devices with low power consumption and small cache hardware.

[0119] Meanwhile, this invention defines only three types of dedicated synchronization messages and does not include general Bluetooth redundant broadcast messages. The three types of messages are: task status message, execution progress message, and feedback control message.

[0120] The lightweight state synchronization communication protocol proposed in this invention has a low-latency retransmission mechanism, uses 10ms short time slot heartbeat packets, and only retransmits key state synchronization messages when a message is lost, without retransmitting audio and video data streams, thus reducing power consumption.

[0121] Its link priority is that the third communication link has a higher priority than the first and second communication links between the wearable device and the mobile terminal device. When the state changes, it can prioritize updating data through the third communication link between the AI ​​headset device and the AI ​​glasses device, reducing the relay latency of the mobile terminal device.

[0122] Appendix Figure 2 This is a schematic diagram of the message layering structure in this invention, which is divided into four layers. The first layer is the device ID frame, which consists of a device type identifier (occupying 1 byte), a binding unique ID (occupying 4 bytes), and a link priority flag (occupying 1 byte).

[0123] The second layer is the status type frame, which consists of a message type code (1 byte), a mode flag (1 byte), and a heartbeat slot identifier (1 byte).

[0124] The third layer is the data payload frame, which consists of a payload length (1 byte) and a synchronization service payload (0~128 bytes).

[0125] The fourth layer is the check frame, which consists of a CRC8 checksum (1 byte), a retransmission flag (1 bit), and a message end character (1 byte).

[0126] This invention designs a novel lightweight state synchronization communication protocol based on the above scheme, which can realize real-time synchronization of task status, execution progress, and feedback status across devices. It solves the problems of the general Bluetooth protocol, such as the lack of dedicated state synchronization message layering, redundant broadcast messages, high latency, and lack of hierarchical retransmission strategy, which cannot meet the real-time collaborative interaction needs of dual wearable devices.

[0127] Furthermore, the present invention also includes a pre-pairing mechanism, which is achieved by writing a unique identification code for each AI earphone device and AI glasses device during the manufacturing stage. After the user connects any wearable device to the mobile terminal device, the mobile terminal device can read the unique identification code, automatically broadcast the pairing signal, and actively establish a third communication link.

[0128] After the device restarts or disconnects, it can automatically reconnect based on the unique identification code, without requiring the user to manually pair it again.

[0129] Please see the appendix Figure 1 The present invention provides a pre-pairing storage unit in the mobile terminal device, which can be used to store the aforementioned unique identification code, so as to identify the corresponding identification code after any wearable device is connected to the mobile terminal device, thereby realizing pairing with another wearable device;

[0130] Under the above-described solution, the present invention, through the factory-pre-bound automatic networking mechanism, can reduce the user's operational threshold and solve the problem in the prior art where headphones and glasses require users to manually pair them with mobile phones and with each other, and the networking is disconnected after changing devices or restarting, and the multi-device networking relationship cannot be automatically restored, resulting in a high user operational threshold.

[0131] Please see the appendix Figure 4 The overall control process of this invention is as follows: device network initialization → modal data acquisition → mode determination → multimodal fusion + task hierarchical scheduling → task distributed execution → three-link status synchronization → interactive feedback output → cyclic acquisition.

[0132] The main steps are as follows:

[0133] 1. Network initialization process;

[0134] Specifically, it is divided into 1-1 device power-on, reading the factory pre-bound ID;

[0135] 1-2 Wearable devices attempt to connect to the mobile terminal and establish a first / second wireless link;

[0136] Terminals 1-3 read the binding ID, search for another wearable device, and attempt to establish a third direct communication link.

[0137] 1-4 Third link established successfully → enter collaborative working mode; Third link failed → enter individual working mode.

[0138] 2. Multimodal acquisition and mode determination process;

[0139] Specifically, it is divided into a 2-1 collaborative mode: the headphones continuously collect voice, the glasses continuously collect visual images, and the two data streams are simultaneously uploaded to the terminal.

[0140] 2-2 Standalone Headphone Mode: Only collects single-modal voice data;

[0141] 2-3 Single Glasses Mode: Only visual single-modal data is collected.

[0142] 3. Multimodal fusion and distributed scheduling algorithm flow, which is the process of executing the multimodal fusion algorithm mentioned above;

[0143] 4. Cross-device status synchronization process;

[0144] Specifically, it is divided into 4-1 terminal generating task execution status messages;

[0145] Message 4-2 is simultaneously sent to headphones and glasses;

[0146] 4-3 Prioritize synchronizing the status of the third direct link to complete the local device status update;

[0147] 5. Interactive feedback output process;

[0148] Specifically, it is divided into: 5-1 Voice-related results are sent to AI earphones for audio playback;

[0149] 5-2 Visual / reminder results are sent to the AI ​​glasses, triggering LED optical prompts or vibration units;

[0150] 6. Mode switching loop detection: Real-time monitoring of the connection status of the third communication link, triggering automatic mode switching, and caching unfinished tasks to ensure uninterrupted interaction.

[0151] Based on the above-described configuration, the present invention has at least the following beneficial effects:

[0152] 1. A distributed dual-wearable hardware + three-link dedicated communication architecture was constructed to solve the shortcomings of existing single-device, single-link communication.

[0153] This invention establishes an AI headset device and an AI glasses device as dual acquisition carriers, designs three independent wireless communication links (first communication link, second communication link, and third communication link), and customizes a lightweight synchronous communication protocol to replace general Bluetooth. Wearable devices can directly connect and synchronize, significantly reducing cross-device state synchronization latency and solving the problems of single-device interaction, general Bluetooth synchronization latency, and state conflict in existing technologies.

[0154] 2. A distributed multimodal large model scheduling algorithm was proposed, which saved computing power;

[0155] This invention uses cross-modal attention fusion features to classify tasks and dynamically allocate computing power. Lightweight tasks are executed on wearable devices, while multimodal fusion tasks are processed through mobile terminal devices. This hierarchical scheduling saves computing power and significantly reduces interaction response latency.

[0156] 3. A dual-mode adaptive switching logic was designed to cover the usage needs of all scenarios;

[0157] This invention can automatically switch between standalone and collaborative working modes, adapting to different usage scenarios of single and dual wearable devices. With complete switching and caching logic, the interaction continuity is stronger, and the range of applicable scenarios is far greater than that of existing technologies.

[0158] 4. A factory-pre-bound automatic networking mechanism was designed to reduce the user's operating threshold and solve the problem of cumbersome pairing of multiple wearable devices;

[0159] This invention achieves its purpose by writing a unique identification code for each AI headset and AI glasses device during the manufacturing stage. After the user connects any wearable device to the mobile terminal device, the mobile terminal device can read the unique identification code, automatically broadcast a pairing signal, and actively establish a third communication link, thus reducing the user's operating threshold.

[0160] 5. A dual-modal synchronous fusion interaction of voice and vision was proposed, which provides more complete information dimensions and significantly outperforms single-modal smart glasses / headphone solutions in terms of the accuracy of large model inference results and the intelligence of interaction.

[0161] This invention features a dual-wearable device consisting of an AI headset and AI glasses, enabling bimodal interaction through voice and vision. Furthermore, via a third communication link, it allows for synchronous information updates. The mobile terminal device analyzes and reasons using the acquired voice and visual information, resulting in more accurate reasoning outcomes. This level of interactive intelligence surpasses that of single-modal smart glasses or smart headset designs.

[0162] It should be noted that the terminology used above is for describing specific embodiments only and is not intended to limit the exemplary embodiments of the present invention. When the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The order of execution of actions, steps, etc., in the apparatus and methods in the specification and drawings can be implemented in any order unless a specific express order is specified, and as long as the output of a previous process is not used in a subsequent process. Similar sequential terms used for ease of description do not imply that such an order must be followed.

[0163] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as constraints. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed wearable multimodal AI interaction system, characterized in that, include: The AI ​​headset device is used to collect user voice input and output audio. The AI ​​headset device has a built-in first wireless communication module for establishing a first communication link with a mobile terminal device. AI glasses device, which is used to collect environmental visual images and video visual information and to provide feedback visual interaction information, has a second wireless communication module built into the AI ​​glasses device to establish a second communication link with the mobile terminal device. And mobile terminal devices, which have a built-in scheduling algorithm module, can obtain user voice input collected by the AI ​​headset device and environmental visual images and video visual information collected by the AI ​​glasses device through the first communication link and the second communication link, and perform multimodal fusion tasks and allocate lightweight tasks to the AI ​​headset device and the AI ​​glasses device; A third wireless communication module is also provided in the AI ​​headset device and the AI ​​glasses device, which is used to establish a third communication link for bidirectional transmission between the AI ​​headset device and the AI ​​glasses device. The AI ​​headset device and the AI ​​glasses device can exchange messages about the task status, execution progress and feedback status of the lightweight task through the third communication link.

2. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The distributed wearable multimodal AI interaction system includes an adaptively switchable individual working mode and a collaborative working mode; When the AI ​​headset and the AI ​​glasses work simultaneously, the third communication link is successfully established, and the distributed wearable multimodal AI interaction system works in collaborative mode. When the AI ​​headset or the AI ​​glasses are working independently, and the establishment of the third communication link fails, the distributed wearable multimodal AI interaction system operates in independent working mode. In the standalone working mode, if the connected wearable device is an AI headset device, a first communication link is established; if the connected wearable device is an AI glasses device, a second communication link is established.

3. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The AI ​​glasses device also integrates a camera module, a feedback module, and a first local microprocessor; The feedback module includes at least a micro vibration unit and an LED array optical prompting unit; The camera module is used to capture visual images and complete the visual lightweighting task locally through the first local microprocessor. The AI ​​glasses device can upload the image preprocessing results to the mobile terminal device via the second communication link, or synchronize the image preprocessing results to the AI ​​earphone device via the third communication link.

4. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The AI ​​headset device also integrates a microphone module, an audio output module, and a second local microprocessor; The microphone module is used to collect user voice input and complete the voice lightweighting task locally through the second local microprocessor; The AI ​​headset device can upload the results of voice interaction to the mobile terminal device through the first communication link, or synchronize the results of voice interaction to the AI ​​glasses device through the third communication link.

5. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The AI ​​headset device shown is the main interaction triggering device of the distributed wearable multimodal AI interaction system. After the AI ​​headset device completes the collection of user voice input, it triggers the mobile terminal device to perform multimodal fusion tasks and assign lightweight tasks to the AI ​​headset device and the AI ​​glasses device.

6. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The algorithm execution steps of the mobile terminal device include: The temporal features in the user's voice input collected by the AI ​​headset device and the image spatial features in the environmental visual images and video visual information collected by the AI ​​glasses device are obtained. The temporal features and the image spatial features are then normalized to generate a fused feature vector. Based on the fusion feature vector, the current task type is identified and the task type is divided into speech lightweight task, vision lightweight task, and multimodal fusion task. The voice lightweighting task is sent to the AI ​​headset device, the vision lightweighting task is sent to the AI ​​glasses device, and the multimodal fusion task is executed through the mobile terminal device; The task status, execution progress, and feedback status of the voice lightweight task, vision lightweight task, and multimodal fusion task are acquired and updated through the first communication link, the second communication link, and the third communication link.

7. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The third wireless communication module adopts a lightweight state synchronization communication protocol. The lightweight state synchronization communication protocol adopts a message layered structure and includes task status messages, execution progress messages, and feedback control messages. Each message has a device ID frame, a status type frame, a data payload frame, and a check frame. The maximum length of a single message is 128 bytes.

8. The distributed wearable multimodal AI interaction system according to claim 7, characterized in that, The lightweight state synchronization communication protocol uses 10ms short-slot heartbeat packets, and only retransmits state synchronization messages when a message is lost. When the AI ​​headset, the AI ​​glasses, and the mobile terminal communicate, the third communication link has a higher priority than the first and second communication links.

9. The distributed wearable multimodal AI interaction system according to claim 1, characterized in that, The AI ​​earphone device and the AI ​​glasses device are bound to a unique identification code; When either the AI ​​headset device or the AI ​​glasses device is connected to the mobile terminal device, the mobile terminal device can read the unique identification code to establish the third communication link.

Images