Method and device for interaction between multifunctional AR glasses and mobile phone

Abstract

The invention discloses a multifunctional AR glasses and mobile phone interaction method and device, and relates to the field of equipment interaction. In the method, a plurality of communication channels are established between an AR glasses end and a mobile phone end through a bidirectional protocol, a protocol encapsulation layer is overlaid on the plurality of communication channels, and a mapping relation between a data type and the plurality of communication channels is established; mapping the user operation into a cross-device instruction stream, and selecting an optimal channel according to the data type and the mapping relationship of the cross-device instruction stream; determining a compression and encryption strategy according to the data type and the real-time network condition, processing the cross-device instruction stream by using the compression and encryption strategy, and transmitting the processed cross-device instruction stream through the optimal channel; and obtaining interaction data and an equipment state between the AR glasses end and the mobile phone end, and when the interaction data or the equipment state is abnormal, switching the optimal channel to the target channel. By implementing the technical scheme provided by the invention, the interaction efficiency of the AR glasses and the mobile phone is improved, and the data transmission reliability is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of device interaction, and in particular to a method and apparatus for interaction between multifunctional AR glasses and a mobile phone. Background Art

[0002] The interactive technology between AR (Augmented Reality) glasses and mobile phones has developed rapidly in recent years. This interaction method not only greatly enriches the user experience but also provides new possibilities for the collaborative operation of smart devices. By combining AR glasses with mobile phones, users can achieve more natural and convenient human-computer interaction, such as controlling mobile applications through gestures, voice, or touch, thus obtaining richer information display and interactive experience in the augmented reality environment.

[0003] In existing technologies, efficient interaction between AR glasses and mobile phones typically involves establishing a single communication channel for data transmission, using fixed data processing strategies, and relying on pre-set network conditions. Specifically, conventional approaches include: first, establishing a single communication link through wireless communication technologies such as Bluetooth or Wi-Fi to transmit all types of interactive data; second, applying uniform compression and encryption methods to different types of data, regardless of the actual network environment and device status; and third, attempting to restore the connection by simply disconnecting and reconnecting or restarting the application when an anomaly occurs during communication.

[0004] However, these conventional methods have obvious flaws, especially when faced with complex and changing usage scenarios. Due to the failure to select appropriate communication channels based on data types and the failure to adjust compression and encryption strategies based on real-time network conditions, data transmission is prone to delays, packet loss, and even complete interruption. Furthermore, when abnormalities occur in interactive data or device status, the lack of an effective channel switching mechanism significantly degrades the user experience. These issues indicate that existing AR glasses and mobile phone interaction technologies urgently need a solution that can dynamically adapt to different data types and network conditions. Summary of the Invention

[0005] The present application provides a method and device for interaction between multifunctional AR glasses and mobile phones, which improves the interaction efficiency between AR glasses and mobile phones, enhances user experience and ensures data transmission reliability.

[0006] In a first aspect of the present application, a method for interacting multifunctional AR glasses with a mobile phone is provided, the method comprising: Establishing multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, overlaying a protocol encapsulation layer on the multiple communication channels, and establishing a mapping relationship between data types and the multiple communication channels; Mapping a user operation into a cross-device instruction stream, and selecting an optimal channel according to a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; determining a compression and encryption strategy based on the data type and real-time network conditions, processing the cross-device instruction stream using the compression and encryption strategy, and transmitting the processed cross-device instruction stream through the optimal channel; The interaction data and device status between the AR glasses and the mobile phone are obtained. When the interaction data or the device status is abnormal, the optimal channel is switched to the target channel, where the target channel is the communication channel with the best performance except the optimal channel in the current state.

[0007] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship includes: Providing a virtual touchpad, a virtual key interface, and a soft keyboard on the mobile phone terminal, and in response to the mobile phone terminal receiving a single-finger or multi-finger sliding operation or an operation of a direction key, a confirmation key, or a return key by the user, mapping the single-finger or multi-finger sliding operation and the direction key, the confirmation key, and the return key operation into a key value instruction of a human body interaction device, and converting text characters input by the user into a key value sequence of the human body interaction device in real time; An optimal channel is determined to be a human body interaction device protocol channel according to the human body interaction device key value instruction, the human body interaction device key value sequence and the mapping relationship.

[0008] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to receiving a navigation request input by a user on the mobile phone, obtaining a starting point, an end point, and an operation mode from the navigation request, calling a third-party map software development toolkit to generate a navigation route on the mobile phone according to the starting point, the end point, and the operation mode, and slicing the position coordinates, turn prompts, and remaining distance in the navigation route to obtain navigation data; The optimal channel is determined to be a low-power Bluetooth channel according to the navigation data and the mapping relationship.

[0009] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to the mobile phone receiving a prompt request input by a user, obtaining a prompt file and prompt parameters from the prompt request, wherein the prompt file includes document content, and the prompt parameters include font size, color, scrolling speed, and speech progress of the teleprompter; According to the prompt file and the mapping relationship, the optimal channel is determined to be the Bluetooth socket channel, and according to the prompt parameters and the mapping relationship, the optimal channel is determined to be the low-power Bluetooth channel, so as to transmit the prompt file and the prompt parameters respectively.

[0010] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: Listening to notification events through the notification listener on the mobile phone, and obtaining notification information from the notification event, the notification information including the notification title, notification content and notification source; Monitor incoming call events through the phone manager on the mobile phone, and obtain incoming call information from the incoming call events, wherein the incoming call information includes the caller number and contact information; The optimal channel is determined to be a low-power Bluetooth channel according to the notification information and / or the incoming call information.

[0011] Optionally, the method further includes: When the incoming call information is sent to the AR glasses, it is displayed on the display interface of the AR glasses, and the user is prompted to answer or reject the call; In response to the user's answering operation, the call audio on the mobile phone is switched to the AR glasses using the call protocol channel.

[0012] Optionally, when the interaction data or the device status is abnormal, switching the optimal channel to the target channel includes: If the low-power Bluetooth channel is abnormal, the navigation data and notification information will be synchronously migrated to the Bluetooth socket channel, and the transmission priority of the human interaction device protocol will be increased; If the Bluetooth socket channel is abnormal, switch the prompt file transmission mode to the low-power Bluetooth channel fragment transmission mode and set the single fragment size threshold; If the call protocol channel is abnormal, the call audio will be replayed to the mobile phone speaker, and an audio replay alarm will be sent to the AR glasses via the low-power Bluetooth channel.

[0013] In a second aspect of the present application, a system for interacting with multifunctional AR glasses and a mobile phone is provided, comprising a communication module, a selection module, a processing module, and a switching module, wherein: A communication module configured to establish multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, superimpose a protocol encapsulation layer on the multiple communication channels, and establish a mapping relationship between data types and the multiple communication channels; a selection module configured to map a user operation into a cross-device instruction stream and select an optimal channel according to a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; a processing module configured to determine a compression and encryption strategy based on the data type and real-time network conditions, process the cross-device instruction stream using the compression and encryption strategy, and transmit the processed cross-device instruction stream through the optimal channel; The switching module is configured to obtain the interaction data and device status between the AR glasses and the mobile phone. When the interaction data or the device status is abnormal, the optimal channel is switched to the target channel. The target channel is the communication channel with the best performance except for the optimal channel in the current state.

[0014] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0015] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions. When the instructions are executed, any one of the methods described above is executed.

[0016] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Establish multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol and superimpose a protocol encapsulation layer to ensure communication stability and reliability. Establish a mapping relationship between data types and multiple communication channels, so that the system can select the most appropriate communication channel based on different data types, thereby optimizing communication efficiency. 2. Map user operations (such as touch, gestures, and voice) into cross-device command streams and select the optimal channel for transmission based on the data type and mapping relationship of the command stream. This ensures the real-time and accuracy of user operations. Compression and encryption strategies are determined based on the data type and real-time network conditions, and cross-device command streams are processed, which not only reduces data transmission volume but also improves data security. 3. Acquire the interaction data and device status between the AR glasses and the mobile phone. When the interaction data or device status is abnormal, the optimal channel can be quickly switched to the target channel. The target channel is the communication channel with the best performance other than the optimal channel in the current state. This ensures that the system can maintain stable communication performance even if the optimal channel has problems. 4. By optimizing communication channel management and cross-device instruction stream processing, data transmission delays and losses are reduced, thereby improving the smoothness and stability of user interaction with AR glasses and mobile phones; the dynamic channel switching mechanism ensures that users can continue to enjoy a seamless interactive experience even in abnormal situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a method for interacting with a mobile phone using multifunctional AR glasses disclosed in an embodiment of the present application; Figure 2 This is a system architecture diagram of the multifunctional AR glasses and mobile phone disclosed in the embodiment of this application; Figure 3 This is a flowchart of the mouse and keyboard simulation of the human body interaction device disclosed in the embodiment of the present application; Figure 4 This is a flowchart of the navigation function data transmission disclosed in the embodiment of this application; Figure 5 This is a flowchart of editing and sending the teleprompter disclosed in the embodiment of the present application; Figure 6 This is a flowchart of mobile phone notification synchronization disclosed in an embodiment of the present application; Figure 7 This is a flowchart of the power acquisition process on the glasses side disclosed in the embodiment of this application; Figure 8 This is a flowchart of incoming call synchronization and answering on the glasses side disclosed in the embodiment of this application; Figure 9 This is a module diagram of a system for interacting with a multifunctional AR glasses and a mobile phone disclosed in an embodiment of the present application; Figure 10 This is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

[0018] Explanation of the reference numerals: 901, communication module; 902, selection module; 903, processing module; 904, switching module; 1001, processor; 1002, communication bus; 1003, user interface; 1004, network interface; 1005, memory. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0020] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0021] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0022] This embodiment discloses a method for interacting with a multifunctional AR glasses and a mobile phone. Figure 1 This is a flow chart of a method for interacting with a mobile phone using multifunctional AR glasses disclosed in an embodiment of the present application. Figure 1 As shown, the method includes the following steps: S101. Establish multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, overlay a protocol encapsulation layer on the multiple communication channels, and establish a mapping relationship between data types and the multiple communication channels; S102: Mapping a user operation into a cross-device instruction stream, and selecting an optimal channel based on a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; S103: Determine a compression and encryption strategy based on the data type and real-time network conditions, process the cross-device instruction stream using the compression and encryption strategy, and transmit the processed cross-device instruction stream through the optimal channel; S104: Acquire interaction data and device status between the AR glasses and the mobile phone. When an abnormality occurs in the interaction data or the device status, switch the optimal channel to a target channel, where the target channel is the communication channel with the best performance other than the optimal channel in the current state.

[0023] Use bidirectional protocols (such as TCP / IP or custom protocols) to establish multiple communication channels between the AR glasses and the phone. These channels can be physical connections (such as Bluetooth or Wi-Fi) or logical connections (such as virtual channels through cloud services). Bidirectional protocols ensure bidirectional data transmission, meaning both the AR glasses and the phone can send and receive data. A protocol encapsulation layer is overlaid on each communication channel to encapsulate and decapsulate the transmitted data. The encapsulation layer can include information such as the data header, data body, and checksum to ensure data integrity and accuracy. The protocol encapsulation layer also handles data segmentation and reassembly to adapt to the data transmission capabilities of different channels. Data types are mapped to multiple communication channels based on their respective characteristics (such as text, navigation, and audio) and the communication channel characteristics (such as bandwidth, latency, and stability). For example, text data may be better suited for transmission over a low-latency, high-reliability channel, while audio data may be better suited for transmission over a high-bandwidth channel. User actions (such as touch, gestures, and voice) on the AR glasses or phone are captured and mapped into a cross-device command stream. The command stream contains information such as the operation type, operation parameters, and data type. Based on the data type of the cross-device command stream and the previously established mapping, the optimal communication channel is selected for transmission. The optimal channel can be selected based on factors such as channel bandwidth, latency, stability, and current network conditions. Based on the data type (e.g., text data, audio data) and real-time network conditions (e.g., bandwidth and latency), appropriate compression and encryption strategies are determined. Compression strategies reduce data transmission volume and improve transmission efficiency, while encryption strategies ensure data security. The cross-device command stream is processed using the determined compression and encryption strategies. Compression removes redundant information and reduces data size, while encryption converts data into ciphertext to prevent data leakage. The processed cross-device command stream is transmitted over the optimal channel. During transmission, the transmission status can be monitored to ensure accurate data arrival. Real-time interaction data (e.g., data volume, transmission latency) and device status (e.g., battery level, network connection status) between the AR glasses and the phone are acquired. The interaction data and device status are monitored for anomalies. Anomalies may include data interruption, excessive transmission latency, or low device battery. When an anomaly occurs, the current optimal channel is switched to the target channel. The target channel is the communication channel with the best performance other than the optimal channel. The switching process can ensure the continuity and stability of data transmission and avoid the collapse of the entire interactive system due to the failure of a single channel.

[0024] Figure 2 This is a system architecture diagram of the multifunctional AR glasses and mobile phone disclosed in the embodiment of this application, such as Figure 2 As shown, the mobile phone in the embodiment of the present application is an Android phone, which includes the core functional modules of the mobile phone app, which include a HID analog input module, a navigation module, a teleprompter module, a notification synchronization module, a power acquisition module, and an incoming call synchronization and answering module. The AR glasses include an HID receiving / parsing unit, a BLE transceiver unit, a navigation display unit, a teleprompter display unit, a notification display unit, a power information service, and an audio and incoming call control module. The Android phone and AR glasses can directly communicate through different protocols such as HID, BLE, Bluetooth Socket, and HFP / HSP.

[0025] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship includes: Providing a virtual touchpad, a virtual key interface, and a soft keyboard on the mobile phone terminal, and in response to the mobile phone terminal receiving a single-finger or multi-finger sliding operation or an operation of a direction key, a confirmation key, or a return key by the user, mapping the single-finger or multi-finger sliding operation and the direction key, the confirmation key, and the return key operation into a key value instruction of a human body interaction device, and converting text characters input by the user into a key value sequence of the human body interaction device in real time; An optimal channel is determined to be a human body interaction device protocol channel according to the human body interaction device key value instruction, the human body interaction device key value sequence and the mapping relationship.

[0026] The mobile app interface provides a virtual touchpad (i.e., touch area) that simulates the operation of a physical touchpad. Users can use this area to perform single- or multi-finger swipes, taps, and double-clicks. The app interface also provides virtual buttons such as up, down, left, right, and right, as well as OK and Back. Users can click these buttons to initiate corresponding commands. For scenarios requiring text input, the app displays a soft keyboard to accept alphanumeric, and symbol input. When users perform operations on the virtual touchpad or virtual key interface, the app monitors these operations and maps them to Human Interface Device (HID) key commands. For example, a single-finger swipe might be mapped to a mouse move command, a tap might be mapped to a mouse click command, and an arrow key operation might be mapped to a keyboard arrow key command. For text input on the soft keyboard, the app converts each character entered into a HID key sequence, representing the text content entered by the user. The app converts each character entered on the soft keyboard into a corresponding HID key value in real time and combines these key values into a key sequence in the order in which they were entered. This allows the glasses to accurately reproduce the text content entered by the user upon receiving these key sequences. The app determines the optimal communication channel based on mapped HID key commands and key sequences, as well as pre-established mappings between data types and communication channels. Because these commands and key sequences are based on the Human Interface Device (HID) protocol, the optimal channel is typically the Human Interface Device protocol channel. This channel may offer low latency and high stability, making it suitable for transmitting HID commands and key sequences.

[0027] Figure 3 This is a flowchart of the mouse and keyboard simulation of the human body interaction device disclosed in the embodiment of this application, such as Figure 3 As shown, the touch / button operation is mapped to the HID key value (mouse / keyboard) through the HID analog input module on the mobile phone, and sent to the AR glasses through the HID Bluetooth protocol. The AR glasses execute the corresponding mouse movement, click or keyboard input through the HID receiving / parsing unit to drive the focus movement / text entry of the application interface on the AR glasses.

[0028] By mapping user actions on the mobile device into cross-device command streams, seamless transmission and interaction of user actions between different devices (such as mobile phones and AR glasses) is achieved. This improves the consistency and convenience of the user experience. Multiple input methods, including a virtual touchpad, virtual key interface, and soft keyboard, are provided to meet user input needs in different scenarios. For example, the virtual touchpad is suitable for delicate operations, the virtual key interface provides quick access to common functions, and the soft keyboard facilitates text input. User-entered text characters are converted into human interface device key-value sequences in real time, ensuring accurate and real-time text input. This is particularly important for applications requiring frequent text input, such as chat and search. Based on the data type of the human interface device key-value commands and key-value sequences, as well as pre-established mappings, the optimal communication channel (such as the human interface device protocol channel) is selected for transmission. This helps reduce data transmission latency and improve data transmission efficiency, thereby enhancing overall system performance. Using the Human Interface Device (HID) protocol as the basis for command transmission makes this technical solution highly compatible and scalable. This means it can be easily integrated with other devices or systems that support the HID protocol, facilitating future functional expansion and upgrades. By providing an intuitive and easy-to-use virtual input interface and a real-time responsive interactive experience, this technology enhances the user-device interaction experience. Users can interact with the device more naturally and smoothly, thereby improving user satisfaction and loyalty.

[0029] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to receiving a navigation request input by a user on the mobile phone, obtaining a starting point, an end point, and an operation mode from the navigation request, calling a third-party map software development toolkit to generate a navigation route on the mobile phone according to the starting point, the end point, and the operation mode, and slicing the position coordinates, turn prompts, and remaining distance in the navigation route to obtain navigation data; According to the navigation data and the mapping relationship, it is determined that the optimal channel is a low-power Bluetooth channel.

[0030] The user enters their starting point (which automatically uses the phone's GPS location) and destination on the app's navigation page and selects walking or cycling mode. This action triggers a navigation request containing the user's desired route from the starting point to the destination. The app extracts the starting point, destination, and the user's selected mode of travel (walking or cycling) from the navigation request. This information forms the basis for generating a navigation route. The app then calls a third-party map SDK (Software Development Kit) interface to generate a navigation route based on the starting point, destination, and mode of travel. The map SDK returns multiple possible routes for the user to choose from. The navigation route is segmented to generate navigation data: the generated navigation route contains key information such as location coordinates, turn instructions, and remaining distance. To facilitate transmission and processing, the app segments this information into navigation data. Segmentation ensures flexible and reliable data transmission. Navigation data is a specific type of data (such as location information and real-time notifications) that requires high real-time and stable transmission. Based on pre-established mappings (data type to communication channel), the system determines the Bluetooth Low Energy (BLE) channel as the optimal channel. BLE channels offer low power consumption, short range, and excellent real-time performance, making them ideal for transmitting navigation data. The app, acting as the BLE server, sends segmented navigation data to the glasses via the BLE channel. This segmented transmission ensures data stability and reliability during transmission, preventing data loss or corruption. Upon receiving the navigation data, the glasses decrypt and decompress it (if the data was encrypted and compressed during transmission). The glasses then display navigation information such as the remaining distance and next turn, providing real-time navigation guidance.

[0031] Figure 4 This is a flowchart of the navigation function data transmission disclosed in the embodiment of this application, such as Figure 4 As shown, the mobile phone selects the starting point and the end point through the navigation module, and can obtain multiple routes. The user selects one and starts navigation, continuously obtains navigation information (including current location, defense line, distance, etc.), compresses and encrypts the navigation information, and sends the navigation data to the glasses via BLE. The navigation display unit on the glasses receives and decrypts the navigation data, and displays the path and prompts in real time on the glasses interface.

[0032] By mapping navigation requests entered by users on their mobile phones into cross-device command streams and selecting appropriate communication channels (such as Bluetooth Low Energy channels) for transmission, this technology enables seamless transmission of navigation information between different devices (such as mobile phones and AR glasses). This provides users with a coherent and consistent navigation experience. Key information such as location coordinates, turn instructions, and remaining distance within the navigation route is processed in fragments and transmitted to the glasses in real time via the optimal channel. This ensures that users always have access to the latest navigation information, enabling them to make accurate navigation decisions. Selecting the Bluetooth Low Energy channel as the optimal channel ensures real-time and stable data transmission while also reducing power consumption. This is particularly important for users who require extended navigation use, as it extends device battery life. Segmented navigation data processing makes data transmission more flexible and reliable. Even if a data segment is lost or corrupted, it does not affect the transmission and display of the entire navigation information. Segmented transmission also improves data transmission efficiency and reduces transmission latency. By presenting navigation information in real time and accurately, this technical solution enhances the user's navigation experience. Users can more clearly understand information such as their current location, next turn, and remaining distance, making navigation tasks easier to complete. Navigation routes are generated using a third-party map software development kit, which makes it highly compatible and extensible. Different map services and data sources can be easily integrated, facilitating future functional expansion and upgrades.

[0033] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to the mobile phone receiving a prompt request input by a user, obtaining a prompt file and prompt parameters from the prompt request, wherein the prompt file includes document content, and the prompt parameters include font size, color, scrolling speed, and speech progress of the teleprompter; According to the prompt file and the mapping relationship, the optimal channel is determined to be the Bluetooth socket channel, and according to the prompt parameters and the mapping relationship, the optimal channel is determined to be the low-power Bluetooth channel, so as to transmit the prompt file and the prompt parameters respectively.

[0034] After the user completes importing and editing the file in the teleprompter interface, they click the "Send to Glasses" button to trigger a prompt request. This action contains the user's instructions to send the prompt file and prompt parameters to the glasses. The prompt file contains the document content—the text information the user wants to prompt. The prompt parameters include settings such as the teleprompter's font size, color, scrolling speed, and speech progress. Together, these details constitute the prompt request. Prompt files typically contain large amounts of data (document content) and require a stable and reliable data transmission channel. Based on a pre-established mapping between data types and communication channels, the system determines the Bluetooth socket channel as the optimal channel for transmitting prompt files. The Bluetooth socket channel provides a classic Bluetooth connection, suitable for transferring large files and ensuring data integrity and stability. Before sending the prompt file, the app can optionally compress and encrypt the file to reduce data transmission volume and improve security. The processed file is written to the Bluetooth socket channel via the OutputStream and sent to the glasses. The glasses receive the prompt file via a BluetoothServerSocket or BluetoothSocket. The received file is decrypted (if encrypted during transmission) and loaded into the local teleprompter display component for user viewing and use. Prompt parameters typically contain a small amount of data (settings information), but they need to be transmitted quickly and in real time to the glasses to adjust the display. Based on a pre-established mapping relationship, the system determines the Bluetooth Low Energy (BLE) channel as the optimal channel for transmitting prompt parameters. The BLE channel has the characteristics of low power consumption and good real-time performance, making it suitable for transmitting small amounts of setting information. The app sends the prompt parameters to the glasses via BLE characteristic values. The glasses receive and parse these parameters in real time to adjust the teleprompter display (such as font size, color, scrolling speed, etc.).

[0035] Figure 5 This is a flowchart of editing and sending the teleprompter disclosed in the embodiment of this application, such as Figure 5 As shown, the mobile phone imports txt files / creates new text content through the teleprompter module and stores the text content in the local database. The user can edit / modify the prompt content and click "Send" to package and encrypt the text content, send the file through the Bluetooth socket, and send the setting content through BLE. The AR glasses decrypt and load the text and setting content through the teleprompter display unit and display it on the glasses interface.

[0036] By transmitting the prompt file and prompt parameters separately, the system can select the optimal transmission channel for different data types. Prompt files are typically large, making them well-suited for stable and reliable data transmission over Bluetooth sockets. Prompt parameters, on the other hand, are smaller and better suited for fast, real-time transmission over Bluetooth LE. Users can seamlessly send both the prompt file and the prompt parameters to the glasses without worrying about data transmission delays or instability. Stable transmission of the prompt file ensures the integrity of the prompt content, while real-time transmission of the prompt parameters allows users to adjust the teleprompter display, such as font size, color, and scrolling speed, on the fly, providing a more personalized prompting experience. Transmitting prompt parameters over Bluetooth LE significantly reduces device power consumption. This is particularly important for users who use the teleprompter for extended periods, extending device battery life and improving user efficiency. This technical solution allows for dynamic selection of the optimal channel based on data type, enhancing the system's flexibility and scalability. In the future, if new data types or transmission requirements are introduced, the system can easily adjust the mapping to select the most appropriate transmission channel. Optional compression and encryption can be used during prompt file transmission to help protect the security and privacy of user data. Even if data is intercepted during transmission, it is difficult for unauthorized personnel to interpret it. By transmitting different types of data separately, the system architecture becomes simpler and clearer. This helps reduce system complexity and improve development and maintenance efficiency.

[0037] Optionally, mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: Listening to notification events through the notification listener on the mobile phone, and obtaining notification information from the notification event, the notification information including the notification title, notification content and notification source; Monitor incoming call events through the phone manager on the mobile phone, and obtain incoming call information from the incoming call events, wherein the incoming call information includes the caller number and contact information; The optimal channel is determined to be a low-power Bluetooth channel according to the notification information and / or the incoming call information.

[0038] The mobile app has a built-in notification listener that captures notification events sent by the system or other apps in real time. When a new notification arrives, the listener is triggered and retrieves relevant information about the notification. Notification information typically includes key data such as the title, content, and source. This information can be important to users, such as new message alerts and schedule updates. Given that notifications are typically small in size and need to be transmitted to the user's other devices (such as smart glasses) in real time, the Bluetooth Low Energy (BLE) channel is chosen as the optimal transmission channel. BLE channels offer low power consumption and excellent real-time performance, making them suitable for transmitting small notifications. The mobile app also integrates a call manager function to monitor incoming call events. When a call arrives, the call manager captures relevant information about the incoming call. Caller information primarily includes the caller number and contact information. This information is equally important to users, especially when they cannot directly access their phone (such as when using smart glasses). Similar to notifications, caller information is also small in size and needs to be transmitted to the user's other devices in real time. Therefore, the low-power Bluetooth channel is also selected as the optimal transmission channel to ensure the real-time and low-power transmission of incoming call information.

[0039] Through the notification listener and phone manager on the phone, the system can monitor and obtain relevant information about notification events and incoming call events in real time. This information includes the notification title, content, source, caller ID, contact information, etc. By mapping this information into a cross-device command stream and selecting a low-power Bluetooth channel for transmission, real-time notification and call synchronization between the phone and the glasses is achieved. Users can get important notifications and call information through the glasses without having to check their phones. Low power consumption and efficient transmission: Selecting the Bluetooth Low Energy (BLE) channel as the optimal channel ensures low power consumption during the transmission of notification and call information. This is particularly important for devices that require long-term connectivity, such as glasses, as it significantly extends their battery life. Furthermore, the BLE channel provides relatively stable transmission performance, ensuring accurate and timely information delivery. Users can access important notifications and call information without frequently checking their phones, significantly improving convenience and efficiency. Especially when driving, exercising, or working, users can quickly access notifications and call status through their glasses, allowing them to respond promptly. By monitoring and transmitting notification and call information in real time, the system ensures that users receive this information immediately. This is crucial for important notifications and calls that require prompt action, preventing users from missing important messages or calls. By monitoring notification and call events on the phone and mapping the relevant information into a cross-device command stream for transmission, the system simplifies the system architecture. Furthermore, due to the widespread application and maturity of Bluetooth Low Energy technology, this solution is also easy to integrate and expand with other devices or systems.

[0040] Optionally, the method further includes: When the incoming call information is sent to the AR glasses, it is displayed on the display interface of the AR glasses, and the user is prompted to answer or reject the call; In response to the user's answering operation, the call audio on the mobile phone is switched to the AR glasses using the call protocol channel.

[0041] When the phone receives an incoming call (e.g., caller ID, contact information, etc.), this information is sent to the AR glasses via a previously established optimal channel (e.g., a Bluetooth Low Energy channel). Upon receiving the incoming call, the AR glasses display the incoming call notification to the user. The interface may also provide options to answer or decline the call, allowing the user to quickly respond. If the user decides to answer the call after seeing the incoming call notification on the AR glasses, they can do so using the answer option on the interface. When the AR glasses detect the user's answer action, the corresponding command flow is triggered, notifying the phone of the user's choice of answer. In response to the user's answer action, the phone switches the call audio from the phone to the AR glasses using a call protocol channel (e.g., a Bluetooth-based audio transmission protocol such as A2DP or HFP). This call protocol channel ensures stable audio transmission, allowing the user to hear the other party's voice through the AR glasses' speakers or headphones and to continue the call through the AR glasses' microphone.

[0042] When an incoming call is sent to the AR glasses, it is displayed on the glasses' display interface, prompting the user to answer or decline the call. This provides an intuitive and convenient way for users to handle incoming calls without having to retrieve their phone. In response to the user's answer, the call audio from the phone is switched to the AR glasses using the call protocol channel. This allows users to make calls directly on the AR glasses without having to hold their phone, freeing their hands and improving the convenience and comfort of the call. Especially when driving, exercising, or in other situations requiring both hands, users can answer calls directly on the AR glasses without having to distract themselves from their phone. This significantly improves user safety and allows them to focus more on their current task or activity. Using the call protocol channel for audio switching ensures real-time and stable audio transmission. Users can enjoy a clear and smooth call experience on the AR glasses without audio delays or interruptions. This technical solution utilizes the existing call protocol channel and the AR glasses' display interface to display incoming call information and switch call audio. This simplifies the system architecture, reduces implementation complexity, and makes the solution easy to integrate and expand with other devices or systems.

[0043] Figure 6This is a flowchart of the mobile phone notification synchronization disclosed in the embodiment of this application, such as Figure 6 As shown, the app requests NotificationListener permissions from the user at the Android system level. When a new notification is triggered (e.g., incoming call, SMS, WeChat, email, etc.), the system triggers a callback to the listener method. The notification content is filtered as necessary (e.g., blacklisted or transmitting only the title and content summary), then compressed and encrypted. It is then sent to the glasses via the BLE server in segments. If the notification is too large, the title and content summary can be transmitted separately. Upon receiving the notification, the glasses display the notification content as a pop-up or floating window on the screen, keeping the user informed.

[0044] Figure 7 This is a flowchart of the glasses end power acquisition disclosed in the embodiment of this application, such as Figure 7 As shown, the mobile app scans and connects to the AR glasses' Battery Service GATT service in client mode, identifying characteristic power values (such as 0x2A19BatteryLevel). The app can obtain power information through polling (regularly reading the power level) or subscription (automatically notifying the glasses when the power level or charging status changes). Once obtained, the power level information is displayed visually on the app interface, and a pop-up or vibration reminder appears if the battery is low.

[0045] Figure 8 This is a flowchart of incoming call synchronization and glasses-side answering disclosed in the embodiment of this application, such as Figure 8 As shown, the app monitors incoming calls on the phone through the TelephonyManager or Call API, obtaining the caller ID and contact name. It then generates an incoming call data packet (e.g., {number, contact, call timestamp, current status}). This data is compressed and encrypted via the BLE server and sent to the glasses. After receiving and decrypting the data, the glasses display the caller ID and name, prompting the user to "Answer" or "Reject." The glasses allow the user to choose to answer or reject the call using a physical button or HID key, which is then transmitted back to the mobile app via the BLE or HID key value. The phone then calls the system dialing API to answer or hang up the call based on the user's instructions. If the glasses have a built-in microphone and speaker and support the Bluetooth calling profile (HFP / HSP), the phone switches the audio route to the glasses when the call is answered. The user can engage in hands-free communication through the glasses, while the phone maintains only call signaling and network connectivity. After the call ends or the call is disconnected, the phone reverts to the default audio route.

[0046] Optionally, when the interaction data or the device status is abnormal, switching the optimal channel to the target channel includes: If the low-power Bluetooth channel is abnormal, the navigation data and notification information will be synchronously migrated to the Bluetooth socket channel, and the transmission priority of the human interaction device protocol will be increased; If the Bluetooth socket channel is abnormal, switch the prompt file transmission mode to the low-power Bluetooth channel fragment transmission mode and set the single fragment size threshold; If the call protocol channel is abnormal, the call audio will be replayed to the mobile phone speaker, and an audio replay alarm will be sent to the AR glasses via the low-power Bluetooth channel.

[0047] The system monitors the status of the Bluetooth LE channel in real time. If an anomaly is detected (such as a connection interruption or unstable signal), a switching mechanism is immediately triggered. Navigation data and notification information originally transmitted over the Bluetooth LE channel are simultaneously migrated to the Bluetooth socket channel for transmission. The Bluetooth socket channel provides a more stable and reliable connection, suitable for transmitting data with high real-time requirements and relatively large data volumes. Furthermore, the protocol transmission priority for human interaction devices is increased to ensure that critical human interaction data (such as touch and button commands) is transmitted first after the channel switch, reducing latency. Similarly, the system monitors the status of the Bluetooth LE channel in real time. If an anomaly is detected (such as a transmission error or connection timeout), a switching mechanism is immediately triggered. Prompt files originally transmitted over the Bluetooth LE channel are switched to fragmented transmission mode over the Bluetooth LE channel. Fragmented transmission reduces the amount of data transmitted per transmission, reduces the risk of transmission errors, and improves transmission flexibility. A single fragment size threshold is set to appropriately divide the fragment size based on the transmission capacity of the Bluetooth LE channel and the size of the prompt file to ensure transmission efficiency and stability. The system monitors the status of the call protocol channel in real time and immediately triggers a switching mechanism if an anomaly is detected (such as audio loss or call interruption). The call audio is replayed back to the phone's speaker, ensuring that the user can continue to hear the call regardless of the channel anomaly. Simultaneously, an audio replay alert is sent to the AR glasses via the Bluetooth Low Energy channel, notifying the user that the call audio has replayed back to the phone and reminding them to check their device status or reconnect to the call protocol channel.

[0048] When an anomaly occurs in the Bluetooth LE channel, Bluetooth socket channel, or call protocol channel, the system automatically detects and switches to a backup channel, ensuring continuous data transmission and service stability. This dynamic switching mechanism significantly enhances system robustness and reduces service interruptions caused by channel failures. In the event of an anomaly in the Bluetooth LE channel, navigation data and notification information are simultaneously migrated to the Bluetooth socket channel, and the human interaction device protocol transmission priority is increased. This ensures timely transmission of critical data while optimizing data transmission efficiency by leveraging the stability and high transmission rate of the Bluetooth socket channel. In the event of an anomaly in the Bluetooth socket channel, the prompt file transmission method is switched to fragmented transmission using the Bluetooth LE channel, with a set single fragment size threshold. This facilitates efficient prompt file transmission within limited bandwidth and avoids transmission failures caused by excessive single data transmission. In the event of an anomaly in the call protocol channel, call audio is reverted to the phone's speaker, and an audio reversion alert is sent to the AR glasses via the Bluetooth LE channel. This ensures call continuity and prevents users from being interrupted by channel failures. At the same time, by sending audio fallback alerts, users can promptly understand changes in call status and make appropriate adjustments. By dynamically switching channels and optimizing data transmission efficiency, this technical solution reduces delays, interruptions, and data loss caused by channel failures, thereby improving the user experience. Users can use functions such as navigation, prompting, and calling in a more stable and smooth environment.

[0049] This embodiment also discloses a system for interaction between multifunctional AR glasses and mobile phones. Figure 9 This is a module diagram of a system for interacting with a mobile phone using multifunctional AR glasses disclosed in an embodiment of the present application. Figure 9 As shown, the system includes a communication module 901, a selection module 902, a processing module 903 and a switching module 904, wherein: Communication module 901 is configured to establish multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, overlay a protocol encapsulation layer on the multiple communication channels, and establish a mapping relationship between data types and the multiple communication channels; A selection module 902 is configured to map a user operation into a cross-device instruction stream and select an optimal channel based on a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; a processing module 903 configured to determine a compression and encryption strategy based on the data type and real-time network conditions, process the cross-device instruction stream using the compression and encryption strategy, and transmit the processed cross-device instruction stream through the optimal channel; The switching module 904 is configured to obtain the interaction data and device status between the AR glasses and the mobile phone. When the interaction data or the device status is abnormal, the optimal channel is switched to the target channel. The target channel is the communication channel with the best performance except the optimal channel in the current state.

[0050] Optionally, the selection module 902 is configured to: Providing a virtual touchpad, a virtual key interface, and a soft keyboard on the mobile phone terminal, and in response to the mobile phone terminal receiving a single-finger or multi-finger sliding operation or an operation of a direction key, a confirmation key, or a return key by the user, mapping the single-finger or multi-finger sliding operation and the direction key, the confirmation key, and the return key operation into a key value instruction of a human body interaction device, and converting text characters input by the user into a key value sequence of the human body interaction device in real time; An optimal channel is determined to be a human body interaction device protocol channel according to the human body interaction device key value instruction, the human body interaction device key value sequence and the mapping relationship.

[0051] Optionally, the selection module 902 is configured to: In response to receiving a navigation request input by a user on the mobile phone, obtaining a starting point, an end point, and an operation mode from the navigation request, calling a third-party map software development toolkit to generate a navigation route on the mobile phone according to the starting point, the end point, and the operation mode, and slicing the position coordinates, turn prompts, and remaining distance in the navigation route to obtain navigation data; The optimal channel is determined to be a low-power Bluetooth channel according to the navigation data and the mapping relationship.

[0052] Optionally, the selection module 902 is configured to: In response to the mobile phone receiving a prompt request input by a user, obtaining a prompt file and prompt parameters from the prompt request, wherein the prompt file includes document content, and the prompt parameters include font size, color, scrolling speed, and speech progress of the teleprompter; According to the prompt file and the mapping relationship, the optimal channel is determined to be the Bluetooth socket channel, and according to the prompt parameters and the mapping relationship, the optimal channel is determined to be the low-power Bluetooth channel, so as to transmit the prompt file and the prompt parameters respectively.

[0053] Optionally, the selection module 902 is configured to: Listening to notification events through the notification listener on the mobile phone, and obtaining notification information from the notification event, the notification information including the notification title, notification content and notification source; Monitor incoming call events through the phone manager on the mobile phone, and obtain incoming call information from the incoming call events, wherein the incoming call information includes the caller number and contact information; The optimal channel is determined to be a low-power Bluetooth channel according to the notification information and / or the incoming call information.

[0054] Optionally, the system further includes a prompt module configured to: When the incoming call information is sent to the AR glasses, it is displayed on the display interface of the AR glasses, and the user is prompted to answer or reject the call; In response to the user's answering operation, the call audio on the mobile phone is switched to the AR glasses using the call protocol channel.

[0055] Optionally, the switching module 904 is configured to: If the low-power Bluetooth channel is abnormal, the navigation data and notification information will be synchronously migrated to the Bluetooth socket channel, and the transmission priority of the human interaction device protocol will be increased; If the Bluetooth socket channel is abnormal, switch the prompt file transmission mode to the low-power Bluetooth channel fragment transmission mode and set the single fragment size threshold; If the call protocol channel is abnormal, the call audio will be replayed to the mobile phone speaker, and an audio replay alarm will be sent to the AR glasses via the low-power Bluetooth channel.

[0056] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0057] This embodiment also discloses an electronic device, referring to Figure 10 The electronic device may include: at least one processor 1001 , at least one communication bus 1002 , a user interface 1003 , a network interface 1004 , and at least one memory 1005 .

[0058] The communication bus 1002 is used to implement the connection and communication between these components.

[0059] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0060] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0061] Processor 1001 may include one or more processing cores. Using various interfaces and circuits, processor 1001 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in memory 1005, as well as accesses data stored in memory 1005, to perform various server functions and process data. Optionally, processor 1001 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). Processor 1001 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display; and the modem handles wireless communications. It is understood that the modem may not be integrated into processor 1001 but implemented as a separate chip.

[0062] Among them, the memory 1005 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 10 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program for a method for interacting with a multifunctional AR glasses and a mobile phone.

[0063] exist Figure 10In the electronic device shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call an application stored in the memory 1005 for a method of interacting with multifunctional AR glasses and a mobile phone. When executed by one or more processors 1001, the electronic device executes one or more methods in the above embodiments.

[0064] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0065] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0067] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0068] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 1005 and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory 1005 includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0070] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for interacting multifunctional AR glasses with a mobile phone, characterized in that: The method comprises: Establishing multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, overlaying a protocol encapsulation layer on the multiple communication channels, and establishing a mapping relationship between data types and the multiple communication channels; Mapping a user operation into a cross-device instruction stream, and selecting an optimal channel according to a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; determining a compression and encryption strategy based on the data type and real-time network conditions, processing the cross-device instruction stream using the compression and encryption strategy, and transmitting the processed cross-device instruction stream through the optimal channel; The interaction data and device status between the AR glasses and the mobile phone are obtained. When the interaction data or the device status is abnormal, the optimal channel is switched to the target channel, where the target channel is the communication channel with the best performance except the optimal channel in the current state.

2. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 1, characterized in that: Mapping the user operation into a cross-device instruction stream and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship includes: Providing a virtual touchpad, a virtual key interface, and a soft keyboard on the mobile phone terminal, and in response to the mobile phone terminal receiving a single-finger or multi-finger sliding operation or an operation of a direction key, a confirmation key, or a return key by the user, mapping the single-finger or multi-finger sliding operation and the direction key, the confirmation key, and the return key operation into a key value instruction of a human body interaction device, and converting text characters input by the user into a key value sequence of the human body interaction device in real time; An optimal channel is determined to be a human body interaction device protocol channel according to the human body interaction device key value instruction, the human body interaction device key value sequence and the mapping relationship.

3. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 1, characterized in that: Mapping the user operation into a cross-device instruction stream, and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to receiving a navigation request input by a user on the mobile phone, obtaining a starting point, an end point, and an operation mode from the navigation request, calling a third-party map software development toolkit to generate a navigation route on the mobile phone according to the starting point, the end point, and the operation mode, and slicing the position coordinates, turn prompts, and remaining distance in the navigation route to obtain navigation data; According to the navigation data and the mapping relationship, it is determined that the optimal channel is a low-power Bluetooth channel.

4. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 1, characterized in that: Mapping the user operation into a cross-device instruction stream, and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: In response to the mobile phone receiving a prompt request input by a user, obtaining a prompt file and prompt parameters from the prompt request, wherein the prompt file includes document content, and the prompt parameters include font size, color, scrolling speed, and speech progress of the teleprompter; According to the prompt file and the mapping relationship, the optimal channel is determined to be the Bluetooth socket channel, and according to the prompt parameters and the mapping relationship, the optimal channel is determined to be the low-power Bluetooth channel, so as to transmit the prompt file and the prompt parameters respectively.

5. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 1, characterized in that: Mapping the user operation into a cross-device instruction stream, and selecting the optimal channel according to the data type of the cross-device instruction stream and the mapping relationship further includes: Listening to notification events through the notification listener on the mobile phone, and obtaining notification information from the notification event, the notification information including the notification title, notification content and notification source; Monitor incoming call events through the phone manager on the mobile phone, and obtain incoming call information from the incoming call events, wherein the incoming call information includes the incoming call number and contact information; The optimal channel is determined to be a low-power Bluetooth channel according to the notification information and / or the incoming call information.

6. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 5, characterized in that: The method further comprises: When the incoming call information is sent to the AR glasses, it is displayed on the display interface of the AR glasses, and the user is prompted to answer or reject the call; In response to the user's answering operation, the call audio on the mobile phone is switched to the AR glasses using the call protocol channel.

7. The method for interaction between multifunctional AR glasses and a mobile phone according to claim 1, characterized in that: When the interaction data or the device state is abnormal, switching the optimal channel to the target channel includes: If the low-power Bluetooth channel is abnormal, the navigation data and notification information will be synchronously migrated to the Bluetooth socket channel, and the transmission priority of the human interaction device protocol will be increased; If the Bluetooth socket channel is abnormal, switch the prompt file transmission mode to the low-power Bluetooth channel fragment transmission mode and set the single fragment size threshold; If the call protocol channel is abnormal, the call audio will be replayed to the mobile phone speaker, and an audio replay alarm will be sent to the AR glasses via the low-power Bluetooth channel.

8. A system for interacting multifunctional AR glasses with a mobile phone, characterized in that: It includes a communication module, a selection module, a processing module and a switching module, wherein: A communication module configured to establish multiple communication channels between the AR glasses and the mobile phone through a bidirectional protocol, superimpose a protocol encapsulation layer on the multiple communication channels, and establish a mapping relationship between data types and the multiple communication channels; a selection module configured to map a user operation into a cross-device instruction stream and select an optimal channel according to a data type of the cross-device instruction stream and the mapping relationship, wherein the operation includes touch, gesture, and voice, and the data type includes text data, navigation data, and audio data; a processing module configured to determine a compression and encryption strategy based on the data type and real-time network conditions, process the cross-device instruction stream using the compression and encryption strategy, and transmit the processed cross-device instruction stream through the optimal channel; The switching module is configured to obtain the interaction data and device status between the AR glasses and the mobile phone. When the interaction data or the device status is abnormal, the optimal channel is switched to the target channel. The target channel is the communication channel with the best performance except the optimal channel in the current state.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, a user interface and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images