Auxiliary method for direct communication between in-vehicle voice assistant and WeChat in virtual machine

Abstract

The embodiment of the invention provides an auxiliary method for direct communication between a vehicle voice assistant and WeChat in a virtual machine, and belongs to the technical field of communication, and the method specifically comprises the steps: 1, collecting and storing the component coordinate information of a WeChat application interface in the virtual machine; step 2, the vehicle voice assistant receives a user voice instruction, and generates a standardized interaction instruction with an authority label after semantic analysis; step 3, performing security verification on the standardized interaction instruction through a security unvarnished transmission layer running between the vehicle machine host system and the virtual machine, and then issuing the standardized interaction instruction to the virtual machine; 4, an intelligent auxiliary interaction agent module in the virtual machine receives the verified instruction and executes a corresponding target operation; and step 5, generating a receipt containing verification information according to an execution result of the target communication operation, feeding back the receipt to the in-vehicle voice assistant through the safe unvarnished transmission layer, and performing result broadcasting or multi-round interactive guidance by the in-vehicle voice assistant according to the receipt. Through the scheme of the invention, the security and reliability of communication are improved.

Description

An auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine. Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine. Background Technology

[0002] Currently, the infotainment systems in existing smart cars are becoming increasingly complex. Users often use the voice assistant installed on the in-vehicle system for navigation, music playback, and other operations. However, within the Android ecosystem provided by the virtual machine, users still need to operate the third-party communication application WeChat through the touchscreen, and cannot directly send or receive messages or make calls with WeChat friends through the in-vehicle voice assistant. This not only increases the operational burden during driving but also affects driving safety. Specifically, the following defects exist:

[0003] 1. Existing patent solutions are mostly focused on local applications for voice control in vehicles, and cannot freely and directly control WeChat operations within the virtual machine.

[0004] 2. Directly using the system's accessibility service to achieve the purpose of automatic application operation, but the conventional system accessibility service has reliability issues such as failure to recognize commands or timeouts when reading application components, resulting in a poor user experience.

[0005] 3. Generally, in-vehicle voice assistants have a certain timeout period after issuing commands. When the voice execution steps are complex and fail, the voice assistant will wait for a timeout, resulting in a very poor user experience.

[0006] It is evident that there is an urgent need for an auxiliary method that can improve the security and reliability of WeChat communication between in-vehicle voice assistants and WeChat within virtual machines. Summary of the Invention

[0007] In view of this, the present disclosure provides an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine, which at least partially solves some of the problems existing in the prior art.

[0008] This disclosure provides an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine, including:

[0009] Step 1: Collect and store the component coordinate information of the WeChat application interface within the virtual machine;

[0010] Step 2: The vehicle's voice assistant receives the user's voice command, and after semantic parsing, generates standardized interactive commands with permission tags.

[0011] Step 3: After the standardized interaction commands are securely verified by the secure pass-through layer running between the vehicle host system and the virtual machine, they are sent to the virtual machine.

[0012] Step 4: The intelligent auxiliary interaction agent module in the virtual machine receives the verified instruction and executes the corresponding target operation, wherein the target operation includes: judging and adapting the status of the WeChat application, and executing the target communication operation related to the contact.

[0013] Step 5: Generate a receipt containing verification information for the execution result of the target communication operation, and feed it back to the vehicle voice assistant through the secure pass-through layer. The vehicle voice assistant will then broadcast the result or provide multi-round interactive guidance based on the receipt.

[0014] According to a specific implementation of an embodiment of this disclosure, step 4 specifically includes:

[0015] Step 4.1: Determine if the WeChat application is installed in the virtual machine. If yes, proceed to step 4.2. If no, return a prompt message and guide the installation process.

[0016] Step 4.2: If the WeChat application is already installed, determine whether the user is currently logged in. If yes, proceed to step 4.3; otherwise, remind the user to log in when the time is right.

[0017] Step 4.3: Based on the component coordinate information and combined with the real-time component coordinates obtained from the system's accessibility service, locate and activate WeChat's contact search function;

[0018] Step 4.4: Enter the target contact information and perform at least one of the following actions based on the search results: sending a message, making a voice call, or making a video call.

[0019] According to a specific implementation of an embodiment of this disclosure, step 4.3 specifically includes:

[0020] Step 4.3.1: Obtain the real-time coordinates of the WeChat search component through the system's accessibility service;

[0021] Step 4.3.2: Compare the real-time coordinates with the corresponding component coordinate information;

[0022] Step 4.3.3: If the comparison result is within the preset tolerance range, then based on the component coordinate information, the click operation is performed by simulating input coordinates to activate the contact search interface.

[0023] According to a specific implementation of an embodiment of this disclosure, step 4.4 specifically includes:

[0024] The system accessibility service retrieves the contact input component; if successful, the component is invoked for input.

[0025] If the input method interface is not successfully retrieved, the contact information will be entered by calling the input method interface.

[0026] If multiple matching contacts are found, the contact list information will be returned to the vehicle's voice assistant to initiate a secondary voice confirmation process.

[0027] According to one specific implementation of this disclosure, the intelligent auxiliary interaction proxy module prioritizes calling the WeChat application interface to perform operations; if the call fails, it switches to simulating user input operations.

[0028] According to a specific implementation of an embodiment of this disclosure, step 4 further includes:

[0029] When performing a video call, determine whether the vehicle is currently in motion.

[0030] If the vehicle is in motion, the video call will be automatically converted to a voice call.

[0031] According to a specific implementation of this disclosure, the standardized interaction instruction is in JSON format and includes a target application identifier, operation type, operation object, and permission tag.

[0032] According to a specific implementation of this disclosure, the security verification includes permission verification and dynamic token verification, and the security pass-through layer runs in a trusted execution environment.

[0033] The auxiliary scheme for direct communication between the vehicle-mounted voice assistant and WeChat within a virtual machine in this embodiment includes: Step 1, collecting and storing component coordinate information of the WeChat application interface within the virtual machine; Step 2, the vehicle-mounted voice assistant receives user voice commands, generates standardized interaction commands with permission tags after semantic parsing; Step 3, the standardized interaction commands are securely verified and sent to the virtual machine through a secure pass-through layer running between the vehicle-mounted host system and the virtual machine; Step 4, the intelligent auxiliary interaction agent module within the virtual machine receives the verified commands and executes the corresponding target operation, wherein the target operation includes: judging and adapting the status of the WeChat application, and executing target communication operations related to contacts; Step 5, generating a receipt containing verification information from the execution result of the target communication operation, feeding it back to the vehicle-mounted voice assistant through the secure pass-through layer, and the vehicle-mounted voice assistant broadcasting the result or providing multi-round interactive guidance based on the receipt.

[0034] The beneficial effects of the embodiments disclosed herein are as follows:

[0035] 1. Adaptive Interaction Flow: Adjusts the interaction mode based on the results returned by the intelligent assisted interaction agent to improve the user experience of the vehicle system;

[0036] 2. Secure and Reliable: Through permission tags and dynamic token mechanisms, the security and controllability of communication information during cross-system interactions are ensured;

[0037] 3. Intelligent Agent: Supports multiple execution paths and has fault tolerance and adaptive capabilities, ensuring high compatibility;

[0038] 4. Robust closed-loop: The receipt mechanism and multi-round interaction design ensure that the interaction process is verifiable and reversible;

[0039] 5. Continuous Evolution: Through a collaborative update mechanism, the long-term compatibility and scalability of the vehicle system and the virtual machine are ensured.

[0040] 6. Fast and accurate: By collecting coordinates in advance, the accuracy of each component of WeChat during communication can be guaranteed, which greatly improves the accuracy of the overall process and shortens the overall process time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 is a flowchart illustrating an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine, provided in an embodiment of this disclosure.

[0043] Figure 2 is a system framework diagram corresponding to an auxiliary method for direct communication between a vehicle voice assistant and WeChat within a virtual machine, provided in an embodiment of this disclosure. Detailed Implementation

[0044] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0045] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0046] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0047] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0048] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0049] This disclosure provides an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine. This method can be applied to the vehicle-mounted interaction process in intelligent vehicle service scenarios.

[0050] Referring to Figure 1, this is a flowchart illustrating an auxiliary method for direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine, provided by an embodiment of this disclosure. As shown in Figures 1 and 2, the method mainly includes the following steps:

[0051] Step 1: Collect and store the component coordinate information of the WeChat application interface within the virtual machine;

[0052] In practice, the intelligent coordinate assistant inside the vehicle's virtual machine uses a special algorithm to collect the coordinates of various WeChat page components on the vehicle's screen and intelligently stores them within the virtual machine.

[0053] Step 2: The vehicle's voice assistant receives the user's voice command, and after semantic parsing, generates standardized interactive commands with permission tags.

[0054] Optionally, the standardized interaction instructions are in JSON format, and the standardized interaction instructions include the target application identifier, operation type, operation object, and permission tag.

[0055] In specific implementation, the intelligent semantic understanding module identifies the communication application type (WeChat), operation category (launching WeChat, WeChat login status judgment, sending messages, sharing location, voice call, video call), and operation object (contact information), as well as various status results returned internally by the virtual machine, and uploads them to the vehicle's voice assistant. The parsed results are mapped to standardized interaction commands (such as JSON structures) with permission tags, ensuring that only predefined communication function operations and return result commands are allowed to be sent. Step 3 involves using a secure pass-through layer running between the vehicle's host system and the virtual machine to perform security verification on the standardized interaction commands before sending them to the virtual machine.

[0056] Furthermore, the security verification includes permission verification and dynamic token verification, and the security pass-through layer runs in a trusted execution environment.

[0057] In practical implementation, a secure mapping channel is established between the vehicle-mounted host system and the virtual machine. The pass-through layer runs in a trusted execution environment, performing permission verification and dynamic token verification on the issued standardized interaction commands to prevent unauthorized operations. Step 4: The intelligent auxiliary interaction agent module in the virtual machine receives the verified commands and executes the corresponding target operations. The target operations include: judging and adapting the status of the WeChat application, and executing target communication operations related to contacts.

[0058] Based on the above embodiments, step 4 specifically includes:

[0059] Step 4.1: Determine if the WeChat application is installed in the virtual machine. If yes, proceed to step 4.2. If no, return a prompt message and guide the installation process.

[0060] Step 4.2: If the WeChat application is already installed, determine whether the user is currently logged in. If yes, proceed to step 4.3; otherwise, remind the user to log in when the time is right.

[0061] Step 4.3: Based on the component coordinate information and combined with the real-time component coordinates obtained from the system's accessibility service, locate and activate WeChat's contact search function;

[0062] Step 4.4: Enter the target contact information and perform at least one of the following actions based on the search results: sending a message, making a voice call, or making a video call.

[0063] Furthermore, step 4.3 specifically includes:

[0064] Step 4.3.1: Obtain the real-time coordinates of the WeChat search component through the system's accessibility service;

[0065] Step 4.3.2: Compare the real-time coordinates with the corresponding component coordinate information;

[0066] Step 4.3.3: If the comparison result is within the preset tolerance range, then based on the component coordinate information, the click operation is performed by simulating input coordinates to activate the contact search interface.

[0067] Furthermore, step 4.4 specifically includes:

[0068] The system accessibility service retrieves the contact input component; if successful, the component is invoked for input.

[0069] If the input method interface is not successfully retrieved, the contact information will be entered by calling the input method interface.

[0070] If multiple matching contacts are found, the contact list information will be returned to the vehicle's voice assistant to initiate a secondary voice confirmation process.

[0071] Furthermore, the intelligent auxiliary interaction proxy module prioritizes calling the WeChat application interface to perform operations. If the call fails, it switches to simulating user input operations.

[0072] Furthermore, step 4 also includes:

[0073] When performing a video call, determine whether the vehicle is currently in motion.

[0074] If the vehicle is in motion, the video call will be automatically converted to a voice call.

[0075] In practice, the intelligent auxiliary interaction agent module within the virtual machine receives verified voice commands and executes the following actions:

[0076] 1. Determine if WeChat is installed. If not, return the result to the car's voice prompt and then automatically redirect to the WeChat installation page in the browser. If it is installed, launch WeChat and execute the step-by-step process. 2.

[0077] 2. If WeChat is installed, the WeChat homepage will be launched. The intelligent auxiliary interaction agent will determine whether the user is currently logged in based on the WeChat page status and return to the vehicle's voice system. If the user is not logged in, the system will remind them to log in when the time is right. If the user is already logged in, the system will proceed with step 3.

[0078] 3. The intelligent assisted interaction agent will first obtain the coordinate system of the WeChat search component through the system accessibility service, and then compare it with the component coordinates collected in the main step process 1. If the difference is not significant, it will be called. Since the click event of the system accessibility service is unreliable, intelligent interaction is used to simulate input coordinates to perform the click operation. The intelligent assisted interaction agent determines whether the current operation is effective based on the "WeChat page status". If the current page is not in the WeChat search page, the intelligent assisted interaction agent will execute the "retry mechanism". When the page is in the WeChat search interface, the sub-step process 4 will be executed.

[0079] 4. Upon being redirected to the WeChat search interface, the intelligent auxiliary interaction module will first retrieve the contact input component through the system's accessibility service. If the component is successfully retrieved, it will be invoked to input the contact information. If not found, the input method will be used directly to input the contact information. When WeChat displays a contact search option, a secondary interaction will be performed based on the number of contacts. If there is only one contact, the intelligent auxiliary interaction will be delegated to execute the operation.

[0080] Step 5: Generate a receipt containing verification information for the execution result of the target communication operation, and feed it back to the vehicle voice assistant through the secure pass-through layer. The vehicle voice assistant will then broadcast the result or provide multi-round interactive guidance based on the receipt.

[0081] In practice, within the virtual machine, the returned voice feedback will generate a receipt containing status information and a verification token, which will be fed back to the vehicle's voice assistant through the interaction agent layer of the main step process 2. When the vehicle's assistant broadcasts the result, it can verify the token to confirm the validity of the execution. If the execution fails, the assistant will guide the user to perform multiple rounds of interaction (such as modifying the message or selecting a specific contact). The XXX method provided in this embodiment overcomes the problems of unreliable recognition and easy timeout in cross-virtual machine environments of the system's accessibility service by pre-collecting and intelligently matching the screen coordinates of the WeChat component in the virtual machine, combined with simulated input operations. This enables the vehicle's voice assistant to accurately and quickly call the core communication functions of WeChat. By establishing a secure pass-through layer running in a trusted execution environment between the vehicle's host system and the virtual machine, and by performing permission tagging and dynamic token verification on all interactive commands, the secure and controllable cross-system communication process is ensured, preventing unauthorized access and information leakage. By deploying an intelligent auxiliary interaction agent module, it has the ability to judge the installation and login status of WeChat, adaptively select the execution path (prioritizing API calls and seamlessly switching to simulated input after failure), and automatically switch operation types based on driving status, which significantly improves the system's compatibility and practicality for different usage scenarios. By establishing a closed-loop feedback mechanism for command execution that includes verification tokens and a multi-round interaction mechanism, the vehicle's voice assistant can confirm the validity of command execution and guide the user to clarify and make choices when the operation is interrupted or ambiguous, thereby greatly improving the reliability of the interaction process and the integrity of the user experience.

[0082] The method of this disclosure will be further described below with reference to two specific embodiments:

[0083] Example 1:

[0084] The driver said, "Car, please send a WeChat message to Li Hua: I have set off." System analysis: Target application = WeChat, Contact = Li Hua, Operation = Send message, Content = "I have set off".

[0085] - The secure pass-through layer verifies the permissions of the command before issuing it;

[0086] - The virtual machine interaction agent prioritizes calling the WeChat API; if that fails, it switches to input simulation.

[0087] - The interactive agent searches for contacts based on input, selects a specific contact, or returns the number of contacts searched for the user to issue a second command;

[0088] - After the WeChat application completes sending the message, it returns a status of "success" and generates a verification token;

[0089] - After the car's infotainment system verifies the token, it will announce in a voice message: "I have sent a WeChat message to Li Hua for you."

[0090] Example 2:

[0091] The driver said, "Car, please help me make a WeChat video call with Li Hua."

[0092] System analysis: Target application = WeChat, contact = Li Hua, operation = video call.

[0093] - The secure pass-through layer verifies the permissions of the command before issuing it;

[0094] - The virtual machine interaction agent prioritizes calling the WeChat API; if that fails, it switches to input simulation.

[0095] - The interactive agent searches for contacts based on input, selects a specific contact, or returns the number of contacts searched for the user to issue a second command;

[0096] - The interaction agent obtains the current driving status and converts the operation to be performed. When the driving status is such that the operation is converted into "voice call";

[0097] - After the WeChat application completes the operation command, the interaction agent determines whether the current WeChat is on the call page. If the call is successful, it returns a status of = success = failure and generates a verification token;

[0098] - After the vehicle's infotainment system verifies the token, it will announce in voice: "Li Hua has been called" or "The current command execution is abnormal, please try again".

[0099] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0100] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for assisting in direct communication between a vehicle-mounted voice assistant and WeChat within a virtual machine, characterized in that, include: Step 1: Collect and store the component coordinate information of the WeChat application interface within the virtual machine; Step 2: The vehicle's voice assistant receives the user's voice command, generates standardized interaction commands with permission tags after semantic parsing; Step 3: After security verification of the standardized interaction commands through a secure pass-through layer running between the vehicle's host system and the virtual machine, the commands are sent to the virtual machine; Step 4: The intelligent auxiliary interaction agent module within the virtual machine receives the verified commands and executes the corresponding target operation, which includes: judging and adapting the status of the WeChat application, and executing target communication operations related to contacts; Step 5: Generate a receipt containing verification information from the execution result of the target communication operation, feed it back to the vehicle's voice assistant through the secure pass-through layer, and the vehicle's voice assistant broadcasts the result or provides multi-round interactive guidance based on the receipt.

2. The method according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1, determining whether the WeChat application is installed in the virtual machine; if yes, proceed to Step 4.2; otherwise, return a prompt message and guide the installation process; Step 4.2, if the WeChat application is already installed, determining whether the user is currently logged in; if yes, proceed to Step 4.3; otherwise, remind the user to log in at an appropriate time; Step 4.3, based on component coordinate information and combined with real-time component coordinates obtained from the system's accessibility service, locate and activate WeChat's contact search function; Step 4.4, inputting the target contact information and performing at least one of the following operations based on the search results: sending a message, making a voice call, or making a video call.

3. The method according to claim 2, characterized in that, Step 4.3 specifically includes: Step 4.3.1, obtaining the real-time coordinates of the WeChat search component through the system accessibility service; Step 4.3.2, comparing the real-time coordinates with the corresponding component coordinate information; Step 4.3.3, if the comparison result is within the preset tolerance range, then based on the component coordinate information, performing a click operation by simulating input coordinates to activate the contact search interface.

4. The method according to claim 3, characterized in that, Step 4.4 specifically includes: obtaining the contact input component through the system's accessibility service; if successful, calling the component for input; if unsuccessful, calling the input method interface to input contact information; if multiple matching contacts are found, returning the contact list information to the vehicle's voice assistant and initiating a secondary voice confirmation process.

5. The method according to claim 1, characterized in that, The intelligent auxiliary interaction proxy module prioritizes calling the WeChat application interface to perform operations. If the call fails, it switches to simulating user input operations.

6. The method according to claim 2, characterized in that, Step 4 further includes: when performing a video call, obtaining whether the vehicle is currently in a driving state; if the vehicle is in a driving state, automatically converting the video call operation into a voice call operation.

7. The method according to claim 1, characterized in that, The standardized interaction instructions are in JSON format and include the target application identifier, operation type, operation object, and permission tag.

8. The method according to claim 1, characterized in that, The security verification includes permission verification and dynamic token verification, and the security pass-through layer runs in a trusted execution environment.

Images