Vehicle linkage control method, device, user terminal and vehicle terminal

Abstract

The present invention relates to a vehicle linkage control method, device, user terminal and vehicle terminal. Among them, the vehicle linkage control method applied to the user terminal includes: obtaining a first control instruction input by the user; converting the first control instruction into a first natural language instruction; using a first large language model to determine whether the first natural language instruction is a linkage control instruction for controlling the vehicle; if the first natural language instruction is a linkage control instruction, sending the first natural language instruction to the vehicle terminal. The embodiment of the present application directly sends the first natural language instruction to the vehicle terminal, so that more information is retained in the first natural language instruction sent to the vehicle terminal, avoiding information loss, facilitating the vehicle terminal to perform more accurate intent recognition based on the first natural language instruction, and then determining the vehicle service to be linked controlled, thereby improving the accuracy of vehicle linkage control. Moreover, since the first natural language instruction is sent directly to the vehicle terminal, the efficiency of vehicle linkage control is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle linkage control method, device, user terminal and vehicle terminal. Background Art

[0002] With the advent of the era of smart cars, cars are no longer just a means of transportation, but a data center and energy center, an important link between people, cars, hands, home, and office. How to organically combine the rich ecology of mobile phones and the rich functions of cars so that users can use the mobile phone ecology in the car and control all car functions on the mobile phone is the future development trend.

[0003] In the traditional C / S (Client / Server) architecture used by mobile phones and in-vehicle systems, interaction between the client and server relies on predefined APIs (Application Programming Interfaces). To enable data transmission and function calls, developers must fully expose the APIs and their corresponding input and output parameters to the client.

[0004] However, under this design model, the API interface acts as a "digital bridge" connecting the client and server, with each parameter acting as a "signpost" for navigation. Any modification to a function on the server—even a minor tweak to a parameter's data type or the interface's calling logic—means a structural change to this "digital bridge." The client, as the caller of the function, must also make corresponding adjustments and adaptations simultaneously. Otherwise, communication with the server will be disrupted, leading to functional anomalies or even system crashes. This drastically impactful architectural feature significantly limits the system's flexibility and scalability, resulting in high costs for feature iteration and maintenance. Summary of the Invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a vehicle linkage control method, device, user terminal and vehicle terminal.

[0006] In a first aspect, the present application provides a vehicle linkage control method, applied to a user terminal, the method comprising:

[0007] Obtaining a first control instruction input by a user;

[0008] converting the first control instruction into a first natural language instruction;

[0009] determining, using a first large language model, whether the first natural language instruction is a linkage control instruction for controlling a vehicle;

[0010] If the first natural language instruction is a linkage control instruction, the first natural language instruction is sent to the vehicle computer.

[0011] Optionally, converting the first control instruction into a first natural language instruction includes:

[0012] If the first control instruction is input by the user by operating a preset first function control, obtaining a first control operation parameter of the preset first function control;

[0013] Converting the first control operation parameter into a first text instruction;

[0014] identifying first control intention information of the first text instruction;

[0015] The first text instruction is converted into a first natural language instruction based on the first control intention information.

[0016] Optionally, converting the first control instruction into a first natural language instruction includes:

[0017] If the first control instruction is input by the user through voice, converting the first control instruction into a first text instruction;

[0018] The control intention information of the first text instruction is extracted, and the first text instruction is converted into a first natural language instruction.

[0019] Optionally, determining whether the first natural language instruction is a linkage control instruction for controlling a vehicle by using the first large language model includes:

[0020] Using the first large language model to perform intent recognition on the first natural language instruction to obtain a first target control object;

[0021] If the first target control object is a vehicle, determining that the first natural language instruction is a linkage control instruction for controlling the vehicle;

[0022] Alternatively, if the first target control object is a user terminal, it is determined that the first natural language instruction is not a linkage control instruction.

[0023] Optionally, the method further includes:

[0024] Determining, using the first large language model, whether the first natural language instruction includes a first linkage reference object;

[0025] Acquire first linkage reference information corresponding to the first linkage reference object;

[0026] When sending the first natural language instruction to the vehicle computer, the first linkage reference information is sent.

[0027] In a second aspect, the present application provides a vehicle linkage control method, which is applied to a vehicle computer, and the method includes:

[0028] receiving a first natural language instruction from a user terminal;

[0029] determining, using the second largest language model, whether the first natural language instruction is a linkage control instruction for controlling a vehicle;

[0030] If the first natural language instruction is a linkage control instruction, determining a fourth target service and a fourth service control instruction corresponding to the first natural language instruction based on the second language model and a preset instruction conversion correspondence;

[0031] The fourth target service is called to execute the fourth service control instruction.

[0032] Optionally, the method further includes:

[0033] Obtaining vehicle status information and / or vehicle surrounding environment information;

[0034] If the vehicle status information and / or the vehicle surrounding environment information and the fourth service control instruction meet the preset prohibition linkage rule, calling the fourth target service to execute the fourth service control instruction is prohibited.

[0035] Alternatively, if the vehicle status information and / or vehicle surrounding environment information does not comply with a preset prohibition linkage rule with the fourth service control instruction, the fourth service control instruction is adjusted according to the vehicle status information and / or vehicle surrounding environment information.

[0036] In a third aspect, the present application provides a vehicle linkage control method, which is applied to a vehicle computer, and the method includes:

[0037] Obtaining a second control instruction input by the user;

[0038] converting the second control instruction into a second natural language instruction;

[0039] determining, using the second largest language model, whether the second natural language instruction is a linkage control instruction for controlling the user terminal;

[0040] If the second natural language instruction is a linkage control instruction, the second natural language instruction is sent to the user terminal.

[0041] Optionally, converting the second control instruction into a second natural language instruction includes:

[0042] If the second control instruction is input by the user on a preset second function control, obtaining a second control operation parameter corresponding to the second control instruction in the preset second function control;

[0043] Converting the second control operation parameter into a second text instruction;

[0044] identifying second control intention information of the second text instruction;

[0045] The second text instruction is converted into a second natural language instruction based on the second control intention information.

[0046] Optionally, converting the second control instruction into a second natural language instruction includes:

[0047] If the second control instruction is input by the user through voice, converting the second control instruction into a second text instruction;

[0048] The control intention information of the second text instruction is extracted, and the second text instruction is converted into a second natural language instruction.

[0049] Optionally, determining whether the first natural language instruction is a linkage control instruction for controlling a vehicle by using the second largest language model includes:

[0050] performing intent recognition on the second natural language instruction using the second largest language model to obtain a second target control object;

[0051] If the second target control object is a user terminal, determining that the second natural language instruction is a linkage control instruction for controlling the user terminal;

[0052] Alternatively, if the second target control object is a vehicle, it is determined that the second natural language instruction is not a linkage control instruction.

[0053] Optionally, the method further includes:

[0054] Determining whether the second natural language instruction includes a second linkage reference object using the second largest language model;

[0055] Acquire second linkage reference information corresponding to the second linkage reference object;

[0056] The second linkage reference information is sent simultaneously with sending the second natural language instruction to the user terminal.

[0057] In a fourth aspect, the present application provides a vehicle linkage control method, applied to a user terminal, the method comprising:

[0058] Receive the second natural language command from the vehicle terminal;

[0059] Determining, using the first large language model, whether the second natural language instruction is a linkage control instruction for controlling a user terminal;

[0060] If the second natural language instruction is a linkage control instruction, determining a second target service and a second service control instruction corresponding to the second natural language instruction based on the first language model and a preset instruction conversion correspondence;

[0061] The second target service is called to execute the second service control instruction.

[0062] In a fifth aspect, the present application provides a user terminal, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0063] Memory for storing computer programs;

[0064] The processor is configured to implement the vehicle linkage control method described in any one of the first aspect or the fourth aspect when executing the program stored in the memory.

[0065] In a sixth aspect, the present application provides a vehicle-mounted terminal, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0066] Memory for storing computer programs;

[0067] The processor is configured to implement the vehicle linkage control method described in any one of the second aspect or the third aspect when executing the program stored in the memory.

[0068] Beneficial effects of the present invention:

[0069] In the user terminal provided by an embodiment of the present invention, the processor converts the first control instruction input by the user into a first natural language instruction by executing the program stored in the memory. When the first natural language instruction is determined to be an interlocking control instruction for controlling the vehicle using the first large language model, the first natural language instruction is sent to the vehicle-machine end. By directly sending the first natural language instruction to the vehicle-machine end, more information is retained in the first natural language instruction sent to the vehicle-machine end, avoiding information loss, facilitating the vehicle-machine end to perform more accurate intent recognition based on the first natural language instruction, and then determine the vehicle service to be interlocked and controlled, thereby improving the accuracy of vehicle interlocking control. Moreover, since the first natural language instruction is sent directly to the vehicle-machine end, the efficiency of vehicle interlocking control is improved.

[0070] Also, when the first large language model is used to determine that the second natural language instruction is a linkage control instruction for controlling the vehicle, the second target service and the second service control instruction corresponding to the second natural language instruction are determined according to the first large language model and the preset instruction conversion correspondence, and the second target service is called to execute the second service control instruction to realize the control of the service in the vehicle. In the embodiment of the present application, more information is retained in the second natural language instruction, which reduces information loss, so that the vehicle-side can perform more accurate intention recognition based on the second natural language instruction, and then determine the vehicle service to be linked and controlled, thereby improving the accuracy of the vehicle linkage control. Since the second natural language instruction is sent directly to the vehicle-side, the efficiency of the vehicle linkage control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the vehicle linkage control more flexible, without the need for users to remember rigid control execution, thereby improving the efficiency of the vehicle linkage control.

[0071] In the vehicle-mounted terminal provided by an embodiment of the present invention, the processor converts the second control instruction input by the user into a second natural language instruction by executing the program stored in the memory. When the second natural language instruction is determined to be a linkage control instruction for controlling the user terminal by using the second largest language model, the second natural language instruction is sent to the user terminal. By directly sending the second natural language instruction to the user terminal, more information is retained in the second natural language instruction sent to the user terminal, avoiding information loss, facilitating the user terminal to perform more accurate intent recognition based on the second natural language instruction, and then determining the service to be linkage controlled, thereby improving the accuracy of vehicle linkage control. Moreover, since the second natural language instruction is sent directly to the user terminal, the efficiency of vehicle linkage control is improved.

[0072] When the second largest language model is used to determine that the first natural language instruction is a linkage control instruction for controlling the vehicle, the fourth target service and the fourth service control instruction corresponding to the first natural language instruction are determined according to the second largest language model and the preset instruction conversion correspondence, and the fourth target service is called to execute the fourth service control instruction to realize the control of the service in the vehicle. In the embodiment of the present application, since more information is retained in the first natural language instruction, information loss is reduced, so that the vehicle-machine end can perform more accurate intention recognition based on the first natural language instruction, and then determine the vehicle service to be linked and controlled, thereby improving the accuracy of the vehicle linkage control. Since the first natural language instruction is sent directly to the vehicle-machine end, the efficiency of the vehicle linkage control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the vehicle linkage control more flexible, without the user having to remember rigid control execution, thereby improving the efficiency of the vehicle linkage control. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0075] Figure 1 This is a schematic diagram of the linkage control between the user terminal and the vehicle terminal;

[0076] Figure 2 A flow chart of a vehicle linkage control method applied to a user terminal provided in an embodiment of the present application;

[0077] Figure 3 Another flow chart of a vehicle linkage control method applied to a user terminal provided in an embodiment of the present application;

[0078] Figure 4 A flow chart of a vehicle linkage control method applied to a vehicle-mounted terminal provided in an embodiment of the present application;

[0079] Figure 5 Another flow chart of a vehicle linkage control method applied to a vehicle-side provided in an embodiment of the present application;

[0080] Figure 6 A flow chart of another vehicle linkage control method applied to a vehicle-mounted terminal provided in an embodiment of the present application;

[0081] Figure 7 Another flow chart of another vehicle linkage control method applied to a vehicle-mounted terminal provided in an embodiment of the present application;

[0082] Figure 8 A flowchart of another vehicle linkage control method applied to a user terminal provided in an embodiment of the present application;

[0083] Figure 9 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0085] In the traditional C / S (Client / Server) architecture used by mobile phones and in-vehicle systems, interaction between the client and server relies on predefined APIs (Application Programming Interfaces). To enable data transmission and function calls, developers must fully expose the APIs and their corresponding input and output parameters to the client.

[0086] Since the API interface in the C / S architecture used by traditional mobile phones and car systems is like a "digital bridge" connecting the client and the server, each parameter is an "instruction signpost" on the bridge. Once the server modifies a function, even if it is just fine-tuning the data type of a parameter or the calling logic of the interface, it means that the structure of this "digital bridge" has changed. As the caller of the function, the client must make corresponding adjustments and adaptations simultaneously, otherwise it will not be able to communicate normally with the server, resulting in functional abnormalities or even system crashes. This kind of architectural feature that affects the entire system greatly limits the flexibility and scalability of the system, making the function iteration and maintenance costs remain high. To this end, the embodiments of the present application provide a vehicle linkage control method, device, user terminal and car terminal.

[0087] The embodiment of the present application provides a vehicle linkage control method, which is applied to a user terminal, such as Figure 1 As shown, the user terminal includes: a first linkage control application APP for controlling vehicle-mounted equipment, an intelligent agent (Agent) module and multiple services. The intelligent agent module includes a semantic adaptation layer, a user terminal agent unit, a context protocol (MCP) server and an A2A protocol unit. The user terminal agent unit includes a first large language model (LLM) and a context protocol (MCP) client. The user terminal can communicate with the A2A protocol server on the vehicle side through the A2A protocol client.

[0088] Among them, the semantic adaptation layer is responsible for conducting a comprehensive semantic analysis of the input data and accurately identifying the user's intentions and specific needs. By extracting keywords, commands or requests and annotating them, the semantic adaptation layer can convert this information into a natural language form that can be understood within the system. Finally, the processed data is transmitted to the large language model for in-depth identification and decision-making. As a key middleware in the system, the semantic adaptation layer assumes the important responsibilities of semantic analysis and data conversion. Through professional processing procedures and cutting-edge natural language processing technology, the semantic adaptation layer ensures that user intentions are accurately identified and converted, providing high-quality input data for the large language model. It enables users to achieve precise control of mobile phones and vehicles through natural language commands.

[0089] Large Language Model (LLM): The Large Language Model (LLM) can parse low-complexity natural language commands, and can parse low-complexity natural language commands based on contextual information. For example, if the system recognizes the command "Turn on assisted driving," it can identify the intention as coordinated vehicle control. The system sends the natural language command to the vehicle computer via the A2A protocol, where it can further recognize the natural language command using the large language model and ultimately complete vehicle control processing.

[0090] like Figure 2 As shown, the vehicle linkage control method may include the following steps:

[0091] Step S101, obtaining a first control instruction input by a user;

[0092] In the embodiment of the present application, the first control instruction is an instruction input by the user through the user terminal device. The user can input the first control instruction through voice, for example, through voice input "turn on the air conditioner in the car". The microphone is responsible for capturing the user's voice instruction. In order to improve the accuracy of voice recognition and the security of the system, noise reduction technology and voiceprint recognition technology can be used. First, noise reduction technology is used to remove background noise to ensure the clarity of the voice signal. Then, voiceprint recognition is used to verify the user's identity and improve the security of the interaction.

[0093] The first control instruction may also be input on a first function control displayed on the user terminal. For example, the user may input the first control instruction by clicking a button in the first linkage control application APP (clicking a button control corresponding to the air conditioning temperature in the first linkage control application APP), setting parameters in the first linkage control application APP, selecting a function in the first linkage control application APP, or entering text in the first linkage control application APP.

[0094] The first control instruction can also be input through body gestures, etc., for example: input "turn on the air conditioner in the car" through sign language, and the user terminal can obtain the first control instruction through a microphone, screen event monitoring, image acquisition equipment, etc.

[0095] Step S102, converting the first control instruction into a first natural language instruction;

[0096] For the first control instruction input by the user through a click operation, the first control instruction can be converted into a data string in a specific format to obtain a first natural language instruction; for the first control instruction input by the user through voice, the voice instruction can be converted into a standardized string through speech big model (STT) to text to obtain a first natural language instruction; for the first control instruction input by the user through body gestures, the body gestures can be matched with each group of body gestures in a preset body gesture set. If the match is successful, the first natural language instruction corresponding to the group of body gestures can be obtained.

[0097] In one embodiment of the present application, step S102 converts the first control instruction into a first natural language instruction, including: if the first control instruction is input by the user by operating a preset first function control, obtaining the first control operation parameter of the preset first function control being operated; converting the first control operation parameter into a first text instruction; identifying the first control intention information of the first text instruction; and converting the first text instruction into a first natural language instruction based on the first control intention information.

[0098] In another embodiment of the present application, step S102 converts the first control instruction into a first natural language instruction, including: if the first control instruction is input by the user through voice, converting the first control instruction into a first text instruction; extracting the control intention information of the first text instruction, and converting the first text instruction into a first natural language instruction.

[0099] Furthermore, after obtaining the first text instruction, keyword matching technology can be used to clean the text, such as filtering sensitive words, filtering dangerous words, etc., to increase the probability of natural language instructions being accurately recognized by the large language model.

[0100] Step S103, using a first large language model to determine whether the first natural language instruction is a linkage control instruction for controlling a vehicle;

[0101] In an embodiment of the present application, the first large language model can be a lightweight model, which can be pre-trained based on multiple user terminal services and multiple target control objects, so as to classify the target control objects of natural language instructions and match the service call requirements of the user terminal. The user terminal can also communicate with the cloud, which is equipped with a heavyweight large language model.

[0102] In this step, the first large language model or the 70B large model can be used to extract key parameters (such as temperature value / location / device name), and the complexity of the first natural language instruction can be identified through the key parameters, such as the number of verbs and nouns. If the number of verbs is greater than or equal to two, or if the number of nouns is greater than or equal to two, for example: "Play the songs played on the mobile phone on the car computer", or if it does not contain verbs or nouns, for example: "I am a little cold", then it can be determined that the complexity of the first natural language instruction is high; otherwise, it can be determined that the complexity of the first natural language instruction is low, for example: "Open the car window", "Play music, etc.", "Turn on assisted driving".

[0103] For a first natural language instruction with low complexity, in one embodiment of the present application, the first large language model can be used to perform intent recognition on the first natural language instruction to obtain a first target control object; if the first target control object is a vehicle, the first natural language instruction is determined to be an interlocking control instruction for controlling the vehicle; or, if the first target control object is a user terminal, the first natural language instruction is determined not to be an interlocking control instruction.

[0104] For a first natural language instruction with a high degree of complexity, in another embodiment of the present application, the intent of the first natural language instruction can be identified through a large language model in the cloud to obtain a first target control object; if the first target control object is a vehicle, the first natural language instruction is determined to be an interlocking control instruction for controlling the vehicle; or, if the first target control object is a user terminal, the first natural language instruction is determined not to be an interlocking control instruction.

[0105] After receiving the first natural language instruction, the large language model in the cloud can perform text correction, such as correcting "open a position" to "open a window", etc. It can also filter sensitive words and use the large language model to perform deep reasoning on the first natural language instruction.

[0106] In an embodiment of the present application, a large language model can be used to identify the user intention in the first natural language instruction to determine whether the user intention is to control the user terminal or the vehicle. For example, if the first natural language instruction is "play music", it is determined that the first natural language instruction is not a linkage control instruction; if the first natural language instruction is "open the car window", it is determined that the first natural language instruction is a linkage control instruction for controlling the vehicle.

[0107] When using a large language model to identify the user intention in the first natural language instruction, the context information of the first natural language instruction can also be obtained, that is, several first natural language instructions whose historical records were generated at the closest time to the current time, and the user intention in the first natural language instruction can be identified in combination with the context information. In actual applications, it can be determined whether the current first natural language instruction is related to the first natural language instruction of the previous round. If so, the first natural language instruction of the previous round can be combined to understand the first natural language instruction of this round, such as: the user says "open the car window" → the first natural language instruction of the previous round is "open the car window 50%", the user says "open it wider" → the current first natural language instruction is "open it wider" → compare the historical records → generate an instruction to open the window 80%.

[0108] Step S104: If the first natural language instruction is a linkage control instruction, the first natural language instruction is sent to the vehicle terminal.

[0109] In this step, in order to enable the vehicle to respond to the first control instruction input by the user, the first natural language instruction can be encapsulated through the A2A protocol used for communication and collaboration between intelligent entities, and the encapsulated instruction can be sent to the vehicle computer end, so that the vehicle computer end can call the vehicle service in the vehicle based on the first natural language instruction.

[0110] The embodiment of the present application converts the first control instruction input by the user into a first natural language instruction. When the first large language model is used to determine that the first natural language instruction is an interlocking control instruction for controlling the vehicle, the first natural language instruction is sent to the vehicle-computer end. By sending the first natural language instruction directly to the vehicle-computer end, more information is retained in the first natural language instruction sent to the vehicle-computer end, avoiding information loss, facilitating the vehicle-computer end to perform more accurate intent recognition based on the first natural language instruction, and then determine the vehicle service to be interlocked and controlled, thereby improving the accuracy of vehicle interlocking control. Moreover, since the first natural language instruction is sent directly to the vehicle-computer end, the efficiency of vehicle interlocking control is improved.

[0111] In another embodiment of the present application, Figure 3 As shown, the method further includes:

[0112] Step S201: using the first large language model to determine whether the first natural language instruction contains a first linkage reference object;

[0113] Since the user status, user terminal status, vehicle driving status, and the environment inside and outside the vehicle may affect the linkage control of the vehicle, in the embodiment of the present application, the first linkage reference object may refer to the user, user terminal, vehicle or environment.

[0114] Step S202: Acquire first linkage reference information corresponding to the first linkage reference object;

[0115] In an embodiment of the present application, the first linkage reference information may include the current user terminal status, the current vehicle status (such as current gear, vehicle speed, etc.), the internal and external environment of the vehicle (such as temperature, rainy days, etc.) and the user status (such as physiological state, psychological state, etc.).

[0116] Step S203: Send the first linkage reference information while sending the first natural language instruction to the vehicle terminal.

[0117] In this step, the first linkage reference information can be sent at the same time as the first natural language instruction is sent, so that the vehicle computer can simultaneously determine the vehicle service to be linked and controlled based on the first natural language instruction and the first linkage reference information, such as "playing the songs played on the mobile phone on the vehicle computer", so that the linkage control of the vehicle is more in line with actual application scenarios and user needs.

[0118] In another embodiment of the present application, the method further includes:

[0119] If the first natural language instruction is not a linkage control instruction, determine the first target service and the first service control instruction corresponding to the first natural language instruction according to the first large language model and the preset instruction conversion correspondence; call the first target service to execute the first service control instruction.

[0120] When using the first language model to determine the user's intent to control the user terminal itself via the first control instruction, the first language model is used to perform parameter refinement processing on the first natural language instruction (i.e., normalizing the parameters, such as modifying half to 50% and the maximum to 100%) and unit standardization processing (i.e., converting them to standard user terminal control units, such as temperature → API value). Based on the processed first natural language instruction, the service to be controlled and the service action are then determined. The MCP server can determine the first target service and the first service control instruction corresponding to the first natural language instruction based on the instruction conversion correspondence, and call the first target service to execute the first service control instruction, thereby achieving control of the service in the user terminal. For example, if the instruction only involves mobile phone functions, such as recognizing "turn off mobile phone Bluetooth," data is sent to the mobile phone's MCP server to call the mobile phone API. The API can be determined through the service, and the corresponding API capability is called to execute the action, thereby completing the control of the mobile phone, and the execution result is fed back to the user.

[0121] In another embodiment of the present application, a vehicle linkage control method is also provided, which is applied to the vehicle terminal, such as Figure 1 As shown, the vehicle-side includes: a second linkage control application APP for vehicle control and some mobile phone control (such as navigation, multimedia, smart assistant, assisted driving, etc.), an intelligent agent (Agent) module and multiple services. The intelligent agent module includes a semantic adaptation layer, a vehicle-side agent unit, a context protocol (MCP) server and an A2A protocol unit. The vehicle-side agent unit includes a second largest language model (LLM) and a context protocol (MCP) client. The vehicle-side can communicate with the A2A protocol client of the user terminal through the A2A protocol server.

[0122] Among them, the semantic adaptation layer is responsible for conducting a comprehensive semantic analysis of the input data and accurately identifying the user's intentions and specific needs. By extracting keywords, commands or requests and annotating them, the semantic adaptation layer can convert this information into a natural language form that can be understood within the system. Finally, the processed data is transmitted to the large language model for in-depth identification and decision-making. As a key middleware in the system, the semantic adaptation layer assumes the important responsibilities of semantic analysis and data conversion. Through professional processing procedures and cutting-edge natural language processing technology, the semantic adaptation layer ensures that user intentions are accurately identified and converted, providing high-quality input data for the large language model. It enables users to achieve precise control of mobile phones and vehicles through natural language commands.

[0123] The second language model (LLM) parses the user's natural language commands, identifying key information and intent. Based on the understood intent, it generates a corresponding text response. Its reasoning and prediction capabilities enable it to make decisions based on contextual information and interpret commands. The model is trained on vehicle-user interactions, improving its ability to analyze content and make intelligent decisions. Ultimately, it outputs text responses that meet expectations. This is key to improving the user experience and a crucial technical support for achieving automotive intelligence. For example, it can recognize the command "Turn on assisted driving," or, in a more complex scenario, if the user says "I'm a little cold," the system can recognize the intended meaning as "Raise the air conditioning temperature."

[0124] like Figure 4 As shown, the vehicle linkage control method may include the following steps:

[0125] Step S301: receiving a first natural language instruction from a user terminal;

[0126] Step S302: using the second largest language model to determine whether the first natural language instruction is a linkage control instruction for controlling a vehicle;

[0127] In an embodiment of the present application, the second largest language model can be a lightweight model, which can be pre-trained based on multiple vehicle services and multiple target control objects, so as to classify the target control objects of natural language instructions and match them with vehicle linkage control scenarios. The vehicle-mounted terminal can also communicate with the cloud, which is equipped with a heavyweight large language model.

[0128] In this step, the second largest language model or the 70B large model can be used to first extract key parameters (such as temperature value / location / device name), and the complexity of the first natural language instruction can be identified through the key parameters, such as the number of verbs and nouns. If the number of verbs is greater than or equal to two, or if the number of nouns is greater than or equal to two, for example: "Play the songs played on the mobile phone on the car computer", or if it does not contain verbs or nouns, for example: "I am a little cold", then it can be determined that the complexity of the first natural language instruction is high; otherwise, it can be determined that the complexity of the first natural language instruction is low, for example: "Open the car window", "Play music, etc.", "Turn on assisted driving".

[0129] For a first natural language instruction with low complexity, in one embodiment of the present application, the second largest language model can be used to identify the intent of the first natural language instruction to obtain a fourth target control object; if the fourth target control object is a vehicle, the first natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0130] For a first natural language instruction with a high degree of complexity, in another embodiment of the present application, the intent of the first natural language instruction can be identified through a large language model in the cloud to obtain a fourth target control object; if the fourth target control object is a vehicle, the first natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0131] After receiving the first natural language instruction, the large language model in the cloud can perform text correction, such as correcting "open a position" to "open a window", etc. It can also filter sensitive words and use the large language model to perform deep reasoning on the first natural language instruction.

[0132] In an embodiment of the present application, a large language model can be used to identify the user intention in the first natural language instruction to determine whether the user intention is to control the user terminal or the vehicle. If the first natural language instruction is "open the car window", the first natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0133] Step S303: If the first natural language instruction is a linkage control instruction, determine a fourth target service and a fourth service control instruction corresponding to the first natural language instruction based on the second language model and a preset instruction conversion correspondence relationship;

[0134] In this step, the second largest language model can be used to perform parameter refinement processing on the first natural language instruction (i.e., normalizing the parameters, such as modifying half to 50% and the maximum modification to 100%) and unit standardization processing (i.e., converting them into standard vehicle control units, such as temperature → CAN value). Based on the processed first natural language instruction, the service to be controlled and the service action are determined. The MCP server can determine the fourth target service and fourth service control instruction corresponding to the first natural language instruction based on the instruction conversion correspondence.

[0135] Step S304: calling the fourth target service to execute the fourth service control instruction.

[0136] The fourth target service includes: seat service, window service, air conditioning service and door service, etc. The fourth service control instruction includes: seat service control instruction, window service control instruction, air conditioning service control instruction and door service control instruction, etc.

[0137] The seat service utilizes a service adaptation protocol to invoke its related service functions. This includes the seat service module, which initializes and performs self-tests on the hardware to ensure proper functioning of all components. It receives user requests for seat adjustment. It interprets the user's instructions and determines the parameters that require adjustment, such as seat position and heating temperature. Based on the interpreted instructions, it controls the motor and other actuators to adjust the seat state. Feedback devices such as position sensors monitor the adjustment results in real time to ensure the desired state is achieved. Detecting hardware failures or abnormal instructions triggers appropriate error handling mechanisms, such as pausing operation or issuing an alarm.

[0138] Window Service: Through the service adaptation protocol, relevant service functions can be invoked and, based on received instructions, corresponding hardware opening and closing operations can be executed. Window status is monitored in real time and fed back to the vehicle computer system. Hardware failures or abnormal conditions can be detected and responded to promptly, ensuring safe window operation.

[0139] Air conditioning service: Through the service adaptation protocol, its related service functions can be called to control the operating status of temperature sensors, humidity sensors, and fans. Supports setting functions such as temperature, wind speed, wind direction, and on / off. Real-time data such as temperature, humidity, and PM2.5 are collected through in-vehicle sensors. External environmental data of the vehicle is obtained through external environmental sensors (such as the external temperature sensor). Based on user instructions and environmental data, the air conditioning parameters that need to be adjusted (such as cooling / heating mode, wind speed, wind direction, etc.) are calculated. The air conditioning compressor, heating system, fan, and blower are controlled to adjust the temperature inside the vehicle. When a sensor failure, insufficient refrigerant, or hardware abnormality is detected, an alarm is triggered or operation is suspended.

[0140] Door Service: Its related service functions can be invoked through the service adaptation protocol. When the vehicle starts, the door service module initializes and self-checks the hardware to ensure the proper functioning of the electric motor, sensors, and locking mechanisms. It then issues door control commands. User commands are transmitted to the control module, which interprets the specific control requirements (such as opening, closing, and locking the door). Based on the interpretation, actuators such as the electric motor and door locks are controlled to perform the corresponding operations. Feedback devices such as door position sensors monitor the operation results in real time, updating door status information. If an anomaly is detected (such as a door not fully closed), an alarm is triggered or corrective action is automatically taken.

[0141] Seat service control instructions: responsible for driving the adjustment of the seat in the front, back, up, down, tilt and other directions.

[0142] Monitors seat position and angle in real time to ensure accurate adjustment. Provides massage functionality through electric vibration or airbag adjustment. Receives user commands and displays seat status.

[0143] Window service control commands: Used to activate and close windows. Real-time window position detection ensures accurate control. Obstacle detection during closing prevents pinching. Communicates with the vehicle's computer system, receives commands, and controls the corresponding hardware.

[0144] Air conditioning service control: This component is responsible for the core refrigeration cycle, regulating the vehicle interior temperature through refrigerant. It uses engine coolant or an electric heater to provide heating to the vehicle interior. It regulates air flow to ensure uniform temperature and air velocity distribution. It includes temperature, humidity, and PM2.5 sensors for real-time monitoring of the interior environment. It adjusts the direction and air velocity of the air conditioning vents. It is used to display the air conditioning status and receive user commands.

[0145] Door Service Control Commands: Used to activate and close the doors. Detects door opening and closing. Controls door lock status. Users control the doors via physical buttons. Connects the Door Service to other vehicle control modules. Processes Door Service commands and data.

[0146] When the embodiment of the present application uses the second largest language model to determine that the first natural language instruction is an interlocking control instruction for controlling the vehicle, the fourth target service and the fourth service control instruction corresponding to the first natural language instruction are determined according to the second largest language model and the preset instruction conversion correspondence, and the fourth target service is called to execute the fourth service control instruction to realize the control of the service in the vehicle. Since the embodiment of the present application retains more information in the first natural language instruction and reduces information loss, the vehicle-machine end can perform more accurate intent recognition based on the first natural language instruction, and then determine the vehicle service to be interlocked and controlled, thereby improving the accuracy of vehicle interlocking control. Since the first natural language instruction is sent directly to the vehicle-machine end, the efficiency of vehicle interlocking control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the interlocking control of the vehicle more flexible, without the user having to remember rigid control execution, thereby improving the efficiency of vehicle interlocking control.

[0147] In another embodiment of the present application, Figure 5 As shown, the method further includes:

[0148] Step S401, obtaining vehicle status information and / or vehicle surrounding environment information;

[0149] In the embodiment of the present application, the vehicle status information may exemplarily refer to information such as the current gear position and vehicle speed, and the vehicle surrounding environment information may exemplarily refer to information such as the external ambient temperature and external obstacles.

[0150] Step S402: If the vehicle state information and / or vehicle surrounding environment information and the fourth service control instruction meet a preset prohibition linkage rule, prohibiting the fourth target service from being called to execute the fourth service control instruction.

[0151] In the embodiment of the present application, linkage prohibition rules can be pre-set, such as prohibiting door opening when the vehicle is not in P gear, prohibiting window opening when the vehicle speed exceeds 30, etc.

[0152] In this step, the vehicle status information and / or vehicle surrounding environment information and the fourth service control instruction can be matched with the prohibition linkage rule respectively. If the match is successful, the fourth target service is prohibited from being called to execute the fourth service control instruction.

[0153] Alternatively, in step S403, if the vehicle status information and / or vehicle surrounding environment information does not comply with a preset prohibition linkage rule with the fourth service control instruction, the fourth service control instruction is adjusted according to the vehicle status information and / or vehicle surrounding environment information.

[0154] In this step, if the vehicle status information and / or the vehicle surrounding environment information and the fourth service control instruction fail to match the prohibition linkage rule respectively, the fourth service control instruction can be adjusted in combination with the vehicle status information and / or the vehicle surrounding environment information, such as: if the current outside temperature is -20 degrees Celsius, the fourth service control instruction "open the window 100%" is adjusted to "open the window 50%".

[0155] The embodiments of the present application can prohibit the target service from executing service control instructions or adjust the service control instructions executed by the target service based on the vehicle status, that is, the vehicle's surrounding environment, so as to make the control of vehicle services more adapted to actual application scenarios and facilitate providing users with a more comfortable and safe vehicle linkage control experience.

[0156] In another embodiment of the present application, a vehicle linkage control method is also provided, which is applied to the vehicle terminal. The internal configuration of the vehicle terminal is the same as that of the above embodiment and will not be repeated here. Figure 6 As shown, the method includes:

[0157] Step S501, obtaining a second control instruction input by a user;

[0158] In the embodiment of the present application, the second control instruction is an instruction input by the user through the vehicle computer. The user can input the second control instruction through voice, for example, through voice input "turn on the air conditioner in the car". The microphone is responsible for capturing the user's voice instruction. In order to improve the accuracy of voice recognition and the security of the system, noise reduction technology and voiceprint recognition technology can be used. First, noise reduction technology is used to remove background noise to ensure the clarity of the voice signal. Then, voiceprint recognition is used to verify the user's identity and improve the security of the interaction.

[0159] The second control instruction may also be input on a second function control displayed on the vehicle terminal. For example, the user may input the first control instruction by clicking a button in the second linkage control application APP (clicking a button control corresponding to the air conditioning temperature in the second linkage control application APP), setting parameters in the second linkage control application APP, selecting a function in the second linkage control application APP, or entering text in the second linkage control application APP.

[0160] The second control instruction can also be input through body gestures, etc. For example, by inputting "turn on the air conditioner in the car" through sign language, the vehicle computer can obtain the first control instruction through a microphone, screen event monitoring, image acquisition equipment, etc.

[0161] Step S502: convert the second control instruction into a second natural language instruction;

[0162] For the second control instruction input by the user through a click operation, the second control instruction can be converted into a data string in a specific format to obtain a second natural language instruction; for the second control instruction input by the user through voice, the voice instruction can be converted into a standardized string through speech big model (STT) to text to obtain a second natural language instruction; for the first control instruction input by the user through body gestures, the body gestures can be matched with each group of body gestures in a preset body gesture set. If the match is successful, the first natural language instruction corresponding to the group of body gestures can be obtained.

[0163] In one embodiment of the present application, step S502 converts the second control instruction into a second natural language instruction, including: if the second control instruction is input by the user on a preset second function control, obtaining the second control operation parameter corresponding to the second control instruction in the preset second function control; converting the second control operation parameter into a second text instruction; identifying the second control intention information of the second text instruction; and converting the second text instruction into a second natural language instruction based on the second control intention information.

[0164] In one embodiment of the present application, step S502 converts the second control instruction into a second natural language instruction, including: if the second control instruction is input by the user through voice, converting the second control instruction into a second text instruction; extracting the control intention information of the second text instruction, and converting the second text instruction into a second natural language instruction.

[0165] Furthermore, after obtaining the second text instruction, keyword matching technology can be used to clean the text, such as filtering sensitive words and dangerous words, to increase the probability that the natural language instruction is accurately recognized by the large language model.

[0166] Step S503: using the second largest language model to determine whether the second natural language instruction is a linkage control instruction for controlling the user terminal;

[0167] In this step, the second largest language model or the 70B large model can be used to first extract key parameters (such as temperature value / location / device name), and the complexity of the second natural language instruction can be identified through the key parameters, such as the number of verbs and nouns. If the number of verbs is greater than or equal to two, or if the number of nouns is greater than or equal to two, for example: "Play the songs played on the mobile phone on the car computer", or if it does not contain verbs or nouns, for example: "I am a little cold", then it can be determined that the complexity of the second natural language instruction is high; otherwise, it can be determined that the complexity of the second natural language instruction is low, for example: "Open the car window", "Play music, etc.", "Turn on assisted driving".

[0168] For a second natural language instruction with low complexity, in one embodiment of the present application, the second large language model can be used to perform intent recognition on the second natural language instruction to obtain a second target control object; if the second target control object is a user terminal, the second natural language instruction is determined to be a linkage control instruction for controlling the user terminal; or, if the second target control object is a vehicle, the second natural language instruction is determined not to be a linkage control instruction.

[0169] For a second natural language instruction with a high degree of complexity, in another embodiment of the present application, the second natural language instruction can be used to identify the intent of the second natural language instruction through a large language model in the cloud to obtain a second target control object; if the second target control object is a user terminal, the second natural language instruction is determined to be a linkage control instruction for controlling the user terminal; or, if the second target control object is a vehicle, the second natural language instruction is determined not to be a linkage control instruction.

[0170] After receiving the second natural language instruction, the large language model in the cloud can perform text correction, such as correcting "open a position" to "open a window", etc. It can also filter sensitive words and use the large language model to perform deep reasoning on the second natural language instruction.

[0171] In an embodiment of the present application, a large language model can be used to identify the user intention in the second natural language instruction to determine whether the user intention is to control the user terminal or the vehicle. For example, if the second natural language instruction is "play music on the mobile phone", the second natural language instruction is determined to be a linkage control instruction; if the second natural language instruction is "open the car window", the first natural language instruction is determined not to be a linkage control instruction.

[0172] When using a large language model to identify the user intention in the second natural language instruction, the context information of the second natural language instruction can also be obtained, that is, several second natural language instructions whose historical records are generated at the closest time to the current time, and the user intention in the second natural language instruction can be identified in combination with the context information. In actual applications, it can be determined whether the current second natural language instruction is related to the second natural language instruction of the previous round. If so, the second natural language instruction of the current round can be understood in combination with the second natural language instruction of the previous round, such as: the user says "open the car window" → the second natural language instruction of the previous round is "open the car window 50%", the user says "open it wider" → the current second natural language instruction is "open it wider" → compare the historical records → generate an instruction to open the window 80%.

[0173] Step S504: If the second natural language instruction is a linkage control instruction, the second natural language instruction is sent to the user terminal.

[0174] In this step, in order to enable the user terminal to respond to the second control instruction input by the user, the second natural language instruction can be encapsulated through the A2A protocol used for communication and collaboration between intelligent agents, and the encapsulated instruction can be sent to the user terminal so that the user terminal calls the service in the user terminal based on the second natural language instruction.

[0175] The embodiment of the present application converts the second control instruction input by the user into a second natural language instruction. When the second natural language instruction is determined by using the second largest language model to be a linkage control instruction for controlling the user terminal, the second natural language instruction is sent to the user terminal. By sending the second natural language instruction directly to the user terminal, more information is retained in the second natural language instruction sent to the user terminal, avoiding information loss, facilitating the user terminal to perform more accurate intent recognition based on the second natural language instruction, and then determine the service to be linkage controlled, thereby improving the accuracy of vehicle linkage control. Moreover, since the second natural language instruction is sent directly to the user terminal, the efficiency of vehicle linkage control is improved.

[0176] In another embodiment of the present application, Figure 7 As shown, the method further includes:

[0177] Step S601: using the second language model to determine whether the second natural language instruction contains a second linkage reference object;

[0178] Since the user status, vehicle driving status, and the environment inside and outside the vehicle may affect the linkage control of the vehicle, in the embodiment of the present application, the second linkage reference object may refer to the user, vehicle, or environment.

[0179] Step S602: Acquire second linkage reference information corresponding to the second linkage reference object;

[0180] In this embodiment of the present application, the second linkage reference information may include the current vehicle status (such as current gear, vehicle speed, etc.), the internal and external environment of the vehicle (such as temperature, rainy days, etc.) and the user status (such as physiological state, psychological state, etc.).

[0181] Step S603: Send the second linkage reference information while sending the second natural language instruction to the user terminal.

[0182] In this step, the second linkage reference information can be sent at the same time as the second natural language instruction, so that the user terminal can determine the service to be linked and controlled based on the second natural language instruction and the second linkage reference information at the same time, so that the linkage control of the vehicle is more in line with actual application scenarios and user needs.

[0183] In another embodiment of the present application, the method also includes: if the second natural language instruction is not a linkage control instruction for controlling the user terminal, determining the third target service and third service control instruction corresponding to the second natural language instruction based on the second large language model and the preset instruction conversion correspondence; calling the third target service to execute the third service control instruction.

[0184] When using the second language model to determine the user's intent to control the vehicle itself via the second control command, the second language model is used to refine the parameters of the second natural language command (i.e., normalize the parameters, such as modifying half to 50% and modifying the maximum to 100%) and standardize the units (i.e., convert them to standard vehicle control units, such as temperature → CAN value). Based on the processed second natural language command, the MCP server can then determine the service to be controlled and the service action. Based on the command conversion relationship, the MCP server can determine the third target service and third service control command corresponding to the second natural language command, and invoke the third target service to execute the third service control command, thereby controlling the vehicle service on the vehicle. For example, if the command only involves vehicle functions, such as recognizing "turn off vehicle Bluetooth," the MCP server sending data to the vehicle performs physical address mapping via the control command, outputs the CAN signal control command, and then completes vehicle control via the CAN signal. The success or failure of the execution is announced via text-to-text transmission (TTS). If the execution fails, a TTS message explaining the reason for the failure to control the vehicle is provided.

[0185] In an embodiment of the present application, the third target service includes: seat service, window service, air conditioning service and door service, etc., and the third service control instruction includes: seat service control instruction, window service control instruction, air conditioning service control instruction and door service control instruction, etc.

[0186] Before calling the third target service to execute the third service control instruction, vehicle status information and / or vehicle surrounding environment information may also be obtained; if the vehicle status information and / or vehicle surrounding environment information and the service control instruction to be executed meet a preset prohibition linkage rule, calling the target service to execute the service control instruction is prohibited. Alternatively, if the vehicle status information and / or vehicle surrounding environment information and the service control instruction to be executed do not meet the preset prohibition linkage rule, the service control instruction is adjusted based on the vehicle status information and / or vehicle surrounding environment information. For details, please refer to the description of steps S401 to S403, which will not be repeated here.

[0187] In another embodiment of the present application, a vehicle linkage control method is also provided, which is applied to a user terminal, such as Figure 8 As shown, the method includes:

[0188] Step S701, receiving a second natural language instruction from the vehicle terminal;

[0189] Step S702: Using the first large language model, determining whether the second natural language instruction is a linkage control instruction for controlling the user terminal;

[0190] In this step, the first large language model or the 70B large model can be used to extract key parameters (such as temperature value / location / device name), and the complexity of the second natural language instruction can be identified through the key parameters, such as the number of verbs and nouns. If the number of verbs is greater than or equal to two, or if the number of nouns is greater than or equal to two, for example: "Play the songs played on the mobile phone on the car computer", or if it does not contain verbs or nouns, for example: "I am a little cold", then it can be determined that the complexity of the second natural language instruction is high; otherwise, it can be determined that the complexity of the second natural language instruction is low, for example: "Open the car window", "Play music, etc.", "Turn on assisted driving".

[0191] For a second natural language instruction with low complexity, in one embodiment of the present application, the second large language model can be used to identify the intent of the second natural language instruction to obtain a third target control object; if the third target control object is a vehicle, the first natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0192] For a first natural language instruction with a high degree of complexity, in another embodiment of the present application, the intent of the first natural language instruction can be identified through a large language model in the cloud to obtain a third target control object; if the third target control object is a vehicle, the first natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0193] After receiving the second natural language instruction, the large language model in the cloud can perform text correction, such as correcting "open a position" to "open a window", etc. It can also filter sensitive words and use the large language model to perform deep reasoning on the second natural language instruction.

[0194] In an embodiment of the present application, a large language model can be used to identify the user intention in the second natural language instruction to determine whether the user intention is to control the user terminal or the vehicle. For example, if the second natural language instruction is "open the car window", the second natural language instruction is determined to be a linkage control instruction for controlling the vehicle.

[0195] Step S703: If the second natural language instruction is a linkage control instruction, determine a second target service and a second service control instruction corresponding to the second natural language instruction based on the first language model and a preset instruction conversion correspondence;

[0196] In this step, the first large language model can be used to perform parameter refinement processing on the second natural language instruction (i.e., normalizing the parameters, such as modifying half to 50% and the maximum modification to 100%) and unit standardization processing (i.e., converting them into standard user terminal control units, such as temperature → API value). Based on the processed second natural language instruction, the service to be controlled and the service action are determined. The MCP server can determine the second target service and second service control instruction corresponding to the second natural language instruction based on the instruction conversion correspondence.

[0197] Step S704: calling the second target service to execute the second service control instruction.

[0198] When the embodiment of the present application uses the first large language model to determine that the second natural language instruction is an interlocking control instruction for controlling the vehicle, the second target service and the second service control instruction corresponding to the second natural language instruction are determined according to the first large language model and the preset instruction conversion correspondence, and the second target service is called to execute the second service control instruction to realize the control of the service in the vehicle. Since the embodiment of the present application retains more information in the second natural language instruction and reduces information loss, the vehicle-machine end can perform more accurate intent recognition based on the second natural language instruction, and then determine the vehicle service to be interlocked and controlled, thereby improving the accuracy of vehicle interlocking control. Since the second natural language instruction is sent directly to the vehicle-machine end, the efficiency of vehicle interlocking control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the interlocking control of the vehicle more flexible, without the user having to remember rigid control execution, thereby improving the efficiency of vehicle interlocking control.

[0199] In another embodiment of the present application, a user terminal is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0200] Memory for storing computer programs;

[0201] The processor is configured to implement any of the aforementioned vehicle linkage control methods applied to a user terminal when executing a program stored in the memory.

[0202] In the user terminal provided by an embodiment of the present invention, the processor converts the first control instruction input by the user into a first natural language instruction by executing the program stored in the memory. When the first natural language instruction is determined to be an interlocking control instruction for controlling the vehicle using the first large language model, the first natural language instruction is sent to the vehicle-machine end. By directly sending the first natural language instruction to the vehicle-machine end, more information is retained in the first natural language instruction sent to the vehicle-machine end, avoiding information loss, facilitating the vehicle-machine end to perform more accurate intent recognition based on the first natural language instruction, and then determine the vehicle service to be interlocked and controlled, thereby improving the accuracy of vehicle interlocking control. Moreover, since the first natural language instruction is sent directly to the vehicle-machine end, the efficiency of vehicle interlocking control is improved.

[0203] Also, when the first large language model is used to determine that the second natural language instruction is a linkage control instruction for controlling the vehicle, the second target service and the second service control instruction corresponding to the second natural language instruction are determined according to the first large language model and the preset instruction conversion correspondence, and the second target service is called to execute the second service control instruction to realize the control of the service in the vehicle. In the embodiment of the present application, more information is retained in the second natural language instruction, which reduces information loss, so that the vehicle-side can perform more accurate intention recognition based on the second natural language instruction, and then determine the vehicle service to be linked and controlled, thereby improving the accuracy of the vehicle linkage control. Since the second natural language instruction is sent directly to the vehicle-side, the efficiency of the vehicle linkage control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the vehicle linkage control more flexible, without the need for users to remember rigid control execution, thereby improving the efficiency of the vehicle linkage control.

[0204] In another embodiment of the present application, a vehicle terminal is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0205] Memory for storing computer programs;

[0206] The processor is used to implement any of the aforementioned vehicle linkage control methods applied to the vehicle-mounted terminal when executing the program stored in the memory.

[0207] In the vehicle-mounted terminal provided by an embodiment of the present invention, the processor converts the second control instruction input by the user into a second natural language instruction by executing the program stored in the memory. When the second natural language instruction is determined to be a linkage control instruction for controlling the user terminal by using the second largest language model, the second natural language instruction is sent to the user terminal. By directly sending the second natural language instruction to the user terminal, more information is retained in the second natural language instruction sent to the user terminal, avoiding information loss, facilitating the user terminal to perform more accurate intent recognition based on the second natural language instruction, and then determining the service to be linkage controlled, thereby improving the accuracy of vehicle linkage control. Moreover, since the second natural language instruction is sent directly to the user terminal, the efficiency of vehicle linkage control is improved.

[0208] When the second largest language model is used to determine that the first natural language instruction is a linkage control instruction for controlling the vehicle, the fourth target service and the fourth service control instruction corresponding to the first natural language instruction are determined according to the second largest language model and the preset instruction conversion correspondence, and the fourth target service is called to execute the fourth service control instruction to realize the control of the service in the vehicle. In the embodiment of the present application, since more information is retained in the first natural language instruction, information loss is reduced, so that the vehicle-machine end can perform more accurate intention recognition based on the first natural language instruction, and then determine the vehicle service to be linked and controlled, thereby improving the accuracy of the vehicle linkage control. Since the first natural language instruction is sent directly to the vehicle-machine end, the efficiency of the vehicle linkage control is improved. Since a large language model is used to identify the intention of the natural language instruction, more diverse voice commands can be recognized, making the vehicle linkage control more flexible, without the user having to remember rigid control execution, thereby improving the efficiency of the vehicle linkage control.

[0209] The communication bus 1140 mentioned above for the user terminal and the vehicle terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0210] The communication interface 1120 is used for communication between the electronic device and other devices.

[0211] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0212] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0213] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0214] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A vehicle linkage control method, characterized in that: The method comprises: The user terminal obtains a first control instruction input by a user, the user terminal including: a first linkage control application for controlling an on-board device, an intelligent agent module, and multiple services, the intelligent agent module including a semantic adaptation layer, a user terminal agent unit, a context protocol (MCP) server, and an A2A protocol unit, the user terminal agent unit including a first large language model (LLM) and a context protocol (MCP) client, and the user terminal communicating with the A2A protocol server on the vehicle side via the A2A protocol client; The user terminal converts the first control instruction into a first natural language instruction; The user terminal determines whether the first natural language instruction is a linkage control instruction for controlling a vehicle using a first large language model, wherein the first large language model is used to extract the number of verbs and nouns in the first natural language instruction; if the complexity of the first natural language instruction is determined to be low based on the number of verbs and nouns, parsing the first natural language instruction with low complexity based on context information of the first natural language instruction; If the first natural language instruction is a linkage control instruction, the user terminal sends the first natural language instruction to the vehicle terminal, where the first natural language instruction is encapsulated and sent using the A2A protocol, a protocol for communication and collaboration between agents; The vehicle-mounted terminal receives a first natural language instruction from a user terminal, where the first natural language instruction is encapsulated and sent using the A2A protocol, a protocol for communication and collaboration between intelligent agents. The vehicle-mounted terminal includes: a second linkage control application for vehicle control and some mobile phone control, an intelligent agent (Agent) module, and multiple services. The intelligent agent module includes a semantic adaptation layer, a vehicle-mounted terminal agent unit, a context protocol (MCP) server, and an A2A protocol unit. The vehicle-mounted terminal agent unit includes a second large language model (LLM) and a context protocol (MCP) client. The vehicle-mounted terminal communicates with the A2A protocol client of the user terminal via the A2A protocol server. The vehicle computer determines, using a second language model, whether the first natural language instruction is a linkage control instruction for controlling the vehicle. The second language model is used to extract the number of verbs and nouns in the first natural language instruction. If the complexity of the first natural language instruction is determined to be low based on the number of verbs and nouns, the first natural language instruction with low complexity is parsed based on context information of the first natural language instruction. If the first natural language instruction is a linkage control instruction, the vehicle computer determines a fourth target service and a fourth service control instruction corresponding to the first natural language instruction based on the second language model and a preset instruction conversion correspondence; The vehicle terminal calls the fourth target service to execute the fourth service control instruction.

2. The vehicle linkage control method according to claim 1, characterized in that: The user terminal converts the first control instruction into a first natural language instruction, including: If the first control instruction is input by the user by operating a preset first function control, the user terminal obtains a first control operation parameter of the preset first function control; Converting the first control operation parameter into a first text instruction; identifying first control intention information of the first text instruction; The first text instruction is converted into a first natural language instruction based on the first control intention information.

3. The vehicle linkage control method according to claim 1, characterized in that: The user terminal converts the first control instruction into a first natural language instruction, including: If the first control instruction is input by the user through voice, the user terminal converts the first control instruction into a first text instruction; The control intention information of the first text instruction is extracted, and the first text instruction is converted into a first natural language instruction.

4. The vehicle linkage control method according to claim 1, characterized in that: The user terminal determines, using the first language model, whether the first natural language instruction is a linkage control instruction for controlling the vehicle, including: The user terminal uses the first large language model to perform intent recognition on the first natural language instruction to obtain a first target control object; If the first target control object is a vehicle, determining that the first natural language instruction is a linkage control instruction for controlling the vehicle; Alternatively, if the first target control object is a user terminal, it is determined that the first natural language instruction is not a linkage control instruction.

5. The vehicle linkage control method according to claim 1, characterized in that: The method further comprises: The user terminal determines whether the first natural language instruction includes a first linkage reference object by using the first large language model; Acquire first linkage reference information corresponding to the first linkage reference object; When sending the first natural language instruction to the vehicle computer, the first linkage reference information is sent.

6. The vehicle linkage control method according to claim 1, characterized in that: The method further comprises: The vehicle computer obtains vehicle status information and / or vehicle surrounding environment information; If the vehicle state information and / or vehicle surrounding environment information and the fourth service control instruction meet a preset prohibition linkage rule, prohibiting the calling of the fourth target service to execute the fourth service control instruction; Alternatively, if the vehicle status information and / or vehicle surrounding environment information does not comply with a preset prohibition linkage rule with the fourth service control instruction, the fourth service control instruction is adjusted according to the vehicle status information and / or vehicle surrounding environment information.

7. A vehicle linkage control method, characterized in that: The method comprises: The vehicle-side computer receives a second control command input by the user. The vehicle-side computer includes: a second linkage control application for vehicle control and some mobile phone control, an intelligent agent module, and multiple services. The intelligent agent module includes a semantic adaptation layer, a vehicle-side agent unit, a context-sensitive protocol (MCP) server, and an A2A protocol unit. The vehicle-side agent unit includes a second large language model (LLM) and a context-sensitive protocol (MCP) client. The vehicle-side computer communicates with the A2A protocol client of the user terminal through the A2A protocol server. The vehicle computer converts the second control instruction into a second natural language instruction; The vehicle computer determines, using a second large language model, whether the second natural language instruction is a linkage control instruction for controlling the user terminal. The second large language model is used to extract the number of verbs and nouns in the second natural language instruction. If the second natural language instruction is determined to be low in complexity based on the number of verbs and nouns, the low-complexity second natural language instruction is parsed based on context information of the second natural language instruction. If the second natural language instruction is a linkage control instruction, the vehicle terminal sends the second natural language instruction to the user terminal, where the second natural language instruction is encapsulated and sent using the A2A protocol, a protocol for communication and collaboration between agents; The user terminal receives a second natural language instruction from the vehicle-mounted terminal, where the second natural language instruction is encapsulated and sent using the A2A protocol, a protocol for communication and collaboration between intelligent agents. The user terminal includes: a first linkage control application for controlling the vehicle-mounted device, an intelligent agent (Agent) module, and multiple services. The intelligent agent module includes a semantic adaptation layer, a user terminal agent unit, a context protocol (MCP) server, and an A2A protocol unit. The user terminal agent unit includes a first large language model (LLM) and a context protocol (MCP) client. The user terminal communicates with the A2A protocol server on the vehicle-mounted terminal via the A2A protocol client. The user terminal determines whether the second natural language instruction is a linkage control instruction for controlling the user terminal using a first large language model, wherein the first large language model is used to extract the number of verbs and nouns in the second natural language instruction; if the second natural language instruction is determined to be of low complexity based on the number of verbs and nouns, parsing the second natural language instruction of low complexity based on context information of the second natural language instruction; If the second natural language instruction is a linkage control instruction, the user terminal determines a second target service and a second service control instruction corresponding to the second natural language instruction based on the first language model and a preset instruction conversion correspondence; The user terminal calls the second target service to execute the second service control instruction.

8. The vehicle linkage control method according to claim 7, characterized in that: The vehicle computer converts the second control instruction into a second natural language instruction, including: If the second control instruction is input by the user on a preset second function control, the vehicle computer obtains a second control operation parameter corresponding to the second control instruction in the preset second function control; Converting the second control operation parameter into a second text instruction; identifying second control intention information of the second text instruction; The second text instruction is converted into a second natural language instruction based on the second control intention information.

9. The vehicle linkage control method according to claim 7, characterized in that: The vehicle computer converts the second control instruction into a second natural language instruction, including: If the second control instruction is input by the user through voice, the vehicle computer converts the second control instruction into a second text instruction; The control intention information of the second text instruction is extracted, and the second text instruction is converted into a second natural language instruction.

10. The vehicle linkage control method according to claim 7, characterized in that: The vehicle computer uses the second language model to determine whether the second natural language instruction is a linkage control instruction for controlling the user terminal, including: The vehicle computer uses the second language model to perform intent recognition on the second natural language instruction to obtain a second target control object; If the second target control object is a user terminal, determining that the second natural language instruction is a linkage control instruction for controlling the user terminal; Alternatively, if the second target control object is a vehicle, it is determined that the second natural language instruction is not a linkage control instruction.

11. The vehicle linkage control method according to claim 7, characterized in that: The method further comprises: The vehicle computer uses the second language model to determine whether the second natural language instruction contains a second linkage reference object; Acquire second linkage reference information corresponding to the second linkage reference object; The second linkage reference information is sent simultaneously with sending the second natural language instruction to the user terminal.

12. A user terminal, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is configured to implement the vehicle linkage control method according to any one of claims 1 to 5, or the vehicle linkage control method according to claim 7, when executing the program stored in the memory.

13. A vehicle terminal, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; The processor is configured to implement the vehicle linkage control method according to claim 1 or 6, or the vehicle linkage control method according to any one of claims 7 to 11, when executing the program stored in the memory.

Images