Vehicle control method, server, and computer-readable storage medium

Abstract

The application discloses a vehicle control method, a server and a computer readable storage medium. The method comprises the following steps: receiving a vehicle voice request, current display content information of a vehicle display component and current perception information of the vehicle. Based on a preset knowledge base, a target execution action is determined according to the voice request, the current display content information and the current perception information. The target execution action is issued to the vehicle to control the vehicle to execute the target execution action. In this way, based on the preset database, the voice request, the current display content information and the current perception information of the vehicle are processed, and the target execution action to be executed is determined, so that the convenience and safety of driving are improved, and the overall user experience is improved.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to a vehicle control method, a server, and a computer-readable storage medium. Background Technology

[0002] In related technologies, in-vehicle voice assistants facilitate human-computer interaction with users, making operation more convenient. However, the design and training of in-vehicle voice assistants are mainly based on training general functions using a limited database, lacking the utilization and analysis of vehicle perception point state information and external knowledge. This limits the depth and efficiency of user interaction with the in-vehicle voice assistant, affecting the user experience. Summary of the Invention

[0003] This application provides a vehicle control method, a server, and a computer-readable storage medium.

[0004] This application provides a vehicle control method, the method comprising:

[0005] Receive voice requests, information on the current display content of vehicle display components, and information on the current perception of the vehicle;

[0006] Based on a preset knowledge base, the target action is determined according to the voice request, the currently displayed content information, and the current perception information.

[0007] The target action is sent to the vehicle to control the vehicle to perform the target action.

[0008] In this way, the server receives voice requests, the current displayed content of vehicle components, and the vehicle's current perception information. Then, based on a pre-defined knowledge base, the server determines the target action based on the voice request, the current displayed content, and the current perception information. Finally, the server sends the target action to the vehicle to control the vehicle to execute it. In this way, by processing voice requests, current displayed content, and the vehicle's current perception information based on a pre-defined database to determine the target action to be performed, the server improves driving convenience and safety, and enhances the overall user experience.

[0009] In some implementations, determining the target action based on a preset knowledge base, according to the voice request, the currently displayed content information, and the current perception information, includes:

[0010] Based on the preset knowledge base, target planning information is generated according to the voice request, the currently displayed content information, and the current perception information;

[0011] Based on the target planning information, determine the target execution action.

[0012] Thus, based on a pre-defined knowledge base, the server generates target planning information according to the voice request, the currently displayed content, and the current perception information. Then, based on the target planning information, the server determines the target execution action. In this way, based on the pre-defined knowledge base, the server can generate accurate target planning information according to the voice request, the currently displayed content, and the current perception information, in order to determine the specific target execution action.

[0013] In some implementations, generating target planning information based on the preset knowledge base, according to the voice request, the currently displayed content information, and the current perception information, includes:

[0014] Based on the first preset model and the preset knowledge base, initial planning information is generated according to the voice request, the currently displayed content information, and the current perception information;

[0015] Based on the second preset model, the target planning information is determined according to the initial planning information.

[0016] Thus, based on the first preset model and the preset knowledge base, the server generates initial planning information according to the voice request, the currently displayed content information, and the current perception information. Based on the second preset model, the server determines the target planning information according to the initial planning information. In this way, by using two preset models and a preset knowledge base, the server can accurately understand and process the voice request, the currently displayed content information, and the current perception information from the vehicle, and generate appropriate and detailed initial planning information in order to determine the target planning information.

[0017] In some implementations, the preset knowledge base includes a pre-built task knowledge base, a perception information base, and a historical information base. The step of generating initial planning information based on the first preset model and the preset knowledge base, according to the voice request, the currently displayed content information, and the current perception information, includes:

[0018] Based on the first preset model and the task knowledge base, the target task type and the operation information corresponding to the target task type are determined according to the voice request.

[0019] Based on the first preset model and the perception information database, a supplementary perception information set is determined according to the current perception information;

[0020] Based on the first preset model and the historical information database, historical planning information and historical execution actions are determined according to the voice request;

[0021] Based on the first preset model, current description information is generated according to the voice request and the currently displayed content information;

[0022] Based on the first preset model, the initial planning information is generated according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution actions, and the current description information.

[0023] Thus, based on the first preset model and the task knowledge base, the server determines the target task type and the corresponding operation information according to the voice request. Next, based on the first preset model and the perception information database, the server determines a supplementary perception information set based on the current perception information. Then, based on the first preset model and the historical information database, the server determines historical planning information and historical execution actions based on the voice request. Subsequently, based on the first preset model, the server generates current description information based on the voice request and the currently displayed content. Finally, based on the first preset model, the server generates initial planning information based on the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions, and current description information. In this way, by determining operation information, supplementary perception information set, historical planning information, and historical execution actions based on the first preset model, the task knowledge base, the perception information database, and the historical information database, and combining the above information with the current perception information, current description information, and voice request, the server can more comprehensively understand the user's intent and the vehicle's environment, generating appropriate initial planning information.

[0024] In some implementations, determining a supplementary perception information set based on the first preset model and the perception information database, according to the current perception information, includes:

[0025] Calculate the similarity between the current perceived information and each piece of perceived information in the perceived information database to obtain multiple similarity values;

[0026] The target similarity value is determined based on a preset similarity threshold and multiple similar values.

[0027] Based on the target similarity value, target perception information is determined to obtain the supplementary perception information set.

[0028] In this way, the server calculates the similarity between the current perceived information and each piece of perceived information in the perceived information database, obtaining multiple similarity values. Next, the server determines the target similarity value based on a preset similarity threshold and the multiple similarity values. Finally, the server determines the target perceived information based on the target similarity value to obtain a supplementary perceived information set. By acquiring a supplementary perceived information set related to the current perceived information, the server can better understand the current environmental conditions, thereby improving the adaptability of the planning scheme to environmental factors.

[0029] In some implementations, determining the target planning information based on the initial planning information and the second preset model includes:

[0030] The initial planning information is evaluated based on the second preset model and the preset evaluation threshold.

[0031] Based on the results of the evaluation process, the target planning information is determined.

[0032] Thus, the server evaluates the initial planning information based on the second preset model and preset evaluation thresholds. Then, based on the evaluation results, the server determines the target planning information. In this way, through evaluation, the system can ensure that the generated target planning information is of high quality and reliable, thereby improving the overall user experience and system performance.

[0033] In some implementations, determining the target planning information based on the result of the evaluation process includes:

[0034] If the initial planning information is determined to be reasonable based on the results of the evaluation process, the initial planning information shall be determined as the target planning information;

[0035] If the initial planning information is determined to be unreasonable based on the evaluation process, the target planning information is generated based on the preset knowledge base, the voice request, the currently displayed content information, and the current perception information.

[0036] Thus, if the initial planning information is deemed reasonable based on the evaluation process, the server designates it as the target planning information. Then, if the initial planning information is deemed unreasonable based on the evaluation process, the server generates target planning information based on a pre-set knowledge base, the voice request, the currently displayed content, and the current perceived information. In this way, through the aforementioned evaluation and adjustment, the server can flexibly generate or adjust planning information according to actual conditions to adapt to constantly changing environments and user needs, thereby enhancing the user's driving experience.

[0037] In some implementations, when the initial planning information is determined to be unreasonable based on the evaluation process, generating the target planning information based on the preset knowledge base, the voice request, the currently displayed content information, and the current perception information includes:

[0038] If the initial planning information is determined to be unreasonable based on the evaluation process, target reflection information is generated based on the voice request, the currently displayed content information, and the current perception information.

[0039] Based on the preset knowledge base, the target planning information is generated according to the target reflection information.

[0040] Thus, if the initial planning information is determined to be unreasonable based on the evaluation results, the server generates target reflection information based on the voice request, currently displayed content, and current perception information. Then, based on a pre-set knowledge base, the server generates target planning information. In this way, by generating target reflection information, the server can identify and correct errors or deficiencies in the planning process, improving the accuracy and reliability of the target planning information, thereby enhancing the user experience.

[0041] This application provides a server, which includes a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, it implements the vehicle control method described above.

[0042] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the vehicle control method described above.

[0043] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0045] Figure 1 This is one of the schematic flowcharts of a vehicle control method according to certain embodiments of this application;

[0046] Figure 2 This is a schematic diagram of the voice request processing flow in some embodiments of this application;

[0047] Figure 3 This is a second schematic flowchart of a vehicle control method according to certain embodiments of this application;

[0048] Figure 4 This is a third schematic flowchart of a vehicle control method according to certain embodiments of this application;

[0049] Figure 5 This is the fourth flowchart of a vehicle control method according to certain embodiments of this application;

[0050] Figure 6 This is the fifth of the flowcharts illustrating a vehicle control method according to certain embodiments of this application;

[0051] Figure 7 This is a schematic flowchart of a vehicle control method according to certain embodiments of this application, number six.

[0052] Figure 8 This is the seventh flowchart of a vehicle control method according to certain embodiments of this application;

[0053] Figure 9 This is the eighth flowchart of a vehicle control method according to certain embodiments of this application. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0055] In the field of intelligent vehicles, in-vehicle voice assistants have become an important technology for improving driving experience and safety. These assistants can typically understand and respond to users' voice commands, performing common functions such as navigation, making phone calls, and playing music. However, while these functions provide convenience for users, the design and training of in-vehicle voice assistants often focus primarily on training common functions based on limited databases, lacking the utilization and analysis of vehicle perception point state information and external knowledge. This, to some extent, limits the depth and efficiency of user interaction with in-vehicle voice assistants, thus affecting the user experience.

[0056] Vehicles are typically equipped with various sensors, such as cameras, radar, and ultrasonic sensors, which provide detailed data about the vehicle's surroundings. However, in-vehicle voice assistants do not fully utilize this data to enhance the interactive experience, lacking the ability to adjust their interaction methods or provide more appropriate suggestions based on changes in vehicle status or driving environment. For example, if the vehicle's sensors detect raindrops, the in-vehicle voice assistant could proactively ask the user whether they need to activate the windshield wipers or adjust their route to avoid flooded areas.

[0057] Based on the above issues, please refer to Figure 1 This application provides a vehicle control method, the method comprising:

[0058] 01: Receive voice requests, information on the current display content of vehicle components, and current perception information of the vehicle;

[0059] 02: Based on a pre-set knowledge base, determine the target action according to the voice request, the currently displayed content information, and the current perception information;

[0060] 03: Issue the target action to the vehicle to control the vehicle to execute the target action.

[0061] This application also provides a server, including a memory and a processor. The vehicle control method of this application can be implemented by the server of this application. Specifically, the memory stores a computer program, and the processor is used to receive voice requests, current display content information of vehicle display components, and current perception information of the vehicle. Based on a preset knowledge base, the processor determines a target action to be performed according to the voice request, the current display content information, and the current perception information. It then sends the target action to the vehicle to control the vehicle to perform the target action.

[0062] This application also provides a vehicle control device. The vehicle control method of this application can be implemented by the vehicle control device of this application. Specifically, the vehicle control device includes a receiving module, a determining module, and a sending module. The receiving module is used to receive voice requests, current display content information of vehicle display components, and current perception information of the vehicle. The determining module is used to determine a target action based on a preset knowledge base, according to the voice request, current display content information, and current perception information. The sending module sends the target action to the vehicle to control the vehicle to execute the target action.

[0063] Specifically, a voice request refers to a voice command issued by a user through the in-vehicle voice assistant during a specific round of voice interaction. This includes controlling a vehicle function, querying information, or requesting a service. Examples include "Turn on the air conditioning," "What's the weather like today?" and "I need the nearest gas station."

[0064] The currently displayed content information of a vehicle's display components refers to the information currently presented on the in-vehicle display screen or other display devices. For example, this could include navigation-related content such as the current driving route, destination, estimated arrival time, and traffic conditions; entertainment system-related content such as music playlists, track information, and radio frequencies; or communication-related content such as incoming call information, text message content, and contact lists. Display content information is a crucial component of the vehicle's intelligent control system, providing drivers with essential driving information and entertainment services, and serving as a vital interface for system-user interaction. It should be noted that in this application's embodiments, the in-vehicle display screen is used as the vehicle display component to describe the vehicle control method; that is, the currently displayed content information is a screenshot of the in-vehicle display screen.

[0065] Current perception information refers to the environmental state information and on-board equipment function status information collected by the vehicle through its sensors. Environmental state information includes data about the vehicle's surrounding environment, such as traffic conditions, road conditions, weather conditions, and geographical location. Traffic conditions refer to road traffic conditions collected by sensors such as cameras, radar, or LiDAR, including the position and speed of other vehicles. Road conditions refer to information such as road slipperiness, potholes, and obstacles. Weather conditions refer to weather information obtained through weather sensors or data exchange with external services, such as temperature, rainfall, and wind speed. Geographical location refers to the vehicle's current location information obtained through positioning systems such as GPS. On-board equipment function status perception information includes status data of the vehicle's internal systems and devices, such as the status of the in-vehicle entertainment system, navigation system settings, air conditioning configuration, engine status, battery charge, fuel level, and tire pressure.

[0066] The target execution action refers to the specific operation determined and executed based on the system's voice request, currently displayed content information, and current perception information. This includes click operations, swipe operations, and typing operations. Click operations refer to clicking icons or buttons to activate a specific function or service. For example, if a user says "turn on music," the system may need to click the music app icon in the in-vehicle entertainment system. Swipe operations refer to actions performed when the current page's information list is collapsed or too long, requiring users to swipe the screen or turn pages to view more information. For example, when the navigation system displays a long route, the system may need to swipe the screen to display the complete route. Typing operations refer to clicking the search box and typing content to find specific information or services within the system. For example, if a user says "search for nearby restaurants," the system may need to click the search box and type "nearby restaurants." Typing operations can also instruct users to enter and send text messages within the in-vehicle system, such as sending text messages or commenting on social media. For example, if a user says "message Zhang San," the system may need to open the messaging app, select the contact Zhang San, and open the chat window.

[0067] The preset knowledge base includes a pre-built task knowledge base, a perception information base, and a historical information base.

[0068] The task knowledge base includes multiple task knowledge entries. Each entry includes a task type and corresponding operation information. The operation information refers to the pre-set steps that may be required to complete the task. For example, {"Task Type": "Search for travel guides using application A", "Operation Information": "1. Open application A 2. Click the search box 3. Search for travel guide information 4. Select a suitable post 5. Summarize post information"}, or {"Task Type": "Book train tickets", "Operation Information": "1. Open application B 2. Click the destination and enter 3. Click the arrival point and enter 4. Select a suitable route 5. Enter the booking information"}, etc.

[0069] The perception information database includes multiple perception information sets. Each perception information set includes perception point information and corresponding supplementary information. Perception point information includes weather, in-vehicle temperature, outside temperature, and road conditions. Supplementary information refers to pre-set additions to the perception point information, which can be used to guide the prompts or feedback that the user may need under that perception point information. For example, {"Perception point information": "Outside temperature", "Supplementary information": "When the outside temperature is low, if the navigation is to an outdoor location, remind the user to keep warm. When the outside temperature is high, if the navigation is to an outdoor location, remind the user to take precautions against sun exposure and heat."}

[0070] The historical information database refers to the database that stores the target planning information and target execution actions generated in each round. The target planning information refers to the steps and actions to complete the user's voice request, and the target execution actions refer to the actual actions performed during the execution of the target planning information, that is, the trajectory of the server in completing the user's voice request.

[0071] Please see Figure 2 The server receives voice requests, information on the current display content of vehicle components, and current perception information of the vehicle.

[0072] Next, based on a pre-set knowledge base, the server processes the above information, that is, it analyzes the content of the voice request, understands the currently displayed content information and the vehicle's current perception information, and then determines the target action to be performed.

[0073] Finally, the server sends the determined target action to the vehicle, and the system on the vehicle receives the instruction and performs the corresponding operation.

[0074] In summary, in the vehicle control method and server provided in this application, the server receives a voice request, the current display content information of the vehicle's display components, and the vehicle's current perception information. Then, based on a preset knowledge base, the server determines the target action to be performed according to the voice request, the current display content information, and the current perception information. Finally, the server sends the target action to the vehicle to control the vehicle to execute the target action. In this way, by processing the voice request, the current display content information, and the vehicle's current perception information based on a preset database to determine the target action to be performed, the convenience and safety of driving are improved, and the overall user experience is enhanced.

[0075] Please see Figure 3 In some implementations, step 02 (determining the target action based on a preset knowledge base, voice request, currently displayed content information, and current perception information) includes:

[0076] 021: Based on a pre-set knowledge base, generate target planning information according to voice requests, currently displayed content information, and current perception information;

[0077] 022: Based on the target planning information, determine the target execution actions.

[0078] In some implementations, the determining module is further configured to generate target planning information based on a preset knowledge base, according to the voice request, currently displayed content information, and current perception information, and to determine the target execution action based on the target planning information.

[0079] In some implementations, the processor is further configured to generate target planning information based on a preset knowledge base, according to the voice request, currently displayed content information, and current perception information, and to determine the target execution action based on the target planning information.

[0080] For details, please refer to [link / reference]. Figure 2 Based on a pre-defined knowledge base, the server generates target planning information according to the voice request, currently displayed content information, and current perception information. The pre-defined knowledge base helps the planning module better understand user intent and scenario requirements, thereby generating accurate and comprehensive target planning information.

[0081] Next, the server determines the target action based on the target planning information.

[0082] Thus, based on a pre-set knowledge base, the server can generate accurate target planning information according to voice requests, currently displayed content information, and current perception information, so as to determine specific target execution actions.

[0083] Please see Figure 4In some implementations, step 021 (generating target planning information based on a preset knowledge base, voice request, currently displayed content information, and current perception information) includes:

[0084] 0211: Based on the first preset model and preset knowledge base, generate initial planning information according to the voice request, the currently displayed content information and the current perception information;

[0085] 0212: Based on the second preset model, determine the target planning information according to the initial planning information.

[0086] In some implementations, the determining module is further configured to generate initial planning information based on a first preset model and a preset knowledge base, according to the voice request, currently displayed content information, and current perception information; and to determine target planning information based on a second preset model and the initial planning information.

[0087] In some implementations, the processor is further configured to generate initial planning information based on a first preset model and a preset knowledge base, according to the voice request, currently displayed content information, and current perception information; and to determine target planning information based on a second preset model and the initial planning information.

[0088] Specifically, the first pre-trained model refers to a pre-trained Visual Language Model (VLM) that combines computer vision and natural language processing techniques to understand and generate cross-modal content, i.e., simultaneously processing image and text data. The first pre-trained model can also utilize Retrieval-Augmented Generation (RAG) technology, leveraging a pre-defined knowledge base to process received information, thereby better understanding user intent and scenario requirements and generating accurate and comprehensive initial planning information. By combining retrieval and generation, Retrieval-Augmented Generation improves the performance of the first pre-trained model when handling complex problems and problems requiring external knowledge.

[0089] The second preset model refers to a model used to evaluate and optimize the initial planning information. It uses a preset evaluation threshold to process the initial planning information to determine whether to use it as target planning information, or, if the initial planning information is unreasonable, to regenerate target planning information based on a preset knowledge base. It should be noted that the first and second preset models can be obtained from the same base model through different training processes, or from different base models trained separately; this is not limited here.

[0090] Please refer to the following: Figure 2After receiving the vehicle's voice request, currently displayed content information, and current perception information, the server processes this information based on a first preset model and a preset knowledge base to generate initial planning information. Next, based on a second preset model and the initial planning information, the server determines target planning information. After determining the target planning information, the server determines the target execution action and sends this action to the vehicle so that the vehicle can execute the target action.

[0091] The following describes the vehicle control method provided in this application using an embodiment. When the outside temperature is 10°C, the user issues a voice request, "Help me search for good Chinese restaurants in application A," and the in-vehicle display is currently on the main interface. Based on a first preset model and a preset knowledge base, the server generates initial planning information according to the received "user voice request: Help me search for good Chinese restaurants in application A; current display information: main interface; current perception information: outside temperature 10°C": "1. Open application A; 2. Click the search box and type 'Chinese restaurant'; 3. Search for restaurant information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user." Then, based on a second preset model, the server determines target planning information according to the initial planning information.

[0092] Thus, by using two preset models and a preset knowledge base, the server can accurately understand and process voice requests from the vehicle, currently displayed content information, and current perception information, generating appropriate and detailed initial planning information in order to determine the target planning information.

[0093] Please see Figure 5 In some implementations, the preset knowledge base includes a pre-built task knowledge base, a perception information base, and a historical information base. Step 0211 (based on the first preset model and the preset knowledge base, generating initial planning information according to the voice request, currently displayed content information, and current perception information) includes:

[0094] 02111: Based on the first preset model and task knowledge base, determine the target task type and the operation information corresponding to the target task type according to the voice request;

[0095] 02112: Based on the first preset model and the perception information database, determine the supplementary perception information set according to the current perception information;

[0096] 02113: Based on the first preset model and historical information database, determine historical planning information and historical execution actions according to the voice request;

[0097] 02114: Based on the first preset model, generate current description information according to the voice request and the currently displayed content information;

[0098] 02115: Based on the first preset model, generate initial planning information according to voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions and current description information.

[0099] In some implementations, the determining module is further configured to, based on a first preset model and a task knowledge base, determine the target task type and corresponding operation information according to the voice request; and, based on the first preset model and a perception information base, determine a supplementary perception information set according to the current perception information; and, based on the first preset model and a historical information base, determine historical planning information and historical execution actions according to the voice request. The determining module is also configured to, based on the first preset model, generate current description information according to the voice request and currently displayed content information; and, based on the first preset model, generate initial planning information according to the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions, and current description information.

[0100] In some embodiments, the processor is further configured to, based on a first preset model and a task knowledge base, determine a target task type and corresponding operation information according to a voice request; and, based on the first preset model and a perception information base, determine a supplementary perception information set according to current perception information; and, based on the first preset model and a historical information base, determine historical planning information and historical execution actions according to a voice request. The processor is also configured to, based on the first preset model, generate current description information according to the voice request and currently displayed content information; and, based on the first preset model, generate initial planning information according to the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions, and current description information.

[0101] For details, please refer to [link / reference]. Figure 2The server, based on a first preset model and a task knowledge base, determines the target task type and corresponding operation information based on the voice request. Continuing the example above, the server, based on the first preset model and the task knowledge base, performs natural language processing on the user's voice request "Help me search for good Chinese restaurants in application A," confirming that the target task type of the user's voice request is "search for guides for application A." Then, using RAG technology, it retrieves the operation information corresponding to the target task type "search for guides for application A" from the task knowledge base: "1. Open application A; 2. Click the search box; 3. Search for guide information; 4. Select a suitable post; 5. Summarize post information." It should be noted that the above operation information is only used to guide the first preset model in generating target planning information. The generated target planning information does not include all the operations in the above operation information, but only one sub-step. For example, if the user is currently on the main interface, the generated initial planning information is "Open application A." If many posts have already appeared, the generated initial planning information is "Select a suitable post."

[0102] In some implementations, the prompt for processing user voice requests in the first preset model may be:

[0103] "Imagine you are an agent planning assistant. Please categorize user requests based on the following information."

[0104] User requests can be categorized into the following types:

[0105] 1. Application A Search Strategy

[0106] 2. Book train tickets

[0107] 3. Order takeout

[0108] 4. Application C sends a message

[0109] Please output the appropriate task category based on the user request.

[0110] User request: <Enter request>

[0111] Task Category:

[0112] It should be noted that the prompt should include all task types from the task knowledge base.

[0113] Next, based on the first preset model and the perception information database, the server determines a supplementary perception information set according to the current perception information. Continuing the example above, based on the first preset model and the perception information database, the server determines the supplementary perception information "perception point information: outside temperature; supplementary information: when the outside temperature is low, if the navigation is to an outdoor location, remind the user to keep warm. when the outside temperature is high, if the navigation is to an outdoor location, remind the user to take precautions against sun exposure and heatstroke."

[0114] Then, based on the first preset model and the historical information database, the server determines the historical planning information and historical execution actions according to the voice request. Continuing the example above, the memory module (i.e., the historical information database) in the first preset model stores the historical planning information and historical execution actions generated in each round. When the first preset model needs to generate initial planning information, it retrieves the historical planning information and historical execution actions from the memory module. For example, if the initial planning information to be generated is "click the search box", then the historical planning information is "open application A", and the historical execution action is "click location A on the main interface (location A is the coordinate of application A on the main interface)".

[0115] Subsequently, based on the first preset model, the server generates current description information according to the voice request and the currently displayed content information. Continuing the above example, the first preset model processes the received voice request and the currently displayed content information to generate current description information specific to the currently displayed content information. For example, if the currently displayed content information is a screenshot of the main interface, it will use text to describe the content on the main interface. In some implementations, the prompt from the first preset model for the currently displayed content information may be as follows:

[0116] "Imagine you are an intelligent assistant. I will give you a picture and ask you to describe the details in the picture."

[0117] You can refer to voice requests to add emphasis to the image description.

[0118] Voice request: <Input request>

[0119] picture:<image_url>

[0120] Based on the information above, please provide a description of the image. Please use Chinese and keep the description concise, within 150 characters.

[0121] Current description information:

[0122] Finally, based on the first preset model, the server generates initial planning information according to the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions, and current description information. Continuing the above example, the first preset model integrates the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions, and current description information to generate the initial planning information "1. Open application A; 2. Click the search box and type 'Chinese restaurant'; 3. Search for guide information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user". If it is determined from the current perception information and current description information that opening application A is not necessary, the generated initial planning information is "1. Click the search box and type 'Chinese restaurant'; 2. Search for guide information; 3. Select a suitable post; 4. Summarize post information; 5. Send a reminder to the user". The planning information "Send a reminder to the user" is obtained from the supplementary perception information set. In some implementations, the prompt for the first preset model to generate the initial planning information may be as follows:

[0123] "Assuming you are an agent planning assistant, please output a suitable planning plan based on the following information."

[0124] Current description information: <Current description information>

[0125] Voice request: <voice request>

[0126] Operation Information: <Operation Information>

[0127] Perception point status: <Current perception information>

[0128] Supplementary information for sensory points: <Supplementary sensory information set>

[0129] Historical plan: <Historical planning information>

[0130] Historical action: <historical action performed>

[0131] Based on the information above, please output the next step plan based on the currently displayed content and historical information:

[0132] Next steps:

[0133] The appropriate planning plan and the next steps both refer to the initial planning information.

[0134] Thus, based on the first preset model, the task knowledge base, the perception information base, and the historical information base, the operation information, the supplementary perception information set, the historical planning information, and the historical execution actions are determined respectively. By combining the above information, the current perception information, the current description information, and the voice request, the server can more comprehensively understand the user's intention and the vehicle's environment and generate appropriate initial planning information.

[0135] Please see Figure 6 In some implementations, step 02112 (based on the first preset model and the perception information database, determining the supplementary perception information set according to the current perception information) includes:

[0136] 021121: Calculate the similarity between the current perceived information and each piece of perceived information in the perceived information database, and obtain multiple similarity values;

[0137] 021122: Determine the target similarity value based on the preset similarity threshold and multiple similar values;

[0138] 021123: Based on the target similarity value, determine the target perception information to obtain a supplementary perception information set.

[0139] In some implementations, the determining module is further configured to calculate the similarity between the current perceived information and each piece of perceived information in the perceived information database, obtaining multiple similarity values; and to determine a target similarity value based on a preset similarity value threshold and the multiple similarity values; and to determine target perceived information based on the target similarity value to obtain a supplementary perceived information set.

[0140] In some implementations, the processor is further configured to calculate the similarity between the current sensed information and each sensed information in the sensed information database, obtaining multiple similarity values; and to determine a target similarity value based on a preset similarity value threshold and the multiple similarity values; and to determine target sensed information based on the target similarity value to obtain a supplementary sensed information set.

[0141] Specifically, the preset similarity threshold refers to a pre-set value used to determine the similarity between the current perceived information and each piece of perceived information in the perceived information database. When calculating the similarity between two data points, if the obtained similarity value is higher than or equal to this threshold, the system considers the two data points to be sufficiently similar and can be classified into the same category or associated in some way. If the similarity value is lower than this threshold, the system considers the two data points to be insufficiently similar and should not be classified into the same category or associated. In this application embodiment, the provided vehicle control method is described with a preset similarity threshold of 0.75.

[0142] Please refer to the following: Figure 2The server calculates the similarity between the current perceived information and each piece of perceived information in the perceived information database, obtaining multiple similarity values. Next, based on a preset similarity threshold and the multiple similarity values, the server determines the target similarity value. Finally, based on the target similarity value, the server determines the target perceived information to obtain a supplementary perceived information set.

[0143] Continuing with the example above, the server calculates the similarity between the current perceived information "outside temperature 10°C" and each piece of perceived information in the perceived information database. In some implementations, the similarity calculation formula is: Where A represents the current perceived information "outside temperature 10℃", and B represents each piece of perceived information in the perceived information database.

[0144] Next, each obtained similarity value is compared with a preset similarity threshold. Similarity values ​​greater than or equal to the preset similarity threshold of 0.75 are identified as target similarity values, and the perceptual information corresponding to these target similarity values ​​is identified as target perceptual information. Finally, a supplementary perceptual information set is constructed based on the obtained target perceptual information.

[0145] In this way, by acquiring a supplementary set of sensing information related to the current sensing information, the server can better understand the current environmental conditions, thereby improving the adaptability of the planning scheme to environmental factors.

[0146] Please see Figure 7 In some implementations, step 0212 (based on the second preset model, determining the target planning information according to the initial planning information) includes:

[0147] 02121: Based on the second preset model and preset evaluation threshold, evaluate and process the initial planning information;

[0148] 02122: Based on the results of the assessment, determine the target planning information.

[0149] In some implementations, the determining module is used to evaluate the initial planning information based on a second preset model and a preset evaluation threshold, and to determine the target planning information based on the results of the evaluation process.

[0150] In some implementations, the processor is further configured to evaluate the initial planning information based on a second preset model and a preset evaluation threshold, and to determine the target planning information based on the results of the evaluation process.

[0151] Specifically, the preset evaluation threshold refers to a pre-set standard used to evaluate the quality of the initial planning information generated by the first preset model. If the evaluation result exceeds this preset evaluation threshold, the system may consider the initial planning information to be reasonable; if the evaluation result is lower than this preset evaluation threshold, the system may consider the output to need improvement or regeneration. It should be noted that the vehicle control method provided in the embodiments of this application is described using a preset evaluation threshold of 0.8.

[0152] Please refer to the following: Figure 2 The server evaluates the initial planning information based on a second preset model and preset evaluation thresholds. Then, based on the evaluation results, the server determines the target planning information.

[0153] Continuing with the example above, after the first preset model generates initial planning information, the second preset model evaluates and processes the initial planning information. In some implementations, the prompt for the second preset model to evaluate and process the initial planning information may be as follows:

[0154] "Assuming you are a scoring assistant helping the agent with planning, please output a score for the initial planning information based on the following information."

[0155] Please rate the effectiveness, feasibility, and risk aspects, with a score range of 0 to 5.

[0156] Effectiveness: Whether the current planning steps meet the task objectives. The score ranges from 0 to 5 points, with higher scores indicating greater effectiveness.

[0157] Feasibility: Whether the current planning steps are feasible under the current resources and environment. The score ranges from 0 to 5 points, with higher scores indicating greater feasibility.

[0158] Risk: Assess whether the current planning steps pose any risks, with a score range of 0 to 5, where a higher score indicates a greater risk.

[0159] Current description information: <Current description information>

[0160] Voice request: <voice request>

[0161] Operation Information: <Operation Information>

[0162] Perception point status: <Current perception information>

[0163] Initial planning information: <Initial planning information>

[0164] Based on the information above, please provide a rating for effectiveness, feasibility, and risk.

[0165] score:".

[0166] It should be noted that the second preset model calculates the obtained effectiveness score, feasibility score, and risk score to obtain a total reasonableness score. Then, this total reasonableness score is compared with a preset evaluation threshold to confirm the reasonableness of the initial planning information. Continuing the example above, after evaluating the initial planning information "1. Open application A; 2. Click the search box and type 'Chinese restaurant'; 3. Search for travel guide information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user," the second preset model obtains an effectiveness score of 5, a feasibility score of 3, and a risk score of 1. Subsequently, the evaluation formula is applied... The calculation yielded a total reasonableness score of 0.82. Then, based on the total reasonableness score of 0.82 and the preset evaluation threshold of 0.8, the evaluation results were confirmed.

[0167] In this way, through evaluation and processing, the system can ensure that the generated target planning information is of high quality and reliable, thereby improving the overall user experience and system performance.

[0168] Please see Figure 8 In some implementations, step 02122 (determining target planning information based on the results of the evaluation process) includes:

[0169] 021221: If the initial planning information is determined to be reasonable based on the results of the evaluation and processing, the initial planning information shall be determined as the target planning information;

[0170] 021222: If the initial planning information is determined to be unreasonable based on the evaluation results, target planning information is generated based on the preset knowledge base, voice request, currently displayed content information, and current perception information.

[0171] In some implementations, the determining module is used to determine the initial planning information as target planning information if the initial planning information is deemed reasonable based on the evaluation process results. Conversely, if the initial planning information is deemed unreasonable based on the evaluation process results, the module generates target planning information based on a preset knowledge base, the voice request, the currently displayed content information, and the current perception information.

[0172] In some implementations, the processor is further configured to, if the initial planning information is determined to be reasonable based on the results of the evaluation process, identify the initial planning information as target planning information; and if the initial planning information is determined to be unreasonable based on the results of the evaluation process, generate target planning information based on a preset knowledge base, the voice request, the currently displayed content information, and the current perception information.

[0173] Specifically, if the evaluation results show that the initial planning information is reasonable, i.e., it meets the preset evaluation criteria, then the server will directly determine the initial planning information as the target planning information. If the evaluation results show that the initial planning information is unreasonable, the server will reconsider the voice request, the currently displayed content information, and the current perception information based on the preset knowledge base to generate new target planning information.

[0174] Continuing the example above, the second preset model uses a total rationality score of 0.82 and a preset evaluation threshold of 0.8. Since the total rationality score of 0.82 is greater than the preset evaluation threshold of 0.8, the initial planning information is confirmed as reasonable. The initial planning information "1. Open application A; 2. Click the search box and type 'Chinese restaurant'; 3. Search for guide information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user" is then determined as the target planning information. Based on this target planning information, the target action is confirmed as "Click on location A on the main interface (location A is the coordinate of application A on the main interface)". If the total rationality score is 0.7, the second preset model confirms that the initial planning information is unreasonable. Based on the preset knowledge base, the server regenerates the target planning information according to the voice request, the currently displayed content information, and the current perception information.

[0175] It should be noted that if the target planning information obtained is "end task", then the server will consider that the current user's voice request has been processed and the process can be terminated.

[0176] Thus, through the above evaluation, processing, and adjustment, the server can flexibly generate or adjust planning information according to the actual situation to adapt to the ever-changing environment and user needs, thereby enhancing the user's driving experience.

[0177] Please see Figure 9 In some implementations, step 021222 (if the initial planning information is determined to be unreasonable based on the evaluation process, generating target planning information based on a preset knowledge base, voice request, currently displayed content information, and current perception information) includes:

[0178] 0212221: If the initial planning information is determined to be unreasonable based on the evaluation results, target reflection information is generated based on the voice request, the currently displayed content information, and the current perception information;

[0179] 0212222: Based on a pre-set knowledge base, generate target planning information according to the target reflection information.

[0180] In some embodiments, the vehicle control device further includes a generation module, which generates target reflection information based on voice requests, currently displayed content information, and current perception information when the initial planning information is determined to be unreasonable based on the results of the evaluation process. It also generates target planning information based on the target reflection information and a preset knowledge base.

[0181] In some implementations, the processor is further configured to generate target reflection information based on the voice request, currently displayed content information, and current perception information if the initial planning information is determined to be unreasonable based on the results of the evaluation process; and to generate target planning information based on the target reflection information and a preset knowledge base.

[0182] Specifically, target reflection information refers to the information generated by the server based on voice requests, currently displayed content information, and current perception information when the server determines that the initial planning information is unreasonable. This information is used to adjust or correct the execution strategy. It reflects the system's analysis of the reasons for the unreasonable initial planning information and its thinking on how to better execute user instructions.

[0183] If the initial planning information is determined to be unreasonable based on the evaluation results, the server generates target reflection information based on the voice request, currently displayed content information, and current perception information. Then, based on a pre-set knowledge base, the server generates target planning information according to the target reflection information.

[0184] In this way, by generating goal reflection information, the server can identify and correct errors or deficiencies in the planning process, improve the accuracy and reliability of goal planning information, and thus enhance the user experience.

[0185] This application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the steps of the vehicle control method described above.

[0186] It is understood that a computer program includes computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, external hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0187] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0188] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of executable request code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0189] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A vehicle control method, characterized in that, The method includes: Receive voice requests, information on the current display content of vehicle display components, and information on the current perception of the vehicle; Based on a preset knowledge base, target planning information is determined according to the voice request, the currently displayed content information, and the current perception information. The preset knowledge base includes a pre-built task knowledge base, a perception information base, and a historical information base. The task knowledge base includes task types and operation information corresponding to the task types. The perception information base includes perception point information and supplementary information corresponding to the perception point information. The historical information base includes historical planning information and historical execution actions. Based on the target planning information, determine the target execution action; The target action is sent to the vehicle to control the vehicle to perform the target action.

2. The vehicle control method according to claim 1, characterized in that, The step of generating target planning information based on the preset knowledge base, according to the voice request, the currently displayed content information, and the current perception information, includes: Based on the first preset model and the preset knowledge base, initial planning information is generated according to the voice request, the currently displayed content information, and the current perception information; Based on the second preset model, the target planning information is determined according to the initial planning information.

3. The vehicle control method according to claim 2, characterized in that, The process of generating initial planning information based on the first preset model and the preset knowledge base, according to the voice request, the currently displayed content information, and the current perception information, includes: Based on the first preset model and the task knowledge base, the target task type and the operation information corresponding to the target task type are determined according to the voice request. Based on the first preset model and the perception information database, a supplementary perception information set is determined according to the current perception information; Based on the first preset model and the historical information database, the historical planning information and the historical execution action are determined according to the voice request; Based on the first preset model, current description information is generated according to the voice request and the currently displayed content information; Based on the first preset model, the initial planning information is generated according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution actions, and the current description information.

4. The vehicle control method according to claim 3, characterized in that, The step of determining a supplementary perception information set based on the first preset model and the perception information database, according to the current perception information, includes: Calculate the similarity between the current perceived information and each piece of perceived information in the perceived information database to obtain multiple similarity values; The target similarity value is determined based on a preset similarity threshold and multiple similar values. Based on the target similarity value, target perception information is determined to obtain the supplementary perception information set.

5. The vehicle control method according to claim 2, characterized in that, The step of determining the target planning information based on the second preset model and the initial planning information includes: The initial planning information is evaluated based on the second preset model and the preset evaluation threshold. Based on the results of the evaluation process, the target planning information is determined.

6. The vehicle control method according to claim 5, characterized in that, The step of determining the target planning information based on the evaluation process includes: If the initial planning information is determined to be reasonable based on the results of the evaluation process, the initial planning information shall be determined as the target planning information; If the initial planning information is determined to be unreasonable based on the evaluation process, the target planning information is generated based on the preset knowledge base, the voice request, the currently displayed content information, and the current perception information.

7. The vehicle control method according to claim 6, characterized in that, If, based on the evaluation process, the initial planning information is determined to be unreasonable, the target planning information is generated based on the preset knowledge base, the voice request, the currently displayed content information, and the current perception information, including: If the initial planning information is determined to be unreasonable based on the evaluation process, target reflection information is generated based on the voice request, the currently displayed content information, and the current perception information. Based on the preset knowledge base, the target planning information is generated according to the target reflection information.

8. A server, characterized in that, The server includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the vehicle control method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images