Vehicle control method, server and computer readable storage medium

Abstract

The invention discloses a vehicle control method, a server and a computer readable storage medium. The method comprises the following steps: receiving a vehicle voice request, current display content information of vehicle display parts and current perception information of a vehicle; and based on a preset knowledge base, according to the voice request, the current display content information and the current perception information, determining a target execution action. And issuing the target execution action to the vehicle to control the vehicle to execute the target execution action. Thus, based on the preset database, the voice request, the current display content information and the current perception information of the vehicle are processed, and the target execution action needing to be executed is determined, so that the driving convenience and safety are improved, and the overall user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of voice interaction technology, and in particular to a vehicle control method, a server and a computer-readable storage medium. Background Art

[0002] In the related art, the in-vehicle voice assistant is used to interact with the user to facilitate the user's operation. However, the design and training of the in-vehicle voice assistant are mainly based on a limited database to train general functions, lacking the utilization and analysis of the vehicle's perception point status information and external knowledge, which limits the depth and efficiency of the user's interaction with the in-vehicle voice assistant and affects the user experience. Summary of the invention

[0003] The present application provides a vehicle control method, a server and a computer-readable storage medium.

[0004] The present application provides a vehicle control method, the method comprising:

[0005] Receiving a voice request, current display content information of a vehicle display component, and current perception information of the vehicle;

[0006] Based on a preset knowledge base, determining a target execution action according to the voice request, the currently displayed content information and the currently perceived information;

[0007] The target execution action is sent to the vehicle to control the vehicle to execute the target execution action.

[0008] In this way, the server receives the voice request, the current display content information of the vehicle display components, and the current perception information of the vehicle. Then, based on the preset knowledge base, the server determines the target execution action according to the voice request, the current display content information, and the current perception information. Finally, the server sends the target execution action to the vehicle to control the vehicle to execute the target execution action. In this way, based on the preset database, the voice request, the current display content information, and the current perception information of the vehicle are processed to determine the target execution action to be executed, so as to improve the convenience and safety of driving and enhance the overall user experience.

[0009] In some implementations, determining the target execution action based on the preset knowledge base, according to the voice request, the currently displayed content information, and the currently perceived information, includes:

[0010] Based on the preset knowledge base, generating target planning information according to the voice request, the currently displayed content information and the currently perceived information;

[0011] The target execution action is determined according to the target planning information.

[0012] In this way, based on the preset knowledge base, the server generates target planning information according to the voice request, the current display content information and the current perception information. Then, the server determines the target execution action according to the target planning information. In this way, based on the preset knowledge base, the server can generate accurate target planning information according to the voice request, the current display content information and the current perception information, so as to determine the specific target execution action.

[0013] In some implementations, generating target planning information based on the preset knowledge base, according to the voice request, the currently displayed content information, and the currently perceived information, includes:

[0014] Based on the first preset model and the preset knowledge base, generating initial planning information according to the voice request, the currently displayed content information and the currently perceived information;

[0015] Based on a second preset model, the target planning information is determined according to the initial planning information.

[0016] In this way, the server generates initial planning information based on the first preset model and the preset knowledge base according to the voice request, the current display content information and the current perception information. Based on the second preset model, the server determines the target planning information according to the initial planning information. In this way, by using two preset models and the preset knowledge base, the server can accurately understand and process the voice request from the vehicle, the current display content information and the current perception information, and generate appropriate and detailed initial planning information in order to determine the target planning information.

[0017] In some embodiments, the preset knowledge base includes a pre-built task knowledge base, a perception information base, and a history information base, and the generating of initial planning information based on the first preset model and the preset knowledge base according to the voice request, the current display content information, and the current perception information includes:

[0018] Based on the first preset model and the task knowledge base, determining a target task type and operation information corresponding to the target task type according to the voice request;

[0019] Based on the first preset model and the perception information library, determining a supplementary perception information set according to the current perception information;

[0020] Based on the first preset model and the historical information library, determine historical planning information and historical execution actions according to the voice request;

[0021] Based on the first preset model, generating current description information according to the voice request and the currently displayed content information;

[0022] Based on the first preset model, the initial planning information is generated according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution action and the current description information.

[0023] In this way, the server determines the target task type and the operation information corresponding to the target task type according to the voice request based on the first preset model and the task knowledge base. Next, based on the first preset model and the perception information base, the server determines the supplementary perception information set according to the current perception information. Then, based on the first preset model and the history information base, the server determines the historical planning information and the historical execution action according to the voice request. Subsequently, based on the first preset model, the server generates the current description information according to the voice request and the current display content information. Finally, based on the first preset model, the server generates the initial planning information according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution action and the current description information. In this way, based on the first preset model, the task knowledge base, the perception information base and the history information base, the operation information, the supplementary perception information set, the historical planning information and the historical execution action are determined respectively. In combination with the above information, the current perception information, the current description information and the voice request, the server can more comprehensively understand the user's intention and the environment of the vehicle and generate appropriate initial planning information.

[0024] In some embodiments, determining the supplementary perception information set based on the first preset model and the perception information library according to the current perception information includes:

[0025] Calculating the similarity between the current perception information and each perception information in the perception information library to obtain multiple similarity values;

[0026] Determine a target similarity value according to a preset similarity value threshold and the plurality of similarity values;

[0027] According to the target similarity value, target perception information is determined to obtain the supplementary perception information set.

[0028] In this way, the server calculates the similarity between the current perception information and each perception information in the perception information library to obtain multiple similarity values. Then, the server determines the target similarity value based on the preset similarity value threshold and the multiple similarity values. Finally, the server determines the target perception information based on the target similarity value to obtain a supplementary perception information set. In this way, by obtaining a supplementary perception information set related to the current perception information, the server can better understand the current environmental conditions, thereby improving the adaptability of the planning scheme to environmental factors.

[0029] In some embodiments, determining the target planning information based on the second preset model and according to the initial planning information includes:

[0030] Based on the second preset model and the preset evaluation threshold, evaluating the initial planning information;

[0031] The target planning information is determined according to the result of the evaluation process.

[0032] In this way, the server evaluates the initial planning information based on the second preset model and the preset evaluation threshold. Then, the server determines the target planning information according to the result of the evaluation process. In this way, through the evaluation process, the system can ensure that the generated target planning information is of high quality and reliable, thereby improving the overall user experience and system performance.

[0033] In some embodiments, determining the target planning information according to the result of the evaluation process includes:

[0034] If it is determined that the initial planning information is reasonable according to the result of the evaluation process, determining the initial planning information as the target planning information;

[0035] In the case where it is determined that the initial planning information is unreasonable according to the result of the evaluation process, the target planning information is generated based on the preset knowledge base, according to the voice request, the currently displayed content information and the current perception information.

[0036] In this way, when the initial planning information is determined to be reasonable according to the result of the evaluation process, the server determines the initial planning information as the target planning information. Then, when the initial planning information is determined to be unreasonable according to the result of the evaluation process, based on the preset knowledge base, the server generates the target planning information according to the voice request, the current display content information and the current perception information. In this way, through the above-mentioned evaluation process and adjustment, the server can flexibly generate or adjust the planning information according to the actual situation to adapt to the ever-changing environment and user needs, thereby enhancing the user's driving experience.

[0037] In some embodiments, when it is determined that the initial planning information is unreasonable according to the result of the evaluation process, generating the target planning information based on the preset knowledge base, according to the voice request, the currently displayed content information and the current perception information includes:

[0038] In the case where it is determined that the initial planning information is unreasonable according to the result of the evaluation process, generating target reflection information according to the voice request, the current display content information and the current perception information;

[0039] Based on the preset knowledge base and according to the target reflection information, the target planning information is generated.

[0040] In this way, when it is determined that the initial planning information is unreasonable according to the result of the evaluation process, the server generates target reflection information according to the voice request, the current display content information and the current perception information. Then, based on the preset knowledge base, the server generates target planning information according to the target reflection information. In this way, by generating target reflection information, the server can identify and correct errors or deficiencies in the planning process, improve the accuracy and reliability of the target planning information, and thus enhance the user experience.

[0041] An embodiment of the present application provides a server, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned vehicle control method is implemented.

[0042] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle control method described above are implemented.

[0043] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0045] Figure 1 It is one of the flowcharts of the vehicle control method of certain embodiments of the present application;

[0046] Figure 2 is a schematic diagram of a processing flow of a voice request in certain embodiments of the present application;

[0047] Figure 3 This is a second flow chart of a vehicle control method according to certain embodiments of the present application;

[0048] Figure 4 This is a third flow chart of a vehicle control method according to certain embodiments of the present application;

[0049] Figure 5 This is a fourth flow chart of a vehicle control method according to certain embodiments of the present application;

[0050] Figure 6 This is a fifth flow chart of a vehicle control method according to certain embodiments of the present application;

[0051] Figure 7 This is the sixth flow chart of the vehicle control method of certain embodiments of the present application;

[0052] Figure 8 This is the seventh flow chart of the vehicle control method of certain embodiments of the present application;

[0053] Fig. 9 This is the eighth flow chart of the vehicle control method of certain embodiments of the present application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0055] In the field of smart cars, in-vehicle voice assistants have become an important technology to enhance driving experience and safety. These assistants are usually able to understand and respond to user voice commands and perform common functions such as navigation, phone calls, music playback, etc. However, although these functions provide convenience to users, the design and training of in-vehicle voice assistants often focus on training common functions based on limited databases, lacking the utilization and analysis of vehicle perception point status information and external knowledge, which to some extent limits the depth and efficiency of user interaction with in-vehicle voice assistants, thus affecting user experience.

[0056] Vehicles are usually equipped with a variety of sensors, such as cameras, radars, and ultrasonic sensors, which can provide detailed data about the vehicle's surroundings. However, in-vehicle voice assistants do not make full use of this data to enhance the interactive experience, and lack the ability to adjust the interaction method or provide more appropriate suggestions based on the vehicle's status information or changes in the driving environment. For example, if the vehicle's sensors detect raindrops, the in-vehicle voice assistant can proactively ask the user whether to activate the wipers or adjust the driving route to avoid flooded sections of the road.

[0057] Based on the above questions, please refer to Figure 1 , the embodiment of the present application provides a vehicle control method, the method comprising:

[0058] 01: Receive voice requests, current display content information of vehicle display components and current perception information of the vehicle;

[0059] 02: Based on the preset knowledge base, the target execution action is determined according to the voice request, the current display content information and the current perception information;

[0060] 03: Send the target execution action to the vehicle to control the vehicle to execute the target execution action.

[0061] The embodiment of the present application also provides a server, including a memory and a processor. The vehicle control method of the embodiment of the present application can be implemented by the server of the embodiment of the present application. Specifically, a computer program is stored in the memory, and the processor is used to receive a voice request, current display content information of a vehicle display component, and current perception information of the vehicle. And based on a preset knowledge base, according to the voice request, the current display content information, and the current perception information, determine a target execution action. And send the target execution action to the vehicle to control the vehicle to execute the target execution action.

[0062] The embodiment of the present application also provides a vehicle control device. The vehicle control method of the embodiment of the present application can be implemented by the vehicle control device of the embodiment of the present application. Specifically, the vehicle control device includes a receiving module, a determining module and a sending module. The receiving module is used to receive voice requests, current display content information of vehicle display components and current perception information of the vehicle. The determining module is used to determine a target execution action based on a preset knowledge base, according to the voice request, the current display content information and the current perception information. The sending module sends the target execution action to the vehicle to control the vehicle to execute the target execution action.

[0063] Specifically, voice requests refer to voice commands issued by users through the in-vehicle voice assistant during a round of voice interaction, including vehicle function control, information query or service request. For example, "Turn on the air conditioner", "What's the weather like today?" and "I need the nearest gas station".

[0064] The current display content information of the vehicle display component refers to the information content currently presented on the vehicle display screen or other display devices. For example, navigation-related content such as the current driving route, destination, estimated arrival time and traffic conditions, or entertainment system-related content such as music playlists, track information, radio station frequencies, etc., and communication-related content such as incoming call information, text message content, and contact lists. Display content information is an important part of the vehicle intelligent control system. The display content information can provide the driver with key driving information and entertainment services, and is also an important interface for interaction between the system and the user. It should be noted that in the implementation manner of the present application, the vehicle control method is described using the vehicle display screen as the vehicle display component, that is, the current display content information is a screenshot of the vehicle display screen.

[0065] Current perception information refers to the environmental status information of the current vehicle and the state perception information of the vehicle's onboard equipment functions collected by the vehicle through its sensors. Environmental status information includes data about the vehicle's surroundings, such as traffic conditions, road conditions, weather conditions, and geographic location. Traffic conditions refer to road traffic conditions collected by sensors such as cameras, radars, or laser radar (LiDAR), including the location and speed of other vehicles. Road conditions refer to information such as the slipperiness, potholes, and obstacles of the road. Weather conditions refer to weather information obtained through weather sensors or data exchange with external services, such as temperature, rainfall, wind speed, etc. Geographic location refers to the current location information of the vehicle obtained through positioning systems such as GPS. Onboard equipment function status perception information includes status data of the vehicle's internal systems and equipment, such as the status of the onboard entertainment system, the settings of the navigation system, the air conditioning configuration, the engine status, the battery power, the fuel level, and the tire pressure.

[0066] The target execution action refers to the specific operation determined and executed by the system voice request, the currently displayed content information and the currently perceived information, including click operation, sliding operation and typing operation. Click operation refers to the operation of clicking an icon or button, which is used to activate a specific function or service. For example, when the user says "open the music", the system may need to execute the operation of clicking the music application icon in the in-vehicle entertainment system. Sliding operation refers to the need to slide the screen or turn the page to view more information when the current page information list is collapsed or too long. For example, when the route information displayed by the navigation system is too long, the system may need to perform the operation of sliding the screen to display the complete route. Typing operation refers to the operation of clicking the search box and typing content, which is used to find specific information or services in the system. For example, when the user says "search for nearby restaurants", the system may need to execute the operation of clicking the search box and typing "nearby restaurants". Typing operation can also indicate the input and sending of text information in the in-vehicle system, such as sending text messages or making comments on social media. For example, when the user says "send a message to Zhang San", the system may need to execute the operation of opening the message application, selecting the contact Zhang San and opening the chat window.

[0067] The preset knowledge base includes a pre-built task knowledge base, a perception information base, and a historical information base.

[0068] The task knowledge base includes multiple pieces of task knowledge information, each of which includes a task type and operation information corresponding to the task type. Operation information refers to the pre-set operation steps that may be required to complete the task. For example, {"task type":"Application A Search Guide","operation information":"1. Open application A 2. Click the search box 3. Search for guide information 4. Select a suitable post 5. Summarize post information"}, or {"task type":"Book a ticket","operation information":"1. Open application B 2. Click the destination and enter 3. Click the destination and enter 4. Select a suitable route 5. Enter the reservation person information"}, etc.

[0069] The perception information library includes multiple perception information, each of which includes perception point information and supplementary information corresponding to the perception point information. The perception point information includes weather, temperature inside the car, temperature outside the car, road conditions, etc. The supplementary information refers to the pre-set supplement to the perception point information, which can be used to guide the prompts or feedback that the user may need under the perception point information. For example, {"Perception point information":"Temperature outside the car","Supplementary information":"When the temperature outside the car is low, if you navigate to an outdoor location, remind the user to keep warm. When the temperature outside the car is high, if you navigate to an outdoor location, remind the user to pay attention to sun protection and cooling."}.

[0070] The historical information base refers to the database that stores the target planning information and target execution actions generated in each round. The target planning information refers to the steps and actions to complete the user's voice request, and the target execution action refers to the action actually performed during the execution of the target planning information, that is, the trajectory of the server completing the user's voice request.

[0071] See also Figure 2 The server receives the voice request, the current display content information of the vehicle display components and the current perception information of the vehicle.

[0072] Next, based on the preset knowledge base, the server processes the above information, that is, analyzes the content of the voice request, understands the current display content information and the current perception information of the vehicle, and then determines the target execution action that needs to be performed.

[0073] Finally, the server sends the determined target execution action to the vehicle, and the system on the vehicle performs the corresponding operation after receiving the instruction.

[0074] In summary, in the vehicle control method and server provided in the embodiments of the present application, the server receives a voice request, current display content information of vehicle display components, and current perception information of the vehicle. Then, based on a preset knowledge base, the server determines a target execution action according to the voice request, current display content information, and current perception information. Finally, the server sends the target execution action to the vehicle to control the vehicle to execute the target execution action. In this way, based on the preset database, the voice request, current display content information, and current perception information of the vehicle are processed to determine the target execution action that needs to be executed, so as to improve the convenience and safety of driving and enhance the overall user experience.

[0075] See also Figure 3 In some embodiments, step 02 (determining the target execution action based on the preset knowledge base, according to the voice request, the current display content information and the current perception information) includes:

[0076] 021: Generate target planning information based on the preset knowledge base, voice request, current display content information and current perception information;

[0077] 022: Determine the target execution action based on the target planning information.

[0078] In some embodiments, the determination module is further configured to generate target planning information based on a preset knowledge base, according to the voice request, the current display content information and the current perception information, and to determine the target execution action based on the target planning information.

[0079] In some embodiments, the processor is further configured to generate target planning information based on a preset knowledge base, according to the voice request, the current display content information and the current perception information, and determine the target execution action according to the target planning information.

[0080] For details, please refer to Figure 2 ,Based on the preset knowledge base, the server generates target planning information according to ,the voice request, the current display content information and the ,current perception information.,The preset knowledge base can help the planning module better understand ,the user's intention and scenario requirements, thereby generating ,accurate and comprehensive target planning information.

[0081] Next, the server determines the target execution action based on the target planning information.

[0082] In this way, based on the preset knowledge base, the server can generate accurate target planning information according to the voice request, the current display content information and the current perception information, so as to determine the specific target execution action.

[0083] See also Figure 4In some embodiments, step 021 (generating target planning information based on a preset knowledge base, according to a voice request, current display content information, and current perception information) includes:

[0084] 0211: Generate initial planning information based on the first preset model and the preset knowledge base according to the voice request, the current display content information and the current perception information;

[0085] 0212: Based on the second preset model and the initial planning information, determine the target planning information.

[0086] In some embodiments, the determination module is further configured to generate initial planning information based on the first preset model and the preset knowledge base, according to the voice request, the current display content information and the current perception information, and to determine target planning information based on the second preset model and the initial planning information.

[0087] In some embodiments, the processor is further configured to generate initial planning information based on the first preset model and the preset knowledge base, according to the voice request, the current display content information and the current perception information, and to determine target planning information based on the second preset model and the initial planning information.

[0088] Specifically, the first preset model refers to a pre-trained visual language model (Visual Language Model, VLM), which combines computer vision technology and natural language processing technology to understand and generate cross-modal content, that is, to process image and text data simultaneously. The first preset model can also process the received information based on the retrieval-augmented generation technology (Retrieval-Augmented Generation, RAG) using a preset knowledge base to better understand user intentions and scenario requirements, thereby generating accurate and comprehensive initial planning information. Retrieval-augmented generation technology improves the performance of the first preset model in dealing with complex problems and problems that require external knowledge by combining retrieval and generation.

[0089] The second preset model refers to a model used to evaluate and optimize the initial planning information, and uses a preset evaluation threshold to evaluate the initial planning information to determine whether to use it as the target planning information, or when the initial planning information is unreasonable, regenerate the target planning information based on the preset knowledge base. It should be noted that the first preset model and the second preset model can be obtained by different training based on the same base model, or by different training of different base models, which is not limited here.

[0090] Please refer to Figure 2After the server receives the voice request, current display content information, and current perception information of the vehicle, the server processes the above information based on the first preset model and the preset knowledge base to generate initial planning information. Then, the server determines the target planning information based on the initial planning information based on the second preset model. After determining the target planning information, the server determines the target execution action based on the target planning information, and sends the target execution action to the vehicle so that the vehicle executes the target execution action.

[0091] The following is an example to illustrate the vehicle control method provided by the implementation method of the present application. When the outside temperature is 10°C, the user issues a voice request "Help me search for a delicious Chinese restaurant in application A", and the current vehicle display screen is in the main interface. Based on the first preset model and the preset knowledge base, the server will generate the initial planning information "1. Open application A; 2. Click the search box and type Chinese restaurant; 3. Search for strategy information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user". Then, based on the second preset model, the server determines the target planning information according to the initial planning information.

[0092] In this way, by using two preset models and a preset knowledge base, the server can accurately understand and process the voice request from the vehicle, the current display content information and the current perception information, and generate appropriate and detailed initial planning information in order to determine the target planning information.

[0093] See also Figure 5 In some embodiments, the preset knowledge base includes a pre-built task knowledge base, a perception information base, and a history information base. Step 0211 (generating initial planning information based on the first preset model and the preset knowledge base, according to the voice request, the current display content information, and the current perception information) includes:

[0094] 02111: determining a target task type and operation information corresponding to the target task type according to a voice request based on a first preset model and a task knowledge base;

[0095] 02112: determining a supplementary perception information set based on the first preset model and the perception information library and according to current perception information;

[0096] 02113: Based on the first preset model and the historical information database, determine the historical planning information and the historical execution action according to the voice request;

[0097] 02114: Based on the first preset model, generate current description information according to the voice request and the current display content information;

[0098] 02115: Based on the first preset model, generate initial planning information according to the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions and current description information.

[0099] In some embodiments, the determination module is also used to determine the target task type and the operation information corresponding to the target task type according to the voice request based on the first preset model and the task knowledge base. And based on the first preset model and the perception information base, determine the supplementary perception information set according to the current perception information. And based on the first preset model and the history information base, determine the historical planning information and the historical execution action according to the voice request. The determination module is also used to generate the current description information based on the first preset model according to the voice request and the current display content information. And based on the first preset model, generate the initial planning information according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution action and the current description information.

[0100] In some embodiments, the processor is also used to determine the target task type and the operation information corresponding to the target task type according to the voice request based on the first preset model and the task knowledge base. And based on the first preset model and the perception information base, determine the supplementary perception information set according to the current perception information. And based on the first preset model and the history information base, determine the historical planning information and the historical execution action according to the voice request. The processor is also used to generate the current description information based on the first preset model according to the voice request and the current display content information. And based on the first preset model, generate the initial planning information according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution action and the current description information.

[0101] For details, please refer to Figure 2, the server determines the target task type and the operation information corresponding to the target task type based on the first preset model and the task knowledge base according to the voice request. Continuing with the above example, the server is based on the first preset model and the task knowledge base. The first preset model performs natural language processing on the user's voice request "Help me search for delicious Chinese restaurants in application A", confirms that the target task type of the user's voice request is "Application A search strategy", and then uses RAG technology to obtain the operation information corresponding to the target task type "Application A search strategy" from the task knowledge base "1. Open application A; 2. Click the search box; 3. Search strategy information; 4. Select a suitable post; 5. Summarize post information". It should be noted that the above operation information is only used to guide the first preset model to generate target planning information, and the generated target planning information does not include all the operations in the above operation information, but only one of the sub-steps. For example, if you are currently in the main interface, the initial planning information generated is "Open application A". If there are already many posts, the initial planning information generated is "Select a suitable post".

[0102] In some implementations, the prompt for processing the user's voice request in the first preset model may be:

[0103] "Suppose you are an agent planning assistant, please classify the user's request based on the following information

[0104] The tasks requested by users are classified into the following categories:

[0105] 1. Application A search strategy

[0106] 2. Book your ticket

[0107] 3. Order takeout

[0108] 4. Application C sends a message

[0109] Please output appropriate task classification based on user requests

[0110] User request: <input request>

[0111] Task category:".

[0112] It should be noted that the prompt should include all task types in the task knowledge base.

[0113] Next, based on the first preset model and the perception information library, the server determines a supplementary perception information set according to the current perception information. Continuing with the above example, based on the first preset model and the perception information library, the server determines the supplementary perception information set according to the current perception information "the temperature outside the car is 10°C" "perception point information: the temperature outside the car; supplementary information: when the temperature outside the car is low, if you navigate to an outdoor location, remind the user to keep warm. When the temperature outside the car is high, if you navigate to an outdoor location, remind the user to pay attention to sun protection and cooling down.".

[0114] Then, based on the first preset model and the historical information library, the server determines the historical planning information and historical execution actions according to the voice request. Continuing with the above example, the memory module (i.e., the historical information library) in the first preset model will store the historical planning information and historical execution actions generated in each round. When the first preset model needs to generate initial planning information, the first preset model will obtain the historical planning information and historical execution actions from the memory module. For example, if the initial planning information that currently needs to be generated is "click the search box", then the historical planning information is "open application A", and the historical execution action is "click position A on the main interface (position A is the coordinate of application A on the main interface)".

[0115] Subsequently, based on the first preset model, the server generates the current description information according to the voice request and the currently displayed content information. Continuing with the above example, the first preset model processes the received voice request and the currently displayed content information to generate the current description information for the currently displayed content information. For example, when the currently displayed content information is a screenshot of the main interface, text will be used to describe the content on the main interface. In some embodiments, the prompt of the first preset model for the currently displayed content information may be as follows:

[0116] “Suppose you are an intelligent assistant and I give you a picture and ask you to describe the details in the picture.

[0117] You can refer to voice requests to add emphasis to the image description.

[0118] Voice request: <input request>

[0119] picture:<image_url>

[0120] Based on the above information, please output a description of the picture. Please use Chinese description, concise language, within 150 words:

[0121] Current description information: ".

[0122] Finally, based on the first preset model, the server generates initial planning information according to the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions and current description information. Continuing with the above example, the first preset model will integrate the voice request, operation information, current perception information, supplementary perception information set, historical planning information, historical execution actions and current description information to generate the initial planning information "1. Open application A; 2. Click the search box and type Chinese restaurant; 3. Search for strategy information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user". If it is determined based on the current perception information and the current description information that it is not necessary to open application A, the generated initial planning information "1. Click the search box and type Chinese restaurant; 2. Search for strategy information; 3. Select a suitable post; 4. Summarize post information; 5. Send a reminder to the user". Among them, the planning information "Send a reminder to the user" is obtained according to the supplementary perception information set. In some embodiments, the prompt for the first preset model to generate the initial planning information may be as follows:

[0123] "Assuming you are an agent planning assistant, please output a suitable planning plan based on the following information.

[0124] Current description information: <Current description information>

[0125] Voice request: <voice request>

[0126] Operation information: <operation information>

[0127] Perception point status: <current perception information>

[0128] Supplementary information of perception point: <Supplementary perception information set>

[0129] Historical plan: <historical planning information>

[0130] Historical action: <historical execution action>

[0131] Based on the above information, please output the next step plan based on the current displayed content information and historical information:

[0132] Next steps:".

[0133] Among them, the appropriate plan and the next plan both refer to the initial planning information.

[0134] In this way, based on the first preset model, task knowledge base, perception information base and historical information base, the operation information, supplementary perception information set, historical planning information and historical execution actions are determined respectively. Combined with the above information, current perception information, current description information and voice requests, the server can more comprehensively understand the user's intentions and the vehicle's environment, and generate appropriate initial planning information.

[0135] See also Figure 6 In some embodiments, step 02112 (determining a supplementary perception information set based on the first preset model and the perception information library and the current perception information) includes:

[0136] 021121: Calculate the similarity between the current perception information and each perception information in the perception information library to obtain multiple similarity values;

[0137] 021122: Determine a target similarity value according to a preset similarity value threshold and multiple similarity values;

[0138] 021123: Determine the target perception information according to the target similarity value to obtain a supplementary perception information set.

[0139] In some embodiments, the determination module is further configured to calculate the similarity between the current perception information and each perception information in the perception information library to obtain a plurality of similarity values, and to determine a target similarity value based on a preset similarity value threshold and the plurality of similarity values, and to determine the target perception information based on the target similarity value to obtain a supplementary perception information set.

[0140] In some embodiments, the processor is further configured to calculate the similarity between the current perception information and each perception information in the perception information library to obtain a plurality of similarity values, and to determine a target similarity value based on a preset similarity value threshold and the plurality of similarity values, and to determine the target perception information based on the target similarity value to obtain a supplementary perception information set.

[0141] Specifically, the preset similarity value threshold refers to a pre-set value used to determine the similarity between the current perception information and each perception information in the perception information library. When calculating the similarity between two data points, if the obtained similarity value is higher than or equal to this threshold, the system will consider the two data points to be similar enough to be classified into the same category or to be associated in some way. If the similarity value is lower than this threshold, the system will consider the two data points to be not similar enough and should not be classified into the same category or associated. In the implementation manner of the present application, the provided vehicle control method is described with a preset similarity value threshold of 0.75.

[0142] Please refer to Figure 2, the server calculates the similarity between the current perception information and each perception information in the perception information library to obtain multiple similarity values. Then, the server determines the target similarity value based on the preset similarity value threshold and the multiple similarity values. Finally, the server determines the target perception information based on the target similarity value to obtain a supplementary perception information set.

[0143] Continuing with the above example, the server calculates the similarity between the current perception information "outside temperature 10°C" and each perception information in the perception information database. In some embodiments, the similarity calculation formula is: A is the current perception information "the temperature outside the vehicle is 10°C", and B is each perception information in the perception information library.

[0144] Next, each similarity value obtained is compared with a preset similarity threshold, similarity values ​​greater than or equal to the preset similarity threshold of 0.75 are confirmed as target similarity values, and the perception information corresponding to these target similarity values ​​is confirmed as target perception information. Finally, a supplementary perception information set is constructed based on the obtained target perception information.

[0145] In this way, by obtaining a supplementary perception information set related to the current perception information, the server can better understand the current environmental conditions, thereby improving the adaptability of the planning scheme to environmental factors.

[0146] See also Figure 7 In some embodiments, step 0212 (determining target planning information based on the second preset model and the initial planning information) includes:

[0147] 02121: Based on the second preset model and the preset evaluation threshold, the initial planning information is evaluated and processed;

[0148] 02122: Determine target planning information based on the results of the evaluation process.

[0149] In some embodiments, the determination module is used to perform evaluation processing on the initial planning information based on the second preset model and the preset evaluation threshold, and determine the target planning information according to the result of the evaluation processing.

[0150] In some embodiments, the processor is further configured to perform evaluation processing on the initial planning information based on the second preset model and the preset evaluation threshold, and determine the target planning information according to the result of the evaluation processing.

[0151] Specifically, the preset evaluation threshold refers to a preset standard for evaluating the quality of the initial planning information generated by the first preset model. If the evaluation result exceeds the preset evaluation threshold, the system may consider the initial planning information to be reasonable; if the evaluation result is lower than the preset evaluation threshold, the system may consider the output to need to be improved or regenerated. It should be noted that the vehicle control method provided in the embodiment of the present application is described with a preset evaluation threshold of 0.8.

[0152] Please refer to Figure 2 The server performs evaluation processing on the initial planning information based on the second preset model and the preset evaluation threshold. Then, the server determines the target planning information according to the result of the evaluation processing.

[0153] Continuing with the above example, after the first preset model generates the initial planning information, the second preset model will evaluate the initial planning information. In some embodiments, the prompt for the second preset model to evaluate the initial planning information may be as follows:

[0154] “Assuming that you are a scoring assistant who assists the agent in planning, please output the score of the initial planning information based on the following information.

[0155] Please rate based on effectiveness, feasibility, and risk, with scores ranging from 0 to 5 points

[0156] Validity: Whether the current planning steps meet the task objectives. The score range is 0 to 5 points. The higher the score, the higher the validity.

[0157] Feasibility: Whether the current planning steps can be implemented under the current resources and environment. The score range is 0 to 5 points. The higher the score, the higher the feasibility.

[0158] Risk: Assess whether the current planning steps will bring risks. The score range is 0 to 5 points. The higher the score, the greater the risk.

[0159] Current description information: <Current description information>

[0160] Voice request: <voice request>

[0161] Operation information: <operation information>

[0162] Perception point status: <current perception information>

[0163] Initial planning information: <initial planning information>

[0164] Based on the above information, please output the scores of effectiveness, feasibility and risk.

[0165] score:".

[0166] It should be noted that the second preset model will calculate the obtained effectiveness score, feasibility score and risk score to obtain a total rationality score. Then, the rationality of the initial planning information is confirmed by comparing the total rationality score with the preset evaluation threshold. Continuing with the above example, the second preset model evaluates the initial planning information "1. Open application A; 2. Click the search box and type Chinese restaurant; 3. Search for guide information; 4. Select a suitable post; 5. Summarize post information; 6. Send a reminder to the user" and obtains an effectiveness score of 5 points, a feasibility score of 3 points, and a risk score of 1 point. Then, according to the evaluation formula The calculation results in a total rationality score of 0.82. Next, the result of the evaluation process is confirmed based on the total rationality score of 0.82 and the preset evaluation threshold of 0.8.

[0167] In this way, through the evaluation process, the system can ensure that the generated target planning information is of high quality and reliable, thereby improving the overall user experience and system performance.

[0168] See also Figure 8 In some embodiments, step 02122 (determining target planning information based on the result of the evaluation process) includes:

[0169] 021221: When the initial planning information is determined to be reasonable according to the result of the evaluation process, the initial planning information is determined as the target planning information;

[0170] 021222: When it is determined that the initial planning information is unreasonable based on the result of the evaluation process, target planning information is generated based on a preset knowledge base, according to the voice request, the current display content information and the current perception information.

[0171] In some embodiments, the determination module is used to determine the initial planning information as the target planning information when the initial planning information is determined to be reasonable according to the result of the evaluation process. And when the initial planning information is determined to be unreasonable according to the result of the evaluation process, the target planning information is generated based on the preset knowledge base, the voice request, the current display content information and the current perception information.

[0172] In some embodiments, the processor is further configured to determine the initial planning information as the target planning information when the initial planning information is determined to be reasonable according to the result of the evaluation process. And when the initial planning information is determined to be unreasonable according to the result of the evaluation process, the target planning information is generated based on the preset knowledge base, the voice request, the current display content information and the current perception information.

[0173] Specifically, if the evaluation result shows that the initial planning information is reasonable, that is, it meets the preset evaluation criteria, then the server will directly determine the initial planning information as the target planning information. If the evaluation result shows that the initial planning information is unreasonable, the server will reconsider the voice request, the current display content information and the current perception information based on the preset knowledge base to generate new target planning information.

[0174] Continuing with the above example, the second preset model confirms that the initial planning information is reasonable based on the total rationality score of 0.82 and the preset evaluation threshold of 0.8. The total rationality score of 0.82 is greater than the preset evaluation threshold of 0.8, and the initial planning information "1. Open application A; 2. Click the search box and type Chinese restaurant; 3. Search for guide information; 4. Select appropriate posts; 5. Summarize post information; 6. Send reminders to users" is determined as the target planning information, and then based on the target planning information, the target execution action is confirmed to be "Click on position A on the main interface (position A is the coordinate of application A on the main interface)". If the total rationality score is 0.7, the second preset model confirms that the initial planning information is unreasonable. Based on the preset knowledge base, the server generates the target planning information again according to the voice request, the currently displayed content information and the current perception information.

[0175] It should be noted that if the target planning information obtained is "end task", the server will consider that the current user voice request has been processed and the process can be ended.

[0176] In this way, through the above-mentioned evaluation processing and adjustment, the server can flexibly generate or adjust planning information according to actual conditions to adapt to the ever-changing environment and user needs, thereby enhancing the user's driving experience.

[0177] See also Fig. 9 In some embodiments, step 021222 (generating target planning information based on a preset knowledge base, voice request, current display content information and current perception information, when the initial planning information is determined to be unreasonable according to the result of the evaluation process) includes:

[0178] 0212221: in a case where it is determined according to the result of the evaluation process that the initial planning information is unreasonable, generating target reflection information according to the voice request, the current display content information and the current perception information;

[0179] 0212222: Based on the preset knowledge base and target reflection information, generate target planning information.

[0180] In some embodiments, the vehicle control device further includes a generation module, which is used to generate target reflection information according to the voice request, the current display content information and the current perception information when the initial planning information is determined to be unreasonable according to the result of the evaluation process, and to generate target planning information according to the target reflection information based on the preset knowledge base.

[0181] In some embodiments, the processor is further configured to generate target reflection information according to the voice request, the current display content information and the current perception information when the initial planning information is determined to be unreasonable according to the result of the evaluation process, and to generate target planning information according to the target reflection information based on the preset knowledge base.

[0182] Specifically, target reflection information refers to the information generated by the server based on voice requests, currently displayed content information and current perception information for adjusting or correcting the execution strategy when the server determines that the initial planning information is unreasonable. This information reflects the system's analysis of the reasons why the initial planning information is unreasonable, as well as its thinking on how to better execute user instructions.

[0183] When the initial planning information is determined to be unreasonable according to the result of the evaluation process, the server generates target reflection information according to the voice request, the current display content information and the current perception information. Then, based on the preset knowledge base, the server generates target planning information according to the target reflection information.

[0184] In this way, by generating target reflection information, the server can identify and correct errors or deficiencies in the planning process, improve the accuracy and reliability of target planning information, and thus enhance the user experience.

[0185] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle control method described above are implemented.

[0186] It is understood that a computer program includes computer program code. The computer program code may be in source code form, object code form, executable file or some intermediate form. Computer readable storage media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution medium.

[0187] In the description of this specification, the descriptions with reference to the terms "specifically", "further", "particularly", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0188] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable requests for implementing specific logical functions or steps of a process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0189] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A vehicle control method, characterized in that: The method comprises: Receiving a voice request, current display content information of a vehicle display component, and current perception information of the vehicle; Based on a preset knowledge base, determining a target execution action according to the voice request, the currently displayed content information and the currently perceived information; The target execution action is sent to the vehicle to control the vehicle to execute the target execution action.

2. The vehicle control method according to claim 1, characterized in that: The step of determining the target execution action based on the preset knowledge base and according to the voice request, the currently displayed content information and the currently perceived information includes: Based on the preset knowledge base, generating target planning information according to the voice request, the currently displayed content information and the currently perceived information; The target execution action is determined according to the target planning information.

3. The vehicle control method according to claim 2, characterized in that: The generating target planning information based on the preset knowledge base according to the voice request, the currently displayed content information and the currently perceived information includes: Based on the first preset model and the preset knowledge base, generating initial planning information according to the voice request, the currently displayed content information and the currently perceived information; Based on a second preset model, the target planning information is determined according to the initial planning information.

4. The vehicle control method according to claim 3, characterized in that: The preset knowledge base includes a pre-built task knowledge base, a perception information base, and a history information base. The initial planning information is generated based on the first preset model and the preset knowledge base according to the voice request, the current display content information, and the current perception information, including: Based on the first preset model and the task knowledge base, determining a target task type and operation information corresponding to the target task type according to the voice request; Based on the first preset model and the perception information library, determining a supplementary perception information set according to the current perception information; Based on the first preset model and the historical information library, determine historical planning information and historical execution actions according to the voice request; Based on the first preset model, generating current description information according to the voice request and the currently displayed content information; Based on the first preset model, the initial planning information is generated according to the voice request, the operation information, the current perception information, the supplementary perception information set, the historical planning information, the historical execution action and the current description information.

5. The vehicle control method according to claim 4, characterized in that: The determining, based on the first preset model and the perception information library and according to the current perception information, a supplementary perception information set includes: Calculating the similarity between the current perception information and each perception information in the perception information library to obtain multiple similarity values; Determine a target similarity value according to a preset similarity value threshold and the plurality of similarity values; According to the target similarity value, target perception information is determined to obtain the supplementary perception information set.

6. The vehicle control method according to claim 3, characterized in that: The determining the target planning information based on the second preset model and according to the initial planning information includes: Based on the second preset model and the preset evaluation threshold, evaluating the initial planning information; The target planning information is determined according to the result of the evaluation process.

7. The vehicle control method according to claim 6, characterized in that: Determining the target planning information according to the result of the evaluation process includes: If it is determined that the initial planning information is reasonable according to the result of the evaluation process, determining the initial planning information as the target planning information; In the case where it is determined that the initial planning information is unreasonable according to the result of the evaluation process, the target planning information is generated based on the preset knowledge base, according to the voice request, the currently displayed content information and the current perception information.

8. The vehicle control method according to claim 7, characterized in that: In the case where it is determined according to the result of the evaluation process that the initial planning information is unreasonable, generating the target planning information based on the preset knowledge base, according to the voice request, the currently displayed content information and the currently perceived information, includes: In the case where it is determined that the initial planning information is unreasonable according to the result of the evaluation process, generating target reflection information according to the voice request, the current display content information and the current perception information; Based on the preset knowledge base and according to the target reflection information, the target planning information is generated.

9. A server, characterized in that: The server includes a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the vehicle control method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images