Vehicle control method, device and storage medium

Abstract

The application provides a vehicle control method and device and a storage medium, and relates to the technical field of vehicles.In the method, the vehicle control device performs structured processing on collected various types of information, such as user information, road information or real-time query information, and inputs a first structured sequence for indicating the various types of information into a preconfigured vehicle control model to obtain a first vehicle control instruction adapted to the various types of information, so as to provide reasonable and accurate driving strategies for users and improve the intelligent degree of vehicle assisted driving.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a vehicle control method, device and storage medium. Background Technology

[0002] Assisted driving refers to the use of a series of technologies and systems to assist drivers in operating vehicles, thereby improving driving safety, comfort, and convenience. It uses sensors, cameras, radar, and advanced computer algorithms to perceive the vehicle's surroundings and provide support to the driver when necessary. Vehicle assisted driving systems can provide functions including adaptive cruise control (ACC), lane keeping assist (LKA), and automatic emergency braking (AEB). Currently, assisted driving systems are not yet suitable for complex and ever-changing driving scenarios, and their level of intelligence needs further improvement. Summary of the Invention

[0003] This application provides a vehicle control method, device, and storage medium to provide users with reasonable and accurate driving strategies, aiming to improve the intelligence level of vehicle assisted driving.

[0004] Firstly, this application provides a vehicle control method applicable to any vehicle control device, which can be integrated into a vehicle controller or operate independently of it. In some embodiments, the vehicle control device can also be deployed in the cloud, which is not limited in this application. For ease of description, the following description uses the vehicle control device as the implementing entity.

[0005] The method includes: a vehicle control device acquiring a first structured sequence, the first structured sequence indicating at least two items of user information, road information, or real-time query information; the vehicle control device inputting the first structured sequence into a vehicle control model, and obtaining a first vehicle control command after processing by the vehicle control model; the first vehicle control command indicating the adjustment of the current vehicle's driving strategy; and the vehicle control device executing or adjusting the first vehicle control command.

[0006] Optionally, the vehicle control model is a machine learning model, including but not limited to neural network models such as recurrent neural networks (RNNs), long short-term memory networks (LSTM), and Transformer models.

[0007] In the above scheme, the vehicle control device performs structured processing on the collected information of various types, and inputs the first structured sequence used to indicate each type of information into a pre-configured vehicle control model to obtain a first vehicle control command adapted to each type of information, thereby providing the user with a reasonable and accurate driving strategy. By executing or adjusting the first vehicle control command, the intelligence level of vehicle assisted driving is improved.

[0008] In one optional embodiment of the first aspect, the first vehicle control instruction is a structured control instruction. Optionally, the first vehicle control instruction includes at least one vehicle control instruction. For example, the first vehicle control instruction is {"intent": "lane change", "lane": 3}, indicating a lane change to lane 3. Another example is [{"intent": "lane change", "lane": 3}, {"intent": acceleration, "max": 80}], indicating a lane change to lane 3 and an increase in vehicle speed to 80 kilometers per hour.

[0009] In one optional embodiment of the first aspect, the user information includes voice information. The vehicle control device acquires a first structured sequence, comprising: converting user-inputted voice information into text information corresponding to the voice information; performing text analysis on the text information to obtain at least one text sequence; and converting the at least one text sequence into a first normalized text sequence. The first normalized text sequence is used to indicate a voice control command input by the user. In this embodiment, the first structured sequence includes a first normalized text sequence.

[0010] In some embodiments, the vehicle control device includes a voice command analysis module, through which the vehicle control device obtains a first standardized text sequence.

[0011] In the above scheme, the vehicle control device recognizes the user's voice input and converts useful voice control commands (such as driving intentions) into a standardized text sequence, which serves as one input to the vehicle control model, providing data support for the model to output accurate vehicle control commands.

[0012] In one optional embodiment of the first aspect, the road information includes at least one of lane information on the road on which the vehicle travels, or vehicle information on the lane. Optionally, the lane information includes the number of lanes on the road, lane type (e.g., overtaking lane / fast lane, driving lane, slow lane, emergency lane, ramp, etc.), and other lane attributes (e.g., lane light intensity, etc.). The vehicle information on the lane includes the number of vehicles on the lane, vehicle position, vehicle speed, and vehicle taillight indication status, etc.

[0013] The vehicle control device acquires a first structured sequence, including: extracting image frames from external video data collected by a camera, performing semantic segmentation on the extracted image frames to obtain a semantic segmentation result; and converting the semantic segmentation result into a second standardized text sequence. The semantic segmentation result is used to indicate road information, and the second standardized text sequence is used to indicate road information.

[0014] In this embodiment, the first structured sequence includes the second normalized text sequence.

[0015] In some embodiments, the vehicle control device includes an on-board video analysis module, through which the vehicle control device obtains a second standardized text sequence.

[0016] In the above scheme, the vehicle control device identifies external video data and converts the lane and vehicle information of the current driving road into a standardized text sequence, which serves as one input to the vehicle control model, providing data support for the model to output accurate vehicle control commands.

[0017] In one alternative embodiment of the first aspect, the real-time query information includes at least one of vehicle data, application data, or user historical driving data.

[0018] The vehicle control device acquires a first structured sequence, including: acquiring vehicle data from onboard sensors; and / or acquiring application data queried by the user from an onboard application; and / or acquiring historical driving data of the user from an onboard database. The vehicle control device converts at least one of the vehicle data, application data, or historical driving data of the user into a third standardized text sequence. The third standardized text sequence is used to indicate real-time query information.

[0019] In this embodiment, the first structured sequence includes the third standardized text sequence.

[0020] In one alternative embodiment of the first aspect, the vehicle data includes at least one of vehicle speed data, tire pressure data, or battery power data.

[0021] In one alternative embodiment of the first aspect, the application data includes at least one of navigation data, weather data, or multimedia data.

[0022] In one alternative embodiment of the first aspect, the user's historical driving data includes at least one of the user's driving habits or historical driving routes.

[0023] In some embodiments, the vehicle control device includes a real-time information processing module, through which the vehicle control device acquires a third standardized text sequence.

[0024] In the above scheme, the vehicle control device collects vehicle data, application data, or user historical driving data, and converts this data into a standardized text sequence, which serves as one input to the vehicle control model, providing data support for the model to output accurate vehicle control commands.

[0025] In one optional embodiment of the first aspect, the user information includes user state information. The vehicle control device acquires a first structured sequence, including: acquiring tone information from the user's voice input and emotional information from image data captured by the camera; and obtaining a fourth standardized text sequence based on the tone information and emotional information. The fourth standardized text sequence is used to indicate the user state information.

[0026] The user's emotional information in the image data collected by the camera includes the user's facial emotional information and the user's body emotional information.

[0027] In this embodiment, the first structured sequence includes the fourth normalized text sequence.

[0028] In some embodiments, the vehicle control device includes a user state analysis module, through which the vehicle control device obtains a fourth standardized text sequence.

[0029] In the above scheme, the vehicle control device obtains the user's overall emotional state data by analyzing the user's voice tone, facial features, and body language. This emotional state data is then converted into a standardized text sequence, which serves as one input to the vehicle control model, providing data support for the model to output accurate vehicle control commands.

[0030] In one optional embodiment of the first aspect, the vehicle control method further includes: the vehicle control device outputting prompt information via voice broadcast and / or interface display, the prompt information indicating a first vehicle control command. The vehicle control device receives user feedback information from the prompt information. If the user feedback information indicates acceptance of the first vehicle control command, the vehicle control device executes the first vehicle control command; or, if the user feedback information indicates rejection of the first vehicle control command, the vehicle control device adjusts the first vehicle control command.

[0031] In the above scheme, the prompts output by the vehicle control device are not voice control commands input by the user alone, but one or more vehicle control commands output by the vehicle control model by integrating various information such as vehicle and road conditions. This provides users with assisted driving suggestions, enhances cockpit interactivity, and provides more driving guidance suggestions, especially for novice drivers.

[0032] In one optional embodiment of the first aspect, if the user feedback information includes a voice control command input by the user, adjusting the first vehicle control command includes: obtaining a fifth standardized text sequence, the fifth standardized text sequence being used to indicate the voice control command in the user feedback information; inputting a second structured sequence into a vehicle control model, and after processing by the vehicle control model, obtaining a second vehicle control command; the second structured sequence includes the fifth standardized text sequence; the second vehicle control command is the adjusted vehicle control command.

[0033] The vehicle control method also includes: executing a second vehicle control command.

[0034] In this embodiment, the vehicle control model can also adjust the vehicle control commands based on the voice control commands in the user feedback information and the structured sequences input to the model in the past, in order to provide the user with vehicle control commands that conform to their driving intentions and improve the intelligence level of vehicle assisted driving.

[0035] Secondly, embodiments of this application provide a vehicle control device, comprising: an acquisition module for acquiring a first structured sequence, the first structured sequence indicating at least two items of user information, road information, or real-time query information; a processing module for inputting the first structured sequence into a vehicle control model, and obtaining a first vehicle control command after processing by the vehicle control model; the first vehicle control command indicating an adjustment to the driving strategy of the current vehicle; a control module for executing the first vehicle control command; or, the processing module for adjusting the first vehicle control command.

[0036] Thirdly, embodiments of this application provide a vehicle control device, including: one or more processors and a memory; the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the vehicle control device to perform the method shown in the first aspect or any optional embodiment of the first aspect.

[0037] Fourthly, embodiments of this application provide a chip system applied to a vehicle control device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the vehicle control device to perform the method shown in the first aspect or any optional embodiment of the first aspect.

[0038] Fifthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on a vehicle control device, cause the vehicle control device to perform the method shown in the first aspect or any optional embodiment of the first aspect.

[0039] In a sixth aspect, embodiments of this application provide a computer program product including computer program code that, when executed on a vehicle control device, causes the vehicle control device to perform the method shown in the first aspect or any optional embodiment of the first aspect. Attached Figure Description

[0040] Figure 1 A flowchart of a vehicle control method provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating the data source for the vehicle control method provided in the embodiments of this application;

[0042] Figure 3 An interactive schematic diagram of a vehicle control method provided in an embodiment of this application;

[0043] Figure 4 An interactive schematic diagram of another vehicle control method provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the structure of a vehicle control device provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of another vehicle control device provided in an embodiment of this application. Detailed Implementation

[0046] With the rapid development of artificial intelligence (AI) technology, its application in the field of driver assistance is becoming increasingly widespread and in-depth. Vehicles use voice recognition technology to recognize user voice commands and implement voice control functions, such as controlling the volume of in-car speakers, the interior temperature, acceleration, and deceleration, providing users with personalized and intelligent services and driving experiences. Currently, driver assistance technology possesses a certain level of voice interaction capability, but voice interaction functions do not cover all intelligent control operations. Due to the lack of multi-dimensional data monitoring and intelligent processing systems, the level of intelligence of the system needs to be improved when the vehicle encounters complex and changing driving environments.

[0047] To provide a better human-computer interaction experience, multiple sensory channels, such as auditory, visual, and tactile channels, can be fully utilized. Different combinations of input forms (e.g., voice, images, gestures, touch, posture, facial expressions, eye movements, brainwaves, etc.) can offer a variety of options for human-computer interaction, improving the naturalness and efficiency of the interaction. Furthermore, considering the application of large-scale model technology, the powerful data processing, generation, and multimodal interaction capabilities of large-scale models can be leveraged to comprehensively process and analyze various types of data, including voice, vision, and touch, in order to enhance the intelligence level of vehicle-assisted driving.

[0048] Therefore, this application provides a vehicle control method that, through a pre-configured vehicle control model, comprehensively processes and analyzes various types of input data (such as user-inputted voice control commands, voice tone, user facial expressions, external environment data, user-queried navigation data, weather data, vehicle status, etc.), and outputs corresponding vehicle control commands to provide assisted driving support for the user and improve the intelligence level of vehicle assisted driving. The input to the vehicle control model is a structured sequence, which indicates at least one type of data information. The output of the vehicle control model is a structured control command, which indicates at least one vehicle control instruction.

[0049] The technical solutions provided in this application will be described in detail below with reference to specific embodiments.

[0050] Figure 1 This is a flowchart illustrating a vehicle control method provided in an embodiment of this application. This vehicle control method can be applied to any vehicle control device, which can be integrated into the vehicle controller or operate independently of it; this application does not limit this. For ease of description, the following description uses the vehicle control device as the implementing entity. Figure 1 As shown, the vehicle control method includes the following steps:

[0051] S101, Obtain a first structured sequence, the first structured sequence being used to indicate at least two items of user information, road information, and real-time query information.

[0052] Optionally, user information may include at least one of the following: user-inputted voice information or user status information.

[0053] The user-input voice information includes both voice control commands and non-voice control information. Voice control commands are those related to vehicle control. Non-voice control information refers to voice information unrelated to vehicle control, such as voice messages from casual conversations within the vehicle. User status information indicates the user's emotional state, including emotions such as happiness, anger, and anxiety.

[0054] In some embodiments, refer to Figure 2 The vehicle control unit obtains a structured sequence of voice control commands from the user information from the voice command analysis module. The voice command analysis module analyzes and processes the user-input voice information and converts the voice commands into a standardized sequence.

[0055] In some embodiments, refer to Figure 2The vehicle control unit obtains a structured sequence of user status information from the user status analysis module. Specifically, the user status analysis module analyzes and processes in-vehicle user image data captured by the in-vehicle camera to obtain a structured sequence indicating the user's facial emotional state. The user status analysis module also performs tone analysis on the user's input voice information to obtain a structured sequence indicating the user's tone and emotional state.

[0056] Optionally, road information includes at least one of the following: lane information on the road the vehicle is traveling on, or vehicle information in the lane. Lane information includes the number of lanes on the road, lane type (e.g., overtaking lane / fast lane, driving lane, slow lane, emergency lane, ramp, etc.), and other lane attributes (e.g., lane light intensity). Vehicle information in the lane includes the number of vehicles in the lane, vehicle position, vehicle speed, and taillight status.

[0057] In some embodiments, refer to Figure 2 The vehicle control unit obtains a structured sequence of road information from the onboard video analysis module. This module analyzes and processes the external video data captured by the onboard camera to obtain a structured sequence indicating the road information outside the vehicle.

[0058] Optionally, the real-time query information includes at least one of vehicle data, application data, or user historical driving data. Vehicle data includes at least one of vehicle speed data, tire pressure data, or battery level data. Application data includes at least one of navigation data, weather data, or multimedia data (such as the volume of multimedia applications). User historical driving data includes at least one of user driving habits or user historical driving routes.

[0059] In some embodiments, refer to Figure 2 The vehicle control unit obtains a structured sequence of real-time query information from the real-time information processing module. The real-time information processing module analyzes and processes data from onboard applications and / or data collected by onboard sensors to obtain a structured sequence indicating the real-time query information.

[0060] S102, the first structured sequence is input into the vehicle control model, and after processing by the vehicle control model, the first vehicle control command is obtained. The first vehicle control command is used to instruct the adjustment of the current vehicle's driving strategy.

[0061] In some embodiments, a vehicle control model is pre-configured in a vehicle control device. The vehicle control device inputs the aforementioned first structured sequence into the vehicle control model. After the vehicle control model analyzes and processes the first structured sequence, it obtains a first vehicle control instruction. The first vehicle control instruction may include at least one vehicle control instruction for instructing the adjustment of the current vehicle's driving strategy.

[0062] In some embodiments, the vehicle control device obtains a first vehicle control command from a cloud server by interacting with the cloud server. The cloud server includes a pre-configured vehicle control model. After obtaining a first structured sequence, the vehicle control device sends the first structured sequence to the cloud server. The cloud server inputs the first structured sequence into the vehicle control model and sends the first vehicle control command output by the model to the vehicle control device.

[0063] In some embodiments, the first vehicle control command is a structured control command.

[0064] For example, the first vehicle control instruction includes a vehicle control instruction, such as instructing the vehicle to change lanes to the rightmost lane of the current road. The first vehicle control instruction can be represented as: {"intent": "lane", "lane":3}, where intent represents the driver assistance intention / strategy, and lane represents the lane number, such as 3 representing the rightmost lane of the current road.

[0065] For example, the first vehicle control instruction includes two vehicle control instructions, such as instructing the vehicle to change lanes to the leftmost lane of the current road, and instructing the vehicle speed to increase to 80 kilometers per hour. The first vehicle control instruction can be represented as: {"intent": "lane change", "lane": 3}, {"intent": acceleration, "max": 80}, where max represents the maximum vehicle speed.

[0066] In some embodiments, the vehicle control model can be a machine learning model, including but not limited to neural network models such as recurrent neural networks (RNNs), long short-term memory networks (LSTM), and Transformer models. The vehicle control model can be trained based on multiple sets of training data, where each set of training data includes a structured sequence of at least two items indicating user information, road information, and real-time query information, as well as the corresponding vehicle control command (which can be a structured control command). In practical applications, the model parameters of the vehicle control model can be updated based on multiple sets of real-time data to continuously optimize the processing capabilities of the vehicle control model.

[0067] S103, execute or adjust the first vehicle control command.

[0068] In some embodiments, the vehicle control device directly executes the first vehicle control command based on the first vehicle control command.

[0069] In some embodiments, the vehicle control device determines whether to adjust the first vehicle control command based on the first vehicle control command and user feedback information regarding the first vehicle control command. If the user accepts the first vehicle control command, the vehicle control device executes the first vehicle control command. If the user does not accept the first vehicle control command, the vehicle control device does not execute the first vehicle control command.

[0070] Optionally, if the user does not accept the first vehicle control command, the vehicle control device may adjust the first vehicle control command based on user feedback (e.g., user feedback includes new voice control commands) and execute the adjusted vehicle control command. For instructions on how to adjust the first vehicle control command, please refer to [reference needed]. Figure 4 Example.

[0071] In the above scheme, various types of data from inside and outside the vehicle are collected and converted into a first structured sequence. This sequence is then input into a pre-configured vehicle control model. After processing by the vehicle control model, a first vehicle control command is output. This first vehicle control command is adapted to the collected data and provides the driver with a reasonable and precise driving strategy. By executing or adjusting the first vehicle control command, the level of intelligence in assisted driving is improved.

[0072] Based on the foregoing embodiments, in some embodiments, the first structured sequence includes a structured sequence for indicating voice control commands input by the user. When the vehicle control device receives the first vehicle control command output by the model, it can directly execute the first vehicle control command to realize assisted driving control based on user voice commands and provide users with personalized assisted driving functions.

[0073] The following is combined with Figure 2 and Figure 3 The vehicle control scheme of this embodiment will be described.

[0074] Figure 3 This is an interactive schematic diagram of a vehicle control method provided in an embodiment of this application. This vehicle control method can be applied to any vehicle control device, which may include a vehicle control module, a voice command analysis module, an in-vehicle video analysis module, and a real-time information processing module.

[0075] like Figure 3 As shown, the vehicle control method involves the interaction between various modules in the vehicle control device, and the method may include the following steps:

[0076] S301, the vehicle control module obtains a first standardized text sequence from the voice command analysis module. The first standardized text sequence is used to indicate the voice control command input by the user.

[0077] In one example, the voice command analysis module can obtain the first standardized text sequence through S3011-S3014.

[0078] S3011, the voice command analysis module receives voice information input by the user.

[0079] The voice command analysis module receives voice information input by the user from the vehicle's microphone.

[0080] S3012, the voice command analysis module converts the user's input voice information into text information corresponding to the voice information.

[0081] In one example, the voice command analysis module uses a speech recognition algorithm to convert the user's input voice information into text information corresponding to the voice information.

[0082] S3013, the voice command analysis module performs text analysis on the text information to obtain at least one text sequence.

[0083] In one example, the voice command analysis module uses a natural language processing model to analyze the text information and obtain at least one text sequence containing voice control commands. That is, at least one text sequence is used to indicate the voice control commands in the user-input voice information.

[0084] Text analysis includes, but is not limited to, text preprocessing (e.g., removing redundant data or punctuation) and text classification (e.g., identifying whether the text is related to vehicle control).

[0085] S3014, the voice command analysis module converts at least one text sequence into a first normalized text sequence, which is used to indicate the voice control command input by the user.

[0086] In one example, the voice command analysis module is pre-configured with a large language model (LLM). The LLM is a pre-trained model. The voice command analysis module can use the LLM to decompose and standardize at least one text sequence to obtain a first standardized text sequence. This example does not limit the model structure of the LLM.

[0087] S302, the vehicle control module obtains a second standardized text sequence from the on-board video analysis module. The second standardized text sequence is used to indicate road information.

[0088] Road information includes at least one of the following: lane information on the road on which the vehicle travels, or vehicle information in the lane, as can be seen in the foregoing embodiments.

[0089] In one example, the vehicle-mounted video analysis module can obtain a second standardized text sequence via S3021-S3023.

[0090] S3021, the vehicle-mounted video analysis module extracts image frames from the external video data collected by the camera.

[0091] S3022, the vehicle video analysis module performs semantic segmentation on the extracted image frames to obtain semantic segmentation results, which are used to indicate road information.

[0092] In one example, the vehicle video analysis module is pre-configured with an image processing model, which is a pre-trained model. The vehicle video analysis module can obtain the semantic segmentation results corresponding to the extracted image frames through the image processing model.

[0093] Optionally, the semantic segmentation results include the category of each image region in the image frame (e.g., region 1 in the image frame is identified as a road, region 2 as a road sign, region 3 as vegetation, region 4 as a vehicle, etc.), and the location of each image region.

[0094] This example does not limit the model structure of the image processing model. For example, the image processing model can use convolutional neural networks (CNN) or more advanced segmentation networks (such as U-NET).

[0095] S3023, the vehicle video analysis module converts the semantic segmentation results into a second standardized text sequence, which is used to indicate road information.

[0096] In one example, the vehicle video analysis module uses a natural language processing model to analyze the semantic segmentation results and generate a second standardized text sequence corresponding to the semantic segmentation results.

[0097] In some embodiments, the vehicle control module may also input the semantic segmentation results into the vehicle control model. The semantic segmentation results may be image data containing annotation information, such as annotations of the categories of different regions in the image.

[0098] S303, the vehicle control module obtains a third standardized text sequence from the real-time information processing module. The third standardized text sequence is used to indicate real-time query information.

[0099] Real-time query information includes at least one of vehicle data, application data, or user historical driving data, as detailed in the aforementioned embodiments.

[0100] In one example, the real-time information processing module can obtain the third standardized text sequence via S3031-S3032.

[0101] S3031, the real-time information processing module obtains at least one of the following information: vehicle data, application data, or user historical driving data.

[0102] In one example, the real-time information processing module acquires vehicle data from onboard sensors. Optionally, the onboard sensors include, but are not limited to, those shown above.

[0103] In one example, the real-time information processing module obtains real-time application data from in-vehicle applications. These in-vehicle applications include, but are not limited to, navigation applications, weather applications, and multimedia applications.

[0104] For example, if a user queries a navigation route through a navigation application, the real-time information processing module can obtain navigation data, including the route, from the navigation application. The real-time information processing module can also obtain real-time weather data from a weather application. Furthermore, it can obtain data such as the volume of music currently playing in the vehicle from a multimedia application.

[0105] It should be noted that the navigation data, weather data, and volume data obtained by the real-time information processing module are only examples. In practical applications, more data from other in-vehicle applications can be obtained as the basis for driver assistance control.

[0106] In one example, the real-time information processing module retrieves the user's historical driving data from an onboard database. The onboard database pre-stores the historical driving data of at least one driver. It is understood that different drivers have different driving habits and routes. For instance, the driver information of the currently driving vehicle can be determined by collecting the driver's facial image or personal information entered by the driver (such as a driver identification ID), and then the historical driving data corresponding to the driver of the currently driving vehicle can be retrieved from the onboard database as basic data for assisted driving control.

[0107] S3032, the real-time information processing module converts at least one of vehicle data, application data, or user historical driving data into a third standardized text sequence.

[0108] In one example, the real-time information processing module uses a natural language processing model to analyze at least one of vehicle data, application data, or user historical driving data to generate a third standardized text sequence.

[0109] It should be noted that the execution order of S301 to S303 is not limited, and they can be executed simultaneously or sequentially. This embodiment does not impose any restrictions on this.

[0110] S304, the vehicle control module inputs the first standardized text sequence, the second standardized text sequence, and the third standardized text sequence into the vehicle control model. After processing by the vehicle control model, a first vehicle control command is obtained. The first vehicle control command is used to instruct the adjustment of the current vehicle's driving strategy.

[0111] Combination Figure 1 In this embodiment, the first structured sequence input to the vehicle control model includes: a first standardized text sequence, a second standardized text sequence, and a third standardized text sequence.

[0112] It should be noted that in some embodiments, the first structured sequence may include at least two of the first normalized text sequence, the second normalized text sequence, and the third normalized text sequence. That is, the first structured sequence is used to indicate at least two of the user-input voice information, road information, and real-time query information. For details, please refer to the scenario examples below.

[0113] In one example, the vehicle control module has a pre-configured vehicle control model.

[0114] S305, the vehicle control module sends a first vehicle control command to the advanced driver assistance system (ADS) so that the ADS controls the vehicle's movement based on the first vehicle control command.

[0115] In this example, the vehicle control module directly controls the ADS to execute the first vehicle control command.

[0116] Since the input of the vehicle control model includes the user's voice commands, and the model also incorporates other types of data, such as the aforementioned road information and real-time query information, the first vehicle control command output by the model integrates the user's voice control intent. Therefore, it can be used as the input of the ADS without user confirmation and be actually executed by the ADS.

[0117] By implementing the above solution, users can use more common and generalized voice commands to control vehicle driving. Voice commands are no longer singular, and multiple voice commands can be input at once. After analysis and processing by the vehicle control model, the accuracy of vehicle assisted driving can be improved.

[0118] Optionally, in some embodiments, such as Figure 3 As shown, before the vehicle control module sends the first vehicle control command to the ADS, it also performs the following:

[0119] S306, The vehicle control module outputs a prompt message, which is used to indicate the first vehicle control command.

[0120] The vehicle control module can output prompts via voice broadcast and / or interface display.

[0121] In one example, the vehicle control module broadcasts prompts via an in-vehicle voice assistant.

[0122] In one example, the vehicle control module pushes prompts via the in-vehicle display screen. For instance, a prompt appears on the in-vehicle display screen interface, and the user can view the pushed driver assistance strategy through the display screen.

[0123] In one example, the vehicle control module simultaneously broadcasts a prompt via the in-vehicle voice assistant and pushes the prompt to the in-vehicle display screen. For instance, a prompt window can pop up on the in-vehicle display screen, displaying the content of the first vehicle control command for user confirmation.

[0124] S307, User feedback information obtained by the vehicle control module from the prompt information.

[0125] In one example, the user inputs feedback information via voice; that is, the user's feedback information is voice information. (See reference...) Figure 3 The vehicle control module obtains user feedback information from the voice command analysis module. The voice command analysis module receives and recognizes the user feedback information.

[0126] In one example, the user inputs feedback information via the vehicle's display screen. For instance, the user clicks the "Confirm" button on the prompt window to indicate agreement to execute the first vehicle control command. Alternatively, the user clicks the "Cancel" button on the prompt window to indicate rejection of the first vehicle control command.

[0127] If the user feedback information indicates acceptance of the first vehicle control command, then S305 is executed.

[0128] If the user feedback information indicates that the first vehicle control command is not accepted, then S305 will not be executed.

[0129] By implementing the above scheme, the vehicle control device will execute the first vehicle control command only after user confirmation, which improves interactivity and accuracy of vehicle control, and avoids the problem that the vehicle control strategy does not conform to the user's intention due to inaccurate voice recognition.

[0130] The following section will analyze several specific application scenarios. Figure 3 The vehicle control scheme shown is illustrated with an example.

[0131] Scenario 1: The vehicle is currently traveling on a highway. The third lane has the fewest vehicles, and the driver feels that the vehicle is moving slowly.

[0132] The driver inputs the voice message: "Change lanes to the lane with the fewest vehicles."

[0133] The vehicle control process includes:

[0134] Voice command analysis module: Performs voice analysis and processing on the voice data collected by the vehicle microphone, and outputs a first standardized text sequence. The first standardized text sequence is used to indicate the user's input voice control command "Change lanes to the lane with the fewest vehicles".

[0135] The vehicle-mounted video analysis module analyzes the video data of the vehicle's exterior captured by cameras and outputs a second normalized text sequence. This sequence indicates the number of vehicles in each lane of the road the vehicle is currently traveling on. For example, if there are three lanes on the road, the second normalized sequence can be represented as {"Lane 1:3", "Lane 2:1", "Lane 3:0"}, where there are three vehicles in lane 1, one vehicle in lane 2, and no vehicle in lane 3. The lanes in the second normalized text sequence can be ordered from left to right according to their position. For example, the left and right lanes of lane 2 are lane 1 and lane 3, respectively.

[0136] The vehicle control module inputs the first standardized text sequence and the second standardized text sequence into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command (which can correspond to the aforementioned first vehicle control command): [{"intent":"lane","lane":3}], indicating a lane change to lane 3.

[0137] Scenario 2: The driver feels that the vehicle in front is moving slowly and wants to speed up while alerting the vehicle in front.

[0138] The driver's voice message: "The car in front is too slow, warn it and help me take off."

[0139] The vehicle control process includes:

[0140] Voice command analysis module: Performs voice analysis and processing on the voice data collected by the vehicle microphone, and outputs a first standardized text sequence. The first standardized text sequence is used to instruct the user to input the voice control command "overtake the vehicle in front, turn on hazard lights, accelerate".

[0141] The vehicle-mounted video analysis module analyzes the video data of the vehicle's exterior captured by the camera and outputs a second standardized text sequence. This second standardized text sequence indicates the vehicle status in each lane of the road the vehicle is currently traveling on. For example, if there are three lanes on the road, the second standardized sequence can be represented as {"Lane 1:3", "Lane 2:1", "Lane 3:0"}.

[0142] The vehicle control module inputs the first and second standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: [{"intent":"overtake","current_lane":2","final_lane":3},{"intent":"hazard lights","switch":true},{"intent":"accelerate","max":80}], indicating that the vehicle changes lanes from lane 2 to lane 3, activates the hazard lights, and controls the speed at 80 kilometers per hour.

[0143] Scene 3: The driver is driving near the scenic area. The scenery along the way is beautiful, and the driver wants to take pictures.

[0144] The driver entered the following voice message: "I want to take some photos later. Could you slow down a bit when we pass some scenic spots so I can take a couple?"

[0145] The vehicle control process includes:

[0146] Voice command analysis module: Performs voice analysis and processing on the voice data collected by the vehicle microphone, and outputs a first standardized text sequence. The first standardized text sequence is used to instruct the user to input the voice control command "Slow down when passing through scenic areas, then resume speed".

[0147] Real-time information processing module: Obtains navigation data from the navigation application, including the location of the upcoming attraction. It acquires the current vehicle speed, vehicle position (e.g., 60 km / h), and a suitable speed for taking photos (e.g., 30 km / h), where the suitable speed for taking photos can be pre-stored in the vehicle database. After acquiring the current speed, vehicle position, and the location of the upcoming attraction, it obtains the following information: at the current speed, there is a first time (denoted as $time1) remaining to reach the attraction, and a second time ($time2) remaining to leave the attraction. After processing by the real-time information processing module, it outputs a third normalized text sequence, which indicates "at the current speed, there is a first time (denoted as $time1) remaining to reach the attraction, and a second time ($time2) remaining to leave the attraction."

[0148] The vehicle control module inputs the first and third standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: [{"intent":"decelerate","speed":30,"time":$time1},{"intent":"accelerate","speed":60,"time":$time2}], which means that the vehicle speed will be reduced to 30 km / h within the first time period, and then accelerate to 60 km / h after the second time period.

[0149] Based on the foregoing embodiments, in some embodiments, the first structured sequence does not include a structured sequence for indicating voice control commands input by the user, that is, the first structured sequence does not include a first standardized text sequence. The vehicle control device can input structured sequences of other types of data (such as user facial expressions, external environment data, navigation data queried by the user, weather data, vehicle status, etc.) into the vehicle control model, obtain the first vehicle control command output by the model, and recommend suitable assisted driving strategies, driving methods and driving routes to the user for selection, thereby improving the user's driving experience.

[0150] The following is combined with Figure 2 and Figure 4 The vehicle control scheme of this embodiment will be described.

[0151] Figure 4 This is an interactive schematic diagram illustrating another vehicle control method provided in an embodiment of this application. This vehicle control method can be applied to any vehicle control device. Figure 3 The difference in the embodiments is that, in addition to including a vehicle control module, a voice command analysis module, an in-vehicle video analysis module, and a real-time information processing module, the vehicle control device also includes... Figure 2 The user status analysis module shown.

[0152] like Figure 4 As shown, the vehicle control method involves the interaction between various modules in the vehicle control device, and the method may include the following steps:

[0153] S401, the vehicle control module obtains the fourth standardized text sequence from the user state analysis module. The fourth standardized text sequence is used to indicate the user's emotional state.

[0154] In some embodiments, the fourth standardized text sequence may also be referred to as a structured sequence of user status information.

[0155] In one example, the user state analysis module can obtain the fourth normalized text sequence via S4011-S4013:

[0156] S4011, The user state analysis module obtains the tone information of the user's voice input.

[0157] In one example, the user state analysis module is pre-configured with a speech tone analysis model. The user state analysis module inputs speech data into the speech tone analysis model, and after processing by the model, it obtains the user's tone information from the speech data. For example, the tone information includes scores for various tones.

[0158] The speech tone analysis model is a pre-trained model, and this example does not impose any restrictions on the model structure of the speech tone analysis model.

[0159] S4012, The user state analysis module obtains the user's emotional information from the image data captured by the camera.

[0160] In one example, the user state analysis module is pre-configured with an emotion analysis model. The user state analysis module inputs image data captured by the camera into the emotion analysis model. After processing by the emotion analysis model, the user's emotion information in the image data is obtained. For example, the emotion information includes scores for various emotions.

[0161] It should be noted that the emotion analysis model can analyze not only a user's facial emotions in image data, but also the user's body language in image data. Combining facial emotions and body language, it outputs the user's emotional information in the image data. That is to say, in this embodiment, the user's emotional information includes the user's body language and / or facial emotions.

[0162] The sentiment analysis model is a pre-trained model, and this example does not impose any restrictions on the model structure of the sentiment analysis model.

[0163] S4013, the user state analysis module obtains the fourth standardized text sequence based on tone information and emotion information.

[0164] In some embodiments, the user state analysis module obtains a fourth standardized text sequence based on tone and emotion information, preset weight values ​​for various tones, and weight values ​​for various emotions (including facial and / or body language emotions). The fourth standardized text sequence includes scores for various emotions.

[0165] In one example, the user state analysis module determines the scores for various emotions based on the scores of different tones in the tone information, the scores of different facial emotions and different body emotions in the emotion information, and the weight values ​​corresponding to different tones, facial emotions, and body emotions. Because it integrates emotional features such as tone, facial expressions, and body language, the output fourth standardized text sequence indicates a more accurate user emotional state, providing data support for subsequent assisted driving.

[0166] For example, the fourth standardized text sequence can be represented as {“happy”: 0.24, “doubtful”: 0.25, “peaceful”: 0.19, “angry”: 0.02, “furious”: 0.3}, where the higher the emotion score, the more obvious or intense the emotion is.

[0167] In this embodiment, the scores of various emotions in the fourth standardized text sequence are obtained based on the user's voice tone, body language, and facial features, providing data support for subsequent assisted driving recommendations.

[0168] In some embodiments, the user state analysis module acquires the user's emotional information from image data captured by the camera, and generates a fourth standardized text sequence based on the user's emotional information. In this embodiment, the scores for various emotions in the fourth standardized text sequence are obtained based on the user's body language and facial features.

[0169] In some embodiments, the user state analysis module obtains the tone information of the user's voice input and generates a fourth standardized text sequence based on the user's tone information. In this embodiment, the scores of various emotions in the fourth standardized text sequence are obtained based on the user's voice tone.

[0170] It should be noted that user status information is not limited to the driver's status information, but may also include the status information of other passengers in the vehicle.

[0171] S402, the vehicle control module obtains a second standardized text sequence from the on-board video analysis module. The second standardized text sequence is used to indicate road information.

[0172] S403, the vehicle control module obtains a third standardized text sequence from the real-time information processing module. The third standardized text sequence is used to indicate real-time query information.

[0173] S402 and S403 in this embodiment can be referred to as S302 and S303 in the previous embodiment, respectively, and will not be repeated here.

[0174] It should be noted that the execution order of S401 to S403 is not limited, and they can be executed simultaneously or sequentially. This embodiment does not limit this.

[0175] S404, the vehicle control module inputs the second, third, and fourth standardized text sequences into the vehicle control model. After processing by the vehicle control model, a first vehicle control command is obtained. The first vehicle control command is used to instruct the adjustment of the current vehicle's driving strategy.

[0176] Combination Figure 1 In this embodiment, the first structured sequence input to the vehicle control model includes: a second standardized text sequence, a third standardized text sequence, and a fourth standardized text sequence.

[0177] It should be noted that in some embodiments, the first structured sequence may include at least two of the second, third, and fourth standardized text sequences, that is, the first structured sequence is used to indicate at least two of the user status information, road information, and real-time query information, as can be seen in the scenario examples below.

[0178] S405, the vehicle control module outputs a prompt message, which is used to indicate the first vehicle control command.

[0179] S406, User feedback information obtained by the vehicle control module from the prompt information.

[0180] S405 and S406 of this embodiment can be referred to as S306 and S307 of the previous embodiment, respectively, and will not be repeated here.

[0181] If the user feedback information indicates acceptance of the first vehicle control command, then S407 is executed.

[0182] If the user feedback information indicates that the first vehicle control command is not accepted, S408 can be executed.

[0183] S407, the vehicle control module sends a first vehicle control command to the ADS, so that the ADS controls the vehicle to drive based on the first vehicle control command.

[0184] S408, the vehicle control module adjusts the first vehicle control command.

[0185] In one example, the vehicle control module can obtain a new vehicle control command (i.e., a second vehicle control command) via S4081-S4082:

[0186] S4081, the vehicle control module obtains the fifth standardized text sequence from the voice command analysis module. The fifth standardized text sequence is used to indicate the voice control commands in the user feedback information.

[0187] In this embodiment, the user feedback information is voice information. The principle of the voice command analysis module obtaining the fifth standardized text sequence can be referred to S3011 to S3014 of the aforementioned embodiment, and will not be elaborated here.

[0188] S4082, the vehicle control module inputs the second structured sequence into the vehicle control model, processes the vehicle control model, and obtains the second vehicle control command. The second structured sequence includes the fifth standardized text sequence.

[0189] Optionally, the second structured sequence may also include structured sequences of other types of data.

[0190] In some embodiments, after S408, the following is also performed:

[0191] S409, execute the second vehicle control command.

[0192] In some embodiments, the vehicle control module determines a second vehicle control command through multiple rounds of interaction with the user.

[0193] By implementing the above scheme, the vehicle control device proactively pushes assisted driving strategies to the user by acquiring user status, road information, and real-time query information. This means that even in scenarios where the user hasn't input voice commands, the vehicle can sense various types of data and push a first vehicle control command suitable for the current driving scenario for the user to choose from. After at least one round of command confirmation, the user-selected vehicle control command is executed. This increases cockpit interactivity while improving the accuracy of assisted driving, meeting the driving needs of different users and being more user-friendly for novice drivers.

[0194] The following section will analyze several specific application scenarios. Figure 4 The vehicle control scheme shown is illustrated with an example.

[0195] Scenario 4: The car in front is driving slowly, but the driver is a novice and dares not change lanes to overtake.

[0196] The vehicle control process includes:

[0197] User Status Analysis Module: Performs frame-by-frame analysis on the video data inside the vehicle captured by the camera (e.g., analyzes the driver's facial emotions in the image frames using an emotion analysis model) to obtain a fourth standardized text sequence. The fourth standardized text sequence is used to indicate the following information: "The driver keeps turning his head, looks anxious, and has the intention to change lanes."

[0198] Vehicle video analysis module: Analyzes video data of the vehicle's exterior captured by the camera and outputs a second standardized text sequence. The second standardized text sequence is used to indicate the vehicle situation (such as the number of vehicles, the speed of the vehicle in front, etc.) and the vehicle's position (e.g., the current vehicle is in lane 2) in each lane on the road where the vehicle is currently traveling.

[0199] In this example, the lanes from left to right are lane 1, lane 2, and lane 3, and there is no car in lane 1.

[0200] The vehicle control module inputs the fourth standardized text sequence and the second standardized text sequence into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: [{"intent":"lane","lane":1}], indicating a lane change to lane 1.

[0201] The vehicle control module uses the in-vehicle voice assistant to play a prompt message such as, "The car in front is a bit slow, but the right lane is empty. Do you need help passing it?" The in-vehicle voice assistant can optimize the content of the structured control commands output by the model to obtain the prompt message, avoiding overly abrupt or abrupt messages.

[0202] The driver entered the user feedback message: "Okay".

[0203] Based on user feedback, the vehicle control module sends structured control commands to the ADS, which then controls the vehicle to change lanes to lane 1.

[0204] Scenario 5: The outdoor weather conditions are good, and the driver is in a good mood.

[0205] The vehicle control process includes:

[0206] User state analysis module: Analyzes video data inside the vehicle captured by the camera frame by frame to obtain a fourth standardized text sequence. The fourth standardized text sequence is used to indicate that the driver's emotional state is pleasant.

[0207] Real-time information processing module: This module acquires the current vehicle speed (40 km / h) and the maximum speed corresponding to a pleasant emotional state (e.g., 60 km / h). The maximum speeds corresponding to different emotional states can be pre-stored in the vehicle database. After processing by the real-time information processing module, a third standardized text sequence is output, indicating "The current speed is 40 km / h, and can be increased to 60 km / h."

[0208] The vehicle control module inputs the fourth and third standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: {"intent":"accelerate","max":60}, indicating acceleration to 60 kilometers per hour.

[0209] The vehicle control module plays a prompt message through the in-vehicle voice assistant: "The current road conditions are good, the weather is nice, do you need to drive faster?"

[0210] The driver entered the user feedback message: "No need".

[0211] Based on user feedback, the vehicle control module does not send structured control commands to the ADS, and the ADS does not execute vehicle acceleration. Optionally, the vehicle control module records relevant data for this process (including whether acceleration is unnecessary when the driver is driving in good weather).

[0212] Scenario 6: Heavy rain is falling, the navigation shows congestion ahead, and the video shows obvious queues in lane 1 and lane 2.

[0213] Real-time information processing module: Obtains navigation data from the navigation application, processes it, and outputs a third standardized text sequence, which is used to indicate congestion ahead of the vehicle.

[0214] The vehicle-mounted video analysis module analyzes the video data of the vehicle's exterior captured by the camera and outputs a second standardized text sequence. This sequence indicates vehicles with illuminated brake lights in lanes 1 and 2, as well as their positions (e.g., the vehicle is currently in lane 1). In this example, the lanes from left to right are lane 1, lane 2, and lane 3.

[0215] The vehicle control module inputs the third and second standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs structured control instructions: [{"intent":"lane","lane":2},{"intent":"lane","lane":3}], which means first change lanes to lane 2, and then change lanes to lane 3.

[0216] The vehicle control module plays a prompt message through the in-vehicle voice assistant: "It's raining heavily right now. Vehicles in lanes 1 and 2 ahead are moving slowly. An accident is estimated. Do you need me to help you change lanes in advance?"

[0217] The driver entered the following user feedback message: "Okay, please hurry up."

[0218] Based on user feedback (which also indicates "accelerate"), the vehicle control module adjusts the structured control command to: [{"intent":"lane change","lane":2},{"intent":"lane change","lane":3},{"intent":"accelerate","speed":60}], indicating that the vehicle first changes lanes to lane 2, then to lane 3, and the speed during the lane change is increased to 60 kilometers per hour.

[0219] Scenario 7: The outdoor temperature is hot and the sun is strong, and the driver's facial expression shows discomfort.

[0220] Real-time information processing module: Obtains the current outdoor temperature (e.g., 38 degrees Celsius) from the weather application or vehicle temperature sensor, and determines that the current in-vehicle air conditioning temperature is set to 20 degrees Celsius. After processing by the real-time information processing module, it outputs a third standardized text sequence, which is used to indicate the indoor and outdoor temperature data.

[0221] The in-vehicle video analysis module analyzes the video data of the vehicle's exterior captured by the camera and outputs a second standardized text sequence. This sequence indicates "the sunlight is currently strong, lanes 1 and 2 are in sunlight, and lane 3 is in shadow" and the vehicle's position (e.g., the vehicle is currently in lane 2). In this example, the lanes from left to right are lane 1, lane 2, and lane 3.

[0222] User state analysis module: Analyzes the video data inside the vehicle captured by the camera frame by frame to obtain the fourth standardized text sequence. The fourth standardized text sequence is used to indicate that the driver's emotional state is agitated.

[0223] The vehicle control module inputs the third, second, and fourth standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: [{"intent":"lane change","lane":3},{"intent":"accelerate","speed":80}], indicating that the vehicle changes lanes to lane 3 and the speed increases to 80 kilometers per hour during the lane change.

[0224] The vehicle control module plays a prompt message through the in-vehicle voice assistant: "In order to avoid being in the sun for too long, do you need me to drive faster and choose a more shady lane?"

[0225] The driver entered the user feedback message: "Okay".

[0226] Based on user feedback, the vehicle control module sends structured control commands to the ADS, which then controls the vehicle to accelerate and change lanes.

[0227] Scene 8: It's raining heavily outside the car, and music is playing inside, but the heavy rain makes it hard to hear clearly.

[0228] The real-time information processing module obtains the user-set volume (e.g., maximum volume) from multimedia applications (e.g., music applications), weather data (e.g., it's raining heavily outside) from weather applications, the actual volume played by the vehicle's infotainment system after echo cancellation, and vehicle speed data (e.g., current speed is 80 km / h). After processing by the real-time information processing module, it outputs a third normalized text sequence, which indicates the user-set volume, the actual played volume, the weather data, and the vehicle speed data.

[0229] User Status Analysis Module: Analyzes the voice tone of the voice data collected by the in-vehicle microphone to obtain the fourth standardized text sequence. The fourth standardized text sequence is used to indicate that the driver's voice tone is complaining that the voice is not loud enough and that the song cannot be heard clearly (i.e., indicating emotional state and voice content).

[0230] The vehicle control module inputs the third and fourth standardized text sequences into the vehicle control model. After processing by the vehicle control model, the model outputs a structured control command: [{"intent":"decelerate","speed":40}], which means decelerating to 40 kilometers per hour.

[0231] The vehicle control module plays a prompt message through the in-vehicle voice assistant: "Do you need me to take over and adjust the speed to a suitable range so you can better enjoy the music?"

[0232] The driver entered the user feedback message: "Okay".

[0233] Based on user feedback, the vehicle control module sends structured control commands to the ADS, which then controls the vehicle to decelerate.

[0234] Scenario 9: The tire pressure of a vehicle is constantly dropping on the highway.

[0235] The real-time information processing module acquires vehicle tire pressure data (e.g., tire pressure continuously decreasing) and vehicle speed data (e.g., current speed is 120 km / h), as well as navigation data from the navigation application (e.g., navigation data shows a nearby rest area). After processing by the real-time information processing model, it outputs a third normalized text sequence, which is used to indicate tire pressure data, vehicle speed data, and navigation data.

[0236] User Status Analysis Module: Analyzes the video data inside the vehicle captured by the camera frame by frame to obtain a fourth standardized text sequence. The fourth standardized text sequence is used to indicate that the driver is in a good mood and has not noticed any abnormalities in the vehicle.

[0237] Vehicle video analysis module: Analyzes video data of the vehicle's exterior captured by the camera and outputs a second standardized text sequence. The second standardized text sequence is used to indicate the vehicle situation and position in each lane of the current vehicle's driving road (e.g., the current vehicle is in the leftmost lane 1).

[0238] The vehicle control module inputs the third, second, and fourth standardized text sequences into the vehicle control model. After processing, the model outputs structured control commands: [{"intent":"decelerate","speed":40},{"navigation":"addWayPoint","destination":"A rest area"},{"intent":"lane change","lane":2},{"intent":"lane change","lane":3},{"intent":"lane change","lane":4}], which respectively represent: decelerate to 40 km / h, add the waypoint "A rest area" to the navigation route, change lanes to lane 2, change lanes to lane 3, and change lanes to lane 4 (i.e., change lanes sequentially from the current lane 1 to lane 4). In this example, the lanes from left to right are lane 1, lane 2, lane 3, and lane 4.

[0239] The vehicle control module plays a prompt message through the in-vehicle voice assistant: "Attention! Your vehicle may have a tire blowout. Would you like us to drive you safely to a repair shop?"

[0240] The driver entered the user feedback message: "Okay".

[0241] Based on user feedback, the vehicle control module sends structured control commands to the ADS, which then controls the vehicle to decelerate, change lanes to the leftmost lane of the current road, and head towards the new waypoint.

[0242] This application provides a vehicle control device. Figure 5 This is a schematic diagram of the structure of a vehicle control device provided in an embodiment of this application, as shown below. Figure 5 As shown, the vehicle control device 500 includes: an acquisition module 501, a processing module 502, and a control module 503.

[0243] The acquisition module 501 is used to acquire a first structured sequence, which is used to indicate at least two items of user information, road information, or real-time query information.

[0244] The processing module 502 is used to input the first structured sequence into the vehicle control model, and after processing by the vehicle control model, obtain the first vehicle control command; the first vehicle control command is used to instruct the adjustment of the current vehicle's driving strategy.

[0245] The control module 503 is used to execute or adjust the first vehicle control command.

[0246] In one optional embodiment, the first vehicle control command is a structured control command.

[0247] In one optional embodiment, the user information includes voice information; the acquisition module 501 is used to convert the voice information input by the user into text information corresponding to the voice information; perform text analysis on the text information to obtain at least one text sequence; convert the at least one text sequence into a first standardized text sequence; the first standardized text sequence is used to indicate the voice control command input by the user;

[0248] The first structured sequence includes the first normalized text sequence.

[0249] In one example, the acquisition module 501 includes Figure 2 The voice command analysis module shown.

[0250] In one optional embodiment, the road information includes at least one of lane information on the road on which the vehicle is traveling, or vehicle information in the lane; the acquisition module 501 is used to extract image frames from the video data outside the vehicle collected by the camera, perform semantic segmentation on the extracted image frames to obtain a semantic segmentation result; convert the semantic segmentation result into a second standardized text sequence; the semantic segmentation result is used to indicate road information; the second standardized text sequence is used to indicate road information; the first structured sequence includes the second standardized text sequence.

[0251] In one example, the acquisition module 501 includes Figure 2 The vehicle-mounted video analysis module shown.

[0252] In one optional embodiment, the real-time query information includes at least one of vehicle data, application data, or user historical driving data. The acquisition module 501 is configured to acquire vehicle data from onboard sensors; and / or acquire application data queried by the user from an onboard application; and / or acquire user historical driving data from an onboard database; and convert at least one of the vehicle data, application data, or user historical driving data into a third standardized text sequence; the third standardized text sequence is used to indicate the real-time query information; the first structured sequence includes the third standardized text sequence.

[0253] In one example, the acquisition module 501 includes Figure 2 The real-time information processing module 502 shown is shown.

[0254] In one alternative embodiment, the vehicle data includes at least one of vehicle speed data, tire pressure data, or battery level data.

[0255] In one alternative embodiment, the application data includes at least one of navigation data, weather data, or multimedia data.

[0256] In one alternative embodiment, the user's historical driving data includes at least one of the user's driving habits or historical driving routes.

[0257] In one optional embodiment, the user information includes user status information; the acquisition module 501 is used to acquire tone information of the user's voice input and the user's emotion information in the image data captured by the camera; a fourth standardized text sequence is obtained based on the tone information and emotion information; the fourth standardized text sequence is used to indicate the user status information; the first structured sequence includes the fourth standardized text sequence.

[0258] In one example, the acquisition module 501 includes Figure 2 The user status analysis module shown.

[0259] In one optional embodiment, the control module 503 is configured to output prompt information via voice broadcast and / or interface display, the prompt information being used to indicate a first vehicle control command.

[0260] The acquisition module 501 is used to receive user feedback information from the prompt information;

[0261] Control module 503 is configured to execute the first vehicle control command if user feedback indicates acceptance of the first vehicle control command; or

[0262] If the user feedback indicates that the first vehicle control command is not accepted, the first vehicle control command should be adjusted.

[0263] In an optional embodiment, the acquisition module 501 is further configured to acquire a fifth standardized text sequence, which is used to indicate the voice control commands in the user feedback information.

[0264] The processing module 502 is also used to input the second structured sequence into the vehicle control model, and after processing by the vehicle control model, obtain the second vehicle control command; the second structured sequence includes the fifth standardized text sequence; the second vehicle control command is the adjusted vehicle control command;

[0265] The control module 503 is also used to execute second vehicle control commands.

[0266] In one example, the acquisition module 501 includes Figure 2 The voice command analysis module shown.

[0267] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0268] This application provides a vehicle control device, which includes one or more processors and a memory; the memory is coupled to one or more processors and is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the vehicle control device to perform the method shown in the foregoing embodiments.

[0269] Figure 6 This is a schematic diagram of another vehicle control device provided in an embodiment of this application. Figure 6 As shown, the vehicle control device 600 includes: a processor 601, a memory 602, and a bus 603.

[0270] The memory 602 is used to store the computer program code of the processor 601; the processor 601 is configured to perform the methods shown in the foregoing embodiments by executing the computer program code.

[0271] Optionally, the memory 602 can be either standalone or integrated with the processor 601.

[0272] Optionally, memory 602 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0273] The memory 602 is connected to the processor 601 via bus 603, enabling communication between them. Bus 603 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus.

[0274] The methods described in the embodiments of this application can be applied to, or implemented by, processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 601 or by instructions in software form. Processor 601 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0275] This application provides a chip or chip system. The chip or chip system can be applied to a vehicle control device. The chip or chip system includes one or more processors, which invoke computer instructions to cause the vehicle control device to perform the methods shown in the foregoing embodiments. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0276] This application also provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions that, when executed on a vehicle control device, cause the vehicle control device to perform the method shown in the foregoing embodiments.

[0277] The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted on a computer-readable medium. A computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. A storage medium can be any target medium accessible by a computer.

[0278] In one possible implementation, a computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact discread-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium targeted to carry or to store required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0279] This application provides a computer program product, which includes computer program code. When the computer program code is run on a vehicle control device, it causes the vehicle control device to perform the method shown in the foregoing embodiments.

[0280] It should be noted that the modules or components described in the above embodiments can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), etc. Furthermore, when a module is implemented through processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors capable of calling program code, such as a controller. Additionally, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0281] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0282] The term "multiple" in this document refers to two or more. The term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the preceding and following related objects; in formulas, " / " indicates a "division" relationship. Additionally, it should be understood that in the description of this application, words such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0283] It should be noted that the user information (including but not limited to user device information, user personal information, such as driver ID) and data (including but not limited to data used for analysis, stored data, and displayed data, including image data, voice data, and video data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0284] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0285] It is understood that, in the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

Claims

1. A vehicle control method, characterized in that, The method includes: Obtain a first structured sequence, wherein the first structured sequence is used to indicate at least two items of user information, road information, or real-time query information; The first structured sequence is input into the vehicle control model, and after processing by the vehicle control model, a first vehicle control command is obtained; the first vehicle control command is used to instruct the adjustment of the current driving strategy of the vehicle. Execute or adjust the first vehicle control command.

2. The method according to claim 1, characterized in that, The first vehicle control command is a structured control command.

3. The method according to claim 1 or 2, characterized in that, The user information includes voice information; obtaining the first structured sequence includes: Convert the user's voice input into text information corresponding to the voice input; The text information is analyzed to obtain at least one text sequence; The at least one text sequence is converted into a first normalized text sequence; the first normalized text sequence is used to indicate the voice control command input by the user. The first structured sequence includes the first standardized text sequence.

4. The method according to any one of claims 1 to 3, characterized in that, The road information includes at least one of the following: lane information on the road on which the vehicle is traveling, or vehicle information in the lane; The step of obtaining the first structured sequence includes: Image frames are extracted from the video data of the vehicle exterior captured by the camera, and semantic segmentation is performed on the extracted image frames to obtain semantic segmentation results; the semantic segmentation results are used to indicate the road information. The semantic segmentation result is converted into a second normalized text sequence; the second normalized text sequence is used to indicate the road information. The first structured sequence includes the second normalized text sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The real-time query information includes at least one of vehicle data, application data, or user historical driving data; The step of obtaining the first structured sequence includes: The vehicle data is acquired from the vehicle sensors; and / or, the application data queried by the user is acquired from the vehicle application; and / or, the user's historical driving data is acquired from the vehicle database. At least one of the vehicle data, the application data, or the user's historical driving data is converted into a third standardized text sequence; the third standardized text sequence is used to indicate the real-time query information. The first structured sequence includes the third standardized text sequence.

6. The method according to claim 5, characterized in that, The vehicle data includes at least one of vehicle speed data, tire pressure data, or battery level data; the application data includes at least one of navigation data, weather data, or multimedia data; and the user's historical driving data includes at least one of user driving habits or historical driving routes.

7. The method according to any one of claims 1 to 6, characterized in that, The user information includes the user status information; obtaining the first structured sequence includes: It acquires tone information from the user's voice input and emotional information from the image data captured by the camera. A fourth standardized text sequence is obtained based on the tone information and the emotion information; the fourth standardized text sequence is used to indicate user status information. The first structured sequence includes the fourth standardized text sequence.

8. The method according to any one of claims 1 to 7, wherein the method further comprises: Prompt information is output through voice broadcast and / or interface display, and the prompt information is used to indicate the first vehicle control command; User feedback information upon receiving the aforementioned prompt; The execution or adjustment of the first vehicle control command includes: If the user feedback information is used to indicate acceptance of the first vehicle control command, then execute the first vehicle control command; or If the user feedback information indicates that the first vehicle control command is not accepted, the first vehicle control command is adjusted.

9. The method according to claim 8, characterized in that, If the user feedback information includes a voice control command input by the user, adjusting the first vehicle control command includes: Obtain a fifth standardized text sequence, which is used to indicate the voice control command in the user feedback information; The second structured sequence is input into the vehicle control model, and after processing by the vehicle control model, a second vehicle control command is obtained; the second structured sequence includes the fifth standardized text sequence. The second vehicle control command is the adjusted vehicle control command; The method further includes: Execute the second vehicle control command.

10. A vehicle control device, characterized in that, include: The acquisition module is used to acquire a first structured sequence, wherein the first structured sequence is used to indicate at least two items of user information, road information, or real-time query information; The processing module is used to input the first structured sequence into the vehicle control model, and after processing by the vehicle control model, obtain a first vehicle control command; the first vehicle control command is used to instruct the adjustment of the current driving strategy of the vehicle. The control module is used to execute the first vehicle control command; or the processing module is used to adjust the first vehicle control command.

11. A vehicle control device, characterized in that, include: One or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the vehicle control device to perform the method as described in any one of claims 1 to 9.

12. A chip system, characterized in that, The chip system is applied to a vehicle control device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the vehicle control device to perform the method as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on a vehicle control device, cause the vehicle control device to perform the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a vehicle control device, causes the vehicle control device to perform the method as described in any one of claims 1 to 9.

Images