Intelligent cockpit user voice command inference methods, intelligent devices and storage media

Abstract

This application relates to the field of intelligent cockpit technology, specifically to an intelligent cockpit user voice command inference method, intelligent device, and readable storage medium, aiming to solve the technical problem of how to achieve flexible and agile response in the inference process of intelligent cockpits to user voice commands. To this end, this application obtains the inference scenario of the intelligent cockpit based on the user's voice command, obtains the corresponding trained external learning module according to the inference scenario, and performs inference on the voice command based on a preset backbone network and the external learning module, obtaining the inference result of the voice command in the intelligent cockpit. This enables the inference of voice commands based on the inference scenario in the intelligent cockpit, calling the external learning module that matches the inference scenario, and combining the backbone network and the external learning module to achieve inference of voice commands. This allows for flexible and agile response in the intelligent cockpit to different inference scenarios corresponding to user voice commands, thereby effectively improving the user experience.

Description

Technical Field

[0001] This application relates to the field of smart cockpit technology, specifically to a smart cockpit user voice command reasoning method, smart device, and readable storage medium. Background Technology

[0002] Intelligent cockpits typically utilize vehicle-cloud collaborative computing to enable users to invoke in-vehicle functions using natural language. Security requirements for multi-user scenarios and open-domain dialogue within the vehicle are also usually addressed through cloud-based RESTful API interfaces, integrated before and after the cockpit's intelligent processing module to filter user and model output content.

[0003] With the enhancement of large language model dialogue capabilities, the increase in vehicle-side computing power, and users' higher demands for intelligent cockpits, deploying large models on the vehicle side to directly serve users' needs for open dialogue, in-vehicle voice control, and personalized configuration is gradually changing the previous edge-cloud collaborative computing model and significantly reducing inference latency. However, in in-vehicle multi-user scenarios and open-domain dialogue content, security requirements still need to be considered. At the content level, judgments need to be made based on user intent, such as whether the topic is safe and timely, and a flexible and agile response is required for the above content.

[0004] However, in scenarios where large-scale models are deployed on the edge of intelligent cockpits, besides limited computing resources, these models typically require periodic updates and optimizations, high quantization and compression, and a single model for multiple scenarios. Traditional content security detection solutions, such as those using BERT (Bidirectional Encoder Representations from Transformers) and TextCNN for detection tasks, still require 50MB to 400MB of memory or GPU memory, and these models lack in context length, natural accuracy, and generalization ability. While applying LLM (Large Language Model) with full or efficient fine-tuning is significantly slower than BERT and TextCNN in inference speed, it consumes excessive computing resources.

[0005] Accordingly, there is a need in the field for a new user instruction reasoning scheme to solve the above problems. Summary of the Invention

[0006] In order to overcome the above-mentioned deficiencies, this application is made to solve, or at least partially solve, the technical problem of how to achieve a flexible and agile response of the reasoning process of a smart cockpit to user voice commands.

[0007] In a first aspect, a method for reasoning through user voice commands in an intelligent cockpit is provided, characterized in that the method includes:

[0008] Obtain the user's voice commands;

[0009] Based on the voice commands, the reasoning scenario of the intelligent cockpit is obtained;

[0010] Based on the reasoning scenario, obtain the external learning module that has been trained and is corresponding to the reasoning scenario;

[0011] Based on the preset backbone network and the external learning module, the voice commands are inferred to obtain the inference results of the intelligent cockpit.

[0012] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the method further includes training the external learning module according to the following steps:

[0013] Obtain the training data corresponding to the inference scenario;

[0014] Connect the external learning module to the backbone network and freeze the network parameters of the backbone network;

[0015] Based on the training data, the external learning module connected to the backbone network is trained to obtain the trained external learning module.

[0016] In one technical solution of the above-mentioned intelligent cockpit user voice command inference method, the step of training the external learning module connected to the backbone network based on the training data includes:

[0017] Obtain the configuration file of the external learning module;

[0018] During the forward propagation phase of the training process, the access point of the backbone network is selected to participate in the forward propagation according to the configuration file.

[0019] During the backpropagation phase of the training process, the access points of the backbone network are selected and weights are adjusted according to the configuration file.

[0020] In one technical solution of the above-mentioned intelligent cockpit user voice command inference method, the external learning module that has completed training includes:

[0021] The module information of the external learning module is obtained by acquiring at least one of the following: module type, configuration file, module weight, and version number of the backbone network, thereby obtaining the trained external learning module.

[0022] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the step of reasoning about the voice command based on the preset backbone network and the external learning module to obtain the reasoning result of the intelligent cockpit includes:

[0023] Obtain the module information of the external learning module;

[0024] Based on the module information, the access location of the backbone network is determined;

[0025] The external learning module is connected to the backbone network according to the access location;

[0026] Based on the backbone network of the external learning module, the voice commands are inferred to obtain the inference results.

[0027] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the step of obtaining the reasoning scenario of the intelligent cockpit based on the voice command includes:

[0028] Based on the backbone network, the voice commands are recognized to obtain the reasoning scenario.

[0029] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the reasoning scenario includes at least one of the following: security requirement scenario, business question and answer scenario, and tool call scenario.

[0030] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the backbone network is a network based on a large language model.

[0031] In one technical solution of the above-mentioned intelligent cockpit user voice command reasoning method, the method further includes:

[0032] When upgrading the prediction model composed of the backbone network and the external learning module in the intelligent cockpit, only the external learning module in the prediction model is updated.

[0033] In a second aspect, a smart device is provided, comprising a smart cockpit; at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described technical solutions of the smart cockpit user voice command reasoning method.

[0034] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions of the intelligent cockpit user voice command reasoning method.

[0035] Solution 1. A method for reasoning user voice commands in an intelligent cockpit, characterized in that the method includes:

[0036] Obtain the user's voice commands;

[0037] Based on the voice commands, the reasoning scenario of the intelligent cockpit is obtained;

[0038] Based on the reasoning scenario, obtain the external learning module that has been trained and is corresponding to the reasoning scenario;

[0039] Based on the preset backbone network and the external learning module, the voice commands are inferred to obtain the inference results of the intelligent cockpit.

[0040] Solution 2. The intelligent cockpit user voice command inference method according to Solution 1, characterized in that the method further includes training the external learning module according to the following steps:

[0041] Obtain the training data corresponding to the inference scenario;

[0042] Connect the external learning module to the backbone network and freeze the network parameters of the backbone network;

[0043] Based on the training data, the external learning module connected to the backbone network is trained to obtain the trained external learning module.

[0044] Solution 3. The intelligent cockpit user voice command reasoning method according to Solution 2, characterized in that,

[0045] The step of training the external learning module connected to the backbone network based on the training data includes:

[0046] Obtain the configuration file of the external learning module;

[0047] During the forward propagation phase of the training process, the access point of the backbone network is selected to participate in the forward propagation according to the configuration file.

[0048] During the backpropagation phase of the training process, the access points of the backbone network are selected and weights are adjusted according to the configuration file.

[0049] Solution 4. The intelligent cockpit user voice command reasoning method according to Solution 2, characterized in that,

[0050] The external learning module that obtains the completed training includes:

[0051] The module information of the external learning module is obtained by acquiring at least one of the following: module type, configuration file, module weight, and version number of the backbone network, thereby obtaining the trained external learning module.

[0052] Solution 5. The intelligent cockpit user voice command inference method according to Solution 1, characterized in that,

[0053] The process of reasoning about the voice commands based on the preset backbone network and the external learning module to obtain the reasoning results of the intelligent cockpit includes:

[0054] Obtain the module information of the external learning module;

[0055] Based on the module information, the access location of the backbone network is determined;

[0056] The external learning module is connected to the backbone network according to the access location;

[0057] Based on the backbone network of the external learning module, the voice commands are inferred to obtain the inference results.

[0058] Solution 6. The intelligent cockpit user voice command reasoning method according to Solution 1, characterized in that,

[0059] The step of obtaining the reasoning scenario of the smart cockpit based on the voice command includes:

[0060] Based on the backbone network, the voice commands are recognized to obtain the reasoning scenario.

[0061] Scheme 7. The intelligent cockpit user voice command reasoning method according to any one of Schemes 1 to 6, characterized in that,

[0062] The reasoning scenarios include at least one of the following: security requirement scenarios, business question and answer scenarios, and tool call scenarios.

[0063] Scheme 8. The intelligent cockpit user voice command reasoning method according to any one of Schemes 1 to 6, characterized in that,

[0064] The backbone network is based on a large language model.

[0065] Solution 9. The intelligent cockpit user voice command reasoning method according to any one of Solutions 1 to 6, characterized in that the method further includes:

[0066] When upgrading the prediction model composed of the backbone network and the external learning module in the intelligent cockpit, only the external learning module in the prediction model is updated.

[0067] Option 10. A smart device, characterized in that it comprises:

[0068] Intelligent cockpit;

[0069] At least one processor;

[0070] And, a memory communicatively connected to the at least one processor;

[0071] The memory stores a computer program, which, when executed by the at least one processor, implements the intelligent cockpit user voice command reasoning method as described in any one of schemes 1 to 9.

[0072] Scheme 11. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to execute the intelligent cockpit user voice command inference method as described in any one of Schemes 1 to 9.

[0073] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0074] In implementing the intelligent cockpit user voice command inference method technical solution provided in this application, this application obtains the inference scenario of the intelligent cockpit based on the user's voice command, obtains the corresponding trained external learning module according to the inference scenario, and performs inference on the voice command based on the preset backbone network and the external learning module to obtain the inference result of the voice command in the intelligent cockpit. Through the above configuration method, this application can realize the inference of voice commands based on the inference scenario in the intelligent cockpit, call the external learning module that conforms to the inference scenario, and combine the backbone network and the external learning module to realize the inference of voice commands. This enables the intelligent cockpit to achieve flexible and agile response to different inference scenarios corresponding to the user's voice command, thereby effectively improving the user experience.

[0075] Furthermore, this application trains external learning modules connected to the backbone network based on training data corresponding to the inference scenario. During the training process, the network parameters of the backbone network are frozen, and only the network parameters of the external learning modules are adjusted. This enables the training of external learning modules that meet the needs of different inference scenarios. Moreover, external learning modules for different inference scenarios share the backbone network, allowing for flexible configuration of the network parameters of the corresponding external learning modules according to the inference scenario. Furthermore, the backbone network does not need to participate in the training, which can effectively reduce the amount of training data, improve efficiency, and enhance the inference accuracy of different inference scenarios, enabling rapid and lightweight deployment of different inference scenarios. Attached Figure Description

[0076] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0077] Figure 1 This is a flowchart illustrating the main steps of a smart cockpit user voice command reasoning method according to an embodiment of this application;

[0078] Figure 2 This is a flowchart illustrating the main steps of the training and inference phases of a prediction model according to one embodiment of this application.

[0079] Figure 3 This is a flowchart illustrating the main steps of a smart cockpit user voice command reasoning method according to one embodiment of the present application.

[0080] Figure 4 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0081] Figure label:

[0082] 11: Memory; 12: Processor. Detailed Implementation

[0083] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0084] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, and memory, and may also include software components, such as program code, or a combination of software and hardware. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular forms of the terms "a" and "this" can also include plural forms.

[0085] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0086] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0087] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0088] Here we will first explain some of the terms used in this application.

[0089] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. Their core idea is to learn patterns and structures of natural language through large-scale unsupervised training, thus mimicking the human language cognition and generation process to some extent.

[0090] Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) technique used to adapt a large language model (LLM) to a specific task without retraining the entire model. Its core idea is to freeze the original weights of the pre-trained model and learn task-specific adjustments by introducing only a small number of trainable low-rank matrices, thereby significantly reducing computational and storage costs.

[0091] VAE (Variational Autoencoder) is a generative deep learning model that introduces probabilistic modeling and variational inference into the traditional autoencoder (AE). This allows it to not only compress and reconstruct data like an autoencoder, but also generate new samples similar to the training data.

[0092] Matrix multiplication refers to the linear transformation of input data, mapping input features to the output space through a weight matrix, thereby enabling the transfer, combination, and extraction of information. In neural networks, matrix multiplication plays a crucial role. Through this operation, the network can efficiently process and transform large amounts of data. In the forward propagation of a neural network, matrix multiplication is present throughout the entire process, from processing input data to calculating hidden layer activation values, and finally generating the output.

[0093] PPO (Proximal Policy Optimization) is a cornerstone technology of early RLHF (Reinforcement Learning Based on Human Feedback). Its core idea is to stably optimize the model policy through "safety constraints" to avoid training crashes caused by excessive update magnitude.

[0094] GRPO (Group Relative Policy Optimization) is an efficient improvement scheme proposed by DeepSeek, which aims to solve the problem of high memory consumption of PPO. Its core is to replace the expensive reward model and value model with "intra-group relative comparison".

[0095] DPO (Direct Preference Optimization) is a disruptive "minimalist" approach that completely bypasses the complexities of reinforcement learning, transforming preference alignment into a simple supervised learning problem.

[0096] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a smart cockpit user voice command reasoning method according to an embodiment of this application. Figure 1 As shown, the intelligent cockpit user voice command reasoning method in this application embodiment mainly includes the following steps S101 to S104.

[0097] Step S101: Obtain the user's voice commands.

[0098] In this embodiment, user voice commands in the smart cockpit can be acquired.

[0099] In one implementation, the user's voice commands can be obtained based on an interactive device installed in the smart cockpit. For example, the interactive device can be a microphone or other similar device installed in the smart cockpit.

[0100] Step S102: Obtain the reasoning scenario of the intelligent cockpit based on voice commands.

[0101] In this embodiment, voice commands can be analyzed to determine the reasoning scenario of the smart cockpit.

[0102] In one implementation, the reasoning scenario can include security requirement scenarios, business question-and-answer scenarios, and tool invocation scenarios. Security requirement scenarios can be those requiring risk assessment of dialogue within the smart cockpit; risk assessment can include timeliness risks, risks associated with specific words / phrases, and content risks targeting specific groups (e.g., children). Business question-and-answer scenarios can be those where the smart cockpit interacts with the user through question-and-answer sessions. Tool invocation scenarios can be those where the user invokes tools within the smart cockpit based on voice commands.

[0103] In one implementation, voice commands are recognized based on a pre-defined backbone network to obtain the current inference scenario of the intelligent cockpit. For example, keywords in the voice commands can be recognized to obtain the inference scenario. Alternatively, the voice features of the voice commands can be recognized to obtain the inference scenario.

[0104] In some other implementations, the inference scenario can be obtained based on the current settings of the smart cockpit. For example, if the current setting of the smart cockpit is a safety requirement scenario, then the inference scenario is the safety requirement scenario.

[0105] In one implementation, the backbone network can be a network based on a large language model.

[0106] Step S103: Based on the reasoning scenario, obtain the external learning module that has been trained and is corresponding to the reasoning scenario.

[0107] In this embodiment, the corresponding external learning module can be invoked based on the identified inference scenario of the intelligent cockpit. This external learning module is trained using training data from the inference scenario. The external learning module can be connected to the backbone network as an adapter.

[0108] In one implementation, the external learning module can be a LORA (low-rank matrix adapter), a VAE (variational autoencoder), or a variant thereof, as long as it can maintain the same dimension as the input and output tensors of the matrix multiplication of the backbone network during inference.

[0109] In one implementation, the external learning module can be trained based on the following steps S201 to S203:

[0110] Step S201: Obtain the training data corresponding to the reasoning scenario.

[0111] In this embodiment, corpus data corresponding to the reasoning scenario can be collected as training data.

[0112] Step S202: Connect the external learning module to the backbone network and freeze the network parameters of the backbone network.

[0113] In this implementation, the type of external learning module can be selected according to the inference scenario, and the configuration file of the external learning module can be obtained. Based on the configuration file, the access position and network parameters of the external learning module to the backbone network are determined. The external learning module is then connected to the backbone network according to the access position, thereby obtaining the prediction model for the inference scenario. The external learning module is initialized, and the prediction model for the current inference scenario is initialized according to the version number of the backbone network. The network parameters of the backbone network in the prediction model are frozen. Notably, any position in the backbone network that involves matrix multiplication can be used as the access position for the external learning module.

[0114] Step S203: Based on the training data, train the external learning module connected to the backbone network to obtain the trained external learning module.

[0115] In this embodiment, an external learning module connected to the backbone network can be trained based on training data to obtain a trained external learning module. During training, the configuration file of the external learning module can be obtained. In the forward propagation phase of training, the access point of the backbone network is selected to participate in forward propagation according to the configuration file. In the backpropagation phase of training, the access point of the backbone network is selected for weight adjustment according to the configuration file. The module type, configuration file, module weights, and backbone network version number of the trained external learning module can be saved as module information, thereby obtaining the training results and realizing the training of the external learning module.

[0116] In one implementation, the process of training the external learning module can employ commonly used model training methods in the field, such as DPO, PPO, and GRPO, to train the external learning module.

[0117] In one implementation, the training process of the external learning module can employ an autoregressive training method.

[0118] In a specific example, taking the inference scenario as a security requirement scenario, the training process of the external learning module corresponding to the security requirement scenario includes:

[0119] Retrieve the version number N of the backbone network; under the security requirement scenario S, prepare the corresponding security corpus data data_s as training data; select the type K_s of the external learning module, and configure the access position and hyperparameters of the external learning module according to the configuration file config_s, initialize the network parameters w_s of the external learning module, and initialize the prediction model according to the version number of the backbone network; freeze the network parameters w_N of the backbone network with version number N, and enable the network parameters w_s of the external learning module; after training begins, perform autoregressive training on the network parameters w_s of the external learning module on the security corpus data data_s. That is, in the forward propagation phase of the training process, determine which external nodes (i.e., access positions) of the backbone network participate in the forward propagation according to the configuration file config_s; in the backpropagation phase of the training process, determine which external nodes of the backbone network are weighted according to the configuration file config_s. After training, save the module information G_s={K_s adapter type, config_s configuration file, w_s adapter weights, N backbone network version number} as the training result.

[0120] Step S104: Based on the preset backbone network and external learning module, reason about the voice commands to obtain the reasoning results of the intelligent cockpit.

[0121] In this embodiment, the acquired external learning module can be connected to the backbone network to obtain the prediction model of the current reasoning scenario, and reasoning can be performed on the voice commands based on the prediction model to obtain the reasoning result of the smart cockpit.

[0122] In one embodiment, step S104 may include steps S1041 to S1044:

[0123] Step S1041: Obtain module information of the external learning module.

[0124] In this embodiment, the module information of the external learning module can be obtained, which may include the module type, configuration file, module weight, and version number of the backbone network of the external learning module.

[0125] Step S1042: Determine the access location of the backbone network based on the module information.

[0126] In this embodiment, the access location of the external learning module to the backbone network can be determined based on the module information.

[0127] Step S1043: Connect the external learning module to the backbone network according to the access location.

[0128] In this embodiment, an external learning module can be connected to the backbone network according to the access location, thereby obtaining a prediction model for the current reasoning scenario of the intelligent cockpit.

[0129] Step S1044: Based on the backbone network of the access external learning module, reason about the voice commands and obtain the reasoning results.

[0130] In this embodiment, the user's voice commands can be inferred based on the prediction model under the current inference scenario of the smart cockpit, thereby obtaining the inference result.

[0131] In one implementation, the intelligent cockpit can have multiple reasoning scenarios during the reasoning process for user voice commands. For multiple reasoning scenarios, external learning modules corresponding to each scenario can be invoked and connected to the backbone network to realize the reasoning process for user voice commands.

[0132] In a specific example, taking a reasoning scenario as a security requirement scenario, the process of reasoning about a user's voice commands includes:

[0133] The system retrieves information from the backbone network N for inference deployment; uploads the module information G_s of the external learning module corresponding to the security requirement scenario; and connects the external learning module to the backbone network according to the config_s configuration file. Under the security requirement scenario, inference is performed on the user's voice commands based on the backbone network with the external learning module connected, thereby obtaining the inference results for the security requirement scenario.

[0134] In one implementation, depending on the reasoning scenario, the external learning module can be connected to or not connected to the backbone network by setting the corresponding Boolean value.

[0135] For a specific example, see Appendix Figure 2 ,like Figure 2 As shown, taking the backbone network, which includes Tokenized Text, Embedding, Block_0, Block_1, Block_N, RMSNorm, and Linear, as an example, the specific process of the training and inference phases of the external learning module is explained. Figure 2 The locations marked by the double trapezoidal boxes can all serve as access points for external learning modules.

[0136] During the training phase, external learning modules can be trained based on training data corresponding to different inference scenarios. For example, for security requirement scenarios, a security dataset can be used, which includes scenario data specific to security requirements; for business question-and-answer scenarios, a free question-and-answer dataset can be used, which includes scenario data specific to business question-and-answer scenarios; and for tool call scenarios, a tool call dataset (FuncitionCall) can be used, which includes scenario data specific to tool call scenarios. Taking Block_0 as an example, external learning modules—adapter groups 1, 2, 3, 4, and 5—can be integrated into the attention module and forward module of Block_0. During training, in the forward propagation phase, the access position of the backbone network can be selected according to the configuration file to participate in forward propagation; in the backward propagation phase, the weights can be adjusted by selecting the access position of the backbone network according to the configuration file. After training is completed, the backbone network and external learning modules can be deployed for the voice command inference process of the intelligent cockpit.

[0137] During the inference phase, based on the deployed backbone network and external learning modules, the inference scenario can be determined according to the user's voice commands. Inference scenarios include security scenarios (security requirement scenarios), business question-and-answer scenarios, and tool call scenarios. The corresponding external learning modules (i.e., adapter groups) can be invoked according to the inference scenario to perform inference on the voice commands, thereby obtaining inference results such as security content determination, business question-and-answer responses, and function call input parameters.

[0138] Please refer to the appendix for further details. Figure 2 Taking Block_0 of the backbone network as an example, we will further explain the access location of the external learning module.

[0139] If Block_0 includes three matrix multiplications: Q, K, and V, then these three matrix multiplications can all be the access points for external learning modules (Adapters).

[0140] In one scenario, each access point can connect to an adapter group, i.e., multiple external learning modules. Each adapter in the adapter group corresponds to an inference scenario, used for fine-tuning for that inference scenario. This can be implemented using the following code:

[0141] Where Block_0 => adapters1 contains "adapters1": {

[0142] "safety": False,

[0143] "business1": True,

[0144] "business2": True

[0145] Multiple adapters each correspond to a reasoning scenario.

[0146] When deploying external learning modules for multiple inference scenarios, their configuration files can be merged into a single overall configuration file.

[0147] ▪ Sub-reasoning scenario 1: Block_0 = {

[0148] "adapters1": {

[0149] "safety": False

[0150] },

[0151] "adapters2": {

[0152] "safety": False

[0153] }, ...

[0155] "adapters5": {

[0156] "safety": True

[0157] },

[0158] }

[0159] ▪ Sub-reasoning scenario 2: Block_0 = {

[0160] "adapters1": {

[0161] "business1": False

[0162] },

[0163] "adapters2": {

[0164] "business1": True

[0165] }, ...

[0167] "adapters5": {

[0168] "business1": True

[0169] },

[0170] }

[0171] ▪ Merged scene configuration: Block_0 = {

[0172] "adapters1": {

[0173] "safety": False,

[0174] "business1": False

[0175] },

[0176] "adapters2": {

[0177] "safety": False,

[0178] "business1": True

[0179] }, ...

[0181] "adapters5": {

[0182] "safety": True

[0183] "business1": True

[0184] },

[0185] }

[0186] Among them, the external learning modules (Adapters) can support various forms, such as LoRa's AB matrix, VAE model, etc., as long as the external learning modules can maintain the consistency of input and output dimensions.

[0187] The advantage of merging into a single configuration file is that developers can use the same configuration file to perform forward and backward propagation of the prediction model consisting of the backbone network and external learning modules during the training and inference phases; and the internal implementation of the configuration file can be determined by the developers, making it more flexible.

[0188] During inference, the backbone network can be shared, while each inference scenario can run independently, reducing GPU (Graphics Processing Unit) resource usage. The specific implementation is shown in the following code:

[0189] def multi_forward(model, x, adapter_blocks_indices: dict):

[0190] # Original branch (backbone network)

[0191] h_original = x

[0192] for i, block in enumerate(model.blocks):

[0193] h_original = block(h_original)

[0194] # adapter branch (external learning module), reasoning again

[0195] h_adapter = x

[0196] for i, block in enumerate(model.blocks):

[0197] if i in find_config(adapter_blocks_indices):

[0198] h_adapter = block(h_adapter, use_adapter=True) # Enable adapter

[0199] else:

[0200] h_adapter = block(h_adapter) # Normal calculation

[0201] return h_original, h_adapter

[0202] The code example above demonstrates how, by inputting inference routing instructions into adapter_blocks_indices through the configuration file config_s, judgments and checks are performed at the access point of each external learning module during an inference process, based on the scenario settings.

[0203] Taking the LORA module as an external learning module as an example, we fine-tune the commonly used 7B-FP16 parameter model for inference scenarios. The model itself occupies 14GB of parameters. In the case of full LORA configuration, that is, the multiplication of the four matrices Q, K, V, and FFN are all external AB matrices with rank (level) = 8, calculated according to the hidden layer dimension d = 4096 and the number of layers L = 32.

[0204] Among them, 7B-FP16 refers to a large language model with 7 billion parameters, which is a parametric model loaded and inferred using the FP16 (half-precision floating-point) format.

[0205] Each external learning module is additionally integrated by an adaptation matrix: 2xdx rank = 2 x 4096 x 8 = 65,536

[0206] Total for each floor: 65,536 x 4 = 262,114

[0207] Total parameters: 262,144 x 32 = 8,388,608 (approximately 8.4M)

[0208] Those skilled in the art can select the rank for actual access and the AB matrix position of LORA, and configure it flexibly. The above is only an example.

[0209] Taking the external learning module as the VAE module as an example, H (dimensionality of hidden layer representation) = 20, Z (dimensionality of latent variables) = 8, and the adaptation matrix of each external learning module is additionally integrated:

[0210] Encode

[0211] ▪# First Linear Layer

[0212] linear1_params = 4096 * 20 + 20 # Weights + Bias

[0213] ▪# Second linear layer

[0214] linear2_params = 20 * 20 + 20

[0215] ▪# Mean layer

[0216] mu_params = 20 * 8 + 8

[0217] ▪# Log-variance layer

[0218] logvar_params = 20 * 8 + 8

[0219] Decode

[0220] ▪# First Linear Layer

[0221] linear1_params = 8 * 20 + 20

[0222] ▪# Second linear layer

[0223] linear2_params = 20 * 20 + 20

[0224] ▪# Output Layer

[0225] output_params = 20 * 4096 + 20

[0226] As can be seen from the above examples, configuring an external learning module for each inference scenario and deploying and inferring the corresponding external learning module for different inference scenarios results in a number of parameters that are much smaller than the number of parameters in the backbone network. This can effectively improve the iteration efficiency of inference scenarios and achieve lightweight deployment for different inference scenarios.

[0227] For a specific application scenario, please refer to the appendix. Figure 3 ,like Figure 3 As shown, in the intelligent cockpit, the user inputs voice commands. The backbone network recognizes these commands and determines that the current inference scenario includes a security requirement scenario and a free-response scenario. Then, it can call the corresponding external learning modules for these scenarios: the security adapter and the free-response adapter. Based on the security adapter, the risk of the user's command is assessed to obtain a security judgment result. The system can then determine whether the scenario is safe. If safe, the free-response adapter can be invoked to participate in a free-response interaction with the user, obtain the result, and send a final response to the user as a safety fallback. If unsafe, the free-response process is skipped, and a safety fallback response is sent directly to the user.

[0228] In one implementation, when it is necessary to upgrade the prediction model composed of the backbone network and external learning modules in the smart cockpit (e.g., OTA (Over-the-Air Technology) upgrade), only the external learning modules in the prediction model need to be upgraded. That is, when upgrading the prediction model, only the upgrade update files for the external learning modules can be issued, without upgrading the backbone network. This can effectively reduce the data volume of the upgrade update files, thereby effectively improving the upgrade efficiency of the prediction model.

[0229] Based on the methods described in steps S101 to S104 above, this embodiment of the application obtains the inference scenario of the intelligent cockpit based on the user's voice command, obtains the external learning module that has been trained corresponding to the inference scenario, and performs inference on the voice command based on the preset backbone network and the external learning module to obtain the inference result of the voice command in the intelligent cockpit. Through the above configuration method, this embodiment of the application can realize the inference of the intelligent cockpit by calling the external learning module that conforms to the inference scenario, and combining the backbone network and the external learning module to realize the inference of the voice command, thereby realizing flexible and agile response in the intelligent cockpit to different inference scenarios corresponding to the user's voice command, and thus effectively improving the user experience.

[0230] Furthermore, in this embodiment, the external learning modules connected to the backbone network are trained based on the training data corresponding to the inference scenario. During the training process, the parameters of the backbone network are frozen, and only the network parameters of the external learning modules are adjusted. This enables the training of external learning modules that meet the needs of different inference scenarios. Moreover, external learning modules for different inference scenarios share the backbone network, which allows for flexible configuration of the network parameters of the corresponding external learning modules according to the inference scenario. Furthermore, the backbone network does not need to participate in the training, which can effectively reduce the amount of training data, improve efficiency, and increase the inference accuracy of different inference scenarios, thereby achieving rapid and lightweight deployment for different inference scenarios.

[0231] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0232] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0233] Another aspect of this application provides a computer-readable storage medium.

[0234] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that executes the intelligent cockpit user voice command inference method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described intelligent cockpit user voice command inference method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device including various electronic devices, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0235] Another aspect of this application provides a smart device.

[0236] In one embodiment of a smart device according to this application, the smart device may include a smart cockpit; at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the methods described in any of the above embodiments. The smart device described in this application may include driving equipment, smart vehicles, robots, and other devices. See appendix. Figure 4 , Figure 4 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0237] In some embodiments of this application, an interactive device may be installed in the smart cockpit to collect the user's voice commands and to conduct free-flowing question-and-answer sessions with the user.

[0238] In some embodiments of this application, the processor may be a central processing unit, a microprocessor, a graphics processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor may be implemented in software, in hardware, or a combination of both.

[0239] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for reasoning user voice commands in an intelligent cockpit, characterized in that, The method includes: Obtain the user's voice commands; Based on the voice commands, the reasoning scenario of the intelligent cockpit is obtained; Based on the reasoning scenario, obtain the external learning module that has been trained and is corresponding to the reasoning scenario; Based on the preset backbone network and the external learning module, the voice commands are inferred to obtain the inference results of the intelligent cockpit.

2. The intelligent cockpit user voice command reasoning method according to claim 1, characterized in that, The method further includes training the external learning module according to the following steps: Obtain the training data corresponding to the inference scenario; Connect the external learning module to the backbone network and freeze the network parameters of the backbone network; Based on the training data, the external learning module connected to the backbone network is trained to obtain the trained external learning module.

3. The intelligent cockpit user voice command reasoning method according to claim 2, characterized in that, The step of training the external learning module connected to the backbone network based on the training data includes: Obtain the configuration file of the external learning module; During the forward propagation phase of the training process, the access point of the backbone network is selected to participate in the forward propagation according to the configuration file. During the backpropagation phase of the training process, the access points of the backbone network are selected and weights are adjusted according to the configuration file.

4. The intelligent cockpit user voice command reasoning method according to claim 2, characterized in that, The external learning module that obtains the completed training includes: The module information of the external learning module is obtained by acquiring at least one of the following: module type, configuration file, module weight, and version number of the backbone network, thereby obtaining the trained external learning module.

5. The intelligent cockpit user voice command reasoning method according to claim 1, characterized in that, The process of reasoning about the voice commands based on the preset backbone network and the external learning module to obtain the reasoning results of the intelligent cockpit includes: Obtain the module information of the external learning module; Based on the module information, the access location of the backbone network is determined; The external learning module is connected to the backbone network according to the access location; Based on the backbone network of the external learning module, the voice commands are inferred to obtain the inference results.

6. The intelligent cockpit user voice command reasoning method according to claim 1, characterized in that, The step of obtaining the reasoning scenario of the smart cockpit based on the voice command includes: Based on the backbone network, the voice commands are recognized to obtain the reasoning scenario.

7. The intelligent cockpit user voice command reasoning method according to any one of claims 1 to 6, characterized in that, The reasoning scenarios include at least one of the following: security requirement scenarios, business question and answer scenarios, and tool call scenarios.

8. The intelligent cockpit user voice command reasoning method according to any one of claims 1 to 6, characterized in that, The backbone network is based on a large language model.

9. The intelligent cockpit user voice command reasoning method according to any one of claims 1 to 6, characterized in that, The method further includes: When upgrading the prediction model composed of the backbone network and the external learning module in the intelligent cockpit, only the external learning module in the prediction model is updated.

10. A smart device, characterized in that, include: Intelligent cockpit; At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the intelligent cockpit user voice command reasoning method according to any one of claims 1 to 9.

Images