An interactive control method and apparatus

Abstract

The application provides an interactive control method and device, applied to the technical field of intelligent cockpit. The method comprises: in response to input data of a user, acquiring current state data of a vehicle. The input data is input to a pre-constructed cockpit alignment model to generate a response instruction for the input data. According to the current state data, the dynamic risk weight of the response instruction is determined. In the case where the dynamic risk weight is lower than a preset threshold, the control execution device is controlled to execute an operation matched with the response instruction, which can solve the problems of lack of safety specification constraints and low individual matching degree in the traditional interactive control method.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit technology, and in particular to an interactive control method and device. Background Technology

[0002] With the rapid development of intelligent cockpit technology, large language models (LLMs) have been gradually applied to scenarios such as cockpit voice interaction and function control. Currently, the industry mainly adopts common alignment training schemes to optimize large cockpit models, such as improving the model's understanding and response capabilities through incremental pre-training or general reinforcement learning from human feedback (RLHF) methods.

[0003] However, this solution still has certain limitations in practical cockpit applications. For example, the model output lacks safety regulations and constraints in cockpit scenarios, it is difficult to balance model inference efficiency and response speed when deployed on the vehicle side, user personalized interaction habits have not been effectively adapted, and the consistency of model understanding and execution under multimodal signals needs to be improved.

[0004] Therefore, there is an urgent need for an interactive control method and device to solve the problems of lack of safety specifications and low personalization matching in traditional interactive control methods. Summary of the Invention

[0005] The purpose of this application is to provide an interactive control method and device to solve the problems of lack of safety specifications and low personalization matching in traditional interactive control methods.

[0006] In a first aspect, embodiments of this application provide an interactive control method, which includes: acquiring current state data of a vehicle in response to user input data; inputting the input data into a pre-built cockpit alignment model to generate a response command for the input data; determining a dynamic hazard weight of the response command based on the current state data; and controlling an execution device to perform an operation matching the response command if the dynamic hazard weight is determined to be lower than a preset threshold.

[0007] The interactive control method provided in this application acquires the vehicle's current state data in real time in response to user input data. The user input data is input into a pre-built cockpit alignment model to obtain a response command based on the input data. Then, based on the current state data, a dynamic hazard weight is determined for the response command in the vehicle's current state. If the dynamic hazard weight is determined to be lower than a preset threshold, the response command is sent to the corresponding execution device to control the execution device to perform a safety operation matching the response command. This application incorporates the vehicle's current state into the decision-making basis for command execution through dynamic hazard weighting, achieving real-time safety verification of the output response command. This can improve the response efficiency and accuracy of cockpit interaction while ensuring driving safety.

[0008] One possible implementation involves determining the dynamic hazard weight of a response command based on current state data. This includes: assessing the risk of the response command and determining its hazard probability value; determining a vehicle speed risk factor based on vehicle speed information and the vehicle's maximum design speed from the current state data; and determining the dynamic hazard weight based on the command hazard probability and the vehicle speed risk factor.

[0009] One possible implementation involves feeding input data into a pre-built cockpit alignment model to generate response commands based on the cockpit input data. This includes: extracting semantic features from the speech data in the input data to obtain a speech feature vector; extracting visual features from the driver state data in the input data to obtain a visual feature vector; fusing the speech feature vector and the visual feature vector to obtain a fused feature vector; and generating response commands based on the fused feature vector.

[0010] One possible implementation involves fusing speech feature vectors and visual feature vectors to obtain a fused feature vector. This includes: transforming the visual feature vector to obtain a query vector; transforming the speech feature vector to obtain a key vector and a value vector; determining an attention score based on the query vector and the key vector; and determining the fused feature vector based on the attention score and the value vector.

[0011] One possible implementation, the interactive control method provided in this application embodiment, further includes: when the dynamic danger weight is determined to be greater than or equal to a preset threshold, determining whether there is a correction scheme for the response command that conforms to preset safety rules. If a correction scheme exists, the response command is corrected according to the correction scheme to obtain a corrected safety response command, and the safety response command is output to the corresponding execution device. If no correction scheme exists, a rejection message is generated and output.

[0012] One possible implementation, the cockpit alignment model construction process, includes: acquiring preset safety rules, input data, and corresponding labeled response instructions to construct a training dataset; adjusting the pre-constructed initial cockpit alignment model based on the training dataset to obtain an intermediate cockpit alignment model; constructing an initial lightweight model with fewer than a threshold parameters based on the intermediate cockpit alignment model; training the initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model; and performing quantization and compression processing on the lightweight alignment model to obtain the final cockpit alignment model.

[0013] One possible implementation involves adjusting a pre-built initial cockpit alignment model based on a training dataset to obtain an intermediate cockpit alignment model. This includes: inputting input data from the training dataset into the initial cockpit alignment model to obtain the predicted response instructions output by the model; determining the intent recognition loss based on the difference between the predicted response instructions and the corresponding labeled response instructions; comparing the predicted response instructions with preset safety rules to determine the degree of deviation, and determining the rule compliance loss based on the degree of deviation; and iteratively updating the parameters of the initial cockpit alignment model based on the intent recognition loss and the rule compliance loss until a preset convergence condition is met, thus obtaining the intermediate cockpit alignment model.

[0014] One possible implementation involves training an initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model. This includes: inputting the input data from the training dataset into both the intermediate cockpit alignment model and the initial lightweight model, respectively, to obtain a first output probability distribution for the intermediate cockpit alignment model and a second output probability distribution for the initial lightweight model. Calculating the distribution difference loss between the first and second output probability distributions. Extracting intermediate layer features from both the intermediate cockpit alignment model and the initial lightweight model, and calculating the feature difference loss between these intermediate layer features. Iteratively updating the parameters of the initial lightweight model based on the distribution difference loss and the feature difference loss until a preset convergence condition is met, thus obtaining the lightweight alignment model.

[0015] One possible implementation of the interactive control method provided in this application embodiment further includes: acquiring user feedback data on response commands. The feedback data includes user satisfaction ratings for the execution results and / or records of manual correction operations. Based on the feedback data, the cockpit alignment model is fine-tuned to generate personalized model parameters. The personalized model parameters are uploaded to a cloud server, allowing the cloud server to aggregate the personalized model parameters uploaded from multiple cockpit terminals to obtain updated global model parameters. The updated global model parameters are received from the cloud server, and the cockpit alignment model is updated accordingly.

[0016] Secondly, embodiments of this application provide an interactive control device, which includes: an acquisition module, a generation module, a determination module, and a control module.

[0017] The acquisition module is used to acquire the current status data of the vehicle in response to user input data.

[0018] The generation module is used to input data into a pre-built cockpit alignment model and generate response instructions for the input data.

[0019] The determination module is used to determine the dynamic danger weight of the response command based on the current status data.

[0020] The control module is used to control the execution device to perform operations that match the response instructions when the dynamic hazard weight is determined to be lower than a preset threshold and the preset safety rules are met.

[0021] One possible implementation involves a generation module that, when feeding input data into a pre-built cockpit alignment model and generating response commands based on the cockpit input data, specifically performs the following: Semantic feature extraction from the speech data in the input data to obtain a speech feature vector; visual feature extraction from the driver state data in the input data to obtain a visual feature vector; fusing the speech feature vector and the visual feature vector to obtain a fused feature vector; and generating response commands based on the fused feature vector.

[0022] One possible implementation involves a generation module that fuses speech feature vectors with visual feature vectors to obtain a fused feature vector. Specifically, this process includes: transforming the visual feature vector to obtain a query vector; transforming the speech feature vector to obtain a key vector and a value vector; determining an attention score based on the query vector and the key vector; and determining the fused feature vector based on the attention score and the value vector.

[0023] In one possible implementation, the interactive control device provided in this application embodiment is further configured to: determine whether there is a correction scheme for the response command that conforms to preset safety rules when the dynamic danger weight is determined to be greater than or equal to a preset threshold; if a correction scheme exists, modify the response command according to the correction scheme to obtain a modified safety response command, and output the safety response command to the corresponding execution device; if no correction scheme exists, generate and output a rejection message.

[0024] One possible implementation involves a module used to construct the cockpit alignment model. Specifically, this module is used to: acquire preset safety rules, input data, and corresponding labeled response instructions to construct a training dataset; adjust the pre-constructed initial cockpit alignment model based on the training dataset to obtain an intermediate cockpit alignment model; construct an initial lightweight model with fewer than a threshold parameters based on the intermediate cockpit alignment model; train the initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model; and perform quantization and compression processing on the lightweight alignment model to obtain the final cockpit alignment model.

[0025] One possible implementation involves a module that, based on a training dataset, adjusts a pre-built initial cockpit alignment model to obtain an intermediate cockpit alignment model. Specifically, this module: inputs the training dataset into the initial cockpit alignment model to obtain the predicted response instructions output by the model. Based on the difference between the predicted response instructions and the corresponding labeled response instructions, it determines the intent recognition loss. It compares the predicted response instructions with preset safety rules to determine the degree of deviation and determines the rule compliance loss based on this deviation. Based on the intent recognition loss and the rule compliance loss, iteratively updates the parameters of the initial cockpit alignment model until a preset convergence condition is met, thus obtaining the intermediate cockpit alignment model.

[0026] One possible implementation involves a module used to train an initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target. Specifically, when obtaining the lightweight alignment model, this module performs the following steps: inputting the input data from the training dataset into both the intermediate cockpit alignment model and the initial lightweight model, respectively, to obtain a first output probability distribution for the intermediate cockpit alignment model and a second output probability distribution for the initial lightweight model. It then calculates the distribution difference loss between the first and second output probability distributions. Next, it extracts intermediate layer features from both the intermediate cockpit alignment model and the initial lightweight model, calculating the feature difference loss between these intermediate layer features. Finally, based on the distribution difference loss and the feature difference loss, iteratively updates the parameters of the initial lightweight model until a preset convergence condition is met, resulting in the lightweight alignment model.

[0027] In one possible implementation, the interactive control device provided in this application embodiment is further configured to: acquire user feedback data on response commands. The feedback data includes user satisfaction ratings for the execution results and / or records of manual correction operations. Based on the feedback data, the cockpit alignment model is fine-tuned to generate personalized model parameters. The personalized model parameters are uploaded to a cloud server, allowing the cloud server to aggregate the personalized model parameters uploaded from multiple cockpit terminals to obtain updated global model parameters. The updated global model parameters are received from the cloud server, and the cockpit alignment model is updated accordingly.

[0028] Thirdly, embodiments of this application provide an interactive control device that has the function of implementing the interactive control method of the first aspect or any possible implementation of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the interactive control method described in the first aspect or any possible implementation thereof.

[0030] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, enable the computer to execute the interactive control method described in the first aspect or any possible implementation thereof.

[0031] The technical effects of any of the design methods in aspects two through five can be found in aspect one or in different possible implementations of aspect one, and will not be repeated here. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 A system structure diagram of an interactive control system provided in an embodiment of this application; Figure 2 A flowchart illustrating an interactive control method provided in this application embodiment; Figure 3 A schematic diagram of an interactive control device provided in an embodiment of this application; Figure 4 Another system architecture diagram of an interactive control system provided in this application embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0036] With the widespread application of large language models in smart cockpits, the level of intelligence in cockpit interaction has been significantly improved. However, related solutions still have many problems in practical applications. Regarding safety constraints, traditional large cockpit models do not incorporate physical parameters such as current vehicle speed and vehicle status as constraints into the decision-making logic when generating response commands. This can lead to conflicts between the model's output operation suggestions and the actual driving state, affecting driving safety. In terms of inference efficiency, mainstream large models have large parameter scales. When deployed to the vehicle-side cockpit domain controller, the inference latency is limited by the computing power of the onboard chip, making it difficult to meet the real-time response requirements of cockpit interaction. Regarding personalized adaptation, related solutions use uniform model parameters and do not personalize them for different users' interaction habits, resulting in deviations between model output and user expectations, affecting interaction efficiency and satisfaction. Regarding multimodal fusion, most related solutions treat speech recognition and visual perception as independent modules, lacking deep fusion and consistency verification of cross-modal features. When there are contradictions between the driver's state and voice commands, the system cannot effectively identify and handle them. In terms of model updates, traditional solutions require uploading user interaction data to the cloud for centralized training, which poses a risk of privacy leaks and conflicts with increasingly stringent data security regulations.

[0037] Based on this, embodiments of this application provide an interactive control method and apparatus. The method includes: acquiring current vehicle state data in response to user input data; inputting the input data into a pre-built cockpit alignment model to generate a response command for the input data; determining a dynamic hazard weight of the response command based on the current state data; and controlling an execution device to perform an operation matching the response command if the dynamic hazard weight is determined to be lower than a preset threshold.

[0038] The interactive control method provided in this application acquires the vehicle's current state data in real time in response to user input data. The user input data is input into a pre-built cockpit alignment model to obtain a response command based on the input data. Then, based on the current state data, a dynamic hazard weight is determined for the response command in the vehicle's current state. If the dynamic hazard weight is determined to be lower than a preset threshold, the response command is sent to the corresponding execution device to control the execution device to perform a safety operation matching the response command. This application incorporates the vehicle's current state into the decision-making basis for command execution through dynamic hazard weighting, achieving real-time safety verification of the output response command. This can improve the response efficiency and accuracy of cockpit interaction while ensuring driving safety.

[0039] The methods provided in the embodiments of this application will now be described in conjunction with the specific accompanying drawings.

[0040] On the one hand, embodiments of this application provide an interactive control system. For example... Figure 1 As shown, the interactive control system 100 may include: a cockpit domain controller 101, a voice acquisition device 102, a driver status acquisition device 103, a vehicle body sensor 104, an in-vehicle display screen 105, a voice broadcast device 106, and a cockpit execution device 107. The cockpit domain controller 101 is communicatively connected to the voice acquisition device 102, the driver status acquisition device 103, the vehicle body sensor 104, the in-vehicle display screen 105, the voice broadcast device 106, and the cockpit execution device 107.

[0041] The cockpit domain controller 101 has a pre-built cockpit alignment model deployed inside. The cockpit domain controller 101 is used to receive and process data from various acquisition devices, run the cockpit alignment model to generate response commands through the interactive control method provided in this application embodiment, and output control signals to the corresponding execution devices after performing security verification on the response commands.

[0042] The voice acquisition device 102 is used to acquire the user's voice commands. The voice acquisition device 102 can be a microphone array deployed in the cockpit, capable of acquiring the sound field in the vehicle from multiple directions, and performing noise reduction and echo cancellation processing on the acquired voice signals, and sending the processed voice data to the cockpit domain controller 101.

[0043] The driver status acquisition device 103 is used to acquire images of the driver's status. The driver status acquisition device 103 can be a driver monitoring system camera installed above the dashboard or inside the A-pillar, capable of capturing real-time images of the driver's face and sending the image data to the cockpit domain controller 101.

[0044] The vehicle body sensor 104 is used to collect the current status data of the vehicle. The vehicle body sensor 104 includes at least one of a vehicle speed sensor, a tire pressure sensor, and a battery level sensor, and can acquire vehicle physical status parameters such as current vehicle speed, tire pressure, and remaining battery level in real time, and send the above status data to the cockpit domain controller 101.

[0045] The in-vehicle display screen 105 is used to display visual feedback information during the interaction process. The in-vehicle display screen 105 can be a touch screen located in the center of the dashboard, used to show the user the current interaction status, the execution result of the response command, and safety prompts.

[0046] The voice broadcast device 106 is used to play voice feedback information during the interaction process. The voice broadcast device 106 may be a speaker system installed in the cockpit, used to broadcast the voice response content generated by the cockpit domain controller 101 to the user.

[0047] The cockpit execution device 107 is used to perform specific cockpit function operations. The cockpit execution device 107 includes at least one of a window controller, a sunroof controller, an air conditioning controller, a seat adjustment controller, and a navigation module. The cockpit execution device 107 receives control commands sent by the cockpit domain controller 101 and executes the physical operations corresponding to the control commands.

[0048] In the aforementioned interactive control system 100, when a user issues an interaction request via voice, the voice acquisition device 102 acquires the user's voice command, the driver status acquisition device 103 simultaneously acquires the driver's facial image, and the vehicle body sensor 104 obtains the current vehicle status parameters. The aforementioned voice data, image data, and status data are sent to the cockpit domain controller 101 in real time. The cockpit domain controller 101 inputs the voice data and image data into a pre-deployed cockpit alignment model, generates corresponding response commands, and performs dynamic safety verification on the response commands based on the current vehicle status parameters. After successful verification, the cockpit domain controller 101 sends the response command to the corresponding cockpit execution device 107 to perform the corresponding operation, or provides voice feedback through the voice broadcast device 106, and simultaneously displays the interaction results on the in-vehicle display screen 105.

[0049] It should be noted that, Figure 1 The connection relationships and quantities of the devices in the interactive control system 100 shown are merely illustrative examples. In practical applications, the quantity and connection methods of the above devices can be adaptively adjusted according to the vehicle configuration and hardware solution. This application embodiment does not impose specific limitations on this.

[0050] On one hand, embodiments of this application provide an interactive control method. This interactive control method can be... Figure 1 The interactive control system 100 shown is executed. For example... Figure 2As shown, the method may include: S201, in response to user input data, obtains the current status data of the vehicle.

[0051] User input data refers to the interactive information generated by the user during cockpit interaction. This input data may include the user's voice command data, as well as sensory data reflecting the user's current state.

[0052] For example, voice command data consists of user-generated operation or query requests via voice, such as "open the sunroof," "set the temperature to 22 degrees," or "how far is it to our destination?" Voice command data can be collected through a microphone array deployed in the cabin. During the collection process, noise reduction algorithms can be used to suppress ambient noise inside the vehicle, thereby improving the signal-to-noise ratio of the voice signal.

[0053] Perception data refers to information that can characterize the driver's real-time state, specifically facial image data. By analyzing the driver's facial images, features such as the driver's gaze direction, eyelid opening and closing, and head posture can be identified, thereby determining whether the driver is currently focused on driving, fatigued, or distracted.

[0054] One possible implementation includes a preprocessing step for the input data before acquiring it. For voice command data, a short-time Fourier transform can be used to convert the voice waveform into a spectrogram, and then a signal-to-noise ratio (SNR) selection formula can be applied. ,in For voice signal power, Low-quality speech segments are removed (based on noise signal power). For image data in the perception data, conventional image preprocessing operations such as size normalization and noise reduction can be performed.

[0055] The vehicle's current status data refers to physical parameters that reflect the vehicle's real-time operating conditions. For example, current status data includes current vehicle speed, tire pressure, remaining battery charge, and engine speed. This status data can be acquired in real time via the vehicle's CAN bus or onboard sensor network.

[0056] One possible implementation involves smoothing the acquired vehicle state data to eliminate noise jumps during sensor acquisition. For example, for vehicle speed signals, a Kalman filter algorithm can be used to predict the current state based on the vehicle state vector from the previous moment, and then correct it using the current observation value to obtain smoothed vehicle speed data. This avoids misjudgments in subsequent safety checks due to sudden changes in vehicle speed values. The specific state update formula for this Kalman filter algorithm is shown below.

[0057]

[0058]

[0059] in, Let A be the vehicle state vector at time k, and let A be the state transition matrix. Let B be the vehicle state vector at time k-1, and let B be the control matrix. This is the control vector at time k-1. This represents the process noise at time k-1. Let H be the observation vector at time k, and H be the observation matrix. To observe noise.

[0060] S202 inputs the input data into a pre-built cockpit alignment model and generates response instructions for the input data.

[0061] The cockpit alignment model is a pre-built neural network model deployed in the cockpit domain controller. It is used to perform semantic understanding and intent reasoning on user input data and generate corresponding response commands. To facilitate understanding of the specific working method of the cockpit alignment model in this application, the construction process of the cockpit alignment model provided in the embodiments of this application is first described.

[0062] One possible implementation, the cockpit alignment model construction process, includes: acquiring preset safety rules, input data, and corresponding labeled response instructions to construct a training dataset; adjusting the pre-constructed initial cockpit alignment model based on the training dataset to obtain an intermediate cockpit alignment model; constructing an initial lightweight model with fewer than a threshold parameters based on the intermediate cockpit alignment model; training the initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model; and performing quantization and compression processing on the lightweight alignment model to obtain the final cockpit alignment model.

[0063] Among them, preset safety rules refer to the constraints defined according to vehicle driving safety regulations. For example, these preset safety rules can be obtained by organizing cabin scenario safety constraints based on GB44495—2024 and HMI safety regulations. For example, preset safety rules may include "When the current vehicle speed exceeds 15km / h, video playback is prohibited," "When the current vehicle speed exceeds 80km / h, sunroof opening is prohibited," and "When the driver is fatigued, complex interactive operations are restricted." The above safety rules can be stored using logical predicate format, for example: Speed>V_threshold→Forbid(Function). Taking a specific scenario as an example, when the rule is "Video is prohibited when the vehicle speed is greater than 15km / h," the logical predicate is represented as: Speed>15→Forbid(Video).

[0064] The input data samples include historical voice command data and corresponding driver status image data. For example, voice command data can be actual voice requests made by the user while driving, such as "open the video," "open the sunroof," or "is there a traffic jam ahead?"; driver status image data can be driver facial images synchronously captured by the driver monitoring system's camera, including image samples from various scenarios such as normal driving, fatigue, and distraction.

[0065] Annotated response instructions refer to standard response content that conforms to safety rules and fulfills the user's intent in the given input data scenario. For example, for the input data "User issues command to open sunroof while driving at high speed," the corresponding labeled response instruction is not "Opening sunroof," but rather a safety-compliant message such as "Opening sunroof while driving at high speed may affect driving safety; it is recommended to slow down before operating," or a similar safety warning. For the input data "User issues command to open video while vehicle is stationary," the corresponding labeled response instruction is "Opening video" and then executing the video playback operation.

[0066] The aforementioned preset security rules, input data samples, and labeled response instructions are associated and stored to form a training dataset.

[0067] The pre-built initial cockpit alignment model is then fine-tuned under safety constraints based on the training dataset to obtain an intermediate cockpit alignment model. The initial cockpit alignment model can be based on an open-source, large-scale pre-trained language model, which already possesses general language understanding and generation capabilities.

[0068] One possible implementation involves adjusting a pre-built initial cockpit alignment model based on a training dataset to obtain an intermediate cockpit alignment model. This includes: inputting input data from the training dataset into the initial cockpit alignment model to obtain the predicted response instructions output by the model; determining the intent recognition loss based on the difference between the predicted response instructions and the corresponding labeled response instructions; comparing the predicted response instructions with preset safety rules to determine the degree of deviation, and determining the rule compliance loss based on the degree of deviation; and iteratively updating the parameters of the initial cockpit alignment model based on the intent recognition loss and the rule compliance loss until a preset convergence condition is met, thus obtaining the intermediate cockpit alignment model.

[0069] Specifically, input data samples from the training dataset are fed into the cockpit alignment initial model, which then generates predicted response commands based on the input data. Subsequently, the predicted response commands are evaluated from two dimensions: accuracy of intent recognition and compliance with safety rules.

[0070] In the intent recognition dimension, the predicted response command is compared with the corresponding labeled response command, and the difference between the two is calculated as the intent recognition loss. This intent recognition loss measures whether the model correctly understands the user's interaction intent. For example, when the user's voice command is "Is there traffic ahead? How long until we reach our destination?", the labeled response command should be a reply containing navigation-related information such as traffic conditions and remaining travel time. If the predicted response command generated by the model is unrelated to navigation, the intent recognition loss is significant.

[0071] In the safety rule compliance dimension, the predicted response command is compared with the preset safety rules to determine whether the operation corresponding to the predicted response command violates the constraints in the preset safety rules, and the degree of violation is quantified. Based on this degree of violation, the rule compliance loss is determined. This rule compliance loss is used to measure whether the response command output by the model meets the safety specifications. For example, for the input scenario "Current vehicle speed is 110km / h, user voice command is to open the sunroof while driving", if the predicted response command output by the model is "Opening the sunroof", then this output violates the constraint "Opening the sunroof is prohibited at high speed" in the preset safety rules, and the rule compliance loss is large. If the predicted response command output by the model is "Opening the sunroof while driving at high speed may affect driving safety, it is recommended to slow down before operating" or a similar refusal prompt, then the output complies with the safety rules, and the rule compliance loss is small.

[0072] After obtaining the intent recognition loss and rule compliance loss, the two are weighted and summed to obtain the total loss for this training iteration. This total loss can be determined using the following security constraint loss function.

[0073]

[0074] in, The total loss value of the safety constraint loss function. In order to identify the loss, The weighting coefficients for the rule compliance loss are... Loss due to rule compliance.

[0075] This is used to adjust the importance of safety rule compliance in the training objective, and its value can be set from 0.5 to 1.0. When the value is large, the training process focuses more on constraining the model output to conform to safety rules; when... When the value is small, the training process focuses more on the model's accurate recognition of user intent. The network parameters of the initial cockpit alignment model are updated via backpropagation based on this total loss. The AdamW optimizer can be used during training, with parameters set as follows: learning rate 2e-5, weight decay 1e-4, and batch size 16. The above iterative process is repeated, and the model's compliance is evaluated after each training round using a safety rule validation set, which can contain more than 1000 constraint instructions. If compliance is not met, the weight coefficient of the rule compliance loss can be increased. The process continues until the total loss converges to below a preset threshold or reaches a preset number of iterations, resulting in a cockpit-aligned intermediate model.

[0076] Through the aforementioned fine-tuning training based on safety constraints, the cockpit-aligned intermediate model, while maintaining its original language understanding capabilities, now possesses the ability to proactively follow preset safety rules when generating response commands. For example, after training, when the input scenario is "vehicle speed 110km / h, user voice command: Is there traffic congestion ahead?", the model can quickly identify the scenario as a highway navigation scenario, prioritize responding to navigation-related questions, and output accurate congestion information and remaining travel time. When the input scenario is "vehicle speed 110km / h, user voice command: Open sunroof while driving", the model can determine that the command poses a safety risk and output a safety warning indicating that execution is not permitted.

[0077] Then, using the output distribution of the cockpit alignment intermediate model as a reference target, the initial lightweight model is trained by knowledge distillation to obtain the lightweight alignment model.

[0078] One possible implementation involves training an initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model. This includes: inputting the input data from the training dataset into both the intermediate cockpit alignment model and the initial lightweight model, respectively, to obtain a first output probability distribution for the intermediate cockpit alignment model and a second output probability distribution for the initial lightweight model. Calculating the distribution difference loss between the first and second output probability distributions. Extracting intermediate layer features from both the intermediate cockpit alignment model and the initial lightweight model, and calculating the feature difference loss between these intermediate layer features. Iteratively updating the parameters of the initial lightweight model based on the distribution difference loss and the feature difference loss until a preset convergence condition is met, thus obtaining the lightweight alignment model.

[0079] Specifically, construct a teacher-student model architecture: use the cockpit-aligned intermediate model as the teacher model, which typically has a large number of parameters, such as reaching billions (≥7B); construct an initial lightweight model with fewer than a preset threshold as the student model, whose network size is significantly smaller than the teacher model, for example, the number of parameters can be controlled at hundreds of millions (≤1B), to adapt to the computing power constraints of the vehicle hardware.

[0080] During distillation training, a distillation loss function is designed, aligning the output distributions and intermediate layer features of the teacher and student models. The distillation loss function can be expressed as:

[0081] in, The total loss during distillation training. The loss is the distribution difference between the output distributions of the teacher model and the student model. These are the feature loss weight coefficients. This represents the feature difference loss between the intermediate layer features of the teacher model and the student model.

[0082] For distributional difference loss KL divergence can be used as a measure of difference, i.e. ,in The first output probability distribution of the teacher model. The second output probability distribution is the output of the student model. KL divergence measures the difference between the two probability distributions. By minimizing the KL divergence, the student model learns the probability distribution characteristics of the teacher model at the output level. The first output probability distribution represents the teacher model's probability estimate for each candidate response instruction, implying the model's decision preference under safety constraints. For example, for the input "open the sunroof while driving at high speed," the probability distribution output by the teacher model shows a higher probability of "output safety prompt" (e.g., 80%) and a lower probability of "execute open sunroof" (e.g., 5%). By learning this probability distribution, the student model can inherit the teacher model's safety decision preference.

[0083] For feature difference loss Mean squared error can be used as a measure of difference, i.e. .in, For the output features of at least one intermediate network layer of the teacher model, This represents the output features of the intermediate network layers corresponding to the student model. By introducing alignment of these intermediate layer features, the student model can learn the knowledge representations of the teacher model at a deeper level, rather than simply imitating the final output.

[0084] In the distillation training strategy, a temperature coefficient T can be used to soften the output distributions of the teacher and student models to improve the distillation effect. The temperature coefficient T can be between 5 and 10. Specifically, before calculating the KL divergence, the original probability values ​​output by the teacher and student models are divided by the temperature coefficient T, making the probability distribution smoother and amplifying the probability weights of secondary candidate responses. During training, the student model can learn richer information about the relative relationships between categories from the smoothed probability distribution, thus better inheriting the decision-making knowledge of the teacher model.

[0085] The above distribution difference loss Compared with feature difference loss Perform a weighted summation to obtain the total loss from the distillation training. The parameters of the initial lightweight model are updated by backpropagation based on the total loss. The above process is repeated iteratively until the total loss converges to below a preset threshold or reaches a preset number of iterations, thus obtaining the lightweight alignment model.

[0086] Finally, the lightweight alignment model undergoes quantization and compression to obtain the final deployed cockpit alignment model. Specifically, the floating-point weight parameters in the lightweight alignment model are converted to low-bit-width integer representations, for example, from 16-bit floating-point format to 8-bit integer format. During quantization, the scaling factor and zero-point offset are determined based on the numerical range of the weight parameters, and the original floating-point weights are converted to integer values ​​through a mapping relationship. For example, the quantization formula can be expressed as:

[0087] in, To convert the original floating-point weights to integer values, For the original floating-point weights, This is the quantization scaling factor. This is the zero-point offset.

[0088] Taking a specific numerical example, if the weights of a certain network layer range from [-3.0, 6.0], then the scaling factor scale = 9.0 / 255 ≈ 0.0353, and the zero-point offset... For the original weights Its quantized integer weight is Through the above mapping, the original floating-point weights are compressed into an 8-bit integer space, significantly reducing the model storage volume. Furthermore, integer operations are more efficient than floating-point operations during inference, making it better suited to the hardware computing power of the cockpit domain controller.

[0089] Simultaneously, operators in the model's computation graph are fused and optimized. For example, the TensorRT framework can be used for operator fusion to reduce memory accesses and computational overhead during inference. After quantization and compression, the cockpit-aligned model significantly reduces both storage size and inference computation while maintaining its original accuracy, meeting the real-time inference requirements of the cockpit domain controller. For instance, the cockpit domain controller needs to have a computing power of at least 10 TOPS and support INT8 quantization inference.

[0090] The above describes the construction process of the cockpit alignment model. After the model is completed and deployed to the cockpit domain controller, the inference process in this step can be executed in the real vehicle environment. The following section details the process of generating response commands.

[0091] One possible implementation involves feeding input data into a pre-built cockpit alignment model to generate response commands based on the cockpit input data. This includes: extracting semantic features from the speech data in the input data to obtain a speech feature vector; extracting visual features from the driver state data in the input data to obtain a visual feature vector; fusing the speech feature vector and the visual feature vector to obtain a fused feature vector; and generating response commands based on the fused feature vector.

[0092] Specifically, a pre-trained speech coding network is used to process the speech signal. For example, the Wav2Vec2 model can be used as a speech feature extractor. First, the original speech waveform is processed through a multi-layer convolutional neural network for feature encoding, extracting local time-frequency features of the speech signal. Then, a Transformer coding layer is used to model the temporal context information, mapping the acoustic features to a high-dimensional semantic space, ultimately obtaining a fixed-dimensional speech feature vector. The dimension of this speech feature vector can be set to 768 dimensions. This speech feature vector encodes the semantic content and interactive intent of the user's voice commands.

[0093] Simultaneously, a convolutional neural network is employed to perform multi-layer convolution and pooling processing on the driver's state images. For example, a ResNet-50 model can be used as the visual feature extractor. The driver's facial image captured by the driver monitoring system's camera is input into the ResNet-50 model. After convolution, batch normalization, and non-linear activation processing through multiple residual blocks, visual features such as facial key points, gaze direction, eyelid opening / closing degree, and head posture are extracted from the image. These features are then mapped into a fixed-dimensional visual feature vector through a fully connected layer. The dimension of this visual feature vector can be set to 512. This visual feature vector encodes the driver's current state information, such as whether the driver is focused, distracted, or fatigued.

[0094] Then, the speech feature vector and the visual feature vector are fused across modalities to obtain the fused feature vector.

[0095] One possible implementation involves fusing speech feature vectors and visual feature vectors to obtain a fused feature vector. This includes: transforming the visual feature vector to obtain a query vector; transforming the speech feature vector to obtain a key vector and a value vector; determining an attention score based on the query vector and the key vector; and determining the fused feature vector based on the attention score and the value vector.

[0096] Specifically, since speech feature vectors and visual feature vectors originate from different feature extraction networks, their vector dimensions may differ. Before fusion, their dimensions can be unified to the same dimension using a feature mapping matrix. For example, if the visual feature vector has a dimension of 512 and the speech feature vector has a dimension of 768, a linear transformation can be performed on the visual feature vector using a mapping matrix of size 512×768 to map it to a 768-dimensional space, resulting in a unified visual feature vector.

[0097] Subsequently, an attention-based cross-modal fusion strategy is employed. The visual feature vector, after unification of dimensions, is used as the query, and the speech feature vector is used as the key and value. Specifically, a linear transformation is performed on the visual feature vector to obtain the query vector Q; two linear transformations are performed on the speech feature vector to obtain the key vector K and the value vector V. These linear transformations can be implemented using a learnable weight matrix. , , Implementation, that is:

[0098]

[0099]

[0100] After obtaining the query vector Q, key vector K, and value vector V, the dot product similarity between the query vector and the key vector is calculated, and then scaled and normalized to obtain the attention score. The formula for calculating the attention score can be expressed as: .

[0101] in, This refers to the dimension of the attention heads. For example, a multi-head attention mechanism can be used, mapping the query vector, key vector, and value vector to multiple attention heads, with the number of attention heads potentially set to 8. Attention scores are calculated independently within each attention head, and feature weighting is applied. Finally, the outputs of the multiple attention heads are concatenated to obtain a fused feature vector.

[0102] Through the aforementioned cross-modal attention fusion, the fused feature vector not only contains the semantic information of the user's voice commands but also incorporates contextual features associated with the driver's current state. This attention score characterizes the degree of correlation between each semantic component in the voice feature vector and the driver's visual state when generating response commands. For example, when the driver is significantly distracted, the dimension representing the degree of attention concentration in the visual feature vector exhibits a specific pattern. This pattern interacts with the voice features through an attention mechanism, enabling the fused feature vector to reflect the contextual information of "the user's current distracted attention," thereby providing richer contextual references for subsequent command generation.

[0103] After obtaining the fused feature vector, it can be further processed through residual connections and layer normalization operations to improve the stability and convergence speed of model training.

[0104] Finally, response commands are generated based on the fused feature vectors. The fused feature vectors are input into the decoding network of the cockpit alignment model, and a word-by-word response text sequence is generated through an autoregressive approach to form a complete response command. This response command represents the cockpit system's understanding of the user input data and the intended response operation. For example, when the user's voice command is "Open the sunroof" and the driver is focused on driving, the generated response command might be "Opening the sunroof for you" with an accompanying control signal for sunroof opening; when the user's voice command is "Open the sunroof" but the driver is fatigued, the fused feature vector already contains fatigue information, and the generated response command might be "You are detected to be fatigued; it is recommended that you rest for a while before operating the sunroof."

[0105] Furthermore, after completing cross-modal attention fusion and obtaining the fused feature vector, before generating response instructions based on the fused feature vector, the consistency of the speech feature vector and the visual feature vector can be checked to determine whether they match at the semantic level.

[0106] One possible implementation involves calculating the semantic consistency score between the visual feature vector and the speech feature vector. The semantic consistency score can be measured using cosine similarity, calculated as follows:

[0107] in, The semantic consistency score ranges from 0 to 1. For visual feature vectors; For speech feature vectors; This is the cosine similarity function.

[0108] Semantic consistency score reflects the degree of semantic matching between the driver's current visual state and voice commands. When the driver is focused on driving and looking straight ahead, if a clear cockpit control command such as "open the sunroof" is issued, the visual and voice features show high semantic consistency, resulting in a high semantic consistency score. When the driver is in a state of closed eyes or significant distraction, if a complex multi-turn dialogue command is issued, the visual and voice features contradict each other semantically, resulting in a low semantic consistency score.

[0109] After obtaining the semantic consistency score, it is compared with a preset consistency threshold. For example, the consistency threshold can be set to 0.6.

[0110] When the semantic consistency score is greater than or equal to the consistency threshold, it indicates that the visual features and speech features have a high degree of semantic consistency, and the driver's current state matches the issued voice command. At this point, the next step is to fuse the feature vectors and input them into the decoding network to generate a response command.

[0111] When the semantic consistency score is below the consistency threshold, it indicates a semantic discrepancy between visual and speech features, meaning the driver's current state does not match the issued voice command. This could be due to voice misrecognition, unconscious vocalization by the user, or user distraction. In this case, the command clarification logic is triggered. Instead of directly generating an execution-type response command, a clarification statement is output to reconfirm the user's intent.

[0112] For example, when the driver's eyes are closed (visual characteristics indicate fatigue) and the voice command is a vague "it's a bit cold," the semantic consistency score may be below 0.6. The system triggers the command clarification logic, outputting clarification statements through the voice broadcasting device, such as: "Your needs are not clearly defined. Do you need to adjust the air conditioning temperature?" or "We have detected that you are quite fatigued. Do you need to activate rest mode?"

[0113] S203, determine the dynamic hazard weight of the response command based on the current status data.

[0114] After generating a response command through the cockpit alignment model, the current state data of the vehicle can be combined to perform a dynamic safety assessment of the response command in order to determine the execution risk level of the command in the current driving scenario.

[0115] One possible implementation involves determining the dynamic hazard weight of a response command based on current state data. This includes: assessing the risk of the response command and determining its hazard probability value; determining a vehicle speed risk factor based on vehicle speed information and the vehicle's maximum design speed from the current state data; and determining the dynamic hazard weight based on the command hazard probability and the vehicle speed risk factor.

[0116] Specifically, a pre-trained dangerous instruction classifier is used to analyze response instructions and determine whether the operation type corresponding to the instruction belongs to the category of high-risk operations. The dangerous instruction classifier can be trained based on historical interaction data. Its input is the text content or semantic feature vector of the response instruction, and its output is the probability value that the instruction belongs to a dangerous instruction, i.e., the dangerous probability value of the instruction. .

[0117] For example, the correspondence between the operation type of the response command and the hazard probability value may include: operations involving opening the sunroof, opening the windows, playing videos, etc., which may distract the driver's attention, have a high hazard probability value, ranging from 0.7 to 1.0; operations involving adjusting the air conditioning temperature, adjusting the volume, etc., which do not affect driving safety, have a medium hazard probability value, ranging from 0.3 to 0.7; and operations involving information services that help driving, such as navigation queries and traffic reports, have a low hazard probability value, ranging from 0 to 0.3.

[0118] Taking a specific scenario as an example, when the response command is "Opening the sunroof for you," this command involves opening the sunroof, which falls into the category of high-risk operations. The dangerous command classifier outputs a dangerous probability value for the command. It can be 0.8; when the response command is "Road ahead is clear, estimated arrival at destination in 20 minutes", this command is a navigation information broadcast and does not involve cockpit hardware control. The hazard probability value output by the hazard command classifier is the hazard probability value of the command. It can be 0.1.

[0119] Then, based on the vehicle speed information in the current status data and the vehicle's maximum design speed, a vehicle speed risk factor is determined. This vehicle speed risk factor quantifies the impact of the current vehicle speed on the risk of instruction execution. In one possible implementation, the vehicle speed risk factor can be determined by the ratio of the current vehicle speed to the vehicle's maximum design speed, i.e.:

[0120] in, As a vehicle speed risk factor, The current vehicle speed is obtained through the vehicle's sensors. This is the vehicle's maximum design speed.

[0121] For example, if the vehicle's maximum design speed The current speed is 200km / h. If the speed is 110 km / h, then the vehicle speed risk factor is... .

[0122] The speed risk factor reflects the degree of danger of the vehicle's current driving state. When the vehicle speed is low, even if a risky operation is performed, the driver has sufficient reaction time; when the vehicle speed is high, any distracting operation may bring greater safety hazards. Therefore, the speed risk factor increases linearly with the increase of vehicle speed.

[0123] Obtain the probability value of the instruction's danger. and vehicle speed risk factor Then, the two are weighted and fused to obtain the dynamic hazard weight of the response command in the current driving scenario. The formula for calculating the dynamic hazard weight can be expressed as:

[0124] in, This is a dynamic hazard weight, with a value ranging from 0 to 1; The weighting coefficient for the probability value of the instruction's danger; This represents the weighting coefficient of the vehicle speed risk factor.

[0125] In one possible implementation, the weighting coefficient and The condition α + β = 1 must be met, and both can be adjusted according to the actual application scenario. For example, if you want to focus more on the risk level of the instruction itself in the safety verification, you can set α = 0.6 and β = 0.4; if you want to focus more on the impact of vehicle speed on the risk, you can set α = 0.4 and β = 0.6.

[0126] S204, if it is determined that the dynamic hazard weight is lower than the preset threshold, control the execution device to perform an operation that matches the response command.

[0127] Specifically, after determining the dynamic hazard weight of the response command, the dynamic hazard weight is compared with a preset threshold, and different processing strategies are adopted for the response command based on the comparison result.

[0128] The preset threshold can be set to 0.5. The value of this preset threshold can be adjusted according to the security requirements of the actual application scenario. When security requirements are high, the preset threshold can be appropriately lowered; when interaction smoothness requirements are high, the preset threshold can be appropriately increased. This application embodiment does not specifically limit this.

[0129] One possible implementation is to determine that if the dynamic hazard weight is below a preset threshold, it indicates that the overall risk of the response command in the current driving scenario is within an acceptable range. In this case, the response command is output to the corresponding execution device to control the device to perform an operation matching the response command.

[0130] For example, when a user issues a "open sunroof" command while driving at low speed, the dynamic hazard weight calculated in S203 is 0.475, which is lower than the preset threshold of 0.5. At this time, the system determines that the operation is risk-controlled in the current scenario and sends a response command to the sunroof controller to execute the sunroof opening operation. When a user issues a navigation query command while driving at high speed, the dynamic hazard weight calculated in S203 is 0.325, which is lower than the preset threshold of 0.5. At this time, the system sends a response command to the navigation module and the voice broadcast device, broadcasting the road conditions ahead and the estimated arrival time.

[0131] Another possible implementation involves determining whether a corrective action exists for the response command that conforms to preset safety rules, provided that the dynamic risk weight is greater than or equal to a preset threshold. If a corrective action exists, the response command is corrected according to the corrective action to obtain a corrected safety response command, which is then output to the corresponding execution device. If no corrective action exists, a rejection message is generated and output.

[0132] Specifically, if the dynamic hazard weight is determined to be greater than or equal to a preset threshold, it indicates that the overall risk of the response command in the current driving scenario exceeds the acceptable range, and safety intervention is required for the command.

[0133] In this case, it is determined whether there is a corrective action that conforms to the preset safety rules for the response command. The preset safety rules are consistent with those used when constructing the training dataset in S202, and can be, for example, compiled based on GB44495—2024 and the HMI safety specification. A corrective action refers to an alternative operation that adjusts the original response command to meet the safety requirements of the current driving scenario without completely rejecting the user's intent.

[0134] For example, the correspondence between preset safety rules and correction schemes may include: when the original response command is "open the sunroof" and the current vehicle speed exceeds 80km / h, there is no safety equivalent correction scheme, because the safety risk of opening the sunroof while driving at high speed cannot be avoided by adjusting parameters.

[0135] When the original response command is "play video" and the current vehicle speed exceeds 15km / h, the correct solution can be "switch video playback to audio playback", that is, only audio content is provided during driving without displaying video, so as to meet the user's entertainment needs while avoiding distraction from driving.

[0136] When the original response command is "adjust the air conditioning temperature to 16 degrees" and the vehicle's current battery level is lower than the preset battery threshold, the corrected solution can be "it is recommended to adjust the air conditioning temperature to 22 degrees to save battery power," guiding the user to make a more reasonable operation choice through voice prompts.

[0137] If a correction plan exists, the response command is modified according to the correction plan to obtain a modified safety response command. This modified safety response command is then output to the corresponding execution device to control the device to perform the operation matching the safety response command. Simultaneously, the reason for the correction can be explained to the user via a voice broadcast device.

[0138] For example, in the above-mentioned "play video" scenario, the system corrects the original "open video" command to "play audio" and prompts the user through a voice broadcast device: "Playing video while driving may affect driving safety. We have switched you to audio playback mode."

[0139] If no corrective action is found, a rejection message will be generated and output. The rejection message can be output verbally via a voice broadcast device and / or displayed as text on an in-vehicle display screen.

[0140] For example, when a user issues the command "open the sunroof" while driving at high speed, the dynamic hazard weight calculated in S203 is 0.675, which is greater than the preset threshold of 0.5. Since there is no safe equivalent correction scheme for opening the sunroof while driving at high speed, the system generates a rejection prompt and outputs through the voice broadcast device: "Opening the sunroof while driving at high speed may affect driving safety. It is recommended to slow down before operating."

[0141] Furthermore, user feedback data on response commands is obtained. This feedback data includes user satisfaction ratings for the execution results and / or records of manual corrections. Based on the feedback data, the cockpit alignment model is fine-tuned to generate personalized model parameters. These personalized model parameters are then uploaded to a cloud server, which aggregates the personalized model parameters from multiple cockpit terminals to obtain updated global model parameters. Finally, the updated global model parameters are received from the cloud server, and the cockpit alignment model is updated accordingly.

[0142] Satisfaction rating refers to the evaluation of the interaction result by the user through voice or touch after the system executes the response command. For example, after the system completes the operation, it can proactively ask "Are you satisfied with this service?" The user can provide feedback by saying "satisfied" or "unsatisfied". The system quantifies the feedback result into a rating of 1 to 5 stars and maps the rating to a weight value (e.g., 1 star is mapped to 0.2, and 5 stars are mapped to 1.0).

[0143] Manual correction operation records refer to the records of user manual adjustments made to the operation within a preset time after the system executes the response command. For example, if the system adjusts the air conditioner temperature to 22 degrees Celsius according to the user's voice command, and the user manually adjusts the temperature to 24 degrees Celsius within a short period of time, this manual correction operation is recorded as feedback data, indicating that there is a deviation between the response command output by the model and the user's actual expectation.

[0144] Specifically, a personalized preference loss function is constructed, which measures the deviation between the model's current output and the user's expected output. The personalized preference loss function can be expressed as:

[0145] in, As expected by the user, This is the current output of the model. Weighting is applied to user ratings. To reduce computational overhead on the terminal, the underlying network parameters of the model can be fixed during fine-tuning, with only the parameters of the upper attention and output layers being updated.

[0146] After completing local customization and fine-tuning, the generated personalized model parameters are uploaded to the cloud server. The cloud server receives personalized model parameters from multiple cockpit terminals and uses a weighted average aggregation algorithm to aggregate the parameters from all parties, obtaining the updated global model parameters. The aggregation algorithm can be expressed as:

[0147] in, These are the aggregated global model parameters. Let be the amount of local feedback data used for fine-tuning in the i-th cockpit terminal. Personalized model parameters uploaded for the i-th cockpit terminal. By weighting by data volume, terminals with larger data volumes contribute more to the global model, improving the statistical reliability of the aggregation.

[0148] After the cloud server completes parameter aggregation, it distributes the updated global model parameters to each cockpit terminal. The cockpit terminal receives the updated global model parameters from the cloud server and updates the locally deployed cockpit alignment model accordingly.

[0149] One possible implementation is to set the aggregation period for federated learning to 7 days to balance model update frequency with communication overhead. Meanwhile, the interaction logs are stored and managed in accordance with GB44495—2024 requirements, using encrypted storage algorithms (such as AES-256) for encryption, with a retention period of no less than 6 months.

[0150] Through the aforementioned federated learning mechanism, this embodiment of the application achieves continuous iterative optimization of the cockpit alignment model without requiring the acquisition of original user interaction data. While each cockpit terminal enjoys personalized services, user privacy data remains locally on the terminal, balancing personalized adaptation effectiveness with data security compliance requirements.

[0151] The above primarily describes the solutions provided in the embodiments of this application from the perspective of the device's working principle. It is understood that, in order to achieve the above functions, the interactive control device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] This application embodiment can divide the interactive control device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module.

[0153] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. When dividing functional modules according to their respective functions, Figure 3 A schematic diagram illustrating a possible composition of the interactive control device involved in the above and embodiment is shown. For example... Figure 3 As shown, the interactive control device 300 may include: an acquisition module 301, a generation module 302, a determination module 303, and a control module 304.

[0154] The acquisition module 301 is used to support the execution of the interactive control device 300. Figure 2 S201 in the schematic interactive control method.

[0155] Generation module 302 is used to support the execution of interactive control device 300. Figure 2 S202 in the schematic interactive control method.

[0156] Determine module 303, used to support the execution of interactive control device 300 Figure 2 S203 in the schematic interactive control method.

[0157] Control module 304 is used to support the execution of interactive control device 300. Figure 2 S204 in the schematic interactive control method.

[0158] One possible implementation involves a generation module that, when feeding input data into a pre-built cockpit alignment model and generating response commands based on the cockpit input data, specifically performs the following: Semantic feature extraction from the speech data in the input data to obtain a speech feature vector; visual feature extraction from the driver state data in the input data to obtain a visual feature vector; fusing the speech feature vector and the visual feature vector to obtain a fused feature vector; and generating response commands based on the fused feature vector.

[0159] One possible implementation involves a generation module that fuses speech feature vectors with visual feature vectors to obtain a fused feature vector. Specifically, this process includes: transforming the visual feature vector to obtain a query vector; transforming the speech feature vector to obtain a key vector and a value vector; determining an attention score based on the query vector and the key vector; and determining the fused feature vector based on the attention score and the value vector.

[0160] In one possible implementation, the interactive control device provided in this application embodiment is further configured to: determine whether there is a correction scheme for the response command that conforms to preset safety rules when the dynamic danger weight is determined to be greater than or equal to a preset threshold; if a correction scheme exists, modify the response command according to the correction scheme to obtain a modified safety response command, and output the safety response command to the corresponding execution device; if no correction scheme exists, generate and output a rejection message.

[0161] One possible implementation involves a module used to construct the cockpit alignment model. Specifically, this module is used to: acquire preset safety rules, input data, and corresponding labeled response instructions to construct a training dataset; adjust the pre-constructed initial cockpit alignment model based on the training dataset to obtain an intermediate cockpit alignment model; construct an initial lightweight model with fewer than a threshold parameters based on the intermediate cockpit alignment model; train the initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model; and perform quantization and compression processing on the lightweight alignment model to obtain the final cockpit alignment model.

[0162] One possible implementation involves a module that, based on a training dataset, adjusts a pre-built initial cockpit alignment model to obtain an intermediate cockpit alignment model. Specifically, this module: inputs the training dataset into the initial cockpit alignment model to obtain the predicted response instructions output by the model. Based on the difference between the predicted response instructions and the corresponding labeled response instructions, it determines the intent recognition loss. It compares the predicted response instructions with preset safety rules to determine the degree of deviation and determines the rule compliance loss based on this deviation. Based on the intent recognition loss and the rule compliance loss, iteratively updates the parameters of the initial cockpit alignment model until a preset convergence condition is met, thus obtaining the intermediate cockpit alignment model.

[0163] One possible implementation involves a module used to train an initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target. Specifically, when obtaining the lightweight alignment model, this module performs the following steps: inputting the input data from the training dataset into both the intermediate cockpit alignment model and the initial lightweight model, respectively, to obtain a first output probability distribution for the intermediate cockpit alignment model and a second output probability distribution for the initial lightweight model. It then calculates the distribution difference loss between the first and second output probability distributions. Next, it extracts intermediate layer features from both the intermediate cockpit alignment model and the initial lightweight model, calculating the feature difference loss between these intermediate layer features. Finally, based on the distribution difference loss and the feature difference loss, iteratively updates the parameters of the initial lightweight model until a preset convergence condition is met, resulting in the lightweight alignment model.

[0164] In one possible implementation, the interactive control device provided in this application embodiment is further configured to: acquire user feedback data on response commands. The feedback data includes user satisfaction ratings for the execution results and / or records of manual correction operations. Based on the feedback data, the cockpit alignment model is fine-tuned to generate personalized model parameters. The personalized model parameters are uploaded to a cloud server, allowing the cloud server to aggregate the personalized model parameters uploaded from multiple cockpit terminals to obtain updated global model parameters. The updated global model parameters are received from the cloud server, and the cockpit alignment model is updated accordingly.

[0165] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0166] The interactive control device 300 provided in this application embodiment is used to perform the above-mentioned... Figure 2 The interactive control method shown can achieve the same effect as the interactive control method described above.

[0167] This application also provides an interactive control device that can execute the interactive control method and related steps described in the above method embodiments.

[0168] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed, perform the interactive control method and related steps in the above method embodiments.

[0169] This application also provides a computer program product that, when run on a computer, causes the computer to execute the interactive control method and related steps described in the above method embodiments.

[0170] In some embodiments, the methods shown in this application can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0171] This application also provides an interactive control system 100, such as... Figure 4 As shown, the interactive control system 100 includes at least one processor 401 and at least one interface circuit 402.

[0172] As an example, when the interactive control system 100 includes a processor and an interface circuit, the processor can be... Figure 4 The processor 401 shown in the solid box (or the processor 401 shown in the dashed box) can be an interface circuit. Figure 4 The interface circuit 402 is shown in the solid box (or the dashed box). When the interactive control system 100 includes two processors and two interface circuits, the two processors include... Figure 4 The processor 401 shown in the solid box and the processor 401 shown in the dashed box, these two interface circuits include Figure 4 Interface circuit 402 is shown in both solid and dashed boxes. No limitations are imposed on this.

[0173] Processor 401 and interface circuit 402 can be interconnected via a line. For example, interface circuit 402 can be used to receive signals. Alternatively, interface circuit 402 can be used to send signals to other devices (e.g., processor 401). For instance, interface circuit 402 can read computer instructions stored in memory and send those instructions to processor 401. Processor 401 executes the instructions and, in conjunction with input / output devices, implements the various steps in the above embodiments, such as implementing... Figure 2 The illustrated method embodiments represent the various steps performed. Of course, this interactive control system may also include other discrete components, and this application embodiment does not specifically limit this.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to it, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interactive control method, characterized in that, The method includes: In response to user input data, obtain the vehicle's current status data; The input data is input into a pre-built cockpit alignment model to generate response commands for the input data; Based on the current status data, determine the dynamic danger weight of the response command; If the dynamic danger weight is determined to be lower than a preset threshold, the control execution device is controlled to perform an operation that matches the response command.

2. The method according to claim 1, characterized in that, Determining the dynamic danger weight of the response command based on the current state data includes: A risk assessment is performed on the response command to determine the probability value of the command's danger. Based on the vehicle speed information in the current status data and the vehicle's maximum design speed, determine the vehicle speed risk factor; The dynamic hazard weight is determined based on the instruction hazard probability and the vehicle speed risk factor.

3. The method according to claim 1, characterized in that, The step of inputting the input data into a pre-built cockpit alignment model and generating response instructions for the input data includes: Semantic features are extracted from the speech data in the input data to obtain a speech feature vector; Visual features are extracted from the driver status data in the input data to obtain a visual feature vector; The speech feature vector and the visual feature vector are fused to obtain a fused feature vector; The response command is generated based on the fused feature vector.

4. The method according to claim 3, characterized in that, The step of fusing the speech feature vector with the visual feature vector to obtain a fused feature vector includes: The visual feature vector is transformed to obtain the query vector; The speech feature vector is transformed to obtain a key vector and a value vector; The attention score is determined based on the query vector and the key vector; The fusion feature vector is determined based on the attention score and the value vector.

5. The method according to claim 1, characterized in that, The method further includes: If the dynamic hazard weight is determined to be greater than or equal to the preset threshold, it is determined whether there is a correction scheme for the response command that conforms to the preset safety rules. If the aforementioned correction scheme exists, the response command is corrected according to the correction scheme to obtain a corrected security response command, and the security response command is output to the corresponding execution device; If the aforementioned solution does not exist, generate and output a rejection message.

6. The method according to claim 1, characterized in that, The process of constructing the cockpit alignment model includes: Obtain preset security rules, input data, and corresponding labeled response instructions to construct a training dataset; Based on the training dataset, the pre-built initial cockpit alignment model is adjusted to obtain an intermediate cockpit alignment model; Based on the aforementioned cockpit alignment intermediate model, an initial lightweight model with fewer than a threshold number of parameters is constructed. Using the output distribution of the intermediate cockpit alignment model as a reference target, the initial lightweight model is trained to obtain a lightweight alignment model; The lightweight alignment model is subjected to quantization and compression to obtain the cockpit alignment model.

7. The method according to claim 6, characterized in that, The step of adjusting the pre-built initial cockpit alignment model based on the training dataset to obtain an intermediate cockpit alignment model includes: Input data from the training dataset is fed into the cockpit alignment initial model to obtain the predicted response instructions output by the model. The intent recognition loss is determined based on the difference between the predicted response command and the corresponding labeled response command; The predicted response command is compared with the preset security rule to determine the degree of deviation between the predicted response command and the preset security rule, and the rule compliance loss is determined based on the degree of deviation. Based on the intent recognition loss and the rule compliance loss, the parameters of the initial cockpit alignment model are iteratively updated until the preset convergence condition is met, thus obtaining the intermediate cockpit alignment model.

8. The method according to claim 6, characterized in that, The step of training the initial lightweight model using the output distribution of the intermediate cockpit alignment model as a reference target to obtain a lightweight alignment model includes: The input data in the training dataset is input into the cockpit alignment intermediate model and the initial lightweight model respectively, to obtain the first output probability distribution of the cockpit alignment intermediate model and the second output probability distribution of the initial lightweight model respectively; Calculate the distribution difference loss between the first output probability distribution and the second output probability distribution; Extract the intermediate layer features of the cockpit alignment intermediate model and the intermediate layer features of the initial lightweight model, and calculate the feature difference loss between the intermediate layer features; Based on the distribution difference loss and the feature difference loss, the parameters of the initial lightweight model are iteratively updated until the preset convergence condition is met, thus obtaining the lightweight alignment model.

9. The method according to claim 1, characterized in that, The method further includes: Obtain user feedback data in response to the response command; the feedback data includes user satisfaction ratings for the execution results and / or records of manual correction operations; Based on the feedback data, the cockpit alignment model is fine-tuned to generate personalized model parameters. The personalized model parameters are uploaded to the cloud server, so that the cloud server can aggregate the personalized model parameters uploaded by multiple cockpit terminals to obtain the updated global model parameters. Receive the updated global model parameters from the cloud server and update the cockpit alignment model.

10. An interactive control device, characterized in that, The device includes: The acquisition module is used to acquire the current status data of the vehicle in response to user input data; The generation module is used to input the input data into a pre-built cockpit alignment model and generate response instructions for the input data; The determination module is used to determine the dynamic danger weight of the response instruction based on the current state data; The control module is used to control the execution device to perform an operation that matches the response command when it is determined that the dynamic hazard weight is lower than a preset threshold and the preset safety rules are met.

Images