Method and apparatus for determining intent

Abstract

This disclosure defines a method and apparatus for determining intent. The method includes: acquiring a voice command sent by a target object and determining the text features corresponding to the voice command; monitoring data frames on a target bus of a vehicle and determining first state data corresponding to the vehicle based on the data frames; collecting facial key points of the target object and determining second state data corresponding to the target object based on the facial key points; calculating a first confidence level of the text features for each candidate intent, a second confidence level of the first state data for each candidate intent, and a third confidence level of the second state data for each candidate intent; and determining the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent. According to this disclosure, the problem of in-vehicle voice assistants' insufficient ability to fuse multimodal information, leading to the inability to accurately determine the user's deep intent, is solved in related technologies.

Description

Technical Field

[0001] This disclosure relates to the field of in-vehicle voice interaction technology, and more specifically, to a method and apparatus for determining intent. Background Technology

[0002] Existing in-vehicle voice systems mostly rely on semantic understanding methods such as keyword matching and template filling, which are insufficient in capturing the rich expressions and implicit intentions of natural language. Furthermore, existing in-vehicle voice systems lack effective long-term contextual memory mechanisms, making it difficult to track topic evolution and changes in user intent across dialogue turns. Therefore, a common problem with existing in-vehicle voice systems is insufficient fusion of multimodal perceptual information. Specifically, voice assistants only process audio signals and fail to effectively integrate Controller Area Network (CAN) bus data (e.g., vehicle speed, battery level, and range), visual perceptual information (e.g., driver status, passenger information), and diverse contextual information such as calendar time and geographical location.

[0003] In summary, in-vehicle voice assistants in related technologies suffer from insufficient ability to fuse multimodal information, resulting in an inability to accurately determine the user's deeper intentions.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a method and apparatus for determining intent, which at least solves the problem in the related art where the inability of in-vehicle voice assistants to accurately determine the user's deep intent is due to insufficient ability to fuse multimodal information.

[0006] According to one embodiment of this disclosure, a method for determining intent is provided, comprising: acquiring a voice command sent by a target object, and extracting features from the voice command to determine text features corresponding to the voice command; monitoring data frames on a target bus of a vehicle, and determining first state data corresponding to the vehicle based on the data frames; acquiring facial key points of the target object, and determining second state data corresponding to the target object based on the facial key points; calculating a first confidence level of the text features for each candidate intent, a second confidence level of the first state data for each candidate intent, and a third confidence level of the second state data for each candidate intent; and determining the target intent corresponding to the voice command from multiple candidate intents based on the first confidence level, the second confidence level, and the third confidence level corresponding to each candidate intent.

[0007] Optionally, feature extraction is performed on the voice command to determine the text features corresponding to the voice command, including: parsing the voice command to determine the text data corresponding to the voice command, and extracting context information corresponding to the voice interaction data between the target object and the vehicle; parsing the text data according to the natural language processing model built into the vehicle to extract semantic elements in the text data; and determining the text features corresponding to the voice command based on the semantic elements and the context information.

[0008] Optionally, determining the first state data corresponding to the vehicle based on the data frame includes: obtaining a first data identifier corresponding to each type of vehicle state data from the database file corresponding to the vehicle; extracting a target data frame corresponding to each data identifier from the data frame, and parsing each target data frame to determine the vehicle state data of each type, wherein the first state data includes: multiple types of vehicle state data, and the first state data includes at least one of the following: vehicle speed data, remaining battery power data, and driving range data.

[0009] Optionally, determining the second state data corresponding to the target object based on the facial key points includes: calculating the gaze direction corresponding to the target object based on the first coordinate of the first key point corresponding to the target object's eyes, the second coordinate of the second key point corresponding to the target object's eyes, and the third coordinate of the center of the target object's pupil; calculating the fatigue level of the target object within a preset time window based on the coordinates of multiple key points corresponding to the target object's eyes, wherein the multiple key points corresponding to the target object's eyes include at least the first key point and the second key point, the facial key points include the multiple key points corresponding to the target object's eyes and the center of the pupil, and the second state data includes at least one of the following: the gaze direction and the fatigue level.

[0010] Optionally, calculating the gaze direction of the target object based on the first coordinates of the first key point corresponding to the target object's eyes, the second coordinates of the second key point corresponding to the target object's eyes, and the third coordinates of the target object's pupil center includes: calculating the geometric center corresponding to the target object's eyes based on the first and second coordinates, and calculating the gaze vector corresponding to the target object based on the geometric center and the third coordinates; calculating the head yaw angle of the target object based on the horizontal difference between the fourth coordinates corresponding to the target part of the target object and the coordinates of the target object's eye centers and the focal length of the image acquisition device in the vehicle; calculating the horizontal gaze angle corresponding to the target object based on the head yaw angle and the gaze vector; and calculating the gaze direction based on the horizontal gaze angle and the bounding box of a preset area.

[0011] Optionally, calculating the fatigue level of the target object within a preset time window based on the coordinates of multiple key points corresponding to the target object's eyes includes: calculating the aspect ratio of the target object's eyes based on the coordinates of multiple key points corresponding to the target object; calculating the duration of each blink of the target object based on the aspect ratio of the eyes, and calculating the total duration of blinks of the target object within the preset time window based on the duration; and calculating the fatigue level based on the total duration and the blink frequency of the target object within the preset time window.

[0012] Optionally, determining the target intent corresponding to the voice command from multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent includes: calculating the comprehensive confidence level corresponding to each candidate intent according to a target formula, wherein the target formula is: , Let i be the overall confidence level corresponding to the i-th candidate intent. Let be the first confidence level corresponding to the i-th candidate intent. Let be the first weight of the first confidence level corresponding to the i-th candidate intent. The second confidence level corresponds to the i-th candidate intent. The second weight is the second confidence level corresponding to the i-th candidate intent. The third confidence level corresponds to the i-th candidate intent. The third weight is the third confidence level corresponding to the i-th candidate intent; the maximum comprehensive confidence level is determined among multiple comprehensive confidence levels, and the candidate intent corresponding to the maximum comprehensive confidence level is determined as the target intent.

[0013] Optionally, after determining the target intent corresponding to the voice command from multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent, the method further includes: determining at least one service with a mapping relationship to the target intent and service parameters corresponding to each service in a target rule base; constructing a target script based on the at least one service and the service parameters corresponding to each service, and generating a visual service combination based on the target script; and, upon receiving a confirmation instruction sent by the target object based on the visual service combination, sending the target script to the scheduler corresponding to the at least one service in the vehicle, so that the scheduler corresponding to the at least one service calls the at least one service.

[0014] Optionally, determining at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service in the target rule base includes: when the network status corresponding to the vehicle is online, determining at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service in a first rule base; when the network status is offline, determining at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service in a second rule base, wherein the target rule base includes: the first rule base and the second rule base.

[0015] According to one embodiment of this disclosure, an intent determination apparatus is also provided, comprising: an acquisition module, configured to acquire a voice command sent by a target object, and perform feature extraction on the voice command to determine text features corresponding to the voice command; and monitor data frames on a target bus of a vehicle, and determine first state data corresponding to the vehicle based on the data frames; and collect facial key points of the target object, and determine second state data corresponding to the target object based on the facial key points; a calculation module, configured to calculate a first confidence level of the text features for each candidate intent, a second confidence level of the first state data for each candidate intent, and a third confidence level of the second state data for each candidate intent; and a determination module, configured to determine the target intent corresponding to the voice command among multiple candidate intents based on the first confidence level, the second confidence level, and the third confidence level corresponding to each candidate intent.

[0016] According to one embodiment of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, the device where the storage medium is located executes the intention determination method of the embodiments of this disclosure.

[0017] According to one embodiment of this disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the intent determination method in the embodiments of this disclosure when it runs.

[0018] According to one embodiment of this disclosure, a computer program product is also provided, including a computer program, and a method for determining the intent of the embodiments of this disclosure when the computer program is executed by a processor.

[0019] The method for determining intent in this embodiment includes: determining the text features corresponding to the voice command, the first state data of the vehicle, and the second state data corresponding to the target object; then calculating the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent; and determining the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, the second confidence level, and the third confidence level corresponding to each candidate intent. According to this embodiment, the problem of in-vehicle voice assistants' insufficient ability to fuse multimodal information, leading to the inability to accurately determine the user's deep intent, is solved, and the ability of in-vehicle voice assistants to fuse multimodal information is improved, thereby enabling accurate determination of the user's deep intent. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0021] Figure 1 This is a flowchart of a method for determining an intent according to one embodiment of the present disclosure;

[0022] Figure 2 This is an architecture diagram of an in-vehicle voice interaction system according to an embodiment of the present disclosure;

[0023] Figure 3 This is a schematic diagram of an information perception layer according to one embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of an intelligent analysis layer according to one embodiment of the present disclosure;

[0025] Figure 5 This is a schematic diagram of a service orchestration engine according to one embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of a service execution and integration layer according to one embodiment of the present disclosure;

[0027] Figure 7 This is a flowchart of an in-vehicle voice interaction method according to one embodiment of the present disclosure;

[0028] Figure 8 This is a schematic diagram of a preferred model of an NLP intent recognition unit according to one embodiment of the present disclosure;

[0029] Figure 9 This is a structural block diagram of an intent-based device according to one embodiment of the present disclosure. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] In-vehicle voice interaction technology has evolved from basic command recognition to natural language understanding and is rapidly transforming towards multimodal interaction. Current in-vehicle voice interaction technologies generally adopt a basic model of "device-side acquisition + cloud processing." This model uses an in-vehicle microphone array to acquire user voice, performs basic signal processing on the device side (e.g., noise reduction, echo cancellation), then uploads the voice data to a cloud server for in-depth processing, and finally returns the commands to the vehicle's infotainment system for execution.

[0033] Existing in-vehicle voice interaction architectures typically consist of four modules: a voice acquisition module, a network transmission module, a cloud processing module, and a command execution module. While this architecture can ideally achieve basic voice interaction, current in-vehicle voice interaction systems overly rely on cloud computing power, resulting in a severe underestimation and underutilization of edge computing capabilities. Existing edge-cloud collaborative solutions complete wake-up verification, privacy data isolation, and basic command recognition on the edge, transmitting only the purified, effective voice signal to the cloud. However, this approach remains limited in edge-side capabilities under complex semantic understanding and multi-service orchestration scenarios. Furthermore, at the algorithm model level, existing systems largely rely on keyword matching and template filling for semantic understanding, lacking the ability to capture the rich expressions and implicit intentions of natural language. Simultaneously, existing in-vehicle voice interaction systems lack effective long-range contextual memory mechanisms, making it difficult to track topic evolution and changes in user intent across dialogue turns. Moreover, at the system integration level, data silos exist between various systems within the vehicle, hindering the voice module's ability to acquire and integrate multi-dimensional data across systems. In addition, multimodal data processing places high demands on the computing resources of the vehicle's infotainment system, making it difficult to achieve real-time response in hardware-constrained environments.

[0034] Existing in-vehicle voice interaction technologies include: 1) Generating and updating virtual characters by collecting user data, enabling voice assistants to have a certain degree of adaptability. However, this method is still limited to adjusting the voice assistant's image and basic interaction mode, with limited support for deep semantic understanding and personalized service combinations. 2) Improving the interactive experience by analyzing personalized information in user voice, but this method mainly focuses on personalization in the speech synthesis stage, with limited improvement on deep semantic understanding and service triggering. 3) Methods based on large language models have significant advantages in terms of semantic understanding depth and contextual grasp capabilities, but their strong dependence on cloud computing power will significantly reduce the system's availability and reliability in driving environments with unstable network signals.

[0035] Therefore, the existing in-vehicle voice interaction architecture has the following shortcomings:

[0036] 1) Poor environmental adaptability and strong network dependence:

[0037] Existing in-vehicle voice interaction systems experience a significant drop in recognition accuracy when faced with complex in-vehicle environments. Tire noise, wind noise, and in-vehicle entertainment system sounds generated during vehicle movement create complex background noise, severely interfering with voice acquisition quality. Despite employing machine learning-based dynamic noise reduction algorithms, voice wake-up rate and recognition accuracy remain difficult to guarantee under real-world road conditions, especially at high speeds. Furthermore, existing technologies are heavily reliant on network connectivity. When vehicles travel in areas with poor network signals, such as underground parking lots, remote mountainous areas, or highway tunnels, the voice assistant essentially fails due to the inability to connect to cloud services. For example, when a user asks "What's the story behind Yunnan Crossing-the-Bridge Rice Noodles?" in an area with poor network signal, the in-vehicle system remains unresponsive for up to 10 seconds, ultimately providing an irrelevant answer.

[0038] 2) Superficial semantic understanding and lack of personalization:

[0039] Existing in-vehicle voice interaction systems suffer from significant shortcomings in semantic depth understanding, particularly in their ability to comprehend ambiguous commands and implicit intentions. For example, when a user utters an abstract word like "entertainment," the system cannot infer whether the user is referring to an in-vehicle game, a short video application (APP), or a storytelling app based on user profile, current time, and vehicle status. It can only provide a standardized response or require the user to specify further instructions. The direct consequence of this lack of personalized service is a mechanical and alienating user experience. Existing systems typically employ a generic response model, failing to adapt to users' unique language habits, behavioral preferences, and scenario needs. For instance, when a young office worker says "take a break," they might expect upbeat music and a seat adjusted to relaxation mode; while a middle-aged business professional saying the same thing might prefer soft background music and a massage function. However, existing systems cannot achieve this differentiated response.

[0040] 3) Insufficient multimodal information fusion:

[0041] Existing in-vehicle voice systems generally suffer from insufficient multimodal perception information fusion. Specifically, voice assistants only process audio signals, failing to effectively integrate vehicle CAN bus data (e.g., vehicle speed, battery level, remaining range), visual perception information (e.g., driver status, passenger information), and diverse contextual information such as calendar time and geographical location. For example, when a user says "I'm tired" during a long drive, ideally, the in-vehicle voice system should comprehensively consider information such as the vehicle being on a highway, the continuous driving time exceeding two hours, and driver eye fatigue, and then intelligently trigger a series of services such as "suggesting a rest stop, lowering the air conditioning temperature, and playing refreshing music." However, existing systems, due to limited multimodal fusion capabilities, are clearly inadequate in understanding and processing such complex scenarios.

[0042] Multimodal fusion refers to the technology of jointly reasoning by comprehensively utilizing information from multiple different sources (modals) such as voice, vehicle data, and visual images.

[0043] In summary, existing in-vehicle voice interaction technologies still have significant shortcomings in key dimensions such as personalized services, weak network adaptability, deep semantic understanding, and multimodal information fusion. These limitations not only affect the user experience but may also pose security risks in certain scenarios.

[0044] According to embodiments of this disclosure, a method embodiment for determining intent is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] This method embodiment can be executed in an electronic device or similar computing device that includes memory and a processor. Taking operation on a computer terminal as an example, the computer terminal may include one or more processors (processors may include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), digital signal processing (DSP) chips, microcontroller units (MCUs), field-programmable gate arrays (FPGAs), neural network processors (NPUs), tensor processors (TPUs), artificial intelligence (AI) type processors, etc.) and memory for storing data. Optionally, the computer terminal may also include transmission devices, input / output devices, and display devices for communication functions. Those skilled in the art will understand that the above structural description is merely illustrative and does not limit the structure of the computer terminal. For example, the computer terminal may include more or fewer components than described above, or have a different configuration than described above.

[0046] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the intent determination method in this embodiment. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby implementing the aforementioned intent determination method. The memory may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] Display devices can be, for example, touchscreen liquid crystal displays (LCDs) and touch displays (also referred to as "touchscreens" or "touch displays"). The LCD allows users to interact with the user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), which allows users to interact with the GUI through finger touches and / or gestures on a touch-sensitive surface.

[0049] Figure 1 This is a flowchart illustrating a method for determining intent according to one embodiment of the present disclosure, such as... Figure 1 As shown, the method includes the following steps:

[0050] Step S11: Obtain the voice command sent by the target object, and extract features from the voice command to determine the text features corresponding to the voice command; and monitor the data frame on the target bus of the vehicle, and determine the first state data corresponding to the vehicle based on the data frame; and collect the facial key points of the target object, and determine the second state data corresponding to the target object based on the facial key points.

[0051] The target bus mentioned above is the CAN bus. The voice commands can be sent by the target object, i.e., the driver, or by someone else, such as the person sitting in the passenger seat.

[0052] The aforementioned first state data includes: vehicle speed signal, remaining battery power, and driving range.

[0053] The aforementioned second state data includes: the driver's line of sight, fatigue level, etc.

[0054] Step S12: Calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent;

[0055] Step S13: Determine the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent.

[0056] Based on the above steps, the text features corresponding to the voice command, the first state data of the vehicle, and the second state data corresponding to the target object are determined. Then, the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent are calculated. Based on the first, second, and third confidence levels corresponding to each candidate intent, the target intent corresponding to the voice command is determined from among multiple candidate intents. According to the embodiments of this disclosure, the problem of in-vehicle voice assistants' insufficient ability to fuse multimodal information, leading to the inability to accurately determine the user's deep intent, is solved, and the in-vehicle voice assistant's ability to fuse multimodal information is improved, thereby enabling accurate determination of the user's deep intent.

[0057] Figure 2 This is an architecture diagram of an in-vehicle voice interaction system according to an embodiment of the present disclosure, used to implement the methods of steps S11-S13 above. The hardware foundation of the in-vehicle voice interaction system is an in-vehicle central computing platform. The core components of the in-vehicle central computing platform include: a high-performance microcontroller unit (MCU), a dedicated digital signal processor (DSP) for audio processing, an in-vehicle network CAN bus interface, a fourth-generation mobile communication technology / fifth-generation mobile communication technology (4G / 5G) cellular network module, a wireless fidelity (Wi-Fi) / Bluetooth module, and a non-volatile large-capacity memory (used to store user models and local rule bases).

[0058] The preferred hardware platform for the in-vehicle voice interaction system is as follows: Main control chip (MCU): NVIDIA Jetson AGX Orin or Qualcomm SA8295P can be used as the core computing platform. These two chips integrate powerful CPU, GPU, and NPU, capable of handling local machine learning inference tasks. Storage: Equipped with at least 64GB of eMMC / UFS high-speed non-volatile memory for storing the operating system, local NLP models, user-personalized rule bases, and user habit data. Network: Supports 5G+Wi-Fi 6 dual-mode high-speed connectivity, ensuring rapid data and model synchronization with the cloud in areas with good network coverage.

[0059] In addition to using knowledge distillation to obtain AutoBERT-Lite, local NLP models can also be implemented by model pruning, quantization, or using more efficient lightweight architectures (such as MobileBERT (lightweight task-independent BERT) and TinyBERT (micro BERT)), as long as the requirements for edge deployment and operation are met.

[0060] In-vehicle voice interaction systems include, for example Figure 2 The following layers are shown: Service Execution and Integration Layer, Service Orchestration Engine, Intelligent Analytics Layer, and Information Awareness Layer.

[0061] Among them, the information perception layer, such as Figure 3 As shown:

[0062] The information perception layer includes: a voice acquisition module, a vehicle status acquisition module, and a visual perception module, used to determine text features, first state data, and second state data, wherein:

[0063] (1) Voice acquisition module: It consists of a 4-microphone array, an audio codec chip and a preamplifier, all located in the cockpit. The output of the microphone array is connected to the analog input of the audio codec chip, which is connected to the MCU via the Inter-IC Sound (I2S bus).

[0064] The voice acquisition module preferentially employs a spectrum mapping noise reduction algorithm based on deep convolutional neural networks. A lightweight deep convolutional neural network (DCNN) model is run on a DSP. This model takes the amplitude spectrum of noisy speech as input and learns an ideal speech mask through a network structure of 5 convolutional layers and 2 fully connected layers. This model is trained in the cloud using millions of hours of in-vehicle noise data and then deployed to the edge. The voice acquisition module can effectively suppress non-stationary noise (such as rain sounds and abnormal noises at specific frequencies), and can improve the speech signal-to-noise ratio by more than 15dB even when driving at high speeds of 80km / h.

[0065] The voice acquisition module is used for: acoustic-to-electrical conversion: the microphone array converts sound wave signals into analog electrical signals; analog-to-digital conversion: the audio codec chip converts analog signals into digital audio streams at a 48kHz sampling rate and 24-bit resolution; real-time noise reduction: the DSP runs a deep learning-based beamforming algorithm, which calculates the phase difference of each microphone signal and focuses on the direction of the driver's mouth to suppress environmental interference such as engine noise and wind noise. Simultaneously, an acoustic echo cancellation algorithm is applied to subtract the audio reference signal played by the vehicle's speakers from the acquired signal to prevent false triggering; and voice activity detection: the processed clean audio stream is sent to the voice activity detection module, which analyzes the short-time energy and zero-crossing rate of the signal to accurately determine the start and end of a speech segment, initiating the subsequent recognition process only when valid speech is detected.

[0066] For example, the voice acquisition module extracts features from the voice command to determine the text features corresponding to the voice command, including: parsing the voice command to determine the text data corresponding to the voice command, and extracting context information corresponding to the voice interaction data between the target object and the vehicle; parsing the text data according to the natural language processing model built into the vehicle to extract semantic elements in the text data; and determining the text features corresponding to the voice command based on the semantic elements and the context information.

[0067] Understandably, the technical solution for determining text features through a voice acquisition module is as follows:

[0068] The noise-reduced audio stream (i.e., voice commands) is converted into specific text strings using an Automatic Speech Recognition (ASR) engine. Simultaneously, current contextual information is extracted from the vehicle status module, visual perception module, and historical dialogue records. Contextual information includes, but is not limited to, current vehicle speed, remaining battery power, driver's line of sight, fatigue level, dialogue history, and geographical location. A lightweight natural language processing module deployed on the vehicle performs deep semantic analysis on the generated text data. The encoding layer understands the meaning of the text, the classification layer identifies the user's core intent (e.g., navigation, entertainment, rest), and the Conditional Random Field (CRF) layer extracts key entities (e.g., destination name, time, nickname). The extracted semantic elements (intent, entity) are then fused with the contextual information. This not only preserves the semantic vector of the text itself but also embeds contextual information as additional features. The final generated text feature is a multi-dimensional data structure that includes what the user said, the context in which it was said, the user's identity, and the current vehicle status.

[0069] (2) Vehicle Status Acquisition Module: Mainly consists of a CAN bus controller and a CAN transceiver, which are physically connected to the vehicle's CAN network. The CAN controller communicates with the MCU through the Serial Peripheral Interface (SPI interface). The CAN transceiver is connected to a twisted pair of the Controller Area Network High (CAN_H) and Controller Area Network Low (CAN_L) signal lines.

[0070] The vehicle status acquisition module handles the differences in CAN protocols across different vehicle models uniformly through the Vehicle Status Abstraction Layer (VSAL). For each vehicle model, a proprietary DBC description file is pre-installed or downloaded online. Based on the DBC file, this module parses the raw CAN ID and signals into unified key-value pairs. For example: parsing the first vehicle model: ID 0x1F0, Startbit 16, Length 16 → VehicleSpeed; parsing the second vehicle model: ID 0x2C0, Startbit 8, Length 12 → VehicleSpeed; output: regardless of the vehicle model, the upper-layer application uniformly receives the data {“VehicleSpeed”: 65}, with the unit uniformly set to km / h.

[0071] The vehicle status acquisition module performs the following steps through a CAN bus parser and a vehicle data decoder: Data monitoring: The module continuously monitors data frames on the CAN bus; Protocol parsing: Based on a pre-set CAN database file (DatabaseCAN, or DBC), it parses the data corresponding to a specific Controller Area Network Identifier (CAN ID). For example, parsing the data frame with ID 0x0CFE extracts the vehicle speed signal in km / h; parsing the data frame with ID 0x07C0 extracts the remaining battery power and driving range; Data decoding and provision: The parsed physical values ​​(e.g., vehicle speed, battery power, Global Positioning System (GPS) coordinates, and time) are provided to the upper-layer module through an internal software interface. The vehicle status acquisition module acts as a unified abstraction layer for vehicle data, shielding the differences in CAN protocols between different vehicle models.

[0072] For example, the vehicle status acquisition module determines the first status data corresponding to the vehicle based on the data frame, including: obtaining a first data identifier corresponding to each type of vehicle status data from the database file corresponding to the vehicle; extracting a target data frame corresponding to each data identifier from the data frame, and parsing each target data frame to determine the vehicle status data of each type, wherein the first status data includes: multiple types of vehicle status data, and the first status data includes at least one of the following: vehicle speed data, remaining battery power data, and driving range data.

[0073] It is understandable that the technical solution for determining the first state data through the vehicle state acquisition module is as follows:

[0074] The system pre-loads or downloads a database file (e.g., a DBC file) corresponding to the specific vehicle model. A mapping table is defined in the database file to establish a correspondence between various types of vehicle status data (including but not limited to vehicle speed, battery level, and door status) and specific data identifiers (i.e., CAN ID and signal bit definitions). The first data identifier includes, but is not limited to, the CAN frame identifier (CAN ID), the start bit of the signal in the frame, the signal length, and the signal scaling factor and offset. The onboard gateway or MCU continuously monitors all broadcast data frames on the CAN bus. Based on the mapping table, target data frames containing key vehicle status data are selected. The extracted target data frames are parsed. According to the signal format defined in the database file, the raw value is extracted from the original binary frame and converted into a value with actual physical meaning (e.g., km / h, %, km) using a preset formula (e.g., physical value = raw value × scaling factor + offset). This ultimately generates structured first status data, including but not limited to: vehicle speed, remaining battery level, and driving range.

[0075] (3) Visual perception module: An infrared camera module installed behind the steering wheel or on the A-pillar, including an image processing module. The camera is connected to the MCU via a MIPI CSI-2 interface (a camera serial interface). The visual perception module is used for: Image acquisition: The camera acquires grayscale or RGB images of the driver's area at a frame rate of 15fps; Key point detection: The MCU runs a lightweight neural network model to detect key facial points of the driver in real time (such as eyelids, corners of the mouth, and head orientation); State inference: Calculates based on key point information: Gaze direction: Determines whether the driver is looking at the central control screen by calculating the relative position of the pupil center and the corner of the eye; Fatigue level: Determines whether the driver is fatigued by calculating the proportion of eyelid closure time.

[0076] For example, the visual perception module determines the second state data corresponding to the target object based on the facial key points, including:

[0077] Step S21: Calculate the gaze direction of the target object based on the first coordinates of the first key point corresponding to the target object's eye, the second coordinates of the second key point corresponding to the target object's eye, and the third coordinates of the target object's pupil center. Specifically:

[0078] The geometric center of the target object's eyes is calculated based on the first and second coordinates, and the gaze vector of the target object is calculated based on the geometric center and the third coordinate. The head yaw angle of the target object is calculated based on the horizontal difference between the fourth coordinate of the target part of the target object and the coordinates of the center of the target object's eyes and the focal length of the image acquisition device in the vehicle. The horizontal angle of the gaze vector of the target object is calculated based on the head yaw angle and the gaze vector. The gaze direction is calculated based on the horizontal angle of the gaze and the bounding box of the preset area.

[0079] Understandably, the process begins with constructing a gaze vector: identifying key points in the driver's eyes (e.g., corners of the eyes, eyelid edges), and extracting the coordinates of these key points for the left and right eyes respectively. The geometric center of each eye (i.e., the midpoint of the line connecting the two eyes) is then calculated using a mathematical average or weighted average. The coordinates of the pupil center are then extracted. Finally, the geometric centers of both eyes are connected to the pupil center to generate an initial gaze vector.

[0080] Geometric center: ,in, Let I be the geometric center, and E be the first coordinate.

[0081] Line of sight vector: , Let P be the line-of-sight vector, and let P be the third coordinate.

[0082] Secondly, calculate the head yaw angle: identify other fixed facial features (e.g., nose tip, ears, or facial contour center) and obtain their coordinates as the fourth coordinate. Calculate the horizontal difference between the reference point and the geometric center of both eyes on the horizontal axis of the image. Combining the known focal length parameters of the image acquisition device (camera), and using geometric optics principles (such as trigonometric relationships), convert the pixel-level horizontal difference into the yaw angle of the driver's head relative to the camera's optical axis.

[0083] Head yaw angle: , The yaw angle of the nose. The x-coordinate in the fourth coordinate system. is the x-coordinate in the coordinate system of the center of both eyes, and f is the focal length.

[0084] Step S22: Calculate the fatigue level of the target object within a preset time window based on the coordinates of multiple key points corresponding to the target object's eyes. The multiple key points corresponding to the target object's eyes include at least: the first key point and the second key point. The facial key points include: the multiple key points corresponding to the target object's eyes and the pupil center. The second state data includes at least one of the following: the gaze direction and the fatigue level. Specifically:

[0085] The aspect ratio of the target object's eyes is calculated based on the coordinates of multiple key points corresponding to the target object; the duration of each blink of the target object is calculated based on the aspect ratio of the eyes, and the total duration of blinking of the target object within a preset time window is calculated based on the duration; the fatigue level is calculated based on the total duration and the blinking frequency of the target object within the preset time window.

[0086] Understandably, the first step is to calculate the eye aspect ratio: identify multiple key points on the driver's face, including the corners of the eyes (inner and outer corners) and the edges of the upper and lower eyelids. Based on the coordinates of these key points, calculate the ratio of the eye's height in the vertical direction to its width in the horizontal direction. When the eyes are open, the eye height is greater, and the aspect ratio is higher and more stable. When the eyes are closed (blinking), the upper and lower eyelids overlap, the height approaches zero, and the aspect ratio drops sharply.

[0087] Calculation of eye aspect ratio: Where EAR is the aspect ratio of the eye. The coordinates of the left eye corner are... , The coordinates are for the upper eyelid. The coordinates of the right eye corner are... , The coordinates are for the lower eyelid.

[0088] Secondly, identify blink events and calculate their duration: set a dynamic threshold. When the EAR value is below the threshold and remains below it for a certain number of frames, a blink is considered to have begun; when the EAR value rises back above the threshold, a blink is considered to have ended. The time span from the start to the end of each blink is recorded to obtain the duration of a single blink. The durations of single blinks within a preset time window are summed to obtain the total blink duration, and the total number of blinks within the preset time window is calculated. The blink frequency is then calculated based on the total blink duration and the total number of blinks.

[0089] That is: when EAR is less than In the case of EAR being less than or equal to, it is marked as closed-eye state; Restored to above In the case of a blink, it is marked as a valid blink, and the duration of each blink is recorded.

[0090] Finally, fatigue level is calculated: fatigue level is assessed by combining blink frequency and the proportion of total blink duration. For example: high frequency short blinks may indicate tension or concentration; low frequency long blinks may indicate drowsiness; excessively long total duration: if the proportion of total eye closure time exceeds the safe threshold, it is judged as high fatigue.

[0091] Total duration: , For the total duration, The time spent with eyes closed within a preset time window. Wherein, the waking state is: <10% and blinking frequency is between [15, 25] times / minute; mild fatigue: 10% ≤ <25%, or blinking frequency >35 times / minute; severe fatigue: ≥25% or single blink time >2 seconds.

[0092] In addition, biosensors (monitoring heart rate, skin conductance, etc. to determine the driver's emotional state), in-vehicle infrared sensors (monitoring the temperature distribution inside the vehicle), or steering wheel grip force sensors can be introduced as additional perception modalities to participate in intention decision-making.

[0093] In summary, existing technologies are mostly based on keyword matching, lacking a deep understanding of the intent behind vague and abstract terms such as "rest and entertainment" and "vacation," and cannot be associated with specific in-vehicle applications (such as games, navigation, and music). They can only provide formatted responses, forcing drivers to distract themselves while driving by thinking about how to organize language that the machine can understand. In contrast, the embodiments disclosed in this disclosure can understand the true intent behind abstract commands such as "entertainment" and "leisure," and accurately associate them with a series of specific application operations (such as starting a game, playing a storytelling program, or activating a seat massage). Through the deep semantic analysis capabilities of the natural language processing unit and the direct mapping of the service mapping rule base, users do not need to painstakingly organize specific language, reducing the cognitive load of interaction and allowing drivers to focus more on driving. This solves the problem of shallow semantic understanding and the inability to parse vague commands, improving the accuracy of understanding users' deep intent and vague semantics.

[0094] In addition to using production rules in the form of "IF-THEN", service mapping rules can also be represented and reasoned using decision trees, directed graphs, or ontology.

[0095] The intelligent analysis layer is responsible for transforming the raw data from the perception layer into structured user intents, such as... Figure 4 As shown, the intelligent analysis layer includes a natural language processing unit and a multimodal fusion unit (including a context manager).

[0096] When the network is smooth, the Natural Language Processing (NLP) unit is trained using a large cloud-based model to generate a lightweight local semantic understanding model, which is then downloaded to the in-vehicle infotainment system. The workflow includes: Automatic Speech Recognition (ASR): The noise-reduced audio stream is input into the on-device ASR engine and converted into text. For example, audio is converted into the text "I'm a little tired." Keyword and Entity Extractor: The local Natural Language Processing (NLP) model analyzes the text and extracts key information: Entities: such as location ("company," "home"), time ("3 PM"), intent: such as "navigation," "entertainment," "rest," user habits: such as "fastest route" preference. Contextual Association: A built-in context manager retains historical information from the last three rounds of dialogue for understanding referential words (such as "there" in "go there").

[0097] The multimodal information fusion unit is used to achieve accurate intent determination. Its working principle is as follows: Information alignment: It receives text features from the NLP unit, real-time data from the vehicle status module (e.g., vehicle speed = 0 km / h, location = underground parking garage), and driver status from the vision module (e.g., gaze directed towards the central control screen); Weighted decision algorithm: It calculates the confidence level of the final intent using the following formula: Intent output: For example, when a user says "navigate home", the fusion unit combines the information that "the vehicle is off" and "the driver is looking at the screen" to increase the final confidence of the intent "plan a route home immediately" and outputs it to the lower layer.

[0098] The aforementioned context-aware multimodal dynamic weighted fusion decision-making mechanism involves a dynamic weight adjustment algorithm for fusing speech, vehicle status (CAN bus), and visual information. The weights are not fixed but dynamically calculated based on real-time context (such as vehicle speed, noise level, and command ambiguity) to arrive at a high-confidence user intent judgment. This addresses the problem of shallow and inaccurate understanding of single-modal speech and improves the ability to understand deeper intents behind ambiguous commands.

[0099] The output of the weighted decision algorithm in the multimodal information fusion unit serves as the trigger for service orchestration.

[0100] In addition to using dynamic weighted averaging, multimodal information fusion can also employ end-to-end fusion networks based on deep learning (such as multimodal Transformer), DS evidence theory, or Bayesian inference to perform information fusion and confidence calculation.

[0101] For example, determining the target intent corresponding to the voice command among multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent includes: calculating the comprehensive confidence level corresponding to each candidate intent according to a target formula, wherein the target formula is: , Let i be the overall confidence level corresponding to the i-th candidate intent. Let be the first confidence level corresponding to the i-th candidate intent. Let be the first weight of the first confidence level corresponding to the i-th candidate intent. The second confidence level corresponds to the i-th candidate intent. The second weight is the second confidence level corresponding to the i-th candidate intent. The third confidence level corresponds to the i-th candidate intent. The third weight is the third confidence level corresponding to the i-th candidate intent; the maximum comprehensive confidence level is determined among multiple comprehensive confidence levels, and the candidate intent corresponding to the maximum comprehensive confidence level is determined as the target intent.

[0102] Understandably, after obtaining the confidence scores of each candidate intent in each modality, the overall confidence score is calculated based on the above formula, and then the target intent is determined based on the overall confidence score of each candidate intent. The first, second, and third weights are dynamically changing.

[0103] In one optional embodiment, when the vehicle is traveling at a low speed (<20km / h), the second weight corresponding to the first state information will be increased because the driver has more energy to perform gesture or eye contact interaction at this time; when a strong noise environment is detected, the first weight corresponding to the voice features will be appropriately reduced, while the second weight corresponding to the first state data will be increased accordingly; when the user command is a vague command such as "I am hungry", "geographical location" and "time" will be given higher weights (i.e., the second weight will be increased) to infer whether to recommend a breakfast, lunch or dinner location.

[0104] In summary, existing technologies process voice commands in isolation, failing to integrate and analyze them with vehicle status (such as speed and battery level), driver status (such as gaze and gestures), calendar time, and geographical location. This prevents the realization of a proactive service experience where a single command triggers multiple services to work collaboratively. However, the embodiments of this disclosure utilize a service orchestration engine, enabling the system to comprehensively analyze voice, vehicle status, and visual information to intelligently generate and execute composite service scripts. For example, upon recognizing address information during a call, it can proactively suggest and initiate navigation, resolving the issues of insufficient multimodal information fusion and inadequate service composition capabilities.

[0105] Among them, service orchestration engines: such as Figure 5As shown, the service orchestration engine is responsible for transforming abstract intents into concrete, executable service compositions. This includes a dynamic script generator and a service mapping rule base. The dynamic script generator comprises rule matching algorithms and script generation logic, while the service mapping rule base includes user-personalized rule sets and a general service rule set. The general service rule set provides the basic logic applicable to all users; the user-personalized rule set is dynamically generated and optimized on the client side through training under smooth network conditions and user feedback. The dynamic script generator's workflow includes: rule matching: receiving user intents output by the fusion unit and matching them in the rule base; parameter binding: binding context information (such as specific addresses and times) to the rule's variable slots; and generating an executable script: outputting a structured script (such as JSON format) that explicitly defines the service sequence to be executed and its parameters.

[0106] In addition to generating linear service sequences, the dynamic script generator can also generate parallel service flows (such as simultaneously lowering media volume and starting navigation) or branch scripts with conditional judgments (such as "if the battery is below 20%, navigate to the nearest charging station; otherwise, navigate to the destination").

[0107] The above embodiments construct a complete voice interaction system with local storage, local learning, and local decision-making. Its core capabilities (NLP model, user rule base) are trained and updated in the cloud when the network is smooth, but can run independently and fully functionally in offline environments. This solves the problem of functional paralysis in weak network environments caused by the strong network dependence of existing technologies. The rule base not only contains predefined rules, but can also be continuously optimized based on user feedback through online learning (such as reinforcement learning), generating executable service composition scripts, achieving a leap from single command to single service to single command to composite service. Through the above technical solution, the problems of service rigidity, lack of personalization, and inability to proactively find users for services are solved.

[0108] For example, after determining the target intent corresponding to the voice command among multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent, the method further includes:

[0109] Step S31: Determine at least one service in the target rule base that has a mapping relationship with the target intent, and the service parameters corresponding to each service; specifically:

[0110] When the network status corresponding to the vehicle is online, at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service are determined in the first rule base; when the network status is offline, at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service are determined in the second rule base, wherein the target rule base includes: the first rule base and the second rule base.

[0111] The first rule base is a set of rules for user personalization, and the second rule base is a set of rules for general services.

[0112] The first rule base includes: rules for responsible service composition generated based on large model inference, rules that require real-time cloud data, and personalized user preference rules;

[0113] The second rule base includes: basic vehicle controls (such as opening windows, adjusting the temperature, and playing local music), basic navigation (based on local map caching), and emergency safety services. In the event of a network outage, the system automatically downgrades to the second rule base.

[0114] Determine one or more services that need to be executed in the first rule base or the second rule base, such as navigation or playing music.

[0115] The construction of the first and second rule bases is a process combining offline and online learning, requiring updates to both. Specifically: when the network is stable, users' historical selection behaviors (e.g., in 10 "off work" commands, the user ultimately manually selected the "fewest traffic lights" route 8 times) are uploaded to the cloud for aggregation and analysis. The cloud uses this anonymized data to optimize the rules through reinforcement learning algorithms and silently distributes the updated personalized rule package to the vehicle, updating both the first and second rule bases.

[0116] Step S32: Construct a target script based on the at least one service and the service parameters corresponding to each service, and generate a visual service combination based on the target script;

[0117] Step S33: Upon receiving a confirmation instruction sent by the target object based on the visualization service combination, the target script is sent to the scheduler corresponding to the at least one service in the vehicle, so that the scheduler corresponding to the at least one service calls the at least one service.

[0118] In summary, existing technologies rely on cloud computing power for speech recognition and semantic analysis. While these technologies work well in areas with good network connectivity, such as home garages and low-speed urban roads, they become ineffective in areas with poor network signals, such as underground parking lots, highway tunnels, and remote areas like those in Qinghai. This disclosed embodiment, through a combination of on-device localized analysis and processing with cloud-based collaborative training when network connectivity is good, allows the system to accurately respond to user commands even in areas with poor network signals (such as tunnels and remote areas). The local speech content analysis, training, and storage mechanism avoids voice function interruptions caused by unstable network connections, significantly improving system reliability and driving safety. It overcomes network environment constraints, ensuring continuous and stable operation of the voice assistant across all scenarios and resolving the problem of functional paralysis in weak network environments caused by strong network dependence.

[0119] Among them, the service execution and integration layer, such as Figure 6As shown, the service execution and integration layer includes a non-intrusive UI prompt module and an API invoker. The non-intrusive UI prompt module includes a floating card rendering engine and a touch / voice confirmation interface. The Application Programming Interface (API) invoker includes an application API pool and a service orchestration script interpreter. To ensure broad compatibility with various in-vehicle applications, a set of in-vehicle service open API standards (VOAAS) has been defined. Any application that wishes to be invoked by the voice assistant must register within the system and declare its supported Actions and Params. The service execution and integration layer is used to execute orchestrated service scripts in a user-friendly and secure manner. The non-intrusive user interface (UI) prompt module is a high-priority graphical layer application of the vehicle's operating system. The workflow includes: rendering floating cards: after receiving the script, visualizing the suggested service combination as a semi-transparent card on the side of the screen, such as displaying "Plan the fastest route home for you and play refreshing music?"; waiting for user confirmation: providing two interaction methods: voice confirmation ("confirm execution") and touch confirmation (clicking the "OK" button). To ensure control remains with the user and prevent accidental operations, the system handles command transmission: once confirmation is received, the executable script is passed to the API scheduler. The API scheduler maintains a list of API interfaces for registered in-vehicle applications. The API scheduler's process is as follows: Script parsing: parses the JSON script from the dynamic script generator; API calls: calls are made sequentially through the standard RESTful API interfaces provided by each application, according to the service sequence in the script; for example, calling the navigation app's `NavigationAPI.startNavigation(destination, preference)`; and calling the music app's `MediaAPI.playPlaylist(playlist_name)`. Exception handling: monitors the execution status of each service. If a service call fails (e.g., the music app is not installed), it logs the failure and attempts to execute the next service, or returns an error message to the UI module.

[0120] Among them, the Vehicle Service Open API (VOAAS) can be implemented not only in a RESTful style, but also based on gRPC (Remote Procedure Call framework), MQTT (Message Queuing Telemetry Transport Protocol) or the specific IPC (Inter-Process Communication) mechanism of the vehicle system, as long as it can achieve the purpose of unified scheduling of different applications.

[0121] Non-intrusive UI refers to presenting information in a way that does not interfere with the main task (driving), and is usually characterized by being concise, short, and quick to dismiss.

[0122] Before a service is executed, in addition to requesting user confirmation via a non-intrusive UI card, it can also support preset voice quick confirmation (such as the user saying "execute as planned"), or set an automatic execution whitelist for specific low-risk services (such as playing music).

[0123] The above technical solution defines a unified open API standard for in-vehicle services, enabling voice assistants to schedule and combine in-vehicle applications (navigation, music, environmental control, etc.) from different developers through a unified interface, achieving true cross-application service integration and solving the problems of low service integration and closed ecosystem of traditional voice systems.

[0124] In one exemplary embodiment, Figure 7 This is a flowchart of an in-vehicle voice interaction method according to one embodiment of the present disclosure, applied to... Figure 2 In-vehicle voice interaction systems, such as Figure 7 As shown:

[0125] Step S71: The information perception layer continuously collects data;

[0126] Step S72, Voice activity detection, to determine whether a voice command sent by the target object has been received;

[0127] If it is determined that a voice command has been received, proceed to step S73; if it is determined that no voice command has been received within the first time period, proceed to step S81.

[0128] Step S73, speech recognition;

[0129] When a voice command is detected, speech recognition is performed on the voice command to determine the text data corresponding to the voice command;

[0130] Step S74, NLP intent recognition;

[0131] To determine multiple candidate intents;

[0132] Among them, the preferred model for the NLP intent recognition unit is as follows: Figure 8 The pruned and quantized variant of the Transformer model (Bidirectional Encoder Representations from Transformers, or BERT model for short) shown is the lightweight BERT model (AutoBERT-Lite).

[0133] Model Training: A large BERT model is trained using a massive amount of in-vehicle voice commands and dialogue text in the cloud. Then, through knowledge distillation, a small, lightweight BERT model (AutoBERT-Lite) learns the output distribution of the large model, thereby compressing the model size to less than 50MB while maintaining high performance, making it suitable for edge deployment.

[0134] Intent recognition: The lightweight BERT model encodes the input text and outputs the intent category (such as NAVIGATION_HOME, ENTERTAINMENT_MUSIC) and its confidence score through a classification layer.

[0135] Entity extraction: A Conditional Random Field (CRF) layer is used to identify named entities in the text, such as location, time, song title, etc.

[0136] When the network is online, the results obtained by the lightweight BERT model are sent to the local model and rules on the vehicle through the cash algorithm. When the network is offline, the local model and rules on the vehicle are updated and adapted to obtain anonymous data on the vehicle. The anonymous data on the vehicle is then aggregated and retrained and sent to the cloud to send the model.

[0137] Step S75, multimodal fusion;

[0138] The text features, first-state data, and second-state data are fused in a multimodal manner, and the comprehensive confidence level corresponding to each candidate intent is determined.

[0139] Step S76, rule matching;

[0140] Determine the service and service parameters corresponding to the target intent;

[0141] Step S77: Generate service script;

[0142] Generate a service script corresponding to the target intent based on the service and service parameters corresponding to the target intent.

[0143] Step S78: Send a UI prompt to the target object so that the target object can confirm the service and service parameters;

[0144] If the target object clicks "Confirm," proceed to step S79.

[0145] The process ends if the target does not click "confirm".

[0146] Step S79: API invokes and executes the service;

[0147] For example: Launch navigation + music service;

[0148] Step S80: Update user preferences;

[0149] Record current user behavior to update user preferences.

[0150] Step S81: Continuous monitoring.

[0151] In one embodiment, the scenario is: a 25-year-old young office worker leaving get off work.

[0152] The vehicle starts in the underground parking garage, and the network connection is good. The user says, "Off work." ASR converts the voice command into the text "Off work." NLP recognizes the intent: INTENT_COMMUTE_HOME. The vehicle's CAN bus obtains the GPS coordinates (company coordinates), Time (18:30), and Day (Wednesday). The camera detects that the driver's face is facing forward, with no obvious signs of fatigue. After calculation using a dynamic weighting algorithm, the final overall confidence level of the target intent reaches 95%. The rule base matches the personalized rule R001_UserA, generating a script. The Route parameter in the script is not fixed to "fastest," but references User_Preference.CommuteRoutePreference. This preference is a dynamic value derived through machine learning by analyzing the user's actual route choices on their way home from get off work over the past 30 times (e.g., on Mondays, the user prefers "fastest," and on Fridays, they prefer "most fuel-efficient"). The UI card displays: "Navigate home, via XX elevated highway, estimated 35 minutes, and play your favorite electronic music playlist?" The user confirms with a voice "OK"; the API scheduler sequentially calls the application interfaces of XX Maps and the music player, seamlessly starting the service.

[0153] In another embodiment, the scenario is: a 30-year-old housewife picking up and dropping off her child.

[0154] The user said, "Go pick up the baby."

[0155] The NLP intent identifies the user-defined nickname "Baby" and resolves it to the entity PERSON_Daughter by querying the local "Family Relationship-Nickname" mapping table. Combining calendar information (known that the daughter gets out of school at 3:00 PM every Monday to Friday), the current time (2:40 PM), and vehicle status (full fuel), the fusion unit determines with high confidence that the user intent is NAVIGATION_TO_SCHOOL. Navigation is initiated, with the destination set to: XX Elementary School. A pre-set message is automatically sent to the daughter's smartwatch: "Mom has set off to pick you up, please wait at the school gate." The car's interior temperature is checked; if too high, the air conditioning is activated to cool it down. Learning that the user usually stops at a nearby bakery to buy snacks before picking up the child, "XX Bakery" is automatically added as a waypoint in the route planning.

[0156] In another embodiment, the scenario is: health care for a 60-year-old retired teacher.

[0157] During a long drive, the user muttered to himself, "It's a bit stuffy."

[0158] NLP identified the vague word "stuffy". The fusion unit combined the following information: visual (detected a slight frown from the driver), vehicle status (2 hours of continuous driving, windows closed, elevated CO2 concentration sensor reading), and user profile (age 60, history of hypertension). The system determined the user's true intent to be ENVIRONMENT_REFRESH and HEALTH_CONCERN. The main service automatically switched the air conditioning to "external circulation" and slightly lowered the temperature. Additional services included a low-volume voice announcement: "Ventilation has been provided. You have been driving continuously for 2 hours. There is a service area 3 kilometers ahead; a 15-minute rest is recommended." Proactive care was indicated on a UI card: "Playing a soothing piano piece and activating gentle seat ventilation." The system not only responded to surface commands but also gained insights into the user's deeper health needs through multimodal information, providing proactive and considerate services.

[0159] Service orchestration refers to the automatic combination, sorting, and execution of multiple individual services based on a complex user intent, forming a coherent and complex task process.

[0160] In summary, existing technologies employ a generic response model, which cannot remember and learn the unique preferences and habits of different users (such as young office workers, business professionals, and the elderly). This results in repetitive setup for each interaction, lacking a sense of personalization and exclusivity. Through the aforementioned exemplary scenarios, by locally storing user habits and continuously learning, the system can provide tailored services for different users. For example, upon recognizing a 25-year-old user saying "off work," the system automatically plans their route home and plays uplifting music; when a 45-year-old business user executes the "welcome" command, the system automatically navigates to the airport and plays elegant music. This solves the problems of missing personalized services and forgotten user habits. The local user profile learning mechanism and multimodal information fusion unit eliminate the tedious process of repeatedly teaching users habits, making the interaction experience natural and smooth.

[0161] Through the above exemplary embodiments, the end-to-end voice command response latency is <800ms (even in a network-free environment). The local NLP model achieves >92% accuracy in intent recognition for commonly used commands, and multimodal fusion can improve the confidence of intent judgment by an average of 25% compared to single voice recognition.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0163] This disclosure also provides an intent-determining device for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0164] Figure 9 This is a structural block diagram of an intent-based determining apparatus according to one embodiment of the present disclosure, such as... Figure 9 As shown, the device includes:

[0165] The acquisition module 901 is used to acquire voice commands sent by the target object, and extract features from the voice commands to determine the text features corresponding to the voice commands; and to monitor data frames on the target bus of the vehicle and determine the first state data corresponding to the vehicle based on the data frames; and to collect facial key points of the target object and determine the second state data corresponding to the target object based on the facial key points.

[0166] The calculation module 902 is used to calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent;

[0167] The determining module 903 is used to determine the target intent corresponding to the voice command among multiple candidate intents based on the first confidence level, the second confidence level and the third confidence level corresponding to each candidate intent.

[0168] Optionally, the acquisition module 901 is further configured to parse the voice command to determine the text data corresponding to the voice command, and extract context information corresponding to the voice interaction data between the target object and the vehicle; parse the text data according to the natural language processing model built into the vehicle to extract semantic elements in the text data; and determine the text features corresponding to the voice command according to the semantic elements and the context information.

[0169] Optionally, the acquisition module 901 is further configured to acquire a first data identifier corresponding to each type of vehicle status data from the database file corresponding to the vehicle; extract a target data frame corresponding to each data identifier from the data frame, and parse each target data frame to determine the vehicle status data of each type, wherein the first status data includes: multiple types of vehicle status data, and the first status data includes at least one of the following: vehicle speed data, remaining battery power data, and driving range data.

[0170] Optionally, the acquisition module 901 is further configured to calculate the gaze direction of the target object based on the first coordinate of the first key point corresponding to the eyes of the target object, the second coordinate of the second key point corresponding to the eyes of the target object, and the third coordinate of the pupil center of the target object; and to calculate the fatigue level of the target object within a preset time window based on the coordinates of multiple key points corresponding to the eyes of the target object, wherein the multiple key points corresponding to the eyes of the target object include at least the first key point and the second key point, the facial key points include the multiple key points corresponding to the eyes of the target object and the pupil center, and the second state data includes at least one of the following: the gaze direction and the fatigue level.

[0171] Optionally, the acquisition module 901 is further configured to calculate the geometric center corresponding to the eyes of the target object based on the first coordinate and the second coordinate, and calculate the gaze vector corresponding to the target object based on the geometric center and the third coordinate; calculate the head yaw angle of the target object based on the horizontal difference between the fourth coordinate corresponding to the target part of the target object and the coordinates of the center of the eyes of the target object and the focal length of the image acquisition device in the vehicle; calculate the gaze horizontal angle corresponding to the target object based on the head yaw angle and the gaze vector; and calculate the gaze direction based on the gaze horizontal angle and the bounding box of the preset area.

[0172] Optionally, the acquisition module 901 is further configured to calculate the eye aspect ratio of the target object based on the coordinates of multiple key points corresponding to the target object; calculate the duration of each blink of the target object based on the eye aspect ratio, and calculate the total blink duration of the target object within a preset time window based on the duration; and calculate the fatigue level based on the total duration and the blink frequency of the target object within the preset time window.

[0173] Optionally, the determining module 903 is further configured to calculate the comprehensive confidence level corresponding to each candidate intent according to the target formula, wherein the target formula is: , Let i be the overall confidence level corresponding to the i-th candidate intent. Let be the first confidence level corresponding to the i-th candidate intent. Let be the first weight of the first confidence level corresponding to the i-th candidate intent. The second confidence level corresponds to the i-th candidate intent. The second weight is the second confidence level corresponding to the i-th candidate intent. The third confidence level corresponds to the i-th candidate intent. The third weight is the third confidence level corresponding to the i-th candidate intent; the maximum comprehensive confidence level is determined among multiple comprehensive confidence levels, and the candidate intent corresponding to the maximum comprehensive confidence level is determined as the target intent.

[0174] Optionally, the determining module 903 is further configured to determine at least one service that has a mapping relationship with the target intent and service parameters corresponding to each service in the target rule base; construct a target script based on the at least one service and the service parameters corresponding to each service, and generate a visual service combination based on the target script; and, upon receiving a confirmation instruction sent by the target object based on the visual service combination, send the target script to the scheduler corresponding to the at least one service in the vehicle, so that the scheduler corresponding to the at least one service calls the at least one service.

[0175] Optionally, the determining module 903 is further configured to, when the network status corresponding to the vehicle is online, determine in the first rule base the at least one service that has a mapping relationship with the target intent and the service parameters corresponding to each service; and when the network status is offline, determine in the second rule base the at least one service that has a mapping relationship with the target intent and the service parameters corresponding to each service, wherein the target rule base includes: the first rule base and the second rule base.

[0176] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0177] According to another aspect of the present disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein the method for determining the intent to control the device where the storage medium is located to perform the above-described actions is provided when the executable program is running.

[0178] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0179] S1, acquire the voice command sent by the target object, and extract features from the voice command to determine the text features corresponding to the voice command; and monitor the data frame on the target bus of the vehicle, and determine the first state data corresponding to the vehicle based on the data frame; and collect the facial key points of the target object, and determine the second state data corresponding to the target object based on the facial key points.

[0180] S2, calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent;

[0181] S3, determine the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, second confidence level and third confidence level corresponding to each candidate intent respectively.

[0182] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0183] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described method for determining intent during runtime.

[0184] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0185] S1, acquire the voice command sent by the target object, and extract features from the voice command to determine the text features corresponding to the voice command; and monitor the data frame on the target bus of the vehicle, and determine the first state data corresponding to the vehicle based on the data frame; and collect the facial key points of the target object, and determine the second state data corresponding to the target object based on the facial key points.

[0186] S2, calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent;

[0187] S3, determine the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, second confidence level and third confidence level corresponding to each candidate intent respectively.

[0188] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program and a method for determining whether the computer program, when executed by a processor, achieves the above-described intent.

[0189] Optionally, in this embodiment, the above-mentioned computer program product can be configured as a computer program that performs the following steps:

[0190] S1, acquire the voice command sent by the target object, and extract features from the voice command to determine the text features corresponding to the voice command; and monitor the data frame on the target bus of the vehicle, and determine the first state data corresponding to the vehicle based on the data frame; and collect the facial key points of the target object, and determine the second state data corresponding to the target object based on the facial key points.

[0191] S2, calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent;

[0192] S3, determine the target intent corresponding to the voice command from among multiple candidate intents based on the first confidence level, second confidence level and third confidence level corresponding to each candidate intent respectively.

[0193] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0197] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0198] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A method for determining intent, characterized in that, include: The voice command sent by the target object is obtained, and the voice command is feature extracted to determine the text features corresponding to the voice command. In addition, monitoring data frames on the target bus of the vehicle and determining first state data corresponding to the vehicle based on the data frames; and collecting facial key points of the target object and determining second state data corresponding to the target object based on the facial key points. Calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent; The target intent corresponding to the voice command is determined from multiple candidate intents based on the first confidence level, the second confidence level, and the third confidence level corresponding to each candidate intent.

2. The method for determining intent according to claim 1, characterized in that, Feature extraction is performed on the voice command to determine the text features corresponding to the voice command, including: The voice command is parsed to determine the text data corresponding to the voice command, and the context information corresponding to the voice interaction data between the target object and the vehicle is extracted. The text data is parsed using the vehicle's built-in natural language processing model to extract semantic elements from the text data; The text features corresponding to the voice command are determined based on the semantic elements and the context information.

3. The method for determining intent according to claim 1, characterized in that, Determining the first state data corresponding to the vehicle based on the data frame includes: Obtain the first data identifier corresponding to each type of vehicle status data from the database file corresponding to the vehicle. Extract the target data frame corresponding to each data identifier from the data frame, and parse each target data frame to determine the vehicle status data of each type. The first status data includes multiple types of vehicle status data, and the first status data includes at least one of the following: vehicle speed data, remaining battery power data, and driving range data.

4. The method for determining intent according to claim 1, characterized in that, Determining the second state data corresponding to the target object based on the facial key points includes: The gaze direction of the target object is calculated based on the first coordinate of the first key point corresponding to the eye of the target object, the second coordinate of the second key point corresponding to the eye of the target object, and the third coordinate of the center of the pupil of the target object. The fatigue level of the target object is calculated based on the coordinates of multiple key points corresponding to the eyes of the target object within a preset time window. The multiple key points corresponding to the eyes of the target object include at least the first key point and the second key point. The facial key points include the multiple key points corresponding to the eyes of the target object and the center of the pupil. The second state data includes at least one of the following: the direction of gaze and the fatigue level.

5. The method for determining intent according to claim 4, characterized in that, The gaze direction of the target object is calculated based on the first coordinate of the first key point corresponding to the target object's eye, the second coordinate of the second key point corresponding to the target object's eye, and the third coordinate of the center of the target object's pupil, including: The geometric center of the target object's eye is calculated based on the first coordinate and the second coordinate, and the gaze vector of the target object is calculated based on the geometric center and the third coordinate. The head yaw angle of the target object is calculated based on the horizontal difference between the fourth coordinate corresponding to the target part of the target object and the coordinates of the center of the eyes of the target object and the focal length of the image acquisition device in the vehicle. Calculate the horizontal angle of the line of sight corresponding to the target object based on the head yaw angle and the line of sight vector; The direction of the line of sight is calculated based on the horizontal angle of the line of sight and the boundary of the preset area.

6. The method for determining intent according to claim 4, characterized in that, The fatigue level of the target object within a preset time window is calculated based on the coordinates of multiple key points corresponding to the target object's eyes, including: Calculate the aspect ratio of the eye of the target object based on the coordinates of multiple key points corresponding to the target object; The duration of each blink of the target object is calculated based on the aspect ratio of the eye, and the total duration of blinks of the target object within a preset time window is calculated based on the duration. The fatigue level is calculated based on the total duration and the blinking frequency of the target object within the preset time window.

7. The method for determining intent according to claim 1, characterized in that, Determining the target intent corresponding to the voice command from multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent, including: The comprehensive confidence level corresponding to each candidate intent is calculated according to the target formula, wherein the target formula is: , Let i be the overall confidence level corresponding to the i-th candidate intent. Let be the first confidence level corresponding to the i-th candidate intent. Let be the first weight of the first confidence level corresponding to the i-th candidate intent. The second confidence level corresponds to the i-th candidate intent. The second weight is the second confidence level corresponding to the i-th candidate intent. The third confidence level corresponds to the i-th candidate intent. The third weight is the third confidence level corresponding to the i-th candidate intent; The maximum overall confidence level is determined among multiple overall confidence levels, and the candidate intent corresponding to the maximum overall confidence level is determined as the target intent.

8. The method for determining intent according to claim 1, characterized in that, After determining the target intent corresponding to the voice command from multiple candidate intents based on the first confidence level, second confidence level, and third confidence level corresponding to each candidate intent, the method further includes: In the target rule base, identify at least one service that has a mapping relationship with the target intent and the service parameters corresponding to each service; A target script is constructed based on the at least one service and the service parameters corresponding to each service, and a visual service combination is generated based on the target script; Upon receiving a confirmation instruction from the target object based on the visualization service combination, the target script is sent to the scheduler corresponding to the at least one service in the vehicle, so that the scheduler corresponding to the at least one service invokes the at least one service.

9. The method for determining intent according to claim 8, characterized in that, Identify at least one service in the target rule base that has a mapping relationship with the target intent, and the service parameters corresponding to each service, including: When the network status corresponding to the vehicle is online, at least one service that has a mapping relationship with the target intent and the service parameters corresponding to each service are determined in the first rule base. When the network status is offline, at least one service that has a mapping relationship with the target intent and the service parameters corresponding to each service are determined in the second rule base, wherein the target rule base includes: the first rule base and the second rule base.

10. An apparatus for determining intent, characterized in that, include: The acquisition module is used to acquire the voice command sent by the target object and extract features from the voice command to determine the text features corresponding to the voice command. In addition, monitoring data frames on the target bus of the vehicle and determining first state data corresponding to the vehicle based on the data frames; and collecting facial key points of the target object and determining second state data corresponding to the target object based on the facial key points. The calculation module is used to calculate the first confidence level of the text features for each candidate intent, the second confidence level of the first state data for each candidate intent, and the third confidence level of the second state data for each candidate intent; The determination module is used to determine the target intent corresponding to the voice command among multiple candidate intents based on the first confidence level, the second confidence level, and the third confidence level corresponding to each candidate intent.

Images