Interaction control method, device, vehicle, storage medium and product

Abstract

This application discloses an interactive control method, device, vehicle, storage medium, and product. Relating to the field of human-computer interaction technology, the method includes: acquiring a sequence of facial images of the driver in response to a head gesture interaction command; extracting spatiotemporal features from the facial image sequence and encoding preset text prompts to obtain first text features, wherein the preset text prompts include candidate head action categories; determining a target head action category from the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features; and executing vehicle control operations according to the user intent corresponding to the target head action category. This application can improve the accuracy of interactive control in noisy environments in smart cockpits, thereby enhancing driving safety and user experience.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to an interactive control method, device, vehicle, storage medium and product. Background Technology

[0002] With the rapid development of the intelligent vehicle industry, the human-machine interaction experience of intelligent cockpits has become a core element in improving the level of vehicle intelligence and user experience.

[0003] Currently, voice interaction, with its ability to keep both hands on the steering wheel and eyes on the road, perfectly meets the dual demands of safety and convenience in automotive scenarios, and has become the mainstream in-vehicle interaction technology. Specifically, users primarily control various vehicle functions by issuing voice commands, such as adjusting the air conditioning temperature, controlling window operation, and starting navigation. However, this control method suffers from lower recognition accuracy in noisy environments (such as high-speed driving, open windows, or multiple conversations), potentially leading to false triggering or delayed response, thus impacting driving safety and user experience.

[0004] Therefore, improving the accuracy of interactive control in noisy environments in smart cockpits to enhance driving safety and user experience is a problem that urgently needs to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide an interactive control method, device, vehicle, storage medium, and product, which aims to improve the accuracy of interactive control in noisy environments in smart cockpits, thereby enhancing driving safety and user experience.

[0006] To achieve the above objectives, this application provides an interactive control method, the interactive control method comprising: In response to head gesture interaction commands, it acquires a sequence of facial images of the driver; Spatiotemporal features are obtained by extracting features from the facial image sequence, and the first text features are obtained by encoding preset text prompts, wherein the preset text prompts include each candidate head action category; Based on the spatiotemporal features and the first text features, the visual language model determines the target head action category from the candidate head action categories, and performs vehicle control operations according to the user intent corresponding to the target head action category.

[0007] In one embodiment, the step of extracting spatiotemporal features from the facial image sequence includes: The facial image sequence is subjected to feature extraction using a visual encoder based on a visual language model to obtain spatiotemporal features; The step of encoding preset text prompt words to obtain the first text feature includes: The preset text prompt words are encoded by the text encoder of the visual language model to obtain the first text feature.

[0008] In one embodiment, the first text feature includes sub-feature vectors corresponding to each of the candidate head action categories, and the step of determining the target head action category from the candidate head action categories using a visual language model based on the spatiotemporal features and the first text feature includes: The similarity between the spatiotemporal features and each of the sub-feature vectors is calculated using a visual language model based on the spatiotemporal features and the first text features. The candidate head action category corresponding to the maximum value among the similarities is determined as the target head action category, and the maximum value is output as the confidence score of the target head action category.

[0009] In one embodiment, before the step of performing vehicle control operations based on the user intent corresponding to the target head movement category, the method further includes: Obtain contextual interaction information in the current driving scenario; The user intent is determined based on the contextual interaction information and the target head action category.

[0010] In one embodiment, the step of determining the user intent based on the contextual interaction information and the target head action category includes: The contextual interaction information is encoded by the text encoder of the visual language model to obtain the second text feature; The spatiotemporal features, the first text features, and the second text features are fused using the cross-attention mechanism of the visual language model to obtain multimodal fusion features; Based on the multimodal fusion features, each candidate intent in the current interaction scenario is determined, and the user intent is determined from each candidate intent based on the target head action category.

[0011] In one embodiment, prior to the step of acquiring a sequence of facial images of the driver in response to a head gesture interaction command, the method further includes: Real-time monitoring of audio data inside the vehicle, and determination of real-time signal-to-noise ratio based on the audio data; When the real-time signal-to-noise ratio is less than a first preset threshold, the head posture interaction command is triggered.

[0012] In one embodiment, after the step of determining the target head action category from each of the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features, the method further includes: When the real-time signal-to-noise ratio is less than a second preset threshold, the amplitude and duration of the head movement performed by the driver are determined, wherein the second preset threshold is less than the first preset threshold. If the amplitude of the action is greater than a preset amplitude threshold and the duration of the action is greater than or equal to a preset duration threshold, then the step of performing vehicle control operation according to the user intent corresponding to the target head action category is executed. If the amplitude of the movement is less than or equal to a preset amplitude threshold or the duration of the movement is less than a preset duration threshold, a prompt message is output, and the process returns to the step of collecting the driver's facial image sequence. The prompt message is used to instruct the driver to increase the amplitude of the movement or repeat the head movement.

[0013] In one embodiment, after the step of determining the target head action category from each of the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features, the method further includes: If the real-time signal-to-noise ratio is less than the second preset threshold, obtain the confidence level of the target head action category; If the confidence level is less than a preset confidence threshold, then return to the step of collecting the driver's facial image sequence.

[0014] Furthermore, to achieve the above objectives, this application also provides an interactive control device, the interactive control device comprising: The acquisition module is used to acquire a sequence of facial images of the driver in response to head posture interaction commands; An encoding module is used to extract spatiotemporal features from the facial image sequence and encode preset text prompts to obtain first text features, wherein the preset text prompts include each candidate head action category; The recognition module is used to determine the target head action category from each of the candidate head action categories based on the spatiotemporal features and the first text features using a visual language model, and to perform vehicle control operations according to the user intent corresponding to the target head action category.

[0015] In addition, to achieve the above objectives, this application also proposes an electronic device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the interactive control method as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a vehicle that includes the electronic equipment described above.

[0017] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a program implementing the interactive control method is stored, and the program implementing the interactive control method is executed by a processor to implement the steps of the interactive control method as described above.

[0018] In addition, to achieve the above objectives, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the interactive control method described above.

[0019] This application provides an interactive control method. First, in response to a head gesture interaction command, the method acquires a sequence of facial images of the driver. Then, it extracts spatiotemporal features from the facial image sequence and encodes preset text prompts to obtain first text features, wherein the preset text prompts include candidate head action categories. Finally, based on the spatiotemporal features and the first text features, a visual language model determines a target head action category from the candidate head action categories and executes vehicle control operations according to the user intent corresponding to the target head action category.

[0020] In summary, this application does not rely on voice commands. Instead, it extracts the spatiotemporal features of facial image sequences and combines them with the first text features obtained by encoding preset text prompts (including candidate head action categories). A visual language model is then used to determine the target head action category and execute the corresponding vehicle control operation. This is unaffected by noisy environments such as high-speed driving, open windows, or multiple conversations inside the vehicle. It avoids accidental triggering of vehicle functions or response delays caused by voice recognition errors, thereby improving the accuracy of in-vehicle interactive control, ensuring driving safety, and enhancing the user experience. Furthermore, this application only collects the driver's facial image sequence for head posture recognition, eliminating the need for continuous monitoring of in-vehicle voice signals or uploading voice data to the cloud for processing. This reduces the risk of privacy leaks associated with voice interaction, further optimizing the security and convenience of user-vehicle interaction. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the first embodiment of the interactive control method of this application; Figure 2 This is a schematic diagram of the system architecture involved in an embodiment of the interactive control method of this application; Figure 3 This is a schematic diagram of the interactive control flow involved in an embodiment of the interactive control method of this application; Figure 4 This is a schematic diagram of the module structure of the interactive control device of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the interactive control method in the embodiments of this application.

[0024] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] Currently, voice interaction, with its ability to keep both hands on the steering wheel and eyes on the road, perfectly meets the dual demands of safety and convenience in automotive scenarios, and has become the mainstream in-vehicle interaction technology. Specifically, users primarily control various vehicle functions by issuing voice commands, such as adjusting the air conditioning temperature, controlling window operation, and starting navigation. However, this control method suffers from lower recognition accuracy in noisy environments (such as high-speed driving, open windows, or multiple conversations), potentially leading to false triggering or delayed response, thus impacting driving safety and user experience.

[0028] Therefore, improving the accuracy of interactive control in noisy in-vehicle environments to enhance driving safety and user experience is a problem that urgently needs to be solved.

[0029] The main solution of this application is as follows: in response to a head gesture interaction command, a sequence of facial images of the driver is acquired; spatiotemporal features are extracted from the facial image sequence, and a first text feature is obtained by encoding a preset text prompt word, wherein the preset text prompt word includes each candidate head action category; based on the spatiotemporal features and the first text feature, a target head action category is determined from each candidate head action category using a visual language model, and a vehicle control operation is performed according to the user intent corresponding to the target head action category.

[0030] This application does not rely on voice commands. Instead, it extracts the spatiotemporal features of facial image sequences and combines them with the first text features obtained by encoding preset text prompts (including candidate head movement categories). A visual language model is then used to determine the target head movement category and execute the corresponding vehicle control operation. This is unaffected by noisy environments such as high-speed driving, open windows, or multiple conversations inside the vehicle. It avoids accidental triggering of vehicle functions or response delays caused by voice recognition errors, thereby improving the accuracy of in-vehicle interactive control, ensuring driving safety, and enhancing the user experience. Furthermore, this application only collects the driver's facial image sequence for head posture recognition, eliminating the need for continuous monitoring of in-vehicle voice signals or uploading voice data to the cloud for processing. This reduces the risk of privacy leaks associated with voice interaction, further optimizing the security and convenience of user-vehicle interaction.

[0031] It should be noted that the executing entity of the interactive control method in various embodiments of this application can be a vehicle control system, or a computing service device with data processing, network communication, and program execution functions, etc. This embodiment does not specifically limit it. The following uses a vehicle control system as the executing entity to describe this embodiment and the following embodiments.

[0032] Based on this, this application proposes an interactive control method according to a first embodiment, please refer to... Figure 1 The interactive control method includes steps S10 to S30: Step S10: In response to the head gesture interaction command, acquire the driver's facial image sequence.

[0033] It should be noted that head posture interaction commands are control signals used to initiate head posture-based interaction. These commands can be automatically triggered by the system (e.g., when excessive ambient noise is detected) or manually activated by the user (e.g., by pressing an interaction button on the steering wheel). Typically, these head posture interaction commands are activated when the vehicle interior environment is noisy or voice interaction is unreliable. Multiple frames of driver facial images (i.e., the aforementioned facial image sequence) are continuously captured by RGB cameras positioned behind the steering wheel or on the A-pillar. Each frame contains information about the driver's facial area and head posture. Furthermore, the frame rate of the facial image sequence can be adaptively adjusted according to the environment; for example, it can be increased to 60fps in noisy environments and decreased to 15fps in quiet environments.

[0034] Upon receiving a head gesture interaction command, the system immediately activates the vehicle's RGB camera to continuously capture images of the driver's face at a preset frame rate (e.g., 30fps), forming an image sequence that includes a temporal dimension. This image sequence records the head's positional changes and motion trajectory across multiple consecutive frames, providing raw data for subsequent extraction of spatiotemporal features. The process of capturing the driver's facial images continues until the interaction ends or the command is canceled.

[0035] Step S20: Extract spatiotemporal features from the facial image sequence and encode preset text prompts to obtain first text features, wherein the preset text prompts include each candidate head action category.

[0036] It should be noted that spatiotemporal features refer to joint feature vectors extracted from multiple consecutive frames of facial images, containing both spatial information (head pose angle and key point positions in a single frame) and temporal information (motion speed, direction, and duration across multiple frames). Spatiotemporal features are also called spatiotemporal feature vectors. Predefined text prompts refer to predefined structured statements containing text descriptions of multiple candidate head movement categories, which can include nodding, shaking, turning the head, and actions without clear meaning. The first text feature refers to the semantic feature vector obtained after inputting the predefined text prompts into the text encoder of the visual language model. Compared to the spatial features of a single frame image, spatiotemporal features can capture the dynamic process of head movement, thus distinguishing between momentary tremors and continuous conscious actions.

[0037] In this embodiment, step S20 may include: Step S201: The facial image sequence is subjected to feature extraction by the visual encoder of the visual language model to obtain spatiotemporal features.

[0038] It's important to note that a Visual Language Model (VLM) refers to a multimodal deep learning model capable of processing both visual and textual data simultaneously. Examples include CLIP (Contrastive Language–Image Pre-training) and BLIP (Bootstrapping Language–Image Pre-training)-2. The visual encoder is the visual processing branch of a VLM, typically employing a Vision Transformer (ViT) or convolutional neural network architecture to convert the input image into a high-dimensional semantic feature vector.

[0039] A sequence of consecutive facial images is fed into the visual encoder of the VLM (Visual Model). The encoder extracts spatial features from each frame through a multi-layer self-attention mechanism and models the motion relationships between frames through a temporal attention module (or positional encoding), ultimately outputting a fixed-dimensional spatiotemporal feature vector. This feature vector encapsulates the dynamic information of the driver's head movements, providing visual input for subsequent cross-modal similarity calculations with text features.

[0040] In one feasible implementation, taking CLIP-ViT as the visual encoder as an example, five consecutive frames of driver facial images (each frame with a resolution of 224×224) are input into the encoder. After processing through block segmentation, linear projection, and a multi-head self-attention layer, a 512-dimensional spatiotemporal feature vector is output. , The calculation formula can be expressed as: .in, This is a sequence of facial images of the driver. The facial image sequence includes multiple frames of facial images; VisionEncoder refers to the visual encoder function, which represents the neural network module in the visual language model (VLM) used to extract image features, such as ViT or CNN (Convolutional Neural Network). Let be the set of real numbers, indicating that each component of the eigenvector is a real number; The dimension of the spatiotemporal feature vector is determined by the network structure of the visual encoder. For example, the output dimension of CLIP-ViT is usually 512 or 768.

[0041] In another feasible implementation, a visual encoder using BLIP-2 is used, which adds a temporal attention layer to ViT, enabling more effective extraction of speed and direction information of head movements.

[0042] Step S202: The preset text prompt words are encoded by the text encoder of the visual language model to obtain the first text feature.

[0043] It should be noted that the text encoder is the text processing branch of VLM, typically employing a Transformer architecture to convert natural language text into high-dimensional semantic feature vectors. Preset text prompts aim to guide the model in aligning visual features with corresponding text semantics. For example, a preset text prompt might be: "Scenario: Cockpit, Environment [Noisy / Quiet], Driver's Head Movement: [Description], Movement Options: A. Nod (Confirmation / Agreement) B. Shake Head (Negation / Refusal) C. Turn Head (Focus on the Side) D. No clear meaning, Question: What is the driver's current intention?". This prompt, by explicitly listing candidate head pose categories and their semantic meanings, and providing a specific scenario and question, enables VLM to perform cross-modal matching between visually observed head movements and the options in the text.

[0044] The structured text prompts mentioned above are pre-constructed, which include the current environmental state (noisy / quiet), placeholders describing the driver's head movements, explicit action options, and questions inquiring about intent. These text prompts are then input into the VLM text encoder, where each word is converted into a vector through a word embedding layer. The semantic relationships between words are then captured through a multi-layer Transformer self-attention mechanism, ultimately outputting a fixed-dimensional text feature vector (i.e., the first text feature). In this first text feature, the sub-feature vectors corresponding to different candidate head pose categories are discriminative in the feature space, enabling accurate matching of corresponding visual features during subsequent similarity calculations.

[0045] In one feasible implementation, the preset text prompt is: "Scenario: Inside the cockpit, noisy environment, driver's head action: [To be identified]. Action options: A. Nod (confirm / agree) B. Shake head (deny / reject) C. Turn head (focus on the side) D. No clear intention. Question: What is the driver's current intention?". This text is input into CLIP's text encoder, and after word segmentation, embedding, and 12-layer Transformer encoding, a 512-dimensional text feature vector is output. , The calculation formula can be expressed as: .in, Preset text prompts; TextEncoder represents the text encoder function; Represent a A dimensional real vector space; This refers to the dimension of the text feature vector.

[0046] Step S30: Based on the spatiotemporal features and the first text features, the visual language model determines the target head action category from the candidate head action categories, and performs vehicle control operations according to the user intent corresponding to the target head action category.

[0047] The visual language model performs cross-modal alignment between spatiotemporal features and first text features to achieve end-to-end understanding from "pixel" to "intent". It selects the category that best matches the current driver's head action from among the candidate head action categories (hereinafter referred to as the target head action category for distinction). Then, it determines the corresponding user intent based on the target head action category and executes the vehicle control operation corresponding to the user intent.

[0048] For example, when the target head action category is nodding, the corresponding user intent could be playing music, answering a phone call, or confirming navigation, etc. The vehicle control operation is the specific function executed according to the user intent, which could be playing music, answering a phone call, or starting navigation, etc.

[0049] In this embodiment, the first text feature includes sub-feature vectors corresponding to each of the candidate head action categories, and step S30 may include: Step S301: Calculate the similarity between the spatiotemporal features and each of the sub-feature vectors based on the spatiotemporal features and the first text features using a visual language model; Step S302: Determine the candidate head action category corresponding to the maximum value among the similarities as the target head action category, and output the maximum value as the confidence score of the target head action category.

[0050] It should be noted that the similarity between spatiotemporal features and sub-feature vectors can be expressed as: The specific calculation formula is as follows:

[0051] in, For sub-feature vectors, specifically they can be , , wait.

[0052] The system sequentially calculates the cosine similarity between the spatiotemporal features and each sub-feature vector, resulting in a set of similarity scores. Higher similarity scores indicate a higher semantic match between the current driver's head movement and the candidate category. The system selects the candidate category corresponding to the maximum similarity score as the target head movement category and outputs this maximum similarity value as the confidence score of the recognition result for subsequent intent confirmation and interactive execution.

[0053] In summary, this embodiment acquires driver facial image sequences in response to head gesture interaction commands, extracts spatiotemporal features, and uses a visual language model to encode preset text prompts (as first text features). Then, based on cross-modal alignment of the spatiotemporal features and the first text features, it calculates similarity to determine the target head gesture category and its confidence level from candidate head gesture categories. Finally, it executes vehicle control operations according to the corresponding user intent. Therefore, this embodiment eliminates the need for voice input or touch operation; the driver can complete the interaction solely through natural head gestures, reducing the risk of driver distraction. Simultaneously, leveraging the pre-trained knowledge of the visual language model, the system can achieve end-to-end head gesture semantic understanding and supports zero-sample or few-sample expansion of new head gesture types without the need to re-collect large amounts of training data. This improves the naturalness, security, and scalability of the interaction while ensuring recognition accuracy.

[0054] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of "performing vehicle control operation according to the user intent corresponding to the target head movement category" in step S30, the interactive control method of this application further includes: Step A10: Obtain contextual interaction information in the current driving scenario; Step A20: Determine the user intent based on the contextual interaction information and the target head action category.

[0055] It should be noted that contextual interaction information refers to various dynamic information related to the interactions the driver may perform in the current driving scenario, including but not limited to proactive inquiries from the intelligent cockpit system (such as navigation prompts such as "Switch route?"), passive event reminders (such as incoming call reminders and low fuel warnings), media playback status (such as currently playing music), and vehicle status (such as vehicle speed and gear). This information reflects the interaction decision points the driver may face at the current moment. User intent refers to the specific operational intention expressed by the driver through head movements, such as "confirm switching navigation routes," "answer the phone," "reject the call," and "play the next song." Compared with the target head action category output in step S30 (such as "nodding"), user intent combines contextual semantics and is a precise interpretation of the meaning of the action.

[0056] First, the system obtains current contextual interaction information from modules such as the cockpit domain controller, navigation application, telephone application, and media player. Examples include "the navigation system is asking whether to avoid traffic jams," "there is an incoming call," and "music is playing." Then, the obtained contextual information, along with the target head action category (such as "nodding"), is used as input. Through preset rule mapping or further reasoning by the visual language model, the system determines the specific user intent corresponding to the action in the current context (e.g., nodding + navigation inquiry = "confirm route change"; nodding + incoming call = "answer the call"; nodding + music playback = "next song").

[0057] In this embodiment, step A20 may include: Step A201: Encode the contextual interaction information using the text encoder of the visual language model to obtain the second text feature; Step A202: The spatiotemporal features, the first text features, and the second text features are fused through the cross-attention mechanism of the visual language model to obtain multimodal fusion features; Step A203: Determine each candidate intent in the current interaction scenario based on the multimodal fusion features, and determine the user intent from each candidate intent based on the target head action category.

[0058] The contextual interaction information is encoded by the text encoder of the visual language model to obtain a text feature vector (hereinafter referred to as the second text feature for distinction). Then, the spatiotemporal features, the first text feature, and the second text feature are deeply fused through the cross-attention mechanism (also known as the cross-modal feature fusion mechanism) of the visual language model to obtain a fused feature vector (hereinafter referred to as the multimodal fusion feature for distinction). Based on the multimodal fusion feature, each candidate intent in the current interaction scenario is determined. The candidate intent refers to the set of user choices allowed by the system for the current user inquiry or system prompt, decoded by the system according to the multimodal fusion feature in the current interaction scenario. The user intent is determined from each candidate intent based on the target head action category.

[0059] For example, when the system asks "Do you want to play music?", the candidate intents are {"Confirm", "Cancel"}. If the target head action category is nodding, then "Confirm" is determined as the user intent from the candidate intents {"Confirm", "Cancel"}.

[0060] In one feasible implementation, after obtaining the target head action category, the system acquires the current scene state of the intelligent cockpit. The current scene state can be a navigation inquiry (e.g., "Avoid congestion?"), an incoming call reminder (Answer the call?), or music playback control, etc. The current scene state is input as contextual interaction information into the VLM's text encoder to obtain a second text feature, which is represented as... ,For example, The cross-attention mechanism of the visual language model uses spatiotemporal features as queries and the concatenation of the first and second text features as keys and values. Attention weights are calculated and weighted summation is performed to generate multimodal fusion features. These fusion features simultaneously encode the driver's head movement patterns, semantic representations of head action categories, and task constraints of the current interaction scenario. Specifically, the multimodal fusion features are represented as follows: Its calculation formula can be expressed as: ;Will The input is fed into a lightweight classifier associated with the current scene (e.g., a fully connected layer followed by Softmax), which outputs a set of candidate intents and their probabilities for the current interaction scene. Since the current scene is "music playback confirmation," two candidate intents are predefined: confirm (probability 0.85) and cancel (probability 0.15). According to a predefined head action-semantic mapping rule, nodding corresponds to the semantics of "confirm / yes." The system identifies the "confirm" option in the candidate intent set that matches the semantics of "confirm" as the final user intent. Subsequently, the system executes vehicle control operations based on this user intent: playing music and displaying "playing" visual feedback via the HUD.

[0061] Table 1 is a preset head action-semantic mapping rule table: Table 1

[0062] The head gestures include single nods, continuous nods, single head shakes, continuous head shakes, turning the head to the left, turning the head to the right, and head tilting + nods, serving as candidate head gesture categories. Basic semantics refers to the fundamental semantic meaning of each head gesture in a general scenario, such as "confirm / yes," "deny / no," "select left," "select right," and "wake up + confirm." Contextual extended semantics refers to the specific functions or instructions that can be extended from the same head gesture in a specific driving scenario. Applicable scenarios refer to the types of interactive environments to which the corresponding head gesture is applicable. When a new head gesture interaction type needs to be added (e.g., "tilting the head to think" to indicate "need suggestion"), there is no need to re-collect a large amount of labeled training data. Only the semantic label of the new action needs to be defined through text description (e.g., the prompt word corresponding to "tilting the head to think"). The visual language model can then utilize its cross-modal alignment capability to match the visual features of the new action with the semantic description, quickly completing the recognition and expansion of the new interaction type. This feature significantly reduces the data cost and training time for system iteration and upgrades, exhibiting good scalability and adaptability.

[0063] In this embodiment, as Figure 2 The diagram shows the system architecture. The vehicle control system of this application includes an in-vehicle camera, an environmental perception module, a visual language model (VLM), a head posture analysis engine, an interaction decision module, and an intelligent cockpit execution layer. The in-vehicle camera (using an RGB / infrared sensor and a wide-angle lens) is used to acquire images of the driver's face. The visual language model includes a visual encoder (such as ViT / CLIP) and a text encoder (such as BERT / GPT) to extract visual and textual features, respectively. The visual language model also includes a multimodal fusion and head posture understanding module to fuse visual, textual, and contextual interaction information. The head posture analysis engine performs head nodding detection, head shaking detection, and posture angle calculation. The environmental perception module is responsible for light and noise detection to identify noisy / quiet environments. The interaction decision module completes intent recognition, command mapping, and feedback generation. Finally, the intelligent cockpit execution layer executes control operations for air conditioning, music, navigation, windows, seats, lights, communication, and driver assistance.

[0064] Furthermore, to ensure driving safety, this application embodiment introduces a safety verification mechanism during head gesture interaction. On one hand, the system acquires the vehicle's speed in real time. When the speed exceeds a preset speed threshold (e.g., 60 km / h), complex interactive functions (such as multi-step menu navigation, in-depth setting adjustments, etc.) are automatically disabled, retaining only safety-related emergency confirmation or denial operations (such as "confirm switching alternative routes" or "cancel hazard warnings"), preventing the driver from being distracted by complex operations. On the other hand, the system uses a Driver Monitoring System (DMS) to detect the driver's fatigue state in real time. When the system determines that the driver is fatigued, it automatically raises the interaction confirmation threshold (e.g., requiring larger head movements, longer durations, or adding repetitive action confirmations), or directly disables unnecessary entertainment-related interactive functions. Through the above dual safety verification, this application embodiment can reduce the risk of driver distraction while ensuring interactive convenience, thereby improving driving safety.

[0065] Thus, this embodiment of the application acquires contextual interaction information in the current driving scenario and encodes it into second text features using a text encoder of a visual language model. Then, a cross-attention mechanism is used to deeply fuse spatiotemporal features, first text features, and second text features. The resulting multimodal fusion feature simultaneously includes head motion dynamics, head action semantics, and current scenario task constraints. Based on this, combined with the identified target head action categories, the user's intent is accurately matched from the candidate intent set defined by the current scenario. This eliminates semantic ambiguity of the same head action in different scenarios, achieving scenario-adaptive intent understanding, thereby improving the accuracy and robustness of head posture interaction and making the interaction process more in line with the driver's natural behavioral habits.

[0066] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description and will not be repeated hereafter. Furthermore, before step S10, the interactive control method of this application further includes: Step B10: Monitor the audio data inside the vehicle in real time, and determine the real-time signal-to-noise ratio based on the audio data; Step B20: When the real-time signal-to-noise ratio is less than a first preset threshold, the head posture interaction command is triggered.

[0067] It should be noted that audio data refers to in-vehicle ambient sound collected through the cockpit array microphones, including driver voice, passenger conversations, wind noise, road noise, etc. Real-time signal-to-noise ratio (SNR) is the power ratio of the effective speech signal to the background noise, used to quantify the noise level of the current environment. The lower the SNR, the noisier the environment. A pre-set threshold for judging noisy environments (e.g., 10dB), i.e., the first preset threshold, is used. When the SNR falls below this first preset threshold, the system determines that the current environment is not suitable for pure voice interaction.

[0068] The system continuously monitors in-vehicle audio data, calculating the real-time signal-to-noise ratio (SNR) every frame or time interval. When the SNR falls below a first preset threshold, the system determines the current environment is noisy, rendering voice interaction unreliable. It then automatically triggers head gesture interaction commands, entering a "head gesture priority" mode. In this mode, the system primarily relies on head movements for intent input, increasing the visual sampling frequency to 60fps. Simultaneously, it can selectively process voice input—specifically, it can use a voice-assisted mode (still collecting voice commands but only as an auxiliary reference, used for disambiguation only when head gesture confidence is low, with final decisions based on head gestures) or a voice-off mode (completely disabling the voice recognition channel, using only head gestures for control to avoid noise-induced false triggers). Through this adaptive triggering mechanism, the system prioritizes gesture-independent head gesture interaction in noisy environments, ensuring accurate acquisition of control commands.

[0069] It should be noted that a threshold for determining a quiet environment (e.g., 20dB) is pre-set, referred to as the third preset threshold for differentiation. When the real-time signal-to-noise ratio (SNR) is greater than or equal to the third preset threshold, the current environment is determined to be quiet, and the system enters a "voice interaction priority" mode. In this mode, the system primarily relies on voice for intent input, while selectively processing head movement input and reducing the visual sampling frequency to 15fps. Specifically, a visual interaction mode is used as an auxiliary mode, or the visual interaction mode is turned off. Furthermore, when the real-time SNR is in an intermediate state—that is, greater than or equal to the first preset threshold and less than the third preset threshold—it is determined to be a normal environment. Head movement control and voice control are processed in parallel, and VLM (Virtual Mode Model) is used for multimodal fusion decision-making.

[0070] For example, in noisy environments (such as high-speed driving, open windows, or multiple people talking), visual feedback (such as HUD head-up display, dashboard icons) or tactile feedback (such as steering wheel vibration) should be prioritized to prevent audio feedback from being masked by ambient noise and thus undetectable to the user. In quiet environments (such as nighttime parking, quiet interior), audio feedback (such as short prompts or voice confirmations) can be used, while visual feedback remains as an auxiliary tool to provide redundant confirmation paths. Through this environment-adaptive feedback strategy, the system can ensure that users receive timely and accurate feedback on interaction results regardless of noise conditions, improving the user-friendliness and reliability of human-computer interaction.

[0071] In the first feasible implementation, in a noisy environment where the vehicle is traveling at high speed, the windows are open, and the external light changes drastically (such as entering or exiting a tunnel), the system first detects that the environment is "noisy," specifically based on a light change rate greater than 30% and background motion detection exceeding a threshold. Subsequently, the system initiates image denoising and dynamic range compression preprocessing to improve the quality of the driver's facial image. Next, the visual language model receives a preset prompt: "In a vehicle environment with changing strong light, identify whether the driver nods (switch to the next song) or shakes their head (stay current)." When the driver nods slightly, the visual language model outputs the recognition result: "Detected a rapid movement to the lower right followed by a return to center, confidence level 0.92, semantics: confirmed switch." Finally, the system plays the next song based on the recognition result and provides voice feedback "Switched to the next song," completing the music control interaction in a noisy environment.

[0072] In the second feasible implementation, in a quiet environment with dim lighting inside the car at night, a mobile phone call triggers interaction: The system first detects that the environment is "quiet," specifically based on light intensity below 50 lux and low signal-to-noise ratio; the system then initiates low-light enhancement and contrast stretching preprocessing to improve image visibility under dim conditions; the visual language model receives the prompt: "In a dimly lit car environment, identify the driver's head movements to handle the call"; the driver makes a compound action of turning their head to the right (looking at the passenger seat) and shaking their head, and the visual language model outputs: "Detected horizontal rightward rotation + swaying action, confidence level 0.88, semantics: refuse to answer"; the system hangs up the phone and remains silent to avoid disturbing the passenger resting in the passenger seat, thus realizing call handling in a quiet environment.

[0073] In the third feasible implementation, during navigation, when the system prompts "Congestion ahead 500 meters, do you want to switch to an alternative route?", the driver first looks up to check the HUD navigation information, then nods twice in succession. At the same time, the temporal modeling module of the visual language model captures the action sequence of "looking up → nodding repeatedly". The system combines the current navigation context (congestion prompt and alternative route inquiry) to infer that the driver's intention is "strongly confirm the switch after checking the information". Finally, the system executes the route switch and announces "Switched to alternative route, expected to save 8 minutes" through voice broadcast, thus completing the accurate identification and response to the compound intention.

[0074] In this embodiment, after the step of "determining the target head action category from each of the candidate head action categories based on the spatiotemporal features and the first text features using a visual language model" in step S30, the interactive control method of this application further includes: Step C10: When the real-time signal-to-noise ratio is less than the second preset threshold, determine the amplitude and duration of the head movement when the driver performs the target head movement category, wherein the second preset threshold is less than the first preset threshold; Step C20: If the amplitude of the action is greater than a preset amplitude threshold and the duration of the action is greater than or equal to a preset duration threshold, then execute the step of performing vehicle control operation according to the user intent corresponding to the target head action category. Step C30: If the amplitude of the movement is less than or equal to a preset amplitude threshold or the duration of the movement is less than a preset duration threshold, then output a prompt message and return to the step of collecting the driver's facial image sequence. The prompt message is used to instruct the driver to increase the amplitude of the movement or repeat the head movement.

[0075] It should be noted that a threshold value lower than the first preset threshold (hereinafter referred to as the second preset threshold for distinction) is set to determine extremely noisy environments. The amplitude of movement refers to the angle or displacement of the driver's head movement, such as the pitch angle of a nod or the yaw angle of a shake. The duration of the movement refers to the length of time the head movement lasts from start to finish, such as the duration of a single nod. The preset amplitude threshold is a pre-set minimum amplitude threshold (e.g., nod angle > 15°) used to distinguish between intentional movements and involuntary shaking. The preset duration threshold is a pre-set minimum duration threshold (e.g., nod duration ≥ 200ms) used to filter out momentary or invalid movements. The prompt message is a feedback signal used to guide the driver to repeat the head movement; the prompt may include "Please increase the nod amplitude" or "Please repeat the nod twice."

[0076] After the system identifies the target head movement category through a visual language model, it further determines whether the current environment is in an extremely noisy state (i.e., whether the real-time signal-to-noise ratio is less than the second preset threshold). If this condition is met, the system enters the "enhanced confirmation mode". In this mode, the system performs quantitative analysis on the identified head movements, calculating the amplitude and duration of the movement. Only when the amplitude of the movement is greater than the preset amplitude threshold and the duration of the movement is greater than or equal to the preset duration threshold, does the system recognize the movement as a valid and intentional interaction command, and then executes vehicle control operations according to the user intent corresponding to the target head movement category. Conversely, if the amplitude or duration of the movement is insufficient (either condition is not met), the system determines that the movement is invalid or the confidence is too low, and will not execute control operations. Instead, it outputs a prompt message (e.g., "Please increase the nod amplitude" displayed through the HUD (Head-Up Display)), requiring the driver to increase the amplitude of the movement or repeat the head movement (e.g., nod twice consecutively), and automatically returns to re-acquire the facial image sequence, repeating the above verification process. This closed-loop mechanism can effectively eliminate false triggers caused by vehicle bumps, fatigued driving, or habitual small movements, ensuring the reliability and safety of head gesture interaction in extremely noisy environments.

[0077] In this embodiment, after the step of "determining the target head action category from each of the candidate head action categories based on the spatiotemporal features and the first text features using a visual language model" in step S30, the interactive control method of this application further includes: Step D10: If the real-time signal-to-noise ratio is less than the second preset threshold, obtain the confidence level of the target head action category; Step D20: If the confidence level is less than a preset confidence threshold, then return to the step of collecting the driver's facial image sequence.

[0078] It should be noted that the preset confidence threshold refers to a pre-set confidence level (e.g., 0.8) used to determine whether the current recognition result is sufficiently reliable. When the confidence level is lower than this preset confidence threshold, the system considers the recognition result to be potentially ambiguous or erroneous.

[0079] This application embodiment further enhances the confidence verification mechanism for extremely noisy environments. Specifically, when the real-time signal-to-noise ratio is less than a second preset threshold (i.e., in an extremely noisy environment), the system acquires the confidence level corresponding to the target head action category output by the visual language model; this confidence level is compared with a preset confidence threshold: if the confidence level is greater than or equal to the preset confidence threshold, the system considers the current recognition result reliable and can continue to execute subsequent intent determination and control operations; if the confidence level is less than the preset confidence threshold, the system determines that the current recognition result lacks confidence and there is a risk of misidentification. In this case, no control operation is executed, but the system automatically returns to re-acquire the driver's facial image sequence, re-extracts features and re-recognizes head actions, forming a closed loop until a high-confidence recognition result is obtained. This mechanism can effectively avoid misidentification caused by image quality degradation, lighting changes, or blurred driver actions in extremely noisy environments, further improving the robustness and safety of head gesture interaction.

[0080] In one feasible implementation, such as Figure 3 The diagram illustrates the interactive control process. It monitors the real-time signal-to-noise ratio (SNR) inside the vehicle. When the SNR is less than a first preset threshold, a head gesture interaction command is triggered, initiating the acquisition of the driver's facial image sequence. The VLM's visual encoder extracts spatiotemporal features from the facial image sequence, and the VLM's text encoder encodes preset text prompts to obtain first text features. The VLM achieves end-to-end understanding from pixels to intent based on the spatiotemporal features and the first text features, obtaining the target head action category. Contextual interaction information is acquired, and the VLM's text encoder encodes this information to obtain second text features. Then, the VLM's cross-attention mechanism fuses the spatiotemporal features, first text features, and second text features to obtain multimodal fusion features. Based on the multimodal fusion features, candidate intents for the current interaction scenario are determined, and the user intent is identified from these candidate intents based on the target head action category. Finally, the corresponding vehicle control operation is executed based on the user intent.

[0081] Thus, this application embodiment, through multi-level signal-to-noise ratio thresholds and dual verification mechanisms, prioritizes the use of head gesture interaction in noisy environments, and jointly verifies the amplitude, duration, and model confidence of head movements in extremely noisy environments. This effectively filters out invalid actions and misidentifications caused by vehicle bumps, changes in lighting, or recognition ambiguity, significantly improving the reliability, robustness, and driving safety of head gesture interaction.

[0082] Furthermore, compared to traditional visual and voice interaction methods, the technical advantages of this application include: Regarding adaptability to noisy environments, traditional visual interaction methods are significantly affected by changes in lighting and exhibit moderate adaptability, while voice interaction methods perform poorly due to noise interference. This application, however, achieves excellent adaptability to noisy environments through multimodal fusion of a visual language model and an adaptive sampling strategy. Regarding adaptability to quiet environments, traditional visual interaction methods perform moderately, while voice interaction methods perform excellently. This application, through the fusion of visual and verbal channels, also achieves excellent adaptability. In terms of semantic understanding, traditional visual interaction methods lack semantic understanding capabilities, and voice interaction methods possess only limited semantic understanding. This application, however, leverages the multimodal fusion of a visual language model... End-to-end semantic understanding enables powerful semantic reasoning capabilities. Regarding hardware cost, traditional visual interaction methods require infrared or depth cameras, resulting in moderate costs, while voice interaction methods only require a microphone, leading to lower costs. This application utilizes only a single RGB camera, resulting in similarly low hardware costs. In terms of interaction naturalness, traditional visual interaction methods require users to learn specific actions, resulting in low naturalness, while voice interaction methods have moderate naturalness. This application aligns with drivers' natural communication habits, resulting in high interaction naturalness. Regarding privacy protection, traditional visual interaction methods use local image processing, resulting in moderate privacy protection, while voice interaction methods often require cloud uploads, resulting in low privacy protection. This application supports local visual language model inference and can optionally run offline, resulting in a high level of privacy protection. In summary, this application achieves significant improvements in multiple dimensions, including environmental adaptability, semantic understanding, hardware cost, interaction naturalness, and privacy protection.

[0083] This application also provides an interactive control device, please refer to... Figure 4 The interactive control device includes: The acquisition module 10 is used to acquire a sequence of facial images of the driver in response to head posture interaction commands; The encoding module 20 is used to extract spatiotemporal features from the facial image sequence and encode preset text prompts to obtain first text features, wherein the preset text prompts include each candidate head action category; The recognition module 30 is used to determine the target head action category from each of the candidate head action categories based on the spatiotemporal features and the first text features using a visual language model, and to perform vehicle control operations according to the user intent corresponding to the target head action category.

[0084] Optionally, the encoding module 20 is further configured to: The facial image sequence is subjected to feature extraction using a visual encoder based on a visual language model to obtain spatiotemporal features; The preset text prompt words are encoded by the text encoder of the visual language model to obtain the first text feature.

[0085] Optionally, the first text feature includes sub-feature vectors corresponding to each of the candidate head action categories, and the recognition module 30 is further configured to: The similarity between the spatiotemporal features and each of the sub-feature vectors is calculated using a visual language model based on the spatiotemporal features and the first text features. The candidate head action category corresponding to the maximum value among the similarities is determined as the target head action category, and the maximum value is output as the confidence score of the target head action category.

[0086] Optionally, the interactive control device further includes a disambiguation module, which is used for: Obtain contextual interaction information in the current driving scenario; The user intent is determined based on the contextual interaction information and the target head action category.

[0087] Optionally, the disambiguation module is further configured to: The contextual interaction information is encoded by the text encoder of the visual language model to obtain the second text feature; The spatiotemporal features, the first text features, and the second text features are fused using the cross-attention mechanism of the visual language model to obtain multimodal fusion features; Based on the multimodal fusion features, each candidate intent in the current interaction scenario is determined, and the user intent is determined from each candidate intent based on the target head action category.

[0088] Optionally, the acquisition module 10 is further configured to: Real-time monitoring of audio data inside the vehicle, and determination of real-time signal-to-noise ratio based on the audio data; When the real-time signal-to-noise ratio is less than a first preset threshold, the head posture interaction command is triggered.

[0089] Optionally, the interactive control device further includes a verification module, the verification module being used for: When the real-time signal-to-noise ratio is less than a second preset threshold, the amplitude and duration of the head movement performed by the driver are determined, wherein the second preset threshold is less than the first preset threshold. If the amplitude of the action is greater than a preset amplitude threshold and the duration of the action is greater than or equal to a preset duration threshold, then the step of performing vehicle control operation according to the user intent corresponding to the target head action category is executed. If the amplitude of the movement is less than or equal to a preset amplitude threshold or the duration of the movement is less than a preset duration threshold, a prompt message is output, and the process returns to the step of collecting the driver's facial image sequence. The prompt message is used to instruct the driver to increase the amplitude of the movement or repeat the head movement.

[0090] Optionally, the verification module is further configured to: If the real-time signal-to-noise ratio is less than the second preset threshold, obtain the confidence level of the target head action category; If the confidence level is less than a preset confidence threshold, then return to the step of collecting the driver's facial image sequence.

[0091] The interactive control device provided in this application, employing the interactive control method described in the above embodiments, can solve the technical problem of how to improve the accuracy of interactive control in noisy environments in smart cockpits, thereby enhancing driving safety and user experience. Compared with the prior art, the beneficial effects of the interactive control device provided in this application are the same as those of the interactive control method provided in the above embodiments, and other technical features in the interactive control device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0092] This application provides an interactive control device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the interactive control method in Embodiment 1 above.

[0093] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the interactive control device in the embodiments of this application. The interactive control device in the embodiments of this application may be an in-vehicle terminal (e.g., an in-vehicle navigation terminal). Figure 5 The interactive control device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0094] like Figure 5As shown, the interactive control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the interactive control device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the interactive control device to communicate wirelessly or wiredly with other devices to exchange data. Although interactive control devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0095] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0096] The interactive control device provided in this application, employing the interactive control method described in the above embodiments, can solve the technical problem of how to improve the accuracy of interactive control in noisy environments in smart cockpits, thereby enhancing driving safety and user experience. Compared with the prior art, the beneficial effects of the interactive control device provided in this application are the same as those of the interactive control method provided in the above embodiments, and other technical features of this interactive control device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0099] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the interactive control method in the above embodiments.

[0100] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0101] The aforementioned computer-readable storage medium may be included in the interactive control device; or it may exist independently and not assembled into the interactive control device.

[0102] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an interactive control device, cause the interactive control device to: in response to a head gesture interaction command, acquire a sequence of facial images of the driver; extract features from the facial image sequence to obtain spatiotemporal features, and encode preset text prompts to obtain first text features, wherein the preset text prompts include candidate head action categories; determine a target head action category from among the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features, and perform vehicle control operations according to the user intent corresponding to the target head action category.

[0103] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0106] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described interactive control method. This solves the technical problem of how to improve the accuracy of interactive control in noisy environments within smart cockpits, thereby enhancing driving safety and user experience. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the interactive control method provided in the above embodiments, and will not be elaborated upon here.

[0107] This application provides a vehicle having the electronic device described above, the electronic device being used to execute the interactive control method in the above embodiments.

[0108] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the interactive control method described above.

[0109] The computer program product provided in this application can improve the accuracy of interactive control in noisy environments in smart cockpits, thereby enhancing driving safety and user experience. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the interactive control method provided in the above embodiments, and will not be repeated here.

[0110] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. An interactive control method, characterized in that, The interactive control method includes: In response to head gesture interaction commands, it acquires a sequence of facial images of the driver; Spatiotemporal features are obtained by extracting features from the facial image sequence, and first text features are obtained by encoding preset text prompts, wherein the preset text prompts include each candidate head action category; Based on the spatiotemporal features and the first text features, the visual language model determines the target head action category from the candidate head action categories, and performs vehicle control operations according to the user intent corresponding to the target head action category.

2. The interactive control method as described in claim 1, characterized in that, The step of extracting spatiotemporal features from the facial image sequence includes: The facial image sequence is subjected to feature extraction using a visual encoder based on a visual language model to obtain spatiotemporal features; The step of encoding preset text prompt words to obtain the first text feature includes: The preset text prompt words are encoded by the text encoder of the visual language model to obtain the first text feature.

3. The interactive control method as described in claim 1, characterized in that, The first text feature includes sub-feature vectors corresponding to each of the candidate head action categories. The step of determining the target head action category from each of the candidate head action categories using a visual language model based on the spatiotemporal features and the first text feature includes: The similarity between the spatiotemporal features and each of the sub-feature vectors is calculated using a visual language model based on the spatiotemporal features and the first text features. The candidate head action category corresponding to the maximum value among the similarities is determined as the target head action category, and the maximum value is output as the confidence score of the target head action category.

4. The interactive control method as described in claim 1, characterized in that, Before the step of performing vehicle control operations based on the user intent corresponding to the target head movement category, the method further includes: Obtain contextual interaction information in the current driving scenario; The user intent is determined based on the contextual interaction information and the target head action category.

5. The interactive control method as described in claim 4, characterized in that, The step of determining the user intent based on the contextual interaction information and the target head action category includes: The contextual interaction information is encoded by the text encoder of the visual language model to obtain the second text feature; The spatiotemporal features, the first text features, and the second text features are fused using the cross-attention mechanism of the visual language model to obtain multimodal fusion features; Based on the multimodal fusion features, each candidate intent in the current interaction scenario is determined, and the user intent is determined from each candidate intent based on the target head action category.

6. The interactive control method as described in claim 1, characterized in that, Prior to the step of acquiring a sequence of facial images of the driver in response to a head gesture interaction command, the method further includes: Real-time monitoring of audio data inside the vehicle, and determination of real-time signal-to-noise ratio based on the audio data; When the real-time signal-to-noise ratio is less than a first preset threshold, the head posture interaction command is triggered.

7. The interactive control method as described in claim 6, characterized in that, After the step of determining the target head action category from the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features, the method further includes: When the real-time signal-to-noise ratio is less than a second preset threshold, the amplitude and duration of the head movement performed by the driver are determined, wherein the second preset threshold is less than the first preset threshold. If the amplitude of the action is greater than a preset amplitude threshold and the duration of the action is greater than or equal to a preset duration threshold, then the step of performing vehicle control operation according to the user intent corresponding to the target head action category is executed. If the amplitude of the movement is less than or equal to a preset amplitude threshold or the duration of the movement is less than a preset duration threshold, a prompt message is output, and the process returns to the step of collecting the driver's facial image sequence. The prompt message is used to instruct the driver to increase the amplitude of the movement or repeat the head movement.

8. The interactive control method as described in claim 7, characterized in that, After the step of determining the target head action category from the candidate head action categories using a visual language model based on the spatiotemporal features and the first text features, the method further includes: If the real-time signal-to-noise ratio is less than the second preset threshold, obtain the confidence level of the target head action category; If the confidence level is less than a preset confidence threshold, then return to the step of collecting the driver's facial image sequence.

9. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the interactive control method as described in any one of claims 1 to 8.

10. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 9.

11. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the interactive control method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the interactive control method as described in any one of claims 1 to 8.

Images