Cabin interaction method and vehicle

Abstract

The application discloses a cockpit interaction method and a vehicle, and relates to the technical field of vehicle control. The application obtains an original lip image sequence of a user in a cockpit and an original vehicle state sequence. Since a cockpit interaction model can represent the correlation between the lip image sequence, the vehicle state sequence and a vehicle control instruction, the correlation between the original lip image sequence and the original vehicle state sequence and the vehicle control instruction is analyzed through the cockpit interaction model, the interaction demand expressed by the user in the cockpit through the lips under a specific vehicle state is accurately understood, and an accurate vehicle control instruction is output, so that the intermediate text recognition and text transmission link are avoided. In addition, the cockpit interaction model deeply fuses the vehicle state sequence as context in the reasoning process, can effectively eliminate lip ambiguity, and thus solves the technical problem that the cockpit interaction experience of the user is poor.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and more particularly to a cockpit interaction method and a vehicle. Background Technology

[0002] Currently, a cascaded processing architecture is typically used to process lip movements to achieve cockpit interaction. Specifically, a visual model is first used to identify natural language text from the user's lip image sequence, and then the identified natural language text is sent to an independent semantic understanding module. The semantic understanding module and the rule matching module then convert the natural language text into vehicle control commands.

[0003] However, in the above method, the visual recognition model and the natural language understanding model are trained independently. The erroneous characters identified from lip reading are directly passed to the natural language understanding model, which amplifies the errors and reduces the accuracy of the finally matched vehicle control commands. Summary of the Invention

[0004] The main purpose of this application is to provide a cockpit interaction method and vehicle, which aims to solve the technical problem of poor user cockpit interaction experience.

[0005] To achieve the above objectives, this application proposes a cockpit interaction method, the cockpit interaction method comprising: Acquire the original lip image sequence of the user in the cockpit, as well as the original vehicle state sequence; Based on the original lip image sequence and the original vehicle state sequence, vehicle control commands are output through the cockpit interaction model to realize cockpit interaction. The cockpit interaction model is used to characterize the correlation between the lip image sequence, the vehicle state sequence and the vehicle control commands.

[0006] In one embodiment, the cockpit interaction model includes an intent recognition module, the original lip image sequence includes the current frame lip image and consecutive preset frame lip images before the current frame, and the original vehicle state sequence includes vehicle state data with the same timestamp as each frame in the lip image sequence. The step of outputting vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence includes: The original lip image sequence and the original vehicle state sequence are input into the cockpit interaction model; Based on the original lip image sequence, the original vehicle state sequence, and the instruction prototype embedding vector built into the cockpit interaction model, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame. The intent recognition result includes intent type, confidence level, and timestamp. Based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames, the corresponding vehicle control command is output.

[0007] In one embodiment, the number of consecutive preset frames is determined based on a target timestamp and the timestamp corresponding to the current frame, wherein the target timestamp is the timestamp corresponding to when it is determined that the user in the cockpit has the intention to interact with the cockpit.

[0008] In one embodiment, the step of outputting a corresponding vehicle control command based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames includes: When the confidence level of the current time step is greater than the preset confidence threshold, the intention evolution state is determined based on the intention type of each time step corresponding to the current time step and the consecutive preset frames, as well as the timestamp of each time step corresponding to the current time step and the consecutive preset frames. The intention evolution state includes non-instruction state, intention exploration state, and specific instruction state. When the intent evolution state is a specific instruction state, the corresponding vehicle control instruction is output according to the intent type of the current time step.

[0009] In one embodiment, the users in the cabin include passengers, and the cabin interaction model further includes a decoder and a semantic understanding module. The step of outputting the corresponding vehicle control command based on the intent type of the current time step includes: Based on the intent type of the current time step, the decoder outputs structured text; The semantic understanding module identifies whether there are referential words, ambiguous words, and error-correction keywords in the structured text; If referential words exist, the passenger riding status is obtained, the structured text is adjusted according to the passenger riding status, and the corresponding vehicle control command is output. If there are ambiguous words, the structured text is adjusted according to the original vehicle state sequence, and the corresponding vehicle control command is output. If there are error correction keywords, the previous vehicle control command is retrieved, the structured text is adjusted according to the previous vehicle control command, and the corresponding vehicle control command is output.

[0010] In one embodiment, the step of outputting the corresponding vehicle control command based on the intent type of the current time step includes: Determine the operation security level corresponding to the intent type of the current time step, wherein the operation security level includes a first level, a second level, and a third level, sorted from low to high; When the operation safety level is the first level, the corresponding vehicle control command is output according to the intent type of the current time step; When the safety level is the second level, it is determined whether there is an intent type in the intent type of each time step corresponding to the consecutive preset frames that is the same as the intent type of the current time step. If there is, a vehicle control command is output according to the intent type of the current time step. When the safety level is level three, based on the original vehicle state sequence, it is determined whether the user in the cabin has pressed the physical confirmation button. If so, a vehicle control command is output according to the intent type of the current time step.

[0011] In one embodiment, the cockpit interaction model further includes an encoding layer and a time alignment module. The step of outputting the intent recognition result of the current frame corresponding to the current time step through the intent recognition module based on the original lip image sequence, the original vehicle state sequence, and the instruction prototype embedding vector built into the cockpit interaction model includes: The original lip image sequence and the original vehicle state sequence are encoded by the encoding layer respectively, and the original lip image embedding sequence and the original vehicle state embedding sequence with the same preset dimension are output. The preset dimension is determined according to the dimension of the instruction prototype embedding vector. The original lip image embedding sequence and the original vehicle state embedding sequence are time-aligned using the time alignment module. Based on the aligned lip image embedding sequence, the aligned vehicle state embedding sequence, and the instruction prototype embedding vector, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame.

[0012] In one embodiment, the training process of the time alignment module includes: Obtain historical valid interaction samples, which include historical facial image sequences of historical users and historical vehicle status sequences; Based on the historical facial image sequence, determine the first timestamp corresponding to when the historical user had the intention to interact with the cockpit, the second timestamp when the historical user's lips began to move, and the third timestamp when the historical user's lips ended to move. Based on the first timestamp, the second timestamp, and the third timestamp, the intention exploration interval, the intention output interval, and the historical lip image sequence are determined; The historical lip image sequence and the historical vehicle state sequence are encoded by a historical coding layer to output the historical lip image embedding sequence and the historical vehicle state embedding sequence. Historical lip image embeddings and historical vehicle state embeddings with the same timestamp within the intent output interval are identified as first positive sample pairs, and historical lip image embeddings within the intent output interval and historical vehicle state embeddings within the intent exploration interval are identified as second positive sample pairs. Historical lip image embeddings and historical vehicle state embeddings with different timestamps were identified as negative sample pairs; Based on the first positive sample pair, the second positive sample pair, and the negative sample pair, the preset contrastive learning loss function is iteratively trained to obtain the time alignment module.

[0013] In one embodiment, prior to the steps of acquiring the original lip image sequence of the user in the cockpit and the original vehicle state sequence, the method further includes: Acquire raw facial image data of users inside the cockpit; Based on the original facial image data, determine the position of the user's line of sight inside the cabin; Based on the location of the line of sight, it is determined whether the user in the cockpit intends to interact with the cockpit.

[0014] In one embodiment, the users in the cockpit include passengers, and after the step of outputting vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence, the method further includes: The passenger status of each passenger in the cabin is obtained, including the passenger status as awake and the passenger status as asleep. When all passengers are awake, the vehicle control results are fed back through voice announcements and icons displayed on the head-up display. When some passengers are awake, the vehicle control results are fed back through icons displayed on the head-up display and steering wheel vibration. When all passengers are asleep, no vehicle control results are fed back or displayed on the instrument panel.

[0015] In addition, to achieve the above objectives, this application also proposes a vehicle comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the cockpit interaction method as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application acquires the original lip image sequence of the user in the cockpit and the original vehicle state sequence; based on the original lip image sequence and the original vehicle state sequence, it outputs vehicle control commands through a cockpit interaction model, thereby realizing cockpit interaction through the vehicle control commands. Since the cockpit interaction model can characterize the correlation between the lip image sequence, the vehicle state sequence, and the vehicle control commands, by analyzing the correlation between the original lip image sequence and the original vehicle state sequence and the vehicle control commands respectively through the cockpit interaction model, the interaction needs expressed by the user in the cockpit through lip movements under a specific vehicle state can be accurately understood, and accurate vehicle control commands can be output, avoiding intermediate text recognition and text transmission links; moreover, the cockpit interaction model deeply integrates the vehicle state sequence as context during the inference process, which can effectively resolve lip-reading ambiguity and improve the accuracy of cockpit interaction. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the cockpit interaction method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the cockpit interaction method of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the cockpit interaction method of this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the cockpit interaction method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] Based on this, embodiments of this application provide a cockpit interaction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the cockpit interaction method of this application.

[0024] In this embodiment, the cockpit interaction method includes steps S10 to S20: Step S10: Obtain the original lip image sequence of the user in the cockpit, as well as the original vehicle state sequence; Step S20: Based on the original lip image sequence and the original vehicle state sequence, output vehicle control commands through the cockpit interaction model to realize cockpit interaction through the vehicle control commands. The cockpit interaction model is used to characterize the correlation between the lip image sequence, the vehicle state sequence and the vehicle control commands.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or smart cockpit domain controller capable of performing the above functions. The following description uses a smart cockpit domain controller as an example to illustrate this embodiment and the subsequent embodiments.

[0026] It should be noted that the intelligent cockpit domain controller is subordinate to the intelligent cockpit interaction system, which is deployed in the vehicle. The intelligent cockpit interaction system also includes a cockpit state bus, which communicates with the vehicle hardware, including air conditioning, audio system, sunshades, windows, seats, etc. This cockpit state bus can be a controller area network (LAN) bus or a local area network (LAN) bus. The intelligent cockpit interaction system may also be equipped with a camera with infrared illumination. This camera automatically activates its infrared illumination function at night or in low-light environments to ensure that the captured lip images are clearly visible.

[0027] Specifically, a camera with infrared illumination can continuously collect facial image data of users in the cabin at a rate of 30 frames per second. Images containing the lip area can be extracted from the facial image data and arranged in chronological order of collection time to form an original lip image sequence. Vehicle status data, including vehicle speed, air conditioning set temperature, media volume, and sunshade position, can be acquired in real time via the cabin status bus at a frequency of 10 times per second. The vehicle status data can be arranged in chronological order of collection time to form an original vehicle status sequence.

[0028] The users inside the cabin can be either the driver or a passenger.

[0029] The intelligent cockpit domain controller inputs the acquired raw lip image sequence and raw vehicle state sequence into a pre-trained cockpit interaction model. This cockpit interaction model can be a deep learning model that integrates multiple functions such as visual feature extraction, semantic understanding, and intent reasoning.

[0030] By analyzing the original lip image sequence using the cockpit interaction model and using the original vehicle state sequence as contextual clues, the user's specific intention is inferred, and a structured vehicle control command is directly output.

[0031] The vehicle control command can be encapsulated into a data frame by the intelligent cockpit domain controller via the cockpit state bus and sent to the corresponding vehicle hardware execution unit. The vehicle control command is used to change the vehicle hardware state, such as setting the air conditioning temperature to 24 degrees or reducing the media volume to 30%.

[0032] Because this cockpit interaction model learns the relationship between lip image sequences, vehicle state sequences, and vehicle control commands, it can use vehicle state to eliminate the ambiguity of lip movements, avoid false triggers, and improve the accuracy of cockpit interaction. Moreover, the driver does not need to make a sound or shift their gaze to operate the touchscreen; they can control vehicle functions simply through natural lip movements. This makes it suitable for quiet scenarios where passengers are resting in the vehicle or driving environments where the environment is noisy and voice recognition is inaccurate.

[0033] In one feasible implementation, the cockpit interaction model can be trained by iteratively training a preset model based on historical lip image sequences, historical vehicle state sequences, and vehicle control command labels.

[0034] In one feasible implementation, the specific way to output vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence can be as follows: The original lip image sequence is normalized by scaling the pixel values ​​of each frame of the lip image to a preset range. Simultaneously, discrete quantities (e.g., gear position) in the original vehicle state sequence are one-hot encoded, and continuous quantities (e.g., vehicle speed) are normalized. The processed lip image sequence and the processed vehicle state sequence are then input into the cockpit interaction model. The cockpit interaction model may include a 3D convolutional neural network, a gated recurrent unit network, a multi-head attention fusion layer, and a lightweight feedforward network. The 3D convolutional neural network extracts spatiotemporal features from the processed lip image sequence, the gated recurrent unit network extracts vehicle state change features from the processed vehicle state sequence, the multi-head attention fusion layer interacts with the spatiotemporal features and vehicle state change features, and the lightweight feedforward network outputs specific vehicle control commands.

[0035] Compared to the traditional cascaded processing method that first identifies text from lip image sequences and then converts the text into vehicle control commands, the above-mentioned method in this embodiment realizes end-to-end mapping between lip image sequences, vehicle state sequences, and vehicle control commands through a cockpit interaction model. This avoids intermediate text recognition and text transmission steps. Furthermore, the cockpit interaction model deeply integrates the vehicle state sequence as context during the inference process, effectively resolving lip-reading ambiguity and improving the accuracy of cockpit interaction.

[0036] In one feasible implementation, after outputting vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence, the intelligent cockpit domain controller can also adaptively select a feedback method to inform the driver or passengers of the corresponding vehicle control results based on the different riding states of different passengers in the cockpit.

[0037] The system acquires the seating status of each passenger in the cabin, including both awake and asleep states. When all passengers are awake, the system provides feedback on vehicle control results via voice announcements and icons displayed on the head-up display. When some passengers are awake, the system provides feedback on vehicle control results via icons displayed on the head-up display and steering wheel vibration. When all passengers are asleep, the system does not provide feedback on vehicle control results, or displays the vehicle control results on the instrument panel.

[0038] Specifically, the intelligent cockpit domain controller can obtain the seating status of each passenger in the cabin, excluding the driver, through seat sensors or in-vehicle cameras, thereby determining whether each passenger is awake or asleep.

[0039] For example, a specific implementation method for obtaining the seating status of each passenger in the cabin can be: by using seat pressure sensors, millimeter-wave radar micro-motion detection, and ambient sound analysis of the in-vehicle microphone array, the seating status of each passenger can be determined through multimodal fusion, avoiding the limitations of visual recognition in nighttime or low-light environments.

[0040] If all passengers are awake, the intelligent cockpit domain controller can provide clear and rich execution feedback to the driver through both voice broadcasts and icons displayed on the head-up display. If some passengers are asleep while others are awake, the intelligent cockpit domain controller can provide feedback only through icons displayed on the head-up display and steering wheel vibration, avoiding voice broadcasts that might disturb sleeping passengers. If all passengers are asleep, the intelligent cockpit domain controller can choose not to provide any feedback at all, or simply display the vehicle control results on the instrument panel with a dim light.

[0041] This embodiment actively suppresses voice and bright light feedback when passengers are resting, achieving silent interaction with zero interference; when passengers are awake, it provides sufficient feedback to ensure that both the driver and passengers know that the command has been executed. This solves the problem of traditional interaction methods where drivers have control needs in scenarios where passengers are sleeping but do not want to wake others with voice feedback.

[0042] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, the cockpit interaction model includes an intent recognition module, the original lip image sequence includes the lip image of the current frame and the lip images of consecutive preset frames preceding the current frame, and the original vehicle state sequence includes vehicle state data with the same timestamp as each frame in the lip image sequence; please refer to... Figure 2 Step S20 includes steps S01 to S02: Step S01: Input the original lip image sequence and the original vehicle state sequence into the cockpit interaction model; Step S02: Based on the original lip image sequence, the original vehicle state sequence, and the instruction prototype embedding vector built into the cockpit interaction model, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame. The intent recognition result includes intent type, confidence level, and timestamp. Step S03: Based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames, output the corresponding vehicle control command.

[0043] Since the original vehicle state sequence includes vehicle state data with the same timestamp as each frame in the lip image sequence, that is, the original vehicle state sequence is strictly aligned with the original lip image sequence in time, it ensures that the cockpit interaction model can accurately establish the correlation between each frame of lip image and each frame of vehicle state.

[0044] It should be noted that the number of consecutive preset frames is determined based on the target timestamp and the timestamp corresponding to the current frame. The target timestamp is the timestamp corresponding to the moment when the user in the cockpit demonstrates an intention to interact with the cockpit. That is, the number of consecutive preset frames is not fixed, but rather the intelligent cockpit domain controller determines the moment when the user in the cockpit expresses an intention to interact with the cockpit, records the moment of such intention as the target timestamp, and dynamically determines the target timestamp based on the time difference between the target timestamp and the timestamp corresponding to the current frame. This time difference reflects the length of time from when the user in the cockpit generates an interaction intention to when the current frame is displayed.

[0045] Among them, cockpit interaction intent refers to the user's intention to interact with the vehicle's hardware while in the cockpit.

[0046] In one feasible implementation, before acquiring the original lip image sequence of the user in the cockpit and the original vehicle state sequence, the method further includes: Acquire raw facial image data of the user inside the cabin; determine the position of the user's gaze based on the raw facial image data; determine whether the user has any intention to interact with the cabin based on the position of the gaze.

[0047] The intelligent cockpit domain controller first continuously acquires raw facial image data of the driver using a driver monitoring camera equipped with infrared illumination. For each frame of the facial image, the intelligent cockpit domain controller runs a gaze-tracking algorithm to extract key eye features, such as the pupil center, eyeball angle, and corner of the eye position. Based on these key eye features, it calculates the driver's current gaze point. The gaze point refers to the specific intersection point between the user's gaze direction within the cockpit and the projection of that gaze direction onto the in-vehicle hardware or the external environment.

[0048] The intelligent cockpit domain controller analyzes the trajectory of the gaze point in consecutive preset frames to determine whether the gaze point has switched from the road area to the area where the vehicle's hardware devices are located, such as the air conditioning vents, the central control screen, the sunroof area, and the rearview mirror. If such a trajectory is detected, it is determined that the user in the cockpit has the intention to interact with the cockpit.

[0049] This embodiment uses gaze behavior as the interaction trigger condition. Only when the user in the cabin actively looks at the in-vehicle hardware devices will the collected original lip image sequence and the original vehicle state sequence be input into the cabin interaction model. This avoids the waste of computing power caused by continuously running the model and significantly reduces system power consumption. At the same time, based on the interaction habit of looking before speaking, it accurately determines whether the user has an interaction intention and effectively filters out unconscious lip movements of the user in the cabin during normal driving or chatting, such as talking to oneself, singing, yawning, etc., thereby greatly reducing the false trigger rate of the system.

[0050] In one feasible implementation, the specific way the intelligent cockpit domain controller inputs the original lip image sequence and the original vehicle state sequence into the cockpit interaction model can be: When the lip image of the current frame is acquired, the vehicle status corresponding to the timestamp of the current frame is acquired simultaneously, and the lip image of the current frame, the lip images of consecutive preset frames before the current frame, the vehicle status data corresponding to the timestamp of the current frame, and the vehicle status data corresponding to the timestamps of consecutive preset frames before the current frame are simultaneously input into the cockpit interaction model.

[0051] This embodiment ensures that the input layer of the cockpit interaction model is naturally aligned with the lip image sequence and the vehicle state sequence in the time dimension when receiving data by simultaneously inputting the original lip action sequence and the original vehicle state sequence. Each frame of lip action has precise vehicle state information at the time of its occurrence. This eliminates the need for additional data alignment or interpolation operations within the cockpit interaction model. It can directly perform cross-modal association learning on the visual features of the lip image sequence and the state features of the vehicle state sequence at the same timestamp, thereby more accurately capturing the causal relationship between lip actions and vehicle state. For example, when the driver says "too high," the cockpit interaction model can immediately know the specific value of the current air conditioning temperature or playback volume, improving the accuracy and response speed of command recognition.

[0052] In another feasible implementation, the specific implementation of the intelligent cockpit domain controller inputting the original lip image sequence and the original vehicle state sequence into the cockpit interaction model can also be as follows: The intelligent cockpit domain controller employs a dual-buffer ping-pong operation for data input. Specifically, it allocates two independent input buffers in memory, each sized to hold a set number of consecutive preset frames of lip images and corresponding vehicle state vectors. Once the first buffer is full, the controller passes its address pointer to the input layer of the cockpit interaction model, triggering inference. Simultaneously, the second buffer begins receiving the next batch of newly acquired raw lip image sequences and raw vehicle state sequences. After the model inference in the first buffer is complete, the controller immediately switches to the second buffer for the next round of inference, while the first buffer is cleared and refilled. This allows data acquisition and model inference to run in parallel, avoiding processing idle time caused by waiting for data to be filled.

[0053] It should be noted that the command prototype embedding vector can be a set of learnable vector parameters pre-stored within the cockpit interaction model. For example, the number of vector parameters can be 100, and each vector parameter has a dimension of 768. These vector parameters represent a set of abstract, open-vocabulary command semantic primitives. The cockpit interaction model can learn how to match and combine the input raw lip image sequence and raw vehicle state sequence with these command semantic primitives to understand various specific vehicle control commands.

[0054] The intelligent cockpit domain controller can send the input original lip image sequence, the original vehicle state sequence, and the embedded command prototype vector within the cockpit interaction model to the intent recognition module. This intent recognition module uses an attention mechanism to temporally correlate the visual features corresponding to the original lip image of the current frame with the visual features of the consecutive preset frames before the current frame. At the same time, it combines the vehicle state data corresponding to the vehicle state features with the same timestamp as the current frame and calculates the similarity with the command prototype vector.

[0055] For the current time step corresponding to the current frame, the intent recognition module outputs a triple: intent type, confidence level, and timestamp. For example, when the user's gaze has just shifted to the air conditioning panel but their lips haven't moved, the intent recognition module outputs an intent type of "exploratory intent" with a confidence level of 0.85; when the user begins to lip-read "lower the temperature," the intent recognition module outputs an intent type of "specific instruction" with a confidence level of 0.95.

[0056] This embodiment does not directly output the final command text, but first outputs the intent recognition result with timing information. This allows the cockpit interaction model to distinguish whether the user in the cockpit is acting unconsciously, thinking and exploring, or issuing a clear command, thereby effectively reducing the false trigger rate caused by the driver's unconscious lip movements.

[0057] In one feasible implementation, the step of outputting the corresponding vehicle control command based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames includes: When the confidence level of the current time step is greater than a preset confidence threshold, the intent evolution state is determined based on the intent type of each time step corresponding to the current time step and the consecutive preset frames, as well as the timestamp of each time step corresponding to the current time step and the consecutive preset frames. The intent evolution state includes a non-instruction state, an intent exploration state, and a specific instruction state. When the intent evolution state is a specific instruction state, the corresponding vehicle control instruction is output based on the intent type of the current time step.

[0058] The intelligent cockpit domain controller can first determine whether the confidence level of the intent recognition result corresponding to the current time step is greater than a preset confidence threshold. If the confidence level of the intent recognition result is greater than the preset confidence threshold, it means that the cockpit interaction model is relatively confident in the current judgment.

[0059] Furthermore, the cockpit interaction model can analyze the changing trends of intent types based on the intent recognition results at the current time step and the intent recognition results at each time step in the consecutive preset frames preceding the current frame, which are output historically by the cockpit interaction model. Intent types include no intent, specific commands, etc.

[0060] For example, if the intent type of multiple consecutive time steps is a specific instruction and the confidence level is higher than 0.9, it is determined that the user in the cabin has indeed issued a specific instruction. That is, the intent evolution state is determined to be a specific instruction state, and then the corresponding vehicle control instruction is generated and output according to the intent type of the current time step.

[0061] If only one time step in a series of consecutive time steps has a specific instruction as its intent type, while the intent types of the preceding and following time steps are all without intent, then it is determined to be a false trigger. That is, the intent evolution state is determined to be a non-instruction state, and no vehicle control command is output.

[0062] This embodiment uses the intent type of consecutive preset frames to determine the intent evolution state, effectively filtering out false intents caused by instantaneous noise or accidental lip movements, and ensuring that vehicle control commands are only output when the intent type is consistently and stably expressed as a "specific instruction".

[0063] In another feasible implementation, the method of outputting the corresponding vehicle control command based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames can also be: A weighted voting mechanism is employed: the intent types of consecutive preset frames are weighted according to proximity (higher weight) and distance (lower weight). The confidence level of each time step is then weighted based on this weight. If the weighted score exceeds a specific value and the intent types of consecutive preset frames are all specific instructions, then a vehicle control command is output. This method involves low computational cost and is suitable for low-cost platforms.

[0064] In one feasible implementation, the cockpit interaction model further includes a decoder and a semantic understanding module, and the implementation of outputting the corresponding vehicle control command according to the intent type of the current time step can be: Based on the intent type of the current time step, the decoder outputs structured text; the semantic understanding module identifies whether there are referential words, ambiguous words, and error-correction keywords in the structured text; if referential words are present, the passenger riding status is obtained, the structured text is adjusted according to the passenger riding status, and the corresponding vehicle control command is output; if ambiguous words are present, the structured text is adjusted according to the original vehicle state sequence, and the corresponding vehicle control command is output; if error-correction keywords are present, the previous vehicle control command is obtained, the structured text is adjusted according to the previous vehicle control command, and the corresponding vehicle control command is output.

[0065] It should be noted that the decoder can employ a lightweight language model based on the Transformer architecture. Structured text refers to the formatted text output by the decoder, which may contain key information such as action type, target object, and parameter values, for example, "adjust," "air conditioning temperature," and "24 degrees." Referential words are used to refer to an object or location in the preceding or surrounding text, such as "over there," "here," "it," and "that." Vague words are those whose meaning is ambiguous in the command and requires context to determine the specific operation, such as "a little higher," "a little smaller," "too hot," and "a little cold." Error-correcting keywords are used to correct words in the previous vehicle control command, such as "incorrect," "wrong," "change to," and "still." The previous vehicle control command refers to the most recently output and executed vehicle control command from the intelligent cockpit domain controller.

[0066] In this embodiment, the intelligent cockpit domain controller can decode the intent type word by word into structured text based on the intent type of the current time step through a decoder; and scan the structured text through a semantic understanding module to identify whether the structured text contains referential words, ambiguous words or error correction keywords.

[0067] For example, if the referential word "the window over there" is detected, the semantic understanding module obtains the riding status of each passenger (sleeping or awake), and selects the most reasonable referent by combining semantic reasoning. For example, "the window over there" is resolved to the window on the side away from the sleeping passenger, and the operation object in the structured text is adjusted.

[0068] If the ambiguous word "a little higher" is detected, the semantic understanding module obtains the current air conditioning temperature value and playback volume from the original vehicle state sequence, determines whether it is more reasonable to increase the temperature or to increase the volume, and supplements and adjusts the specific parameter values ​​of the structured text.

[0069] If the error correction keyword "incorrect, 23 degrees" is detected, the semantic understanding module obtains the previous vehicle control command. If the current specific intent is to correct the previous command, the parameter value of the previous command is overwritten or modified.

[0070] This embodiment introduces multi-dimensional context (passenger status, vehicle status, historical commands) for semantic disambiguation, enabling users to express their intentions in the most natural and conversational way, without having to memorize precise command formats, thus improving the accuracy of command understanding and user experience.

[0071] In one feasible implementation, the method of outputting the corresponding vehicle control command based on the intent type of the current time step can be: The operation safety level corresponding to the intent type of the current time step is determined. The operation safety level includes a first level, a second level, and a third level, sorted from low to high. When the operation safety level is the first level, a corresponding vehicle control command is output according to the intent type of the current time step. When the safety level is the second level, it is determined whether there is an intent type in the intent types of each time step corresponding to the consecutive preset frames that is the same as the intent type of the current time step. If so, a vehicle control command is output according to the intent type of the current time step. When the safety level is the third level, it is determined whether the user in the cabin has pressed the physical confirmation button according to the original vehicle state sequence. If so, a vehicle control command is output according to the intent type of the current time step.

[0072] It should be noted that the operational safety level refers to a pre-defined classification based on the degree to which the types of operations involved in vehicle control commands affect driving safety. A higher level indicates a greater safety risk that could result from misoperation. Level 1 can be the lowest level, corresponding to operation types that do not directly affect vehicle driving safety and whose consequences are minor, such as querying information or adjusting non-safety-related comfort features. Level 2 can be an intermediate level, corresponding to operation types with some safety impact that require additional confirmation, such as adjusting air conditioning temperature, audio volume, or seat position. Level 3 can be the highest level, corresponding to operation types that directly affect vehicle dynamics or may lead to safety risks, such as switching driving modes or controlling the opening and closing of windows or sunroof.

[0073] The intelligent cockpit domain controller can obtain the operation type corresponding to the current time step intent type and look up the operation security level to which the operation type belongs in the internally pre-stored security level mapping table.

[0074] If the operational safety level is Level 1, the intelligent cockpit domain controller can directly output vehicle control commands based on the intent type of the current time step without additional confirmation.

[0075] If the operational safety level is Level 2, the intelligent cockpit domain controller can further check whether there is an intent type that is the same as the intent type of the current time step in the intent recognition results of each time step corresponding to consecutive preset frames. For example, whether at least one frame in the past preset frames also recognized the intent to adjust the air conditioner. If so, it means that the user has expressed the same intent multiple times within a period of time. The vehicle control command can be output according to the intent type of the current time step. The existence of a similar intent type in the past preset frames indicates that the user has confirmed the intent again in the current frame; otherwise, no vehicle control command is output.

[0076] If the operational safety level is Level 3, the intelligent cockpit domain controller can read the real-time status of the physical confirmation button via the cockpit state bus. The physical confirmation button refers to a physical button deployed on the vehicle's steering wheel or center console. The user needs to press this button to clearly express their intention to confirm a high-risk operation. The status of this button is reported to the intelligent cockpit domain controller in real time via the cockpit state bus. Only when the press of this physical confirmation button is detected can vehicle control commands be output according to the intent type of the current time step.

[0077] This embodiment avoids the cumbersome operation caused by uniformly using physical buttons through a hierarchical confirmation mechanism, and also avoids the security risks caused by uniform non-confirmation, thus balancing security and convenience.

[0078] Based on the first and second embodiments of this application, the same or similar content as the above embodiments in the third embodiment of this application can be referred to the above description and will not be repeated hereafter. In addition, the cockpit interaction model also includes an encoding layer and a time alignment module, please refer to... Figure 3 Step S02 includes steps A1-A3: Step A1: The original lip image sequence and the original vehicle state sequence are encoded by the encoding layer respectively, and the original lip image embedding sequence and the original vehicle state embedding sequence with the same preset dimension are output. The preset dimension is determined according to the dimension of the instruction prototype embedding vector. Step A2: The original lip image embedding sequence and the original vehicle state embedding sequence are time-aligned using the time alignment module. Step A3: Based on the aligned lip image embedding sequence, the aligned vehicle state embedding sequence, and the instruction prototype embedding vector, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame.

[0079] It should be noted that the coding layer can consist of parallel visual encoders and state visual encoders. The visual encoders can be three-dimensional convolutional neural networks or temporal self-attention networks, while the state encoders can typically be fully connected networks or gated recurrent unit networks.

[0080] The intelligent cockpit domain controller can feed the raw lip image sequence into the encoding layer. The visual encoder in the encoding layer can extract the spatiotemporal features of each frame of the lip image through 3D convolution or temporal attention mechanisms, and output a feature vector sequence, namely the raw lip image embedding sequence. The dimension of each feature vector in the raw lip image embedding sequence is a preset dimension (e.g., 768 dimensions). At the same time, the state encoding layer in the encoding layer can map the state data of each time step to the same preset dimension through a fully connected network, and output the raw vehicle state embedding sequence.

[0081] In this embodiment, after mapping the original lip image sequence and the original vehicle state sequence to the same dimensional space as the instruction prototype embedding vector through the encoding layer, the cockpit interaction model can perform similarity calculation and attention interaction in a unified representation space, thereby learning the correlation between the lip image sequence and the vehicle state sequence more effectively.

[0082] Furthermore, since the lip image acquisition frame rate is 30 frames per second, while the vehicle state sampling frequency is 10 Hz, the two do not correspond one-to-one in their original timestamps. In a feasible implementation, a time alignment module can resample the original lip image embedding sequence and the original vehicle state embedding sequence along the time axis using algorithms such as dynamic time warping or linear interpolation, based on the timestamp information attached to each embedding vector in the original lip image embedding sequence and each embedding vector in the original vehicle state embedding sequence. This ensures that the aligned lip image embedding sequence and the vehicle state embedding sequence have a strictly corresponding temporal relationship at each time step. This solves the temporal matching problem of multi-source heterogeneous data and improves the model's recognition accuracy.

[0083] In another feasible implementation, the training process of the time alignment module can be: The process involves acquiring historical valid interaction samples, including historical facial image sequences and historical vehicle state sequences of historical users. Based on the historical facial image sequences, a first timestamp corresponding to the historical user's cockpit interaction intent, a second timestamp indicating the start of the historical user's lip movement, and a third timestamp indicating the end of the historical user's lip movement are determined. Based on the first, second, and third timestamps, an intent exploration interval, an intent output interval, and a historical lip image sequence are determined. The historical lip image sequence and the historical vehicle state sequence are encoded using a historical encoding layer to output historical lip image embedding sequences and historical vehicle state embedding sequences. Historical lip image embeddings and historical vehicle state embeddings with the same timestamp within the intent output interval are identified as first positive sample pairs, and historical lip image embeddings within the intent output interval and historical vehicle state embeddings within the intent exploration interval are identified as second positive sample pairs. Historical lip image embeddings and historical vehicle state embeddings with different timestamps are identified as negative sample pairs. Based on the first positive sample pairs, the second positive sample pairs, and the negative sample pairs, a preset contrastive learning loss function is iteratively trained to obtain the time alignment module.

[0084] It should be noted that historical valid interaction samples refer to the complete process data collected from real driving scenarios during the model training phase, which includes the user successfully completing lip-reading interactions. Specifically, this includes historical facial image sequences and corresponding historical vehicle state sequences.

[0085] During the training phase, the changes in the gaze position can be analyzed based on historical facial image sequences to determine the first timestamp of the user's intention to interact with the cockpit. Simultaneously, lip motion energy can be analyzed based on historical facial image sequences. This lip motion energy is a quantitative indicator of the intensity and activity of the user's lip region across multiple consecutive frames. Based on the lip motion energy, the second timestamp of the start of the lip movement and the third timestamp of the end of the lip movement can be determined. The corresponding image set from the second timestamp to the third timestamp is then used as the historical lip image sequence.

[0086] Based on the first, second, and third timestamps, the interval from the first to the second timestamp can be considered the intent exploration interval, i.e., the user has expressed an interaction intent but has not yet begun issuing lip-reading commands. The interval from the second to the third timestamp can be considered the intent output interval, i.e., the user is issuing lip-reading commands. A historical encoding layer maps historical facial image sequences and corresponding historical vehicle state sequences to historical lip image embedding sequences and historical vehicle state embedding sequences, respectively, and constructs positive and negative sample pairs. Positive sample pairs can include a first positive sample and a second positive sample. The first positive sample pair can be a visual embedding and a state embedding with the same timestamp within the intent output interval, used to train the model to learn the strong correlation between vision and state during instruction execution. The second positive sample pair can be a visual embedding within the intent output interval and a state embedding within the intent exploration interval, used to train the model to learn the temporal causal relationship from exploration to execution. Negative sample pairs can be visual embeddings and state embeddings with different timestamps, used to train the model to distinguish samples that are not temporally aligned.

[0087] The goal of the contrastive learning loss function is to bring positive sample pairs closer together in the feature space while widening the distance between negative sample pairs. Based on the first positive sample pair, the second positive sample pair, and the negative sample pair, iterative training is performed using the contrastive learning loss function, enabling the temporal alignment module to learn to bring temporally related visual and state features closer together in the embedding space, while distancing temporally unrelated features from each other.

[0088] It is understandable that by using gaze switching as a natural weak supervision anchor point, the model can learn the temporal correspondence between lip movements and vehicle states autonomously without the need for manual annotation of a large amount of precisely aligned data. This allows for accurate time alignment even when facing new users and new scenarios that have never been seen before during the inference stage.

[0089] After the original lip image embedding sequence and the original vehicle state embedding sequence are time-aligned by the time alignment module, the intelligent cockpit domain controller can concatenate or interact with the aligned lip image embedding sequence and the aligned vehicle state embedding sequence with the built-in instruction prototype embedding vector and send them to the intent recognition module.

[0090] The intent recognition module can capture cross-modal associations between visual features and state features through a multi-head attention mechanism, and at the same time calculate the similarity between these features and the instruction prototype embedding vector. Based on cross-modal associations and similarity, the intent recognition module outputs intent recognition results for the current time step, including intent type, confidence level and timestamp.

[0091] This embodiment achieves multimodal context-aware intent understanding by fusing time-aligned visual features and state features. Even if the lip image itself is blurry, the cockpit interaction model can still use the aligned vehicle state features to help determine the user's intent, thus maintaining a high recognition accuracy in complex environments.

[0092] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the cockpit interaction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0093] This application provides a vehicle, the vehicle including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the cockpit interaction method in Embodiment 1 above.

[0094] The following is for reference. Figure 4 The diagram illustrates a structural schematic of a vehicle suitable for implementing embodiments of this application. The vehicle in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, tablets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 4 The vehicle shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0095] like Figure 4As shown, the vehicle may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for vehicle operation. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the vehicle to communicate wirelessly or wiredly with other devices to exchange data. Although vehicles with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0096] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0097] The vehicle provided in this application, employing the cockpit interaction method described in the above embodiments, can solve the technical problem of poor user cockpit interaction experience. Compared with the prior art, the beneficial effects of the vehicle provided in this application are the same as those of the cockpit interaction method provided in the above embodiments, and other technical features of the vehicle are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0098] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0099] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0100] The above descriptions are merely some embodiments of this application and do not limit the scope of protection of this application. Any equivalent structural transformations made based on the technical concept of this application and the content of this specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection laws and policies of the country where the application is located and with authorization from the owner of the corresponding device.

Claims

1. A cockpit interaction method, characterized in that, The cockpit interaction method includes: Obtain the original lip image sequence of the user in the cockpit, as well as the original vehicle state sequence; Based on the original lip image sequence and the original vehicle state sequence, vehicle control commands are output through the cockpit interaction model to realize cockpit interaction. The cockpit interaction model is used to characterize the correlation between the lip image sequence, the vehicle state sequence and the vehicle control commands.

2. The cockpit interaction method as described in claim 1, characterized in that, The cockpit interaction model includes an intent recognition module, the original lip image sequence includes the current frame lip image and consecutive preset frame lip images before the current frame, and the original vehicle state sequence includes vehicle state data with the same timestamp as each frame in the lip image sequence. The step of outputting vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence includes: The original lip image sequence and the original vehicle state sequence are input into the cockpit interaction model; Based on the original lip image sequence, the original vehicle state sequence, and the instruction prototype embedding vector built into the cockpit interaction model, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame. The intent recognition result includes intent type, confidence level, and timestamp. Based on the intent recognition result of the current time step and the intent recognition results of each time step corresponding to the consecutive preset frames, the corresponding vehicle control command is output.

3. The cockpit interaction method as described in claim 2, characterized in that, The number of consecutive preset frames is determined based on the target timestamp and the timestamp corresponding to the current frame. The target timestamp is the timestamp corresponding to when it is determined that the user in the cockpit has the intention to interact with the cockpit.

4. The cockpit interaction method as described in claim 2, characterized in that, The step of outputting the corresponding vehicle control command based on the intent recognition result of the current time step and the intent recognition results of each time step in the consecutive preset frames includes: When the confidence level of the current time step is greater than the preset confidence threshold, the intention evolution state is determined based on the intention type of each time step corresponding to the current time step and the consecutive preset frames, as well as the timestamp of each time step corresponding to the current time step and the consecutive preset frames. The intention evolution state includes non-instruction state, intention exploration state, and specific instruction state. When the intent evolution state is a specific instruction state, the corresponding vehicle control instruction is output according to the intent type of the current time step.

5. The cockpit interaction method as described in claim 4, characterized in that, The users in the cockpit include passengers, and the cockpit interaction model further includes a decoder and a semantic understanding module. The step of outputting the corresponding vehicle control command based on the intent type at the current time step includes: Based on the intent type of the current time step, the decoder outputs structured text; The semantic understanding module identifies whether there are referential words, ambiguous words, and error-correction keywords in the structured text; If referential words exist, the passenger riding status is obtained, the structured text is adjusted according to the passenger riding status, and the corresponding vehicle control command is output. If there are ambiguous words, the structured text is adjusted according to the original vehicle state sequence, and the corresponding vehicle control command is output. If there are error correction keywords, the previous vehicle control command is retrieved, the structured text is adjusted according to the previous vehicle control command, and the corresponding vehicle control command is output.

6. The cockpit interaction method as described in claim 4, characterized in that, The step of outputting the corresponding vehicle control command based on the intent type of the current time step includes: Determine the operation security level corresponding to the intent type of the current time step, wherein the operation security level includes a first level, a second level, and a third level, sorted from low to high; When the operation safety level is the first level, the corresponding vehicle control command is output according to the intent type of the current time step; When the safety level is the second level, it is determined whether there is an intent type in the intent type of each time step corresponding to the consecutive preset frames that is the same as the intent type of the current time step. If there is, a vehicle control command is output according to the intent type of the current time step. When the safety level is level three, based on the original vehicle state sequence, it is determined whether the user in the cabin has pressed the physical confirmation button. If so, a vehicle control command is output according to the intent type of the current time step.

7. The cockpit interaction method as described in claim 2, characterized in that, The cockpit interaction model further includes an encoding layer and a time alignment module. The step of outputting the intent recognition result of the current frame corresponding to the current time step through the intent recognition module based on the original lip image sequence, the original vehicle state sequence, and the instruction prototype embedding vector built into the cockpit interaction model includes: The original lip image sequence and the original vehicle state sequence are encoded by the encoding layer respectively, and the original lip image embedding sequence and the original vehicle state embedding sequence with the same preset dimension are output. The preset dimension is determined according to the dimension of the instruction prototype embedding vector. The original lip image embedding sequence and the original vehicle state embedding sequence are time-aligned using the time alignment module. Based on the aligned lip image embedding sequence, the aligned vehicle state embedding sequence, and the instruction prototype embedding vector, the intent recognition module outputs the intent recognition result corresponding to the current time step of the current frame.

8. The cockpit interaction method as described in claim 7, characterized in that, The training process of the time alignment module includes: Obtain historical valid interaction samples, which include historical facial image sequences of historical users and historical vehicle status sequences; Based on the historical facial image sequence, determine the first timestamp corresponding to when the historical user had the intention to interact with the cockpit, the second timestamp when the historical user's lips began to move, and the third timestamp when the historical user's lips ended to move. Based on the first timestamp, the second timestamp, and the third timestamp, the intention exploration interval, the intention output interval, and the historical lip image sequence are determined; The historical lip image sequence and the historical vehicle state sequence are encoded by a historical coding layer to output the historical lip image embedding sequence and the historical vehicle state embedding sequence. Historical lip image embeddings and historical vehicle state embeddings with the same timestamp within the intent output interval are identified as first positive sample pairs, and historical lip image embeddings within the intent output interval and historical vehicle state embeddings within the intent exploration interval are identified as second positive sample pairs. Historical lip image embeddings and historical vehicle state embeddings with different timestamps were identified as negative sample pairs; Based on the first positive sample pair, the second positive sample pair, and the negative sample pair, the preset contrastive learning loss function is iteratively trained to obtain the time alignment module.

9. The cockpit interaction method as described in claim 1, characterized in that, Before the steps of acquiring the original lip image sequence of the user in the cockpit and the original vehicle state sequence, the method further includes: Acquire raw facial image data of users inside the cockpit; Based on the original facial image data, determine the position of the user's line of sight inside the cabin; Based on the location of the line of sight, it is determined whether the user in the cockpit intends to interact with the cockpit.

10. The cockpit interaction method as described in claim 1, characterized in that, The users in the cockpit include passengers. After the step of outputting vehicle control commands through the cockpit interaction model based on the original lip image sequence and the original vehicle state sequence, the method further includes: The passenger status of each passenger in the cabin is obtained, including the passenger status as awake and the passenger status as asleep. When all passengers are awake, the vehicle control results are fed back through voice announcements and icons displayed on the head-up display. When some passengers are awake, the vehicle control results are fed back through icons displayed on the head-up display and steering wheel vibration. When all passengers are asleep, no vehicle control results are fed back or displayed on the instrument panel.

11. A vehicle, characterized in that, The vehicle includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the cockpit interaction method as described in any one of claims 1 to 10.

Images