Intelligent cockpit interaction control method and system based on multi-modal perception

Abstract

The application discloses an intelligent cockpit interaction control method and system based on multi-modal perception, the method comprises the following steps: collecting cockpit multi-modal interaction data and vehicle operating state data according to the same sampling time window, and marking the starting subject for the multi-modal interaction data; identifying the environmental interference factors received by each interaction mode in the sampling time window, and generating the real-time credibility of each interaction mode; performing semantic analysis on the interaction instruction, and judging whether the interaction instruction belongs to an explicit object instruction or a fuzzy reference instruction, and the application relates to the technical field of intelligent automobile cockpit human-computer interaction control. The intelligent cockpit interaction control method and system based on multi-modal perception express the same direction confirmation edge, object conflict edge, action conflict edge, subject authority edge, safety inhibition edge and time sequence dependence edge through an object level intention conflict graph, and output results based on the edge types of the graph, so that balance can be achieved among interaction accuracy, response continuity and vehicle operation safety.

Description

Technical Field

[0001] This invention relates to the field of human-machine interaction control in intelligent vehicle cockpits, and more specifically, to an intelligent cockpit interaction control method and system based on multimodal perception. Background Technology

[0002] As intelligent vehicle cockpits evolve towards collaborative interaction via voice, gestures, gaze, touch, and posture, the objects controlled within the cockpit have expanded from traditional buttons to include windows, sunroofs, air conditioning, seats, lights, audio systems, HUDs, navigation interfaces, and driver assistance interfaces. In existing technologies, CN115509366A discloses a multimodal human-machine interaction control method for intelligent cockpits, which achieves control intent recognition through voice recognition, gesture recognition, and a cockpit model. However, this solution focuses on the combined recognition of voice and gestures and lacks a real-time reliability correction mechanism for noise, lighting changes, object occlusion, hand overlap, accidental touches, and occupant posture deviations. CN108334871A discloses an interaction method for head-up display devices based on an intelligent cockpit platform, which controls the HUD display through facial expressions and actions. However, the control objects and interaction entry points are relatively singular, making it difficult to resolve object conflicts and permission conflicts when multiple occupants and multiple modalities input simultaneously. Therefore, there is an urgent need for an intelligent cockpit interaction control scheme that combines modal reliability, fuzzy referencing binding, object-level intent conflict mapping, and security permission arbitration. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent cockpit interaction control method and system based on multimodal perception, which solves the problems of insufficient reliability of modal recognition under environmental interference, difficulty in accurately binding fuzzy referencing target objects, difficulty in uniformly handling conflicts of multiple occupant interaction intentions, and lack of collaborative arbitration between vehicle operation risks and occupant permissions in existing intelligent cockpit interaction control.

[0004] This invention achieves the above objective through the following technical solution: an intelligent cockpit interaction control method based on multimodal perception, the method comprising:

[0005] Cockpit multimodal interaction data and vehicle operating status data are collected within the same sampling time window, and the issuing entity is marked for the multimodal interaction data;

[0006] The multimodal interaction data includes voice commands, gaze direction, head posture, gesture trajectory, touch operation, seat pressure, and occupant position;

[0007] The environmental interference factors affecting each interaction mode within the sampling time window are identified. Based on the interference mapping relationship between the environmental interference factors and the interaction modes, the initial identification confidence of each interaction mode is corrected to generate the real-time confidence of each interaction mode.

[0008] Semantic parsing is performed on the interactive instructions corresponding to voice commands or touch operations to extract control actions, object words, location words and pronouns, and to determine whether the interactive instructions are explicit object instructions or vague pronoun instructions;

[0009] When the interaction command is a specific object command, the target control object is determined from the cockpit object space library based on the object word;

[0010] When the interaction command is a vague reference command, the target control object is determined from the cockpit object space library based on the gaze direction, gesture trajectory, touch operation, current display interface and interaction history that meet the binding trust threshold.

[0011] Based on the issuing entity, target control object, control action, source of interaction modality, real-time reliability, vehicle operation risk level, and occupant permission level, construct object-level interaction intent nodes;

[0012] Based on object consistency, action compatibility, subject permission constraints, security risk constraints, and timing execution constraints among multiple object-level interaction intent nodes, an object-level intent conflict graph is generated.

[0013] The control arbitration outcome is determined based on the object-level intent conflict map, and the control arbitration outcome is direct execution, delayed execution, secondary confirmation, downgraded execution, or refusal to execute.

[0014] The cockpit control operation is executed according to the control arbitration result, and the target control object, control action, real-time reliability, control arbitration result and user confirmation result are written into the interaction history database.

[0015] Furthermore, cockpit multimodal interaction data and vehicle operating status data are collected within the same sampling time window, including:

[0016] Voice commands are collected via microphone array; gaze direction, head posture, gesture trajectory and occupant position are collected via in-vehicle camera or infrared sensing unit; touch operation is collected via touch screen, steering wheel button or seat control button; seat pressure is collected via seat pressure sensor; and vehicle operating status data is collected via vehicle bus.

[0017] Timestamps are added to voice commands, gaze direction, head posture, gesture trajectory, touch operation, seat pressure, occupant position and vehicle operating status data respectively;

[0018] Data whose timestamps fall within the same sampling time window are merged into a group of interactive samples, and the commanding entity of the interactive sample is determined based on the occupant's position, the direction of the sound source, and the seat pressure.

[0019] Furthermore, the vehicle operating status data includes:

[0020] Vehicle speed, steering angle, braking status, gear status, window status, air conditioning status, navigation status, and driver assistance status;

[0021] The vehicle operation risk level is determined based on vehicle operation status data, specifically as follows:

[0022] When the vehicle is in park and the speed is 0 km / h, it is classified as the first risk level.

[0023] When the vehicle is in driving gear and the speed does not exceed the low speed threshold, it is determined to be the second risk level;

[0024] When a vehicle is in driving gear and its speed exceeds the low speed threshold, or when any of the following conditions—braking status, steering angle, or driver assistance status—meets the risk triggering conditions, it is classified as a third-level risk.

[0025] The first risk level, the second risk level, and the third risk level each correspond to different cockpit control authority ranges.

[0026] Furthermore, the real-time credibility of each interaction modality is generated, including:

[0027] Establish an interference mapping relationship between environmental interference factors and interaction modes. The environmental interference factors include cabin noise, lighting changes, object occlusion, hand overlap, accidental touch, and occupant posture deviation.

[0028] Obtain initial recognition confidence scores for voice modality, gaze modality, gesture modality, touch modality, and posture modality;

[0029] The confidence correction coefficient for each interaction mode is determined based on the interference mapping relationship, and the initial recognition confidence of the corresponding interaction mode is corrected using the confidence correction coefficient to obtain the real-time confidence.

[0030] Interaction modalities with real-time credibility not lower than the execution credibility threshold are marked as execution-available modalities; interaction modalities with real-time credibility lower than the execution credibility threshold but not lower than the binding credibility threshold are marked as binding-available modalities; and interaction modalities with real-time credibility lower than the binding credibility threshold are marked as invalid modalities.

[0031] Furthermore, semantic parsing is performed on the interactive instructions corresponding to voice commands or touch operations, including:

[0032] The voice command is converted into text to obtain the voice text.

[0033] Perform control recognition on touch operations to obtain touch semantic text;

[0034] Perform word segmentation and intent recognition on voice text or touch semantic text to extract control actions, object words, location words and referential information;

[0035] When the semantic parsing result contains an object word and the object word matches an object identifier in the cockpit object space library, the interaction command is determined to be an explicit object command;

[0036] When the semantic parsing result contains any one of the following: near reference information, far reference information, location reference information, object substitution reference information, and historical object reference information, and no object word matching the object identifier in the cockpit object space library is identified, the interaction instruction is determined to be a fuzzy reference instruction.

[0037] When the semantic parsing results do not extract the control action, an invalid instruction flag is generated.

[0038] Furthermore, the cockpit object space library includes multiple cockpit object records, each cockpit object record including object identifier, three-dimensional position range, interface control identifier, set of executable actions, action safety level, and set of allowed command entities;

[0039] The cockpit objects include the left front window, right front window, left rear window, right rear window, sunroof, driver's seat, passenger seat, rear seats, air conditioning temperature control, air conditioning fan speed control, interior lighting control, audio control, HUD control, navigation interface control, and driver assistance interface control.

[0040] Furthermore, when the interactive command is a vague reference command, the target control object is determined from the cockpit object space library, including:

[0041] Based on the intersection relationship between the line of sight direction and the three-dimensional position range of each cockpit object, the line of sight anchoring result is generated;

[0042] Based on the intersection of the extension direction of the gesture trajectory and the three-dimensional position range of each cockpit object, the gesture projection result is generated;

[0043] Based on the interface control identifiers corresponding to the touch operation and the interface control identifiers in the cockpit object record, a touch hit result is generated.

[0044] Based on the currently displayed focus control and the target control object that was most recently successfully executed in the interaction history library, generate the interface focus result and the historical continuous result;

[0045] Based on the gaze anchoring results, gesture projection results, touch hit results, interface focus results, historical continuous results, and the real-time reliability of the corresponding interaction modality, an object binding value is generated for each candidate target object.

[0046] The candidate target object with the highest object binding value and an object binding value not lower than the object binding threshold is identified as the target control object;

[0047] An object conflict flag is generated when the difference between the candidate target object with the highest object binding value and the candidate target object with the second highest object binding value is lower than the object discrimination threshold.

[0048] Furthermore, an object-level intent conflict graph is generated, including:

[0049] Each object-level interaction intent node is treated as a graph node;

[0050] When two graph nodes have the same target control object and the same control action or belong to the same action direction, generate a confirmation edge in the same direction between the two graph nodes.

[0051] When two graph nodes originate from the same fuzzy referencing instruction and are bound to different target control objects, or when object conflict markers are generated, object conflict edges are generated.

[0052] When two graph nodes have the same target control object and opposite control actions or do not belong to the same action direction, an action conflict edge is generated.

[0053] When the issuing entity does not belong to the set of allowed issuing entities of the target controlled object, an entity permission edge is generated;

[0054] When the vehicle's operational risk level does not meet the target control object's action safety level, a safety inhibition edge is generated.

[0055] When the control actions of two graph nodes have a preceding action and a following action, a temporal dependency edge is generated.

[0056] Further, determining the control arbitration outcome based on the object-level intent conflict graph includes:

[0057] When there are no object conflict edges, action conflict edges, subject permission edges, or security inhibition edges in the object-level intent conflict graph, and the source of the interaction modality corresponding to the target control object belongs to the execution available modality, the direct execution result is output.

[0058] When there are unidirectional confirmation edges in the object-level intent conflict graph, and the graph nodes connected by the unidirectional confirmation edges correspond to the same target control object, the execution priority of the target control object is increased, and the direct execution result is output when there are no subject permission edges and security suppression edges.

[0059] When there are object conflict edges or action conflict edges in the object-level intent conflict graph, output the secondary confirmation result;

[0060] If no user confirmation is received within the confirmation waiting time, output the delayed execution result;

[0061] When there is a subject permission edge in the object-level intent conflict graph, output a rejection result, or output a downgraded execution result when the target controlled object has a low-permission alternative action.

[0062] When a safety-suppressing edge exists in the object-level intent conflict graph, output a rejection result, or output a delayed execution result after the vehicle operation risk level is reduced to an allowable level.

[0063] A multimodal perception-based intelligent cockpit interaction control system is applied to execute the aforementioned multimodal perception-based intelligent cockpit interaction control method. The system includes:

[0064] The multimodal data acquisition module is used to collect cockpit multimodal interaction data and vehicle operating status data according to the same sampling time window, and to mark the issuing entity for the multimodal interaction data;

[0065] The interference identification and credibility generation module is used to identify the environmental interference factors affecting each interaction modality within the sampling time window and generate the real-time credibility of each interaction modality.

[0066] The semantic parsing module is used to perform semantic parsing on the interactive instructions corresponding to voice commands or touch operations, and to determine whether the interactive instructions are explicit object instructions or vague reference instructions.

[0067] The object space management module is used to store the cockpit object space library and provides a query interface for cockpit object records;

[0068] The target object binding module is used to determine the target control object when the interaction instruction is a clear object instruction, and to determine the target control object based on the gaze direction, gesture trajectory, touch operation, current display interface and interaction history when the interaction instruction is a vague reference instruction.

[0069] The intent node building module is used to build object-level interactive intent nodes;

[0070] The conflict graph generation module is used to generate object-level intent conflict graphs.

[0071] The control arbitration module is used to determine the control arbitration outcome based on the object-level intent conflict graph;

[0072] The cockpit control execution module is used to execute cockpit control operations according to the control arbitration result and write the interaction information after execution into the interaction history database.

[0073] The beneficial effects of this invention are as follows:

[0074] 1. Before multimodal interaction data enters the control process, this invention identifies cabin noise, lighting changes, object occlusion, hand overlap, accidental touch, and occupant posture deviation. It also corrects the initial recognition confidence of each interaction modality based on environmental interference factors to obtain real-time reliability. This enables the system to dynamically determine the availability level of voice modality, gaze modality, gesture modality, touch modality, and posture modality in complex cabin environments, thereby improving the reliability of multimodal interaction information.

[0075] 2. This invention addresses ambiguous referential interaction commands such as "this," "that," "there," "it," and "that one just now." By combining gaze anchoring results, gesture projection results, touch hit results, current interface focus results, historical continuous results, and the cockpit object space library, it calculates the object binding value of candidate target objects. This enables the determination of the control object even when the target object is not explicitly stated, reducing the probability of miscontrol and repeated confirmation.

[0076] 3. This invention encapsulates the issuing entity, target control object, control action, source of interaction modality, real-time reliability, vehicle operation risk level, and occupant permission level into object-level interaction intent nodes, thereby elevating cockpit control from command-level recognition to object-level intent expression. This is beneficial for unified modeling of continuous interaction, multi-occupant interaction, and multi-modal interaction.

[0077] 4. This invention expresses unidirectional confirmation edges, object conflict edges, action conflict edges, subject authority edges, security inhibition edges, and temporal dependency edges through an object-level intent conflict graph. Based on the graph edge type, it outputs direct execution, secondary confirmation, delayed execution, degraded execution, or rejection results, which can achieve a balance between interaction accuracy, response continuity, and vehicle operation safety.

[0078] 5. This invention writes the target control object, control action, real-time reliability, control arbitration result, and user confirmation result into the interaction history database, enabling subsequent continuous interactions to use historical records for omitted object parsing and fuzzy referencing binding, thereby improving the continuity and personalized adaptability of intelligent cockpit interaction. Attached Figure Description

[0079] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0080] Figure 1 This is a diagram showing the overall system structure of the present invention;

[0081] Figure 2 This is a flowchart illustrating the overall process of the method of the present invention.

[0082] Figure 3This is a flowchart of the multimodal acquisition and credibility generation process of the present invention;

[0083] Figure 4 This is a flowchart illustrating the fuzzy referencing target object binding process of the present invention;

[0084] Figure 5 This is a flowchart of the object-level intent conflict map and control arbitration process of the present invention. Detailed Implementation

[0085] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0086] Example 1:

[0087] Please see Figures 1 to 5 This invention provides an intelligent cockpit interaction control method based on multimodal perception. This method is applied to intelligent vehicle cockpit domain controllers, vehicle-machine interaction control platforms, or vehicle edge computing units. When the driver, front passenger, or rear passenger issues cockpit control intentions through voice, line of sight, gesture, touch, or posture, the method combines the vehicle's operating status, cockpit environmental interference, the authority of the issuing entity, and the security level of the cockpit object to determine the target control object and output a safe and executable control result.

[0088] In this embodiment, the intelligent cockpit interaction control system uses the vehicle cockpit coordinate system as a unified spatial reference. The vehicle cockpit coordinate system has its origin at the center point of the front edge of the driver's seat, with the vehicle's forward direction as the positive X-axis, the left-side direction as the positive Y-axis, and the vehicle's height as the positive Z-axis. The spatial unit is uniformly mm. The left front window, right front window, left rear window, right rear window, sunroof, driver's seat, passenger seat, rear seats, air conditioning temperature control, air conditioning fan speed control, interior lighting control, audio control, HUD control, navigation interface control, and driver assistance interface control in the cockpit object space library are all stored as three-dimensional bounded regions. These three-dimensional bounded regions are determined by the object's center coordinates, object length, object width, and object height.

[0089] During the data acquisition phase, the system collects cockpit multimodal interaction data and vehicle operating status data within the same sampling time window, and labels the issuing entity for the multimodal interaction data. Multimodal interaction data includes voice commands, gaze direction, head posture, gesture trajectory, touch operations, seat pressure, and occupant position. Vehicle operating status data includes vehicle speed, steering angle, braking status, gear status, window status, air conditioning status, navigation status, and driver assistance status.

[0090] Specifically, voice commands are collected by a microphone array with a sampling rate set to 16 kHz; gaze direction, head posture, gesture trajectory, and occupant position are collected by an in-vehicle camera or infrared sensing unit with a frame rate set to 30 fps; seat pressure is collected by a seat pressure sensor with a sampling frequency set to 20 Hz; touch operations are collected by the touchscreen, steering wheel buttons, or seat control buttons, and touch events are recorded with millisecond-level timestamps; vehicle operating status data is read through the vehicle bus with a refresh cycle set to 100 ms. All data from these different sources are aligned according to timestamps within the same sampling time window, with the timestamp unit uniformly set to milliseconds.

[0091] In this embodiment, the sampling time window is set to 500 ms, and the sliding step size between adjacent sampling time windows is set to 250 ms. This parameter is determined based on engineering rules that the cockpit control response time should not exceed 1 s, the voice tail detection delay should not exceed 300 ms, and the synchronization error between gaze and gesture recognition should not exceed 200 ms. If the vehicle platform has low computing power, the sampling time window can be adjusted to 800 ms; if the vehicle platform deploys a high-performance cockpit domain controller, the sampling time window can be adjusted to 300 ms.

[0092] The system merges voice commands, gaze direction, head posture, gesture trajectory, touch operation, seat pressure, occupant position, and vehicle operating status data that fall within the same sampling time window into a set of interaction samples. It then determines the issuing entity of the interaction sample based on occupant position, sound source direction, and seat pressure. When the sound source direction located by the microphone array corresponds to the driver's area, and the driver's seat pressure is greater than the occupant threshold, the issuing entity is identified as the driver; when the sound source direction corresponds to the front passenger area, and the front passenger seat pressure is greater than the occupant threshold, the issuing entity is identified as the front passenger; and when the sound source direction corresponds to the rear passenger area, and the rear seat pressure is greater than the occupant threshold, the issuing entity is identified as the rear passenger. The occupant threshold is set to 80 N, which is determined based on the seat unloaded pressure, the minimum effective pressure for child occupants, and the zero-point drift range of the seat pressure sensor.

[0093] During the environmental interference identification phase, the system identifies environmental interference factors affecting each interaction modality within the sampling time window and establishes an interference mapping relationship between these factors and the interaction modality. Environmental interference factors include cabin noise, lighting changes, object occlusion, hand overlap, accidental touches, and occupant posture shifts. Interaction modalities include voice modality, gaze modality, gesture modality, touch modality, and posture modality.

[0094] Specifically, for the voice modality, the system detects cabin noise intensity, music playback status, wind noise from open windows, and simultaneous speech by multiple people; for the gaze modality, the system detects changes in lighting, glare, eyeglass occlusion, and head tilt; for the gesture modality, the system detects overlapping hands, interrupted gesture trajectories, and gestures leaving the recognition area; for the touch modality, the system detects continuous touch, touch drift, and touch on non-target areas; and for the posture modality, the system detects occupant posture deviation, body occlusion, and abnormal seat pressure.

[0095] To avoid dimension inconsistencies caused by different physical quantities directly participating in calculations, this embodiment first performs dimensionless normalization on environmental interference factors. For any environmental interference factor, its normalized interference intensity is determined according to the following formula:

[0096]

[0097] In the formula, Indicates the first The normalized interference intensity of environmental interference factors, with a value range of [value range missing]. to , which is a dimensionless quantity; Indicates the first Measured values ​​of environmental interference factors; Indicates the first Reference values ​​for environmental disturbance factors under normal cabin conditions; Indicates the first Environmental interference factors allow deviations from the calibration range of the baseline value. Because... , and Using the same units, the above formula yields... It is a dimensionless quantity.

[0098] Specifically, cabin noise Units are benchmark value Pick Calibration range Pick Light intensity Units are benchmark value Pick Calibration range Pick Object occlusion is represented by the proportion of the occlusion area, with a value range of [value missing]. to The overlap of hands is represented by the proportion of the overlapping area, with a value ranging from [value range missing]. to Accidental touches are represented by touch offset distance in mm, with a baseline value of 0 mm and a calibration range of 30 mm. Occupant posture offset is represented by head center offset distance in mm, with a baseline value of 0 mm and a calibration range of 150 mm. The above baseline values ​​and calibration ranges can be determined through cabin calibration tests before mass production of the vehicle. The calibration samples cover daytime, nighttime, window-open, music playback, multi-person seating, and occupant height scenarios.

[0099] In the real-time confidence generation stage, the system first obtains the initial recognition confidence of each interaction modality, and then corrects it according to the influence of corresponding environmental interference factors. This embodiment does not use the method of adding multiple weights, but instead uses the maximum interference suppression method to reduce the influence of manual weight setting on the results. The real-time confidence of each interaction modality is determined according to the following formula:

[0100]

[0101] In the formula, Indicates the first The real-time reliability of each interaction modality, with a value range of [value range missing]. to , which is a dimensionless quantity; Indicates the first The initial recognition confidence score for each interaction modality is output by the corresponding recognition model, and its value ranges from [value range missing]. to ; Indicates the impact of the first A set of environmental interference factors for each interaction modality; The first expression represents the first expression obtained from the previous expression. Normalized interference intensity of similar environmental interference factors; Indicates the impact of the first The maximum interference intensity for each interaction modality. The rationale for adopting the maximum interference suppression method is that when a key interference factor in cockpit interaction reaches a high level, the reliability of single interaction modality recognition will significantly decrease. For example, strong noise will significantly affect the voice modality, and strong occlusion will significantly affect the line-of-sight modality.

[0102] In this embodiment, a trusted threshold is applied. Set as Binding trusted threshold Set as Real-time reliability Not less than At that time, mark the interaction modality as an executable available modality; real-time reliability Below and not less than At that time, mark the interaction modality as a bound available modality; real-time reliability Below If this occurs, mark the interaction modality as invalid. Apply the trust threshold. and binding trust threshold Determined jointly by the false execution rate and the object misbinding rate: In the calibration dataset, the false execution rate of directly executed class instructions should not exceed [a certain value]. The minimum confidence level is used as the execution confidence threshold; ensuring that the accuracy of binding fuzzy referential objects is not lower than... The minimum level of confidence is used as the binding confidence threshold. All of the above thresholds are dimensionless values.

[0103] During the semantic parsing phase, the system performs semantic parsing on the interactive instructions corresponding to voice commands or touch operations to determine whether the interactive instructions are explicit object instructions or vague referential instructions. For voice commands, the system first performs speech-to-text conversion to obtain speech text; for touch operations, the system identifies the interface controls corresponding to the touch location to obtain touch semantic text. The system then performs word segmentation and intent recognition on the speech text or touch semantic text, extracting control actions, object words, location words, and referential words.

[0104] Control actions include opening, closing, raising, lowering, switching, confirming, canceling, and adjusting to a specified state. When the semantic parsing result contains an object word that matches an object identifier in the cockpit object space library, the system identifies the interaction command as an explicit object command. For example, in "open the left rear window," the object word is "left rear window," and the control action is "open," so the system directly identifies the left rear window as a candidate target control object. When the semantic parsing result contains any of the pronouns "this," "that," "there," "it," or "that one," and no object word matching an object identifier in the cockpit object space library is identified, the system identifies the interaction command as a vague pronoun. For example, "lower this a little" does not explicitly specify the target object; object binding needs to be completed by combining gaze direction, gesture trajectory, touch operation, current display interface, and interaction history. When the semantic parsing result does not extract a control action, the system generates an invalid command flag and outputs a rejection result or a secondary confirmation result.

[0105] In this embodiment, the cockpit object space library includes multiple cockpit object records. Each cockpit object record includes an object identifier, a three-dimensional position range, an interface control identifier, a set of executable actions, an action safety level, and a set of allowed command-issuing entities. Cockpit objects include the left front window, right front window, left rear window, right rear window, sunroof, driver's seat, passenger seat, rear seats, air conditioning temperature control, air conditioning fan speed control, interior lighting control, and audio control. Controls, navigation interface controls, and driver assistance interface controls. Object identifiers are used to match object words in the semantic parsing results; three-dimensional position ranges are used for spatial matching with line-of-sight direction and gesture trajectory; interface control identifiers are used to match touch operations or the currently displayed interface; the set of executable actions is used to determine whether the control action is executable; the action safety level is used to match with the vehicle operation risk level; and the set of allowed command entities is used to determine whether the driver, front passenger, or rear passenger has control authority.

[0106] When the interaction command is a explicitly defined object command, the system determines the target control object from the cockpit object space library based on the object term. When the interaction command is a vaguely descriptive command, the system determines the target control object from the cockpit object space library based on the gaze direction, gesture trajectory, touch operation, current display interface, and interaction history that meet the binding trust threshold.

[0107] During the fuzzy referential object binding phase, the system generates an object binding value for each candidate cockpit object. Candidate cockpit objects come from the cockpit object space library, and their set of executable actions includes the control actions obtained from the current semantic parsing. The system uses gaze anchoring results, gesture projection results, touch hit results, current interface focus results, and historical continuous results as object binding evidence, and calculates the object binding value using an evidence accumulation method instead of a simple weighted summation method.

[0108] Specifically, the system generates line-of-sight anchoring results based on the intersection of the line-of-sight direction and the three-dimensional position range of each cockpit object; generates gesture projection results based on the intersection of the extension direction of the gesture trajectory and the three-dimensional position range of each cockpit object; generates touch hit results based on the interface control identifiers corresponding to the touch operation and the interface control identifiers in the cockpit object record; generates interface focus results based on the focus control of the currently displayed interface; and generates historical continuous results based on the most recently successfully executed target control object in the interaction history database. The object binding value of each candidate target object is determined according to the following formula:

[0109]

[0110] In the formula, Indicates candidate target object The object binding value can take the following range: to , which is a dimensionless quantity; Indicates pointing to the candidate target object A valid set of binding evidence; This refers to a type of binding evidence, specifically including eye contact evidence, gesture evidence, touch evidence, interface focus evidence, and historical continuity evidence; Present evidence Real-time reliability of the corresponding interaction modality; Present evidence With candidate target objects Spatial or semantic matching degree between them, with a value range of 100%. to It is a dimensionless quantity. Regarding visual evidence and gesture evidence, The distance is determined based on the shortest distance between the line of sight or gesture projection line and the three-dimensional bounding area of ​​the candidate object; the smaller the distance, the better. The higher the ray, the better; when the ray passes through the three-dimensional enclosed region, Pick For touch evidence, when the touch control identifier matches the interface control identifier of the candidate object, Pick Otherwise take Regarding interface focus evidence, when the currently displayed focus control is a candidate object, Pick Otherwise take For continuous historical evidence, when the candidate object is the target object that was most recently successfully executed in the interaction history library, and the time interval between the current instruction and the previous instruction does not exceed [a certain value], [the following criteria apply]. hour, Pick Otherwise take .

[0111] Object binding threshold Set as object differentiation threshold Set as The rules for determining the target control object are as follows:

[0112]

[0113] In the formula, This indicates the final target control object; Indicates the highest object binding value; Indicates the second highest object binding value; Indicates the object binding threshold; This represents the threshold for distinguishing objects. If... Below This indicates insufficient valid binding evidence, and the system outputs a secondary confirmation result; if and The difference between them is lower than This indicates that the two candidate target objects cannot be reliably distinguished, and the system generates an object conflict flag. Object binding threshold. Distinguishing threshold between objects The cabin object binding calibration test determined the following rule: in the calibration dataset, the accuracy of fuzzy referencing binding should not be lower than [the specified standard]. The secondary confirmation rate is no higher than The threshold combinations are used as candidate threshold combinations, and then the group with the shortest average response delay is selected from the candidate threshold combinations as the final threshold combination.

[0114] During the vehicle operation risk level determination phase, the system does not perform a weighted summation of vehicle speed, steering angle, braking status, and driver assistance status. Instead, it uses a tiered rule for judgment to avoid mixing calculations of different dimensions. Specifically, when the gear is park and the vehicle speed is... At that time, the vehicle's operational risk level is Level 1; when the gear is driving and the vehicle speed is greater than... and not exceeding And the absolute value of the steering wheel angle is less than And the normalized braking pressure is less than And the driver assistance warning sign is At that time, the vehicle's operational risk level was the second risk level; when the vehicle speed exceeded... Or the absolute value of the steering wheel angle is not less than Or the normalized braking pressure is not less than Or the driver assistance warning sign is At that time, the vehicle's operational risk level was Level 3. The unit of vehicle speed was... The steering wheel angle is in degrees, and the normalized braking pressure is... to The dimensionless value, the driving assistance warning sign is or Low speed threshold Determined based on cockpit control safety requirements for low-speed parking, park driving, and congested following scenarios; steering angle threshold. Determined based on vehicle sharp turn recognition requirements; normalized braking pressure threshold. Determined based on the requirements for identifying moderate to high braking conditions.

[0115] During the occupant access level determination phase, the driver corresponds to the first access level, the front passenger to the second access level, and the rear passengers to the third access level. The first access level allows control of the left front window, right front window, left rear window, right rear window, sunroof, driver's seat, front passenger seat, rear seats, air conditioning temperature controls, air conditioning fan speed controls, interior lighting controls, and audio controls. The first level of access allows control over the right front window, passenger seat, air conditioning temperature control, air conditioning fan speed control, interior lighting control, audio control, and navigation interface control; the second level of access allows control over the left rear window, right rear window, rear seats, air conditioning fan speed control, interior lighting control, and audio control. If the issuing entity does not belong to the set of allowed issuing entities for the target control object, the system generates a subject access permission edge in the object-level intent conflict graph.

[0116] During the object-level interaction intent node construction phase, the system constructs object-level interaction intent nodes based on the issuing entity, target control object, control action, interaction modality source, real-time reliability, vehicle operation risk level, and occupant permission level. Each object-level interaction intent node includes at least a node number, issuing entity, target control object, control action, interaction modality source, real-time reliability, vehicle operation risk level, occupant permission level, and timestamp. The node number is generated by combining the timestamp and the target object identifier to distinguish different control intents within the same sampling time window. If multiple object-level interaction intent nodes exist within the same sampling time window, or if object-level interaction intent nodes with temporal correlation exist within consecutive sampling time windows, the system enters the object-level intent conflict graph generation process.

[0117] During the object-level intent conflict graph generation phase, the system uses each object-level interactive intent node as a graph node and generates graph edges based on object consistency, action compatibility, subject permission constraints, security risk constraints, and timing execution constraints among multiple object-level interactive intent nodes.

[0118] When two graph nodes have the same target control object and the same control action or belong to the same action direction, a same-direction confirmation edge is generated between the two graph nodes; when two graph nodes originate from the same fuzzy reference instruction and are bound to different target control objects, or when object conflict markers are generated according to the aforementioned target control object determination rules, an object conflict edge is generated; when two graph nodes have the same target control object and opposite control actions or do not belong to the same action direction, an action conflict edge is generated; when the issuing entity does not belong to the set of allowed issuing entities of the target control object, an entity permission edge is generated; when the vehicle operation risk level does not meet the action safety level of the target control object, a safety suppression edge is generated; when the control actions of two graph nodes have pre-action and post-action, a time-series dependency edge is generated.

[0119] During the control arbitration phase, the system determines the control arbitration result based on the object-level intent conflict graph. Control arbitration results include direct execution, delayed execution, secondary confirmation, degraded execution, and refusal to execute. When there are no object conflict edges, action conflict edges, subject authority edges, or security suppression edges in the object-level intent conflict graph, and the interaction modality source corresponding to the target control object belongs to an executable available modality, the system outputs a direct execution result. When there are same-direction confirmation edges in the object-level intent conflict graph, and the graph nodes connected by these same-direction confirmation edges correspond to the same target control object, the system increases the execution priority of that target control object and outputs a direct execution result when there are no subject authority edges or security suppression edges. When there are object conflict edges or action conflict edges in the object-level intent conflict graph, the system outputs a secondary confirmation result, with the confirmation waiting time set to [value missing]. ;like If no user confirmation is received, the system outputs a delayed execution result. When a principal permission edge exists in the object-level intent conflict graph, the system outputs a rejection execution result; if the target controlled object has a low-privilege alternative action, a downgraded execution result is output. When a safety suppression edge exists in the object-level intent conflict graph, the system outputs a rejection execution result; if the vehicle operation risk level is within... If the risk level is reduced to the allowable level, the output will be delayed, and the process will be re-arbitrated after the risk level is reduced.

[0120] During the cockpit control execution phase, the system executes the corresponding cockpit control operation according to the control arbitration result, and writes the target control object, control action, real-time reliability, control arbitration result, and user confirmation result into the interaction history database. The interaction history database is used to support the parsing of subsequent continuous interaction commands and the binding of fuzzy reference objects. For example, if the user first issues "open the left rear window," the system will successfully execute it and record the target control object as the left rear window; if the user then issues "close it a little smaller," the system will bind the omitted object to the left rear window based on the most recent successful execution record, and determine whether to execute it based on the vehicle operation risk level and occupant permission level.

[0121] In a specific application example, the vehicle is at the highest risk level, and the driver issues the voice command "Turn this down a bit." Within the same sampling time window, the system detects that the driver's gaze is directed towards the air conditioning temperature controls, and the gesture trajectory is towards the air conditioning area on the central control screen; the current focus of the displayed interface is the air conditioning interface. The initial confidence level for voice modality recognition is [insert confidence level here]. The cabin noise is According to the normalized interference intensity calculation rule, the normalized interference intensity of the noise is obtained as follows: The real-time credibility of the speech modality is obtained according to the real-time credibility generation rules. It belongs to the bound available modality; the real-time reliability of the gaze modality is... This belongs to the available execution modalities; the real-time reliability of the gesture modal is... This belongs to the available execution modality. The system calculates the object binding value of the air conditioner temperature control. The highest object binding value of other candidate objects is The system identifies the target control object as the air conditioner temperature control. After constructing the object-level interactive intent node, it does not generate object conflict edges, action conflict edges, subject permission edges, or security suppression edges. Ultimately, it outputs the direct execution result and lowers the air conditioner temperature. .

[0122] In another specific application example, the vehicle is in the third risk level, and a rear passenger issues the voice command "Turn off driver assistance." The system identifies the driver assistance interface controls as the target control object and "turn off" as the control action. Since the rear passenger is not part of the set of allowed command subjects for the driver assistance interface controls, and the third risk level does not allow the action of turning off driver assistance, the system generates subject permission edges and safety inhibition edges in the object-level intent conflict graph, and the control arbitration result is a refusal to execute. The system outputs a message to the user "This operation is not allowed in the current state" and writes the refusal result to the interaction history database.

[0123] Example 2:

[0124] Based on Example 1, this embodiment further provides a processing procedure for a co-driver issuing a vague reference instruction at low speed, which is used to illustrate how the present invention achieves secure arbitration through real-time credibility, object binding value and intent conflict graph when multimodal evidence is not completely consistent.

[0125] In this embodiment, the vehicle is in driving gear, with a speed of 12 km / h, an absolute steering wheel angle of 18°, a normalized braking pressure of 0.12, and a driver assistance warning sign of 0. Since the vehicle speed is greater than 0 km / h but does not exceed 15 km / h, the absolute steering wheel angle is less than 45°, the normalized braking pressure is less than 0.4, and the driver assistance warning sign is 0, the system determines the vehicle's operational risk level to be the second risk level.

[0126] Within the same 500 ms sampling time window, the co-pilot issued the voice command "Open that area a little." The microphone array identified the sound source as falling into the co-pilot's area, and the co-pilot's seat pressure was 360 N, exceeding the occupant threshold of 80 N. The system identified the sender as the co-pilot. The semantic parsing module performed text conversion and word segmentation on the voice command, extracting the control action as "open" and the pronoun as "there," but no explicit object word matching the object identifier in the cockpit object space library was extracted. Therefore, the interaction command was determined to be a fuzzy referential command.

[0127] The system further collects and analyzes multimodal information within the same sampling time window. The line of sight falls into the three-dimensional surrounding area of ​​the right front window, the extension direction of the gesture trajectory is close to the three-dimensional surrounding area of ​​the sunroof, the current focus control of the central control display interface is the window control interface, the target object of the most recent successful execution in the interaction history library is the right front window, and the time interval between the current command and the current command is 3 seconds.

[0128] During the environmental interference identification phase, the system detected a cabin noise level of 50 dB, an illumination intensity of 800 lux, and slight hand overlap in the gesture area, with an overlap area ratio of 0.35. According to the normalized interference intensity calculation rules, the noise normalized interference intensity corresponding to the speech modality was 0.17, the illumination normalized interference intensity corresponding to the gaze modality was 0.12, and the hand overlap normalized interference intensity corresponding to the gesture modality was 0.35. The initial recognition confidence scores were 0.88 for the speech modality, 0.90 for the gaze modality, and 0.86 for the gesture modality. Following the real-time reliability correction rules, the real-time reliability scores were 0.74 for the speech modality, 0.80 for the gaze modality, and 0.61 for the gesture modality.

[0129] Since the execution confidence threshold is 0.75 and the binding confidence threshold is 0.55, the system marks the gaze modality as an execution-available modality and the voice and gesture modalities as binding-available modalities. At this point, although the voice modality is insufficient as a direct basis for execution, it can serve as a basis for semantic parsing and object binding; the gesture modality is affected by hand overlap, resulting in decreased confidence, and is only used as auxiliary binding evidence.

[0130] During the target object binding phase, the system selects candidate objects from the cockpit object space library that can perform the "open" action, including the right front window, sunroof, interior lighting controls, and audio controls. The spatial matching degree between line-of-sight evidence and the right front window is 1; the spatial matching degree between gesture evidence and the sunroof is 0.72; the semantic matching degree between the current interface focus evidence and the right front window is 0.90; and the semantic matching degree between historical continuous evidence and the right front window is 1. Based on the object binding value calculation rules, the system obtains an object binding value of 0.94 for the right front window, 0.44 for the sunroof, 0.18 for the interior lighting controls, and 0.10 for the audio controls.

[0131] Since the object binding value of the right front window (0.94) is higher than the object binding threshold of 0.65, and the difference in object binding value between the right front window and the second highest candidate object (sunroof) is 0.50, which is higher than the object differentiation threshold of 0.15, the system identifies the right front window as the target control object and does not generate an object conflict flag.

[0132] Subsequently, the system constructs an object-level interaction intent node. In this node, the issuing entity is the front passenger, the target control object is the right front window, the control action is to open it, and the interaction modal sources include voice modality, gaze modality, gesture modality, interface focus evidence, and historical continuous evidence. The main execution basis corresponding to the target object is gaze modality, the vehicle operation risk level is the second risk level, and the occupant permission level is the second permission level.

[0133] During the object-level intent conflict graph generation phase, the system determined that the front passenger occupant belonged to the set of permitted entities for issuing commands to the right front window, and no entity permission edge was generated; the second risk level allowed the partial opening of the right front window, and no safety suppression edge was generated; no other occupant issued opposing control actions to the right front window within the current sampling time window, and no action conflict edge was generated; the object binding result for the right front window was clear, and no object conflict edge was generated. Therefore, the object-level intent conflict graph does not contain object conflict edges, action conflict edges, entity permission edges, or safety suppression edges.

[0134] During the arbitration phase, the system outputs direct execution results based on the object-level intent conflict map. Considering the vehicle is in the second risk level, the system does not execute the full window opening operation. Instead, according to the second risk level control strategy corresponding to the right front window in the cabin object space library, it maps "opening slightly" to a window drop height of 30 mm. This drop height is set based on wind noise control requirements and occupant comfort requirements under low-speed driving conditions, and falls within the permissible control range for the right front window under the second risk level.

[0135] After execution, the system writes the interaction record to the interaction history database. The record includes: the issuing entity is the front passenger, the target control object is the right front window, the control action is to lower it by 30 mm, the real-time reliability of the voice modality is 0.74, the real-time reliability of the eye-tracking modality is 0.80, the real-time reliability of the gesture modality is 0.61, the control arbitration result is direct execution, and the user confirmation result is no confirmation required. This record is used for subsequent continuous interaction parsing; for example, if the front passenger then says "open it a little more" within 5 seconds, the system can combine the interaction history database to continue binding the omitted object to the right front window, and re-determine whether to execute based on the vehicle's operational risk level and the passenger's permission level.

[0136] As can be seen from this embodiment, even though the three modalities of voice, gaze, and gesture have different levels of credibility in a complex cockpit environment, and the gesture evidence and gaze evidence do not point to the same direction, the system can still determine the credible target control object through real-time credibility after environmental interference correction, object binding value, object differentiation threshold, and object-level intent conflict graph, and execute cockpit control under the premise of satisfying permission and security constraints, thereby improving the control accuracy and security in fuzzy referential interaction scenarios.

[0137] Based on the same inventive concept, this embodiment also provides an intelligent cockpit interaction control system based on multimodal perception. For example... Figure 1 As shown, the system includes a multimodal data acquisition module, an interference identification and credibility generation module, a semantic parsing module, an object space management module, a target object binding module, an intent node construction module, a conflict graph generation module, a control arbitration module, and a cockpit control execution module.

[0138] The multimodal data acquisition module collects cockpit multimodal interaction data and vehicle operating status data within the same sampling time window, and labels the issuing entity for the multimodal interaction data. The interference identification and credibility generation module identifies environmental interference factors affecting each interaction modality within the sampling time window and generates the real-time credibility of each interaction modality. The semantic parsing module performs semantic parsing on the interaction commands corresponding to voice commands or touch operations, determining whether the interaction command is an explicit object command or a vague reference command. The object space management module stores the cockpit object space library and provides a query interface for cockpit object records. The target object binding module determines the target control object when the interaction command is an explicit object command, and determines the target control object based on gaze direction, gesture trajectory, touch operation, current display interface, and interaction history when the interaction command is a vague reference command. The intent node construction module constructs object-level interaction intent nodes. The conflict graph generation module generates an object-level intent conflict graph. The control arbitration module determines the control arbitration result based on the object-level intent conflict graph. The cockpit control execution module is used to execute cockpit control operations according to the control arbitration result and write the interaction information after execution into the interaction history database.

[0139] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A smart cockpit interaction control method based on multimodal perception, characterized in that, The method includes: Cockpit multimodal interaction data and vehicle operating status data are collected within the same sampling time window, and the issuing entity is marked for the multimodal interaction data; The multimodal interaction data includes voice commands, gaze direction, head posture, gesture trajectory, touch operation, seat pressure, and occupant position; The environmental interference factors affecting each interaction mode within the sampling time window are identified. Based on the interference mapping relationship between the environmental interference factors and the interaction modes, the initial identification confidence of each interaction mode is corrected to generate the real-time confidence of each interaction mode. Semantic parsing is performed on the interactive instructions corresponding to voice commands or touch operations to extract control actions, object words, location words and pronouns, and to determine whether the interactive instructions are explicit object instructions or vague pronoun instructions; When the interaction command is a specific object command, the target control object is determined from the cockpit object space library based on the object word; When the interaction command is a vague reference command, the target control object is determined from the cockpit object space library based on the gaze direction, gesture trajectory, touch operation, current display interface and interaction history that meet the binding trust threshold. Based on the issuing entity, target control object, control action, source of interaction modality, real-time reliability, vehicle operation risk level, and occupant permission level, construct object-level interaction intent nodes; Based on object consistency, action compatibility, subject permission constraints, security risk constraints, and timing execution constraints among multiple object-level interaction intent nodes, an object-level intent conflict graph is generated. The control arbitration result is determined based on the object-level intent conflict map, and the control arbitration result is direct execution, delayed execution, secondary confirmation, downgraded execution, or refusal to execute. The cockpit control operation is executed according to the control arbitration result, and the target control object, control action, real-time reliability, control arbitration result and user confirmation result are written into the interaction history database.

2. The intelligent cockpit interaction control method based on multimodal perception according to claim 1, characterized in that, Cockpit multimodal interaction data and vehicle operating status data were collected within the same sampling time window, including: Voice commands are collected via microphone array; gaze direction, head posture, gesture trajectory and occupant position are collected via in-vehicle camera or infrared sensing unit; touch operation is collected via touch screen, steering wheel button or seat control button; seat pressure is collected via seat pressure sensor; and vehicle operating status data is collected via vehicle bus. Timestamps are added to voice commands, gaze direction, head posture, gesture trajectory, touch operation, seat pressure, occupant position and vehicle operating status data respectively; Data whose timestamps fall within the same sampling time window are merged into a group of interactive samples, and the commanding entity of the interactive sample is determined based on the occupant's position, the direction of the sound source, and the seat pressure.

3. The intelligent cockpit interaction control method based on multimodal perception according to claim 2, characterized in that, The vehicle operating status data includes: Vehicle speed, steering angle, braking status, gear status, window status, air conditioning status, navigation status, and driver assistance status; The vehicle operation risk level is determined based on vehicle operation status data, specifically as follows: When the vehicle is in park and the speed is 0 km / h, it is classified as the first risk level. When the vehicle is in driving gear and the speed does not exceed the low speed threshold, it is classified as the second risk level. When a vehicle is in driving gear and its speed exceeds the low speed threshold, or when any of the following conditions—braking status, steering angle, or driver assistance status—meets the risk triggering conditions, it is classified as a third-level risk. The first risk level, the second risk level, and the third risk level each correspond to different cockpit control authority ranges.

4. The intelligent cockpit interaction control method based on multimodal perception according to claim 3, characterized in that, Generate real-time credibility for each interaction modality, including: Establish an interference mapping relationship between environmental interference factors and interaction modes. The environmental interference factors include cabin noise, lighting changes, object occlusion, hand overlap, accidental touch, and occupant posture deviation. Obtain initial recognition confidence scores for voice modality, gaze modality, gesture modality, touch modality, and posture modality; The confidence correction coefficient for each interaction mode is determined based on the interference mapping relationship, and the initial recognition confidence of the corresponding interaction mode is corrected using the confidence correction coefficient to obtain the real-time confidence. Interaction modalities with real-time credibility not lower than the execution credibility threshold are marked as execution-available modalities; interaction modalities with real-time credibility lower than the execution credibility threshold but not lower than the binding credibility threshold are marked as binding-available modalities; and interaction modalities with real-time credibility lower than the binding credibility threshold are marked as invalid modalities.

5. The intelligent cockpit interaction control method based on multimodal perception according to claim 4, characterized in that, Semantic parsing of interactive instructions corresponding to voice commands or touch operations, including: The voice command is converted into text to obtain the voice text. Perform control recognition on touch operations to obtain touch semantic text; Perform word segmentation and intent recognition on voice text or touch semantic text to extract control actions, object words, location words and referential information; When the semantic parsing result contains an object word and the object word matches an object identifier in the cockpit object space library, the interaction command is determined to be an explicit object command; When the semantic parsing result contains any one of the following: near reference information, far reference information, location reference information, object substitution reference information, and historical object reference information, and no object word matching the object identifier in the cockpit object space library is identified, the interaction instruction is determined to be a fuzzy reference instruction. When the semantic parsing results do not extract the control action, an invalid instruction flag is generated.

6. The intelligent cockpit interaction control method based on multimodal perception according to claim 5, characterized in that: The cockpit object space library includes multiple cockpit object records. Each cockpit object record includes an object identifier, a three-dimensional position range, an interface control identifier, a set of executable actions, an action safety level, and a set of allowed commanding entities. The cockpit objects include the left front window, right front window, left rear window, right rear window, sunroof, driver's seat, passenger seat, rear seats, air conditioning temperature control, air conditioning fan speed control, interior lighting control, audio control, HUD control, navigation interface control, and driver assistance interface control.

7. The intelligent cockpit interaction control method based on multimodal perception according to claim 6, characterized in that, When the interaction command is a vague reference command, the target control object is determined from the cockpit object space library, including: Based on the intersection relationship between the line of sight direction and the three-dimensional position range of each cockpit object, the line of sight anchoring result is generated; Based on the intersection of the extension direction of the gesture trajectory and the three-dimensional position range of each cockpit object, the gesture projection result is generated; Based on the interface control identifiers corresponding to the touch operation and the interface control identifiers in the cockpit object record, a touch hit result is generated. Based on the currently displayed focus control and the target control object that was most recently successfully executed in the interaction history library, generate the interface focus result and the historical continuous result; Based on the gaze anchoring results, gesture projection results, touch hit results, interface focus results, historical continuous results, and the real-time reliability of the corresponding interaction modality, an object binding value is generated for each candidate target object. The candidate target object with the highest object binding value and an object binding value not lower than the object binding threshold is identified as the target control object; An object conflict flag is generated when the difference between the candidate target object with the highest object binding value and the candidate target object with the second highest object binding value is lower than the object discrimination threshold.

8. The intelligent cockpit interaction control method based on multimodal perception according to claim 7, characterized in that, Generate an object-level intent conflict graph, including: Each object-level interaction intent node is treated as a graph node; When two graph nodes have the same target control object and the same control action or belong to the same action direction, generate a confirmation edge in the same direction between the two graph nodes. When two graph nodes originate from the same fuzzy referencing instruction and are bound to different target control objects, or when object conflict markers are generated, object conflict edges are generated. When two graph nodes have the same target control object and opposite control actions or do not belong to the same action direction, an action conflict edge is generated. When the issuing entity does not belong to the set of allowed issuing entities of the target controlled object, an entity permission edge is generated; When the vehicle's operational risk level does not meet the target control object's action safety level, a safety inhibition edge is generated. When the control actions of two graph nodes have a preceding action and a following action, a temporal dependency edge is generated.

9. The intelligent cockpit interaction control method based on multimodal perception according to claim 8, characterized in that, Determining the control arbitration outcome based on the object-level intent conflict map includes: When there are no object conflict edges, action conflict edges, subject permission edges, or security inhibition edges in the object-level intent conflict graph, and the source of the interaction modality corresponding to the target control object belongs to the execution available modality, the direct execution result is output. When there are unidirectional confirmation edges in the object-level intent conflict graph, and the graph nodes connected by the unidirectional confirmation edges correspond to the same target control object, the execution priority of the target control object is increased, and the direct execution result is output when there are no subject permission edges and security suppression edges. When there are object conflict edges or action conflict edges in the object-level intent conflict graph, output the secondary confirmation result; If no user confirmation is received within the confirmation waiting time, output the delayed execution result; When there is a subject permission edge in the object-level intent conflict graph, output a rejection result, or output a downgraded execution result when the target controlled object has a low-permission alternative action. When a safety-suppressing edge exists in the object-level intent conflict graph, output a rejection result, or output a delayed execution result after the vehicle operation risk level is reduced to an allowable level.

10. An intelligent cockpit interaction control system based on multimodal perception, characterized in that, The system is applied to the intelligent cockpit interaction control method based on multimodal perception as described in any one of claims 1 to 9, the system comprising: The multimodal data acquisition module is used to collect cockpit multimodal interaction data and vehicle operating status data according to the same sampling time window, and to mark the issuing entity for the multimodal interaction data; The interference identification and credibility generation module is used to identify the environmental interference factors affecting each interaction modality within the sampling time window and generate the real-time credibility of each interaction modality. The semantic parsing module is used to perform semantic parsing on the interactive instructions corresponding to voice commands or touch operations, and to determine whether the interactive instructions are explicit object instructions or vague reference instructions. The object space management module is used to store the cockpit object space library and provides a query interface for cockpit object records; The target object binding module is used to determine the target control object when the interaction instruction is a clear object instruction, and to determine the target control object based on the gaze direction, gesture trajectory, touch operation, current display interface and interaction history when the interaction instruction is a vague reference instruction. The intent node building module is used to build object-level interactive intent nodes; The conflict graph generation module is used to generate object-level intent conflict graphs. The control arbitration module is used to determine the control arbitration outcome based on the object-level intent conflict graph; The cockpit control execution module is used to execute cockpit control operations according to the control arbitration result and write the interaction information after execution into the interaction history database.

Images