Multi-mode consciousness identification predictive vehicle control system and method and electronic equipment

Abstract

The invention discloses a predictive vehicle control system and method for multi-modal consciousness recognition and electronic equipment, and relates to the field of remote monitoring, and the system comprises a multi-modal sensing module, an intention prediction module, a cross-domain pre-control module and a false trigger suppression module. The multi-mode sensing module is used for collecting three-mode data of sight, gestures and voice and dynamically distributing weights based on environmental parameters; the intention prediction module is used for performing time-space synchronization verification on the three-mode data and calculating comprehensive confidence in a grading manner; the cross-domain pre-control module outputs pre-control instructions to a cabin domain, a vehicle body domain and a power domain based on the effective intention; the false triggering suppression module is used for dynamically adjusting a confidence coefficient threshold value based on a driving scene and has a failure backspacing learning function; wherein each module performs data interaction based on a vehicle-mounted Ethernet / CAN bus.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote monitoring, and in particular to a multi-modal awareness recognition predictive vehicle control system, a multi-modal awareness recognition predictive vehicle control method, an electronic device, a storage medium, and a vehicle cabin. BACKGROUND

[0002] The current vehicle human-computer interaction and control technology has the following core defects:

[0003] Insufficient multi-modal fusion capability: existing technologies (such as CN113226886A and CN111966320B) rely on "voice + image" dual-mode static matching, and do not realize dynamic spatio-temporal fusion of gaze, gesture, and voice - for example, only gesture recognition is used to determine the operation intention, and the relevance of gaze focus and action target is ignored, resulting in an intention prediction accuracy of less than 85%, and easy misjudgment of "accidental touch without operation intention".

[0004] Limited pre-control range: existing pre-control schemes (such as CN111968631B and CN118894104A) only cover a single system (such as cabin UI preloading or power real-time control), and do not build a "cabin-vehicle body-power" cross-domain collaborative pre-control link - for example, when predicting a lane change intention, only the lane keeping system (LKA) parameters are adjusted, and the sensitivity of the blind area monitoring system (BSD) is not optimized synchronously, resulting in a response delay of 0.5-1s for lane change operation.

[0005] Low robustness of false trigger suppression: existing technologies use fixed thresholds to determine intention validity (such as the action matching threshold in CN111966320B), and lack environmental adaptability - for example, when the gaze tracking accuracy decreases in a strong light environment, the original confidence threshold is still used, resulting in a false trigger rate of more than 15%, affecting driving safety and user experience.

[0006] To address the above-mentioned defects, the present application proposes a vehicle control system with multi-modal dynamic fusion, cross-domain pre-control, and environment-adaptive false trigger suppression, realizing a smooth interaction experience of "people want to move, and the vehicle knows". SUMMARY

[0007] The purpose of the present application is to provide a multi-modal awareness recognition predictive vehicle control system, a multi-modal awareness recognition predictive vehicle control method, an electronic device, a storage medium, and a vehicle cabin, which at least solves one of the technical problems.

[0008] For example, to solve the problems of insufficient multi-modal fusion capability, limited pre-control range, and low robustness of false trigger suppression.

[0009] The present application provides the following solutions:

[0010] According to a first aspect of the present application, a multi-modal awareness recognition predictive vehicle control system is provided, comprising:

[0011] a multi-modal awareness module, an intention prediction module, a cross-domain pre-control module, and a false trigger suppression module;

[0012] The multi-modal awareness module collects three modal data of sight, gesture, and voice, and dynamically allocates weights based on environmental parameters;

[0013] The intention prediction module performs temporal and spatial synchronization verification on the three modal data and calculates a comprehensive confidence level in stages;

[0014] The cross-domain pre-control module outputs pre-control instructions to the three domains of the cockpit, the vehicle body, and the power based on the effective intention;

[0015] The false trigger suppression module dynamically adjusts the confidence threshold based on the driving scene and has a failure rollback learning function;

[0016] Each module performs data interaction based on a vehicle-mounted Ethernet / CAN bus.

[0017] Further, comprising:

[0018] The multi-modal awareness module includes a sight tracking sub-module, a gesture recognition sub-module, and a voice recognition sub-module;

[0019] The sight tracking sub-module includes collecting driver eye images using a near-infrared camera, outputting sight focal point coordinates and a confidence level C1 through a Transformer model, and connecting a light sensor to detect light intensity L;

[0020] The gesture recognition sub-module includes collecting hand movements using an RGB camera, outputting gesture types and a confidence level C2 through a spatio-temporal graph neural network ST-GCN, and connecting an IMU to detect hand movement acceleration a;

[0021] The voice recognition sub-module includes collecting voice instructions using a microphone array, outputting semantic text and a confidence level C3 through a CTC model, and connecting a noise sensor to detect environmental noise N;

[0022] Wherein, the confidence levels of C1, C2, and C3 are valued between 0 and 1.

[0023] Further, comprising:

[0024] The rules for dynamically allocating weights include:

[0025] W1+W2+W3=1;

[0026] Wherein, W1 is the sight weight, W2 is the gesture weight, and W3 is the voice weight;

[0027] If L>5000 lux, then W1=0.2, W2=0.5, W3=0.3;

[0028] If N>65 dB, then W1=0.4, W2=0.5, W3=0.1;

[0029] If L≤5000 lux and N≤65 dB, then W1=0.35, W2=0.35, W3=0.3.

[0030] Further comprising:

[0031] The space-time synchronization verification comprises a space verification and a time verification.

[0032] The space verification comprises calculating the Euclidean distance D between the line-of-sight focus and the gesture target region.

[0033] When D<150px, it is determined that the space matches.

[0034] The time verification comprises calculating the time difference Δt between the line-of-sight focus stable time and the gesture action start time.

[0035] When Δt<200ms, it is determined that the time is synchronized.

[0036] When the space matches and the time is synchronized, the comprehensive confidence calculation link is entered.

[0037] Further comprising:

[0038] The comprehensive confidence C=W1×C1+W2×C2+W3×C3, and the grading rules comprise:

[0039] When C≥0.8, it is determined as high confidence.

[0040] Under high confidence, the pre-control is directly triggered.

[0041] When 0.6≤C<0.8, it is determined as medium confidence.

[0042] Under medium confidence, the continuous 3-frame timing verification is started.

[0043] If all the 3 frames satisfy the space-time synchronization, it is determined as valid.

[0044] When C<0.6, it is determined as low confidence.

[0045] Under low confidence, the pre-control is inhibited.

[0046] Further comprising:

[0047] The cabin UI pre-control logic of the cross-domain pre-control module comprises:

[0048] When the air conditioning adjustment intention is predicted and the confidence requirement is met, the air conditioning control interface is highlighted and the preset commonly used adjustment item is preloaded through the IVI system of the vehicle before the driver's finger reaches the central control screen.

[0049] Among them, the driver's finger reaches the central control screen within 150-200ms, and the program of highlighting the air conditioning control interface is started;

[0050] Among them, highlighting the air conditioning control interface includes, each time highlighting, increasing the original brightness by 30%;

[0051] The preset commonly used adjustment item temperature and air volume slider;

[0052] When the driver's finger does not reach the central control screen for more than a preset delay length, the brightness returns to the non-highlighted state.

[0053] Further, it includes:

[0054] The vehicle body electronic pre-control logic of the cross-domain pre-control module includes:

[0055] When the lane change intention is predicted and the confidence requirement is met, an instruction is sent to the BSD system to command to expand the detection range to 4m+(C-0.7)×2m and increase the sensitivity by 20%;

[0056] Also includes, sending an instruction to the LKA system to command to relax the lateral intervention threshold to ±0.3m+(C-0.7)×0.2m.

[0057] Further, it includes:

[0058] The power chassis pre-control logic of the cross-domain pre-control module includes:

[0059] When the acceleration intention is predicted in a congestion scenario and C≥0.75, an instruction is sent to the TCU to command to pre-switch the gear to the lower gear and adjust the clutch half-linkage stroke to 30%;

[0060] Also includes, sending an instruction to the ECU to command to increase the injection advance angle by 2°.

[0061] Further, it includes:

[0062] The scene adaptive threshold adjustment rule includes:

[0063] The comprehensive confidence trigger threshold of the lane change intention in the highway scenario is C≥0.7;

[0064] The comprehensive confidence trigger threshold of the power pre-adjustment in the urban congestion scenario is C≥0.8;

[0065] Also includes:

[0066] The failure fallback learning function comprises:

[0067] When the actual operation is not detected after 5 consecutive pre-control triggers, the confidence threshold in the current scene is automatically increased by 0.05, and the multi-modal data such as illumination, noise, gesture type and the like in the scene are recorded for weight algorithm optimization.

[0068] According to a second aspect of the present application, a predictive vehicle control method for multi-modal awareness recognition is provided, comprising:

[0069] S1: multi-modal data acquisition and dynamic weight distribution;

[0070] The eye image of the driver is collected by a near-infrared camera, the line-of-sight focal point coordinates and the confidence C1 are output by a Transformer model, and the light intensity L is detected by a light sensor;

[0071] The hand action is collected by an RGB camera, the gesture type and the confidence C2 are output by a space-time graph neural network ST-GCN, and the hand motion acceleration a is detected by an IMU;

[0072] The voice instruction is collected by a microphone array, the semantic text and the confidence C3 are output by a CTC model, and the environmental noise N is detected by a noise sensor;

[0073] The confidence values of C1, C2 and C3 are between 0 and 1;

[0074] The rules for dynamically distributing weights according to environmental parameters include: W1+W2+W3=1;

[0075] W1 is the line-of-sight weight, W2 is the gesture weight, and W3 is the voice weight;

[0076] If L>5000lux, then W1=0.2, W2=0.5, and W3=0.3;

[0077] If N>65dB, then W1=0.4, W2=0.5, and W3=0.1;

[0078] If L≤5000lux and N≤65dB, then W1=0.35, W2=0.35, and W3=0.3;

[0079] S2: time-space synchronization verification and confidence level decision;

[0080] The Euclidean distance D between the line-of-sight focal point and the gesture target region is calculated;

[0081] When D<150px, it is determined that the space matches;

[0082] The time verification comprises calculating a time difference Δt between the line-of-sight focus stable moment and the gesture action starting moment;

[0083] When Δt<200 ms, it is determined that the time is synchronized;

[0084] When the space is matched and the time is synchronized, the comprehensive confidence calculation link is entered;

[0085] The comprehensive confidence C=W1×C1+W2×C2+W3×C3, and the grading rules comprise:

[0086] When C≥0.8, it is determined that the confidence is high;

[0087] Under the high confidence, the pre-control is directly triggered;

[0088] When 0.6≤C<0.8, it is determined that the confidence is medium;

[0089] Under the medium confidence, the continuous 3-frame time sequence verification is started;

[0090] If all the 3 frames satisfy the space-time synchronization, it is determined to be valid;

[0091] When C<0.6, it is determined that the confidence is low;

[0092] Under the low confidence, the pre-control is inhibited;

[0093] S3: Cross-domain pre-control instruction output;

[0094] When the air conditioning intention is predicted and the confidence requirement is met, before the driver's finger reaches the center control screen, the air conditioning control interface is highlighted through the vehicle IVI system, and the preset commonly used adjustment item is preloaded;

[0095] The 150-200 ms before the driver's finger reaches the center control screen starts the program of highlighting the air conditioning control interface;

[0096] The highlighting of the air conditioning control interface comprises that each time the original brightness is increased by 30%;

[0097] The preset commonly used adjustment item temperature and air volume slider;

[0098] When the driver's finger does not reach the center control screen for more than a preset delay length, the brightness returns to the non-highlighted state;

[0099] When the lane changing intention is predicted and the confidence requirement is met, an instruction is sent to the BSD system to command to expand the detection range to 4m+(C-0.7)×2m and increase the sensitivity by 20%;

[0100] Also included is sending an instruction to the LKA system, commanding to relax the lateral intervention threshold to ±0.3m+(C-0.7)×0.2m;

[0101] When the congestion scenario predicts the acceleration intention and C is greater than or equal to 0.75, an instruction is sent to the TCU, commanding to pre-switch the gear to the lower gear by one level in advance, and adjust the clutch semi-engagement stroke to 30%;

[0102] Also included is sending an instruction to the ECU, commanding to increase the fuel injection advance angle by 2°;

[0103] S4: Scene adaptive threshold adjustment and failure fallback learning;

[0104] The comprehensive confidence trigger threshold of the lane change intention in the highway scenario is C greater than or equal to 0.7;

[0105] The comprehensive confidence trigger threshold of the power pre-regulation in the urban congestion scenario is C greater than or equal to 0.8;

[0106] When the actual operation is not detected after the pre-control is triggered for 5 times in succession, the confidence threshold in the current scenario is automatically increased by 0.05, and the multi-modal data such as light, noise, gesture type and the like in the scenario are recorded for weight algorithm optimization.

[0107] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0108] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the predictive vehicle control method of the modal awareness recognition.

[0109] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the predictive vehicle control method of the multi-modal awareness recognition.

[0110] According to a fifth aspect of the present application, a vehicle cabin is provided, comprising:

[0111] An electronic device is used to implement the steps of the predictive vehicle control method of the multi-modal awareness recognition;

[0112] A processor runs a program, and when the program runs, the data output from the electronic device executes the steps of the predictive vehicle control method of the multi-modal awareness recognition;

[0113] A storage medium for storing a program that, when executed, performs the steps of a predictive vehicle control method as modal awareness recognition on data output from an electronic device.

[0114] Through the above scheme, the following beneficial technical effects are obtained:

[0115] The present application makes up for the modal limitation of the prior art which only relies on "voice + image" by constructing a "line-of-sight tracking + gesture recognition + voice recognition" three-modal perception system.

[0116] The present application solves the problem that a fixed weight cannot adapt to a complex environment by designing an environment adaptive dynamic weight algorithm, which adjusts the three-modal weight in real time according to the light intensity (L) and the environmental noise (N) (such as reducing the line-of-sight weight to 0.2 and increasing the gesture weight to 0.5 in strong light).

[0117] The present application solves the precision defect of the prior art that only relies on a single modal feature to judge the intention by adding a space-time synchronization verification mechanism, which verifies space matching through Euclidean distance (D<150px) and verifies timing synchronization through time difference (Δt<200ms).

[0118] The present application solves the problem that the cockpit preloading is disconnected from the actual demand by realizing the precise linkage between the cockpit UI and the operation intention (such as highlighting the control interface 150-200ms in advance when predicting the air conditioning adjustment intention).

[0119] The present application solves the decision gap problem of the prior art that ignores BSD optimization when adjusting LKA by establishing a vehicle body electronic system cooperative pre-control logic (such as synchronously adjusting the BSD detection range and the LKA intervention threshold when changing lanes).

[0120] The present application solves the response delay problem of single power control that cannot adapt to intention prediction by opening up the powertrain pre-adjustment link (such as pre-switching the gear position of the gearbox and adjusting the injection advance angle when following a car in congestion).

[0121] The present application solves the problem that a fixed threshold cannot adapt to different driving scenarios by proposing a scene adaptive threshold adjustment strategy (such as reducing the highway lane change threshold to C≥0.7 and increasing the urban congestion power pre-adjustment threshold to C≥0.8).

[0122] The present application solves the problem of low robustness of a single threshold by constructing a three-level confidence level decision (C≥0.8 directly triggers, 0.6≤C<0.8 timing verification, and C<0.6 suppression).

[0123] The present application solves the defect of the prior art that has no closed-loop optimization by adding a failure rollback learning mode (increasing the threshold by 0.05 when the actual operation is not detected for 5 consecutive times). BRIEF DESCRIPTION OF DRAWINGS

[0124] Figure 1 This is a flowchart of a predictive vehicle control method based on modal awareness recognition provided in one or more embodiments of the present invention.

[0125] Figure 2 This is a structural diagram of a predictive vehicle control system for modal awareness recognition provided by one or more embodiments of the present invention.

[0126] Figure 3 This is a schematic diagram of a vehicle control system provided in a specific embodiment of the present invention.

[0127] Figure 4 This is a schematic diagram of a three-level confidence level classification provided in a specific embodiment of the present invention.

[0128] Figure 5 This is a schematic diagram of a dynamic weight allocation provided in a specific embodiment of the present invention.

[0129] Figure 6 This is a schematic diagram of a scene-adaptive threshold adjustment provided in a specific embodiment of the present invention.

[0130] Figure 7 This is an electronic device structural block diagram of a predictive vehicle control method for multimodal consciousness recognition provided in one or more embodiments of the present invention. Detailed Implementation

[0131] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0132] Figure 2 This is a structural diagram of a predictive vehicle control system for modal awareness recognition provided by one or more embodiments of the present invention.

[0133] like Figure 2 As shown in Figure 1, a predictive vehicle control system based on multimodal consciousness recognition, comprising:

[0134] Multimodal perception module, intent prediction module, cross-domain pre-control module, and false triggering suppression module;

[0135] The multimodal perception module collects data in three modalities: gaze, gesture, and voice, and dynamically assigns weights based on environmental parameters.

[0136] The intent prediction module performs spatiotemporal synchronization verification on the three-modal data and calculates the comprehensive confidence level in a hierarchical manner.

[0137] The cross-domain pre-control module outputs pre-control instructions to the cockpit, vehicle body and power domains based on an effective intention.

[0138] The false triggering suppression module dynamically adjusts the confidence threshold based on the driving scene and has a failure fallback learning function.

[0139] Each module performs data interaction based on a vehicle-mounted Ethernet / CAN bus.

[0140] In this embodiment, the following is included:

[0141] The multi-modal perception module includes a line-of-sight tracking sub-module, a gesture recognition sub-module and a voice recognition sub-module.

[0142] The line-of-sight tracking sub-module includes using a near-infrared camera to collect driver eye images, outputting a line-of-sight focal point coordinate and a confidence C1 through a Transformer model, and connecting a light sensor to detect light intensity L.

[0143] The gesture recognition sub-module includes using an RGB camera to collect hand movements, outputting a gesture type and a confidence C2 through a spatio-temporal graph neural network ST-GCN, and connecting an IMU to detect hand movement acceleration a.

[0144] The voice recognition sub-module includes using a microphone array to collect voice instructions, outputting a semantic text and a confidence C3 through a CTC model, and connecting a noise sensor to detect environmental noise N.

[0145] The confidence values of C1, C2 and C3 are between 0 and 1.

[0146] In this embodiment, the following is included:

[0147] The rules for dynamically assigning weights include:

[0148] W1+W2+W3=1;

[0149] W1 is the line-of-sight weight, W2 is the gesture weight, and W3 is the voice weight.

[0150] If L>5000lux, then W1=0.2, W2=0.5 and W3=0.3.

[0151] If N>65dB, then W1=0.4, W2=0.5 and W3=0.1.

[0152] If L≤5000lux and N≤65dB, then W1=0.35, W2=0.35 and W3=0.3.

[0153] In this embodiment, the following is included:

[0154] The space-time synchronization verification includes a space verification and a time verification;

[0155] The space verification includes calculating the Euclidean distance D between the line-of-sight focus and the gesture target region;

[0156] When D < 150px, it is determined that the space matches;

[0157] The time verification includes calculating the time difference At between the time when the line-of-sight focus is stable and the time when the gesture action is started;

[0158] When At < 200ms, it is determined that the time is synchronized;

[0159] When the space matches and the time is synchronized, the comprehensive confidence calculation link is entered.

[0160] In the embodiment, the following is included:

[0161] The comprehensive confidence C = W1xC1 + W2xC2 + W3xC3, and the grading rules include:

[0162] When C > 0.8, it is determined that the confidence is high;

[0163] Under the high confidence, the pre-control is directly triggered;

[0164] When 0.6 < C < 0.8, it is determined that the confidence is medium;

[0165] Under the medium confidence, the continuous 3-frame time sequence verification is started;

[0166] If all of the 3 frames satisfy the space-time synchronization, it is determined to be valid;

[0167] When C < 0.6, it is determined that the confidence is low;

[0168] Under the low confidence, the pre-control is inhibited.

[0169] In the embodiment, the following is included:

[0170] The cockpit UI pre-control logic of the cross-domain pre-control module includes:

[0171] When the air conditioning adjustment intention is predicted and the confidence requirement is met, before the driver's finger reaches the center control screen, the air conditioning control interface is highlighted through the vehicle IVI system, and the preset commonly used adjustment item is preloaded;

[0172] Among them, 150-200ms before the driver's finger reaches the center control screen, the program of highlighting the air conditioning control interface is started;

[0173] Among them, highlighting the air conditioning control interface includes increasing the original brightness by 30% each time;

[0174] Pre-set commonly used adjustment item temperature and air volume slider;

[0175] When the driver's finger does not reach the central control screen for more than a preset delay length, the brightness returns to the non-highlighted state.

[0176] In the embodiment, it comprises:

[0177] The vehicle body electronic pre-control logic of the cross-domain pre-control module comprises:

[0178] When the lane changing intention is predicted and the confidence requirement is met, an instruction is sent to the BSD system, commanding to expand the detection range to 4m+(C-0.7)×2m and increase the sensitivity by 20%;

[0179] Further, an instruction is sent to the LKA system, commanding to relax the lateral intervention threshold to ±0.3m+(C-0.7)×0.2m.

[0180] In the embodiment, it comprises:

[0181] The power chassis pre-control logic of the cross-domain pre-control module comprises:

[0182] When the acceleration intention is predicted in the congestion scenario and C is greater than or equal to 0.75, an instruction is sent to the TCU, commanding to pre-switch the gear to the lower gear and adjust the clutch half-linkage stroke to 30%;

[0183] Further, an instruction is sent to the ECU, commanding to increase the fuel injection advance angle by 2°.

[0184] In the embodiment, it comprises:

[0185] The scenario adaptive threshold adjustment rule comprises:

[0186] The comprehensive confidence trigger threshold of the lane changing intention in the highway scenario is C≥0.7;

[0187] The comprehensive confidence trigger threshold of the power pre-adjustment in the urban congestion scenario is C≥0.8;

[0188] Further, it comprises:

[0189] The failure fallback learning function comprises:

[0190] When the actual operation is not detected after the pre-control is triggered for 5 times in succession, the confidence threshold in the current scenario is automatically increased by 0.05, and the multi-modal data such as light, noise, gesture type and the like in the scenario are recorded for weight algorithm optimization.

[0191] Figure 2 is a structural diagram of a modal awareness recognition predictive vehicle control system provided by one or more embodiments of the application.

[0192] As Figure 2 shown in the prediction vehicle control method of multi-modal consciousness recognition includes:

[0193] S1: multi-modal data acquisition and dynamic weight distribution;

[0194] The near-infrared camera is used to collect the driver's eye image, the line-of-sight focal point coordinates and the confidence C1 are output through the Transformer model, and the light sensor is connected to detect the light intensity L;

[0195] RGB camera is used to collect hand action, gesture type and confidence C2 are output through space-time graph neural network ST-GCN, and IMU is connected to detect hand motion acceleration a;

[0196] The microphone array is used to collect the voice instruction, the semantic text and the confidence C3 are output through the CTC model, and the noise sensor is connected to detect the environmental noise N;

[0197] Among them, the confidence values of C1, C2 and C3 are between 0 and 1;

[0198] The rule of dynamically distributing weights according to environmental parameters includes: W1+W2+W3=1;

[0199] Among them, W1 is the line-of-sight weight, W2 is the gesture weight, and W3 is the voice weight;

[0200] If L>5000lux, then W1=0.2, W2=0.5, W3=0.3;

[0201] If N>65dB, then W1=0.4, W2=0.5, W3=0.1;

[0202] If L≤5000lux and N≤65dB, then W1=0.35, W2=0.35, W3=0.3;

[0203] S2: space-time synchronization verification and confidence level decision;

[0204] Calculate the Euclidean distance D between the line-of-sight focal point and the gesture target area;

[0205] When D<150px, it is determined that the space matches;

[0206] The time verification includes calculating the time difference Δt between the line-of-sight focal point stable time and the gesture action start time;

[0207] When Δt<200ms, it is determined that the time is synchronized;

[0208] When the space matches and the time is synchronized, then enter the comprehensive confidence calculation link;

[0209] Wherein, the comprehensive confidence C = W1xC1 + W2xC2 + W3xC3, the grading rules include:

[0210] When C ≥ 0.8, it is determined as high confidence;

[0211] Under high confidence, directly trigger pre-control;

[0212] When 0.6 ≤ C < 0.8, it is determined as medium confidence;

[0213] Under medium confidence, start continuous 3 frame timing verification;

[0214] If 3 frames all meet space-time synchronization, it is determined as valid;

[0215] When C < 0.6, it is determined as low confidence;

[0216] Under low confidence, inhibit pre-control;

[0217] S3: Cross-domain pre-control instruction output;

[0218] When predicting the air conditioning intention and meeting the confidence requirement, before the driver's finger reaches the center control screen, the air conditioning control interface is highlighted through the vehicle IVI system, and the preset commonly used adjustment item is preloaded;

[0219] Wherein, 150-200ms before the driver's finger reaches the center control screen, the program of highlighting the air conditioning control interface is started;

[0220] Wherein, highlighting the air conditioning control interface includes, each time highlighting, the original brightness is increased by 30%;

[0221] The preset commonly used adjustment item temperature and air volume slider;

[0222] When the driver's finger does not reach the center control screen for more than a preset delay length, the brightness returns to the non-highlighted state;

[0223] When predicting the lane change intention and meeting the confidence requirement, an instruction is sent to the BSD system (full name Blind Spot Detection, i.e. blind spot monitoring system), and the detection range is extended to 4m+ (C-0.7) ×2m, and the sensitivity is increased by 20%;

[0224] Also includes, an instruction is sent to the LKA system (full name Lane Keeping Assist, i.e. lane keeping assist system), and the lateral intervention threshold is widened to ±0.3m+ (C-0.7) ×0.2m;

[0225] When the acceleration intention is predicted under the congestion scene and C≥0.75, an instruction is sent to the TCU to command the gear to be pre-switched to the next lower gear and the clutch semi-linkage stroke to be adjusted to 30% in advance;

[0226] Further comprising, sending an instruction to the ECU to command the fuel injection advance angle to be increased by 2°;

[0227] S4: Scene adaptive threshold adjustment and failure fallback learning;

[0228] The comprehensive confidence trigger threshold of lane change intention under the highway scene is C≥0.7;

[0229] The comprehensive confidence trigger threshold of power pre-regulation under the urban congestion scene is C≥0.8;

[0230] When the actual operation is not detected after the pre-control is triggered for 5 times in succession, the confidence threshold under the current scene is automatically increased by 0.05, and the multi-modal data such as light, noise, gesture type and the like under the scene are recorded for weight algorithm optimization.

[0231] It is worth noting that although the system / device only discloses the above-mentioned module / unit module, it does not mean that the system / device is limited to the above-mentioned basic function modules. On the contrary, the meaning expressed by the present application is that on the basis of the above-mentioned basic function modules, a person skilled in the art can add one or more function modules to form infinite embodiments or technical solutions in combination with the prior art. That is to say, the system is open rather than closed, and it cannot be considered that the protection scope of the present application is limited to the above-mentioned basic function modules just because the present embodiment discloses only individual basic function modules.

[0232] In one specific embodiment, a vehicle control system is disclosed, which includes four core units of multi-modal perception module, intention prediction module, cross-domain pre-control module and false trigger suppression module, each unit realizes data interaction through vehicle-mounted Ethernet / CAN bus, end-to-end delay≤50ms (based on NVIDIA Orin edge domain controller), as shown in Figure 3 The specific scheme is as follows:

[0233] Multi-modal perception module: dynamic weight three-modal data acquisition and preprocessing;

[0234] Submodule composition:

[0235] Gaze tracking submodule: near-infrared camera (frame rate 60fps) is used to collect driver eye images, and the gaze focal point coordinates (accuracy ±10px) and confidence C1 (0-1) are output through the Transformer model, and the light sensor data (detecting light intensity L, unit lux) are connected;

[0236] Gesture recognition sub-module: RGB camera (frame rate 30fps) is used to collect hand actions, and spatio-temporal graph neural network (ST-GCN) is used to output gesture type (such as "pointing" and "pressing") and confidence C2 (0-1), and IMU data (detecting hand motion acceleration a, unit m / s²) is inputted;

[0237] Voice recognition sub-module: microphone array (noise reduction level 40dB) is used to collect voice instructions, and CTC model is used to output semantic text (such as "turn on air conditioner") and confidence C3 (0-1), and noise sensor data (detecting environmental noise N, unit dB) is inputted.

[0238] As shown in Figure 5 , the dynamic weight distribution algorithm is:

[0239] According to the environmental parameters, the three-modal weights (W1+W2+W3=1) are adjusted in real time:

[0240] When L>5000lux (strong light): W1=0.2 (reduce the line of sight weight), W2=0.5 (increase the gesture weight), and W3=0.3;

[0241] When N>65dB (high noise): W1=0.4, W2=0.5, and W3=0.1 (reduce the voice weight);

[0242] Normal environment (L≤5000lux and N≤65dB): W1=0.35, W2=0.35, and W3=0.3.

[0243] Intention prediction module: spatio-temporal synchronization verification and confidence grading;

[0244] Spatio-temporal synchronization verification:

[0245] Spatio-temporal correlation verification is performed on multi-modal data:

[0246] Spatial verification: calculate the Euclidean distance D (unit px) between the line of sight focus and the gesture target area, and when D<150px, it is determined that the space matches;

[0247] Time verification: calculate the time difference Δt (unit ms) between the stable time of the line of sight focus and the starting time of the gesture action, and when Δt<200ms, it is determined that the time is synchronized;

[0248] Only when the space matches and the time is synchronized, the confidence calculation link is entered.

[0249] As shown in Figure 4 , three-level confidence grading:

[0250] Calculate the comprehensive intention confidence C=W1×C1+W2×C2+W3×C3, and grade:

[0251] High confidence (C≥0.8): directly determine the intention is valid, trigger pre-control;

[0252] Medium confidence (0.6≤C<0.8): start continuous 3 frames (frame interval 33 ms) timing verification, if 3 frames meet space-time synchronization, determine valid;

[0253] Low confidence (C<0.6): determine that the intention is invalid, inhibit pre-control.

[0254] Cross-domain pre-control module: cockpit-car-body-power collaborative pre-preparation;

[0255] Based on the predicted intention type (such as "adjusting air conditioner" "lane change" "acceleration"), output pre-control instructions to the three domains:

[0256] Cockpit UI pre-control:

[0257] When predicting "adjusting air conditioner" intention (C≥0.8): before the driver's finger reaches the center screen (150-200 ms in advance), highlight the air conditioner control interface through the vehicle IVI system (brightness increased by 30%), and preload common adjustment items (such as temperature, air volume slider), reduce interface switching delay;

[0258] Car body electronic pre-control:

[0259] When predicting "lane change" intention (continuous 3 frames C≥0.7):

[0260] Send instructions to BSD system: expand the detection range from the original 4m to 4m+(C-0.7)×2m (such as C=0.9, expand to 4.4m), sensitivity increased by 20%;

[0261] Send instructions to LKA system: relax the lateral intervention threshold from the original ±0.3m to ±0.3m+(C-0.7)×0.2m (such as C=0.9, relax to ±0.34m), avoid frequent intervention when changing lanes;

[0262] Power chassis pre-control:

[0263] When predicting "acceleration" intention (C≥0.75 in congestion scenario):

[0264] Send instructions to transmission control unit (TCU): pre-switch the gear from the current gear to the lower gear (such as current 3 gears → pre-mount 2 gears), and adjust the clutch semi-linkage stroke to 30% (original 15%);

[0265] Send instructions to engine control unit (ECU): increase the fuel injection advance angle by 2°, improve power response speed, and reduce the jerk when following in congestion.

[0266] False trigger suppression module: environment adaptive threshold and failure fallback;

[0267] As shown in Figure 6 Scene adaptive threshold adjustment:

[0268] Dynamic adjustment of confidence threshold according to vehicle scene (acquire road type through V2X or GPS):

[0269] Highway scene: lane change intention trigger threshold is reduced to C≥0.7 (originally 0.8), to adapt to the rapid operation requirement when driving at high speed;

[0270] Urban congestion scene: power pre-adjustment trigger threshold is raised to C≥0.8 (originally 0.75), to avoid frequent misadjustment;

[0271] Failure fallback mechanism:

[0272] When no actual operation is detected (such as the finger does not reach the screen, the steering wheel is not turned) after 5 consecutive pre-control triggers:

[0273] Automatically enter the learning mode, and raise the confidence threshold in the current scene by 0.05 (such as from 0.8→0.85);

[0274] Record the multi-modal data (such as light, noise, gesture type) in this scene for subsequent weight algorithm optimization.

[0275] In another specific embodiment, a predictive vehicle control in multiple scenes is disclosed, comprising:

[0276] Embodiment 1: air conditioner pre-starting scene;

[0277] Perception stage:

[0278] The light sensor detects the light intensity L=3000 lux (regular environment), the noise sensor detects N=50 dB (regular environment), and the multi-modal weight distribution is W1=0.35, W2=0.35, W3=0.3;

[0279] The line-of-sight tracking sub-module detects that the driver's line-of-sight focus falls in the center control screen air conditioner area (coordinates (450, 320)) for 3 frames (99 ms), C1=0.92;

[0280] The gesture recognition sub-module detects that the driver's right hand moves towards the center control screen (acceleration a=0.8 m / s²), and the gesture type is "pointing", C2=0.88;

[0281] The speech recognition sub-module does not detect valid instructions, C3=0.1.

[0282] Prediction stage:

[0283] Spatial verification: Euclidean distance between the gaze focus and the air conditioning area D = 80px < 150px, spatial matching;

[0284] Temporal verification: Δt = 120ms < 200ms, time synchronization;

[0285] Comprehensive confidence C = 0.35 × 0.92 + 0.35 × 0.88 + 0.3 × 0.1 = 0.676, enter the medium confidence timing verification, continuous 3 frames all meet the space-time synchronization, determine that the "adjust air conditioner" intention is valid.

[0286] Pre-control stage:

[0287] The cross-domain pre-control module sends instructions to the IVI system, and highlights the air conditioning control interface 180ms in advance (brightness from 500cd / m² to 650cd / m²), preloads the temperature (24℃) and air volume (3 gears) sliders;

[0288] When the driver's finger reaches the screen, it directly enters the adjustment interface, and the operation delay is reduced from the original 300ms to 120ms.

[0289] False trigger suppression:

[0290] This scenario is an urban road (not congested), the trigger threshold is maintained C ≥ 0.6, the failure fallback mechanism is not triggered, and there is no additional adjustment.

[0291] Example 2: Lane change pre-control scenario;

[0292] Perception stage:

[0293] The light sensor detects L = 6000lux (strong light), the noise sensor detects N = 60dB, and the multi-modal weight distribution is W1 = 0.2, W2 = 0.5, W3 = 0.3;

[0294] The gaze tracking sub-module detects that the driver's gaze moves to the right side mirror direction, C1 = 0.75;

[0295] The gesture recognition sub-module detects that the driver's left hand holds the steering wheel lightly (preparing to turn), and the gesture type is "turning preparation", C2 = 0.9;

[0296] The speech recognition sub-module detects the instruction "turn on the right turn signal", C3 = 0.85.

[0297] Prediction stage:

[0298] Spatial verification: Euclidean distance between the gaze focus and the right side mirror area D = 120px < 150px, spatial matching;

[0299] Time verification: line-of-sight movement and gesture initiation Δt = 180 ms < 200 ms, time synchronization;

[0300] Comprehensive confidence C = 0.2 x 0.75 + 0.5 x 0.9 + 0.3 x 0.85 = 0.855, high confidence, determine that the "lane change" intention is valid.

[0301] Pre-control stage:

[0302] Send instructions to the BSD system: the detection range is expanded from 4 m to 4 + (0.855 - 0.7) x 2 = 4.31 m, and the sensitivity is improved by 20%;

[0303] Send instructions to the LKA system: the lateral intervention threshold is relaxed from ± 0.3 m to ± 0.3 + (0.855 - 0.7) x 0.2 = ± 0.331 m;

[0304] When the driver turns on the turn signal and turns, the BSD detects a right-side oncoming vehicle 0.8 s in advance, and the LKA does not produce unnecessary intervention, and the lane change smoothness is improved by 40%.

[0305] False trigger suppression:

[0306] This scenario is on a highway, and the lane change intention trigger threshold is reduced to C ≥ 0.7. The current C = 0.855 meets the requirements, and there is no false trigger.

[0307] Three-mode dynamic weight fusion technology (perception layer core)

[0308] Core features: based on lighting (L) and noise (N) environmental parameters, real-time adjustment of weight proportion of line-of-sight tracking (W1), gesture recognition (W2), and voice recognition (W3) (such as W1 = 0.2, W2 = 0.5 when strong light L > 5000 lux, and W3 = 0.1 when high noise N > 65 dB), rather than fixed weights in existing technologies;

[0309] Technical value: solves the intention misjudgment caused by precision fluctuations of a single mode (such as line-of-sight in strong light and voice in high noise), and is the basis for achieving 98% prediction accuracy.

[0310] Time and space synchronization verification + three-level confidence decision (prediction layer core);

[0311] Core features: first verify the multi-modal correlation through "Euclidean distance D < 150 px (spatial matching) + time difference Δt < 200 ms (time synchronization)", and then make a hierarchical decision according to the comprehensive confidence C (C ≥ 0.8 directly triggers, 0.6 ≤ C < 0.8 times verification, and C < 0.6 suppression);

[0312] Technical value: filters out invalid intentions such as "line-of-sight-gesture deviation" and "instantaneous misrecognition", and avoids the robustness defects of single threshold judgment in existing technologies.

[0313] Cockpit-vehicle-power cross-domain pre-control coordination logic (control layer core);

[0314] Core features: based on predicted intention to realize multi-system cascade pre-adjustment, such as lane change intention to synchronize expand BSD detection range (4m→4.31m), relax LKA intervention threshold (±0.3m→±0.331m), linkage TCU pre-shift (3rd→2nd) and ECU injection adjustment (+2°) when following in congestion;

[0315] Technical value: break through the existing technology "single domain pre-control" limitation, shorten 0.5-1s response delay, reduce 60% jerk.

[0316] Scene adaptive + failure backoff false trigger suppression mechanism (fault tolerance layer core);

[0317] Core features: dynamically adjust confidence threshold according to scene (high-speed lane change C≥0.7, congestion power C≥0.8), automatically increase threshold 0.05 and optimize mode weight after 5 consecutive pre-control failures, form a closed loop learning;

[0318] Technical value: reduce false trigger rate from 15% to 3%, while adapting to different driver operation habits, solve the scene inadaptability of existing technology fixed threshold.

[0319] Alternative solution 1: single mode reinforced perception scheme (only rely on high-precision gaze tracking);

[0320] Implementation principle: use 120fps high-resolution near-infrared camera (such as OV5640), improve gaze tracking accuracy to ±5px through deep learning model, predict intention based on gaze focus change only (such as gaze stays in air conditioning area for 300ms to determine adjustment intention);

[0321] Limitations:

[0322] Poor environmental adaptability: even with high-precision cameras, gaze accuracy still drops by 25% in strong light (L>5000lux), false positive rate rises to 20% (3% in this application);

[0323] Intention ambiguity cannot be solved: gaze stays in navigation area may be "view route" or "adjust navigation volume", cannot be distinguished without gesture / voice assistance, invalid trigger increases by 50%.

[0324] Alternative solution 2: fixed weight multi-modal + single domain pre-control scheme;

[0325] Principle: Adopt "line of sight + gesture" dual mode, fixed weight W1=0.5, W2=0.5, only pre-control cockpit UI (such as pre-loading interface when predicting air conditioning intent), vehicle body / power system still uses real-time control;

[0326] Limitations:

[0327] Narrow pre-control range: only LKA adjusts parameters when changing lanes, BSD is not synchronized and optimized, lane changing response delay is still 0.8s (only 0.2s in this application);

[0328] Weight rigidity: speech cannot assist in high noise, but there is no weight adjustment mechanism, intent prediction accuracy is only 82% (98% in this application);

[0329] Power jerk is not solved: no TCU / ECU pre-adjustment when following in congestion, jerk frequency is still 4 times per minute (only 2 times in this application).

[0330] Through the above embodiments, "precision improvement, delay reduction, false trigger reduction, experience optimization" four categories of beneficial effects are directly brought about, all effects are necessarily produced by specific technical solutions, and can be verified by quantitative data or actual scene:

[0331] 1. Multi-modal fusion precision is significantly improved, and intent prediction accuracy is significantly improved

[0332] The combination design of three-modal dynamic fusion + time-space verification makes the intent prediction accuracy improve from ≤85% of the prior art to ≥98%, and the misjudgment rate decreases by 13 percentage points - in the strong light (L=6000 lux) scene, by reducing the line of sight weight and increasing the gesture weight, the influence of the 30% decrease in line of sight tracking accuracy is offset, and the intent recognition accuracy is still maintained above 96%;

[0333] The time-space synchronous verification mechanism avoids "false touch without intent" and "line of sight deviation still triggers" misjudgment, such as when the driver accidentally stretches to the center control screen but the line of sight is not focused on the air conditioning area (D=200px>150px), the system automatically suppresses pre-control, reducing more than 60% of invalid triggers compared to the prior art.

[0334] 2. Cross-domain pre-control shortens response delay, and operation fluency improves by more than 40%

[0335] Cockpit UI pre-loading shortens the operation delay from 300ms of the prior art to 120ms (60% reduction), such as adjusting the air conditioner, the driver can directly operate the highlighted interface when touching the screen, without waiting for the interface to switch;

[0336] The body electronic cooperative pre-control makes the lane changing response delay reduce 0.5-1s, like after the BSD detection range is expanded, the side rear vehicle is identified 0.8s in advance, the LKA intervention threshold is widened to avoid frequent correction of direction when changing lanes, and the lane changing fluency is improved by 40%.

[0337] The power chassis pre-regulation makes the power response delay reduce from 0.6s to 0.2s (67% reduction) when following a car in congestion, and the jerk frequency reduces from 5 times / min to 2 times / min (60% reduction), so that the driving comfort is significantly improved.

[0338] 3. The robustness of false triggering suppression is improved, and the false activation rate is controlled within 3%

[0339] The scene adaptive threshold + three-level confidence grading makes the false triggering rate reduce from more than 15% of the prior art to ≤3%, such as in a high noise (N=70dB) scene, reducing the voice weight to 0.1 and improving the confidence threshold to avoid air conditioner / window false start caused by voice misrecognition.

[0340] The failure fallback learning mode makes the system have self-optimization capability, such as after 2 consecutive false triggers, the threshold is automatically increased by 0.05, and the false triggering probability is reduced by 20% in the same scene subsequently, which adapts to the operation habit differences of different drivers.

[0341] 4. System resource utilization optimization, adaptive functional safety standard

[0342] Dynamic weight allocation reduces invalid modal data processing, such as no need to calculate line-of-sight data under strong light, the algorithm power occupancy of the vehicle-mounted domain controller (NVIDIA Orin) reduces by 15%, and the energy consumption reduces by 8%;

[0343] Cross-domain pre-control adopts the logic of "preparation instead of direct execution" (such as pre-hanging gear without direct gear shifting, pre-highlight interface without triggering function), to avoid the influence of false operation on driving safety, which meets the ISO26262 ASILB level functional safety standard, and balances experience and safety.

[0344] Figure 3 An electronic device structure block diagram of a multi-modal awareness recognition predictive vehicle control method provided by one or more embodiments of the present application.

[0345] As shown in Figure 3 The present application provides an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0346] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a multi-modal awareness recognition predictive vehicle control method.

[0347] The application also provides a computer readable storage medium storing a computer program executable by an electronic device, which, when executed on the electronic device, causes the electronic device to perform the steps of the method for predictive vehicle control based on multi-modal awareness recognition.

[0348] The application also provides a vehicle cabin comprising:

[0349] An electronic device for implementing the steps of the method for predictive vehicle control based on multi-modal awareness recognition;

[0350] A processor, which, when running a program, performs the steps of the method for predictive vehicle control based on multi-modal awareness recognition from data output by the electronic device;

[0351] A storage medium for storing a program, which, when running, performs the steps of the method for predictive vehicle control based on multi-modal awareness recognition from data output by the electronic device.

[0352] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0353] The electronic device comprises a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer comprises central processing units (CPUs), memory management units (MMUs), memories, and other hardware. The operating system can be any one or more computer operating systems that implement control of the electronic device through processes, such as Linux, Unix, Android, iOS, or windows operating systems, etc. In the embodiments of the application, the electronic device can be a handheld device such as a smartphone or a tablet computer, or an electronic device such as a desktop computer or a portable computer, and is not particularly limited in the embodiments of the application.

[0354] The execution subject of the electronic device control in the embodiment of the present application can be an electronic device, or a functional module in the electronic device capable of calling and executing a program. The electronic device can acquire firmware corresponding to the storage medium, the firmware corresponding to the storage medium is provided by a supplier, and the firmware corresponding to different storage media can be the same or different, which is not limited herein. After the electronic device acquires the firmware corresponding to the storage medium, the electronic device can write the firmware corresponding to the storage medium into the storage medium, specifically, burn the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented by using the prior art, which is not described in detail in the embodiment of the present application.

[0355] The electronic device can also acquire a reset command corresponding to the storage medium, the reset command corresponding to the storage medium is provided by a supplier, and the reset commands corresponding to different storage media can be the same or different, which is not limited herein.

[0356] At this time, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium in which the corresponding firmware is written, so that the electronic device resets the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by using the prior art, which is not described in detail in the embodiment of the present application.

[0357] For the convenience of description, the above device is described as various units and modules in terms of functions. Of course, the functions of the units and modules can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0358] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood as having meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0359] For the method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the action order described, because according to the embodiments of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0360] Those skilled in the art can clearly understand the application by the description of the above embodiments that the application can be implemented by means of software and the necessary universal hardware platform. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each of the embodiments or some parts of the embodiments of the application.

[0361] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, rather than limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A predictive vehicle control system for multi-modal awareness recognition, characterized by, The predictive vehicle control system of the multi-modal consciousness recognition comprises: A multi-modal perception module, an intention prediction module, a cross-domain pre-control module, and a false trigger suppression module; The multi-modal perception module collects three modal data of sight line, gesture, and voice, and dynamically allocates weights based on environmental parameters; The intention prediction module performs space-time synchronization verification on the three modal data and calculates a comprehensive confidence level in stages; The cross-domain pre-control module outputs pre-control instructions to the three domains of cockpit, vehicle body, and power based on effective intentions; The false trigger suppression module dynamically adjusts the confidence threshold based on the driving scene and has a failure rollback learning function; Each module performs data interaction based on a vehicle-mounted Ethernet / CAN bus.

2. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Comprise: The multi-modal perception module comprises a sight line tracking sub-module, a gesture recognition sub-module, and a voice recognition sub-module; The sight line tracking sub-module Comprises collecting driver eye images using a near-infrared camera, outputting sight line focal point coordinates and a confidence level C1 through a Transformer model, and connecting a light sensor to detect light intensity L; The gesture recognition sub-module comprises collecting hand movements using an RGB camera, outputting gesture types and a confidence level C2 through a space-time graph neural network ST-GCN, and connecting an IMU to detect hand movement acceleration a; The voice recognition sub-module comprises collecting voice instructions using a microphone array, outputting semantic text and a confidence level C3 through a CTC model, and connecting a noise sensor to detect environmental noise N; Wherein, the confidence levels of C1, C2, and C3 are between 0 and 1.

3. The predictive vehicle control system of multimodal awareness recognition of claim 2, wherein, Comprise: The rules for dynamically allocating weights comprise: W1+W2+W3=1; Wherein, W1 is the sight line weight, W2 is the gesture weight, and W3 is the voice weight; If L>5000lux, then W1=0.2, W2=0.5, and W3=0.3; If N>65dB, then W1=0.4, W2=0.5, and W3=0.1; If L≤5000lux and N≤65dB, then W1=0.35, W2=0.35, and W3=0.

3.

4. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Comprise: The space-time synchronization verification comprises spatial verification and temporal verification; The spatial verification comprises calculating the Euclidean distance D between the sight line focal point and the gesture target area; When D<150px, it is determined that the space is matched; The temporal verification comprises calculating the time difference Δt between the sight line focal point stable time and the gesture action start time; When Δt<200ms, it is determined that the time is synchronized; When the space is matched and the time is synchronized, the comprehensive confidence level calculation link is entered.

5. The predictive vehicle control system of multimodal awareness recognition of claim 4, wherein, Comprise: The comprehensive confidence level C=W1×C1+W2×C2+W3×C3, and the hierarchical rules comprise: When C≥0.8, it is determined to be a high confidence level; Under high confidence level, pre-control is directly triggered; When 0.6≤C<0.8, it is determined to be a medium confidence level; Under medium confidence level, continuous 3-frame time sequence verification is started; If 3 frames all meet the space-time synchronization, it is determined to be valid; When C<0.6, it is determined to be a low confidence level; Under low confidence level, pre-control is suppressed.

6. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Comprise: The cockpit UI pre-control logic of the cross-domain pre-control module comprises: When the air conditioning adjustment intention is predicted and the confidence requirement is met, the air conditioning control interface is highlighted and the preset commonly used adjustment item is preloaded through the vehicle IVI system before the driver's finger reaches the central control screen; Among them, 150-200ms before the driver's finger reaches the central control screen, the program of highlighting the air conditioning control interface is started; Among them, highlighting the air conditioning control interface includes, each time highlighting, the original brightness is increased by 30%; The preset commonly used adjustment item temperature and air volume slider; When the driver's finger does not reach the central control screen for more than a preset delay length, the brightness returns to the non-highlighted state.

7. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Including: The vehicle body electronic pre-control logic of the cross-domain pre-control module includes: When the lane changing intention is predicted and the confidence requirement is met, an instruction is sent to the BSD system to command the detection range to be expanded to 4m+(C-0.7)×2m and the sensitivity to be increased by 20%; Also including, sending an instruction to the LKA system to command the lateral intervention threshold to be relaxed to ±0.3m+(C-0.7)×0.2m.

8. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Including: The power chassis pre-control logic of the cross-domain pre-control module includes: When the acceleration intention is predicted in a congestion scenario and C≥0.75, an instruction is sent to the TCU to command the gear to be pre-switched to a lower gear and the clutch half-linkage stroke to be adjusted to 30%; Also including, sending an instruction to the ECU to command the fuel injection advance angle to be increased by 2°.

9. The predictive vehicle control system of multimodal awareness recognition of claim 1, wherein, Including: The scene adaptive threshold adjustment rule includes: The comprehensive confidence trigger threshold for lane changing intention in highway scenario is C≥0.7; The comprehensive confidence trigger threshold for power pre-regulation in urban congestion scenario is C≥0.8; Also including: The failure fallback learning function includes: When the actual operation is not detected after 5 consecutive pre-control triggers, the confidence threshold in the current scenario is automatically increased by 0.05, and the multi-modal data such as light, noise, gesture type in the scenario is recorded for weight algorithm optimization.

10. A predictive vehicle control method of multi-modal awareness recognition, characterized by, The predictive vehicle control method of multi-modal awareness recognition includes: S1: Multi-modal data acquisition and dynamic weight distribution; An infrared camera is used to collect driver eye images, and a Transformer model is used to output the line of sight focus coordinates and confidence C1, and a light sensor is used to detect the light intensity L; An RGB camera is used to collect hand movements, and a spatio-temporal graph neural network ST-GCN is used to output gesture types and confidence C2, and an IMU is used to detect hand movement acceleration a; A microphone array is used to collect voice instructions, and a CTC model is used to output semantic text and confidence C3, and a noise sensor is used to detect environmental noise N; Among them, the confidence values of C1, C2 and C3 are between 0 and 1; The rule of dynamically distributing weights according to environmental parameters includes: W1+W2+W3=1; Among them, W1 is the line of sight weight, W2 is the gesture weight, and W3 is the voice weight; If L>5000lux, then W1=0.2, W2=0.5, and W3=0.3; If N>65dB, then W1=0.4, W2=0.5, and W3=0.1; If L≤5000 lux and N≤65 dB, then W1=0.35, W2=0.35, W3=0.3; S2: Time-space synchronization verification and confidence level decision-making; Calculate the Euclidean distance D between the line-of-sight focus and the gesture target area; When D<150px, determine space matching; The time verification includes calculating the time difference Δt between the line-of-sight focus stabilization time and the gesture action start time; When Δt<200ms, determine time synchronization; When space matching and time synchronization, enter the comprehensive confidence level calculation link; The comprehensive confidence level C=W1×C1+W2×C2+W3×C3, and the grading rules include: When C≥0.8, determine high confidence level; Under high confidence level, directly trigger pre-control; When 0.6≤C<0.8, determine medium confidence level; Under medium confidence level, start continuous 3-frame timing verification; If all 3 frames meet time-space synchronization, determine validity; When C<0.6, determine low confidence level; Under low confidence level, suppress pre-control; S3: Cross-domain pre-control instruction output; When predicting air conditioning adjustment intention and meeting confidence level requirements, highlight the air conditioning control interface through the vehicle IVI system before the driver's finger reaches the center control screen, and preload the preset common adjustment items; The program of highlighting the air conditioning control interface is started 150-200ms before the driver's finger reaches the center control screen; Each time the highlight display increases the original brightness by 30%; The preset common adjustment items include temperature and air volume sliders; When the driver's finger does not reach the center control screen for more than a preset delay length, the brightness returns to the non-highlighted state; When predicting lane change intention and meeting confidence level requirements, send instructions to the BSD system to expand the detection range to 4m+(C-0.7)×2m and increase the sensitivity by 20%; Also include sending instructions to the LKA system to relax the lateral intervention threshold to ±0.3m+(C-0.7)×0.2m; When predicting acceleration intention in a congested scenario and C≥0.75, send instructions to the TCU to pre-switch the gear to the next lower gear and adjust the clutch half-link travel to 30%; Also include sending instructions to the ECU to increase the injection advance angle by 2°; S4: Scene-adaptive threshold adjustment and failure fallback learning; The comprehensive confidence level trigger threshold for lane change intention in highway scenarios is C≥0.7; The comprehensive confidence level trigger threshold for power pre-adjustment in urban congestion scenarios is C≥0.8; When pre-control is triggered continuously for 5 times without detecting actual operation, automatically increase the confidence level threshold in the current scenario by 0.05, and record the multi-modal data such as illumination, noise, and gesture type in this scenario for weight algorithm optimization.

Images