Gesture intelligent recognition processing method for vehicle-mounted fast charging

Abstract

The application provides a kind of gesture intelligent recognition processing method of vehicle-mounted fast charging, vehicle state information is acquired in real time based on CAN bus, and dynamically adjusts sensor weight according to environmental conditions, improves data acquisition accuracy using hardware level time synchronization and multi-sensor data fusion technology, combines background difference with deep learning model to output accurate gesture mask, and predicts gesture trajectory, extracts the space-time characteristics of gesture and encodes, adopts Kalman filter to compensate inertia influence, classifies users, responds to gesture instructions in a personalized way, predefines gesture instruction table to ensure operation safety in combination with confidence verification and gear check, dynamically adjusts charging power according to gesture intensity, and establishes overload protection mechanism, instrument panel displays charging information and highlights gesture area through green frame, provides voice feedback, adjusts model parameters daily, adds noise to generate adversarial samples, evaluates and updates the model comprehensively every month, so as to continuously improve the robustness and user experience of the system.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics technology, and in particular to a gesture intelligent recognition processing method for in-vehicle fast charging. Background Technology

[0002] With the popularization of electric vehicles, fast charging technology has become one of the key factors in improving user experience. At the same time, the development of intelligent interaction technology has brought new possibilities to in-vehicle systems, especially the application of gesture recognition technology. It can not only provide a more intuitive and convenient human-computer interaction method, but also reduce the risk of driver distraction during driving.

[0003] However, existing gesture recognition systems face many challenges when applied to in-vehicle environments. Frequent changes in lighting inside vehicles, such as entering and exiting tunnels, vibrations, and external interference such as direct sunlight can affect the accuracy of camera-captured gestures. To improve the robustness and accuracy of gesture recognition, it is often necessary to fuse data from multiple sensors, such as cameras and millimeter-wave radar. However, achieving efficient time synchronization and spatial alignment is a technical challenge.

[0004] Different users have significantly different needs for fast charging. For example, aggressive users may prefer fast charging, while conservative users are more concerned about battery life. Therefore, a system that can automatically adjust according to user behavior patterns is needed. When performing gesture control operations, the stability and safety of the system must be ensured to avoid safety hazards caused by misoperation. Summary of the Invention

[0005] The purpose of this invention is to provide a gesture intelligent recognition processing method for in-vehicle fast charging.

[0006] The problem this invention aims to solve is to address the issues of low accuracy, poor user experience, and insufficient security in existing gesture recognition technologies. By collecting and fusing multi-sensor data in real time, dynamically adjusting sensor weights based on vehicle status, performing gesture segmentation and feature extraction, and optimizing charging operations according to user behavior patterns, an overload protection mechanism is employed to ensure charging safety. Based on daily usage data, the invention also features self-learning and model updates to adapt to the personalized needs of different users.

[0007] A gesture intelligent recognition processing method for in-vehicle fast charging, the technical solution of which is as follows:

[0008] S1: Based on the CAN bus, the vehicle speed, acceleration, steering wheel angle, gear status and remaining battery power are read in real time to construct the vehicle state vector, and dynamic weight allocation is performed based on rules. Combined with light intensity, temperature and vehicle status, the sensor weights are output.

[0009] S2: Based on hardware-level time synchronization and calibration matrix, it performs spatiotemporal synchronous acquisition of multi-sensor data and performs joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared and visible light fusion.

[0010] S3: Perform gesture region segmentation, including background subtraction and depth segmentation, use Gaussian mixture model to dynamically update the background, use MobileNetV3 neural network with added SE Block to output gesture mask, and combine physical model of vehicle acceleration to predict gesture trajectory.

[0011] S4: Spatiotemporal feature encoding is performed based on the spatial and temporal features of gestures. PCA is used to reduce the feature dimension, the gesture coordinate system is aligned with the vehicle coordinate system, and inertia is compensated by fixed-gain Kalman filtering.

[0012] S5: Based on the neural network output feature vector of MobileNetV3 with added SE Block, K-means is used to classify users into three categories: conservative, moderate, and aggressive. Dynamic convolution kernels are used to establish a multi-task loss function for incremental learning.

[0013] S6: Predefined gesture command table, setting trigger conditions, including confidence threshold and continuity verification, and verifying gear position via CAN bus;

[0014] S7: Dynamically adjusts charging parameters, combines gesture intensity for power control, calculates gesture intensity based on gesture area, establishes overload protection mechanism, and controls the charging module via CAN bus.

[0015] S8: The dashboard displays charging power and remaining time, and the gesture recognition area is highlighted with a green box, with pre-recorded voice prompts.

[0016] S9: Every night when the vehicle is stationary, the model is fine-tuned using the day's gesture data, parameters are updated, user feedback data is collected regularly, lighting and occlusion noise are added to the user gesture data, adversarial examples are generated, and the model is adjusted monthly.

[0017] Furthermore, in S1, a vehicle state vector is constructed, dynamic weight allocation is performed based on rules, and sensor weights are output, including:

[0018] S11: Real-time reading of vehicle speed v, acceleration a, steering wheel angle θ, gear status G (including P, N, R, D), and remaining battery power E via the vehicle CAN bus interface to construct a vehicle state vector. ;

[0019] S12: Measuring light intensity based on an ambient light sensor Based on thermistor measurement of ambient temperature Define the basic weight matrix The basic weights include millimeter-wave radar, infrared sensors, and cameras, respectively. ;

[0020] S13: Adjust weights based on vehicle status. , , ;

[0021] S14: Normalize the above weights. ,in Let represent the normalized sensor weight vector.

[0022] Furthermore, in S2, based on hardware-level time synchronization and calibration matrix, multi-sensor data is acquired spatiotemporally synchronously, and joint compensation is performed based on light intensity and vehicle vibration, including:

[0023] S21: It adopts an in-vehicle Ethernet switch that supports the IEEE 1588 protocol, allocates an independent physical clock to each sensor, and records the data acquisition time through a hardware-level timestamp module. All sensors send time synchronization requests to the master control unit via Ethernet. The master control unit synchronizes the master clock to all slave devices through the "master-slave" mode of IEEE 1588. The difference between the transmission time of each sensor data frame and the reception time of the master control unit is compensated by software.

[0024] S22: Use a laser rangefinder to perform calibration when the vehicle is stationary. Fix the calibration plate in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the intrinsic parameter matrix of the camera based on Zhang Zhengyou's calibration method, calculate the extrinsic parameter matrix of the camera and millimeter-wave radar using 3D point cloud matching, and convert the camera coordinate system points to the radar coordinate system.

[0025] S23: Establish a compensation formula model , ,in Let be the coordinate point. Light intensity, The vehicle vibration amplitude is obtained based on accelerometer readings. , is the illumination compensation coefficient. , is the vibration attenuation coefficient, read in real time. and Adjust the image brightness according to the formula;

[0026] S24: Dynamically adjust weights based on light intensity. ,when hour, ,otherwise ,in The intensity values ​​of the fused image. These are the raw image intensity values ​​acquired by the visible light sensor. The raw image intensity values ​​acquired by the infrared sensor. These are the weighting coefficients for the visible light image.

[0027] Furthermore, in step S3, gesture region segmentation is performed, and gesture trajectory prediction is performed in conjunction with a physical model of vehicle acceleration, including:

[0028] S31: Use Gaussian Mixture Model (GMM) to generate a background model, update it in real time, update it every 5 frames, initially segment the foreground region through background subtraction, and use a MobileNetV3 neural network with added SE Block to output the final gesture mask.

[0029] S32: Based on the current gesture position and vehicle status, predict the gesture trajectory at time T in the future, and calculate the current gesture speed based on continuous frame difference. , ,in Let T be the coordinates of the gesture at time T. The coordinates of the current gesture position. To accelerate the vehicle, The acceleration direction is a unit vector. The prediction result is combined with the current gesture mask to generate a prediction of the future gesture region.

[0030] Furthermore, in step S4, spatiotemporal feature encoding is performed based on the spatial and temporal features of the gesture, aligning the gesture coordinate system with the vehicle coordinate system, and compensating for inertia through fixed-gain Kalman filtering, including:

[0031] S41: Use MobileNetV3 to extract the spatial features of the predicted gesture trajectory described in S3, and enhance the extracted spatial features based on SE Block;

[0032] S42: Extract the dynamic temporal features of the predicted gesture trajectory described in S3 from continuous frames, select a fixed time window, calculate the velocity features and acceleration features, and combine them with the vehicle steering wheel angle to generate a time feature vector;

[0033] S43: Spatial and temporal features are concatenated and input into a fully connected layer to generate preliminary encoded features. The final features are then output based on the spatiotemporal attention module.

[0034] S44: Based on the vehicle's state, including velocity v, acceleration a, and steering wheel angle θ, construct the transformation matrix for the future time T. , , , , where is the translational amount calculated based on the vehicle's acceleration 'a'. These are the components of acceleration a in the x and y directions, respectively. The aligned gesture position is obtained by multiplying the gesture's position in the camera coordinate system with the transformation matrix.

[0035] S45: Based on the current gesture state of the vehicle acceleration, the gesture trajectory after inertia compensation is obtained by using the gesture prediction value at a future time T with a fixed gain and the actual observation value.

[0036] Furthermore, in S5, K-means is used to classify users into three categories: conservative, moderate, and aggressive. A dynamic convolutional kernel is used to establish a multi-task loss function, including:

[0037] Based on the distance to the cluster centers, users are divided into three categories: conservative, moderate, and aggressive. Conservative is defined as low acceleration, low speed, small steering wheel angle, and stable battery power fluctuation. Moderate is defined as medium acceleration, medium speed, moderate steering wheel angle, and small battery power fluctuation. Aggressive is defined as high acceleration, high speed, large steering wheel angle, and large battery power fluctuation.

[0038] The criteria for classifying acceleration as low, medium, and high are as follows: acceleration ≤ 1.0 m / s² 2 1.0m / s 2 <Acceleration ≤ 2.0 m / s 2 Acceleration > 2.0 m / s² 2 ;

[0039] The classification of low, medium, and high speeds is based on speeds ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, and speeds > 80 km / h, respectively.

[0040] The criteria for classifying steering wheel angles as small, moderate, and large are: ≤30° for the absolute value of the angle, 30° < ≤60° for the absolute value of the angle, and >60° for the absolute value of the angle.

[0041] The criteria for classifying battery remaining power fluctuations as stable, small, and large are respectively: fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, and fluctuation > 10%.

[0042] The user features are converted into adjustment signals, a dynamic convolution kernel is calculated based on the user features, and the kernel is used to convolve the input features to generate adaptive features.

[0043] The network is optimized by jointly optimizing gesture recognition, user classification, and battery status prediction. The tasks are defined as gesture recognition, user classification, and remaining battery power prediction, respectively. The loss of each task is calculated, and the total loss is combined according to the weights. The network is then optimized through backpropagation.

[0044] Furthermore, the predefined gesture command table in S6 sets trigger conditions and verifies the gear position via the CAN bus, including:

[0045] S61: Query the instruction table based on the gesture classification results, obtain the corresponding function, and determine whether the instruction is effective by combining the user type and the remaining battery power.

[0046] A fist gesture activates the fast charging mode, which is triggered when the remaining battery level is less than 20%.

[0047] The OK gesture corresponds to the function of stopping charging / switching to normal mode, and can be triggered unconditionally.

[0048] A swipe to the right gesture corresponds to the function of increasing charging power, which is triggered when the remaining battery level is less than 50% and the user is classified as aggressive.

[0049] A swipe to the left gesture corresponds to the function of reducing charging power, which is triggered when the remaining battery level is greater than 80% and the user is classified as conservative.

[0050] S62: Check the relationship between gesture confidence and threshold. The threshold is 0.95 when the user is classified as aggressive, 0.85 when the user is classified as moderate, and 0.75 when the user is classified as conservative. Verify gesture continuity. The gesture must be recognized for 5 consecutive frames and the gesture trajectory must conform to the predefined path.

[0051] S63: Reads the vehicle gear and speed in real time via the CAN bus to determine if safety conditions are met. If they are met, the instruction is executed; otherwise, it is rejected.

[0052] Furthermore, in step S7, power control is performed in conjunction with gesture intensity. The gesture intensity is calculated based on the area of ​​the gesture region, and an overload protection mechanism is established, including:

[0053] S71: Calculate the gesture region area A based on the gesture region segmentation described in S3, and calculate the gesture movement speed based on the gesture temporal characteristics described in S4. and gesture acceleration Establish gesture strength Calculation formula , , These are the weighting coefficients, The maximum values ​​of the gesture area, gesture movement speed, and gesture acceleration, respectively;

[0054] S72: Dynamically adjusts charging power based on gesture strength and user type. , ,in This is the default charging power. This is the user type coefficient; when the user is conservative... When the user is of medium size When the user is aggressive ;

[0055] S73: Reads battery temperature and remaining battery power in real time via CAN bus to determine whether overload conditions are triggered. Overload conditions include temperature not exceeding 60 degrees Celsius and remaining battery power not exceeding 95%.

[0056] Furthermore, in S9, the model is fine-tuned using the gesture data of the day, parameters are updated, illumination and occlusion noise are added to the user gesture data, adversarial examples are generated, and the model is adjusted monthly, including:

[0057] Check if the vehicle status meets the stationary condition. If it does, load the gesture dataset for the day, including gesture images and their labels, and fine-tune the existing model using the daily accumulated gesture data.

[0058] For each gesture sample, random lighting changes and partial occlusion operations are applied to generate new samples. Based on the generative adversarial network, adversarial samples are further generated, and the data of adversarial samples are added to the training set.

[0059] A comprehensive model evaluation is conducted at the end of each month, including accuracy, recall, and F1 score. If the current model performance deteriorates, the model is retrained using all accumulated gesture data, including daily fine-tuning data, adversarial examples, and user feedback data.

[0060] The beneficial effects of this invention are: by adjusting the weights of sensors based on vehicle status and environmental conditions, it can more effectively cope with different driving conditions, improve the quality of data acquisition, and maintain high-precision data acquisition capabilities under different light intensities and vehicle vibration conditions.

[0061] By using Gaussian mixture model for background subtraction and depth segmentation, and combining it with physical model for gesture trajectory prediction, the accuracy and real-time performance of gesture recognition are improved. By fusing spatial and temporal features and aligning coordinate systems, the robustness of gesture recognition is enhanced, and it can work effectively even in vehicle motion.

[0062] Users are categorized into conservative, moderate, and aggressive driving types based on their driving behavior, thus providing a more personalized and safer charging experience. The predefined gesture command list allows users to control the charging process through simple gesture operations, improving convenience.

[0063] By converting gesture intensity into charging power adjustment parameters, an intuitive and flexible charging power control mechanism is achieved. By monitoring battery temperature and remaining battery power in real time, safety hazards caused by overcharging are prevented.

[0064] By continuously fine-tuning the model using data accumulated daily, and by adding noise to generate adversarial examples to enhance the model's generalization ability, the performance stability of the model is guaranteed for long-term use. Attached Figure Description

[0065] Figure 1 This is a flowchart of a gesture intelligent recognition processing method for in-vehicle fast charging. Detailed Implementation

[0066] The present invention will be further described clearly and completely below, but the scope of protection of the present invention is not limited thereto.

[0067] A gesture intelligent recognition processing method for in-vehicle fast charging, the technical solution of which is as follows:

[0068] S1: Based on the CAN bus, the vehicle speed, acceleration, steering wheel angle, gear status and remaining battery power are read in real time to construct the vehicle state vector, and dynamic weight allocation is performed based on rules. Combined with light intensity, temperature and vehicle status, the sensor weights are output.

[0069] S2: Based on hardware-level time synchronization and calibration matrix, it performs spatiotemporal synchronous acquisition of multi-sensor data and performs joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared and visible light fusion.

[0070] S3: Perform gesture region segmentation, including background subtraction and depth segmentation, use Gaussian mixture model to dynamically update the background, use MobileNetV3 neural network with added SE Block to output gesture mask, and combine physical model of vehicle acceleration to predict gesture trajectory.

[0071] S4: Spatiotemporal feature encoding is performed based on the spatial and temporal features of gestures. PCA is used to reduce the feature dimension, the gesture coordinate system is aligned with the vehicle coordinate system, and inertia is compensated by fixed-gain Kalman filtering.

[0072] S5: Based on the neural network output feature vector of MobileNetV3 with added SE Block, K-means is used to classify users into three categories: conservative, moderate, and aggressive. Dynamic convolution kernels are used to establish a multi-task loss function for incremental learning.

[0073] S6: Predefined gesture command table, setting trigger conditions, including confidence threshold and continuity verification, and verifying gear position via CAN bus;

[0074] S7: Dynamically adjusts charging parameters, combines gesture intensity for power control, calculates gesture intensity based on gesture area, establishes overload protection mechanism, and controls the charging module via CAN bus.

[0075] S8: The dashboard displays charging power and remaining time, and the gesture recognition area is highlighted with a green box, with pre-recorded voice prompts.

[0076] S9: Every night when the vehicle is stationary, the model is fine-tuned using the day's gesture data, parameters are updated, user feedback data is collected regularly, lighting and occlusion noise are added to the user gesture data, adversarial examples are generated, and the model is adjusted monthly.

[0077] refer to Figure 1 The diagram shown is a flowchart of a gesture intelligent recognition processing method for in-vehicle fast charging.

[0078] Furthermore, in S1, a vehicle state vector is constructed, dynamic weight allocation is performed based on rules, and sensor weights are output, including:

[0079] S11: Real-time reading of vehicle speed v, acceleration a, steering wheel angle θ, gear status G (including P, N, R, D), and remaining battery power E via the vehicle CAN bus interface to construct a vehicle state vector. ;

[0080] S12: Measuring light intensity based on an ambient light sensor Based on thermistor measurement of ambient temperature Define the basic weight matrix The basic weights include millimeter-wave radar, infrared sensors, and cameras, respectively. ;

[0081] S13: Adjust weights based on vehicle status. , , ;

[0082] S14: Normalize the above weights. ,in Let represent the normalized sensor weight vector.

[0083] Furthermore, in S2, based on hardware-level time synchronization and calibration matrix, multi-sensor data is acquired spatiotemporally synchronously, and joint compensation is performed based on light intensity and vehicle vibration, including:

[0084] S21: It adopts an in-vehicle Ethernet switch that supports the IEEE 1588 protocol, allocates an independent physical clock to each sensor, and records the data acquisition time through a hardware-level timestamp module. All sensors send time synchronization requests to the master control unit via Ethernet. The master control unit synchronizes the master clock to all slave devices through the "master-slave" mode of IEEE 1588. The difference between the transmission time of each sensor data frame and the reception time of the master control unit is compensated by software.

[0085] S22: Use a laser rangefinder to perform calibration when the vehicle is stationary. Fix the calibration plate in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the intrinsic parameter matrix of the camera based on Zhang Zhengyou's calibration method, calculate the extrinsic parameter matrix of the camera and millimeter-wave radar using 3D point cloud matching, and convert the camera coordinate system points to the radar coordinate system.

[0086] S23: Establish a compensation formula model , ,in Let be the coordinate point. Light intensity, The vehicle vibration amplitude is obtained based on accelerometer readings. , is the illumination compensation coefficient. , is the vibration attenuation coefficient, read in real time. and Adjust the image brightness according to the formula;

[0087] S24: Dynamically adjust weights based on light intensity. ,when hour, ,otherwise ,in The intensity values ​​of the fused image. These are the raw image intensity values ​​acquired by the visible light sensor. The raw image intensity values ​​acquired by the infrared sensor. These are the weighting coefficients for the visible light image.

[0088] Furthermore, in step S3, gesture region segmentation is performed, and gesture trajectory prediction is performed in conjunction with a physical model of vehicle acceleration, including:

[0089] S31: Use Gaussian Mixture Model (GMM) to generate background model, update in real time, update once every 5 frames, initially segment the foreground region through background subtraction, and use a MobileNetV3 neural network with added SE Block to output the final gesture mask;

[0090] S32: Based on the current gesture position and vehicle status, predict the gesture trajectory at time T in the future, and calculate the current gesture speed based on continuous frame difference. , ,in Let T be the coordinates of the gesture at time T. The coordinates of the current gesture position. To accelerate the vehicle, The acceleration direction is a unit vector. The prediction result is combined with the current gesture mask to generate a prediction of the future gesture region.

[0091] Furthermore, in step S4, spatiotemporal feature encoding is performed based on the spatial and temporal features of the gesture, aligning the gesture coordinate system with the vehicle coordinate system, and compensating for inertia through fixed-gain Kalman filtering, including:

[0092] S41: Use MobileNetV3 to extract the spatial features of the predicted gesture trajectory described in S3, and enhance the extracted spatial features based on SE Block;

[0093] S42: Extract the dynamic temporal features of the predicted gesture trajectory described in S3 from continuous frames, select a fixed time window, calculate the velocity features and acceleration features, and combine them with the vehicle steering wheel angle to generate a time feature vector;

[0094] S43: Spatial and temporal features are concatenated and input into a fully connected layer to generate preliminary encoded features. The final features are then output based on the spatiotemporal attention module.

[0095] S44: Based on the vehicle's state, including velocity v, acceleration a, and steering wheel angle θ, construct the transformation matrix for the future time T. , , , , where is the translational amount calculated based on the vehicle's acceleration 'a'. These are the components of acceleration a in the x and y directions, respectively. The aligned gesture position is obtained by multiplying the gesture's position in the camera coordinate system with the transformation matrix.

[0096] S45: Based on the current gesture state of the vehicle acceleration, the gesture trajectory after inertia compensation is obtained by using the gesture prediction value at a future time T with a fixed gain and the actual observation value.

[0097] Furthermore, in S5, K-means is used to classify users into three categories: conservative, moderate, and aggressive. A dynamic convolutional kernel is used to establish a multi-task loss function, including:

[0098] Based on the distance to the cluster centers, users are divided into three categories: conservative, moderate, and aggressive. Conservative is defined as low acceleration, low speed, small steering wheel angle, and stable battery power fluctuation. Moderate is defined as medium acceleration, medium speed, moderate steering wheel angle, and small battery power fluctuation. Aggressive is defined as high acceleration, high speed, large steering wheel angle, and large battery power fluctuation.

[0099] The criteria for classifying acceleration as low, medium, and high are as follows: acceleration ≤ 1.0 m / s² 2 1.0m / s 2 <Acceleration ≤ 2.0 m / s 2 Acceleration > 2.0 m / s² 2 ;

[0100] The classification of low, medium, and high speeds is based on speeds ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, and speeds > 80 km / h, respectively.

[0101] The criteria for classifying steering wheel angles as small, moderate, and large are: ≤30° for the absolute value of the angle, 30° < ≤60° for the absolute value of the angle, and >60° for the absolute value of the angle.

[0102] The criteria for classifying battery remaining power fluctuations as stable, small, and large are respectively: fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, and fluctuation > 10%.

[0103] The user features are converted into adjustment signals, a dynamic convolution kernel is calculated based on the user features, and the kernel is used to convolve the input features to generate adaptive features.

[0104] The network is optimized by jointly optimizing gesture recognition, user classification, and battery status prediction. The tasks are defined as gesture recognition, user classification, and remaining battery power prediction, respectively. The loss of each task is calculated, and the total loss is combined according to the weights. The network is then optimized through backpropagation.

[0105] Furthermore, the predefined gesture command table in S6 sets trigger conditions and verifies the gear position via the CAN bus, including:

[0106] S61: Query the instruction table based on the gesture classification results, obtain the corresponding function, and determine whether the instruction is effective by combining the user type and the remaining battery power.

[0107] A fist gesture activates the fast charging mode, which is triggered when the remaining battery level is less than 20%.

[0108] The OK gesture corresponds to the function of stopping charging / switching to normal mode, and can be triggered unconditionally.

[0109] A swipe to the right gesture corresponds to the function of increasing charging power, which is triggered when the remaining battery level is less than 50% and the user is classified as aggressive.

[0110] A swipe to the left gesture corresponds to the function of reducing charging power, which is triggered when the remaining battery level is greater than 80% and the user is classified as conservative.

[0111] S62: Check the relationship between gesture confidence and threshold. The threshold is 0.95 when the user is classified as aggressive, 0.85 when the user is classified as moderate, and 0.75 when the user is classified as conservative. Verify gesture continuity. The gesture must be recognized for 5 consecutive frames and the gesture trajectory must conform to the predefined path.

[0112] S63: Reads the vehicle gear and speed in real time via the CAN bus to determine if safety conditions are met. If they are met, the instruction is executed; otherwise, it is rejected.

[0113] Furthermore, in step S7, power control is performed in conjunction with gesture intensity. The gesture intensity is calculated based on the area of ​​the gesture region, and an overload protection mechanism is established, including:

[0114] S71: Calculate the gesture region area A based on the gesture region segmentation described in S3, and calculate the gesture movement speed based on the gesture temporal characteristics described in S4. and gesture acceleration Establish gesture strength Calculation formula , , These are the weighting coefficients, The maximum values ​​of the gesture area, gesture movement speed, and gesture acceleration, respectively;

[0115] S72: Dynamically adjusts charging power based on gesture strength and user type. , ,in This is the default charging power. This is the user type coefficient; when the user is conservative... When the user is of medium size When the user is aggressive ;

[0116] S73: Reads battery temperature and remaining battery power in real time via CAN bus to determine whether overload conditions are triggered. Overload conditions include temperature not exceeding 60 degrees Celsius and remaining battery power not exceeding 95%.

[0117] Furthermore, in S9, the model is fine-tuned using the gesture data of the day, parameters are updated, illumination and occlusion noise are added to the user gesture data, adversarial examples are generated, and the model is adjusted monthly, including:

[0118] Check if the vehicle status meets the stationary condition. If it does, load the gesture dataset for the day, including gesture images and their labels, and fine-tune the existing model using the daily accumulated gesture data.

[0119] For each gesture sample, random lighting changes and partial occlusion operations are applied to generate new samples. Based on the generative adversarial network, adversarial samples are further generated, and the data of adversarial samples are added to the training set.

[0120] A comprehensive model evaluation is conducted at the end of each month, including accuracy, recall, and F1 score. If the current model performance deteriorates, the model is retrained using all accumulated gesture data, including daily fine-tuning data, adversarial examples, and user feedback data.

[0121] This invention provides a gesture intelligent recognition processing method for in-vehicle fast charging. It acquires vehicle status information in real time via the CAN bus and dynamically adjusts sensor weights according to environmental conditions. Hardware-level time synchronization and multi-sensor data fusion technologies improve data acquisition accuracy. Background subtraction and deep learning models are combined to output accurate gesture masks and predict gesture trajectories. The spatiotemporal features of the gestures are extracted and encoded. Kalman filtering compensates for inertial effects. Users are categorized and gesture commands are responded to in a personalized manner. A predefined gesture command table, combined with confidence verification and gear position checks, ensures operational safety. Charging power is dynamically adjusted based on gesture intensity, and an overload protection mechanism is established. Charging information is displayed on the dashboard, and the gesture area is highlighted with a green box. Voice feedback is provided. Model parameters are fine-tuned daily, noise is added to generate adversarial examples, and the model is comprehensively evaluated and updated monthly, thereby continuously improving the system's robustness and user experience.

Claims

1. A gesture intelligent recognition processing method for vehicle fast charging, characterized in that, include: S1: Based on the CAN bus, the vehicle speed, acceleration, steering wheel angle, gear status and remaining battery power are read in real time to construct the vehicle state vector, and dynamic weight allocation is performed based on rules. Combined with light intensity, temperature and vehicle status, the sensor weights are output. S2: Based on hardware-level time synchronization and calibration matrix, it performs spatiotemporal synchronous acquisition of multi-sensor data and performs joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared and visible light fusion. S3: Perform gesture region segmentation, including background subtraction and depth segmentation, use Gaussian mixture model to dynamically update the background, use MobileNetV3 neural network with added SE Block to output gesture mask, and combine physical model of vehicle acceleration to predict gesture trajectory. S4: Spatiotemporal feature encoding is performed based on the spatial and temporal features of gestures. PCA is used to reduce the feature dimension, the gesture coordinate system is aligned with the vehicle coordinate system, and inertia is compensated by fixed-gain Kalman filtering. S5: Based on the neural network output feature vector of MobileNetV3 with added SE Block, K-means is used to classify users into three categories: conservative, moderate, and aggressive. Dynamic convolution kernels are used to establish a multi-task loss function for incremental learning. S6: Predefined gesture command table, setting trigger conditions, including confidence threshold and continuity verification, and verifying gear position via CAN bus; S7: Dynamically adjusts charging parameters, combines gesture intensity for power control, calculates gesture intensity based on gesture area, establishes overload protection mechanism, and controls the charging module via CAN bus. S8: The dashboard displays charging power and remaining time, and the gesture recognition area is highlighted with a green box, with pre-recorded voice prompts. S9: Every night when the vehicle is stationary, the model is fine-tuned using the day's gesture data, parameters are updated, user feedback data is collected regularly, lighting and occlusion noise are added to the user gesture data, adversarial examples are generated, and the model is adjusted monthly.

2. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, In step S1, a vehicle state vector is constructed, dynamic weight allocation is performed based on rules, and sensor weights are output, including: S11: Real-time reading of vehicle speed v, acceleration a, steering wheel angle θ, gear status G (including P, N, R, D), and remaining battery power E via the vehicle CAN bus interface to construct a vehicle state vector. ; S12: Measuring light intensity based on an ambient light sensor Based on thermistor measurement of ambient temperature Define the basic weight matrix The basic weights include millimeter-wave radar, infrared sensors, and cameras, respectively. ; S13: Adjust weights based on vehicle status. , , ; S14: Normalize the above weights. ,in Let represent the normalized sensor weight vector.

3. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, S2, based on hardware-level time synchronization and calibration matrix, performs spatiotemporal synchronous acquisition of multi-sensor data and performs joint compensation based on light intensity and vehicle vibration, including: S21: It adopts an in-vehicle Ethernet switch that supports the IEEE 1588 protocol, allocates an independent physical clock to each sensor, and records the data acquisition time through a hardware-level timestamp module. All sensors send time synchronization requests to the master control unit via Ethernet. The master control unit synchronizes the master clock to all slave devices through the "master-slave" mode of IEEE 1588. The difference between the transmission time of each sensor data frame and the reception time of the master control unit is compensated by software. S22: Use a laser rangefinder to perform calibration when the vehicle is stationary. Fix the calibration plate in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the intrinsic parameter matrix of the camera based on Zhang Zhengyou's calibration method, calculate the extrinsic parameter matrix of the camera and millimeter-wave radar using 3D point cloud matching, and convert the camera coordinate system points to the radar coordinate system. S23: Establish a compensation formula model , ,in Let be the coordinate point. Light intensity, The vehicle vibration amplitude is obtained based on accelerometer readings. , is the illumination compensation coefficient. , is the vibration attenuation coefficient, read in real time. and Adjust the image brightness according to the formula; S24: Dynamically adjust weights based on light intensity. ,when hour, ,otherwise ,in The intensity values ​​of the fused image. These are the raw image intensity values ​​acquired by the visible light sensor. The raw image intensity values ​​acquired by the infrared sensor. These are the weighting coefficients for the visible light image.

4. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, In step S3, gesture region segmentation is performed, and gesture trajectory prediction is performed based on a physical model of vehicle acceleration, including: S31: Use Gaussian Mixture Model (GMM) to generate background model, update in real time, update once every 5 frames, initially segment the foreground region through background subtraction, and use a MobileNetV3 neural network with added SE Block to output the final gesture mask; S32: Based on the current gesture position and vehicle status, predict the gesture trajectory at time T in the future, and calculate the current gesture speed based on continuous frame difference. , ,in Let T be the coordinates of the gesture at time T. The coordinates of the current gesture position. To accelerate the vehicle, The acceleration direction is a unit vector. The prediction result is combined with the current gesture mask to generate a prediction of the future gesture region.

5. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, In step S4, spatiotemporal feature encoding is performed based on the spatial and temporal features of the gesture, aligning the gesture coordinate system with the vehicle coordinate system, and compensating for inertia through fixed-gain Kalman filtering, including: S41: Use MobileNetV3 to extract the spatial features of the gesture trajectory prediction described in S3, and enhance the extracted spatial features based on SE Block; S42: Extract the dynamic temporal features of the gesture trajectory prediction described in S3 from continuous frames, select a fixed time window, calculate the velocity features and acceleration features, and combine them with the vehicle steering wheel angle to generate a time feature vector; S43: Spatial and temporal features are concatenated and input into a fully connected layer to generate preliminary encoded features. The final features are then output based on the spatiotemporal attention module. S44: Based on the vehicle's state, including velocity v, acceleration a, and steering wheel angle θ, construct the transformation matrix for the future time T. , , , , where is the translational amount calculated based on the vehicle's acceleration 'a'. These are the components of acceleration a in the x and y directions, respectively. The aligned gesture position is obtained by multiplying the gesture's position in the camera coordinate system with the transformation matrix. S45: Based on the current gesture state of the vehicle acceleration, the gesture trajectory after inertia compensation is obtained by using the gesture prediction value at a future time T with a fixed gain and the actual observation value.

6. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, In S5, K-means is used to classify users into three categories: conservative, moderate, and aggressive. A dynamic convolutional kernel is used to establish a multi-task loss function, including: Based on the distance to the cluster centers, users are divided into three categories: conservative, moderate, and aggressive. Conservative is defined as low acceleration, low speed, small steering wheel angle, and stable battery power fluctuation. Moderate is defined as medium acceleration, medium speed, moderate steering wheel angle, and small battery power fluctuation. Aggressive is defined as high acceleration, high speed, large steering wheel angle, and large battery power fluctuation. The criteria for classifying acceleration as low, medium, and high are as follows: acceleration ≤ 1.0 m / s² 2 1.0m / s 2 <Acceleration ≤ 2.0 m / s 2 Acceleration > 2.0 m / s² 2 ; The classification of low, medium, and high speeds is based on speeds ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, and speeds > 80 km / h, respectively. The criteria for classifying steering wheel angles as small, moderate, and large are: ≤30° for the absolute value of the angle, 30° < ≤60° for the absolute value of the angle, and >60° for the absolute value of the angle. The criteria for classifying battery remaining power fluctuations as stable, small, and large are respectively: fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, and fluctuation > 10%. The user features are converted into adjustment signals, a dynamic convolution kernel is calculated based on the user features, and the kernel is used to convolve the input features to generate adaptive features. The network is optimized by jointly optimizing gesture recognition, user classification, and battery status prediction. The tasks are defined as gesture recognition, user classification, and remaining battery power prediction, respectively. The loss of each task is calculated, and the total loss is combined according to the weights. The network is then optimized through backpropagation.

7. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, The S6 predefined gesture command table sets trigger conditions and verifies the gear position via the CAN bus, including: S61: Query the instruction table based on the gesture classification results, obtain the corresponding function, and determine whether the instruction is effective by combining the user type and the remaining battery power. A fist gesture activates the fast charging mode, which is triggered when the remaining battery level is less than 20%. The OK gesture corresponds to the function of stopping charging / switching to normal mode, and can be triggered unconditionally. A swipe to the right gesture corresponds to the function of increasing charging power, which is triggered when the remaining battery level is less than 50% and the user is classified as aggressive. A swipe to the left gesture corresponds to the function of reducing charging power, which is triggered when the remaining battery level is greater than 80% and the user is classified as conservative. S62: Check the relationship between gesture confidence and threshold. The threshold is 0.95 when the user is classified as aggressive, 0.85 when the user is classified as moderate, and 0.75 when the user is classified as conservative. Verify gesture continuity. The gesture must be recognized for 5 consecutive frames and the gesture trajectory must conform to the predefined path. S63: Reads the vehicle gear and speed in real time via the CAN bus to determine if safety conditions are met. If they are met, the instruction is executed; otherwise, it is rejected.

8. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, In S7, power control is performed in conjunction with gesture intensity, which is calculated based on the gesture area. An overload protection mechanism is established, including: S71: Calculate the gesture region area A based on the gesture region segmentation described in S3, and calculate the gesture movement speed based on the gesture temporal characteristics described in S4. and gesture acceleration Establish gesture strength Calculation formula , , These are the weighting coefficients, The maximum values ​​of the gesture area, gesture movement speed, and gesture acceleration, respectively; S72: Dynamically adjusts charging power based on gesture strength and user type. , ,in This is the default charging power. This is the user type coefficient; when the user is conservative... When the user is of medium size When the user is aggressive ; S73: Reads battery temperature and remaining battery power in real time via CAN bus to determine whether overload conditions are triggered. Overload conditions include temperature not exceeding 60 degrees Celsius and remaining battery power not exceeding 95%.

9. The gesture intelligent recognition processing method for vehicle fast charging as described in claim 1, characterized in that, The S9 method uses daily gesture data to fine-tune the model, update parameters, add illumination and occlusion noise to user gesture data, generate adversarial examples, and adjust the model monthly, including: Check if the vehicle status meets the stationary condition. If it does, load the gesture dataset for the day, including gesture images and their labels, and fine-tune the existing model using the daily accumulated gesture data. For each gesture sample, random lighting changes and partial occlusion operations are applied to generate new samples. Based on the generative adversarial network, adversarial samples are further generated, and the data of adversarial samples are added to the training set. A comprehensive model evaluation is conducted at the end of each month, including accuracy, recall, and F1 score. If the current model performance deteriorates, the model is retrained using all accumulated gesture data, including daily fine-tuning data, adversarial examples, and user feedback data.

Images