Intelligent gesture recognition processing method for vehicle-mounted fast charging

Abstract

The invention provides an intelligent gesture recognition processing method for vehicle-mounted fast charging, which is characterized in that vehicle state information is acquired in real time based on a CAN bus, the weight of a sensor is dynamically adjusted according to environmental conditions, and the data acquisition precision is improved by using hardware-level time synchronization and multi-sensor data fusion technologies; a precise gesture mask is output by combining background difference and a deep learning model, a gesture track is predicted, spatial and temporal features of gestures are extracted and coded, inertial influence is compensated by adopting Kalman filtering, users are classified, gesture instructions are responded in a personalized mode, a predefined gesture instruction list is combined with confidence verification and gear check, and operation safety is ensured. Charging power is dynamically adjusted according to gesture intensity, an overload protection mechanism is established, an instrument panel displays charging information, a gesture area is highlighted through a green frame, voice feedback is provided, model parameters are finely adjusted every day, noise is added to generate confrontation samples, the model is comprehensively evaluated and updated every month, and therefore the robustness of the system and user experience are continuously improved.

Description

Technical Field

[0001] The present invention relates to the field of automotive electronics technology, and particularly to a method for intelligent gesture recognition and processing of in-vehicle fast charging. Background Art

[0002] With the popularization of electric vehicles, fast charging technology has become one of the key factors to improve user experience. At the same time, the development of intelligent interaction technology has brought new possibilities to in-vehicle systems. In particular, the application of gesture recognition technology can not only provide a more intuitive and convenient man-machine interaction method, but also reduce the risk of driver distraction during driving.

[0003] However, the existing gesture recognition systems face many challenges when applied to in-vehicle environments. Factors such as frequent changes in vehicle interior light when entering and exiting tunnels, vibrations, and external interferences such as direct sunlight will affect the accuracy of the camera to capture gestures. In order to improve the robustness and accuracy of gesture recognition, it is often necessary to fuse data from multiple sensors, such as cameras, millimeter-wave radars, etc. However, how to achieve efficient time synchronization and spatial alignment is a technical difficulty.

[0004] There are significant differences in the fast charging needs of different users. For example, aggressive users may be more inclined to fast charging, while conservative users are more concerned about battery life. Therefore, a system that can automatically adjust according to user behavior patterns is needed. When performing gesture control operations, it is necessary to ensure the stability and safety of the system to avoid potential safety hazards caused by misoperations. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for intelligent gesture recognition and processing of in-vehicle fast charging.

[0006] The problems to be solved by the present invention are: aiming to solve the problems of low gesture recognition accuracy, poor user experience and insufficient safety in the prior art. By collecting and fusing multi-sensor data in real time, dynamically adjusting the sensor weights in combination with the vehicle state, performing gesture segmentation and feature extraction, and classifying and optimizing the charging operation according to the user behavior pattern, while adopting an overload protection mechanism to ensure charging safety, and self-learning and model updating based on daily usage data to adapt to the personalized needs of different users.

[0007] A method for intelligent gesture recognition and processing of in-vehicle fast charging, the adopted technical solution is as follows: S1: Based on the CAN bus, read the vehicle speed, acceleration, steering wheel angle, gear state, and remaining battery power in real time, construct a vehicle state vector, perform dynamic weight allocation based on rules, and combine the light intensity, temperature and vehicle state to output the sensor weights; S2: Based on hardware-level time synchronization and calibration matrix, perform spatio-temporal synchronous acquisition of multi-sensor data, and perform joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared and visible light fusion; S3: Perform gesture area segmentation, including background difference and depth segmentation, dynamically update the background using Gaussian mixture model, output gesture masks using the neural network of MobileNetV3 with added SE Block, and predict gesture trajectories by combining the physical model of vehicle acceleration; S4: Perform spatio-temporal feature encoding based on the spatial and temporal features of gestures, reduce the feature dimension using PCA, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through fixed-gain Kalman filter; S5: Output feature vectors using the neural network of MobileNetV3 with added SE Block, classify users into three categories: conservative, medium, and radical using K-means, use dynamic convolution kernels, establish a multi-task loss function, and perform incremental learning; S6: Pre-define a gesture instruction table, set trigger conditions, including confidence threshold and continuity verification, and verify the gear position through the CAN bus; S7: Dynamically adjust charging parameters, perform power control in combination with gesture intensity, calculate gesture intensity through the gesture area, establish an overload protection mechanism, and control the charging module through the CAN bus; S8: Display the charging power and remaining time on the dashboard, highlight the gesture recognition area with a green box, and give a pre-recorded voice prompt; S9: Every night when the vehicle is stationary, fine-tune the model using the gesture data of the day, update the parameters, regularly collect user feedback data, add light and occlusion noise to the user gesture data to generate adversarial samples, and adjust the model monthly.

[0008] Furthermore, in the above S1, construct a vehicle state vector, perform dynamic weight allocation based on rules, and output sensor weights, including: S11: Real-time read the vehicle speed v, acceleration a, steering wheel angle θ, gear state G including P, N, R, D, and remaining battery power E through the in-vehicle CAN bus interface, and construct a vehicle state vector ; S12: Measure the light intensity based on the ambient light sensor , measure the ambient temperature based on the thermistor , define a basic weight matrix , including the basic weights of millimeter-wave radar, infrared sensor, and camera, which are respectively ; S13: Adjust the weights according to the vehicle state, , , ; S14: Normalize the above weights, , where represents the sensor weight vector after normalization.

[0009] Furthermore, in S2, based on hardware-level time synchronization and calibration matrix, spatio-temporal synchronous acquisition of multi-sensor data is performed, and joint compensation is performed based on light intensity and vehicle vibration, including: S21: Use a vehicle-mounted Ethernet switch that supports the IEEE 1588 protocol to assign an independent physical clock to each sensor. The sensor records the data acquisition time through a hardware-level timestamp module. All sensors send time synchronization requests to the main control unit through Ethernet. The main control unit synchronizes the master clock to all slave devices through the "master-slave" mode of IEEE 1588. The difference between the transmission time of each sensor data frame and the reception time of the main control unit is compensated by software; S22: Use a laser rangefinder to perform calibration when the vehicle is stationary. Fix the calibration board in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the internal parameter matrix of the camera based on Zhang Zhengyou calibration method, calculate the external parameter matrix of the camera and millimeter-wave radar using three-dimensional point cloud matching, and convert the points in the camera coordinate system to the radar coordinate system; S23: Establish a compensation formula model , , where is the coordinate point, is the light intensity, is the vehicle vibration amplitude, obtained based on the accelerometer, , is the light compensation coefficient, , is the vibration attenuation coefficient, read in real time and , and adjust the image brightness according to the formula; S24: Dynamically adjust the weights according to the light intensity, , when , , otherwise , where is the fused image intensity value, is the original image intensity value collected by the visible light sensor, is the original image intensity value collected by the infrared sensor, is the weight coefficient of the visible light image.

[0010] Furthermore, in S3, gesture area segmentation is performed, and gesture trajectory prediction is performed in combination with the physical model of vehicle acceleration, including: S31: Generate a background model using the GMM Gaussian mixture model, which is updated in real time every 5 frames. Initially segment the foreground region through background difference, and use a neural network of MobileNetV3 with an added SE Block to output the final gesture mask; S32: Predict the gesture trajectory at the future T moment based on the current gesture position and vehicle state, and calculate the current gesture speed based on the continuous frame difference , , where is the gesture coordinate at the T moment, is the current gesture position coordinate, is the vehicle acceleration, is the unit vector of the acceleration direction. Combine the prediction result with the current gesture mask to generate a prediction of the future gesture area.

[0011] Further, in step S4, spatio-temporal feature encoding is performed based on the spatial and temporal features of the gesture, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through a fixed-gain Kalman filter, including: S41: Use MobileNetV3 to extract the spatial features of the predicted gesture trajectory in S3, and enhance the extracted spatial features based on the SE Block; S42: Extract the dynamic time features of the predicted gesture trajectory in S3 from consecutive frames, select a fixed time window, calculate the speed feature and acceleration feature, and combine the vehicle steering wheel angle to generate a time feature vector; S43: Concatenate the spatial and time features and input them into a fully connected layer to generate preliminary encoded features, and output the final features based on the spatio-temporal attention module; S44: According to the vehicle state including speed v, acceleration a, and steering wheel angle θ, construct a transformation matrix at the future T moment , , , , the translation amount calculated according to the vehicle acceleration a, are the components of the acceleration a in the x and y directions respectively. Multiply the position of the gesture in the camera coordinate system by the transformation matrix to obtain the aligned gesture position; S45: According to the current gesture state of the vehicle acceleration, use the fixed-gain predicted value and actual observation value of the gesture at the future T moment to obtain the gesture trajectory after compensating for inertia.

[0012] Further, in step S5, use K-means to classify users into three categories: conservative, medium, and radical, and use a dynamic convolution kernel to establish a multi-task loss function, including: According to the distance from the clustering center, users are divided into three categories: conservative, medium, and radical. The conservative type is defined as low acceleration, low speed, small steering wheel angle, and stable fluctuation of the remaining battery power. The medium type is defined as medium acceleration, medium speed, moderate steering wheel angle, and small fluctuation of the remaining battery power. The radical type is defined as high acceleration, high speed, large steering wheel angle, and large fluctuation of the remaining battery power; The basis for dividing low, medium, and high acceleration is acceleration ≤ 1.0 m / s 2 , 1.0 m / s 2 <acceleration ≤ 2.0 m / s 2 , acceleration > 2.0 m / s 2 ; The basis for dividing low, medium, and high speed is speed ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, speed > 80 km / h; The basis for dividing small, moderate, and large steering wheel angles is the absolute value of the angle ≤ 30°, 30° < the absolute value of the angle ≤ 60°, the absolute value of the angle > 60°; The basis for dividing stable, small, and large fluctuations of the remaining battery power is fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, fluctuation > 10%; Convert user characteristics into adjustment signals, calculate a dynamic convolution kernel based on user characteristics, and use this kernel to convolve the input features to generate adaptive features; Jointly optimize gesture recognition, user classification, and battery state prediction. The tasks are respectively defined as gesture recognition, user classification, and prediction of the remaining battery power. Calculate the losses of each task, combine the total loss according to the weights, and optimize the network through backpropagation.

[0013] Further, in S6, a predefined gesture instruction table is set, and trigger conditions are set. Verify the gear through the CAN bus, including: S61: Query the instruction table according to the gesture classification result, obtain the corresponding function, and judge whether the instruction is effective in combination with the user type and the remaining battery power; Making a fist gesture corresponds to the function of starting the fast charging mode, which is triggered when the remaining power < 20%; The OK gesture corresponds to the function of stopping charging / switching to the normal mode, which is triggered unconditionally; The gesture of swiping right corresponds to the function of increasing the charging power, which is triggered when the remaining power < 50% and the user is classified as radical; The gesture of swiping left corresponds to the function of reducing the charging power, which is triggered when the remaining power > 80% and the user is classified as conservative; S62: Check the relationship between the gesture confidence and the threshold. When the user is classified as aggressive, the threshold is 0.95; when the user is classified as medium, the threshold is 0.85; when the user is classified as conservative, the threshold is 0.75. Verify the gesture continuity. The gesture needs to be recognized for 5 consecutive frames, and the gesture trajectory needs to conform to the predefined path. S63: Read the vehicle gear and vehicle speed in real time through the CAN bus, and determine whether the safety conditions are met. If they are met, execute the instruction; otherwise, reject it.

[0014] Furthermore, in S7, power control is combined with the gesture intensity. The gesture intensity is calculated through the gesture area. An overload protection mechanism is established, including: S71: Calculate the gesture area A based on the gesture area segmentation in S3, and calculate the gesture movement speed based on the time characteristics of the gesture in S4 and the gesture acceleration , establish the gesture intensity calculation formula , , are the weight coefficients respectively, are the maximum values of the gesture area, gesture movement speed, and gesture acceleration respectively; S72: Dynamically adjust the charging power according to the gesture intensity and user type , , where is the default charging power, is the user type coefficient. When the user is conservative , when the user is medium , when the user is aggressive ; S73: Read the battery temperature and the remaining battery power in real time through the CAN bus, and determine whether the overload condition is triggered. The overload condition includes that the temperature does not exceed 60 degrees and the remaining battery power does not exceed 95%.

[0015] Furthermore, in S9, the model is fine-tuned using the gesture data of the day, the parameters are updated, light and occlusion noise are added to the user gesture data, and adversarial samples are generated. The model is adjusted monthly, including: Check whether the vehicle state meets the stationary condition. If it meets, load the gesture dataset of the day, including gesture images and their labels, and fine-tune the existing model using the gesture data accumulated daily; Apply random light changes and partial occlusion operations to each gesture sample to generate new samples. Based on the generative adversarial network, further generate adversarial samples, and add the data of the adversarial samples to the training set; Conduct a comprehensive model evaluation at the end of each month, including precision, recall, and F1-score metrics. If the performance of the current model deteriorates, retrain the model using all accumulated gesture data, including daily fine-tuning data, adversarial samples, and user feedback data.

[0016] The beneficial effects of the present invention are as follows: By adjusting the weights of sensors based on vehicle status and environmental conditions, it can more effectively cope with different driving conditions, improve the quality of data collection, and maintain high-precision data collection capabilities even under different light intensities and vehicle vibrations. Use a Gaussian mixture model for background difference and depth segmentation, and combine a physical model for gesture trajectory prediction to improve the accuracy and real-time performance of gesture recognition. Through the fusion of spatial and temporal features and coordinate system alignment, the robustness of gesture recognition is enhanced, and it can work effectively even during vehicle movement. Classify users into conservative, medium, and aggressive types according to their driving behaviors, so as to provide a more personalized and safe charging experience. The predefined gesture instruction table allows users to control the charging process through simple gesture operations, improving convenience. Convert gesture intensity into a charging power adjustment parameter to achieve an intuitive and flexible charging power control mechanism. By real-time monitoring of battery temperature and remaining battery power, it can prevent safety hazards caused by overcharging. Continuously fine-tune the model using daily accumulated data, and enhance the generalization ability of the model by adding noise to generate adversarial samples, ensuring the performance stability for long-term use. Brief Description of the Drawings

[0017] Figure 1 It is a flowchart of a gesture intelligent recognition and processing method for in-vehicle fast charging. Detailed Embodiments

[0018] The following further clearly and completely describes the present invention, but the protection scope of the present invention is not limited thereto.

[0019] A gesture intelligent recognition and processing method for in-vehicle fast charging, and the adopted technical solutions are as follows: S1: Based on the CAN bus, read the vehicle speed, acceleration, steering wheel angle, gear state, and remaining battery power in real time, construct a vehicle state vector, perform dynamic weight allocation based on rules, and combine the light intensity, temperature, and vehicle state to output sensor weights. S2: Based on hardware-level time synchronization and calibration matrix, perform spatio-temporal synchronous acquisition of multi-sensor data, and perform joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared and visible light fusion. S3: Perform gesture area segmentation, including background difference and depth segmentation. Dynamically update the background using the Gaussian mixture model, and use a neural network of MobileNetV3 with an added SE Block to output a gesture mask. Combine the physical model of vehicle acceleration to predict the gesture trajectory; S4: Encode spatio-temporal features based on the spatial and temporal features of the gesture. Use PCA to reduce the dimensionality of the features, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through a fixed-gain Kalman filter; S5: Based on the feature vectors output by the neural network of MobileNetV3 with an added SE Block, use K-means to classify users into three categories: conservative, medium, and radical. Use dynamic convolution kernels, establish a multi-task loss function, and perform incremental learning; S6: Pre-define a gesture instruction table, set trigger conditions, including confidence threshold and continuity verification, and verify the gear through the CAN bus; S7: Dynamically adjust the charging parameters, perform power control in combination with the gesture intensity, calculate the gesture intensity through the gesture area, establish an overload protection mechanism, and control the charging module through the CAN bus; S8: Display the charging power and remaining time on the dashboard, highlight the gesture recognition area with a green box, and give a pre-recorded voice prompt; S9: Every night when the vehicle is stationary, use the gesture data of the day to fine-tune the model and update the parameters. Regularly collect user feedback data, add illumination and occlusion noise to the user gesture data to generate adversarial samples, and adjust the model monthly.

[0020] Reference Figure 1 As shown, it is a flowchart of a gesture intelligent recognition and processing method for in-vehicle fast charging.

[0021] Further, in the above S1, a vehicle state vector is constructed, and dynamic weight allocation is performed based on rules to output sensor weights, including: S11: Real-time read the vehicle speed v, acceleration a, steering wheel angle θ, gear state G (including P, N, R, D), and remaining battery power E through the in-vehicle CAN bus interface to construct a vehicle state vector ; S12: Measure the illumination intensity based on the ambient light sensor , measure the ambient temperature based on the thermistor , define a basic weight matrix , including the basic weights of the millimeter-wave radar, infrared sensor, and camera, which are respectively ; S13: Adjust the weights according to the vehicle state, , , ; S14: Normalize the above weights, , where represents the sensor weight vector after normalization.

[0022] Furthermore, in S2, based on hardware-level time synchronization and calibration matrix, spatio-temporal synchronous acquisition of multi-sensor data is performed, and joint compensation is performed based on light intensity and vehicle vibration, including: S21: Use an in-vehicle Ethernet switch that supports the IEEE 1588 protocol to assign an independent physical clock to each sensor. The sensor records the data acquisition time through a hardware-level timestamp module, and all sensors send time synchronization requests to the main control unit through Ethernet. The main control unit synchronizes the master clock to all slave devices through the "master-slave" mode of IEEE 1588, and the difference between the transmission time of each sensor data frame and the reception time of the main control unit is compensated by software; S22: Use a laser rangefinder to perform calibration when the vehicle is stationary. Fix the calibration board in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the internal parameter matrix of the camera based on Zhang Zhengyou calibration method, calculate the external parameter matrix of the camera and millimeter-wave radar using three-dimensional point cloud matching, and convert the points in the camera coordinate system to the radar coordinate system; S23: Establish a compensation formula model , , where is the coordinate point, is the light intensity, is the vehicle vibration amplitude, obtained based on the accelerometer, , is the light compensation coefficient, , is the vibration attenuation coefficient, read in real time and , and adjust the image brightness according to the formula; S24: Dynamically adjust the weights according to the light intensity, , when , , otherwise , where is the fused image intensity value, is the original image intensity value collected by the visible light sensor, is the original image intensity value collected by the infrared sensor, is the weight coefficient of the visible light image.

[0023] Furthermore, in S3, gesture area segmentation is performed, and gesture trajectory prediction is performed in combination with the physical model of vehicle acceleration, including: S31: Generate a background model using the GMM Gaussian mixture model, which is updated in real time, once every 5 frames. Initially segment the foreground area through background difference, and use a neural network of MobileNetV3 with an added SE Block to output the final gesture mask; S32: Based on the current gesture position and vehicle state, predict the gesture trajectory at the future T moment, and calculate the current gesture speed based on the continuous frame difference , , where is the gesture coordinate at the T moment, is the current gesture position coordinate, is the vehicle acceleration, is the unit vector of the acceleration direction. Combine the prediction result with the current gesture mask to generate a prediction of the future gesture area.

[0024] Further, in S4, perform spatio-temporal feature encoding based on the spatial and temporal features of the gesture, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through a fixed-gain Kalman filter, including: S41: Use MobileNetV3 to extract the spatial features of the predicted gesture trajectory in S3, and enhance the extracted spatial features based on the SE Block; S42: Extract the dynamic time features of the predicted gesture trajectory in S3 from continuous frames, select a fixed time window, calculate speed features and acceleration features, and combine the vehicle steering wheel angle to generate a time feature vector; S43: Concatenate the spatial and time features and input them into a fully connected layer to generate preliminary encoded features, and output the final features based on the spatio-temporal attention module; S44: According to the vehicle state including speed v, acceleration a, and steering wheel angle θ, construct a transformation matrix at the future T moment , , , , the translation amount calculated according to the vehicle acceleration a, are the components of the acceleration a in the x and y directions respectively. Multiply the position of the gesture in the camera coordinate system by the transformation matrix to obtain the aligned gesture position; S45: According to the current gesture state of the vehicle acceleration, use the predicted value and actual observation value of the gesture at the future T moment with a fixed gain to obtain the gesture trajectory after compensating for inertia.

[0025] Further, in S5, use K-means to classify users into three categories: conservative, medium, and radical, and use a dynamic convolution kernel to establish a multi-task loss function, including: According to the distance from the clustering center, users are divided into three categories: conservative, medium, and radical. The conservative type is defined as low acceleration, low speed, small steering wheel angle, and stable fluctuation of the remaining battery power. The medium type is defined as medium acceleration, medium speed, moderate steering wheel angle, and small fluctuation of the remaining battery power. The radical type is defined as high acceleration, high speed, large steering wheel angle, and large fluctuation of the remaining battery power; The criteria for dividing low, medium, and high acceleration are acceleration ≤ 1.0 m / s 2 , 1.0 m / s 2 <acceleration ≤ 2.0 m / s 2 , acceleration > 2.0 m / s 2 ; The criteria for dividing low, medium, and high speed are speed ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, and speed > 80 km / h; The criteria for dividing small, moderate, and large steering wheel angles are the absolute value of the angle ≤ 30°, 30° < the absolute value of the angle ≤ 60°, and the absolute value of the angle > 60°; The criteria for dividing stable, small, and large fluctuations of the remaining battery power are fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, and fluctuation > 10%; Convert user characteristics into adjustment signals, calculate the dynamic convolution kernel according to user characteristics, and use this kernel to convolve the input features to generate adaptive features; Jointly optimize gesture recognition, user classification, and battery status prediction. The tasks are defined as gesture recognition, user classification, and prediction of the remaining battery power respectively. Calculate the losses of each task, combine the total loss according to the weights, and optimize the network through backpropagation.

[0026] Further, in S6, a predefined gesture instruction table is set, and a trigger condition is set. Verify the gear through the CAN bus, including: S61: Query the instruction table according to the gesture classification result, obtain the corresponding function, and judge whether the instruction is effective in combination with the user type and the remaining battery power; The gesture of making a fist corresponds to the function of starting the fast charging mode, which is triggered when the remaining power < 20%; The gesture of making an "OK" corresponds to the function of stopping charging / switching to the normal mode, which is triggered unconditionally; The gesture of swiping right corresponds to the function of increasing the charging power, which is triggered when the remaining power < 50% and the user is classified as radical; The gesture of swiping left corresponds to the function of decreasing the charging power, which is triggered when the remaining power > 80% and the user is classified as conservative; S62: Check the relationship between the gesture confidence and the threshold. When the user is classified as aggressive, the threshold is 0.95; when the user is classified as moderate, the threshold is 0.85; when the user is classified as conservative, the threshold is 0.75. Verify the gesture continuity. The gesture needs to be recognized for 5 consecutive frames, and the gesture trajectory needs to conform to the predefined path. S63: Read the vehicle gear and vehicle speed in real time through the CAN bus, and determine whether the safety conditions are met. If they are met, execute the instruction; otherwise, reject it.

[0027] Further, in S7, power control is combined with the gesture intensity. The gesture intensity is calculated through the gesture area, and an overload protection mechanism is established, including: S71: Calculate the gesture area A based on the gesture area segmentation in S3, and calculate the gesture movement speed based on the time characteristics of the gesture in S4 and the gesture acceleration , and establish the gesture intensity calculation formula , , are weight coefficients respectively, are the maximum values of the gesture area, gesture movement speed, and gesture acceleration respectively; S72: Dynamically adjust the charging power according to the gesture intensity and user type , , where is the default charging power, is the user type coefficient. When the user is conservative , when the user is moderate , when the user is aggressive ; S73: Read the battery temperature and the remaining battery power in real time through the CAN bus, and determine whether the overload condition is triggered. The overload condition includes that the temperature does not exceed 60 degrees and the remaining battery power does not exceed 95%.

[0028] Further, in S9, the model is fine-tuned using the gesture data of the day, the parameters are updated, light and occlusion noise are added to the user gesture data to generate adversarial samples, and the model is adjusted monthly, including: Check whether the vehicle state meets the stationary condition. If it does, load the gesture data set of the day, including gesture images and their labels, and fine-tune the existing model using the gesture data accumulated daily; Apply random light changes and partial occlusion operations to each gesture sample to generate new samples, and further generate adversarial samples based on the generative adversarial network, and add the data of the adversarial samples to the training set; A comprehensive model evaluation is conducted at the end of each month, including precision, recall, and F1-score metrics. If the performance of the current model degrades, all accumulated gesture data, including daily fine-tuning data, adversarial samples, and user feedback data, is used to retrain the model.

[0029] The present invention provides a method for gesture intelligent recognition and processing of in-vehicle fast charging. It obtains vehicle status information in real time based on the CAN bus, dynamically adjusts the sensor weights according to environmental conditions, uses hardware-level time synchronization and multi-sensor data fusion technology to improve data acquisition accuracy, combines background difference with a deep learning model to output a precise gesture mask, predicts the gesture trajectory, extracts the spatio-temporal features of the gesture and encodes them, uses Kalman filtering to compensate for inertial effects, classifies users, responds to gesture commands in a personalized manner, combines a predefined gesture command table with confidence verification and gear check to ensure operation safety, dynamically adjusts the charging power according to the gesture intensity, establishes an overload protection mechanism, the dashboard displays charging information and highlights the gesture area with a green frame, provides voice feedback, fine-tunes the model parameters daily, adds noise to generate adversarial samples, comprehensively evaluates and updates the model monthly, thereby continuously improving the robustness of the system and the user experience.

Claims

1. A gesture intelligent recognition and processing method for in-vehicle fast charging, characterized in that Including: S1: Based on the CAN bus, read the vehicle speed, acceleration, steering wheel angle, gear state, and remaining battery power in real time, construct a vehicle state vector, perform dynamic weight allocation based on rules, and combine the light intensity, temperature, and vehicle state to output sensor weights; S2: Based on hardware-level time synchronization and calibration matrices, perform spatio-temporal synchronous acquisition of multi-sensor data, and perform joint compensation based on light intensity and vehicle vibration, including dynamic exposure compensation and infrared-visible light fusion; S3: Perform gesture region segmentation, including background difference and depth segmentation, dynamically update the background using a Gaussian mixture model, use a neural network of MobileNetV3 with an added SE Block to output a gesture mask, and predict the gesture trajectory in combination with the physical model of vehicle acceleration; S4: Perform spatio-temporal feature encoding based on the spatial and temporal features of the gesture, use PCA to reduce the dimensionality of the feature dimensions, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through a fixed-gain Kalman filter; S5: Based on the neural network of MobileNetV3 with an added SE Block, output feature vectors, use K-means to classify users into three categories: conservative, medium, and aggressive, use dynamic convolution kernels, establish a multi-task loss function, and perform incremental learning; S6: Pre-define a gesture instruction table, set trigger conditions, including confidence threshold and continuity verification, and verify the gear through the CAN bus; S7: Dynamically adjust the charging parameters, perform power control in combination with the gesture intensity, calculate the gesture intensity through the gesture region area, establish an overload protection mechanism, and control the charging module through the CAN bus; S8: The dashboard displays the charging power and remaining time, the gesture recognition area is highlighted with a green box, and a prerecorded voice prompt is used; S9: Every night when the vehicle is stationary, use the gesture data of the day to fine-tune the model and update the parameters, regularly collect user feedback data, add light and occlusion noise to the user gesture data to generate adversarial samples, and adjust the model monthly.

2. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In the S1, constructing the vehicle state vector, performing dynamic weight allocation based on rules, and outputting sensor weights includes: S11: Read the vehicle speed v, acceleration a, steering wheel angle θ, gear state G including P, N, R, D, and remaining battery power E in real time through the in-vehicle CAN bus interface, and construct a vehicle state vector ; S12: Measure the light intensity based on the ambient light sensor , measure the ambient temperature based on the thermistor , define the basic weight matrix , including the basic weights of the millimeter-wave radar, infrared sensor, and camera, which are respectively ; S13: Adjust the weight according to the vehicle state, , , ; S14: Normalize the above weights, , where represents the sensor weight vector after normalization.

3. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In the S2, based on hardware-level time synchronization and calibration matrices, performing spatio-temporal synchronous acquisition of multi-sensor data, and performing joint compensation based on light intensity and vehicle vibration, including: S21: Use an in-vehicle Ethernet switch that supports the IEEE 1588 protocol to assign an independent physical clock to each sensor. The sensor records the data acquisition time through a hardware-level timestamp module. All sensors send time synchronization requests to the main control unit through the Ethernet. The main control unit synchronizes the master clock to all slave devices in the "master-slave" mode of IEEE 1588. The difference between the sending time of each sensor data frame and the receiving time of the main control unit is compensated by software; S22: Perform calibration using a laser rangefinder when the vehicle is stationary. Fix the calibration board in front of the vehicle, record the observation data of the camera and millimeter-wave radar, calculate the internal parameter matrix of the camera based on the Zhang Zhengyou calibration method, calculate the external parameter matrix between the camera and the millimeter-wave radar using 3D point cloud matching, and convert the points in the camera coordinate system to the radar coordinate system; S23: Establish a compensation formula model , , where is a coordinate point, is the light intensity, is the vehicle vibration amplitude, obtained based on an accelerometer, is the light compensation coefficient, is the vibration attenuation coefficient, read in real time and , and adjust the image brightness according to the formula; S24: Dynamically adjust the weight according to the light intensity, When At that time, Otherwise Among them Is the fused image intensity value, Is the original image intensity value collected by the visible light sensor, Is the original image intensity value collected by the infrared sensor, Is the weight coefficient of the visible light image.

4. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In step S3, gesture area segmentation is performed, and gesture trajectory prediction is carried out in combination with the physical model of vehicle acceleration, including: S31: Use the GMM Gaussian mixture model to generate a background model, which is updated in real time, updated every 5 frames. Initially segment the foreground area through background difference, and use the neural network of MobileNetV3 with an added SE Block to output the final gesture mask; S32: Predict the gesture trajectory at the future time T based on the current gesture position and vehicle state, and calculate the current gesture speed based on the continuous frame difference , , where is the gesture coordinate at time T, is the current gesture position coordinate, is the vehicle acceleration, is the unit vector of the acceleration direction. Combine the prediction result with the current gesture mask to generate the prediction of the future gesture area.

5. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In step S4, spatio-temporal feature encoding is performed based on the spatial and temporal features of the gesture, align the gesture coordinate system with the vehicle coordinate system, and compensate for inertia through a fixed-gain Kalman filter, including: S41: Use MobileNetV3 to extract the spatial features of the predicted gesture trajectory in S3, and enhance the extracted spatial features based on the SE Block; S42: Extract the dynamic time features of the predicted gesture trajectory in S3 from consecutive frames, select a fixed time window, calculate the speed feature and acceleration feature, and combine with the vehicle steering wheel angle to generate a time feature vector; S43: Concatenate the spatial and temporal features and input them into a fully connected layer to generate preliminary encoded features, and output the final features based on the spatio-temporal attention module; S44: Construct a transformation matrix at future time T based on vehicle states including speed v, acceleration a, and steering wheel angle θ , , , , which is the translation amount calculated based on vehicle acceleration a are the components of acceleration a in the x and y directions respectively. The gesture position after alignment is obtained by multiplying the position of the gesture in the camera coordinate system by the transformation matrix S45: According to the current gesture state of the vehicle acceleration, use the predicted value and actual observation value of the gesture at the future T moment with a fixed gain to obtain the gesture trajectory after compensating for inertia.

6. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In step S5, use K-means to divide users into three categories: conservative, medium, and aggressive. Use a dynamic convolution kernel to establish a multi-task loss function, including: According to the distance from the clustering center, divide users into three categories: conservative, medium, and aggressive. The conservative type is defined as low acceleration, low speed, small steering wheel angle, and stable battery remaining power fluctuation. The medium type is defined as medium acceleration, medium speed, moderate steering wheel angle, and small battery remaining power fluctuation. The aggressive type is defined as high acceleration, high speed, large steering wheel angle, and large battery remaining power fluctuation; The basis for dividing acceleration into low, medium, and high is that the acceleration ≤ 1.0 m / s 2 , 1.0 m / s 2 <acceleration ≤ 2.0 m / s 2 , acceleration > 2.0 m / s 2 ; The criteria for dividing low, medium, and high speeds are speed ≤ 40 km / h, 40 km / h < speed ≤ 80 km / h, and speed > 80 km / h respectively; The criteria for dividing small, moderate, and large steering wheel angles are the absolute value of the angle ≤ 30°, 30° < the absolute value of the angle ≤ 60°, and the absolute value of the angle > 60° respectively; The criteria for dividing stable, small, and large fluctuations of the remaining battery power are fluctuation ≤ 5%, 5% < fluctuation ≤ 10%, and fluctuation > 10% respectively; Convert the user features into adjustment signals, calculate the dynamic convolution kernel according to the user features, and use this kernel to convolve the input features to generate adaptive features; Jointly optimize gesture recognition, user classification, and battery state prediction. The tasks are respectively defined as gesture recognition, user classification, and prediction of the remaining battery power. Calculate the losses of each task, combine the total loss according to the weights, and optimize the network through backpropagation.

7. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that In S6, a predefined gesture instruction table is set with trigger conditions, and the gear is verified through the CAN bus, including: S61: Query the instruction table according to the gesture classification result, obtain the corresponding function, and judge whether the instruction is effective by combining the user type and the remaining battery power; The gesture of making a fist corresponds to the function of starting the fast charging mode, which is triggered when the remaining power < 20%; The gesture of making an "OK" corresponds to the function of stopping charging / switching to the normal mode, which is triggered unconditionally; The gesture of swiping right corresponds to the function of increasing the charging power, which is triggered when the remaining power < 50% and the user is classified as aggressive; The gesture of swiping left corresponds to the function of decreasing the charging power, which is triggered when the remaining power > 80% and the user is classified as conservative; S62: Check the relationship between the gesture confidence and the threshold. The threshold is 0.95 when the user is classified as aggressive, 0.85 when the user is classified as medium, and 0.75 when the user is classified as conservative. Verify the gesture continuity. The gesture needs to be recognized continuously for 5 frames, and the gesture trajectory needs to conform to the predefined path; S63: Read the vehicle gear and vehicle speed in real time through the CAN bus, judge whether the safety conditions are met, and execute the instruction if they are met, otherwise reject it.

8. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, characterized in that, In S7, power control is combined with the gesture strength. The gesture strength is calculated by the gesture area, and an overload protection mechanism is established, including: S71: Calculate the area A of the gesture area based on the gesture area segmentation described in S3, and calculate the gesture movement speed based on the time feature of the gesture described in S4 and the gesture acceleration , establish the gesture intensity calculation formula , , are the weight coefficients respectively, are the maximum values of the gesture area, gesture movement speed and gesture acceleration respectively; S72: Dynamically adjust the charging power according to the gesture intensity and user type , , where is the default charging power, is the user type coefficient, when the user is conservative , when the user is medium , when the user is aggressive ; S73: Read the battery temperature and the remaining battery power in real time through the CAN bus, and judge whether the overload condition is triggered. The overload condition includes that the temperature does not exceed 60 degrees and the remaining battery power does not exceed 95%.

9. The gesture intelligent recognition and processing method for in-vehicle fast charging according to claim 1, wherein In S9, the model is fine-tuned using the daily gesture data, the parameters are updated, light and occlusion noise are added to the user gesture data to generate adversarial samples, and the model is adjusted monthly, including: Check whether the vehicle state meets the stationary condition. If it does, load the gesture data set of the day, including gesture images and their labels, and fine-tune the existing model using the gesture data accumulated daily; Apply random light changes and partial occlusion operations to each gesture sample to generate new samples. Based on the generative adversarial network, further generate adversarial samples, and add the data of the adversarial samples to the training set; A comprehensive model evaluation is carried out at the end of each month, including accuracy, recall rate, and F1 score metrics. If the performance of the current model deteriorates, then use all the accumulated gesture data, including daily fine-tuning data, adversarial samples, and user feedback data, to retrain the model.

Images