Recognition method and system for human-vehicle interaction

Abstract

The invention discloses an identification method for human-vehicle interaction, and relates to the field of human-vehicle interaction, and the method comprises the steps: collecting the original data of bc types of sensors; performing time synchronization on the original data of the bc types of sensors by using a timestamp alignment method; performing spatial synchronization on the original data of the bc types of sensors by using coordinate transformation to obtain preprocessed data after data alignment; establishing a deep neural network architecture, and performing feature-level fusion on the preprocessed data of each sensor through the deep neural network architecture to obtain a current environment state outside the vehicle; the system interacts with a driver, performs warning and reminding display on the driver according to the current environment state outside the vehicle, realizes judgment of the environment state outside the vehicle body in a complex environment, controls the vehicle body according to intelligent judgment, and realizes interaction control between the vehicle and personnel or objects outside and inside the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of human-vehicle interaction, and more specifically, to a human-vehicle interaction recognition method and system. Background Art

[0002] In human-vehicle interaction recognition, the recognition accuracy issue is a key challenge, especially in complex environments, where there are problems of misidentification or missed recognition, difficulty in multi-target tracking, and delays in dynamic scenes.

[0003] In the case of misidentification or missed identification, the system may mistake background objects for pedestrians or vehicles, causing the car to respond incorrectly. For example, emergency braking may be performed when there are no pedestrians, affecting traffic flow. The system fails to identify real pedestrians or vehicles, which may lead to failure to brake or evade in time, leading to accidents. Especially when pedestrians suddenly appear in front of the car, multi-target tracking is difficult. The system may lose the position of a pedestrian or confuse two different targets, causing deviations in the vehicle's response. For example, the system may think that a pedestrian has left the lane, but in fact the pedestrian is still in a dangerous area. It may also ignore certain key targets, especially those that are obscured or smaller, such as children or bicycles, which may cause the vehicle to fail to evade or brake in time. In addition, the acquisition time and coordinate system of different sensors may be inconsistent, resulting in inaccurate information during data fusion. Therefore, how to design recognition for human-vehicle interaction is now an urgent problem to be solved. Summary of the invention

[0004] In view of the problems existing in the prior art, the purpose of the present invention is to provide an identification method for human-vehicle interaction to realize intelligent human-vehicle interaction.

[0005] To achieve the above object, the present invention provides the following technical solution: The human-vehicle interaction recognition method comprises:

[0006] Step S1: Collecting raw data from b_c types of sensors;

[0007] Step S2: aligning the raw data of b_c types of sensors to obtain aligned pre-processed data;

[0008] Step S3: perform feature-level fusion on the pre-processed data of each sensor according to the deep neural network architecture to obtain the current environment status outside the vehicle;

[0009] Step S4: collecting the current status data of the vehicle body, and displaying warnings and reminders according to the current environmental status outside the vehicle;

[0010] Preferably, the b_c types of sensors include cameras, lidars, millimeter-wave radars and infrared sensors arranged on the vehicle body, the cameras are used to provide high-resolution 2D images, the lidars are used to provide high-precision 3D point cloud data, the millimeter-wave radars are used to provide distance, speed and relative motion information of objects, and the infrared sensors are used to provide temperature or thermal imaging data.

[0011] Preferably, the method for aligning raw data of b_c types of sensors includes:

[0012] Step A1: Use the timestamp alignment method to synchronize the raw data of b_c types of sensors;

[0013] Step A2: Use coordinate transformation to spatially synchronize the raw data of b_c types of sensors.

[0014] Preferably, the method of performing time synchronization on raw data of b_c types of sensors using a timestamp alignment method comprises:

[0015] One of the sensors is selected as the base sensor, the time delay between the base sensor and any other sensors is calculated, the cycle time T and the delay assumption value set are set, the delay assumption value set includes a preset time delay value, and the raw data of the base sensor and other sensors within the cycle time T are formed into two sets of time series A[T] and B[T];

[0016] Calculate the signal means μA and μB of A[T] and B[T], and decentralize the time series A[T] and B[T] to obtain A 1 [T] and B 1 [T], calculate the cross-correlation function Assign τ to the values ​​in the delay hypothesis value set and perform calculations, and select the cross-correlation function R AB The value of τ when (τ) is the maximum is taken as the actual delay time. The actual delay time is used to perform time correction on other sensors compared with the base sensor, and the corrected time series is obtained as B[T-τ];

[0017] The corrected time series of each sensor is obtained in turn, and the data of all sensors are aligned at the same time point by interpolation, discarding or repeating the data.

[0018] Preferably, the method of spatially synchronizing raw data of b_c types of sensors using coordinate transformation comprises:

[0019] Set the base coordinate system, which can be the vehicle coordinate system or the world coordinate system;

[0020] Convert the camera coordinate system to the base coordinate system, obtain the 2D image captured by the camera, and add the coordinate system to form a 2D image coordinate system. The pixel points are mapped to the base coordinate system through the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0021] Among them, the camera's internal parameter matrix Among them, (f x , f y ) is the focal length, (c x , c y ) is the optical center position. The camera's external parameter matrix is ​​calibrated using a calibration plate. By taking z_p calibration plate images at different angles and using the camera calibration function to calculate the relationship between the pixel coordinates of the corner points in the calibration plate image and the base coordinates, the camera's external parameter matrix can be obtained.

[0022] The method of converting the infrared sensor coordinate system to the base coordinate system is the same as that of the camera;

[0023] Convert the laser radar coordinate system to the base coordinate system, and convert the laser radar data to the base coordinate system through the laser radar's external parameter matrix and translation vector;

[0024] The method for obtaining the external parameter matrix and translation vector of the laser radar is as follows: using the dot array as the calibration target, using the laser radar and the camera to collect data at different angles and positions, using the point cloud registration technology to align the point cloud data of the laser radar with the image data of the camera, obtaining the external parameter matrix and translation vector, and optimizing the external parameter matrix by the nonlinear least squares method;

[0025] The method of converting the millimeter wave radar coordinate system into the base coordinate system is the same as that of the laser radar.

[0026] Preferably, the method of fusing the preprocessed data of each sensor at the feature level includes:

[0027] Set up a comprehensive sample set, convert the images captured by the camera into RGB images or grayscale images of standard size, and perform standardization processing; voxelize the laser radar point cloud data to generate a bird's-eye view or depth map; convert the millimeter wave radar data into image form; convert the infrared sensor data into a thermal map or grayscale map, and perform standardization processing; all collected data form a comprehensive sample set;

[0028] Build a deep neural network architecture for multi-sensor fusion, set the input layer to a comprehensive sample set, use CNN network to extract features from camera and infrared sensor data, use 3D convolutional neural network to extract features from lidar and millimeter wave radar data, and then obtain the feature vector of each sensor. Use weighted summation to fuse each feature vector into a large overall vector, input the large overall vector into the fully connected layer, and set the output layer to u_p classification task outputs and u_k regression task outputs. The classification task is used to identify objects of u_p categories, and the regression task output is used to predict the continuous attributes of the object.

[0029] The output of the integrated deep neural network architecture is used to obtain the current state of the environment outside the vehicle;

[0030] The classification task uses the Softmax function for activation, the regression task uses the linear activation function for activation, and the combined loss function is used The classification task uses cross entropy loss, and the regression task uses mean square error loss, where ε(j) represents the weight of the j-th classification task, CE_loss(j) represents the cross entropy loss of the j-th classification task, θ(u) represents the weight of the u-th classification task, and MSE_loss(u) represents the mean square error loss of the u-th regression task. Adam or SGD is used to optimize the model.

[0031] Preferably, the method for extracting features from the data of the camera and the infrared sensor using a CNN network comprises:

[0032] The input is set to the data of the camera and infrared sensor, and the size of the original image is (H, W, C), where H is the height, W is the width, and C is the number of channels;

[0033] Use multiple convolutional layers to extract local features Use the ReLU activation function to introduce nonlinearity, obtain the transformed output F_relu(k,h,m), downsample the feature map through the pooling layer, and repeat the convolution and pooling operations to obtain a low-resolution feature map (H 11 ,W 11 ,C);

[0034] Among them, H 11 is the height, W 11 is the width, F_con(k,h,m) represents the feature value of the mth channel at position (k,h) in the output feature map, I(k+p,h+r,c) is the pixel value of the input image at position (k,h) and (p,r) on channel c, W_m(p,r,c) represents the weight of the mth convolution kernel at position (p,r) and channel c, Bm represents the bias term of the mth convolution kernel, and K_P represents the size of the convolution kernel;

[0035] Use bilinear interpolation to restore the low-resolution feature map to the resolution of the original image to obtain the feature map F_up(k,h,m), jump connect the high-resolution feature map F_con(k,h,m) of the early convolutional layer with the upsampled feature map F_up(k,h,m) to obtain the feature map F_sk(k,h,m), perform global averaging on the feature map F_sk(k,h,m), and traverse each feature map to generate a feature vector of fixed length.

[0036] Preferably, the method for extracting features from the laser radar and millimeter wave radar data using a 3D convolutional neural network includes:

[0037] The image data of the laser radar or millimeter wave radar is normalized and processed in size, and the continuous S_c frame images are stacked to form a four-dimensional tensor;

[0038] Design l_h convolutional layers, introduce adaptive pooling after each convolutional layer, use four-dimensional convolution kernel to convolve the input four-dimensional tensor, extract spatiotemporal features, output feature maps, add a timing modeling layer after the convolutional layer, the timing modeling layer is based on LSTM or GRU, set the output of the timing modeling layer as the motion information of the target, design a fully connected layer after the timing modeling layer, and the output of the fully connected layer is a feature vector representing the comprehensive feature information of the target.

[0039] Preferably, the current status data of the vehicle body includes the driving status of the vehicle and various control parameters. The current environmental status outside the vehicle is displayed through a display screen inside the vehicle, or a voice broadcast is used for warning reminders. The driver controls the vehicle body according to the display and warning.

[0040] A recognition method for human-vehicle interaction, comprising:

[0041] Data acquisition module: used to collect raw data from b_c types of sensors;

[0042] Data processing module: used to align the raw data of each sensor in time and space, and obtain pre-processed data after data alignment;

[0043] Vehicle state fitting prediction module: Build a deep neural network architecture, use the deep neural network architecture to fuse the pre-processed data of each sensor at the feature level, and obtain the current environment state outside the vehicle;

[0044] Control output module: used to interact with the driver and provide warnings and reminders to the driver based on the current environmental status outside the vehicle.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] In the present invention, data from multiple sensors are combined to provide high-precision real-time environmental perception, and the accuracy of identifying objects (such as pedestrians, vehicles, obstacles, etc.) is significantly improved. The driving status of the vehicle and the status of the surrounding environment generate intelligent warnings and reminders, reducing the cognitive burden of the driver and enhancing safety. Utilizing deep learning technology, a large amount of sensor data is automatically processed and analyzed, and environmental information and alarms are generated in real time to improve the intelligence level of the vehicle. The system can automatically adjust the alarm content according to real-time environmental changes, making human-vehicle interaction more natural and improving user experience;

[0047] By building a deep neural network architecture, the pre-processed data of each sensor is fused at the feature level through the deep neural network architecture, and the CNN network is used to extract features based on image seams for the data of the camera and infrared sensor, thereby increasing the data processing efficiency and quality. The 3D convolutional neural network is used to extract features based on time series and adaptive image processing size for the data of the lidar and millimeter wave radar, so as to capture the dynamic features of objects with high efficiency and quality, and make the prediction of objects more accurate.

[0048] The adaptive control and optimization mechanism continuously improves the overall efficiency of intelligent analysis. The cooperation of multiple modules can realize comprehensive data collection, processing and analysis, provide real-time and accurate analysis of detection data, and realize efficient coordinated control of multi-joint arms in spider cranes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A structural schematic diagram of a human-vehicle interaction recognition method proposed by the present invention;

[0050] Figure 2 It is a schematic diagram of the identification method used for human-vehicle interaction in the present invention. DETAILED DESCRIPTION

[0051] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0052] Embodiment 1

[0053] Reference Figure 1 and Figure 2 , Example 1 further illustrates a human-vehicle interaction recognition method proposed by the present invention.

[0054] In human-vehicle interaction recognition, the recognition accuracy issue is a key challenge, especially in complex environments, where there are problems of misidentification or missed recognition, difficulty in multi-target tracking, and delays in dynamic scenes.

[0055] In the case of misidentification or missed identification, the system may mistake background objects for pedestrians or vehicles, causing the car to make incorrect responses. For example, emergency braking may be performed when there are no pedestrians, affecting traffic flow. The system fails to identify real pedestrians or vehicles, which may result in failure to brake or evade in time, leading to accidents, especially when pedestrians suddenly appear in front of the car. The reasons for this include:

[0056] 1. In low-light environments such as at night or in tunnels, insufficient light causes the camera's visual recognition ability to drop significantly. Although infrared cameras or lidar can provide assistance, these sensors also have their limitations, such as low resolution or limited detection distance.

[0057] 2. In severe weather such as heavy rain or snow, rain or snow will interfere with the camera's field of view and affect image clarity. In addition, raindrops or snow may gather on the sensor surface, causing the sensor to fail. Radar and lidar signals may also be distorted due to reflection or scattering of rain and snow.

[0058] 3. When sunlight shines directly on the camera, the strong light may cause overexposure, making it impossible for the camera to correctly identify pedestrians or vehicles. In addition, strong light may produce reflections or shadows, which will also interfere with the system's recognition.

[0059] 4. In urban environments, the background is often very complex, including moving people, vehicles, billboards, trees, buildings, etc. It may be difficult for cameras or sensors to distinguish between pedestrians, vehicles, and background objects. When the background is similar in color or texture to pedestrians or vehicles, the probability of misidentification increases.

[0060] Multi-target tracking is difficult, and the system may lose the position of a pedestrian or confuse two different targets, causing the vehicle's response to be biased. For example, the system may think that the pedestrian has left the lane, but in fact the pedestrian is still in the danger zone. It may also ignore certain key targets, especially those that are obscured or small, such as children or bicycles, which may cause the vehicle to fail to avoid or brake in time. The reasons for this include:

[0061] 1. On busy city streets or crowded areas, there are many pedestrians, vehicles, bicycles and other targets, and the speeds, directions and behaviors of these targets are different. In this case, the camera or sensor needs to detect and track multiple targets at the same time, but the algorithm has limited capabilities and may have difficulty handling too many targets.

[0062] 2. When multiple targets occlude each other (for example, a pedestrian walks beside a car, or a car drives behind a pedestrian), the system may have difficulty tracking the occluded targets. This occlusion is especially common during rush hour or on narrow streets.

[0063] 3. When the appearance, size, color and other features of pedestrians or vehicles are too similar, the system may have difficulty distinguishing different targets. This may cause the system to merge multiple targets into one, or mistakenly identify one target as multiple targets between different frames.

[0064] These algorithms require a lot of computing resources, and the computing time will increase significantly, resulting in delays. There may also be delays in the data transmission process, especially in multi-sensor fusion. The data update frequency of each sensor is different, and the transmission path and data processing speed are also inconsistent, which can easily cause time delays, especially in complex scenarios. It may lead to delayed decisions. Due to the limitations of computing and sensor processing capabilities, it may not be possible to respond in time when faced with rapidly changing scenarios. Delayed decisions may prevent the vehicle from avoiding danger in time, increasing the risk of accidents.

[0065] The following is a technical solution to solve the above problems, the specific contents are as follows:

[0066] The b_c types of sensors include cameras, lidars, millimeter-wave radars and infrared sensors installed on the vehicle body. The cameras are used to provide high-resolution 2D images, the lidars are used to provide high-precision 3D point cloud data, the millimeter-wave radars are used to provide distance, speed and relative motion information of objects, and the infrared sensors are used to provide temperature or thermal imaging data.

[0067] Cameras are used to provide high-resolution visual information and can capture detailed images and videos. They are suitable for use in daytime and well-lit environments, and are mainly used for visual features such as color, texture, and edge detection. Laser radar (LiDAR) is used to provide three-dimensional point cloud data of the environment. It can accurately perceive the shape and relative distance of objects, detect the geometric shape, size, and distance of objects, and is suitable for target detection in complex backgrounds and dynamic scenes. Millimeter-wave radar has the characteristics of strong penetration and can work normally in bad weather (such as heavy rain, heavy snow, haze, etc.), and is particularly suitable for detecting fast-moving targets. Infrared sensors effectively detect pedestrians and vehicles at night or in low-light environments through thermal imaging technology, and can detect objects in low-light or night environments.

[0068] Multi-sensor fusion can effectively improve the robustness of the system in various complex environments and overcome the limitations of a single sensor by integrating different types of sensor data.

[0069] The method for aligning the raw data of b_c types of sensors includes:

[0070] Step A1: Use the timestamp alignment method to synchronize the raw data of b_c types of sensors;

[0071] Step A2: Use coordinate transformation to spatially synchronize the raw data of b_c types of sensors;

[0072] The method of using the timestamp alignment method to synchronize the raw data of b_c types of sensors includes:

[0073] Select one of the sensors as the base sensor (the base sensor is generally selected to collect data at the same time point, and the original data transmission time is the shortest, that is, the sensor that is transmitted the fastest), calculate the time delay between the base sensor and any other sensors, set the cycle time T and the delay assumption value set, the delay assumption value set includes a preset time delay value, which can be set according to experimental data analysis or experience, and the original data of the base sensor and other sensors within the cycle time T are formed into two sets of time series A[T] and B[T];

[0074] Calculate the signal means μA and μB of A[T] and B[T], Where N represents the number of raw data in the period, i represents the time index, a_t(i) represents the raw data value at time i in the time series A[T], and b_t(i) represents the raw data value at time i in the time series B[T]. Decentralize the time series A[T] and B[T] to obtain A 1 [T] and B 1 [T],A 1 [T] = A[T] - μA, B 1 [T] = B[T] - μB, calculate the cross-correlation function Among them, a 1 _t(i) represents A 1 [T] is the data value at time i, b 1 _t(i+τ) represents the data value after time i is delayed by τ. τ is assigned to the value in the delay hypothesis value set and calculated. The cross-correlation function R is selected. AB The value of τ when (τ) is the maximum is taken as the actual delay time. The actual delay time is used to perform time correction on other sensors compared with the base sensor, and the corrected time series is obtained as B[T-τ];

[0075] Obtain the corrected time series of each sensor in turn, and use interpolation, discarding or repeating data to align the data of all sensors at the same time point;

[0076] For example: If the camera captures 30 frames of images per second, the LiDAR captures 10 frames of 3D point clouds per second, and the millimeter-wave radar obtains 20 speed samples per second, these data need to be discarded or repeated, and aligned according to the time stamp to ensure that there is valid data from each sensor at the same time point.

[0077] The data from different sensors are usually in their own coordinate systems, so coordinate transformation is required to map all the data into the same reference coordinate system.

[0078] The method of spatially synchronizing the raw data of b_c types of sensors using coordinate transformation includes:

[0079] Set the base coordinate system, which can be the vehicle coordinate system or the world coordinate system;

[0080] Convert the camera coordinate system to the base coordinate system, obtain the 2D image captured by the camera, and add the coordinate system to form a 2D image coordinate system. The pixel points are mapped to the base coordinate system through the camera intrinsic parameter matrix and extrinsic parameter matrix.

[0081] Among them, the camera's internal parameter matrix Among them, (f x , f y ) is the focal length, (c x , c y ) is the optical center position. The camera's external parameter matrix is ​​calibrated using a calibration plate (such as a chessboard). By taking z_p calibration plate images at different angles and using a camera calibration function (such as OpenCV) to calculate the relationship between the pixel coordinates of the corner points in the calibration plate image and the base coordinates, the camera's external parameter matrix can be obtained.

[0082] The method of converting the infrared sensor coordinate system to the base coordinate system is the same as that of the camera;

[0083] Convert the LiDAR coordinate system to the base coordinate system. LiDAR directly provides 3D point cloud data. Convert the LiDAR data to the base coordinate system through the LiDAR's external parameter matrix and translation vector;

[0084] Among them, the method for obtaining the external parameter matrix and translation vector of the laser radar is: use a known dot array to ensure that it can be captured by the laser radar and camera at different positions at the same time, use the dot array as the calibration target, and use the laser radar and camera to collect data at different angles and positions. Use point cloud registration technology to align the point cloud data of the laser radar with the image data of the camera to obtain the external parameter matrix and translation vector. The external parameter matrix is ​​optimized by nonlinear least squares method (such as Levenberg-Marquardt algorithm), which can be implemented using open source libraries (such as PCL, Open3D).

[0085] Point cloud registration technology can be ICP (Iterative Closest Point) or feature matching. ICP iteratively minimizes the distance between two sets of point clouds and calculates rotation and translation. Feature matching extracts feature points in the lidar and camera data for matching.

[0086] The method of converting the millimeter wave radar coordinate system into the base coordinate system is the same as that of the laser radar.

[0087] The extrinsic matrix and translation vector are calibrated by extrinsic parameters: The extrinsic parameters (position and rotation relationship) of sensors such as cameras, LiDAR, millimeter-wave radars, and infrared sensors relative to the base coordinate system need to be obtained through the calibration process. Calibration can be completed by collecting calibration targets (images and point clouds of calibration plates or known markers). After that, the data of LiDAR, millimeter-wave radar, and cameras are projected into the base coordinate system using the calibration results.

[0088] For example, suppose the camera is located 1 meter in front of the car, and the LiDAR is on the top of the car. Through calibration, their relative positions to the base coordinates (external parameter matrix) are obtained, and then the 3D point cloud coordinates of the LiDAR and the 2D image of the camera are projected into the base coordinate system for easy fusion. Then the coordinate systems of all sensors are converted to the same coordinate system to achieve spatial unification and alignment.

[0089] Multimodal deep learning models can fuse and process data from different sensors to improve the accuracy of multi-target detection and recognition.

[0090] Methods for fusing the preprocessed data of each sensor at the feature level include:

[0091] Set a comprehensive sample set, convert the image captured by the camera into a standard-sized RGB image or grayscale image, and perform standardization processing (such as normalization); voxelize the lidar point cloud data to generate a bird's-eye view or depth map; convert the millimeter-wave radar data into an image form (such as a radar reflection intensity map); convert the infrared sensor data into a heat map or grayscale map, and perform standardization processing; all collected data form a comprehensive sample set;

[0092] Build a deep neural network architecture for multi-sensor fusion, set the input of the input layer as a comprehensive sample set, design an independent feature extraction network for each sensor, use the CNN network to extract features from the data of the camera and infrared sensor, and use the 3D convolutional neural network to extract features from the data of the lidar and millimeter-wave radar, and then obtain the feature vector of each sensor, use weighted summation to fuse each feature vector into a large overall vector, input the large overall vector into the fully connected layer, and set the output layer to u_p classification task outputs and u_k regression task outputs. The classification task is used to identify objects of u_p categories, and the regression task output is used to predict the continuous attributes of the object, such as distance, speed, and direction;

[0093] The output of the integrated deep neural network architecture is used to obtain the current state of the environment outside the vehicle;

[0094] The weights of the weighted sum of each feature vector are obtained according to the self-attention mechanism. During the fusion process, the self-attention mechanism allows the feature vector of each sensor to interact with itself and the feature vectors of other sensors, and dynamically learn the correlation between the features of different sensors.

[0095] The specific steps are as follows:

[0096] Setting input:

[0097] Assume we have feature vectors from different sensors:

[0098] Camera features: F_ca, dimension is (B, D);

[0099] LiDAR feature: F_LiD, dimension is (B, D);

[0100] Millimeter wave radar feature: F_ra, dimension is (B, D);

[0101] Infrared feature: F_inf, dimension is (B, D);

[0102] Where B is the batch size and D is the dimension of the feature vector.

[0103] Feature matrix construction:

[0104] Combine these feature vectors into a matrix F, F = [F_ca, F_LiD, F_ra, F_inf], with shape (B, N, D), where N is the number of sensors (for example, 4 sensors) and D is the feature dimension of each sensor:

[0105] Generate query (Q), key (K) and value (V):

[0106] The input feature matrix F is linearly transformed to obtain the query matrix Q, key matrix K and value matrix V. The dimension of each matrix is ​​(B, N, D), indicating that each sensor generates independent query, key and value vectors, Q = W_q*F, Q = W_k*F, V = W_v*F, where W_q, W_k and W_v are learned weight matrices, usually of size (D, D).

[0107] Attention weight calculation:

[0108] Calculate the dot product of the query and the key, normalize the result through Softmax, and get the attention score matrix Here, K T is the transpose of the key matrix, is a scaling factor to ensure numerical stability.

[0109] Weighted sum:

[0110] Use the attention weight matrix A to perform weighted summation on the value matrix V and calculate the output feature matrix F_y = A·V;

[0111] Output:

[0112] The feature matrix F_y after attention is used as the fused feature vector and passed through the fully connected layer or directly as the input of the subsequent layer.

[0113] The classification task uses the Softmax function for activation, the regression task uses the linear activation function for activation, and the combined loss function is used The classification task uses cross entropy loss, and the regression task uses mean square error loss, where ε(j) represents the weight of the j-th classification task, CE_loss(j) represents the cross entropy loss of the j-th classification task, θ(u) represents the weight of the u-th classification task, and MSE_loss(u) represents the mean square error loss of the u-th regression task. Adam or SGD is used to optimize the model.

[0114] Image segmentation technology can be used to effectively separate the background and the target, thereby reducing the misidentification of objects in the background. Image segmentation technology is integrated into multi-sensor CNN feature extraction to output feature vectors. The methods of using CNN networks to extract features from camera and infrared sensor data include:

[0115] The input is set to the data of the camera and infrared sensor, and the size of the original image is (H, W, C), where H is the height, W is the width, and C is the number of channels;

[0116] Use multiple convolutional layers to extract local features Use the ReLU activation function to introduce nonlinearity, obtain the transformed output F_relu(k,h,m), downsample the feature map through the pooling layer, and repeat the convolution and pooling operations to obtain a low-resolution feature map (H 11 ,W 11 ,C);

[0117] Among them, H 11 is the height, W 11 is the width, F_con(k,h,m) represents the feature value of the mth channel at position (k,h) in the output feature map, I(k+p,h+r,c) is the pixel value of the input image at position (k,h) and (p,r) on channel c, W_m(p,r,c) represents the weight of the mth convolution kernel at position (p,r) and channel c, Bm represents the bias term of the mth convolution kernel, and K_P represents the size of the convolution kernel;

[0118] Use bilinear interpolation to restore the low-resolution feature map to the resolution of the original image to obtain the feature map F_up(k,h,m), jump connect the high-resolution feature map F_con(k,h,m) of the early convolutional layer with the upsampled feature map F_up(k,h,m) to obtain the feature map F_sk(k,h,m), perform global averaging on the feature map F_sk(k,h,m), and traverse each feature map to generate a feature vector of fixed length.

[0119] By integrating image segmentation technology into multi-sensor CNN feature extraction, technical problems such as background and target separation, feature extraction complexity, and spatial information loss are solved, efficient and accurate target recognition capabilities are achieved, and the robustness and feature representation capabilities of the model are enhanced.

[0120] Methods for extracting features from LiDAR and millimeter-wave radar data using 3D convolutional neural networks include:

[0121] The image data of the laser radar or millimeter wave radar is normalized and processed in size, and the continuous S_c frame images are stacked to form a four-dimensional tensor;

[0122] Design l_h convolutional layers, introduce adaptive pooling after each convolutional layer, use four-dimensional convolution kernel to convolve the input four-dimensional tensor, extract spatiotemporal features, output feature maps, add a timing modeling layer after the convolutional layer, the timing modeling layer is based on LSTM or GRU, and is used to capture the dynamic changes of time series data and enhance the understanding of target state changes. The output of the timing modeling layer is set to the motion information of the target. A fully connected layer is designed after the timing modeling layer, and the output of the fully connected layer is a feature vector representing the comprehensive feature information of the target.

[0123] By introducing time series modeling and adaptive pooling, dynamic changes can be better captured and the model's ability to understand the target can be improved while maintaining the integrity of feature information.

[0124] The current status data of the vehicle body includes the driving status of the vehicle and various control parameters, such as driving speed, direction and other equipment parameters that can control the vehicle. The current environmental status outside the vehicle is displayed on the display screen inside the vehicle, or voice broadcast is used for warning reminders. The driver controls the vehicle body according to the display and warning.

[0125] Embodiment 2

[0126] Reference Figure 1 and Figure 2 , Example 2 further illustrates a human-vehicle interaction recognition method proposed by the present invention.

[0127] The human-vehicle interaction recognition method comprises the following steps:

[0128] Step S1: Collecting raw data from b_c types of sensors;

[0129] Step S2: aligning the raw data of b_c types of sensors to obtain aligned pre-processed data;

[0130] Step S3: perform feature-level fusion on the pre-processed data of each sensor according to the deep neural network architecture to obtain the current environment status outside the vehicle;

[0131] Step S4: Collect the current status data of the vehicle body, and display warnings and reminders according to the current environmental status outside the vehicle.

[0132] The human-vehicle interaction recognition system is applied to a human-vehicle interaction recognition method and includes:

[0133] Data acquisition module: used to collect raw data from b_c types of sensors;

[0134] Data processing module: used to align the raw data of each sensor in time and space, and obtain pre-processed data after data alignment;

[0135] Vehicle state fitting prediction module: Build a deep neural network architecture, use the deep neural network architecture to fuse the pre-processed data of each sensor at the feature level, and obtain the current environment state outside the vehicle;

[0136] Control output module: used to interact with the driver and provide warnings and reminders to the driver based on the current environmental status outside the vehicle;

[0137] The modules are connected to each other via wired and / or wireless means.

[0138] In addition, according to the implementation of the present application, the process described in the figure of a method for identifying human-vehicle interaction can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application. Of course, the architecture shown in the figure of a method for identifying human-vehicle interaction is only exemplary. When implementing different devices, adaptive selection or adjustment can be made according to actual needs.

[0139] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0140] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A recognition method for human-vehicle interaction, characterized in that: The human-vehicle interaction recognition method comprises: Step S1: Collecting raw data from b_c types of sensors; Step S2: aligning the raw data of b_c types of sensors to obtain aligned pre-processed data; Step S3: perform feature-level fusion on the pre-processed data of each sensor according to the deep neural network architecture to obtain the current environment status outside the vehicle; Step S4: Collect the current status data of the vehicle body, and display warnings and reminders according to the current environmental status outside the vehicle.

2. The human-vehicle interaction recognition method according to claim 1, characterized in that: The b_c types of sensors include cameras, lidars, millimeter-wave radars and infrared sensors installed on the vehicle body. The cameras are used to provide high-resolution 2D images, the lidars are used to provide high-precision 3D point cloud data, the millimeter-wave radars are used to provide distance, speed and relative motion information of objects, and the infrared sensors are used to provide temperature or thermal imaging data.

3. The human-vehicle interaction recognition method according to claim 2, characterized in that: The method for aligning the raw data of b_c types of sensors includes: Step A1: Use the timestamp alignment method to synchronize the raw data of b_c types of sensors; Step A2: Use coordinate transformation to spatially synchronize the raw data of b_c types of sensors.

4. The human-vehicle interaction recognition method according to claim 3, characterized in that: The method for time synchronization of raw data of b_c types of sensors using a timestamp alignment method comprises: selecting one of the sensors as a base sensor, calculating the time delay between the base sensor and any other sensors, setting a cycle time T and a delay assumption value set, wherein the delay assumption value set includes a preset time delay value, and forming two sets of time series A[T] and B[T] with the raw data of the base sensor and other sensors within the cycle time T; Calculate the signal means μA and μB of A[T] and B[T], and decentralize the time series A[T] and B[T] to obtain A 1 [T] and B 1 [T], calculate the cross-correlation function Assign τ to the values ​​in the delay hypothesis value set and perform calculations, and select the cross-correlation function R AB The value of τ when (τ) is the maximum is taken as the actual delay time. The actual delay time is used to perform time correction on other sensors compared with the base sensor, and the corrected time series is obtained as B[T-τ]; The corrected time series of each sensor is obtained in turn, and the data of all sensors are aligned at the same time point by interpolation, discarding or repeating the data.

5. The human-vehicle interaction recognition method according to claim 4, characterized in that: The method for spatially synchronizing raw data of b_c types of sensors using coordinate transformation includes: Set the base coordinate system, which can be the vehicle coordinate system or the world coordinate system; Convert the camera coordinate system to the base coordinate system, obtain the 2D image captured by the camera, and add the coordinate system to form a 2D image coordinate system. The pixel points are mapped to the base coordinate system through the camera intrinsic parameter matrix and extrinsic parameter matrix. Among them, the camera's internal parameter matrix Among them, (f x , f y ) is the focal length, (c x , c y ) is the optical center position. The camera's external parameter matrix is ​​calibrated using a calibration plate. By taking z_p calibration plate images at different angles and using the camera calibration function to calculate the relationship between the pixel coordinates of the corner points in the calibration plate image and the base coordinates, the camera's external parameter matrix can be obtained. The method of converting the infrared sensor coordinate system to the base coordinate system is the same as that of the camera; Convert the laser radar coordinate system to the base coordinate system, and convert the laser radar data to the base coordinate system through the laser radar's external parameter matrix and translation vector; The method for obtaining the external parameter matrix and translation vector of the laser radar is as follows: using the dot array as the calibration target, using the laser radar and the camera to collect data at different angles and positions, using the point cloud registration technology to align the point cloud data of the laser radar with the image data of the camera, obtaining the external parameter matrix and translation vector, and optimizing the external parameter matrix by the nonlinear least squares method; The method of converting the millimeter wave radar coordinate system into the base coordinate system is the same as that of the laser radar.

6. The human-vehicle interaction recognition method according to claim 5, characterized in that: The method for fusing the preprocessed data of each sensor at the feature level includes: Set up a comprehensive sample set, convert the images captured by the camera into RGB images or grayscale images of standard size, and perform standardization processing; voxelize the laser radar point cloud data to generate a bird's-eye view or depth map; convert the millimeter wave radar data into image form; convert the infrared sensor data into a thermal map or grayscale map, and perform standardization processing; all collected data form a comprehensive sample set; Build a deep neural network architecture for multi-sensor fusion, set the input layer to a comprehensive sample set, use CNN network to extract features from camera and infrared sensor data, use 3D convolutional neural network to extract features from lidar and millimeter wave radar data, and then obtain the feature vector of each sensor. Use weighted summation to fuse each feature vector into a large overall vector, input the large overall vector into the fully connected layer, and set the output layer to u_p classification task outputs and u_k regression task outputs. The classification task is used to identify objects of u_p categories, and the regression task output is used to predict the continuous attributes of the object. The output of the integrated deep neural network architecture is used to obtain the current state of the environment outside the vehicle; The classification task uses the Softmax function for activation, the regression task uses the linear activation function for activation, and the combined loss function is used The classification task uses cross entropy loss, and the regression task uses mean square error loss, where ε(j) represents the weight of the j-th classification task, CE_loss(j) represents the cross entropy loss of the j-th classification task, θ(u) represents the weight of the u-th classification task, and MSE_loss(u) represents the mean square error loss of the u-th regression task. Adam or SGD is used to optimize the model.

7. The human-vehicle interaction recognition method according to claim 6, characterized in that: The method for extracting features from the camera and infrared sensor data using a CNN network includes: The input is set to the data of the camera and infrared sensor, and the size of the original image is (H, W, C), where H is the height, W is the width, and C is the number of channels; Use multiple convolutional layers to extract local features Use the ReLU activation function to introduce nonlinearity, obtain the transformed output F_relu(k,h,m), downsample the feature map through the pooling layer, and repeat the convolution and pooling operations to obtain a low-resolution feature map (H 11 ,W 11 ,C); Among them, H 11 is the height, W 11 is the width, F_con(k,h,m) represents the feature value of the mth channel at position (k,h) in the output feature map, I(k+p,h+r,c) is the pixel value of the input image at position (k,h) and (p,r) on channel c, W_m(p,r,c) represents the weight of the mth convolution kernel at position (p,r) and channel c, Bm represents the bias term of the mth convolution kernel, and K_P represents the size of the convolution kernel; Use bilinear interpolation to restore the low-resolution feature map to the resolution of the original image to obtain the feature map F_up(k,h,m), jump connect the high-resolution feature map F_con(k,h,m) of the early convolutional layer with the upsampled feature map F_up(k,h,m) to obtain the feature map F_sk(k,h,m), perform global averaging on the feature map F_sk(k,h,m), and traverse each feature map to generate a feature vector of fixed length.

8. The human-vehicle interaction recognition method according to claim 7, characterized in that: The method for extracting features of the laser radar and millimeter wave radar data using a 3D convolutional neural network includes: performing size standardization and normalization processing on the image data of the laser radar or millimeter wave radar, and stacking continuous S_c frame images to form a four-dimensional tensor; Design l_h convolutional layers, introduce adaptive pooling after each convolutional layer, use four-dimensional convolution kernel to convolve the input four-dimensional tensor, extract spatiotemporal features, output feature maps, add a timing modeling layer after the convolutional layer, the timing modeling layer is based on LSTM or GRU, set the output of the timing modeling layer as the motion information of the target, design a fully connected layer after the timing modeling layer, and the output of the fully connected layer is a feature vector representing the comprehensive feature information of the target.

9. The human-vehicle interaction recognition method according to claim 8, characterized in that: The current status data of the vehicle body includes the driving status of the vehicle and various control parameters. The current environmental status outside the vehicle is displayed on the display screen inside the vehicle, or a voice broadcast is used for warning reminders. The driver controls the vehicle body according to the display and warning.

10. A recognition system for human-vehicle interaction, applied to any one of claims 1 to 9, characterized in that: The human-vehicle interaction recognition method comprises: Data acquisition module: used to collect raw data from b_c types of sensors; Data processing module: used to align the raw data of each sensor in time and space, and obtain pre-processed data after data alignment; Vehicle state fitting prediction module: Build a deep neural network architecture, use the deep neural network architecture to fuse the pre-processed data of each sensor at the feature level, and obtain the current environmental state outside the vehicle; Control output module: used to interact with the driver and provide warnings and reminders to the driver based on the current environmental status outside the vehicle.

Images