Gesture action recognition and control method and device for vehicle-mounted electric tailgate

Abstract

The embodiment of the application provides a gesture action recognition and control method and device for a vehicle-mounted electric tail gate, accurate recognition of gestures is realized through radar and vision fusion. A control mechanism is constructed, reliable action recognition strategies are established by combining feature fusion and space-time analysis. Control optimization is introduced, and continuous improvement of operation is ensured through intention confirmation and safety inspection. The method effectively solves the shortcomings of traditional technologies in aspects of perception alignment, feature fusion and control execution, and provides technical support for intelligent control of the vehicle-mounted electric tail gate.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method and device for gesture recognition and control of vehicle power tailgates. Background Technology

[0002] Existing gesture control methods for vehicle power tailgates have significant shortcomings. Traditional systems perform poorly in multimodal perception and data alignment, failing to effectively and accurately recognize gestures and thus affecting control performance.

[0003] Furthermore, existing technologies face bottlenecks in feature fusion and action recognition. Most systems lack robust attention fusion mechanisms and spatiotemporal feature extraction strategies, resulting in less than ideal gesture recognition accuracy.

[0004] Existing systems have technical shortcomings in control execution. The lack of in-depth analysis of user intent makes it difficult to achieve efficient safety control through state machines, thus affecting operational reliability. Solving these problems is crucial for improving the intelligent control capabilities of electric tailgates. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides a gesture recognition and control method and device for vehicle power tailgates, which can effectively solve the shortcomings of traditional technologies in terms of perception alignment, feature fusion and control execution, and provide technical support for intelligent control of vehicle power tailgates.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] In a first aspect, this application provides a gesture recognition and control method for vehicle power tailgates, including:

[0008] Based on the echo signal collected by the millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, a range Doppler map is generated by range dimension transformation and velocity dimension transformation. The range Doppler map is then subjected to constant false alarm rate detection and angle estimation to obtain radar point cloud frames. Based on the image frame collected by the near-infrared camera and the hand region detection, a hand region image block is obtained. The radar point cloud frame and the hand region image block are aligned and paired according to the timestamp to obtain a multimodal perception data frame.

[0009] The radar point cloud frame sequence in the multimodal perception data frame is associated with target tracking through spatial clustering to obtain radar gesture feature vectors. The hand region image block sequence is input into a visual feature encoding network to obtain visual gesture feature vectors. Based on the radar gesture feature vectors and the visual gesture feature vectors, a multimodal fused gesture feature vector is obtained through cross-attention fusion. The multimodal fused gesture feature vector sequence is input into a spatiotemporal convolutional network to obtain gesture category probability distribution and action completion estimation. Based on the gesture category probability distribution and the action completion estimation, a multi-level confidence screening is performed to obtain effective recognition results.

[0010] Based on the preset intent confirmation mode, the valid recognition result is used to perform user intent confirmation to obtain intent confirmation result. The intent confirmation result is input into the power tailgate control state machine and combined with the vehicle status to perform safety checks and state transition determination to obtain power tailgate control command. The power tailgate control command is then sent to the power tailgate actuator.

[0011] Furthermore, it also includes: reading intermediate frequency signal data based on a millimeter-wave radar sensor deployed in the rear area of ​​the vehicle at a preset sampling period, obtaining range spectrum data by performing a range dimension fast Fourier transform on the intermediate frequency signal data, generating a range Doppler map by performing a velocity dimension fast Fourier transform on the range spectrum data, and characterizing the two-dimensional energy distribution of the target with distance and velocity as coordinate axes;

[0012] Based on the range Doppler image, constant false alarm rate (CFAR) detection is performed according to a configurable detection threshold to obtain a set of target detection points exceeding the threshold. Angle estimation is performed on each detection point in the target detection point set based on the phase difference of multiple receiving antennas to obtain the azimuth and elevation angle estimates of each detection point. The range, velocity, signal strength, azimuth, and elevation angle estimates of each detection point are assembled and a frame timestamp is added to obtain a radar point cloud frame.

[0013] Furthermore, it also includes: acquiring image frames at a preset frame rate based on a near-infrared camera deployed in the rear area of ​​the vehicle; obtaining preprocessed image frames by distortion correction and grayscale normalization of the image frames; inputting the preprocessed image frames into a lightweight hand detection network to obtain hand bounding box coordinates and detection confidence; and performing region cropping and size normalization on the preprocessed image frames based on the hand bounding box coordinates to obtain hand region image blocks.

[0014] Based on the frame timestamp of the radar point cloud frame and the frame timestamp of the hand region image block, time alignment matching is performed. Radar point cloud frames and hand region image blocks with timestamp differences within a preset pairing tolerance threshold are marked as valid pairs. The validly paired radar point cloud frames and hand region image blocks are assembled to obtain a multimodal sensing data frame and written into the multimodal data buffer.

[0015] Furthermore, it also includes: reading multiple consecutive radar point cloud frames from a multimodal data buffer to form a radar point cloud frame sequence; performing density-based spatial clustering on the radar point cloud frame sequence within a preset temporal aggregation window to obtain a target cluster set; performing cross-frame target tracking association on the target cluster set using Kalman filtering and Hungarian matching algorithm to obtain a target trajectory; and calculating the mean distance, mean velocity, angle distribution range, and mean signal strength of the cluster centroid based on the target trajectory, and extracting the distance change curve and velocity change curve of the trajectory to assemble a radar gesture feature vector.

[0016] A sequence of hand region image blocks is formed by reading multiple consecutive frames of hand region image blocks from a multimodal data buffer. Each frame in the hand region image block sequence is input into a visual feature encoding network based on a depthwise separable convolutional architecture to extract spatial feature vectors. The spatial feature vector sequence is then passed through a temporal convolutional layer to extract temporal features and obtain a visual gesture feature vector.

[0017] Furthermore, it also includes: mapping the radar gesture feature vector and the visual gesture feature vector to a common feature space through a linear projection layer to obtain a radar projection feature vector and a visual projection feature vector, performing bidirectional cross-attention weighting with the radar projection feature vector and the visual projection feature vector as queries to obtain an attention-enhanced feature vector pair, and inputting the attention-enhanced feature vector pair into a fully connected layer after residual connection and concatenation to obtain a multimodal fusion gesture feature vector;

[0018] The multimodal fused gesture feature vector sequence is input into a spatiotemporal convolutional network based on a causal dilated convolutional architecture to obtain a gesture category probability distribution and an estimated action completion degree. The highest probability value in the gesture category probability distribution is compared with a preset recognition confidence threshold to perform recognition confidence filtering. The estimated action completion degree is compared with a preset completion threshold to perform action integrity filtering. The gesture category labels of multiple consecutive inference cycles are subjected to temporal consistency filtering to obtain an effective recognition result.

[0019] Furthermore, it also includes: reading the current intent confirmation mode identifier according to the system configuration, wherein the intent confirmation mode identifier corresponds to one of the instant confirmation mode, hover confirmation mode, and secondary gesture confirmation mode, and retrieving the corresponding confirmation processing rules and confirmation parameter configuration based on the intent confirmation mode identifier;

[0020] In the instant confirmation mode, the valid recognition result is directly marked as intent confirmation passed. In the hover confirmation mode, the radar speed measurement value is used to determine whether the user's hand remains in a relatively stationary state below the hover speed threshold within the preset hover confirmation time. In the secondary gesture confirmation mode, the user is detected to perform the preset confirmation gesture within the preset confirmation waiting time. Valid recognition results that meet the corresponding confirmation conditions are marked as intent confirmation passed and a confirmation timestamp is added to obtain the intent confirmation result.

[0021] Furthermore, it also includes: converting the gesture category in the intent confirmation result into a state transition event input to the power tailgate control state machine; querying the state transition table based on the current state of the power tailgate control state machine and the state transition event to determine the target state; reading the vehicle gear state and vehicle speed state according to the target state to perform a safety check; and generating a power tailgate control command from the state transition request that has passed the safety check.

[0022] The power tailgate control command is sent to the power tailgate actuator via the drive interface to perform opening, closing, and pausing actions. During the execution process, the clamping force and obstacle detection signal are continuously monitored. If an abnormal monitoring signal is detected, a safety interlock is triggered and an emergency stop command is generated and sent to the power tailgate actuator.

[0023] Secondly, this application provides a gesture recognition and control device for vehicle power tailgates, comprising:

[0024] The data perception module is used to collect echo signals based on millimeter-wave radar sensors deployed in the rear area of ​​the vehicle, generate a range Doppler map through range dimension transformation and velocity dimension transformation, obtain radar point cloud frames by constant false alarm rate detection and angle estimation of the range Doppler map, acquire image frames based on near-infrared cameras and obtain hand region image blocks by hand region detection, and perform alignment and pairing of radar point cloud frames and hand region image blocks according to timestamps to obtain multimodal perception data frames;

[0025] The gesture recognition module is used to obtain radar gesture feature vectors by spatially clustering and associating the radar point cloud frame sequence in the multimodal perception data frame with target tracking; input the hand region image block sequence into a visual feature encoding network to obtain a visual gesture feature vector; fuse the radar gesture feature vector and the visual gesture feature vector through cross-attention to obtain a multimodal fused gesture feature vector; input the multimodal fused gesture feature vector sequence into a spatiotemporal convolutional network to obtain a gesture category probability distribution and an action completion estimate; and obtain an effective recognition result by multi-level confidence filtering based on the gesture category probability distribution and the action completion estimate.

[0026] The tailgate control module is used to perform user intent confirmation on the valid recognition result according to the preset intent confirmation mode to obtain the intent confirmation result, input the intent confirmation result into the electric tailgate control state machine and perform safety checks and state transition determination in combination with the vehicle status to obtain the electric tailgate control command, and send the electric tailgate control command to the electric tailgate actuator.

[0027] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the gesture recognition and control method for vehicle-mounted electric tailgates.

[0028] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gesture recognition and control method for a vehicle-mounted electric tailgate.

[0029] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the gesture recognition and control method for vehicle-mounted electric tailgates.

[0030] As described above, this application provides a gesture recognition and control method and device for vehicle power tailgates. It achieves accurate gesture recognition through radar and vision fusion. A control mechanism is constructed, combining feature fusion and spatiotemporal analysis to establish a reliable action recognition strategy. Control optimization is introduced, ensuring continuous improvement of operation through intent confirmation and safety checks. This method effectively addresses the shortcomings of traditional technologies in perception alignment, feature fusion, and control execution, providing technical support for intelligent control of vehicle power tailgates. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the gesture recognition and control method for vehicle-mounted electric tailgates in an embodiment of this application. Figure 2 This is a structural diagram of the gesture recognition and control device for vehicle power tailgate according to an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0035] In view of the problems existing in the prior art, this application provides a gesture recognition and control method and device for vehicle power tailgates. It achieves accurate gesture recognition through radar and vision fusion. A control mechanism is constructed, combining feature fusion and spatiotemporal analysis to establish a reliable action recognition strategy. Control optimization is introduced, ensuring continuous improvement of operation through intent confirmation and safety checks. This method effectively solves the shortcomings of traditional technologies in perception alignment, feature fusion, and control execution, providing technical support for intelligent control of vehicle power tailgates.

[0036] To effectively address the shortcomings of traditional technologies in areas such as perception alignment, feature fusion, and control execution, and to provide technical support for intelligent control of vehicle power tailgates, this application provides an embodiment of a gesture recognition and control method for vehicle power tailgates. See [link to embodiment]. Figure 1 The gesture recognition and control method for vehicle power tailgates specifically includes the following:

[0037] Step S101: Based on the echo signal collected by the millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, a range Doppler map is generated by range dimension transformation and velocity dimension transformation. The range Doppler map is then subjected to constant false alarm rate detection and angle estimation to obtain a radar point cloud frame. Based on the image frame collected by the near-infrared camera and the hand area detection, a hand area image block is obtained. The radar point cloud frame and the hand area image block are aligned and paired according to the timestamp to obtain a multimodal perception data frame.

[0038] In this embodiment, echo signals are read from a millimeter-wave radar sensor deployed in the rear area of ​​the vehicle. The millimeter-wave radar sensor outputs intermediate frequency (IF) signal data according to a preset sampling period. The IF signal data is then subjected to a range-dimensional fast Fourier transform to obtain range spectrum data, which characterizes the echo energy distribution at different range units.

[0039] After the range spectrum data is generated, this embodiment performs a velocity-dimensional Fast Fourier Transform on it, performing spectral analysis along the slow time dimension on the range spectrum data of multiple chirped periods to generate a range Doppler map. The range Doppler map forms a two-dimensional energy distribution matrix with distance and velocity as coordinate axes, and the value of each cell in the matrix corresponds to the target echo intensity at a specific combination of distance and velocity.

[0040] Based on the range Doppler image, this embodiment performs constant false alarm rate (CFAR) detection according to a configurable detection threshold. CFAR detection compares the energy value of each cell in the range Doppler image with the statistical characteristics of its neighboring cells, marking cells exceeding an adaptive threshold as target detection points. Each detection point in the target detection point set carries a corresponding distance value, velocity value, and signal strength value.

[0041] After the target detection point set is determined, this embodiment performs angle estimation processing on each detection point. Angle estimation is based on the phase difference information between the multiple receiving antennas of the millimeter-wave radar, and the azimuth and elevation angle estimates for each detection point are calculated using a beamforming algorithm. This embodiment assembles the distance, velocity, signal strength, azimuth, and elevation angle estimates for each detection point into a five-tuple structure and adds a frame timestamp to form a radar point cloud frame.

[0042] This embodiment simultaneously acquires image frames from a near-infrared camera deployed at the rear of the vehicle at a preset frame rate. The near-infrared camera is equipped with an active near-infrared illumination module, enabling clear imaging even in low-visible-light environments. The image frames, after distortion correction and grayscale normalization, are input into a lightweight hand detection network, which outputs the hand bounding box coordinates and detection confidence score. Based on the hand bounding box coordinates, this embodiment performs region cropping and size normalization on the preprocessed image frames to obtain hand region image blocks.

[0043] After the radar point cloud frame and the hand region image block are generated, this embodiment performs time alignment matching based on their frame timestamps. The alignment strategy is to find the hand region image block with the smallest timestamp difference for each radar point cloud frame as a candidate pair. When the timestamp difference is within a preset pairing tolerance threshold, the pair is marked as a valid pair. The validly paired radar point cloud frame and the hand region image block are assembled to form a multimodal sensing data frame. The multimodal sensing data frame is written into a multimodal data buffer for subsequent steps S201 to read and perform feature extraction and fusion processing.

[0044] Step S102: The radar point cloud frame sequence in the multimodal perception data frame is associated with target tracking through spatial clustering to obtain radar gesture feature vectors. The hand region image block sequence is input into a visual feature encoding network to obtain visual gesture feature vectors. Based on the radar gesture feature vectors and the visual gesture feature vectors, multimodal fusion gesture feature vectors are obtained through cross-attention fusion. The multimodal fusion gesture feature vector sequence is input into a spatiotemporal convolutional network to obtain gesture category probability distribution and action completion estimation. Based on the gesture category probability distribution and the action completion estimation, effective recognition results are obtained through multi-level confidence filtering.

[0045] In this embodiment, the multimodal sensing data frames generated in step S101 are read from the multimodal data buffer, and radar point cloud frames from multiple consecutive frames are extracted to form a radar point cloud frame sequence. The radar point cloud frame sequence is subjected to density-based spatial clustering within a preset temporal aggregation window. The clustering process groups point clouds that are spatially adjacent and have similar motion characteristics into the same target cluster, and outputs a target cluster set.

[0046] After the target cluster set is generated, this embodiment performs cross-frame target tracking association. The tracking association uses a Kalman filter to predict the motion state of each target cluster and a Hungarian matching algorithm to establish a correspondence between the target clusters in the current frame and the tracking trajectories in previous frames. Successfully matched target clusters inherit existing trajectory identifiers and update their trajectory states, while unmatched target clusters are initialized with new trajectories. Based on the tracked and associated target trajectories, this embodiment calculates the average distance, average velocity, angular distribution range, and average signal strength of the cluster centroids, and extracts the distance and velocity variation curves of the trajectory within a time window. These features are then assembled to obtain a radar gesture feature vector.

[0047] This embodiment simultaneously extracts multiple consecutive frames of hand region image patches from the multimodal sensing data frames to form a hand region image patch sequence. Each frame in the hand region image patch sequence is input to a visual feature encoding network based on a depthwise separable convolutional architecture. This visual feature encoding network extracts spatial feature vectors from single-frame image patches. The spatial feature vectors of each frame are arranged temporally and then passed through a temporal convolutional layer to extract inter-frame dynamic change features, outputting a visual gesture feature vector.

[0048] After both the radar gesture feature vector and the visual gesture feature vector are generated, this embodiment performs cross-attention fusion on them. The fusion process first maps the two feature vectors to a common feature space via linear projection layers, obtaining radar projection feature vectors and visual projection feature vectors. In this embodiment, the radar projection feature vector is used as a query to perform attention weighting on the visual projection feature vector, and simultaneously, the visual projection feature vector is used as a query to perform attention weighting on the radar projection feature vector, forming a bidirectional cross-attention enhanced feature vector pair. This feature vector pair is then fed into a fully connected layer after residual connection and concatenation to compress its dimensions, resulting in a multimodal fused gesture feature vector.

[0049] In this embodiment, the multimodal fused gesture feature vectors from multiple consecutive time points are organized temporally into a multimodal fused gesture feature vector sequence, which is then input into a spatiotemporal convolutional network based on a causal dilated convolutional architecture. The causal convolutional constraints of the spatiotemporal convolutional network ensure that the output at each time point depends only on the current and historical inputs, and the dilation factor increases layer by layer to expand the temporal receptive field. The spatiotemporal convolutional network outputs a gesture category probability distribution after global temporal pooling and a classification head, and outputs an estimated action completion value after an action completion evaluation head.

[0050] Based on the gesture category probability distribution and the estimated action completion rate, this embodiment performs multi-level confidence filtering. The first level of filtering compares the highest probability value in the gesture category probability distribution with a preset recognition confidence threshold; recognition results below the threshold are marked as low confidence and subsequent processing is terminated. The second level of filtering compares the estimated action completion rate with a preset completion threshold; recognition results below the threshold are marked as incomplete action and await subsequent inference updates. The third level of filtering performs a temporal consistency check on the gesture category labels across multiple consecutive inference cycles, requiring that the category labels remain consistent across consecutive cycles. Recognition results that pass all three levels of filtering are marked as valid recognition results, which will be input into the intent confirmation module in subsequent step S103 for user intent verification.

[0051] Step S103: Perform user intent confirmation on the valid recognition result according to the preset intent confirmation mode to obtain intent confirmation result, input the intent confirmation result into the power tailgate control state machine and perform safety checks and state transition determination in combination with the vehicle status to obtain power tailgate control command, and send the power tailgate control command to the power tailgate actuator.

[0052] This embodiment reads the current intent confirmation mode identifier based on the system configuration. The intent confirmation mode identifier corresponds to one of three types: instant confirmation mode, hover confirmation mode, and secondary gesture confirmation mode. Based on the intent confirmation mode identifier, the corresponding confirmation processing rules and confirmation parameter configuration are retrieved, and user intent confirmation is performed on the valid recognition result output in the aforementioned step S102.

[0053] In instant confirmation mode, this embodiment directly marks the valid recognition result as intent confirmation passed. This mode is suitable for usage scenarios with high response speed requirements. In hover confirmation mode, this embodiment determines whether the user's hand remains relatively stationary based on radar speed measurement values. When the speed measurement value remains below the hover speed threshold for a preset hover confirmation time, the hover state is determined to be established, and the valid recognition result is marked as intent confirmation passed. If the hand moves out of the detection area or the speed exceeds the threshold during hovering, the recognition is canceled.

[0054] In the secondary gesture confirmation mode, this embodiment detects whether the user has performed a preset confirmation gesture within a preset confirmation waiting time. The recognition process for the confirmation gesture reuses the recognition path of the aforementioned step S102. When an action matching the preset confirmation gesture is detected, the valid recognition result is marked as intent confirmation passed. If no confirmation gesture is detected after the confirmation waiting time, the recognition is canceled. This embodiment adds a confirmation timestamp to the result that meets the corresponding confirmation conditions to form an intent confirmation result.

[0055] After the intent confirmation result is generated, this embodiment converts the gesture category into a state transition event input to the power tailgate control state machine. The power tailgate control state machine includes a set of state nodes comprising idle state, door opening waiting state, door opening execution state, door opening completed state, door closing waiting state, door closing execution state, door closing completed state, paused state, and fault state. This embodiment uses the current state of the power tailgate control state machine and the state transition event to query the state transition table to determine the target state and corresponding output action.

[0056] After the target state is determined, this embodiment reads the vehicle's gear position and speed status based on the target state and performs a safety check. The safety check items include whether the vehicle is in park, whether the vehicle speed is below a safety threshold, and whether the tailgate locking mechanism is in an operable state. When all safety check items pass, this embodiment generates a power tailgate control command and executes a state transition. When any safety check item fails, the power tailgate control state machine maintains its current state and triggers an audible and visual alert to notify the user.

[0057] In this embodiment, the power tailgate control command is sent to the power tailgate actuator via the drive interface. The power tailgate actuator performs opening, closing, and pausing actions according to the command. During execution, this embodiment continuously monitors the clamping force signal and obstacle detection signal. The clamping force signal is obtained from the current feedback of the power tailgate drive motor, and the obstacle detection signal is generated by the fusion judgment of millimeter-wave radar and near-infrared camera. When the clamping force signal exceeds a preset threshold or the obstacle detection signal is triggered, this embodiment activates the safety interlock mechanism to generate an emergency stop command and sends it to the power tailgate actuator, and transfers the power tailgate control state machine to the pause state. The status feedback after execution is written to the control log storage area for the system adaptive module to read.

[0058] As described above, the gesture recognition and control method for vehicle power tailgates provided in this application can achieve accurate gesture recognition through radar and vision fusion. A control mechanism is constructed, combining feature fusion and spatiotemporal analysis to establish a reliable action recognition strategy. Control optimization is introduced, ensuring continuous improvement of operation through intent confirmation and safety checks. This method effectively solves the shortcomings of traditional technologies in perception alignment, feature fusion, and control execution, providing technical support for intelligent control of vehicle power tailgates.

[0059] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0060] Step S201: Based on the millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, read the intermediate frequency signal data according to the preset sampling period, and obtain the range spectrum data by performing a range dimension fast Fourier transform on the intermediate frequency signal data. Then, generate a range Doppler map by performing a velocity dimension fast Fourier transform on the range spectrum data. The range Doppler map represents the two-dimensional energy distribution of the target with distance and velocity as coordinate axes.

[0061] Step S202: Based on the range Doppler image, perform constant false alarm rate detection according to the configurable detection threshold to obtain a set of target detection points exceeding the threshold. Perform angle estimation on each detection point in the target detection point set according to the phase difference of multiple receiving antennas to obtain the azimuth angle estimate and elevation angle estimate of each detection point. Assemble the distance value, velocity value, signal strength value, azimuth angle estimate, and elevation angle estimate of each detection point and add a frame timestamp to obtain a radar point cloud frame.

[0062] This embodiment reads intermediate frequency (IF) signal data from a millimeter-wave radar sensor deployed at the rear of the vehicle at a preset sampling period. The millimeter-wave radar sensor operates using a frequency-modulated continuous wave (FMCV) system. The transmitting end outputs a chirped signal whose frequency varies linearly with time. The receiving end mixes the target's reflected echo with the local oscillator signal to generate an IF signal. The frequency of the IF signal data carries target distance information, and the phase carries target velocity information.

[0063] After the intermediate frequency (IF) signal data is read, this embodiment performs a range-dimensional Fast Fourier Transform (FFT) on it. The range-dimensional FFT performs spectral analysis along the sampling point sequence within a single chirped period, converting the time-domain IF signal into frequency-domain range spectrum data. Each frequency component in the range spectrum data corresponds to a specific range cell, and the amplitude of the frequency component reflects the echo energy intensity at that range cell. The range resolution is determined by the bandwidth of the chirped signal; the larger the bandwidth, the smaller the interval between adjacent range cells.

[0064] Based on the range spectrum data, this embodiment performs a velocity-dimensional Fast Fourier Transform (FFT) along the slow time dimension on the range spectrum of multiple consecutive chirped cycles. The velocity-dimensional FFT extracts the phase change pattern of the same range unit across multiple chirped cycles, converting the phase change rate into a Doppler frequency shift to calculate the target's radial velocity. The output of the velocity-dimensional FFT, together with the range spectrum data, constitutes a range-Doppler map, which represents the two-dimensional energy distribution of the target with distance as the vertical axis and velocity as the horizontal axis. For example, when a user waves their hand towards the rear of the vehicle, it appears as an energy accumulation region with a positive velocity component within a specific distance range in the range-Doppler map.

[0065] After the range Doppler map is generated, this embodiment performs constant false alarm rate (CFAR) detection based on a configurable detection threshold. CFAR detection establishes a local reference window for each cell in the range Doppler map, and uses the mean or ordered statistic of the energy values ​​of the cells within the reference window as a background noise estimate. This background noise estimate is then multiplied by a threshold factor to obtain an adaptive detection threshold. When the energy value of a cell to be detected exceeds the adaptive detection threshold, the cell is marked as a target detection point, and its corresponding distance, velocity, and signal strength values ​​are recorded. All cells exceeding the threshold constitute the target detection point set.

[0066] Based on the target detection point set, this embodiment performs angle estimation processing on each detection point. The millimeter-wave radar sensor is equipped with multiple receiving antennas. When the echo signal from the same target reaches each receiving antenna, there is a path difference, which causes a phase difference between the received signals of each antenna. This embodiment calculates the azimuth and elevation angle estimates for each detection point using a beamforming algorithm, based on the correspondence between the phase difference and the antenna array geometric parameters. The azimuth angle estimate represents the target's deviation angle from the radar normal in the horizontal plane, and the elevation angle estimate represents the target's deviation angle from the radar normal in the vertical plane.

[0067] In this embodiment, the distance, velocity, signal strength, azimuth estimate, and elevation estimate of each detection point are assembled into a quintuple data structure. The quintuples of all detection points within the same frame are arranged in sequence and a frame timestamp is added to form a radar point cloud frame. The radar point cloud frame will be time-aligned and paired with the hand region image block in subsequent step S301 to jointly constitute a multimodal sensing data frame.

[0068] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0069] Step S301: Based on the near-infrared camera deployed in the rear area of ​​the vehicle, image frames are acquired at a preset frame rate. The image frames are processed by distortion correction and grayscale normalization to obtain preprocessed image frames. The preprocessed image frames are input into a lightweight hand detection network to obtain hand bounding box coordinates and detection confidence. Based on the hand bounding box coordinates, the preprocessed image frames are cropped and normalized to obtain hand region image blocks.

[0070] Step S302: Perform time alignment matching based on the frame timestamp of the radar point cloud frame and the frame timestamp of the hand region image block. Mark the radar point cloud frame and the hand region image block with the timestamp difference within the preset pairing tolerance threshold as valid pairings. Assemble the validly paired radar point cloud frame and the hand region image block to obtain a multimodal sensing data frame and write it into the multimodal data buffer.

[0071] This embodiment acquires image frames from a near-infrared camera deployed at the rear of the vehicle at a preset frame rate. The near-infrared camera is sensitive to near-infrared light and, in conjunction with an active near-infrared illumination module, can obtain clear images at night and in low-visibility light environments. The field of view of the acquired image frames covers the spatial range overlapping with the detection area of ​​the millimeter-wave radar, ensuring that the same target can be observed simultaneously by both sensors.

[0072] After the image frame is acquired, this embodiment performs distortion correction processing on it. Distortion correction compensates for radial and tangential distortions caused by lens optical characteristics in the image based on camera calibration parameters, restoring curved straight lines to geometrically correct shapes. The distortion-corrected image frame then undergoes grayscale normalization processing. Grayscale normalization maps image pixel values ​​to a uniform numerical range, eliminating the influence of brightness differences under different lighting conditions on subsequent processing. After distortion correction and grayscale normalization processing, a preprocessed image frame is obtained.

[0073] Based on the preprocessed image frames, this embodiment inputs them into a lightweight hand detection network to perform hand region localization. The lightweight hand detection network employs an embedded platform-optimized object detection architecture, performing multi-scale feature extraction and candidate box regression on the input image. The lightweight hand detection network outputs hand bounding box coordinates and detection confidence scores. The hand bounding box coordinates represent the spatial extent of the hand region in the image using the pixel positions of the top-left and bottom-right corners, and the detection confidence score characterizes the probability that the bounding box contains the hand target.

[0074] After the coordinates of the hand bounding box are determined, this embodiment performs region cropping on the preprocessed image frame based on these coordinates. Region cropping extracts the rectangular region defined by the hand bounding box from the preprocessed image frame and removes background pixels unrelated to gesture recognition. The cropped region then undergoes size normalization processing, scaling hand regions of different sizes to a fixed size to obtain hand region image blocks. A corresponding frame timestamp is appended to each hand region image block for subsequent time alignment.

[0075] In this embodiment, time alignment matching is performed based on the frame timestamp of the radar point cloud frame generated in step S202 and the frame timestamp of the hand region image block. The time alignment strategy is to traverse the sequence of hand region image blocks for each radar point cloud frame, calculate the absolute value of the difference between the frame timestamp of each image block and the timestamp of the radar point cloud frame, and select the image block with the smallest absolute difference as the candidate pairing object.

[0076] After the candidate pairings are determined, this embodiment compares the absolute value of the timestamp difference with a preset pairing tolerance threshold. When the absolute value of the timestamp difference is within the preset pairing tolerance threshold range, the radar point cloud frame and the corresponding hand region image block are marked as a valid pairing. When the absolute value of the timestamp difference exceeds the preset pairing tolerance threshold, the pairing is marked as invalid and discarded. The validly paired radar point cloud frame and the hand region image block are assembled to form a multimodal sensing data frame. The multimodal sensing data frame is written into a multimodal data buffer for subsequent steps S401 to read and perform feature extraction processing.

[0077] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0078] Step S401: Read multiple consecutive radar point cloud frames from the multimodal data buffer to form a radar point cloud frame sequence. Perform density-based spatial clustering on the radar point cloud frame sequence within a preset temporal aggregation window to obtain a target cluster set. Perform cross-frame target tracking association on the target cluster set using Kalman filtering and Hungarian matching algorithm to obtain the target trajectory. Calculate the mean distance, mean velocity, angle distribution range, and mean signal strength of the cluster centroid based on the target trajectory, and extract the distance change curve and velocity change curve of the trajectory to assemble a radar gesture feature vector.

[0079] Step S402: Read consecutive frames of hand region image blocks from the multimodal data buffer to form a hand region image block sequence. Input each frame in the hand region image block sequence into a visual feature encoding network based on a depth-separable convolutional architecture to extract spatial feature vectors. Extract temporal features from the spatial feature vector sequence through a temporal convolutional layer to obtain a visual gesture feature vector.

[0080] In this embodiment, the multimodal sensing data frames written in step S302 are read from the multimodal data buffer, and radar point cloud frames from multiple consecutive frames are extracted to form a radar point cloud frame sequence. The number of frames in the radar point cloud frame sequence is determined by the preset temporal aggregation window length, which covers the typical time span of the gesture action from start to completion.

[0081] After reading the radar point cloud frame sequence, this embodiment performs density-based spatial clustering processing. Spatial clustering uses the three-dimensional spatial coordinates formed by the distance, azimuth, and elevation angle estimates of each detection point as the basis, grouping spatially adjacent detection points into the same cluster. The clustering process starts from any unvisited detection point, searching for neighboring points within its neighborhood radius. When the number of points in the neighborhood exceeds a density threshold, the point is marked as a core point, and cluster members are recursively expanded. After processing all detection points, a target cluster set is output, where each cluster corresponds to an independent moving target within the detection area.

[0082] Based on the aforementioned target cluster set, this embodiment performs cross-frame target tracking association. The tracking association first applies a Kalman filter to each established trajectory, predicting the expected position and velocity of the target in the current frame based on the trajectory's historical state. This embodiment calculates the distance cost between the centroid of each target cluster and the predicted position of each trajectory in the current frame, constructs the cost matrix, and then uses the Hungarian matching algorithm to solve for the optimal matching scheme. Successfully matched target clusters are associated with their corresponding trajectories, and their trajectory states are updated. Unmatched target clusters are initialized as new trajectories, and unmatched trajectories across multiple consecutive frames are marked as terminated.

[0083] After the target trajectory is generated, this embodiment calculates its cluster-level statistical features and trajectory-level dynamic features. The cluster-level statistical features include the average distance, average velocity, angular distribution range, and average signal strength of the cluster centroid. The angular distribution range is characterized by the range of azimuth and elevation angles of each point within the cluster. The trajectory-level dynamic features include the distance variation curve and velocity variation curve of the trajectory within a time-series aggregation window, with each curve represented by equally spaced sampling points. This embodiment assembles the cluster-level statistical features and trajectory-level dynamic features in a preset order to obtain a radar gesture feature vector.

[0084] In this embodiment, multiple consecutive frames of hand region image blocks are read from the multimodal data buffer to form a hand region image block sequence. The number of frames in the hand region image block sequence corresponds to the temporal aggregation window length of the radar point cloud frame sequence, ensuring that the feature extraction of the two modalities covers the same time interval.

[0085] Based on the sequence of hand region image blocks, this embodiment sequentially inputs each frame's image blocks into a visual feature encoding network based on a depthwise separable convolutional architecture. The visual feature encoding network uses depthwise separable convolution instead of standard convolution, decomposing spatial convolution into two stages: channel-wise depthwise convolution and pointwise pointwise convolution. This reduces the number of parameters and computational cost while maintaining feature extraction capabilities. The visual feature encoding network outputs a corresponding spatial feature vector for each frame's hand region image block. This spatial feature vector encodes static visual information such as hand shape, finger posture, and palm orientation.

[0086] After extracting the spatial feature vectors for each frame, this embodiment arranges them temporally to form a spatial feature vector sequence. The spatial feature vector sequence is input into a temporal convolutional layer, which performs sliding window convolution on the spatial feature vectors of adjacent frames along the time dimension to extract dynamic change patterns between frames. The output of the temporal convolutional layer is compressed using global pooling to obtain a visual gesture feature vector. This visual gesture feature vector and the radar gesture feature vector will undergo cross-attention fusion in subsequent step S501.

[0087] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0088] Step S501: Map the radar gesture feature vector and the visual gesture feature vector to the common feature space through a linear projection layer to obtain the radar projection feature vector and the visual projection feature vector, respectively. Perform bidirectional cross-attention weighting with the radar projection feature vector and the visual projection feature vector as mutual queries to obtain attention-enhanced feature vector pairs. Input the attention-enhanced feature vector pairs into a fully connected layer after residual connection and concatenation to obtain multimodal fusion gesture feature vectors.

[0089] Step S502: Input the multimodal fused gesture feature vector sequence into a spatiotemporal convolutional network based on a causal dilated convolutional architecture to obtain the gesture category probability distribution and the action completion estimate. Compare the highest probability value in the gesture category probability distribution with a preset recognition confidence threshold to perform recognition confidence filtering. Compare the action completion estimate with a preset completion threshold to perform action integrity filtering. Perform temporal consistency filtering on the gesture category labels of multiple consecutive inference cycles to obtain effective recognition results.

[0090] In this embodiment, linear projection is performed on the radar gesture feature vector generated in step S401 and the visual gesture feature vector generated in step S402, respectively. The linear projection layer maps the feature vectors of the two modalities from their original dimensions to a common feature space of the same dimension through a learnable weight matrix, eliminating the dimensional differences between the modalities. The radar gesture feature vector is linearly projected to obtain the radar projection feature vector, and the visual gesture feature vector is linearly projected to obtain the visual projection feature vector.

[0091] After the radar projection feature vector and the visual projection feature vector are generated, this embodiment performs bidirectional cross-attention weighting on them. In the first direction, using the radar projection feature vector as the query vector and the visual projection feature vector as the key and value vectors, the attention weights of the radar features on each dimension of the visual features are calculated and weighted aggregation is performed to obtain the visually enhanced radar feature vector. In the second direction, using the visual projection feature vector as the query vector and the radar projection feature vector as the key and value vectors, the attention weights of the visual features on each dimension of the radar features are calculated and weighted aggregation is performed to obtain the radar-enhanced visual feature vector. The outputs of the two directions constitute an attention-enhanced feature vector pair.

[0092] Based on the attention-enhanced feature vector pair, this embodiment performs residual connection and concatenation operations. The residual connection adds the visually enhanced radar feature vector to the radar projection feature vector element-wise, and adds the radar-enhanced visual feature vector to the visual projection feature vector element-wise, preserving the original projection feature information while introducing cross-modal enhancement information. The two feature vectors after residual connection are concatenated along the feature dimension, and the concatenation result is input into a fully connected layer for dimensionality compression and feature recombination to obtain a multimodal fusion gesture feature vector.

[0093] In this embodiment, the multimodal fused gesture feature vectors generated at multiple consecutive time points are organized temporally into a multimodal fused gesture feature vector sequence, which is then input into a spatiotemporal convolutional network based on a causal dilated convolutional architecture. The spatiotemporal convolutional network consists of multiple stacked spatiotemporal convolutional blocks, each containing a cascaded structure of spatial feature transformation layers and temporal convolutional layers. The temporal convolutional layers employ causal convolution constraints, with the convolutional kernels only covering the input at the current and historical time points, ensuring that the inference process does not depend on future information. The dilation factor of each temporal convolutional layer is configured exponentially, allowing shallow layers to capture local temporal patterns while deeper layers capture long-range temporal dependencies.

[0094] After the encoding output of the spatiotemporal convolutional network is generated, this embodiment inputs it into a classification head and an action completion evaluation head, respectively. The classification head consists of a fully connected network and a normalized exponential function, outputting a gesture category probability distribution. Each element in the gesture category probability distribution corresponds to the probability of belonging to each category in a preset gesture category set. The action completion evaluation head consists of an independent fully connected network, outputting an action completion estimate. The action completion estimate represents the execution progress of the gesture action within the current observation window relative to a complete standard action.

[0095] Based on the gesture category probability distribution and the estimated action completion rate, this embodiment performs multi-level confidence filtering. The first level of filtering extracts the highest probability value from the gesture category probability distribution and compares it with a preset recognition confidence threshold. Recognition results with a highest probability value lower than the threshold are marked as low confidence and subsequent processing is terminated. The second level of filtering compares the estimated action completion rate with a preset completion threshold. Recognition results with an estimated action completion rate lower than the threshold are marked as incomplete actions and await updates in subsequent inference cycles. The third level of filtering performs a temporal consistency check on the gesture category labels for multiple consecutive inference cycles, requiring that the category labels remain consistent for a preset number of consecutive cycles and that the highest probability value in each cycle exceeds the recognition confidence threshold. Recognition results that pass all three levels of filtering are marked as valid recognition results. These valid recognition results will be input into the intent confirmation module in subsequent step S601 to perform user intent verification.

[0096] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0097] Step S601: Read the current intent confirmation mode identifier according to the system configuration. The intent confirmation mode identifier corresponds to one of the instant confirmation mode, hover confirmation mode, and secondary gesture confirmation mode. Retrieve the corresponding confirmation processing rules and confirmation parameter configuration based on the intent confirmation mode identifier.

[0098] Step S602: In the instant confirmation mode, the valid recognition result is directly marked as intent confirmation passed. In the hover confirmation mode, the radar speed measurement value is used to determine whether the user's hand remains in a relatively stationary state below the hover speed threshold within the preset hover confirmation time. In the secondary gesture confirmation mode, the user is detected to perform the preset confirmation gesture within the preset confirmation waiting time. Valid recognition results that meet the corresponding confirmation conditions are marked as intent confirmation passed and a confirmation timestamp is added to obtain the intent confirmation result.

[0099] This embodiment reads the current intent confirmation mode identifier based on the system configuration. The intent confirmation mode identifier is stored in the gesture control configuration file of the vehicle terminal and is determined by user preference settings or the vehicle model's default configuration.

[0100] The intent confirmation mode identifier corresponds to one of three types: instant confirmation mode, hover confirmation mode, and secondary gesture confirmation mode. These three modes offer differentiated configuration options in terms of response speed and false trigger protection strength.

[0101] Based on the intent confirmation mode identifier, this embodiment retrieves the corresponding confirmation processing rules and confirmation parameter configurations. The parameter configuration for the instant confirmation mode is an empty set. The parameter configuration for the hover confirmation mode includes a hover speed threshold and a hover confirmation duration. The parameter configuration for the secondary gesture confirmation mode includes a confirmation gesture category definition and a confirmation waiting time.

[0102] In the instant confirmation mode, this embodiment directly marks the valid recognition result output in the aforementioned step S502 as intent confirmation passed. This mode skips the additional verification step and is suitable for use scenarios where users have high requirements for response speed.

[0103] In hover confirmation mode, this embodiment continuously reads the speed measurement value from the millimeter-wave radar to determine whether the user's hand remains relatively stationary. The determination logic is to detect whether the speed measurement value remains below a hover speed threshold for a preset hover confirmation duration.

[0104] When the hovering state is determined to be valid, the valid recognition result is marked as intent confirmation passed. If the hand moves out of the detection area or the speed measurement value exceeds the hovering speed threshold during the hovering period, the recognition is canceled and the confirmation process is reset.

[0105] In the secondary gesture confirmation mode, this embodiment detects whether the user has performed a preset confirmation gesture within a preset confirmation waiting time. The detection of the confirmation gesture reuses the recognition path of the aforementioned steps S401 to S502, and matches the recognition output with the preset confirmation gesture category.

[0106] When a gesture matching the preset confirmation gesture is detected within the confirmation waiting time, the valid recognition result is marked as intent confirmation passed. If no confirmation gesture is detected after the confirmation waiting time, the recognition is cancelled.

[0107] In this embodiment, a confirmation timestamp is appended to the valid recognition results that meet the corresponding confirmation conditions to form an intent confirmation result. The intent confirmation result includes three items: a confirmation status flag, the type of confirmation gesture, and a confirmation timestamp. These will be input into the power tailgate control state machine in subsequent step S701 to trigger a state transition.

[0108] In one embodiment of the gesture recognition and control method for vehicle power tailgates in this application, the method may further include the following:

[0109] Step S701: Convert the gesture category in the intent confirmation result into a state transition event input to the power tailgate control state machine. Based on the current state of the power tailgate control state machine and the state transition event, query the state transition table to determine the target state. Based on the target state, read the vehicle gear state and vehicle speed state to perform a safety check. Generate a power tailgate control command from the state transition request that has passed the safety check.

[0110] Step S702: The power tailgate control command is sent to the power tailgate actuator via the drive interface to perform opening, closing, and pausing actions. During the execution process, the clamping force and obstacle detection signal are continuously monitored. If an abnormal monitoring signal is detected, a safety interlock is triggered and an emergency stop command is generated and sent to the power tailgate actuator.

[0111] In this embodiment, the gesture categories in the intent confirmation result generated in step S602 are converted into state transition events. A waving gesture to open a door is converted into a door-opening gesture confirmation event, a waving gesture to close a door is converted into a door-closing gesture confirmation event, a pause gesture is converted into a pause gesture confirmation event, and a cancel gesture is converted into a cancel gesture confirmation event.

[0112] The state transition event is input into the electric tailgate control state machine. The electric tailgate control state machine includes a set of state nodes consisting of idle state, door opening waiting state, door opening execution state, door opening completed state, door closing waiting state, door closing execution state, door closing completed state, paused state, and fault state.

[0113] This embodiment queries the state transition table based on the current state of the electric tailgate control state machine and the state transition events. The state transition table uses the combination of the current state and the input event as an index to record the corresponding target state and output action.

[0114] For example, if the current state is idle and the input event is a door opening gesture confirmation event, the target state returned by the state transition table is the door opening waiting state. If the current state is the door opening execution state and the input event is a pause gesture confirmation event, the target state returned by the state transition table is the paused state.

[0115] Once the target state is determined, this embodiment performs a safety check based on the vehicle's gear position and speed. The safety check items include whether the vehicle is in park and whether the speed is below a safety threshold.

[0116] When all safety checks are passed, this embodiment will generate a power tailgate control command from the state transition request. The power tailgate control command includes a command type and a target position parameter. The command type is distinguished as an open command, a close command, and a pause command.

[0117] When a safety check fails, the power tailgate control state machine maintains its current state. In this embodiment, the audible and visual alert module is simultaneously triggered to provide the user with feedback on the reason for the safety restriction.

[0118] In this embodiment, the power tailgate control command is sent to the power tailgate actuator via the drive interface. The power tailgate actuator drives the motor to perform tailgate opening, closing, and pausing actions according to the command type.

[0119] During execution, this embodiment continuously monitors the clamping force signal and the obstacle detection signal. The clamping force signal is obtained from the current feedback of the electric tailgate drive motor; a sudden increase in current indicates that the tailgate movement is obstructed.

[0120] The obstacle detection signal is generated by the fusion of millimeter-wave radar and near-infrared camera. The obstacle detection signal is triggered when an obstacle is detected in the tailgate's movement path.

[0121] When the clamping force signal exceeds a preset threshold or an obstacle detection signal is triggered, this embodiment activates the safety interlock mechanism. The safety interlock mechanism immediately generates an emergency stop command and sends it to the electric tailgate actuator, terminating the current movement and transferring the electric tailgate control state machine to a paused state.

[0122] After execution, status feedback and safety event records are written to the control log storage area for the system adaptive module to read and use for subsequent parameter adjustments.

[0123] To effectively address the shortcomings of traditional technologies in areas such as perception alignment, feature fusion, and control execution, and to provide technical support for intelligent control of vehicle power tailgates, this application provides an embodiment of a gesture recognition and control device for vehicle power tailgates, which implements all or part of the aforementioned gesture recognition and control method. See [link to embodiment]. Figure 2The gesture recognition and control device for the vehicle's power tailgate specifically includes the following components:

[0124] The data perception module 10 is used to collect echo signals based on millimeter-wave radar sensors deployed in the rear area of ​​the vehicle, generate a range Doppler map through range dimension transformation and velocity dimension transformation, obtain a radar point cloud frame by constant false alarm rate detection and angle estimation of the range Doppler map, collect image frames based on a near-infrared camera and obtain a hand region image block by hand region detection, and perform alignment and pairing of the radar point cloud frame and the hand region image block according to the timestamp to obtain a multimodal perception data frame.

[0125] The gesture recognition module 20 is used to obtain radar gesture feature vectors by associating the radar point cloud frame sequence in the multimodal perception data frame with target tracking through spatial clustering; input the hand region image block sequence into a visual feature encoding network to obtain a visual gesture feature vector; fuse the radar gesture feature vector and the visual gesture feature vector through cross-attention to obtain a multimodal fused gesture feature vector; input the multimodal fused gesture feature vector sequence into a spatiotemporal convolutional network to obtain a gesture category probability distribution and an action completion estimate; and obtain an effective recognition result by filtering the gesture category probability distribution and the action completion estimate through multi-level confidence.

[0126] The tailgate control module 30 is used to perform user intent confirmation on the valid recognition result according to the preset intent confirmation mode to obtain the intent confirmation result, input the intent confirmation result into the electric tailgate control state machine and perform safety checks and state transition determination in combination with the vehicle status to obtain the electric tailgate control command, and send the electric tailgate control command to the electric tailgate actuator.

[0127] As described above, the gesture recognition and control device for vehicle power tailgates provided in this application embodiment can achieve accurate gesture recognition through radar and vision fusion. A control mechanism is constructed, combining feature fusion and spatiotemporal analysis to establish a reliable action recognition strategy. Control optimization is introduced, ensuring continuous improvement of operation through intent confirmation and safety checks. This method effectively solves the shortcomings of traditional technologies in perception alignment, feature fusion, and control execution, providing technical support for intelligent control of vehicle power tailgates.

[0128] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the gesture recognition and control method for a vehicle-mounted electric tailgate.

[0129] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described gesture recognition and control method for vehicle-mounted electric tailgates.

[0130] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described gesture recognition and control method for vehicle-mounted electric tailgates.

[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A gesture recognition and control method for vehicle-mounted electric tailgates, characterized in that, The method includes: Based on the echo signal collected by the millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, a range Doppler map is generated by range dimension transformation and velocity dimension transformation. The range Doppler map is then subjected to constant false alarm rate detection and angle estimation to obtain radar point cloud frames. Based on the image frame collected by the near-infrared camera and the hand region detection, a hand region image block is obtained. The radar point cloud frame and the hand region image block are aligned and paired according to the timestamp to obtain a multimodal perception data frame. The radar point cloud frame sequence in the multimodal perception data frame is associated with target tracking through spatial clustering to obtain radar gesture feature vectors. The hand region image block sequence is input into a visual feature encoding network to obtain visual gesture feature vectors. Based on the radar gesture feature vectors and the visual gesture feature vectors, a multimodal fused gesture feature vector is obtained through cross-attention fusion. The multimodal fused gesture feature vector sequence is input into a spatiotemporal convolutional network to obtain gesture category probability distribution and action completion estimation. Based on the gesture category probability distribution and the action completion estimation, a multi-level confidence screening is performed to obtain effective recognition results. Based on the preset intent confirmation mode, the valid recognition result is used to perform user intent confirmation to obtain intent confirmation result. The intent confirmation result is input into the power tailgate control state machine and combined with the vehicle status to perform safety checks and state transition determination to obtain power tailgate control command. The power tailgate control command is then sent to the power tailgate actuator.

2. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The method involves collecting echo signals from a millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, generating a range Doppler image through range-dimensional and velocity-dimensional transformations, and then obtaining a radar point cloud frame by constant false alarm rate detection and angle estimation of the range Doppler image, including: Based on the millimeter-wave radar sensor deployed in the rear area of ​​the vehicle, the intermediate frequency signal data is read at a preset sampling period. The intermediate frequency signal data is then subjected to a range dimension fast Fourier transform to obtain range spectrum data. The range spectrum data is then subjected to a velocity dimension fast Fourier transform to generate a range Doppler map. The range Doppler map represents the two-dimensional energy distribution of the target with distance and velocity as coordinate axes. Based on the range Doppler image, constant false alarm rate (CFAR) detection is performed according to a configurable detection threshold to obtain a set of target detection points exceeding the threshold. Angle estimation is performed on each detection point in the target detection point set based on the phase difference of multiple receiving antennas to obtain the azimuth and elevation angle estimates of each detection point. The range, velocity, signal strength, azimuth, and elevation angle estimates of each detection point are assembled and a frame timestamp is added to obtain a radar point cloud frame.

3. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The process involves acquiring image frames using a near-infrared camera and detecting the hand region to obtain hand region image blocks. Then, based on timestamps, the radar point cloud frames and the hand region image blocks are aligned and paired to obtain multimodal perception data frames, including: Based on the near-infrared camera deployed in the rear area of ​​the vehicle, image frames are acquired at a preset frame rate. The image frames are then processed by distortion correction and grayscale normalization to obtain preprocessed image frames. The preprocessed image frames are then input into a lightweight hand detection network to obtain hand bounding box coordinates and detection confidence. Based on the hand bounding box coordinates, the preprocessed image frames are then cropped and normalized to obtain hand region image blocks. Based on the frame timestamp of the radar point cloud frame and the frame timestamp of the hand region image block, time alignment matching is performed. Radar point cloud frames and hand region image blocks with timestamp differences within a preset pairing tolerance threshold are marked as valid pairs. The validly paired radar point cloud frames and hand region image blocks are assembled to obtain a multimodal sensing data frame and written into the multimodal data buffer.

4. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The step of obtaining radar gesture feature vectors by spatially clustering and associating radar point cloud frame sequences from the multimodal sensing data frames with target tracking, and inputting hand region image block sequences into a visual feature encoding network to obtain visual gesture feature vectors, includes: A radar point cloud frame sequence is formed by reading multiple consecutive radar point cloud frames from a multimodal data buffer. The radar point cloud frame sequence is then subjected to density-based spatial clustering within a preset temporal aggregation window to obtain a target cluster set. The target cluster set is then subjected to cross-frame target tracking association using Kalman filtering and Hungarian matching algorithm to obtain the target trajectory. Based on the target trajectory, the mean distance, mean velocity, angle distribution range, and mean signal strength of the cluster centroid are calculated, and the distance change curve and velocity change curve of the trajectory are extracted and assembled to obtain a radar gesture feature vector. A sequence of hand region image blocks is formed by reading multiple consecutive frames of hand region image blocks from a multimodal data buffer. Each frame in the hand region image block sequence is input into a visual feature encoding network based on a depthwise separable convolutional architecture to extract spatial feature vectors. The spatial feature vector sequence is then passed through a temporal convolutional layer to extract temporal features and obtain a visual gesture feature vector.

5. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The process involves fusing the radar gesture feature vector and the visual gesture feature vector through cross-attention to obtain a multimodal fused gesture feature vector. This multimodal fused gesture feature vector sequence is then input into a spatiotemporal convolutional network to obtain a gesture category probability distribution and an estimated action completion rate. Based on the gesture category probability distribution and the estimated action completion rate, a multi-level confidence screening process is performed to obtain an effective recognition result, including: The radar gesture feature vector and the visual gesture feature vector are respectively mapped to a common feature space through a linear projection layer to obtain radar projection feature vector and visual projection feature vector. The radar projection feature vector and the visual projection feature vector are used as queries to perform bidirectional cross-attention weighting to obtain attention-enhanced feature vector pairs. The attention-enhanced feature vector pairs are fed into a fully connected layer after residual connection and concatenation to obtain multimodal fusion gesture feature vectors. The multimodal fused gesture feature vector sequence is input into a spatiotemporal convolutional network based on a causal dilated convolutional architecture to obtain a gesture category probability distribution and an estimated action completion degree. The highest probability value in the gesture category probability distribution is compared with a preset recognition confidence threshold to perform recognition confidence filtering. The estimated action completion degree is compared with a preset completion threshold to perform action integrity filtering. The gesture category labels of multiple consecutive inference cycles are subjected to temporal consistency filtering to obtain an effective recognition result.

6. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The step of performing user intent confirmation on the valid recognition result according to the preset intent confirmation mode to obtain the intent confirmation result includes: The system reads the current intent confirmation mode identifier based on the system configuration. The intent confirmation mode identifier corresponds to one of the instant confirmation mode, hover confirmation mode, and secondary gesture confirmation mode. Based on the intent confirmation mode identifier, the corresponding confirmation processing rules and confirmation parameter configuration are retrieved. In the instant confirmation mode, the valid recognition result is directly marked as intent confirmation passed. In the hover confirmation mode, the radar speed measurement value is used to determine whether the user's hand remains in a relatively stationary state below the hover speed threshold within the preset hover confirmation time. In the secondary gesture confirmation mode, the user is detected to perform the preset confirmation gesture within the preset confirmation waiting time. Valid recognition results that meet the corresponding confirmation conditions are marked as intent confirmation passed and a confirmation timestamp is added to obtain the intent confirmation result.

7. The gesture recognition and control method for vehicle-mounted electric tailgates according to claim 1, characterized in that, The process of inputting the intent confirmation result into the power tailgate control state machine and performing safety checks and state transition determinations in conjunction with the vehicle status to obtain the power tailgate control command, and then issuing the power tailgate control command to the power tailgate actuator, includes: The gesture category in the intent confirmation result is converted into a state transition event and input into the power tailgate control state machine. Based on the current state of the power tailgate control state machine and the state transition event, the state transition table is queried to determine the target state. Based on the target state, the vehicle gear state and vehicle speed state are read to perform a safety check. The state transition request that passes the safety check is generated into a power tailgate control command. The power tailgate control command is sent to the power tailgate actuator via the drive interface to perform opening, closing, and pausing actions. During the execution process, the clamping force and obstacle detection signal are continuously monitored. If an abnormal monitoring signal is detected, a safety interlock is triggered and an emergency stop command is generated and sent to the power tailgate actuator.

8. A gesture recognition and control device for vehicle power tailgates, characterized in that, The device includes: The data perception module is used to collect echo signals based on millimeter-wave radar sensors deployed in the rear area of ​​the vehicle, generate a range Doppler map through range dimension transformation and velocity dimension transformation, obtain radar point cloud frames by constant false alarm rate detection and angle estimation of the range Doppler map, acquire image frames based on near-infrared cameras and obtain hand region image blocks by hand region detection, and perform alignment and pairing of radar point cloud frames and hand region image blocks according to timestamps to obtain multimodal perception data frames; The gesture recognition module is used to obtain radar gesture feature vectors by spatially clustering and associating the radar point cloud frame sequence in the multimodal perception data frame with target tracking; input the hand region image block sequence into a visual feature encoding network to obtain a visual gesture feature vector; fuse the radar gesture feature vector and the visual gesture feature vector through cross-attention to obtain a multimodal fused gesture feature vector; input the multimodal fused gesture feature vector sequence into a spatiotemporal convolutional network to obtain a gesture category probability distribution and an action completion estimate; and obtain an effective recognition result by multi-level confidence filtering based on the gesture category probability distribution and the action completion estimate. The tailgate control module is used to perform user intent confirmation on the valid recognition result according to the preset intent confirmation mode to obtain the intent confirmation result, input the intent confirmation result into the electric tailgate control state machine and perform safety checks and state transition determination in combination with the vehicle status to obtain the electric tailgate control command, and send the electric tailgate control command to the electric tailgate actuator.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the gesture recognition and control method for vehicle-mounted electric tailgate as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the gesture recognition and control method for vehicle-mounted electric tailgates as described in any one of claims 1 to 7.

Images