Action detection model training method, driving action detection method and equipment

Abstract

A motion detection model training method, a driving motion detection method and a device, comprising: inputting own vehicle motion information into a TRN for feature extraction to obtain output global features; inputting the multiple frames of vehicle acquisition images into an encoding module for encoding to obtain vehicle visual features; performing feature conversion on the output global features and the vehicle visual features to obtain enhanced fusion features; inputting the enhanced fusion features into a classification head for classification prediction, obtaining a driving action prediction classification result, and constructing a driving action classification loss function; inputting the enhanced fusion feature into an image prediction network for image prediction to obtain a future image prediction result and construct a future image prediction loss function; and constructing a joint loss function according to the driving action classification loss function and the future image prediction loss function, and training the action detection model based on the joint loss function to obtain a trained action detection model. According to the method, the robustness and the stability of the action detection model in a severe scene are improved.

Description

Technical Field

[0001] This invention relates to the field of motion detection technology, specifically to a motion detection model training method, a driving motion detection method, and a device. Background Technology

[0002] The development of autonomous driving applications requires a large and diverse traffic scenario database. This database is used to extensively validate autonomous driving systems, ensuring their safety and robustness in complex and ever-changing traffic environments. In real-world autonomous driving scenarios, data collection primarily relies on vehicles on the road. While the collected data is massive, the majority of these scenarios are simple and require annotation before testing. However, manual annotation is time-consuming and costly. Using action detection algorithms such as TRN, OadTR, Colar, GateHUB, and E2E-LOAD for traffic scenario classification can significantly reduce manual annotation and improve data annotation efficiency, which is crucial for increasing the size and enriching the content of traffic scenario datasets. However, action detection algorithms for traffic scenario classification cannot effectively handle the large amount of redundant information within traffic scenarios, resulting in poor performance in action detection tasks within traffic scenarios. Summary of the Invention

[0003] The present invention aims to improve the robustness and stability of motion detection models in harsh environments.

[0004] According to a first aspect, one embodiment provides a method for training an action detection model, applied in an autonomous driving scenario to detect driving actions. The action detection model includes an encoding module, a TRN, a cross-attention mechanism, a classification head, and an image prediction network. The method for training the action detection model includes: Acquire multiple frames of vehicle images and vehicle motion information; wherein, the multiple frames of vehicle images include images reflecting the surrounding environment of the currently controlled vehicle within the current time period; The vehicle motion information is input into the TRN for feature extraction, resulting in the output global features. The multi-frame vehicle images are input into the encoding module for encoding processing to obtain vehicle visual features; Based on the cross-attention mechanism, feature transformation is performed on the output global features and the vehicle visual features to obtain enhanced fusion features; The enhanced fusion features are input into the classification head for classification prediction to obtain the driving action prediction classification result, and a driving action classification loss function is constructed based on the driving action prediction classification result and the preset real driving action classification result. The enhanced fusion features are input into the image prediction network to perform image prediction, thereby obtaining future image prediction results. A future image prediction loss function is then constructed based on the future image prediction results and the preset future image real results. The future image prediction results include images of the surrounding environment of the currently controlled vehicle in the future time period. A joint loss function is constructed based on the driving action classification loss function and the future image prediction loss function, and the action detection model is trained based on the joint loss function to obtain a trained action detection model.

[0005] In some embodiments, the step of performing feature transformation on the output global features and the vehicle visual features based on the cross-attention mechanism to obtain enhanced fusion features includes: The output global feature is used as the query vector in the cross-attention mechanism, and the vehicle visual feature is used as the key vector and value vector in the cross-attention mechanism. The output global feature is enhanced based on the query vector, the key vector and the value vector to obtain the enhanced output global feature. The enhanced output global features are stitched together with the vehicle visual features along a preset channel dimension to obtain enhanced fusion features.

[0006] In some embodiments, the step of enhancing the output global features based on the query vector, the key vector, and the value vector to obtain enhanced output global features includes: Calculate the similarity between the query vector and the key vector, and normalize the similarity to obtain the first attention weight; The value vector is weighted using the first attention weight, and the weighted value vector is fused with the query vector to obtain the enhanced output global feature.

[0007] In some embodiments, the encoding module includes a first encoding submodule, a second spatial attention module, and a third spatiotemporal attention module; the step of inputting the multi-frame vehicle acquisition images into the encoding module for encoding processing to obtain vehicle visual features includes: The multi-frame vehicle acquisition images are input into the first encoding submodule for block embedding processing to obtain the first feature representation; The query vector, key vector, and value vector in the second spatial attention module are determined based on the first feature representation. Spatial feature enhancement processing is then performed based on the query vector, key vector, and value vector in the second spatial attention module to obtain the second feature representation. The query vector, key vector, and value vector in the third spatial attention module are determined based on the second feature representation. Spatiotemporal feature enhancement processing is then performed based on the query vector, key vector, and value vector in the third spatial attention module to obtain vehicle visual features.

[0008] In some embodiments, the step of inputting the multi-frame vehicle acquisition images into the first encoding submodule for block embedding processing to obtain a first feature representation includes: For each frame of vehicle image acquisition, the vehicle image is slid segmented with a preset step size to obtain multiple image blocks; Spatial features of each image patch are extracted using a convolutional neural network, and each image patch is flattened into a token sequence; wherein, the token sequence is used to characterize the spatial features of the image patch; A cls token is inserted at the beginning of each frame of the token sequence, and a preset position code is added to the multi-frame token sequence after the insertion of the cls token to obtain the first feature representation.

[0009] In some embodiments, the spatial feature enhancement processing based on the query vector, key vector, and value vector in the second spatial attention module to obtain the second feature representation includes: The second attention weight is calculated based on the query vector and key vector in the second spatial attention module; The value vector in the second spatial attention module is weighted using the second attention weight to obtain the second feature representation.

[0010] In some embodiments, the temporal decoder in the TRN includes a single-step future prediction branch; the step of inputting the vehicle motion information into the TRN for feature extraction to obtain output global features includes: Feature extraction is performed on the vehicle motion information to obtain a feature sequence; The feature sequence is input into the self-attention mechanism of the TRN for temporal fusion to obtain temporal fusion features; The temporal fusion features are input into the single-step future prediction branch of the temporal decoder for feature prediction, resulting in the output global features.

[0011] According to a second aspect, one embodiment provides a driving action detection method, including: Acquire vehicle images and vehicle motion information of the currently moving vehicle; The vehicle-captured images and vehicle motion information are input into a trained motion detection model to obtain the driving motion detection results of the currently driving vehicle; wherein, the motion detection model is trained using the motion detection model training method described above.

[0012] According to a third aspect, one embodiment provides an action detection model training device, comprising: Memory, used to store programs; A processor for implementing the action detection model training method by executing a program stored in the memory.

[0013] According to the fourth aspect, one embodiment provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the action detection model training method.

[0014] The action detection model training method, driving action detection method, action detection model training device, and computer program product based on the above embodiments, by introducing vehicle motion information, provide absolute motion prior information, preventing the inability to distinguish between relative motion and relative stillness by relying solely on multiple frames of vehicle images. Furthermore, it enables the detection of vehicle motion even under adverse visual conditions such as nighttime or heavy rain, improving the robustness of action detection. Simultaneously, during the training phase, an image prediction network is used to predict future image prediction results. This image prediction network shares an encoding module with the classification head. Through a joint loss function, it forces the encoding module to simultaneously capture both visual and motion information of the vehicle, improving the generalization ability of spatial-temporal features and enhancing the instructions for spatial and temporal feature extraction, thereby improving action detection accuracy. Attached Figure Description

[0015] Figure 1 This is a flowchart of the action detection model training method in the embodiments of this application; Figure 2 This is a flowchart illustrating how to perform feature transformation on output global features and vehicle visual features based on a cross-attention mechanism in one embodiment to obtain enhanced fusion features. Figure 3 This is a flowchart illustrating how to enhance the output global features based on the query vector, key vector, and value vector in one embodiment, resulting in the enhanced output global features. Figure 4 This is a flowchart illustrating how multiple frames of vehicle images are input into an encoding module for encoding processing to obtain vehicle visual features in one embodiment. Figure 5 This is a flowchart of one embodiment of inputting multiple frames of vehicle-captured images into a first encoding submodule for block embedding processing to obtain a first feature representation; Figure 6 This is a model architecture diagram of block embedding processing in one embodiment; Figure 7This is a flowchart illustrating how spatial feature enhancement processing is performed on the query vector, key vector, and value vector in the second spatial attention module in one embodiment to obtain a second feature representation. Figure 8 This is a flowchart illustrating how vehicle motion information is input into a TRN for feature extraction in one embodiment, resulting in the output of global features. Figure 9 This is a model architecture diagram of an action detection model training method in one embodiment; Figure 10 This is a flowchart of a driving action detection method in one embodiment. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0017] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0018] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0019] Action detection algorithms cannot handle a large amount of redundant information in traffic scenes well. They have problems such as broken temporal information, difficulty in focusing on key features, and limitations of single-modal input, resulting in poor performance in traffic scene action detection tasks. The general workflow of action detection algorithms consists of three steps: first, encoding the input data to extract spatial features, then extracting temporal features, and finally classifying. In order to achieve more effective traffic scene classification, action detection algorithms also need to overcome the following challenges: (1) Extracting key spatial features: The visual input of traffic scenes contains rich semantic information, such as road information, information of other traffic participants, weather information, etc. However, the appearance variance of the same type of scene is large under different lighting, weather, and seasons. Moreover, key targets such as drivable areas and dynamic participants that are directly related to driving tasks often only occupy a small part of the pixels in the image, while redundant areas such as the sky and distant buildings account for a high proportion and are easy to dominate the network judgment. It is a difficult problem for the model to effectively learn key features. (2) Extracting effective temporal features: Traffic scene behavior usually requires 3-5 seconds or longer of context to judge, but the original long temporal data has a large amount of redundancy that drowns out key actions. Processing long temporal data is a major problem. Starting with OadTR, action detection algorithms have introduced transformers to handle long temporal information. Thanks to the global receptive field of the self-attention mechanism, transformers offer significant improvements compared to previous CNN and RNN algorithms. However, the raw long temporal data suffers from significant redundancy that obscures key actions. Global self-attention treats every frame equally, leading to the model remembering more and more clutter as it goes further. Dynamic traffic scenes contain a large amount of non-action and non-causal noise, making it difficult for models to distinguish between genuine dynamic driving cues and irrelevant temporal factors. Furthermore, the confusion between relative stillness and real motion in traffic scenes—for example, the relative stillness and motion of surrounding dynamic participants and the vehicle itself—can affect the autonomous driving system's misjudgment of motion information and remains to be addressed.

[0020] To address the aforementioned issues, this application proposes a method for training an action detection model, applied in autonomous driving scenarios to detect driving actions. The action detection model includes an encoding module, a TRN, a cross-attention mechanism, a classifier head, and an image prediction network. The action detection model training method includes: acquiring multiple frames of vehicle-captured images and vehicle motion information; wherein the multiple frames of vehicle-captured images include images reflecting the surrounding environment of the currently controlled vehicle within the current time period; inputting the vehicle motion information into the TRN for feature extraction to obtain output global features; inputting the multiple frames of vehicle-captured images into the encoding module for encoding processing to obtain vehicle visual features; and performing feature transformation on the output global features and vehicle visual features based on the cross-attention mechanism to obtain enhanced fusion. Features: Enhanced fusion features are input into a classification head for classification prediction to obtain driving action prediction classification results. A driving action classification loss function is constructed based on the driving action prediction classification results and preset real driving action classification results. Enhanced fusion features are input into an image prediction network for image prediction to obtain future image prediction results. A future image prediction loss function is constructed based on the future image prediction results and preset real future image results. The future image prediction results include images of the surrounding environment of the currently controlled vehicle in the future time period. A joint loss function is constructed based on the driving action classification loss function and the future image prediction loss function. The action detection model is trained based on the joint loss function to obtain a trained action detection model.

[0021] The action detection model training method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0022] This application provides a method for training an action detection model, applied in autonomous driving scenarios to detect driving actions. The action detection model includes an encoding module, a TRN, a cross-attention mechanism, a classification head, and an image prediction network. Figure 1 A flowchart of an action detection model training method provided in an embodiment of this application is shown, which will be described in detail below.

[0023] Step S10: Acquire multiple frames of vehicle images and vehicle motion information.

[0024] Specifically, the forward-facing camera of the currently controlled vehicle captures multiple frames of vehicle images. These frames include images reflecting the surrounding environment of the currently controlled vehicle within the current time period, which is a pre-set fixed duration, such as 30 seconds. During this period, the forward-facing camera of the currently controlled vehicle continuously captures multiple frames of vehicle images. Vehicle motion information is key data in the autonomous driving system describing the motion state of the currently controlled vehicle. This motion information includes the position, speed, and acceleration of the currently controlled vehicle.

[0025] Step S20: Input the vehicle motion information into the TRN for feature extraction to obtain the output global features.

[0026] Specifically, when processing sensor inputs, i.e., vehicle motion information, through a recursive structure, Temporal Recurrent Networks (TRNs) can capture long-term dependencies in vehicle motion information, significantly improving the accuracy and real-time performance of motion detection in complex dynamic scenes. In this embodiment, a simplified TRN is used. The temporal decoder of the TRN is simplified by removing the original multi-step future prediction branch and retaining only the single-step future prediction branch, and the temporal decoder parameters are shared across each time step.

[0027] By incorporating vehicle motion information, the relative stillness and absolute motion can be decoupled. In purely visual input (i.e., the aforementioned multi-frame vehicle images), when the preceding vehicle and the following vehicle are traveling at the same speed, the relative position of the preceding vehicle remains almost unchanged. Its visual appearance is almost indistinguishable from the stationary state of the two vehicles. The network must rely on subtle contextual cues (such as shadow changes, small parallax, and distant stationary references) for implicit inference, which is not only inefficient but also prone to failure when scene textures are sparse, lighting is poor, or references are missing. It also makes it difficult for the model to learn key features. However, by adding vehicle motion information, the motion information of the currently controlled vehicle can be directly obtained. The model no longer needs to infer the motion state of the currently controlled vehicle from images, and can thus focus on more discriminative dynamic and scene key cues, such as traffic markings on the ground and changes in road shape.

[0028] In extreme and harsh environments such as darkness, heavy rain, blizzards, dense fog, or dust storms, the quality of visual input degrades drastically. Dynamic range compression leads to extremely low contrast, rain streaks and snow particles cause numerous artifacts, and road markings and surrounding vehicles are severely obscured or even completely invisible. Under these conditions, the confidence of pure visual action detection networks drops significantly, and false positives and false negatives increase exponentially. To address this bottleneck, this application's embodiments introduce vehicle motion information. Even when pure visual input fails or becomes unreliable, it still provides the model with high-frequency, low-latency, absolute motion ground truth values ​​unaffected by lighting and weather conditions, thereby significantly improving the robustness and stability of the action detection model in harsh scenarios.

[0029] Step S30: Input multiple frames of vehicle images into the encoding module for encoding processing to obtain vehicle visual features.

[0030] Specifically, in order to fully explore the spatial semantics and temporal relationships between multiple frames of vehicle-captured images, this application embodiment refers to the E2E-LOAD structure and uses MViT to process multiple frames of vehicle-captured images. The encoding module may include a PatchEmbedding module, a Spatial Attention module, and a Spatial-Temporal Attention module, which are used together to perform encoding processing to obtain vehicle visual features.

[0031] Step S40: Based on the cross-attention mechanism, perform feature transformation on the output global features and vehicle visual features to obtain enhanced fusion features.

[0032] Specifically, a cross-attention mechanism is introduced, using the output global features as the query and the vehicle visual features as the key and value. By calculating spatial-temporal attention weights, the information in the output global features most relevant to the current action is dynamically enhanced. Then, the enhanced output global features and the vehicle visual features are concatenated along the channel dimension to obtain the enhanced fused features. Specifically, the concatenation process involves combining the CLS_Token from the vehicle visual features with the enhanced output global features.

[0033] In this embodiment, while maintaining the advantages of historical-future context modeling, adaptive weighting of the output global features is achieved, which improves the consistency and discriminative power of multimodal features.

[0034] Step S50: Input the enhanced fusion features into the classification head for classification prediction to obtain the driving action prediction classification result, and construct the driving action classification loss function based on the driving action prediction classification result and the preset real driving action classification result.

[0035] Specifically, the enhanced fusion features are input into the classification head to complete the probability estimation of the current action and obtain the driving action prediction classification result. Driving actions refer to the quantifiable control behaviors taken by a vehicle during driving to achieve a goal. Examples of driving actions include acceleration, deceleration, constant speed driving, emergency braking, turning, lane changing, and obstacle avoidance. A driving action classification loss function, Loss, is constructed based on the driving action prediction classification result and the preset real driving action classification result. cls The loss function can be the cross-entropy loss function.

[0036] Step S60: Input the enhanced fusion features into the image prediction network to perform image prediction, obtain the future image prediction result, and construct the future image prediction loss function based on the future image prediction result and the preset future image real result.

[0037] Specifically, by adding a self-supervised branch, the image prediction network, during training, the encoding module is forced to capture both visual and motion context information simultaneously. This improves the generalization ability of spatial-temporal features and enhances the quality of both spatial and temporal feature extraction, indirectly improving action detection accuracy. The image prediction network, instead of directly reconstructing pixels, first reconstructs high-order motion tokens in the enhanced fusion features. This identifies a few key image frames from multiple vehicle acquisition images that truly influence the vehicle's future motion, accurately acquiring vehicle motion information. Only after generating pixel frames using this motion information can future frames be accurately predicted. This approach constrains attention weights to high-information regions and suppresses redundant information.

[0038] The image prediction network shares the same encoding module with the classification head. It uses the image prediction network to generate future image prediction results. The structure of the image prediction network includes a multilayer perceptron (MLP) and deconvolution (Deconv).

[0039] Furthermore, because the differences between static regions in multi-frame vehicle images are relatively small, only abrupt changes in location, such as changes in the relative position of the vehicle, will result in significant losses in image prediction tasks. Therefore, in image prediction tasks, gradients will prioritize updating attention heads that significantly reduce prediction errors; that is, the Spatial Attention module and the Spatial-Temporal Attention module that significantly reduces prediction errors will be updated, while redundant information tokens will be ignored. Additionally, since the model cannot see real future frame information and can only utilize past tokens that truly carry motion semantic cues, short-term artifacts such as light flickering lack predictability, are small in pixel area, and are thus deweighted in image prediction tasks, allowing the model to further focus on motion information.

[0040] Furthermore, the enhanced fusion features are input into the image prediction network for image prediction to obtain future image prediction results. These future image prediction results include images of the surrounding environment of the currently controlled vehicle within a future time period. A future image prediction loss function, Loss, is constructed based on the future image prediction results and the preset true future image results. MSE .

[0041] For example, future image prediction results include images of the surrounding environment of the currently controlled vehicle in the next 2-3 minutes.

[0042] In some embodiments, a mean squared error loss function is used to calculate the pixel-by-pixel difference between the predicted future image and the preset true future image. Therefore, the image prediction loss function... The future image prediction result is the prediction of the future image. The true future image is I, and N represents the number of pixels in the predicted or true future image. This indicates the prediction of the i-th pixel in a future image. Let represent the i-th pixel in the true future image. The image prediction loss function described above measures the average of the squared differences between the predicted and actual future image at the pixel level. It guides the training of the image prediction network and assists in the training of the main network in the entire model, enabling it to reconstruct the future image more accurately.

[0043] Step S70: Construct a joint loss function based on the driving action classification loss function and the future image prediction loss function, and train the action detection model based on the joint loss function to obtain a trained action detection model.

[0044] Specifically, based on the driving action classification loss function Loss cls Loss function for future image prediction MSE Construct a joint loss function Loss, Loss = Loss cls +α Loss MSE Where α is a preset weight parameter. The action detection model is trained based on the joint loss function until the trained action detection model meets the preset convergence requirement. The action detection model that meets the convergence requirement is taken as the trained action detection model.

[0045] The action detection model training method based on the above embodiments, by introducing vehicle motion information, provides absolute motion prior information, preventing the inability to distinguish between relative motion and relative stillness by relying solely on multiple frames of vehicle images. Furthermore, it enables the detection of vehicle motion even under adverse visual conditions such as nighttime or heavy rain, improving the robustness of action detection. Simultaneously, during the training phase, an image prediction network is used to predict future image prediction results. This image prediction network shares an encoding module with the classification head. Through a joint loss function, it forces the encoding module to simultaneously capture both visual and motion information of the vehicle, improving the generalization ability of spatial-temporal features and enhancing the instructions for spatial and temporal feature extraction, thereby improving action detection accuracy.

[0046] Please refer to Figure 2 In some embodiments, step S40: based on the cross-attention mechanism, the output global features and vehicle visual features are transformed to obtain enhanced fusion features, including steps S41 to S42, which are described in detail below.

[0047] Step S41: Use the output global features as the query vector in the cross-attention mechanism, and the vehicle visual features as the key vector and value vector in the cross-attention mechanism. Enhance the output global features based on the query vector, key vector, and value vector to obtain the enhanced output global features.

[0048] Specifically, the output of the cross-attention mechanism can be directly added to or concatenated with the output global features to achieve information enhancement and obtain enhanced output global features. Alternatively, a gating mechanism can be introduced to dynamically control the fusion ratio between the output global features and the output of the cross-attention mechanism to achieve information enhancement and obtain enhanced output global features.

[0049] Step S42: The enhanced output global features are stitched together with the vehicle visual features along the preset channel dimension to obtain the enhanced fusion features.

[0050] Specifically, the enhanced output global features are concatenated with the CLS_Token in the vehicle visual features along the preset channel dimension to obtain the enhanced fused features.

[0051] Please refer to Figure 3 In some embodiments, step S41: Information enhancement is performed on the output global features based on the query vector, key vector and value vector to obtain the enhanced output global features, including steps S411 to S412, which are described in detail below.

[0052] Step S411: Calculate the similarity between the query vector and the key vector, and normalize the similarity to obtain the first attention weight.

[0053] For example, the similarity between query vector Q1 and key vector K1 is calculated and normalized to obtain the first attention weight W1.

[0054] Step S412: Use the first attention weight to weight the value vector, and then fuse the weighted value vector with the query vector to obtain the enhanced output global feature.

[0055] For example, the value vector V1 is weighted using the first attention weight W1, and the weighted value vector is fused with the query vector Q1 to obtain the enhanced output global feature.

[0056] Please refer to Figure 4 In some embodiments, the encoding module includes a first encoding submodule, a second spatial attention module, and a third spatiotemporal attention module; Step S30: Input multiple frames of vehicle acquisition images into the encoding module for encoding processing to obtain vehicle visual features, including steps S31 to S33, which are described in detail below.

[0057] Step S31: Input the multi-frame vehicle acquisition images into the first encoding submodule for block embedding processing to obtain the first feature representation.

[0058] Specifically, the first encoding submodule is the Patch embedding module, which performs block embedding processing on each frame of vehicle acquisition image to obtain the first feature representation.

[0059] Step S32: Determine the query vector, key vector, and value vector in the second spatial attention module based on the first feature representation, and perform spatial feature enhancement processing based on the query vector, key vector, and value vector in the second spatial attention module to obtain the second feature representation.

[0060] Specifically, the second spatial attention module is the Spatial Attention module. In this module, five transformer layers are repeated to extract multi-scale features from the spatial dimension of the first feature representation. The spatial resolution is gradually reduced and the channel dimension is increased through pooling attention to extract multi-scale spatial features. The spatial features of a single frame are aggregated through cls tokens while keeping the temporal dimension unchanged, providing high-quality spatial features for subsequent temporal modeling.

[0061] Step S33: Determine the query vector, key vector, and value vector in the third spatial attention module based on the second feature representation, and perform spatiotemporal feature enhancement processing based on the query vector, key vector, and value vector in the third spatial attention module to obtain vehicle visual features.

[0062] Specifically, the third spatial attention module is the Spatial-Temporal Attention module. This module repeats 11 transformer layers to gradually refine the features. Each transformer layer dynamically adjusts the pooling kernel and stride of the query vector, key vector, and value vector in the third spatial attention module, keeping the temporal dimension unchanged while performing only one 2×2 downsampling on the spatial dimension, and simultaneously expanding the channel dimension. The cls token, along with all patch tokens, participates in global spatiotemporal self-attention, resulting in spatiotemporal features that are spatially compressed and interactive between frames while maintaining the same temporal length.

[0063] Please refer to Figure 5 In some embodiments, step S31: inputting multiple frames of vehicle acquisition images into the first encoding submodule for block embedding processing to obtain the first feature representation, including steps S311 to S313, which are described in detail below.

[0064] Step S311: For each frame of vehicle acquisition image, slide segment the vehicle acquisition image with a preset step size to obtain multiple image blocks.

[0065] For example, for each frame of vehicle image acquisition, sliding patch segmentation is performed along the spatial dimension with a step size of 4*4, resulting in 56*56 image blocks, or patches, in each frame.

[0066] Step S312: Use a convolutional neural network to extract the spatial features of each image block and flatten each image block into a token sequence.

[0067] Specifically, spatial features within each patch are extracted using a Convolutional Neural Network (CNN), and the patch is flattened into a token sequence, where the token sequence is used to characterize the spatial features of the image patch.

[0068] Step S313: Insert a cls token at the beginning of each frame token sequence, and add a preset position code to the multi-frame token sequence after inserting the cls token to obtain the first feature representation.

[0069] Specifically, by adding a preset position encoding, the spatial position information within the frame is preserved, and a learnable cls token is inserted at the beginning of each frame's token sequence to obtain the first feature representation.

[0070] Please refer to Figure 6 , combined Figure 6 The block embedding process in step S31 is explained as follows: the selected vehicle image frame is subjected to sliding segmentation to obtain multiple patches. The spatial features of each patch are extracted using CNN and flattened into a token sequence. A cls token is inserted at the beginning of each frame's token sequence. The cls token can be 0, 1, 2, ..., HW, where HW represents the total number of patches after image segmentation, H is the number of patches segmented in the image height direction, and W is the number of patches segmented in the image width direction. Finally, position embedding is added to obtain the first feature representation.

[0071] Please refer to Figure 7 In some embodiments, step S32: spatial feature enhancement processing is performed based on the query vector, key vector and value vector in the second spatial attention module to obtain the second feature representation, including steps S321 to S322, which are described in detail below.

[0072] Step S321: Calculate the second attention weights based on the query vector and key vector in the second spatial attention module.

[0073] Step S322: Use the second attention weight to weight the value vector in the second spatial attention module to obtain the second feature representation.

[0074] Specifically, the implementation steps of steps S321 to S322 are described in steps S411 to S412, and will not be repeated here.

[0075] In some embodiments, the temporal decoder in the TRN includes a single-step future prediction branch; please refer to Figure 8 Step S20: Input the vehicle motion information into the TRN for feature extraction to obtain the output global features, including steps S21 to S23, which are explained in detail below.

[0076] Step S21: Extract features from the vehicle's motion information to obtain a feature sequence.

[0077] Step S22: Input the feature sequence into the self-attention mechanism of TRN for temporal fusion to obtain temporal fusion features.

[0078] Step S23: Input the temporal fusion features into the single-step future prediction branch of the temporal decoder for feature prediction to obtain the output global features.

[0079] Specifically, the temporal decoder in TRN usually contains multi-step prediction, such as predicting the next 3-5 frames, which requires recursive or iterative generation of sequences and has a large number of parameters. However, the embodiments of this application simplify TRN. The temporal decoder in the simplified TRN contains a single-step future prediction branch, which only predicts the state at the next moment. The decoder structure is simple, the number of parameters is reduced, and it pays more attention to local dynamics, which makes it easier to analyze the model's response to motion information.

[0080] Please refer to Figure 9 , Figure 9 This is a model architecture diagram of the action detection model training method according to an embodiment of this application, combined with... Figure 9This document describes the main technical approach of the action detection model training method in this application. For the acquired multi-array vehicle images, encoding is performed using a Patch embedding module, a Spatial Attention module, and a Spatial-Temporal Attention module. The output of the Patch embedding module is used as the query vector, key vector, and value vector in the attention mechanism, i.e., as Q, K, and V in the Spatial Attention module. The output of the Spatial Attention module is then used as Q, K, and V in the Spatial-Temporal Attention module to obtain vehicle visual features. Simultaneously, for the sensor input, i.e., the vehicle motion information in this application embodiment, feature extraction is performed using TRN. The extracted global features are used as Q in the cross-attention mechanism, and the vehicle visual features are used as K and V to obtain enhanced global output features. The CLS_Token from the vehicle visual features is concatenated with the enhanced global output features to obtain enhanced fusion features. The enhanced fusion features are input into the classification head, Classification, and a driving action classification loss function, CE LOSS, is constructed. The enhanced fusion features are input into the image prediction network, Image Prediction, which is composed of a multilayer perceptron (MLP) and a deconvolutional network (Deconv), and a future image prediction loss function, MSE LOSS, is constructed.

[0081] In this application embodiment, a training method for an action detection model combining image prediction task and vehicle motion information is proposed. It mainly includes a self-supervised branch for predicting the next frame. Through training with a joint loss function, the generalization ability of spatial-temporal features is improved, and the quality of spatial feature extraction and temporal feature extraction is improved. Through a cross-attention mechanism, the vehicle motion information is enhanced and fused to solve the problem that pure visual input cannot distinguish relative motion. Compared with pure visual algorithms, this application embodiment can obtain more vehicle motion information in weak visual environments, which enhances the robustness of the action detection algorithm. Experimental results show that the method of this application embodiment improves the performance of the state-of-the-art method by 3% under the action detection metric mAP. In addition, it is worth reiterating that the key point and protection point of this application is (1) the development of a self-supervised auxiliary branch that improves the performance of the action detection task by predicting the next frame image during the training phase. This branch shares the underlying spatiotemporal encoder with the main network of the action detection model and improves the generalization ability of spatial-temporal features through joint loss. (2) By introducing vehicle motion information as prior information through a simplified TRN structure and enhancing it through a cross-attention mechanism, and concatenating it with vehicle visual features, the relative motion lacking in visual space is transformed into absolute motion determined in the vehicle coordinate system, thus eliminating the ambiguity of motion information input by pure vision. Under adverse visual conditions (night, heavy rain, glare), the prior vehicle motion information is independently available, forming a "weak vision-strong motion" degradation mode, solving the problem that traditional pure vision cannot extract key motion features.

[0082] Please refer to Figure 10 This application provides a driving action detection method, including steps S80 to S90, which are described in detail below.

[0083] Step S80: Acquire vehicle images and vehicle motion information of the currently moving vehicle.

[0084] Specifically, the vehicle image acquired by the currently moving vehicle includes images of the surrounding environment of the currently moving vehicle within the current time period, and the vehicle motion information includes the position, speed, and acceleration of the currently moving vehicle.

[0085] Step S90: Input the vehicle acquisition image and vehicle motion information into a trained motion detection model to obtain the driving motion detection results of the currently driving vehicle.

[0086] Specifically, the action detection model is trained using the action detection model training method in steps S10 to S70, and the driving action detection results of the currently driving vehicle are obtained through the trained action detection model. These driving action detection results include, but are not limited to, no driving action detected, the currently driving vehicle passing through an intersection, turning left, turning right, the currently driving vehicle changing from the current lane to the left to the adjacent lane, the currently driving vehicle changing from the current lane to the right to the adjacent lane, the currently driving vehicle entering a left-hand branch lane from the main lane, the currently driving vehicle entering a right-hand branch lane from the main lane, the currently driving vehicle passing through a pedestrian crossing area, the currently driving vehicle passing through a level crossing between a railway and a highway, the currently driving vehicle merging from a side road into the main road, and the currently driving vehicle making a U-turn on the road.

[0087] This application provides an action detection model training device, including: Memory, used to store programs; A processor is used to implement a method for training an action detection model by executing a program stored in memory.

[0088] This application provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement an action detection model training method.

[0089] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0090] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for training an action detection model, applied in autonomous driving scenarios to detect driving actions, characterized in that, The action detection model includes an encoding module, a TRN, a cross-attention mechanism, a classification head, and an image prediction network; The method for training the action detection model includes: Acquire multiple frames of vehicle images and vehicle motion information; wherein, the multiple frames of vehicle images include images reflecting the surrounding environment of the currently controlled vehicle within the current time period; The vehicle motion information is input into the TRN for feature extraction, resulting in the output global features. The multi-frame vehicle images are input into the encoding module for encoding processing to obtain vehicle visual features; Based on the cross-attention mechanism, feature transformation is performed on the output global features and the vehicle visual features to obtain enhanced fusion features; The enhanced fusion features are input into the classification head for classification prediction to obtain the driving action prediction classification result, and a driving action classification loss function is constructed based on the driving action prediction classification result and the preset real driving action classification result. The enhanced fusion features are input into the image prediction network to perform image prediction, thereby obtaining future image prediction results. A future image prediction loss function is then constructed based on the future image prediction results and the preset future image real results. The future image prediction results include images of the surrounding environment of the currently controlled vehicle in the future time period. A joint loss function is constructed based on the driving action classification loss function and the future image prediction loss function, and the action detection model is trained based on the joint loss function to obtain a trained action detection model.

2. The action detection model training method as described in claim 1, characterized in that, The step of performing feature transformation on the output global features and the vehicle visual features based on the cross-attention mechanism to obtain enhanced fusion features includes: The output global feature is used as the query vector in the cross-attention mechanism, and the vehicle visual feature is used as the key vector and value vector in the cross-attention mechanism. The output global feature is enhanced based on the query vector, the key vector and the value vector to obtain the enhanced output global feature. The enhanced output global features are stitched together with the vehicle visual features along a preset channel dimension to obtain enhanced fusion features.

3. The action detection model training method as described in claim 2, characterized in that, The step of enhancing the output global features based on the query vector, the key vector, and the value vector to obtain the enhanced output global features includes: Calculate the similarity between the query vector and the key vector, and normalize the similarity to obtain the first attention weight; The value vector is weighted using the first attention weight, and the weighted value vector is fused with the query vector to obtain the enhanced output global feature.

4. The action detection model training method as described in claim 1, characterized in that, The encoding module includes a first encoding submodule, a second spatial attention module, and a third spatiotemporal attention module; The step of inputting the multi-frame vehicle images into the encoding module for encoding processing to obtain vehicle visual features includes: The multi-frame vehicle acquisition images are input into the first encoding submodule for block embedding processing to obtain the first feature representation; The query vector, key vector, and value vector in the second spatial attention module are determined based on the first feature representation. Spatial feature enhancement processing is then performed based on the query vector, key vector, and value vector in the second spatial attention module to obtain the second feature representation. The query vector, key vector, and value vector in the third spatial attention module are determined based on the second feature representation. Spatiotemporal feature enhancement processing is then performed based on the query vector, key vector, and value vector in the third spatial attention module to obtain vehicle visual features.

5. The action detection model training method as described in claim 4, characterized in that, The step of inputting the multi-frame vehicle acquisition images into the first encoding submodule for block embedding processing to obtain the first feature representation includes: For each frame of vehicle image acquisition, the vehicle image is slid segmented with a preset step size to obtain multiple image blocks; Spatial features of each image patch are extracted using a convolutional neural network, and each image patch is flattened into a token sequence; wherein, the token sequence is used to characterize the spatial features of the image patch; A cls token is inserted at the beginning of each frame of the token sequence, and a preset position code is added to the multi-frame token sequence after the insertion of the cls token to obtain the first feature representation.

6. The action detection model training method as described in claim 4, characterized in that, The spatial feature enhancement processing based on the query vector, key vector, and value vector in the second spatial attention module to obtain the second feature representation includes: The second attention weight is calculated based on the query vector and key vector in the second spatial attention module; The value vector in the second spatial attention module is weighted using the second attention weight to obtain the second feature representation.

7. The action detection model training method as described in claim 1, characterized in that, The time-series decoder in the TRN includes a single-step future prediction branch; The step of inputting the vehicle motion information into the TRN for feature extraction to obtain output global features includes: Feature extraction is performed on the vehicle motion information to obtain a feature sequence; The feature sequence is input into the self-attention mechanism of the TRN for temporal fusion to obtain temporal fusion features; The temporal fusion features are input into the single-step future prediction branch of the temporal decoder for feature prediction, resulting in the output global features.

8. A method for detecting driving actions, characterized in that, include: Acquire vehicle images and vehicle motion information of the currently moving vehicle; The vehicle-acquired image and the vehicle motion information are input into a trained motion detection model to obtain the driving motion detection result of the currently driving vehicle; wherein the motion detection model is trained by the motion detection model training method as described in any one of claims 1-7.

9. A motion detection model training device, characterized in that, include: Memory, used to store programs; A processor for implementing the action detection model training method as described in any one of claims 1-7 by executing a program stored in the memory.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the action detection model training method as described in any one of claims 1-7.

Images