A dual attention network model training method and a vehicle control method

Abstract

The application discloses a double-attention network model training method and a vehicle control method, and the method comprises the following steps: acquiring an RGB camera image and LiDAR point cloud data of a target vehicle object, and using different backbone networks to perform feature extraction to obtain a target feature set; using a self-attention mechanism to capture global context features of a single mode in the target feature set, and using a cross-attention mechanism to realize fusion of global context features of multiple modes; using a road point prediction network based on a Transformer to perform trajectory prediction on the target vehicle object according to the fused features, and adding an auxiliary supervision task. The application can fully exploit the feature complementarity of camera images and LiDAR point cloud data, is compatible with feature data fusion of multiple data source types, enhances the ability of the ego intelligent agent to process various information elements in a complex traffic scene, and can be widely applied to the technical field of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a dual attention network model training method and a vehicle control method. Background Technology

[0002] In recent years, end-to-end autonomous driving has attracted considerable interest from researchers. Compared to traditional modular autonomous driving, end-to-end autonomous driving can jointly optimize perception, prediction, and planning tasks, significantly reducing the required computational resources. In complex traffic scenarios, autonomous vehicle agents must be able to simultaneously monitor multiple areas, such as traffic lights overhead and other dynamic agents ahead. Compared to single-modal perception systems that rely solely on cameras or LiDAR, multimodal fusion techniques that integrate information from both sensors can provide autonomous vehicles (AVs) with more comprehensive and robust perception information, thereby enhancing subsequent prediction and planning tasks. However, achieving accurate alignment between the two modalities is often very challenging. Many previous studies have largely utilized geometric projection transformations between image space and LiDAR projection space to achieve feature alignment, such as in bird's-eye view (BEV) images. Establishing a one-to-one hard association between sparse point clouds and dense image pixels through geometric projection not only leads to the loss of a significant amount of image features but also heavily relies on high-quality calibration between the two sensors. Achieving accurate calibration is often very challenging due to the inherent spatiotemporal misalignment problem, and methods based on convolutional neural networks (CNNs) are limited to aggregating multimodal features within a local neighborhood.

[0003] Previous end-to-end autonomous driving research has largely employed gated recurrent unit (GRU) networks for predicting future waypoints for vehicles. These networks are initialized using low-dimensional tensors obtained from sensor inputs extracted via a backbone feature extraction network. However, this approach may result in the loss of perception information related to small objects such as pedestrians and bicycles. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a dual attention network model training method and a vehicle control method to enhance the ability of autonomous vehicle agents to process various information elements in complex traffic scenarios.

[0005] One aspect of this invention provides a method for training a dual-attention network model, comprising:

[0006] RGB camera images and LiDAR point cloud data of the target vehicle object are acquired, and feature extraction is performed using different backbone networks to obtain the target feature set;

[0007] The global context features of a single modality in the target feature set are captured using a self-attention mechanism, and the global context features of multiple modalities are fused using a cross-attention mechanism to obtain fused features.

[0008] Based on the fusion features, a Transformer-based waypoint prediction network is used to predict the trajectory of the target vehicle object, and an auxiliary supervision task is added to enhance the learning ability of the dual attention network model, thus completing the training of the dual attention network model.

[0009] Optionally, the step of acquiring RGB camera images and LiDAR point cloud data of the target vehicle object, and performing feature extraction using different backbone networks to obtain a target feature set, includes:

[0010] For image branch input, three camera images with a field of view of 60° are acquired from the left front, front and right front of the target vehicle, and stitched together to form a wide-angle image with a total field of view of 180°.

[0011] For the point cloud branch input, the acquired 3D LiDAR point cloud is converted into a histogram on a 2D BEV mesh using a coordinate transformation matrix;

[0012] The target feature set was obtained by using a pre-trained ResNet-34 as the backbone network for image feature extraction and a ResNet-18 as the backbone network for point cloud BEV feature extraction.

[0013] Optionally, the step of converting the acquired 3D LiDAR point cloud into a histogram on a 2D BEV mesh using a coordinate transformation matrix includes:

[0014] For any LiDAR point cloud spatial coordinates, perform a perspective transformation to obtain the corresponding image coordinates; for any LiDAR point cloud spatial coordinates, perform a rotation transformation to obtain the corresponding BEV grid coordinates.

[0015] Based on the pixel range of each camera, a corresponding mask is constructed for the BEV grid to divide the BEV grid into multiple regions, so that each region corresponds to the viewpoint of a single camera.

[0016] We establish potential spatial connections between perspective space and BEV space by learningable geometric link location embeddings, and then integrate features of complementary dimensions through weighted averaging operations.

[0017] Optionally, the step of using a self-attention mechanism to capture the global context features of a single modality in the target feature set, and using a cross-attention mechanism to fuse the global context features of multiple modalities to obtain fused features, includes:

[0018] A self-attention mechanism is used to calculate the query, key, and value of a single modality of BEV features in an image or point cloud to determine the feature associations within that single modality.

[0019] A cross-attention mechanism is used to calculate the query, key, and value for different modalities, and key-value queries are performed between different modalities to achieve the fusion of features from different modalities and obtain fused features.

[0020] Optionally, the self-attention mechanism is employed to calculate the query, key, and value of a single modality of BEV features in an image or point cloud to determine the feature associations within that single modality, specifically:

[0021] The intermediate features obtained by the backbone extraction network are flattened into feature sequences, and linear mapping transformation is used to convert the feature sequences into query matrices, key matrices, and value matrices.

[0022] The method employs a cross-attention mechanism to calculate queries, keys, and values ​​for different modalities, and performs key-value queries between different modalities to achieve the fusion of features from different modalities, resulting in fused features. Specifically:

[0023] By combining BEV masks and geometric link position embeddings as auxiliary information for attention calculation, queries, keys, and values ​​between different modalities are calculated using a cross-attention formula.

[0024] Image features are added to BEV features through cross-modal dependencies to enable the LiDAR branch to fuse geometric information and RGB semantic information;

[0025] The fusion process is applied to four ResNet layers in the feature extraction backbone to achieve multimodal fusion at four different resolutions.

[0026] Optionally, the step of using a Transformer-based waypoint prediction network to predict the trajectory of the target vehicle object based on the fused features, and adding an auxiliary supervision task to enhance the learning ability of the dual-attention network model, to complete the training of the dual-attention network model, includes:

[0027] A self-attention mechanism is used to model these long-term time dependencies, and a mask is used to prevent the model from receiving ground truth waypoints during the training phase.

[0028] All waypoints are embedded into a cross-attention mechanism to query the corresponding intermediate features obtained from the image and the BEV branch;

[0029] Two auxiliary tasks, semantic segmentation and BEV mapping, were added to enhance the model's learning ability.

[0030] Optionally, the mask W The expression for (i,j) is:

[0031]

[0032] The formula for calculating the self-attention MSA(Q,K,V) of the mask is:

[0033]

[0034] Where Q represents the query matrix; K represents the key matrix; V represents the value matrix; d represents the number of feature channels; i represents the i-th time step; j represents the j-th waypoint query; Mask W Represents the mask;

[0035] The formula for calculating the loss function of the waypoint prediction network in the dual-attention network model is as follows:

[0036]

[0037] in, The loss function representing the waypoint prediction network; w t This represents the waypoint predicted at time t; This represents the truth path from the expert at time t.

[0038] Another aspect of this invention provides a vehicle control method, comprising:

[0039] Acquire RGB camera images and LiDAR point cloud data of the target vehicle object;

[0040] The RGB camera images and LiDAR point cloud data are input into a dual attention network model to generate road network trajectory prediction results.

[0041] Based on the road network trajectory prediction results, control instructions are generated for the target vehicle object to control the travel trajectory of the target vehicle object;

[0042] The dual attention network model is trained according to the dual attention network model training method described above.

[0043] Another aspect of this invention provides a dual-attention network model training device, comprising:

[0044] The first module is used to acquire RGB camera images and LiDAR point cloud data of the target vehicle object, and to extract features using different backbone networks to obtain a target feature set.

[0045] The second module is used to capture the global context features of a single modality in the target feature set using a self-attention mechanism, and to fuse the global context features of multiple modalities using a cross-attention mechanism to obtain fused features.

[0046] The third module is used to predict the trajectory of the target vehicle object using a Transformer-based waypoint prediction network based on the fused features, and to add an auxiliary supervision task to enhance the learning ability of the dual attention network model, thereby completing the training of the dual attention network model.

[0047] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0048] The memory is used to store programs;

[0049] The processor executes the program to implement the method described above.

[0050] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0051] The embodiments of the present invention first acquire RGB camera images and LiDAR point cloud data of the target vehicle object, and extract features using different backbone networks to obtain a target feature set; then, a self-attention mechanism is used to capture the global context features of a single modality in the target feature set, and a cross-attention mechanism is used to fuse the global context features of multiple modalities to obtain fused features; finally, based on the fused features, a Transformer-based waypoint prediction network is used to predict the trajectory of the target vehicle object, and an auxiliary supervision task is added to enhance the learning ability of the dual-attention network model, thus completing the training of the dual-attention network model. The present invention can fully exploit the feature complementarity of camera images and LiDAR point cloud data, is compatible with feature data fusion of multiple data source types, and enhances the ability of autonomous vehicle agents to process various information elements in complex traffic scenarios. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the overall steps of an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0055] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0056] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0058] The dual-attention network model training method and vehicle control method provided in this invention relate to the field of autonomous driving. These methods can be applied to terminals, servers, or software running on either. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the dual-attention network model training method and vehicle control method, but is not limited to these forms.

[0059] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0060] To address the problems existing in the prior art, the purpose of this invention is to propose a dual-attention Transformer for end-to-end autonomous driving. This network combines self-attention and cross-attention, effectively solving the misalignment problem caused by hard associations formed by sensor calibration, enhancing the ability of the autonomous vehicle agent to process various information elements in complex traffic scenarios. At the same time, a waypoint prediction network based on Transformer is designed to achieve an effective combination of the temporal relationship between waypoints and the spatial features extracted from sensor input.

[0061] Specifically, such as Figure 1 As shown, one aspect of this invention provides a method for training a dual-attention network model, comprising:

[0062] RGB camera images and LiDAR point cloud data of the target vehicle object are acquired, and feature extraction is performed using different backbone networks to obtain the target feature set;

[0063] The global context features of a single modality in the target feature set are captured using a self-attention mechanism, and the global context features of multiple modalities are fused using a cross-attention mechanism to obtain fused features.

[0064] Based on the fusion features, a Transformer-based waypoint prediction network is used to predict the trajectory of the target vehicle object, and an auxiliary supervision task is added to enhance the learning ability of the dual attention network model, thus completing the training of the dual attention network model.

[0065] Optionally, the step of acquiring RGB camera images and LiDAR point cloud data of the target vehicle object, and performing feature extraction using different backbone networks to obtain a target feature set, includes:

[0066] For image branch input, three camera images with a field of view of 60° are acquired from the left front, front and right front of the target vehicle, and stitched together to form a wide-angle image with a total field of view of 180°.

[0067] For the point cloud branch input, the acquired 3D LiDAR point cloud is converted into a histogram on a 2D BEV mesh using a coordinate transformation matrix;

[0068] The target feature set was obtained by using a pre-trained ResNet-34 as the backbone network for image feature extraction and a ResNet-18 as the backbone network for point cloud BEV feature extraction.

[0069] Optionally, the step of converting the acquired 3D LiDAR point cloud into a histogram on a 2D BEV mesh using a coordinate transformation matrix includes:

[0070] For any LiDAR point cloud spatial coordinates, perform a perspective transformation to obtain the corresponding image coordinates; for any LiDAR point cloud spatial coordinates, perform a rotation transformation to obtain the corresponding BEV grid coordinates.

[0071] Based on the pixel range of each camera, a corresponding mask is constructed for the BEV grid to divide the BEV grid into multiple regions, so that each region corresponds to the viewpoint of a single camera.

[0072] We establish potential spatial connections between perspective space and BEV space by learningable geometric link location embeddings, and then integrate features of complementary dimensions through weighted averaging operations.

[0073] Optionally, the step of using a self-attention mechanism to capture the global context features of a single modality in the target feature set, and using a cross-attention mechanism to fuse the global context features of multiple modalities to obtain fused features, includes:

[0074] A self-attention mechanism is used to calculate the query, key, and value of a single modality of BEV features in an image or point cloud to determine the feature associations within that single modality.

[0075] A cross-attention mechanism is used to calculate the query, key, and value for different modalities, and key-value queries are performed between different modalities to achieve the fusion of features from different modalities and obtain fused features.

[0076] Optionally, the self-attention mechanism is employed to calculate the query, key, and value of a single modality of BEV features in an image or point cloud to determine the feature associations within that single modality, specifically:

[0077] The intermediate features obtained by the backbone extraction network are flattened into feature sequences, and linear mapping transformation is used to convert the feature sequences into query matrices, key matrices, and value matrices.

[0078] The method employs a cross-attention mechanism to calculate queries, keys, and values ​​for different modalities, and performs key-value queries between different modalities to achieve the fusion of features from different modalities, resulting in fused features. Specifically:

[0079] By combining BEV masks and geometric link position embeddings as auxiliary information for attention calculation, queries, keys, and values ​​between different modalities are calculated using a cross-attention formula.

[0080] Image features are added to BEV features through cross-modal dependencies to enable the LiDAR branch to fuse geometric information and RGB semantic information;

[0081] The fusion process is applied to four ResNet layers in the feature extraction backbone to achieve multimodal fusion at four different resolutions.

[0082] Optionally, the step of using a Transformer-based waypoint prediction network to predict the trajectory of the target vehicle object based on the fused features, and adding an auxiliary supervision task to enhance the learning ability of the dual-attention network model, to complete the training of the dual-attention network model, includes:

[0083] A self-attention mechanism is used to model these long-term time dependencies, and a mask is used to prevent the model from receiving ground truth waypoints during the training phase.

[0084] All waypoints are embedded into a cross-attention mechanism to query the corresponding intermediate features obtained from the image and the BEV branch;

[0085] Two auxiliary tasks, semantic segmentation and BEV mapping, were added to enhance the model's learning ability.

[0086] Optionally, the mask W The expression for (i,j) is:

[0087]

[0088] The formula for calculating the self-attention MSA(Q,K,V) of the mask is:

[0089]

[0090] Where Q represents the query matrix; K represents the key matrix; V represents the value matrix; d represents the number of feature channels; i represents the i-th time step; j represents the j-th waypoint query; Mask W Represents the mask;

[0091] The formula for calculating the loss function of the waypoint prediction network in the dual-attention network model is as follows:

[0092]

[0093] in, The loss function representing the waypoint prediction network; w t This represents the waypoint predicted at time t; This represents the truth path from the expert at time t.

[0094] Another aspect of this invention provides a vehicle control method, comprising:

[0095] Acquire RGB camera images and LiDAR point cloud data of the target vehicle object;

[0096] The RGB camera images and LiDAR point cloud data are input into a dual attention network model to generate road network trajectory prediction results.

[0097] Based on the road network trajectory prediction results, control instructions are generated for the target vehicle object to control the travel trajectory of the target vehicle object;

[0098] The dual attention network model is trained according to the dual attention network model training method described above.

[0099] The following describes the implementation steps of the present invention in detail using a specific application scenario as an example:

[0100] This invention provides a dual-attention Transformer for end-to-end autonomous driving, the method comprising the following steps:

[0101] S1. Acquire RGB camera images and LiDAR point cloud data, and extract features using different backbone networks respectively;

[0102] S2. Use a self-attention mechanism to capture the global context of a single modality, and use a cross-attention mechanism to achieve the fusion of features from multiple modalities;

[0103] S3. Use a Transformer-based waypoint prediction network for trajectory prediction and add auxiliary supervision tasks to enhance the model's learning ability.

[0104] Optionally, the step of acquiring RGB camera images and LiDAR point cloud data, and performing feature extraction using different backbone networks respectively, specifically includes:

[0105] S11. For the image branch input, acquire images from three cameras with a 60° field of view: left front, front, and right front. These images are then stitched together to form a wide-angle image with a total field of view of 180°. Each camera captures an image with a resolution of 224×300 pixels and contains three parameters: K... k R k and t k (k is the camera number, with values ​​of 1, 2, 3), where K k ∈R 3×3 R represents the camera intrinsic parameter matrix. k ∈R 3×3 Let t represent the camera extrinsic matrix. k ∈R 3×3 This indicates the offset between the origin of the camera coordinate system and the origin of the LiDAR point cloud coordinate system.

[0106] S12. For the point cloud branch input, the acquired 3D LiDAR point cloud is converted into a histogram on a 2D BEV mesh using a coordinate transformation matrix.

[0107] Specifically, for any LiDAR point cloud spatial coordinates P (L) ∈R 3 The corresponding image coordinates P can be obtained by perspective transformation. (l) ∈R 3 The corresponding BEV grid coordinates P can be obtained by rotation transformation. (B) ∈R 3 :

[0108] d c P (l) =K k R k (P (E) -t k )

[0109] Among them, P (E)= (x,y,z), P (l) =(u i ,v i ,1). d c Represents each pixel in the camera coordinate system (u i ,v i The depth value of ). P (B) =(u b ,v b Let (u, z) represent the BEV coordinate system, where z represents each pixel point (u, z) in the BEV coordinate system. b ,v b The height value of ).

[0110] The converted 2D BEV mesh is 192×192 pixels with a resolution of pixels per meter, meaning the BEV view range is equivalent to 48 meters in front of the vehicle and 24 meters to each side. However, since the BEV mesh involves the combined viewpoints of multiple cameras, a BEV mask is needed to divide the mesh into different regions, each corresponding to the viewpoint of a single camera. Based on the pixel range of each camera, a corresponding mask can be constructed for the BEV mesh:

[0111]

[0112] in, This represents the width and height range of the k-th (k = 1, 2, 3) camera. By using a BEV mask, the BEV mesh can be accurately divided into multiple regions to ensure that each region corresponds to the viewpoint of a single camera.

[0113] To achieve alignment between perspective space and BEV space and accurately establish the correlation between the two different modalities, this invention utilizes learnable geometric link position embedding to establish potential spatial connections between the two modalities:

[0114]

[0115]

[0116] Where K(·) represents a linear mapping operation, F I and F B These represent image features and BEV features, respectively. Since the BEV space lacks a height dimension and the perspective space lacks a depth dimension, embodiments of this invention augment the Z-axis and D-axis to compensate for the missing dimensions. Finally, this invention integrates features of complementary dimensions through a weighted average operation.

[0117] S13. Use pre-trained ResNet-34 as the backbone network for image feature extraction and ResNet-18 as the backbone network for point cloud BEV feature extraction.

[0118] Using the classic ResNet as the backbone for feature extraction of images and point cloud BEVs can effectively extract features from the original information of images and point cloud BEVs.

[0119] Optionally, the step of capturing the global context of a single modality using a self-attention mechanism and fusing features from multiple modalities using a cross-attention mechanism specifically includes:

[0120] S21. Employ a self-attention mechanism to calculate the query, key, and value (Q, K, V) of a single modality of BEV features in an image or point cloud, in order to calculate the feature associations within a single modality.

[0121] Specifically, the intermediate features obtained by the backbone extraction network are flattened into a feature sequence F∈R. N×C The feature sequence is transformed into a query matrix Q, a key matrix K, and a value matrix V using a linear mapping transformation, as shown in the following equation:

[0122] Q=FM q +E,K=FM k +E,V=FM v

[0123] Among them, M q M k M v ∈R C×D Let E be the mapping matrix, and E∈R N×C The position embedding is performed in step S12. The formula for calculating self-attention is as follows:

[0124]

[0125] S22. Employ a cross-attention mechanism to calculate queries, keys, and values ​​(Q, K, V) for different modalities, and perform key-value queries between different modalities to achieve the fusion of features from different modalities.

[0126] Specifically, for self-attention, the query, key, and value (Q, K, V) are calculated only within a single modality, while cross-attention is calculated between two modalities. The formula for cross-attention is as follows:

[0127]

[0128] Among them, Q B Let K be the query matrix under the BEV feature. I and V I These are the key matrix and value matrix for image features.

[0129] Cross-attention mechanism by considering Q B and K I Focusing on cross-modal similarity between VI By incorporating BEV masks and positional embeddings of geometric links as auxiliary information for attention computation, this learning model can easily identify such similarities.

[0130] Furthermore, image features V are integrated through cross-modal dependencies. I Add to BEV feature F B In this process, the LiDAR branch integrates geometric and RGB semantic information. Furthermore, the fusion module is applied to four ResNet layers in the feature extraction backbone, resulting in a total of four different resolutions for multimodal fusion.

[0131] Optionally, the step of using a Transformer-based waypoint prediction network for trajectory prediction and adding an auxiliary supervision task to enhance the model's learning ability specifically includes:

[0132] S31. Model the long-term temporal dependence of waypoints using masked self-attention, predict all future n waypoints in parallel, and use a cross-attention mechanism to query perceptual features that are more relevant to waypoint prediction.

[0133] Specifically, given a series of n waypoints: W (in) =(w0,w1,w2,…,w n-2 ,w n-1 The objective of this invention is to predict n future waypoints: W (out) =(w1,w2,w3,…,w n-1 ,w n The specific method used is to predict the next waypoint by using previous waypoints: w0→w1, w1→w2,…,w n-2 →w n-1 ,w n-1 →w n Where w0 represents the origin of the vehicle coordinate system, which is fixed at (0,0), indicating the origin of the vehicle coordinate system.

[0134] Considering the inherent correlations between waypoints, a self-attention mechanism is employed to model these long-term temporal dependencies. A mask is then used. W To prevent the model from receiving ground truth waypoints during the training phase, Masked Self-Attention (MSA) is calculated as follows:

[0135]

[0136]

[0137] Where Q,K,V∈R n×d , which is the d-dimensional embedding vector obtained by encoding two-dimensional waypoints using a shared MLP layer.W ∈R n×n This ensures that the t-th waypoint can only calculate its attention weight with itself and the previous t-1 waypoints (1, 2, ...). Subsequently, all waypoint embeddings utilize the cross-attention mechanism in step S22 to query the corresponding intermediate features obtained from the image and the BEV branch.

[0138] S32. Two auxiliary tasks, semantic segmentation and BEV mapping, are added to enhance the model's learning ability.

[0139] Specifically, for semantic segmentation, seven categories were considered: (1) unlabeled, (2) vehicles, (3) roads, (4) red lights, (5) pedestrians, (6) lane markings, and (7) sidewalk objects. For BEV maps, four categories were considered: (1) unlabeled, (2) vehicles, (3) roads, and (4) lane markings.

[0140] The waypoint prediction network uses the L1 loss function, calculated as follows:

[0141]

[0142] Among them, w t w represents the waypoint predicted at time t. gt This indicates the point at which the expert finds the truth value.

[0143] The loss functions for semantic segmentation and BEV mapping are cross-entropy loss and focus loss, respectively. The overall loss function is calculated as follows:

[0144]

[0145] Wherein, λ pt ,λ Seg and λ Map These are the weighting factors for each task, and their values ​​need to be adjusted by observing the validation errors of multiple experiments.

[0146] In summary, this invention proposes a dual-attention Transformer for end-to-end autonomous driving. By capturing global context through a self-attention mechanism and achieving multimodal feature fusion using a cross-attention mechanism, it effectively solves the misalignment problem caused by hard correlations formed by sensor calibration, enhancing the ability of the autonomous vehicle agent to handle various information elements in complex traffic scenarios. Using this invention, the feature complementarity of camera images and LiDAR point cloud data can be fully exploited, and it is compatible with feature data fusion from multiple data source types, possessing strong practical application value. This invention can be widely applied in the field of end-to-end autonomous driving.

[0147] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned dual-attention network model training method and vehicle control method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0148] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dual-attention network model training method and vehicle control method.

[0150] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0151] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0153] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0154] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0156] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0157] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0158] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0159] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0160] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for training a dual-attention network model, characterized in that, include: RGB camera images and LiDAR point cloud data of the target vehicle object are acquired, and feature extraction is performed using different backbone networks to obtain the target feature set; The global context features of a single modality in the target feature set are captured using a self-attention mechanism, and the global context features of multiple modalities are fused using a cross-attention mechanism to obtain fused features. Based on the fusion features, a Transformer-based waypoint prediction network is used to predict the trajectory of the target vehicle object, and an auxiliary supervision task is added to enhance the learning ability of the dual attention network model, thus completing the training of the dual attention network model. The method involves using a self-attention mechanism to capture the global context features of a single modality in the target feature set, and utilizing a cross-attention mechanism to fuse the global context features of multiple modalities to obtain fused features, including: A self-attention mechanism is used to calculate the query, key, and value of a single modality of BEV features in an image or point cloud to determine the feature associations within that single modality. A cross-attention mechanism is used to calculate the query, key, and value for different modalities, and key-value queries are performed between different modalities to achieve the fusion of features from different modalities and obtain fused features. The method employs a self-attention mechanism to calculate the query, key, and value of a single modality of BEV features in an image or point cloud, in order to determine the feature associations within that single modality. Specifically: The intermediate features obtained by the backbone extraction network are flattened into feature sequences, and linear mapping transformation is used to convert the feature sequences into query matrices, key matrices, and value matrices. The method employs a cross-attention mechanism to calculate queries, keys, and values ​​for different modalities, and performs key-value queries between different modalities to achieve the fusion of features from different modalities, resulting in fused features. Specifically: By combining BEV masks and geometric link position embeddings as auxiliary information for attention calculation, queries, keys, and values ​​between different modalities are calculated using a cross-attention formula. Image features are added to BEV features through cross-modal dependencies to enable the LiDAR branch to fuse geometric information and RGB semantic information; The fusion process is applied to four ResNet layers in the feature extraction backbone to achieve multimodal fusion at four different resolutions.

2. The dual-attention network model training method according to claim 1, characterized in that, The process involves acquiring RGB camera images and LiDAR point cloud data of the target vehicle object, and then using different backbone networks to extract features to obtain a target feature set, including: For image branch input, three camera images with a field of view of 60° are acquired from the left front, front and right front of the target vehicle, and stitched together to form a wide-angle image with a total field of view of 180°. For the point cloud branch input, the acquired 3D LiDAR point cloud is converted into a histogram on a 2D BEV mesh using a coordinate transformation matrix; The target feature set was obtained by using a pre-trained ResNet-34 as the backbone network for image feature extraction and a ResNet-18 as the backbone network for point cloud BEV feature extraction.

3. The dual-attention network model training method according to claim 2, characterized in that, The process of converting the acquired 3D LiDAR point cloud into a histogram on a 2D BEV mesh using a coordinate transformation matrix includes: For any LiDAR point cloud spatial coordinates, perform a perspective transformation to obtain the corresponding image coordinates; for any LiDAR point cloud spatial coordinates, perform a rotation transformation to obtain the corresponding BEV grid coordinates. Based on the pixel range of each camera, a corresponding mask is constructed for the BEV grid to divide the BEV grid into multiple regions, so that each region corresponds to the viewpoint of a single camera. We establish potential spatial connections between perspective space and BEV space by learningable geometric link location embeddings, and then integrate features of complementary dimensions through weighted averaging operations.

4. The dual-attention network model training method according to claim 1, characterized in that, The step involves using a Transformer-based waypoint prediction network to predict the trajectory of the target vehicle object based on the fused features, and adding an auxiliary supervision task to enhance the learning ability of the dual-attention network model, thereby completing the training of the dual-attention network model, including: A self-attention mechanism is used to model these long-term time dependencies, and a mask is used to prevent the model from receiving ground truth waypoints during the training phase. All waypoints are embedded into a cross-attention mechanism to query the corresponding intermediate features obtained from the image and the BEV branch; Two auxiliary tasks, semantic segmentation and BEV mapping, were added to enhance the model's learning ability.

5. The dual-attention network model training method according to claim 4, characterized in that, The mask The expression is: The self-attention of the mask The calculation formula is: Where Q represents the query matrix; K represents the key matrix; and V represents the value matrix; Represents the number of feature channels; Representing the One time step; Representing the Query waypoints; Represents the mask; The formula for calculating the loss function of the waypoint prediction network in the dual-attention network model is as follows: in, The loss function representing the waypoint prediction network; This represents the waypoint predicted at time t; This represents the truth path from the expert at time t.

6. A vehicle control method, characterized in that, include: Acquire RGB camera images and LiDAR point cloud data of the target vehicle object; The RGB camera images and LiDAR point cloud data are input into a dual attention network model to generate road network trajectory prediction results. Based on the road network trajectory prediction results, control instructions are generated for the target vehicle object to control the travel trajectory of the target vehicle object; The dual attention network model is trained using the dual attention network model training method according to any one of claims 1-5.

7. An apparatus for implementing the dual-attention network model training method as described in any one of claims 1-5, characterized in that, include: The first module is used to acquire RGB camera images and LiDAR point cloud data of the target vehicle object, and to extract features using different backbone networks to obtain a target feature set. The second module is used to capture the global context features of a single modality in the target feature set using a self-attention mechanism, and to fuse the global context features of multiple modalities using a cross-attention mechanism to obtain fused features. The third module is used to predict the trajectory of the target vehicle object using a Transformer-based waypoint prediction network based on the fused features, and to add an auxiliary supervision task to enhance the learning ability of the dual attention network model, thereby completing the training of the dual attention network model.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

Images