End-to-end autonomous driving system and method based on diffusion model and secure boot

Abstract

The present disclosure relates to the field of end-to-end autonomous driving, and particularly relates to an end-to-end autonomous driving system and method based on a diffusion model and a safety guide, wherein in the process of generating a trajectory plan by the diffusion model, based on the bird's eye energy graph obtained by the ego vehicle front-end perceiving the traffic environment, the safety guide mechanism is used to adjust the trajectory plan, so as to obtain a trajectory with safety guarantee, avoid or reduce the collision between the planned trajectory and other traffic participants, and improve the safety of autonomous driving while making the planned trajectory conform to the data distribution learned by the network.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of end-to-end autonomous driving, and in particular to an end-to-end autonomous driving system and method based on diffusion model and safety guidance. BACKGROUND

[0002] End-to-end autonomous driving is an autonomous driving technology that directly maps raw data obtained from sensors (such as cameras, lidar, etc.) to vehicle control signals (such as steering, throttle, brake, etc.) using a single, unified model. End-to-end autonomous driving systems exhibit impressive performance in complex urban environments, but lack explicit mechanisms to ensure safe decision-making in dynamic and uncertain traffic environments, making the safety of end-to-end autonomous driving a key challenge for actual deployment.

[0003] Existing end-to-end methods either lack explicit safety guarantee mechanisms, leading to collision between planned trajectories and other traffic participants, or rely heavily on rules to optimize or filter generated trajectories, resulting in rigid and inflexible behavior patterns that cannot be adjusted according to the environment, which to some extent deviates from the original intention of using learning to replace predefined rules. SUMMARY

[0004] To solve the above problems in the prior art, the purpose of the present disclosure is to propose an end-to-end autonomous driving system and method based on diffusion model and safety guidance. In the process of generating trajectory planning by diffusion model, a safety guidance mechanism based on traffic environment is used to adjust the trajectory planning, so as to obtain a trajectory with safety guarantee, avoid or reduce collision between planned trajectory and other traffic participants, and improve the safety of autonomous driving. The specific technical solutions are as follows.

[0005] In a first aspect, the present disclosure proposes an end-to-end autonomous driving system based on diffusion model and safety guidance, which comprises a diffusion planner and a safety guidance module; the diffusion planner is configured to use a conditional diffusion model to generate a noisy trajectory based on a noisy trajectory The enhanced perception feature is fused and enhanced based on the current time perception feature and the historical state, and the historical state is integrated based on GRU for multiple historical perception features; the perception feature is obtained based on the fusion of two-dimensional images and navigation information obtained by the ego vehicle; the safety guidance module is configured to adjust the trajectory based on the noise-free trajectory obtained by the conditional diffusion model at the i-th denoising iteration time step The boundary rectangular frame of the ego vehicle at each time on the trajectory is projected onto the corresponding bird's eye energy graph, and the total energy is calculated based on the energy of the boundary rectangular frame coverage area of the ego vehicle in each bird's eye energy graph obtain a trajectory update gradient ; obtain a trajectory update gradient g and a preset guidance intensity guide adjust the noise-free trajectory generate a new noise-free trajectory calculate a post-guidance prediction noise the post-guidance prediction noise a noisy trajectory for obtaining a next denoising iteration time step diffusion model , , , is a hyperparameter; wherein the bird's eye energy map is generated based on a bird's eye view obtained by enhanced perception.

[0006] In an embodiment of the above technical solution, the bird's eye energy map, the obtaining step comprises: obtaining an avoidance target area of the bird's eye view, the avoidance target area being a minimum rectangular frame area surrounding the vehicle and / or the pedestrian; calculating the energy based on the shortest distance of each pixel position in the bird's eye view relative to the boundary of the avoidance target area, the energy calculation formula being: , and is a scaling coefficient, is the shortest distance, the distance from the inside of the avoidance target to the boundary being calculated as a negative value.

[0007] In an embodiment of the above technical solution, the avoidance target area is obtained by performing a union operation on the vehicle area and the pedestrian area in the segmentation map of the bird's eye view.

[0008] In an embodiment of the above technical solution, the perception feature is obtained based on fusion of a two-dimensional image obtained by the ego vehicle and navigation information, and the step comprises: obtaining two-dimensional image features and depth estimation based on the two-dimensional image obtained by the ego vehicle; projecting and compressing the two-dimensional image features to the bird's eye view plane based on the depth estimation, and then obtaining image perception features through Resnet compression; obtaining navigation encoding based on navigation information through a multilayer perception machine, and fusing the navigation encoding with the image perception features to obtain the perception feature.

[0009] In an embodiment of the above technical solution, the down-sampling layer of the conditional diffusion model adopts an AdaIN-Down module, and the up-sampling layer adopts an AdaIN-Up module, and the AdaIN-Down and AdaIN-Up modules are both composed of a ResBlock and an AdainBlock, wherein the ResBlock is responsible for embedding the denoising iteration time step and calculating and processing the trajectory feature, and the AdainBlock is responsible for injecting the enhanced perception feature.

[0010] Based on the above system technical scheme, the present disclosure also obtains an end-to-end automatic driving method, in which a conditional diffusion model is used to obtain a noise-free trajectory based on a noisy trajectory The planning trajectory is iteratively generated based on enhanced perception features, and the trajectory is adjusted using a safety guide mechanism at each denoising iteration time step to finally obtain a trajectory with safety guarantee; the enhanced perception features are fused and enhanced based on current perception features and historical states, the historical states being integrated based on GRU of multiple historical perception features; the perception features are obtained based on fusion of two-dimensional images and navigation information obtained by the ego vehicle; the safety guide mechanism is used to adjust the trajectory at each denoising iteration time step, and the steps include: obtaining a noise-free trajectory from the conditional diffusion model at the i-th denoising iteration time step The boundary rectangular frame of the ego vehicle at each time on the trajectory is projected onto the corresponding bird's eye energy graph, and the total energy is calculated based on the energy of the boundary rectangular frame covering area of the ego vehicle in each bird's eye energy graph The trajectory update gradient is obtained; based on the trajectory update gradient g and a preset guide intensity , the noise-free trajectory is adjusted to generate a new noise-free trajectory The prediction noise after the guide is calculated, and the prediction noise after the guide is used to obtain a noisy trajectory of the diffusion model at the next denoising iteration time step , , , is a hyperparameter; wherein the bird's eye energy graph is generated based on the bird's eye graph obtained based on enhanced perception.

[0011] In a third aspect, the present disclosure provides a computer-readable storage medium storing a computer program capable of being loaded and executed by a processor to perform the method according to any one of the above aspects.

[0012] The present disclosure has the beneficial technical effects that the safety of the planning trajectory can be improved, and the trajectory planning can be flexibly adjusted according to the environment. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0014] Figure 1 It is a schematic diagram of a trajectory planning model structure of end-to-end automatic driving in an embodiment.

[0015] Figure 2 An end-to-end autonomous driving backbone structure schematic diagram in an embodiment.

[0016] Figure 3 A diffusion planner structure schematic diagram in an embodiment.

[0017] Figure 4 Generating a bird's eye energy map based on the bird's eye view generated from the end-to-end driving backbone output in an embodiment. DETAILED DESCRIPTION

[0018] To avoid collision between the planned trajectory and other traffic participants, and without relying on rules or screening to generate trajectories, adaptive adjustments can be made according to the environment, and the present application proposes an end-to-end autonomous driving method based on diffusion model and safety guidance. By using the bird's eye energy map calculated by the bird's eye view to guide the trajectory denoising and avoid dangerous areas in the process of generating a planned trajectory based on enhanced perception features in the conditional diffusion model, the trajectory generated by the end-to-end autonomous driving system can not only conform to the distribution of learning data, but also ensure safety.

[0019] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of the present application.

[0020] Figure 1 A trajectory planning model structure schematic diagram in an end-to-end autonomous driving system is shown, which includes an end-to-end autonomous driving backbone module, a diffusion planner, and a safety guidance mechanism module, as follows.

[0021] (I) End-to-end autonomous driving backbone module

[0022] Since trajectory planning is closely related to the prediction of other traffic participants, the present disclosure uses the end-to-end autonomous driving backbone module to output a bird's eye view (BEV) and a bird's eye optical flow map about traffic participants, which serve as interpretable intermediate perception results and provide a check reference for safety guidance. The bird's eye view is a BEV semantic segmentation map, and the bird's eye optical flow map is a segmentation occupancy of surrounding vehicles and / or pedestrians at a future time in the BEV semantic segmentation map perspective.

[0023] For the convenience of description, the bird's eye view light flow diagram is described as a bird's eye view of the future time, and the bird's eye view is described as a bird's eye view of the current time, since the safety driving mechanism designed in the present application focuses on the influence of the traffic participants on the safety driving of the ego vehicle. That is, the bird's eye view of the vehicle and / or the pedestrian at each time is used in the present application.

[0024] In an implementation, the end-to-end autonomous driving backbone includes a perception navigation module, a time fusion module, and a decoder.

[0025] The perception navigation module is configured to take the image obtained by the ego vehicle and the ego vehicle navigation information as input and output the perception feature.

[0026] The time fusion module is configured to perform fusion enhancement based on the current perception feature and the historical state h to obtain the enhanced perception feature. In an implementation, a GRU-based module is used to integrate the historical multi-time perception features into the historical state, and the enhanced perception feature is obtained by fusing the historical state with the current perception feature. Meanwhile, the time fusion module uses the perception features of the past multiple times to update the historical state in sequence.

[0027] The decoder is configured to output the bird's eye view of the current time and several future times based on the enhanced perception feature. The decoder is used to capture dynamic environment information and predict the future occupancy of dynamic objects in the BEV space. The bird's eye view of the current time and several future times corresponds to the time of the trajectory point of the noisy trajectory.

[0028] An implementation of the above-mentioned perception navigation module is as follows: the image obtained by the ego vehicle is subjected to feature extraction by using an image encoder composed of a ResNet backbone network to generate two-dimensional image features and depth estimation. Based on the estimated depth, the two-dimensional image features are projected and compressed to the BEV plane, and then processed by multiple ResBlocks to obtain a compact image perception feature representation. The navigation information is processed by a navigation encoder composed of multiple layers of perception machine MLP, and then fused with the image perception feature to become the final perception feature. See Figure 2 The perception navigation module shown in the upper left outputs the perception feature. Figure 2 h in the above-mentioned equation is the historical state. Figure 2 The lower part illustrates that the perception feature is input into the time fusion module and fused and enhanced by using the historical state, and meanwhile the historical state is updated by using the perception feature in the time perception module. The graph decoded by the current perception feature and the current historical state in the decoder is composed of the bird's eye view and the bird's eye light flow diagram (i.e., the bird's eye view of the future time) (see Figure 2 the upper right illustration).

[0029] It should be noted that the end-to-end driving backbone in the present application does not need to specify a specific end-to-end autonomous driving algorithm, and only needs to meet the above-mentioned tasks.

[0030] (ii) Diffusion planner

[0031] The diffusion planner generates the final trajectory using a conditional diffusion model iteratively for denoising, and adjusts the trajectory using a safety guide mechanism at each denoising iteration time step to obtain a final trajectory with safety guarantee. Figure 1 and Figure 3 The "xN" in indicates the iteration.

[0032] The conditional diffusion model injects enhanced perception features as conditions, which are fused and enhanced based on the current perception features and historical state h, which is integrated based on GRU for multiple historical perception features.

[0033] In an embodiment, the conditional diffusion model structure is as shown in Figure 3 The conditional diffusion model uses an AdaIN-based conditional injection mechanism, the down-sampling layer uses an AdaIN-Down module, and the up-sampling layer uses an AdaIN-Up module. The AdaIN-Down and AdaIN-Up module structures are basically the same, and are mainly composed of ResBlock and AdainBlock, forming an alternating stacking structure. The difference is that the AdaIN-Down module uses down-sampling convolution at the tail of the network to compress the time dimension of the trajectory, and the AdaIN-Up module uses up-sampling convolution at the head of the network to expand the time dimension of the trajectory. The time in the time dimension in this part refers to the planning time length of the trajectory, which corresponds to the time of the bird's eye view, which is different from the denoising time in Figure 3 The ResBlock is responsible for embedding the denoising iteration time step and calculating and processing the trajectory features, and the AdainBlock is responsible for injecting enhanced perception features. The bottleneck layer of the diffusion model uses an AdaIN-Mid module, which is only composed of ResBlock and AdainBlock, and is responsible for processing bottleneck information. Figure 3 The time embedding in is to convert the denoising iteration time step into a feature representation. Here, the denoising iteration time step refers to the current denoising iteration time step, which is 10. The time embedding will convert this denoising iteration time step into a 256-dimensional length feature, and then pass it to the AdainBlock. Figure 3 The condition compressor in is an MLP, whose input is the enhanced perception feature, and its output is input into the AdainBlock.

[0034] When using the conditional diffusion model, first, the noisy trajectory The input is input into the Unet network composed of Adain-Down and Adain-Up modules, each Adain-Down and Adain-Up module continuously injects enhanced perception feature condition information, and calculates and processes trajectory features, and finally the Unet network outputs predicted noise .

[0035] In the training or inference process, each intermediate module embeds information and enhanced perception features as conditions in the denoising iteration time step, wherein the enhanced perception features are additional injection conditions designed by the present application, and the denoising iteration time step is a condition model inherent embedding condition.

[0036] Specifically, during training, a Gaussian noise with the same dimension as the trajectory data is first initialized and a denoising iteration time step i is obtained according to the diffusion scheduler parameter The original trajectory is diffused to obtain a noisy trajectory , the noisy trajectory, denoising iteration time step, and enhanced perception feature are input into the diffusion planner to output predicted noise , and the initialized Gaussian noise is used for loss calculation and gradient backpropagation for network learning. The diffusion planner is retrained after the end-to-end automatic driving backbone module is trained.

[0037]

[0038] In application, a Gaussian noise with the same dimension as the trajectory data is first initialized as a noisy trajectory, and the noisy trajectory , enhanced perception feature, and denoising iteration time step i are input into the diffusion planner to output predicted noise , and the diffusion scheduler parameter is used to denoise the noisy trajectory to obtain a noisy trajectory , and the cycle is repeated until the final trajectory is obtained.

[0039]

[0040] In the above, , , are hyperparameters.

[0041] (Three) Safety Guidance Mechanism

[0042] The safety guidance mechanism, as an additional safety guarantee mechanism and diffusion planner, runs together to modify and guide the predicted noise of the diffusion planner so that the final planning trajectory avoids collision.

[0043] In the system implementation, the safety boot mechanism is designed as a safety boot module. The safety boot module is configured to pass the condition diffusion model based on the noise trajectory obtained from the noise trajectory at each denoising iteration time step. The boundary rectangle frame of the ego vehicle at each time on the trajectory is projected onto the corresponding bird's eye energy graph. Based on the energy of the ego vehicle boundary rectangle frame coverage area in each bird's eye energy graph, the energy sum is calculated to obtain the trajectory update gradient. Based on the trajectory update gradient and the guide intensity, the noise-free trajectory is guided to adjust Generate a new noise-free trajectory, and then calculate the noise used to obtain the noise trajectory of the next denoising iteration time step diffusion model.

[0044] The above guide intensity is preset, and the preset value can be 0.1, 0.5, etc.

[0045] The above each time of the ego vehicle corresponds to a bird's eye energy graph, and each time of the bird's eye energy graph is generated based on the end-to-end driving backbone output of the current time and the future time.

[0046] The above noise-free trajectory is obtained by denoising the predicted noise obtained by the diffusion model at each denoising iteration time step.

[0047] The above energy calculation projects the ego vehicle occupied area onto the bird's eye energy graph according to the trajectory position and the direction calculated according to the trajectory to calculate the total energy of the area coverage.

[0048] The above trajectory update gradient is obtained by calculating the gradient of the energy sum with respect to the trajectory. The trajectory update gradient is used to offset the noise-free trajectory to avoid high-energy areas, and the new trajectory obtained is the guided noise-free trajectory. The new trajectory is calculated back to the new predicted noise, which is the guided predicted noise, and then the trajectory sample is normally denoised by the diffusion scheduler. Finally, after multiple denoising iteration time steps, the final safety trajectory is obtained.

[0049] In one embodiment, the perception features are obtained based on the current traffic environment image obtained by the ego vehicle, and the enhanced perception features are obtained by fusing the historical state. The current time and the future 2 time bird's eye view are generated using the enhanced perception features. The number of future time is set according to the need. Referring to Figure 1 As shown, the bird's eye energy graph in the safety guide mechanism is obtained by using the end-to-end automatic driving backbone output bird's eye view and bird's eye optical flow graph, and energy graph generation, Figure 1 Three time bird's eye energy graphs are illustrated in 、 、 It can be understood that corresponding to the bird's eye view, 、 corresponding to the bird's eye optical flow graph.

[0050] The energy map generation from the bird's eye view to the bird's eye energy map includes: first, performing a union operation on the vehicle region and the pedestrian region in the BEV segmentation map to obtain an avoidance target region on the BEV plane, the avoidance target region being a minimum rectangular frame region surrounding the vehicle and / or the pedestrian. Taking the boundary of the minimum rectangular frame region as a reference, the shortest distances of each pixel position in the BEV plane relative to the boundaries of all avoidance target regions are calculated respectively, and the energy values are calculated according to the distances , and is a scaling coefficient, is the shortest distance, and the energy value is higher when the distance is closer. Considering the randomness of the diffusion denoising process, the trajectory sample may overlap with the target, and the distance from the inside of the avoidance target to the boundary is also calculated and taken as the negative value of the distance for calculation to obtain a higher energy, and thus the BEV energy map is obtained. Figure 4 The image obtained by the ego vehicle is shown, as well as the bird's eye view and the bird's eye energy map output by the end-to-end automatic driving backbone module.

[0051] Since the traffic participants with a long distance or a short distance but no collision risk actually have little effect on the planning, only the energy field near the traffic participants is retained to avoid affecting the normal planning and driving behavior. The range of the near energy field can be set, and in one implementation, the energy of all pixel positions with a distance less than 1 meter is retained, so that the energy within the external distance of 1 meter and the internal distance (both negative values) is retained.

[0052] After obtaining the BEV energy map, the noise-free trajectory and the prediction noise of the diffusion planner are recovered from the noisy trajectory This operation is the reverse process of the forward diffusion, and the original trajectory is directly recovered according to the prediction noise and the noisy trajectory. The subscript 0 represents the noise-free trajectory, and the superscript i represents the noisy trajectory based on the i-th denoising iteration time step. Then, the vehicle orientation at each time is calculated according to the noise-free trajectory, and the ego vehicle boundary rectangular frame is projected onto the BEV energy map according to the noise-free trajectory and the orientation, and the energy sum of the ego vehicle boundary rectangular frame coverage region at each time in the trajectory is calculated, and then the gradient of the energy sum relative to the noise-free trajectory is calculated, that is, the energy sum change caused by the unit horizontal and vertical trajectory point position change, as the trajectory update gradient g. Finally, the noise-free trajectory g is updated according to the trajectory update gradient and the guidance intensity , a new noise-free trajectory is obtained by adjustment, and the new noise-free trajectory is the guided noise-free trajectory in Figure 1 ​ , and the noisy trajectory is denoised to obtain a denoised trajectory and the noisy trajectory , and the noisy trajectory is denoised to obtain a denoised trajectory , and the noisy trajectory is denoised to obtain a denoised trajectory , and the noisy trajectory is denoised to obtain a denoised trajectory , , , and the noisy trajectory is denoised to obtain a denoised trajectory . The process is repeated to obtain a final trajectory , which is taken as a planning trajectory. The calculation formula involved in the foregoing process is as follows.

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] The present scheme can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon, for causing a processor to implement aspects of the present application.

[0059] Exemplarily, in a method embodiment, an end-to-end autonomous driving method is provided, in which a conditional diffusion model is used to generate a planning trajectory based on a noisy trajectory , and a safety guide mechanism is used to adjust the trajectory at each denoising iteration time step to finally obtain a trajectory with safety guarantee; the enhanced perception feature is fused and enhanced based on the current time perception feature and the historical state, and the historical state is integrated based on the GRU of the historical multiple perception features; the perception feature is obtained based on the fusion of the two-dimensional image and the navigation information obtained by the ego vehicle; and the safety guide mechanism is used to adjust the trajectory at each denoising iteration time step, and the steps include: obtaining a denoised trajectory , projecting the boundary rectangular frame of the ego vehicle at each time on the corresponding bird's eye energy map, calculating the total energy based on the energy of the covered area of the ego vehicle boundary rectangular frame in each bird's eye energy map, and obtaining a trajectory update gradient ; based on the trajectory update gradient g and the preset guide intensity , the denoised trajectory is adjusted generate a new noise-free trajectory , calculate the post-guided prediction noise , the post-guided prediction noise noise trajectory for obtaining a diffusion model of the next denoising iteration time step , , , is a hyperparameter; wherein the bird's eye energy map is generated based on a bird's eye view obtained based on enhanced perception conditions.

[0060] Exemplarily, in an embodiment of a system, an end-to-end autonomous driving system based on diffusion model and safe guidance, the system comprises a diffusion planner and a safe guidance module; the diffusion planner is configured to use a conditional diffusion model to generate a noise trajectory based on a noise trajectory , and adjust the trajectory at each denoising iteration time step using a safe guidance module to finally obtain a trajectory with safety guarantee; the enhanced perception features are based on the fusion of current time perception features and historical state, and the historical state is based on the integration of multiple historical perception features based on GRU; the perception features are obtained based on the fusion of two-dimensional images and navigation information obtained by the ego vehicle; the safe guidance module is configured to obtain a noise-free trajectory based on the conditional diffusion model of the i-th denoising iteration time step , project the boundary rectangular frame of the ego vehicle at each time on the trajectory onto the corresponding bird's eye energy map, and calculate the total energy based on the energy of the area covered by the boundary rectangular frame of the ego vehicle in each bird's eye energy map , obtain a trajectory update gradient ; based on the trajectory update gradient g and the preset guidance intensity , adjust the noise-free trajectory generate a new noise-free trajectory , calculate the post-guided prediction noise , the post-guided prediction noise noise trajectory for obtaining a diffusion model of the next denoising iteration time step , , , is a hyperparameter; wherein the bird's eye energy map is obtained based on the bird's eye view of the vehicle and / or the pedestrian.

[0061] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0062] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0063] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0064] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0065] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0066] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0067] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0068] Embodiments of the present application have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the disclosed embodiments are possible in light of the above teachings. It is therefore to be understood that within the scope of the disclosed embodiments, modifications and variations of the disclosed embodiments can be practiced. It is also to be understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary processes. Based upon the description and illustrations provided herein, those skilled in the art will understand that changes can be made to the order of steps in the processes and that many of the individual steps can be modified or eliminated. Additionally, the description and illustrations provided herein are not meant to limit the scope of the disclosed embodiments. The scope of the disclosed embodiments is limited only by the claims.

Claims

1. An end-to-end autonomous driving system based on diffusion model and secure boot, characterized in that, The system comprises a diffusion planner and a safety guide module. The diffusion planner is configured to use a conditional diffusion model to generate a planning trajectory based on a noisy trajectory The planning trajectory is iteratively generated using enhanced perception features, and the trajectory is adjusted at each denoising iteration time step using a safety guide module to finally obtain a trajectory with safety guarantee; the enhanced perception features are fused and enhanced based on current perception features and historical states, the historical states being integrated based on GRU for multiple historical perception features; the perception features are obtained based on fusion of two-dimensional images and navigation information obtained by the ego vehicle; The safety guiding module is configured to obtain a noise-free trajectory according to an i-th denoising iteration time step conditional diffusion model projecting the boundary rectangular frame of the ego vehicle at each time on the trajectory onto the corresponding bird's eye energy map, and calculating an energy sum based on the energy of the boundary rectangular frame of the ego vehicle covering area in each bird's eye energy map obtaining a trajectory update gradient based on the trajectory update gradient g and a preset guiding intensity guiding and adjusting the noise-free trajectory to generate a new noise-free trajectory calculating a post-guiding prediction noise the post-guiding prediction noise is used to obtain a noisy trajectory of a next denoising iteration time step diffusion model , , , is a hyperparameter; The aerial energy map is generated based on enhanced perception features.

2. The end-to-end autonomous driving system of claim 1, wherein, The aerial energy map, the acquisition step comprises: An avoidance target area of the aerial view is acquired, and the avoidance target area is a minimum rectangular frame area surrounding the vehicle and / or the pedestrian. The energy is calculated based on the shortest distance of each pixel position in the bird's eye view relative to the boundary of the avoidance target region, the energy calculation formula is: , and is a scaling coefficient, is the shortest distance, the distance from the inside of the avoidance target to the boundary is calculated as a negative value.

3. The end-to-end autonomous driving system of claim 2, wherein, The avoidance target area is acquired by performing a union operation on the vehicle area and the pedestrian area in the segmentation map of the aerial view.

4. The end-to-end autonomous driving system of claim 1, wherein, The perception features are acquired based on fusion of two-dimensional images and navigation information acquired by the ego vehicle, and the step comprises: Two-dimensional image features and depth estimation are acquired based on two-dimensional images acquired by the ego vehicle. The two-dimensional image features are projected and compressed to the aerial view plane based on the depth estimation, and then image perception features are acquired through Resnet compression. Navigation encoding is acquired based on navigation information through a multi-layer perception machine, and the navigation encoding is fused with the image perception features to acquire the perception features.

5. The end-to-end automatic driving system according to claim 1, characterized in that: The down-sampling layer of the conditional diffusion model adopts an AdaIN-Down module, and the up-sampling layer adopts an AdaIN-Up module, and the AdaIN-Down and AdaIN-Up modules are both composed of a ResBlock and an AdainBlock, wherein the ResBlock is responsible for embedding a denoising iteration time step and calculating and processing trajectory features, and the AdainBlock is responsible for injecting enhanced perception features.

6. An end-to-end automatic driving method, characterized in that: In the method, a conditional diffusion model is used to generate a planning trajectory based on a noisy trajectory The planning trajectory is iteratively generated based on enhanced perception features, and the trajectory is adjusted at each de-noising iteration time step using a safety guide mechanism to finally obtain a trajectory with safety guarantee; the enhanced perception features are fused and enhanced based on current perception features and historical states, the historical states being integrated based on GRU for multiple historical perception features; the perception features are obtained based on fusion of two-dimensional images and navigation information obtained by the ego vehicle. The trajectory is adjusted at each denoising iteration time step using a safety guide mechanism, and the step comprises: a noise-free trajectory obtained according to a condition diffusion model of the i-th denoising iteration time step projecting the boundary rectangular frame of the ego vehicle at each time on the trajectory onto the corresponding bird's eye energy map, and calculating an energy sum based on the energy of the area covered by the boundary rectangular frame of the ego vehicle in each bird's eye energy map obtaining a trajectory update gradient based on the trajectory update gradient g and a preset guidance intensity guiding and adjusting the noise-free trajectory to generate a new noise-free trajectory calculating a post-guidance prediction noise the post-guidance prediction noise is used to obtain a noisy trajectory of a diffusion model of a next denoising iteration time step , , , is a hyperparameter; The aerial energy map is generated based on enhanced perception features.

7. The method of claim 6, wherein, The aerial energy map, the acquisition step comprises: An avoidance target area of the aerial view is acquired, and the avoidance target area is a minimum rectangular frame area surrounding the vehicle and / or the pedestrian. The energy is calculated based on the shortest distance of each pixel position in the bird's eye view relative to the boundary of the avoidance target region, the energy calculation formula is: , and is a scaling coefficient, is the shortest distance, the distance from the inside of the avoidance target to the boundary is calculated as a negative value.

8. The method of claim 7, wherein, The avoidance target area is acquired by performing a union operation on the vehicle area and the pedestrian area in the segmentation map of the aerial view.

Images