Generative End-to-End Automated Driving Method and System Based on Neuromorphic Neural Circuits

Abstract

This application relates to the field of autonomous driving technology, and particularly to a generative end-to-end autonomous driving method and system based on neuromorphic neural circuits. The method includes the following steps: using a temporal bird's-eye view as input to a generative world model, encoding it through an encoder to obtain a feature distribution; sampling the feature distribution using a reparameterization method to obtain sampled features; decoding the sampled features using a decoder to obtain a generative bird's-eye view output; simulating the perception, planning, and control operations of a nematode neural network to establish a neuromorphic neural circuit network; and inputting the sampled features into the neuromorphic neural circuit network to obtain the control input for the autonomous vehicle. The end-to-end autonomous driving method provided in this application solves the integrated planning and control problem of autonomous driving using generative methods and a neuromorphic neural circuit model.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a generative end-to-end autonomous driving method and system based on brain-like neural circuits. Background Technology

[0002] Generative artificial intelligence (AI), as a significant advancement in general AI, is a milestone, marking a shift in AI development from perceptual intelligence to cognitive intelligence. Generative AI has enormous potential to empower the real economy, especially in areas such as autonomous driving. Autonomous driving embodies the cross-integration of the automotive industry with next-generation information technologies such as AI, automated control, and big data in the transportation sector. High-level autonomous driving systems need to cope with almost all complex traffic environments and complete driving tasks safely and efficiently.

[0003] In recent years, bird's-eye view generation technology has developed rapidly, enabling self-driving cars to convert photos of the surrounding environment obtained by cameras into "God's-eye view" bird's-eye views. For example, Tesla's recently launched occupied grid map is a type of bird's-eye view, which allows self-driving cars to obtain surrounding environmental information in an occupied grid manner.

[0004] However, after obtaining a bird's-eye view, how to safely plan a drivable path for an autonomous vehicle and successfully control its movement remains a recognized problem. Traditional methods use a hierarchical approach, planning the trajectory by transforming the positional relationship between the autonomous vehicle and surrounding traffic participants in the bird's-eye view, and then using control algorithms to track the planned trajectory. This method has high interpretability but poor transferability. Summary of the Invention

[0005] This application provides a generative end-to-end autonomous driving method and system based on neuromorphic neural circuits, which uses generative methods and neuromorphic neural circuit models to solve the problem of integrated planning and control in autonomous driving.

[0006] To address the aforementioned technical problems, this application provides a generative end-to-end autonomous driving method based on neuromorphic neural circuits, comprising the following steps: First, a temporal bird's-eye view is used as input to a generative world model, and encoded by an encoder to obtain a feature distribution; next, the feature distribution is sampled using a reparameterization method to obtain sampled features; then, the sampled features are decoded using a decoder to obtain a generative bird's-eye view output; next, a neuromorphic neural circuit network is established by simulating the perception, planning, and control processes of a nematode neural network; finally, the sampled features are input into the neuromorphic neural circuit network to obtain the control quantities of the autonomous vehicle.

[0007] In some exemplary embodiments, the temporal bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

[0008] In some exemplary embodiments, the process of obtaining the generative bird's-eye view output is represented as follows:

[0009]

[0010] Among them, z k Indicates sampling features; z k ~N(μ) k , σ k ) indicates that the encoder output characteristics conform to a Gaussian distribution; μ k σ represents the mean of the characteristic distribution; k It represents the variance of the characteristic distribution.

[0011] In some exemplary embodiments, the brain-like neural circuit network includes four layers of neurons; wherein, the four layers of neurons are: N s One sensory neuron, N i N internal neurons c N instruction neurons, m n motor neurons; between any two consecutive layers, for any source neuron, insert n so-t n synapses; of which, n so-t Satisfy: n so-t ≤N t The synaptic polarity follows a Bernoulli distribution, where N t Indicates the number of target neurons; n so-t The target neurons are randomly selected using a binomial distribution; between any two consecutive layers, any target neuron j without synapses is inserted with mso-t synapses, m so-t satisfy: in, Let m be the number of synapses connected to the target neuron i, and let m be the number of synapses whose polarity follows a Bernoulli distribution. so-t Each source neuron is randomly selected using a binomial distribution; instruction neurons are cyclically connected, and for any instruction neuron, l is inserted. so-t There are synapses, of which l so-t Satisfy: l so-t ≤N c The synaptic polarity follows a Bernoulli distribution, where N c Indicates the number of instruction neurons, l so-t Each source neuron is randomly selected using a binomial distribution.

[0012] In some exemplary embodiments, each neuron is modeled as follows, based on the characteristics of the current transmission between neuronal synapses:

[0013]

[0014] Where x(t) represents the neuronal synaptic current, I(t) represents the external input to the synapse, A is the bias matrix, and f I Let τ represent a neural network, and let τ represent the time constant.

[0015] In some exemplary embodiments, simulating the perception, planning, and control operations of a nematode neural network to establish a brain-like neural circuit network includes: using a function g to represent the brain-like neural circuit network, and using the sampled features to convert them into control action information at the current moment, thereby realizing the conversion process from perception to control. This conversion process is represented by the following formula:

[0016] a k+n =g(z) k , z k+1 , ..., z k+n (3)

[0017] Among them, a k+n This indicates the control action information at the current moment; z k z k+1 z k+n These represent the sampling features at times k, k+1, and k+n, respectively.

[0018] In some exemplary embodiments, after obtaining the generative bird's-eye view output and before inputting the sampled features into the brain-like neural circuit network, the method further includes: inputting the sampled features from time k to k+n into the brain-like neural circuit network to train the generative world model.

[0019] In some exemplary embodiments, training the generative world model includes: inputting sampled features from time k to k+n into the neuromorphic neural circuit network for model training, so that the control quantity and bird's-eye view sequence at the current time are close to the true values, and the encoder output features are close to a Gaussian distribution, and the resulting loss value is:

[0020]

[0021] Where MSE represents the L2 distance between two values, and KL represents the relative entropy between two distributions. and These represent the true values ​​of the actions and the true values ​​of the bird's-eye view output, respectively; the generative world model is trained by minimizing the loss value using gradient descent.

[0022] This application also provides a generative end-to-end autonomous driving system based on neuromorphic neural circuits, including a generative world model module, a neuromorphic neural circuit network module, and a control output module connected in sequence. The generative world model module includes an encoding unit, a sampling unit, and a decoding unit. The encoding unit uses a temporal bird's-eye view as input to the generative world model and encodes it through an encoder to obtain a feature distribution. The sampling unit uses a reparameterization method to sample the feature distribution to obtain sampled features. The decoding unit uses a decoder to decode the sampled features to obtain a generative bird's-eye view output. The neuromorphic neural circuit network module simulates the operation process of perception, planning, and control in a nematode neural network to establish a neuromorphic neural circuit network. The control output module inputs the sampled features into the neuromorphic neural circuit network to obtain the control quantity of the autonomous vehicle.

[0023] In some exemplary embodiments, the temporal bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

[0024] In some exemplary embodiments, the above-described generative end-to-end autonomous driving system based on brain-like neural circuits further includes a model training module; the model training module is used to train the generative world model using the brain-like neural circuit network.

[0025] The technical solution provided in this application has at least the following advantages:

[0026] This application provides a generative end-to-end autonomous driving method and system based on neuromorphic neural circuits. The method includes the following steps: First, a temporal bird's-eye view is used as input to a generative world model and encoded by an encoder to obtain a feature distribution; next, the feature distribution is sampled using a reparameterization method to obtain sampled features; then, the sampled features are decoded using a decoder to obtain a generative bird's-eye view output; next, a neuromorphic neural circuit network is established by simulating the perception, planning, and control operations of a nematode neural network; finally, the sampled features are input into the neuromorphic neural circuit network to obtain the control quantities of the autonomous vehicle.

[0027] This application proposes a generative integrated planning and control method for autonomous driving based on neuromorphic neural circuits. It utilizes a bird's-eye view rich in semantic information as input and directly derives the control information for autonomous driving through an integrated planning and control approach. After the temporal bird's-eye view is input, a decoder first obtains the feature distribution and samples the feature values. The decoder then decodes these sampled feature values ​​to obtain the generative bird's-eye view output. This application simulates the perception, planning, and control processes of a nematode neural network, establishing a neuromorphic neural circuit network. The sampled feature values ​​within the time series are input into this network to obtain the control quantities for the autonomous vehicle, thus completing the integrated planning and control autonomous driving task. Attached Figure Description

[0028] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0029] Figure 1 A flowchart illustrating a generative end-to-end autonomous driving method based on neuromorphic neural circuits provided in an embodiment of this application;

[0030] Figure 2 A flowchart illustrating a generative end-to-end autonomous driving architecture based on neuromorphic neural circuits provided in an embodiment of this application;

[0031] Figure 3 This is a bird's-eye view schematic diagram provided for an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of a brain-like neural circuit architecture provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the structure of a generative end-to-end autonomous driving system based on neuromorphic neural circuits, provided in an embodiment of this application. Detailed Implementation

[0034] As the background technology indicates, traditional methods use a hierarchical approach, planning trajectories by transforming the positional relationships between the autonomous vehicle and surrounding traffic participants in a bird's-eye view, and then using control algorithms to track the planned trajectory. This method offers high interpretability but poor transferability. To obtain a method with strong generality and transferability, a generative approach can be used to design an integrated planning and control method for autonomous driving.

[0035] To address the aforementioned technical problems, this application provides a generative end-to-end autonomous driving method based on neuromorphic neural circuits, comprising the following steps: First, a temporal bird's-eye view is used as input to a generative world model, and encoded by an encoder to obtain a feature distribution; next, a reparameterization method is used to sample the feature distribution to obtain sampled features; then, a decoder is used to decode the sampled features to obtain a generative bird's-eye view output; next, a neuromorphic neural circuit network is established by simulating the perception, planning, and control operations of a nematode neural network; finally, the sampled features are input into the neuromorphic neural circuit network to obtain the control input for the autonomous vehicle. This application provides a generative end-to-end autonomous driving method based on neuromorphic neural circuits, utilizing generative methods and a neuromorphic neural circuit model to solve the integrated planning and control problem of autonomous driving.

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0037] See Figure 1 This application provides a generative end-to-end autonomous driving method based on neuromorphic neural circuits, including the following steps:

[0038] Step S1: Using the temporal bird's-eye view as input to the generative world model, the feature distribution is obtained by encoding through an encoder.

[0039] Step S2: Use the reparameterization method to sample the feature distribution to obtain sampled features.

[0040] Step S3: Use a decoder to decode the sampled features to obtain a generative bird's-eye view output.

[0041] Step S4: Simulate the operation process of nematode neural network in perception, planning and control, and establish a brain-like neural circuit network.

[0042] Step S5: Input the sampled features into the brain-like neural circuit network to obtain the control quantity of the autonomous vehicle.

[0043] This application proposes a generative integrated planning and control method for autonomous driving based on neuromorphic neural circuits. This method uses a semantically rich bird's-eye view as input and directly derives the control information for autonomous driving through an integrated planning and control approach. The method architecture diagram is shown below. Figure 2 As shown. The bird's-eye view information includes vehicle location information, traffic participant location information, lane line information, road boundary information, traffic light information, etc., such as... Figure 3As shown, after the temporal bird's-eye view is input, it first passes through a decoder to obtain the feature distribution and samples the feature values. The decoder then decodes the sampled feature values ​​to obtain the generative bird's-eye view output. For example... Figure 4 As shown, the operation process of perception, planning and control by nematode neural network is simulated. A brain-like neural circuit network is established, and the sampled feature values ​​in time series are input into the brain-like neural circuit network to obtain the control quantity of autonomous vehicle, thus completing the integrated planning and control autonomous driving task.

[0044] Specifically, the present invention can be implemented through the following technical solutions:

[0045] Figure 2 A flowchart illustrating the architecture of a generative world model is shown. Figure 2 As shown, the bird's-eye view at time k is input into I. k Encoding is performed using an encoder, assuming the encoder output features follow a Gaussian distribution, i.e., z k ~N(μ) k , σ k If the encoder generates the mean μ of the feature distribution, then... k and variance σ k The sampled features z are obtained by sampling the feature distribution using a reparameterization method. k , z k Substitute into the decoder to obtain the bird's-eye view output O k .

[0046] In some embodiments, the process of obtaining the generative bird's-eye view output is represented as follows:

[0047]

[0048] Among them, z k Indicates sampling features; z k ~N(μ) k , σ k ) indicates that the encoder output characteristics conform to a Gaussian distribution; μ k σ represents the mean of the characteristic distribution; k It represents the variance of the characteristic distribution.

[0049] This method takes a time-series bird's-eye view as input, that is, the input is the bird's-eye view input I from time k to k+n. k I k+1 to I k+n The sampling features z are obtained through the generative model. k z k+1 To z k+n and bird's-eye view output O k O k+1 To O k+n .

[0050] Figure 3 A bird's-eye view is shown. In some embodiments, such as Figure 3 As shown, the time-series bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

[0051] This application simulates the perception, planning, and control processes of a nematode neural network, establishing a brain-like neural circuit network. The sampled feature values ​​within a time series are input into this network to obtain the control inputs for an autonomous vehicle. *Caenorhabditis elegans* is a very small animal that performs perception and movement functions through its nearly perfect nervous system structure. Many neural circuits in its nervous system are modeled as four-layer hybrid topologies. This application mimics these neural circuits to establish a brain-like neural circuit architecture, such as... Figure 4 As shown.

[0052] See Figure 4 In some embodiments, the brain-like neural circuit network includes four layers of neurons; wherein the four layers of neurons are: N s One sensory neuron, Ni internal neurons, N c N instruction neurons, m n motor neurons; between any two consecutive layers, for any source neuron, insert n so-t n synapses; of which, n so-t Satisfy: n so-t ≤N t The synaptic polarity follows a Bernoulli distribution, where N t Indicates the number of target neurons; n so-t The target neurons are randomly selected using a binomial distribution; between any two consecutive layers, any target neuron j without a synapse is inserted into m. so-t One synapse, m so-t satisfy: in, Let m be the number of synapses connected to the target neuron i, and let m be the number of synapses whose polarity follows a Bernoulli distribution. so-t Each source neuron is randomly selected using a binomial distribution; instruction neurons are cyclically connected, and for any instruction neuron, l is inserted. so-t There are synapses, of which l so-t Satisfy: l so-t ≤N c The synaptic polarity follows a Bernoulli distribution, where N c Indicates the number of instruction neurons, l so-t Each source neuron is randomly selected using a binomial distribution.

[0053] In some embodiments, each neuron is modeled as follows, based on the characteristics of the intersynaptic current transmission:

[0054]

[0055] Where x(t) represents the neuronal synaptic current, I(t) represents the external input to the synapse, A is the bias matrix, and f I Let τ represent a neural network, and let τ represent the time constant.

[0056] In some embodiments, simulating the perception, planning, and control processes of a nematode neural network to establish a brain-like neural circuit network includes: using a function g to represent the brain-like neural circuit network, and employing sampling features z. k z k+1 To z k+n Convert to control action information at the current moment a k+n This process transforms perception into control, thereby achieving integrated planning and control based on a bird's-eye view. The transformation process is represented by the following formula:

[0057] a k+n =g(z) k , z k+1 , ..., z k+n (3)

[0058] Among them, a k+n This indicates the control action information at the current moment; z k z k+1 z k+n These represent the sampling features at times k, k+1, and k+n, respectively.

[0059] In some embodiments, after obtaining the generative bird's-eye view output and before inputting the sampled features into the brain-like neural circuit network, the method further includes: inputting the sampled features from time k to k+n into the brain-like neural circuit network to train the generative world model.

[0060] In some embodiments, training the generative world model includes: inputting sampled features from time k to k+n into the neuromorphic neural circuit network for model training, such that the control quantity and bird's-eye view sequence at the current time are close to the true values, and that the encoder output features are close to a Gaussian distribution, with the resulting loss value being:

[0061]

[0062] Where MSE represents the L2 distance between two values, and KL represents the relative entropy between two distributions. and These represent the true values ​​of the actions and the true values ​​of the bird's-eye view output, respectively; the generative world model is trained by minimizing the loss value using gradient descent.

[0063] See Figure 5 This application also provides a generative end-to-end autonomous driving system based on neuromorphic neural circuits, including a generative world model module 101, a neuromorphic neural circuit network module 102, and a control output module 103 connected in sequence. The generative world model module 101 includes an encoding unit 1011, a sampling unit 1012, and a decoding unit 1013. The encoding unit 1011 uses a temporal bird's-eye view as input to the generative world model and encodes it through an encoder to obtain a feature distribution. The sampling unit 1012 uses a reparameterization method to sample the feature distribution to obtain sampled features. The decoding unit 1013 uses a decoder to decode the sampled features to obtain a generative bird's-eye view output. The neuromorphic neural circuit network module 102 simulates the operation process of perception, planning, and control in a nematode neural network to establish a neuromorphic neural circuit network. The control output module 103 inputs the sampled features into the neuromorphic neural circuit network to obtain the control quantity of the autonomous vehicle.

[0064] In some embodiments, such as Figure 3 As shown, the time-series bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

[0065] In some embodiments, the above-described generative end-to-end autonomous driving system based on neuromorphic neural circuits further includes a model training module; the model training module is used to train the generative world model using a neuromorphic neural circuit network. Data from time k to k+n is input into the model for training, ensuring that the control input and bird's-eye view sequence at the current time are close to the true values, and that z approximates a Gaussian distribution. Gradient descent is used to minimize the loss value to train the entire model.

[0066] Based on the above technical solutions, this application provides a generative end-to-end autonomous driving method and system based on neuromorphic neural circuits. The method includes the following steps: First, a temporal bird's-eye view is used as the input to a generative world model and encoded by an encoder to obtain a feature distribution; next, the feature distribution is sampled using a reparameterization method to obtain sampled features; then, the sampled features are decoded using a decoder to obtain a generative bird's-eye view output; next, the operation process of perception, planning, and control by a nematode neural network is simulated to establish a neuromorphic neural circuit network; finally, the sampled features are input into the neuromorphic neural circuit network to obtain the control quantity of the autonomous vehicle.

[0067] This application proposes a generative integrated planning and control method for autonomous driving based on neuromorphic neural circuits. It utilizes a bird's-eye view rich in semantic information as input and directly derives the control information for autonomous driving through an integrated planning and control approach. After the temporal bird's-eye view is input, a decoder first obtains the feature distribution and samples the feature values. The decoder then decodes these sampled feature values ​​to obtain the generative bird's-eye view output. This application simulates the perception, planning, and control processes of a nematode neural network, establishing a neuromorphic neural circuit network. The sampled feature values ​​within the time series are input into this network to obtain the control quantities for the autonomous vehicle, thus completing the integrated planning and control autonomous driving task.

[0068] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A generative end-to-end autonomous driving method based on brain-like neural circuits, characterized in that, Includes the following steps: Using a temporal bird's-eye view as input to a generative world model, the feature distribution is obtained by encoding through an encoder. The feature distribution is sampled using a reparameterization method to obtain sampled features; The sampled features are decoded using a decoder to obtain a generative bird's-eye view output; Simulating the perception, planning, and control processes of a nematode neural network, a brain-like neural circuit network is established, including: Use functions This represents a brain-like neural circuit network. The sampled features are converted into control action information at the current moment, realizing the perception-control conversion process. This conversion process is represented by the following formula: （3） in, This indicates the control action information at the current moment; , , They represent , , Sampling characteristics at any given time; Will arrive The sampled features at each time point are input into the brain-like neural circuit network to train the generative world model, including: Will arrive The sampled features at each time step are input into the neuromorphic neural circuit network for model training, so that the control quantity and bird's-eye view sequence at the current time step are close to the true values, and the encoder output features are close to a Gaussian distribution. The resulting loss value is: （4） in, Indicates the relationship between two values distance, Represents the relative entropy of two distributions. and These represent the true values ​​of the actions and the true values ​​of the bird's-eye view output, respectively; the generative world model is trained by minimizing the loss value using gradient descent. The sampled features are input into the neuromorphic neural circuit network to obtain the control quantities for the autonomous vehicle.

2. The generative end-to-end autonomous driving method based on neuromorphic neural circuits according to claim 1, characterized in that, The time-series bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

3. The generative end-to-end autonomous driving method based on neuromorphic neural circuits according to claim 1, characterized in that, The process of obtaining the generative bird's-eye view output is represented as follows: （1） in, Indicates sampling characteristics; This indicates that the encoder output characteristics conform to a Gaussian distribution; Represents the mean of the characteristic distribution; It represents the variance of the characteristic distribution.

4. The generative end-to-end autonomous driving method based on neuromorphic neural circuits according to claim 1, characterized in that, The brain-like neural circuit network comprises four layers of neurons; wherein... The four layers of neurons are as follows: A sensory neuron, One internal neuron, One instruction neuron, One motor neuron; Between any two consecutive layers, for any source neuron, insertion One synapse; among them satisfy: The synaptic polarity follows a Bernoulli distribution, where, Each target neuron is randomly selected using a binomial distribution; Between any two consecutive layers, any target neuron without a synapse insert One synapse, satisfy: ;in, To synapse the target neuron The number of synapses and their polarity follow a Bernoulli distribution. Each source neuron is randomly selected using a binomial distribution; Circular connections exist between instruction neurons; for any instruction neuron, insertion... One synapse, of which satisfy: Synaptic polarity follows a Bernoulli distribution. Each source neuron is randomly selected using a binomial distribution.

5. The generative end-to-end autonomous driving method based on neuromorphic neural circuits according to claim 4, characterized in that, Based on the characteristics of current transmission between neuronal synapses, each neuron is modeled as follows: （2） in, Represents neuronal synaptic currents. A represents the external input to the synapse, and A is the deviation matrix. Represents a neural network. This represents the time constant.

6. A generative end-to-end autonomous driving system based on neuromorphic neural circuits, the system being used to implement the generative end-to-end autonomous driving method based on neuromorphic neural circuits as described in any one of claims 1 to 5, characterized in that, It includes a generative world model module, a brain-like neural circuit network module, and a control output module connected in sequence; The generative world model module includes an encoding unit, a sampling unit, and a decoding unit. The encoding unit uses a temporal bird's-eye view as input to the generative world model and encodes it using an encoder to obtain a feature distribution. The sampling unit uses a reparameterization method to sample the feature distribution to obtain sampled features. The decoding unit uses a decoder to decode the sampled features to obtain a generative bird's-eye view output. The brain-like neural circuit network module is used to simulate the operation process of perception, planning and control of the nematode neural network and establish a brain-like neural circuit network. The control output module is used to input the sampled features into the brain-like neural circuit network to obtain the control quantity of the autonomous vehicle.

7. The generative end-to-end autonomous driving system based on neuromorphic neural circuits according to claim 6, characterized in that, The time-series bird's-eye view includes vehicle location information, traffic participant location information, lane line information, road boundary information, and traffic light information.

8. The generative end-to-end autonomous driving system based on neuromorphic neural circuits according to claim 6, characterized in that, It also includes a model training module; the model training module is used to train the generative world model using the brain-like neural circuit network.

Images