Vehicle control method, vehicle, and computer-readable storage medium

Abstract

The embodiment of the application provides a vehicle control method, a vehicle and a computer readable storage medium. The method comprises the following steps: acquiring multi-modal sensor data of a vehicle, wherein the multi-modal sensor data comprises asynchronous event data and / or synchronous perception data; performing prediction processing on the multi-modal sensor data by using a target pulse neural network model to obtain a target perception result, wherein the target perception result is used to represent perception attribute information of a perceived object, the perception attribute information comprises at least one of the following: boundary box information, category information and predicted trajectory information, and the target pulse neural network model is obtained by iteratively training a plurality of sets of sample label data; and performing a control operation on the vehicle based on the target perception result. The application solves the technical problems of high power consumption, large delay and poor robustness in extreme environments when performing automatic driving environment perception in the related art.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to a vehicle control method, a vehicle, and a computer-readable storage medium. Background Technology

[0002] With the development of autonomous driving technology, environmental perception systems are facing increasingly higher demands for real-time performance, accuracy, and robustness. Related technologies generally rely on global shutter cameras, LiDAR, and millimeter-wave radar, combined with convolutional neural networks (CNNs) for multimodal data fusion to achieve target detection and tracking. However, these technologies, due to their reliance on dense frame data and high computing power, suffer from problems such as high power consumption, large latency, and susceptibility to failure under extreme lighting conditions. Furthermore, the asynchronous sparse event streams captured by event cameras are difficult to process efficiently by traditional frame-based CNNs.

[0003] There is currently no good solution to the above problems. Summary of the Invention

[0004] This application provides a vehicle control method, a vehicle, and a computer-readable storage medium to at least solve the technical problems of high power consumption, large latency, and poor robustness in extreme environments when performing autonomous driving environmental perception.

[0005] According to one aspect of the embodiments of this application, a vehicle control method is provided, comprising: acquiring multimodal sensor data of a vehicle, wherein the multimodal sensor data includes asynchronous event data and / or synchronous sensing data; performing predictive processing on the multimodal sensor data using a target spiking neural network model to obtain a target perception result, wherein the target perception result is used to represent the perception attribute information of the perceived object, the perception attribute information including at least one of the following: bounding box information, category information, and predicted trajectory information, the target spiking neural network model being obtained by iterative training using multiple sets of sample labeled data; and performing control operations on the vehicle based on the target perception result.

[0006] Furthermore, the target spiking neural network model is used to predict and process the multimodal sensor data to obtain target perception results, including: determining pulse sequence data based on the multimodal sensor data, wherein the pulse sequence data is used to determine the spatiotemporal semantic features of the perceived object; using the target spiking neural network model to predict and process the pulse sequence data to obtain decision pulse data, wherein the decision pulse data includes: pulse spatial distribution data and pulse frequency, wherein the pulse spatial distribution data is used to determine the detection location and detection category of the perceived object, and the pulse frequency is used to determine the detection confidence of the perceived object; and decoding the decision pulse data to obtain the target perception results.

[0007] Further, determining pulse sequence data based on multimodal sensor data includes: preprocessing asynchronous event data to obtain a first pulse sequence, wherein the first pulse sequence represents the pulse sequence after time-domain alignment and noise filtering of the asynchronous event data; responding to the multimodal sensor data containing synchronous sensing data, pulse encoding is performed on the synchronous sensing data to obtain a second pulse sequence; and determining pulse sequence data based on the first pulse sequence and the second pulse sequence, wherein the second pulse sequence represents the pulse sequence corresponding to the synchronous sensing data.

[0008] Furthermore, determining the pulse sequence data based on the multimodal sensor data includes: in response to the multimodal sensor data not containing synchronous sensing data, determining the pulse sequence data based on the first pulse sequence.

[0009] Furthermore, pulse coding is performed on the synchronous sensing data to obtain the second pulse sequence, including: in response to the synchronous sensing data being image frame data, differential coding is performed on the image frame data, or frequency coding is performed on the image frame data to obtain the second pulse sequence; in response to the synchronous sensing data being radar point cloud data, Poisson coding is performed on the radar point cloud data to obtain the second pulse sequence.

[0010] Furthermore, the target spiking neural network model includes an input layer, a hidden layer, and an output layer. The target spiking neural network model is used to predict and process pulse sequence data to obtain decision pulse data. This includes: receiving pulse sequence data using the input layer; performing spatiotemporal pulse integration on the pulse sequence data using the hidden layer to obtain a pulse spatial temporal representation, where the pulse spatial temporal representation is used to represent the spatiotemporal data set of pulse events; and outputting decision pulse data based on the pulse spatial temporal representation using the output layer.

[0011] Furthermore, the hidden layer includes multiple spiking neurons. The hidden layer is used to perform spatiotemporal pulse integration on the pulse sequence data to obtain a pulse spatial temporal representation. This includes: in the hidden layer, multiple spiking neurons are used to perform spatiotemporal pulse integration on the pulse sequence data to obtain a pulse spatial temporal representation. The multiple spiking neurons are constructed based on the leak integral firing model.

[0012] Furthermore, performing control operations on the vehicle based on the target perception results includes: generating vehicle decision commands based on the target perception results, wherein the vehicle decision commands are used to control multiple vehicle actuators, and the multiple vehicle actuators include at least one of the following: a braking mechanism, a steering mechanism, and a throttle mechanism; and performing control operations on the vehicle based on the vehicle decision commands.

[0013] According to another aspect of the embodiments of this application, a vehicle control device is also provided, comprising: an acquisition module for acquiring multimodal sensor data of a vehicle, wherein the multimodal sensor data includes asynchronous event data and / or synchronous sensing data; a processing module for performing predictive processing on the multimodal sensor data using a target spiking neural network model to obtain a target perception result, wherein the target perception result is used to represent the perception attribute information of the perceived object, the perception attribute information including at least one of the following: bounding box information, category information, and predicted trajectory information, and the target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data; and a control module for performing control operations on the vehicle based on the target perception result.

[0014] Furthermore, the processing module is also used to: determine pulse sequence data based on multimodal sensor data, wherein the pulse sequence data is used to determine the spatiotemporal semantic features of the sensed object; perform predictive processing on the pulse sequence data using a target spiking neural network model to obtain decision pulse data, wherein the decision pulse data includes: pulse spatial distribution data and pulse frequency, the pulse spatial distribution data is used to determine the detection location and detection category of the sensed object, and the pulse frequency is used to determine the detection confidence of the sensed object; and decode the decision pulse data to obtain the target sensing result.

[0015] Furthermore, the processing module is also configured to: preprocess the asynchronous event data to obtain a first pulse sequence, wherein the first pulse sequence is used to represent the pulse sequence after time-domain alignment and noise filtering of the asynchronous event data; in response to the multimodal sensor data containing synchronous sensing data, pulse encoding the synchronous sensing data to obtain a second pulse sequence, and determine pulse sequence data based on the first pulse sequence and the second pulse sequence, wherein the second pulse sequence is used to represent the pulse sequence corresponding to the synchronous sensing data.

[0016] Furthermore, the processing module is also configured to: determine pulse sequence data based on the first pulse sequence in response to the multimodal sensor data not containing synchronous sensing data.

[0017] Furthermore, the processing module is also configured to: in response to the synchronous sensing data being image frame data, perform differential coding on the image frame data, or perform frequency coding on the image frame data to obtain a second pulse sequence; in response to the synchronous sensing data being radar point cloud data, perform Poisson coding on the radar point cloud data to obtain a second pulse sequence.

[0018] Furthermore, the target spiking neural network model includes an input layer, a hidden layer, and an output layer. The processing module is also used to: receive pulse sequence data using the input layer; perform spatiotemporal pulse integration on the pulse sequence data using the hidden layer to obtain a pulse spatial temporal representation, wherein the pulse spatial temporal representation is used to represent the spatiotemporal data set of pulse events; and output decision pulse data based on the pulse spatial temporal representation using the output layer.

[0019] Furthermore, the hidden layer includes multiple spiking neurons and a processing module, which is also used to: integrate the pulse sequence data in the hidden layer using multiple spiking neurons to obtain a pulse spatial temporal representation, wherein the multiple spiking neurons are constructed according to the leak integral firing model.

[0020] Furthermore, the control module is also used to: generate vehicle decision commands based on the target perception results, wherein the vehicle decision commands are used to control multiple vehicle actuators, the multiple vehicle actuators including at least one of the following: braking mechanism, steering mechanism and throttle mechanism; and perform control operations on the vehicle based on the vehicle decision commands.

[0021] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0022] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0023] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0024] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0025] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0026] In this embodiment, multimodal sensor data of a vehicle is acquired, including asynchronous event data and / or synchronous perception data. Then, a target spiking neural network model is used to predict the multimodal sensor data to obtain a target perception result. The target perception result represents the perception attribute information of the perceived object, which includes at least one of the following: bounding box information, category information, and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data. Based on the target perception result, control operations are performed on the vehicle, achieving the goal of directly fusing event camera and multi-sensor asynchronous data in the pulse domain. This achieves the technical effects of ultra-low power consumption, microsecond-level response, and high robustness perception in extreme environments, thus solving the technical problems of high power consumption, large latency, and poor robustness in extreme environments in related technologies when performing autonomous driving environmental perception. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 This is a flowchart of an optional vehicle control method according to an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of an optional pulse coding process according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of an optional multimodal data fusion according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram illustrating an optional working principle of a spiking neuron according to an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of an optional vehicle control system hardware architecture according to an embodiment of this application;

[0033] Figure 6 This is a flowchart of another optional vehicle control method according to an embodiment of this application;

[0034] Figure 7 This is a structural block diagram of an optional vehicle control device according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] According to an embodiment of this application, a method embodiment for vehicle control is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] This embodiment provides a vehicle control method. Figure 1 This is a flowchart of an optional vehicle control method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0039] Step S11: Acquire multimodal sensor data of the vehicle, wherein the multimodal sensor data includes: asynchronous event data and / or synchronous sensing data;

[0040] Step S12: The target spiking neural network model is used to predict and process the multimodal sensor data to obtain the target perception result. The target perception result is used to represent the perception attribute information of the perceived object. The perception attribute information includes at least one of the following: bounding box information, category information and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data.

[0041] Step S13: Perform control operations on the vehicle based on the target perception results.

[0042] The aforementioned asynchronous event data represents a set of discrete data points output by the event camera, consisting of pixel coordinates, polarity, and timestamps. It is the original sensing signal generated after the actual perceived change in the brightness of the physical scene.

[0043] The aforementioned synchronous sensing data refers to structured sensing data with a clear timestamp, collected by traditional sensors such as global shutter cameras or millimeter-wave radars at fixed time intervals or under system-level synchronous triggering with event cameras. This includes: complete image frames output by the global shutter camera, which consist of a pixel matrix, with each pixel having a grayscale or color value, reflecting the static light intensity distribution of the scene at the sampling time; or target point cloud data output by millimeter-wave radar, which consists of multiple target points, each containing distance, azimuth, pitch, and radial velocity information, reflecting the position and motion state of the target in the scene. This synchronous sensing data is aligned with the aforementioned asynchronous event data under the system clock reference, providing global semantic, static structural, or velocity information that the event camera cannot directly acquire. Its data format is frame-based or list-based, rather than an event stream. Its acquisition behavior is controlled by an external clock, complementing the asynchronous triggering mechanism of the event camera and providing time-consistent auxiliary input for the multimodal fusion of the spiking neural network.

[0044] The aforementioned acquisition of multimodal sensor data from the vehicle indicates that this embodiment captures asynchronous event data via an event camera. This asynchronous event data consists of pixel coordinates, polarity, and timestamps, reflecting microsecond-level dynamic information about brightness changes in the scene. Simultaneously, this embodiment optionally acquires synchronous sensing data, which originates from a global shutter camera or millimeter-wave radar. The global shutter camera provides frame-based image data, while the millimeter-wave radar provides target point cloud data. Their timestamps are aligned with the event data at the system level, forming a spatiotemporally consistent multimodal input. That is, the aforementioned multimodal sensor data can contain event camera data alone, or it can contain both event camera data and radar / image data. The acquisition of these two types of data breaks through the traditional perception system's dependence on a single sensor modality or reconstructed frame, allowing asynchronous event data and synchronous sensing data to coexist as input without being forcibly unified into a frame structure. This preserves the high dynamic range and low latency advantages of the event camera and is compatible with the existing automotive sensor ecosystem, providing a raw data foundation for subsequent pulse domain fusion.

[0045] The aforementioned target spiking neural network model represents a pre-trained spiking neural network (SNN) model, which is a deep network structure formed by iterative training with multiple sets of labeled sample data, consisting of multiple spiking neurons. Its input layer neurons correspond one-to-one with the pixel array of the event camera, and the hidden layer consists of multiple layers of spiking neurons connected by synaptic weights. It maintains a low-power silent state when there is no input pulse, triggering the integration process only when an input pulse is received. When the membrane potential exceeds the firing threshold, an output pulse is generated and the system resets. Its neurons strictly adhere to temporal dependence characteristics. The pulse frequency of the output layer neurons is mapped to the bounding box (BBox) position, category confidence, or trajectory prediction value of the perceived object. This target spiking neural network model uses pulses as the basic information carrier, achieving efficient and sparse signal processing through the dynamic threshold response and time-dependent mechanism of neurons.

[0046] The aforementioned iterative training indicates that the target spiking neural network model optimizes synaptic weights on a labeled dataset through supervised learning algorithms, such as Spike-Time Backpropagation (STBP), or unsupervised learning algorithms, such as Spike-Timing-Dependent Plasticity (STDP), to establish a stable mapping between the spatiotemporal patterns of spiking and real-world perceived attributes. Its computation process is based on the propagation and integration of discrete spiking events, without relying on floating-point operations or continuous value calculations of activation functions. It possesses the characteristics of event-driven, sparse computation, low power consumption, and low latency. Furthermore, its structural design is adapted to be deployed on Field-Programmable Gate Array (FPGA) or Application-Specific Integrated Circuit (ASIC) hardware for accelerated deployment, providing a computational carrier for the native fusion of multimodal sensor data in the spiking domain.

[0047] The above-mentioned sample labeled data represents a real-world perception scene dataset that has been precisely labeled manually or semi-automatically for training the target spiking neural network model. It contains raw sensor inputs, such as event streams, image frames, and radar point clouds, as well as their corresponding ground truth (GT) perception results, and is used to achieve end-to-end learning of the spiking neural network from "raw pulse signals" to "structured perception outputs".

[0048] For example, in this embodiment, the data collection and processing flow of the sample annotation data is as follows: First, a test vehicle equipped with an event camera, a global shutter camera, and a millimeter-wave radar is used to synchronously collect raw sensor data in typical autonomous driving scenarios such as urban roads, highways, strong light, and low light. Then, all sensors achieve microsecond-level time alignment through hardware synchronization triggers, such as Global Positioning System Pulse Per Second (GPS PPS), and complete spatial coordinate unification through extrinsic parameter calibration. Subsequently, using professional annotation platforms such as Computer Vision Annotation Tool (CVAT), the event stream, image frames, and radar target list within each 5–20 ms time window are jointly annotated manually to accurately annotate the bounding boxes (x_min, y_min, x_max, ...) of dynamic targets. The dataset includes data on y_max, category (vehicle, pedestrian, non-motorized vehicle), ID, and future trajectory points, and marks abnormal states such as occlusion and sudden changes in illumination. Events in each time window are converted into binary tensors with dimensions of 346×260×2, where two channels represent positive and negative polarities. To enhance the model's generalization ability, data augmentation operations such as time stretching, noise injection, and local occlusion are further applied to the event stream. Finally, a multimodal pulse sensing dataset containing tens of thousands of hours of real-world scenes and millions of precisely labeled windows is constructed. This ensures high-quality, highly synchronous, and high-scene-coverage data support for STBP supervised training, pulse coding calibration, and multimodal fusion optimization of the spiking neural network model.

[0049] The bounding box information, category information, and predicted trajectory information described above represent the structured perception results decoded by the output layer of the spiking neural network. These constitute the complete semantic expression for the autonomous driving system to detect, classify, and predict the behavior of dynamic targets, which can be vehicles, pedestrians, or non-motorized vehicles. The bounding box information is used to accurately locate the spatial position of the target on the image plane; the category information is used to identify the semantic type of the target; and the predicted trajectory information is used to model the future movement trend of the target. These three types of information are generated by frequency encoding and decoding of the pulse activity of the spiking neural network output layer and serve as direct inputs to subsequent decision-making and control modules.

[0050] The above-mentioned use of a target spiking neural network model to predict and process the multimodal sensor data to obtain target perception results means that the asynchronous sensor data output by the event camera, global shutter camera and millimeter-wave radar, which is spatiotemporally aligned and encoded into pulse sequences, is directly input into the target spiking neural network model. The model completes end-to-end pulse propagation and fusion calculations based on the pulse timing dependency mechanism without global clock driving, and generates a structured perception result containing bounding box information, category information and predicted trajectory information by decoding the pulse firing frequency of the output layer neurons.

[0051] The aforementioned control operations on the vehicle based on target perception results involve transmitting the decoded target perception results to the vehicle decision and control module via a standardized communication protocol, such as Robot Operating System 2 (ROS2) or Controller Area Network (CAN) bus. The control module then uses the bounding box information from the perception results to determine the target's position, category information to identify the target type, and trajectory prediction information to infer the motion trend, thereby generating steering, braking, or acceleration commands to complete closed-loop control. This process achieves direct connection between perception output and control input, ensuring that the perception results drive downstream systems in a structured and executable format.

[0052] Based on steps S11 to S13, this embodiment first acquires multimodal sensor data of the vehicle, including asynchronous event data and / or synchronous perception data. Then, a target spiking neural network model is used to predict the multimodal sensor data to obtain a target perception result. The target perception result is used to represent the perception attribute information of the perceived object, which includes at least one of the following: bounding box information, category information, and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data. Then, based on the target perception result, control operations are performed on the vehicle, achieving the goal of directly fusing event camera and multi-sensor asynchronous data in the pulse domain. This achieves the technical effects of ultra-low power consumption, microsecond-level response, and high robustness perception in extreme environments, thereby solving the technical problems of high power consumption, large latency, and poor robustness in extreme environments in related technologies when performing autonomous driving environment perception.

[0053] Furthermore, the target spiking neural network model is used to predict and process the multimodal sensor data to obtain target perception results, including: determining pulse sequence data based on the multimodal sensor data, wherein the pulse sequence data is used to determine the spatiotemporal semantic features of the perceived object; using the target spiking neural network model to predict and process the pulse sequence data to obtain decision pulse data, wherein the decision pulse data includes: pulse spatial distribution data and pulse frequency, wherein the pulse spatial distribution data is used to determine the detection location and detection category of the perceived object, and the pulse frequency is used to determine the detection confidence of the perceived object; and decoding the decision pulse data to obtain the target perception results.

[0054] The aforementioned pulse sequence data refers to the data composed of the asynchronous event stream output by the event camera, the pulse sequence output by the global shutter camera, and the spatial pulse sequence output by the millimeter-wave radar. It is used to characterize the semantic features of the perceived object in the spatiotemporal dimension, such as brightness changes, motion edges, and velocity distribution.

[0055] The aforementioned decision pulse data represents the pulse data generated by the activation of output layer neurons of the target spiking neural network model. It includes pulse spatial distribution data and pulse frequency. The pulse spatial distribution data maps the bounding box position and category of the perceived object through the spatial index of the output neurons in the preset grid. The pulse frequency reflects the pulse firing density of each neuron within the time window and is used to quantify the detection confidence of the perceived object.

[0056] The above-described decoding process of decision pulse data yields target perception results. This means that the detection position of the perceived object is determined based on the spatial coordinates of high-frequency activated neurons in the pulse spatial distribution data, the detection category is determined based on the highest frequency group of the corresponding neuron group for different categories, and the predicted trajectory is formed by connecting the position sequences of high-frequency neurons in the continuous time step output layer. Finally, a structured perception result containing bounding box information, category information, and predicted trajectory information is output.

[0057] Based on the above optional embodiments, the embodiments of this application complete the end-to-end mapping from raw sensor data to semantic perception results in the pulse domain. It does not require image reconstruction, feature stitching, or floating-point operations. It achieves low-latency, low-power encoding and decoding of perception information only through the spatiotemporal distribution and firing intensity of pulse events. At the same time, it realizes sparse, asynchronous, and highly robust target perception and semantic extraction of multimodal sensor data based on an event-driven mechanism without global clock driving.

[0058] Further, determining pulse sequence data based on multimodal sensor data includes: preprocessing asynchronous event data to obtain a first pulse sequence, wherein the first pulse sequence represents the pulse sequence after time-domain alignment and noise filtering of the asynchronous event data; responding to the multimodal sensor data containing synchronous sensing data, pulse encoding is performed on the synchronous sensing data to obtain a second pulse sequence; and determining pulse sequence data based on the first pulse sequence and the second pulse sequence, wherein the second pulse sequence represents the pulse sequence corresponding to the synchronous sensing data.

[0059] The aforementioned first pulse sequence represents a sequence of pulses generated by aggregating asynchronous event data through a sliding time window and applying a spatial-temporal filtering algorithm to remove isolated noise pulses. This preprocessing of the asynchronous event data to obtain the first pulse sequence involves aggregating the raw asynchronous event data output from the event camera according to a fixed time window. Within each time window, all events are sorted by their timestamps, ensuring strict temporal consistency and achieving temporal alignment. Subsequently, a spatial-temporal filtering algorithm is applied to remove isolated pulses occurring within a very short time interval at the same or adjacent pixels without subsequent related events. These isolated pulses are typically generated by sensor noise, charge leakage, or environmental interference. After filtering, real event pulses with continuous spatiotemporal evolution characteristics are retained, forming a structured, high signal-to-noise ratio first pulse sequence. This first pulse sequence contains only pulse events with spatiotemporal continuity caused by real dynamic changes, providing an accurate and reliable input foundation for subsequent spiking neural networks.

[0060] The above response to the multimodal sensor data including synchronous sensing data involves pulse coding of the synchronous sensing data to obtain a second pulse sequence. This indicates that when the system simultaneously receives synchronous sensing data from a global shutter camera or millimeter-wave radar, the spatial location, category confidence, or motion state information in the synchronous sensing data is converted into discrete pulse activation patterns of neurons in the input layer of a spiking neural network, generating a set of pulse events that are compatible with the asynchronous pulse sequence output by the event camera in terms of time reference and data format.

[0061] The aforementioned determination of pulse sequence data based on the first and second pulse sequences means aligning and concatenating the first and second pulse sequences along the timestamp dimension and aligning the pixel arrays along the spatial dimension to form a unified multi-source pulse input stream. This pulse sequence data retains the high dynamic response capability of the event camera to high-speed motion and extreme lighting conditions, while also incorporating the global semantics and target position priors provided by the synchronous sensor. This achieves native fusion of heterogeneous sensor data in the pulse domain, providing a structurally consistent and semantically complementary unified input representation for the subsequent parallel, sparse, and event-driven processing of the target spiking neural network model.

[0062] Based on the above optional embodiments, this application embodiment eliminates the computational redundancy of image reconstruction and feature stitching in traditional methods by uniformly encoding asynchronous event streams and synchronous sensing data into pulse sequences. This enables all sensor data to enter the subsequent neural network processing stage in the same event-driven format, realizing the spatiotemporal alignment and format unification of heterogeneous sensor data in the pulse domain. This provides a structurally consistent and time-accurate input foundation for the end-to-end fusion processing of subsequent spiking neural networks.

[0063] Furthermore, determining the pulse sequence data based on the multimodal sensor data includes: in response to the multimodal sensor data not containing synchronous sensing data, determining the pulse sequence data based on the first pulse sequence.

[0064] The above response that the multimodal sensor data does not include synchronous sensing data indicates that the system only connects to the event camera as a sensing input source, and does not receive frame-based or point cloud-based data from the global shutter camera, millimeter-wave radar or other synchronous sensors.

[0065] The above-mentioned determination of pulse sequence data based on the first pulse sequence indicates that the system skips the generation and fusion process of the second pulse sequence and directly uses the first pulse sequence, which has undergone temporal alignment and noise filtering, as the final pulse sequence data. The first pulse sequence is formed by processing the raw asynchronous event data output by the event camera through time window aggregation and space-time filtering algorithms. Isolated noise pulses have been removed and the orderliness of events in the time dimension has been ensured. It retains all spatiotemporal semantic information of dynamic environmental changes. Moreover, in scenarios without synchronous sensing data, the embodiments of this application can still rely entirely on the first pulse sequence for subsequent spiking neural network inference without introducing redundant data interfaces or forcibly filling pseudo-data, thus avoiding system degradation or performance degradation due to sensor absence.

[0066] Based on the above optional embodiments, the embodiments of this application realize the adaptive capability of sensor configuration, construct a pure event-driven perception path with the event camera as the core and no dependence on synchronous data. When the autonomous driving system faces extreme conditions such as sensor failure, communication interruption or cost limitation, it can still maintain basic but reliable dynamic target perception capability, ensuring the functional integrity and operational robustness of the system in single-modal input scenarios.

[0067] Furthermore, pulse coding is performed on the synchronous sensing data to obtain the second pulse sequence, including: in response to the synchronous sensing data being image frame data, differential coding is performed on the image frame data, or frequency coding is performed on the image frame data to obtain the second pulse sequence; in response to the synchronous sensing data being radar point cloud data, Poisson coding is performed on the radar point cloud data to obtain the second pulse sequence.

[0068] The above response, assuming the synchronous sensing data is image frame data, involves differential encoding or frequency encoding of the image frame data to obtain a second pulse sequence. This indicates that when the synchronous sensing data originates from 30fps continuous image frames output by a global shutter camera, differential encoding calculates the grayscale intensity difference of each pixel between the current and previous image frames. When the difference exceeds a preset intensity threshold, a pulse event with a precise timestamp is generated at the corresponding pixel location. The pulse polarity is determined by the direction of grayscale change (increase or decrease), thus transforming moving edges and dynamic regions in the image frame into a spatiotemporally sparse pulse sequence. Alternatively, frequency encoding maps the grayscale value of each pixel in the image frame to the average pulse firing frequency per unit time. The higher the grayscale value, the higher the pulse firing frequency, forming a continuous pulse stream proportional to the image brightness distribution. Both encoding methods transform the static spatial information of the image frame into a pulse expression aligned with the event camera's time reference. The aforementioned image pulse sequence is fused with event pulse features through a lightweight convolutional structure after the first hidden layer of the SNN to generate a multimodal feature map with spatial semantic enhancement.

[0069] The above response is that the synchronous sensing data is radar point cloud data. The radar point cloud data is Poisson encoded to obtain the second pulse sequence. This means that when the synchronous sensing data comes from the target point cloud output by the millimeter-wave radar, Poisson encoding is used to map the spatial coordinates of each target, i.e., azimuth and distance, to a specific cluster of neurons in the input layer of the spiking neural network. The pulse firing frequency is set according to the radial velocity of the target, and the pulse firing frequency is proportional to the radial velocity. The specific formula is: pulse frequency = k × v.

[0070] Here, pulse frequency represents the number of pulses fired per second, i.e., the pulse firing frequency, measured in Hertz (Hz). Velocity (v) represents the radial velocity of the target detected by the radar, measured in meters per second (m / s). Scaling factor (k) represents a trainable or preset scaling factor used to adjust the mapping sensitivity from velocity to frequency. Simultaneously, the pulse timestamp is synchronized with the event stream, thereby encoding the radar target's motion state into a pulse sequence with spatial localization and velocity modulation characteristics. This second pulse sequence is element-wise fused with the original pulse stream from the event camera in the second hidden layer of the SNN. The fusion weights are adaptively optimized through training, enabling the radar's velocity information to effectively enhance the SNN's ability to estimate target depth and motion status.

[0071] For example, Figure 2 This is a schematic diagram of an optional pulse coding process according to an embodiment of this application. Specifically, Figure 2The process of encoding a millimeter-wave radar target list into a pulse sequence is described. In the embodiments of this application, the millimeter-wave radar output target list contains several dynamic targets, such as vehicles and pedestrians. Each target is represented by structured data, which includes three key parameters: azimuth angle θ, radial velocity v, and target identifier ID.

[0072] The specific encoding process is as follows: First, each target in the radar target list, such as target ID 1 and target ID 2, carries its azimuth angle θ, radial velocity v, and unique target ID. Then, the system uses a preset mapping table to bind the target ID to the target's azimuth angle θ in the radar detection coordinate system. For example, ID 1 corresponds to θ = +15°, and this azimuth angle determines the position of the neuron cluster activated in the SNN input layer. For example, 15° is located in neuron cluster 2: 10°-20°, meaning the target is assigned to neuron clusters 1-2; ID 2 corresponds to θ = +55°, located in neuron cluster 6: 50°-60°, meaning the target is assigned to neuron clusters 5-6. Subsequently, the target's relative velocity v is used to calculate the pulse firing frequency. According to the formula "pulse frequency = k × v", the velocity information is converted into the pulse time density. The higher the velocity, the more pulses fired by the neuron cluster per unit time, thus reflecting the target's dynamic characteristics in the pulse sequence. Furthermore, the target presence confidence is achieved through a normalized probability gating mechanism for pulse output: if the confidence is below a threshold, for example, less than 0.6, the target's pulse sequence is suppressed, and no pulses are injected into the SNN; if the confidence is above the threshold, pulses are generated according to the aforementioned velocity-frequency relationship. The higher the confidence, the more likely a scaling factor can be added to further increase the pulse frequency. For example, the pulse frequency can be calculated using the formula "pulse frequency = k × v × Score", where Score is the original detection confidence from the radar output, resulting in stronger neuronal activation signals for high-confidence targets in the SNN. Finally, this neuron cluster outputs a series of asynchronous pulses on the time axis. The spatial location (i.e., the neuron cluster index implies the target's orientation), the pulse density (implicitly indicating the target's velocity), and the presence or absence of pulses (implying the target's existence) are fused into a unified, event-driven pulse sequence. For example, pulse sequences A and B, and pulse sequences E and F, are spatiotemporally aligned with the pulses output by the event camera, merging into a unified pulse event stream that is directly input into the SNN for joint pulse integration and feature extraction. The above encoding process is completely frameless and floating-point arithmetic-free, relying solely on hardware logic mapping and pulse counting, thus achieving lightweight, low-latency, and low-power pulsed expression of radar information.

[0073] Based on the above optional embodiments, the embodiments of this application realize that image frame data and radar point cloud data are accurately converted into spatiotemporally aligned pulse sequences by differential coding, frequency coding or Poisson coding, so that synchronous sensing data can be uniformly adapted to the asynchronous and sparse input format of spiking neural networks without changing the original semantics. This avoids the computational redundancy and delay accumulation caused by image reconstruction, feature stitching and time interpolation in traditional methods, and provides a consistent, time-accurate and semantically preserved input foundation for multimodal pulse fusion.

[0074] Furthermore, the target spiking neural network model includes an input layer, a hidden layer, and an output layer. The target spiking neural network model is used to predict and process pulse sequence data to obtain decision pulse data. This includes: receiving pulse sequence data using the input layer; performing spatiotemporal pulse integration on the pulse sequence data using the hidden layer to obtain a pulse spatial temporal representation, where the pulse spatial temporal representation is used to represent the spatiotemporal data set of pulse events; and outputting decision pulse data based on the pulse spatial temporal representation using the output layer.

[0075] The aforementioned target spiking neural network model adopts an end-to-end event-driven structure, including an input layer, hidden layers, and an output layer. Specifically, the input layer consists of 69,200 neurons, corresponding one-to-one with the 346×260 pixel array and positive and negative polarity channels of the event camera, directly receiving the original asynchronous event stream; the hidden layer extracts features through two fully connected Leaky Integrate-and-Fire Neuron Model (LIF) spiking neuron layers, each containing 512 neurons, and employs a non-linear activation mechanism of Rectified Linear Unit (ReLU) to enhance the expressive power of the impulse response; the output layer is a multi-task target detection layer, mapping neurons to the category (e.g., vehicle, pedestrian) and bounding box parameters (center coordinates, width and height) of a preset anchor box (perceived object), and its impulse firing frequency represents the detection confidence, achieving efficient decoding from sparse impulses to continuous semantic output. Furthermore, during training, the target spiking neural network model employs a supervised learning algorithm based on temporal backpropagation (STBP), using an adaptive moment estimation optimization algorithm (ADAM) optimizer and a weighted cross-entropy loss function. This is accomplished within the PyTorch Deep Learning Framework with Dynamic Computation Graphs Based on Tensors (PyTorch) framework, combined with dedicated SNN libraries such as the Spike Neural Network Extension Library for PyTorch (snnTorch). The optimizer has a learning rate of 0.001, and the training process operates in 5-millisecond event windows, sequentially executing event pulse encoding, SNN forward propagation, output pulse decoding as detection results, loss calculation, and backpropagation to update weights. This achieves end-to-end optimization from the raw event stream to high-precision target detection.

[0076] The above-mentioned use of the input layer to receive pulse sequence data means that the input layer is composed of neurons that correspond one-to-one with the pixel array of the event camera. Each neuron corresponds to a pixel position and its positive and negative polarity channels. Each pulse event in the pulse sequence data carries a precise timestamp, spatial coordinates and polarity information. When the input layer neuron receives a pulse that matches its spatial position, it immediately generates a response and transmits it to the next layer, realizing lossless reception and spatial mapping of the original pulse event, ensuring that the spatiotemporal distribution of the event is completely preserved.

[0077] The above-described method utilizes hidden layers to perform spatiotemporal pulse integration on pulse sequence data, resulting in a pulse spatial temporal representation. This means the hidden layer, acting as an intermediate processing unit in the target spiking neural network model, receives pulse sequence data from the input layer. Based on predefined connection mapping relationships and spatiotemporal weight configurations, it performs correlation calculations between the spatial coordinates and timestamps of pulse events, aggregating pulse events scattered across different pixel locations and time points into a response pattern with spatiotemporal continuity, and outputting a pulse spatial temporal representation. The pulse sequence data undergoes nonlinear integration in the hidden layer through predefined spatiotemporal rules, outputting a pulse spatial temporal representation with spatiotemporal structural integrity and consistent event response, providing accurate dynamic feature inputs for the generation of decision pulses in the output layer.

[0078] Figure 3 This is a schematic diagram of an optional multimodal data fusion according to an embodiment of this application. The target spiking neural network (SNN) comprises a three-layer structure: an input layer, a hidden layer (containing fusion points), and an output layer. Before inputting the data into the spiking neural network (SNN), event camera data and radar / image data are first acquired. The event camera pulse input stream consists of asynchronous events output by the event camera, and the asynchronous event stream data includes the pixel coordinates (x, y), polarity (positive / negative, p), and microsecond-level timestamp (t) of each event. After preprocessing isolated noise pulses through time window sliding aggregation and space-time filtering algorithms, a pulse sequence A is generated and passed into the input layer of the SNN. The multi-sensor pulse input stream includes preprocessed pulse sequences from millimeter-wave radar and global shutter camera: the target list of the millimeter-wave radar is converted into a pulse sequence through spatial distribution-based Poisson coding. The pulse firing position is mapped to a specific cluster of neurons in the input layer according to the target azimuth angle. The pulse frequency is proportional to the radial velocity of the target, thus encoding the velocity information into temporal pulse density. The image frames of the global shutter camera generate pulse sequences of changing regions through differential coding. The timestamps are aligned with the event stream, and the pulse positions correspond to the coordinates of pixels with brightness changes in the image, forming an auxiliary dynamic feature stream consistent with the spatiotemporal of the event stream, which is also mapped to the input layer neurons.

[0079] Furthermore, the pulse sequences from the two types of non-event-based sensors mentioned above, i.e., pulse sequence B, are synchronously injected into the hidden layer of the SNN via independent channels. These pulse sequences are then fused with the raw pulse stream from the event camera at the neuron level through element-level pulse superposition. Specifically, if a neuron simultaneously receives multiple pulses from the event stream, radar stream, or image stream within the same time window, its total input pulse count is the sum of these three. The neuron's membrane potential integral is accumulated based on this combined pulse intensity, achieving native fusion of multi-source information. The fused pulse sequence undergoes further spatiotemporal nonlinear processing in the hidden layer neuron LIF model, namely pulse convolution and recursive connections. Finally, it outputs decision pulses through the output layer and is decoded into unified perception result data, representing the category and confidence level of the perceived object, such as: "vehicle, confidence level 95%", "pedestrian, confidence level 80%".

[0080] The above-described fusion process in this application embodiment is deployed on an FPGA. It significantly reduces power consumption in static scenarios through dynamic clock adjustment and reduces end-to-end response latency by utilizing hardware interrupt mechanisms. Furthermore, under extreme lighting conditions, such as 105dB strong light and 10dB weak light, the measured target detection accuracy is significantly improved compared to traditional CNN schemes, thereby achieving low power consumption, high real-time performance, and strong robustness.

[0081] The aforementioned pulse spatial temporal representation represents a set of data that characterizes the overall structure of the pulse events in terms of spatial distribution and temporal evolution. This data is formed by associating and aggregating the spatial coordinates and timestamp information carried by each pulse event in the input pulse sequence data during the hidden layer processing of the target spiking neural network model, using a preset spatiotemporal mapping rule. This pulse spatial temporal representation does not rely on image reconstruction or fixed-frame sampling; instead, it is directly composed of the spatiotemporal sequence relationship of pulse events. Its content includes the spatial distribution of all pulse events within a specific time window in the pixel array, and the chronological order and density variation trend of these pulse events according to their timestamps. It fully preserves the directional, velocity changes, and edge evolution characteristics of the target motion in a dynamic scene, and is the unique form of the hidden layer output.

[0082] The above-mentioned output layer uses pulse spatial temporal representation to output decision pulse data, indicating that the output layer neurons correspond one-to-one with the perception task target, such as target category, bounding box position or trajectory prediction point. The pulse firing frequency of each output neuron is proportional to the confidence of the corresponding target. The pulse sequence is decoded by frequency to form the final perception result. The above-mentioned output layer realizes continuous tracking and dynamic updating of the target state through the temporal dependence of pulse spatial temporal representation, avoiding the information gap caused by the fixed frame rate in traditional neural networks.

[0083] Based on the above optional embodiments, this application embodiment constructs an end-to-end pulse domain perception process by pixel-level pulse reception in the input layer, time-dependent pulse integration in the hidden layer, and frequency coding decision-making in the output layer. This avoids image reconstruction and feature stitching, making the computation of the neural network entirely event-driven. It realizes the layer-by-layer spatiotemporal integration and semantic extraction of pulse sequence data between the input layer, hidden layer, and output layer with pulse events as the basic unit. Finally, it outputs decision pulse data with temporal continuity, spatial localization, and confidence modulation, providing a structured pulse expression basis for low-latency and high-energy-efficiency perception decision-making.

[0084] Furthermore, the hidden layer includes multiple spiking neurons. The hidden layer is used to perform spatiotemporal pulse integration on the pulse sequence data to obtain a pulse spatial temporal representation. This includes: in the hidden layer, multiple spiking neurons are used to perform spatiotemporal pulse integration on the pulse sequence data to obtain a pulse spatial temporal representation. The multiple spiking neurons are constructed based on the leak integral firing model.

[0085] In the hidden layer, multiple spiking neurons are used to spatiotemporally integrate the pulse sequence data to obtain a pulse spatial temporal representation. These multiple spiking neurons are constructed based on a leaky integral firing model, indicating that the hidden layer consists of multiple independent spiking neurons. Each spiking neuron receives asynchronous pulse events from the input layer, and its input corresponds to the spatial position of the event camera pixel array. When a pulse event arrives, the spiking neuron accumulates its internal state according to the leaky integral firing model. This model causes the neuron's state to decay exponentially over time, triggering a pulse output only when the accumulated value exceeds a preset threshold. The spiking neurons are spatially distributed across the hidden dimension, responding to pulse events in local regions. Multiple spiking neurons collaboratively process pulse sequences from different spatial locations and timestamps. Through the temporal dependency between their state accumulation and pulse firing, the dispersed pulse events are aggregated into local activation patterns in the spatial dimension and form a continuous response sequence in the temporal dimension. The final output pulse set constitutes the pulse spatial temporal representation.

[0086] The aforementioned leakage integral output model represents a pulse calculation mechanism based on the dynamic characteristics of biological neurons. Its specific workflow is as follows: asynchronous input pulse events are injected into the neuron synapse by an event camera or upper neuron layer. The input pulse causes an instantaneous jump in the membrane potential integration unit. This integration process forms a dynamic equilibrium under the continuous exponential decay effect of the leakage decay path, so that the membrane potential exhibits a composite form of stepwise accumulation and slow decay over time. When the membrane potential reaches a preset threshold after integration accumulation, the threshold comparator triggers the pulse output trigger to generate an output pulse with standardized amplitude and width, and simultaneously activates the membrane potential reset mechanism to forcibly pull the membrane potential back to the preset reset potential. Subsequently, the neuron enters a recovery state and restarts the integration process of receiving the next round of input pulses. The above-mentioned leakage integral firing model operates entirely on an event-driven basis. When there is no input pulse, the membrane potential continuously leaks to the resting level, and the system is in a zero-power state. The entire process does not require global clock synchronization, does not rely on frame structure, and only uses hardware circuitry to realize continuous integration of membrane potential, exponential leakage, threshold judgment and state reset. This ensures that the neuron has selective response capability to the spatiotemporal distribution of input pulses, while maintaining extremely low operating power consumption and deterministic pulse firing behavior. This provides a basic, efficient and biomimetic computing unit support for the spiking neural network constructed in the embodiments of this application.

[0087] like Figure 4The diagram illustrates an optional working principle of a spiking neuron according to an embodiment of this application. Specifically, the input pulse represents an asynchronous pulse event from an event camera or fusion layer. Each pulse is input to the synaptic connection of the spiking neuron in the form of discrete spikes. When the pulse arrives, its intensity is applied to the membrane potential integration unit through a weighting coefficient, causing a momentary jump in the membrane potential. The membrane potential V(t) register is a state register whose value accumulates continuously over time and is summed by integration by an adder. Its rate of change is determined by the amplitude and frequency of the input pulse. Parallel to the integration path is a leakage decay path, which exhibits a constant-rate exponential decay process implemented by a resistor-capacitor equivalent circuit. The membrane potential slowly leaks towards zero with a fixed time constant, ensuring that the contribution of historical pulses decays over time and avoiding ineffective accumulation. This mechanism gives the neuron short-term memory characteristics, making it sensitive only to recent dense pulse sequences. The membrane potential integration result is fed into a threshold comparator, which continuously monitors whether the membrane potential has reached a preset firing threshold. This threshold is a fixed voltage value set by hardware circuitry and cannot be dynamically adjusted, ensuring the determinism and repeatability of the pulse firing behavior. When the membrane potential exceeds the threshold, the threshold comparator triggers a pulse output trigger, generating an output pulse with standardized amplitude and width. This pulse is transmitted as a signal to subsequent neurons or decoding modules. Simultaneously, a membrane potential reset mechanism is immediately activated, forcibly pulling the membrane potential state back to the initial reset potential. This reset potential is typically the resting potential or a fixed value slightly below the threshold. After the reset, the membrane potential restarts the integration process, completing one full firing cycle.

[0088] Based on the above optional embodiments, the embodiments of this application realize the generation of pulse spatial temporal representation in the hidden layer in an event-driven, spatiotemporal parallel, and frameless reconstruction manner. This enables the system to directly output spatiotemporal codes with dynamic structural integrity while maintaining the sparsity and asynchronicity of the original event stream, significantly reducing the computational redundancy and latency accumulation caused by inter-frame interpolation and feature pooling in traditional visual processing.

[0089] Furthermore, performing control operations on the vehicle based on the target perception results includes: generating vehicle decision commands based on the target perception results, wherein the vehicle decision commands are used to control multiple vehicle actuators, and the multiple vehicle actuators include at least one of the following: a braking mechanism, a steering mechanism, and a throttle mechanism; and performing control operations on the vehicle based on the vehicle decision commands.

[0090] The aforementioned generation of vehicle decision commands based on target perception results indicates that, in this embodiment, the target bounding box, target category probability, or set of future trajectory points obtained by decoding the pulse spatial temporal representation output by the spiking neural network processing unit is used as the input data source. Based on preset decision logic rules, the spatial position, speed, and category information in the target perception results are mapped into structured vehicle decision commands. These vehicle decision commands are a set of discrete control signals, containing the control quantities required to be applied to at least one actuator among the braking, steering, and throttle mechanisms. For example, a braking request signal is generated when the target distance is less than a safety threshold; a steering angle compensation command is generated when the target's lateral deviation exceeds the allowable range; and a throttle opening maintenance signal is generated when there are no obstacles ahead. This generation process does not rely on a manual rule base or complex path planning module but is directly driven by the spatiotemporal characteristics of the perception results, ensuring that the decision basis originates entirely from the output characteristics of the event-driven perception system.

[0091] The aforementioned braking mechanism, steering mechanism, and throttle mechanism represent the control objects of vehicle decision-making commands, namely the core execution units of the vehicle chassis system.

[0092] The aforementioned vehicle control operations based on vehicle decision commands mean that after receiving the vehicle decision commands, the vehicle controller converts the commands into electrical or hydraulic signals that conform to the vehicle's dynamics model. These signals directly drive the braking mechanism to generate braking force, the steering mechanism to adjust the front wheel angle, and the throttle mechanism to adjust the valve opening, enabling the vehicle to complete deceleration, steering, or acceleration actions. This control process is responded to via hard interrupts in an automotive-grade real-time operating system, ensuring that the end-to-end delay from command generation to execution does not exceed the system's designed millisecond time limit. Without introducing additional decision networks or central planning units, it achieves a direct, low-latency mapping between perception output and execution control, constructing a lightweight closed-loop control path from event stream input to physical action output.

[0093] Based on the above optional embodiments, the embodiments of this application realize the direct generation of control commands for the braking mechanism, steering mechanism and throttle mechanism and drive their execution by using the sparse, low-latency target perception results output by the spiking neural network as the only input source. This makes the vehicle control operation completely based on the output of the event-driven perception system. While maintaining control accuracy, it significantly reduces the computational complexity and response latency of the control decision link and improves the system's execution response speed and energy efficiency ratio in high-speed dynamic scenarios.

[0094] Figure 5 This is a schematic diagram of an optional vehicle control system hardware architecture according to an embodiment of this application. Specifically, the system of this embodiment consists of four core parts: an event camera module, a spiking neural network processing unit (SNN), a multi-sensor interface module, and a multi-sensor fusion and decision planning module.

[0095] The event camera module, whose physical entity is an event camera or a similar bionic vision sensor, includes a pixel-level event detection array and a built-in clock synchronization circuit. Its output is directly connected to the spiking neural network processing unit through the high-speed serial interface of Mobile Industry Processor Interface Camera Serial Interface 2 (MIPI CSI-2) to continuously capture asynchronous event stream data in the environment. The event stream data includes the pixel coordinates (x, y), polarity (positive / negative, p), and microsecond-level timestamp (t) of each event. The module has no frame rate limit and only generates data when a change in brightness is detected. Its power consumption is close to zero in static scenes, providing the system with low-latency, high-dynamic-range raw perception input.

[0096] The Spiking Neural Network (SNN) processing unit, implemented using a Field-Programmable Gate Array (FPGA), has its input physically connected to the output interface of the event camera module, receiving the raw asynchronous event stream from the event camera. This SNN comprises an input layer, hidden layers, and an output layer. The input layer is mapped and aligned one-to-one with the event camera pixels. The hidden layers consist of multiple layers of spiking neurons, primarily performing nonlinear pulse convolution and recursive computation. The output layer outputs a decision pulse sequence. Additionally, this SNN processing unit simultaneously receives pulse-coded signals from the multi-sensor interface module via an internal data bus, performs multimodal pulse fusion computation, and sends the final output pulse sequence to the multi-sensor fusion and decision planning module via a low-latency communication interface. This processing unit operates as an independent coprocessor, not relying on the main computing platform's Graphics Processing Unit (GPU) or Central Processing Unit (CPU) for core inference. Its computation process is entirely event-triggered, with zero power consumption when there are no events, significantly different from traditional frame-based convolutional neural network processing units.

[0097] A multi-sensor interface module is used to connect to traditional sensors such as global shutter cameras and millimeter-wave radars. This module includes a data preprocessing circuit and a pulse encoder. Its inputs receive image frame data from the global shutter camera via MIPI CSI-2 or Gigabit Multimedia Serial Link 2 (GMSL2) interfaces, and target point cloud data from the millimeter-wave radar via a CAN bus or Gigabit Ethernet interface. This multi-sensor interface module uses differential encoding for the image frame data, calculating the pixel difference between consecutive frames and converting pixels with changes exceeding a set threshold into timestamped pulse sequences. For radar target data, it uses a spatially distributed Poisson encoding method, mapping the distance, azimuth, and velocity of each target to the spatial region of the spiking neural network input layer. The pulse firing frequency is set according to the target velocity, outputting a pulse sequence spatiotemporally aligned with the event stream. The output pulse sequence is injected into a designated hidden layer of the spiking neural network processing unit via a parallel bus, achieving cross-modal pulse domain fusion. This transforms non-event-type sensor data into a pulse expression form isomorphic to that of the event camera, eliminating data format differences between modalities.

[0098] The multi-sensor fusion and decision-making module is directly connected to the output of the spiking neural network processing unit. Internally, it includes a pulse frequency decoder and a target trajectory mapping unit, used to convert the pulse sequence from the spiking neural network output layer into structured perception results, including target bounding box coordinates (BBox), category probability, and a set of future trajectory points. This fusion decision output module encapsulates data packets according to a preset automotive-grade communication protocol, such as ROS2 or Automotive Open System Architecture (AUTOSAR), and sends the final perception results to the main autonomous driving computing platform via CAN bus or Ethernet interface for the decision-making module to use. Its output format is a standardized message structure to ensure compatibility with existing vehicle architectures, and it controls the vehicle's control actuators to perform braking, steering, and throttle operations.

[0099] In addition, the intelligent driving domain controller receives data from event cameras and multiple sensors, and processes it in the spiking neural network processing unit (SNN) and the multi-sensor fusion and decision planning module to output structured perception results in order to control the vehicle control actuators.

[0100] The above Figure 5The hardware topology and data interaction relationships between the event camera module, the spiking neural network processing unit (SNN), the multi-sensor interface module, and the multi-sensor fusion and decision planning module are demonstrated. The core processing position of the spiking neural network processing unit in the system and the implementation path of unified encoding and fusion of multimodal data in the pulse domain are clarified, providing a clear physical carrier support for the ultra-low power consumption, low latency, event-driven dynamic sensing fusion method proposed in the embodiments of this application.

[0101] Figure 6 This is a flowchart of another optional vehicle control method according to an embodiment of this application. The specific process is as follows: First, asynchronous event stream data generated by an event camera is acquired. The event stream data includes pixel coordinates, polarity, and timestamp information. This step is continuously executed by the event camera module. Whenever a change in ambient light intensity triggers a pixel response, the sensor immediately outputs an independent event, without the need for frame synchronization or a global clock, achieving microsecond-level response.

[0102] Second, the asynchronous event stream data is preprocessed and then input into a pre-trained spiking neural network model (SNN). This model has completed weight optimization through STBP or STDP algorithms. Its input layer neurons correspond one-to-one with the event camera pixel array. The event stream directly drives the neurons to fire pulses, eliminating the need to reconstruct pseudo-images and removing computational redundancy from traditional frame-based processing.

[0103] Third, the spiking neural network model performs pulse encoding on the input event stream and performs dynamic, pulse-time-dependent feature extraction and computation through its internal spiking neuron layer. The spiking neural network model includes an input layer, hidden layers, and an output layer. The input layer receives pulses, the hidden layers perform feature extraction and fusion, and the output layer outputs decision pulses. In this step, the spiking neurons in the hidden layer adopt a leakage integral firing model, where their membrane potential accumulates with the input pulses and decays exponentially, triggering the output pulse only when a threshold is reached. Multiple hidden layer neurons work together to nonlinearly integrate the spatiotemporal pulse sequence, outputting a high-dimensional pulse spatial temporal representation that preserves the target's motion direction, velocity changes, and contour evolution features.

[0104] On the other hand, a multimodal fusion step is performed before inputting the spiking neural network model. Specifically, synchronous frame data or target point cloud data from at least one other sensor are acquired simultaneously. This data originates from a global shutter camera or millimeter-wave radar, and its timestamp is synchronized with the event stream. Furthermore, the data from the other sensors is converted into a pulse sequence spatiotemporally aligned with the event stream. Specifically, image frame data is used to generate pulses in changing regions through differential coding, and radar point cloud data is mapped to a specific region of the spiking neural network input layer based on spatial distribution Poisson coding, according to the target azimuth angle. The pulse frequency is proportional to the target velocity. Then, in a specific layer of the spiking neural network, the pulse sequence from the event stream and the pulse sequence from the other sensors are fused. The fusion method is element-level pulse superposition, and the weights are adaptively learned during the training process to achieve lightweight information complementarity in the pulse domain.

[0105] Fourth, the pulse signals output by the spiking neural network model are decoded to obtain the perception results for detecting, classifying, or tracking dynamic targets in the environment. This step converts the pulse firing rate of the output layer neurons into target confidence scores through a frequency decoder, maps the neuron index to preset anchor box coordinates, and outputs a structured target list, including bounding boxes (BBoxes), class probabilities, or sets of future trajectory points. Finally, the perception results are sent to a system-on-a-chip (SoC) to execute subsequent vehicle planning and control operations.

[0106] For example, the embodiments of this application can be adjusted in the following ways during specific implementation: First, at the model structure level, the spiking neural network (SNN) can be simplified from a double hidden layer to a single hidden layer structure. By reducing the number of neurons and computational levels, hardware resource consumption can be significantly reduced while ensuring basic target detection capabilities, making it suitable for low-order domain controllers with limited computing power. Second, at the sensor level, the event camera can be replaced with other types of biomimetic vision sensors, such as Dynamic Vision Sensors (DVS) and asynchronous optical flow sensors. Only the mapping relationship between the input layer neurons and the sensor pixel array needs to be adjusted accordingly to adapt to biomimetic input sources with different resolutions or polarity coding rules, maintaining the consistency of the system architecture. Third, at the fusion strategy level, the original element-level additive fusion can be replaced with an attention mechanism based on pulse timing. By introducing a learnable pulse attention weight module, the contribution of each modality (event, radar, image pulse) is dynamically evaluated in the pulse domain to achieve adaptive information weighted fusion, thereby further improving detection accuracy and robustness in complex interference scenarios. The above solutions do not require changes to the core idea of ​​the system's pulse domain processing and can be used independently or in combination, effectively expanding the adaptability and scalability of the embodiments of this application in different automotive-grade scenarios.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0108] According to an embodiment of this application, an optional vehicle control device embodiment is provided, such as... Figure 7 As shown, it should be noted that this device can be used to execute the above-described vehicle control method. The device includes: an acquisition module 701, used to acquire multimodal sensor data of the vehicle, wherein the multimodal sensor data includes asynchronous event data and / or synchronous perception data; a processing module 702, used to perform predictive processing on the multimodal sensor data using a target spiking neural network model to obtain a target perception result, wherein the target perception result represents the perception attribute information of the perceived object, and the perception attribute information includes at least one of the following: bounding box information, category information, and predicted trajectory information; the target spiking neural network model is obtained through iterative training using multiple sets of sample labeled data; and a control module 703, used to perform control operations on the vehicle based on the target perception result.

[0109] Furthermore, the processing module 702 is also used to: determine pulse sequence data based on multimodal sensor data, wherein the pulse sequence data is used to determine the spatiotemporal semantic features of the sensed object; perform predictive processing on the pulse sequence data using a target spiking neural network model to obtain decision pulse data, wherein the decision pulse data includes: pulse spatial distribution data and pulse frequency, the pulse spatial distribution data is used to determine the detection location and detection category of the sensed object, and the pulse frequency is used to determine the detection confidence of the sensed object; and perform decoding processing on the decision pulse data to obtain the target sensing result.

[0110] Furthermore, the processing module 702 is also configured to: preprocess the asynchronous event data to obtain a first pulse sequence, wherein the first pulse sequence is used to represent the pulse sequence after time-domain alignment and noise filtering of the asynchronous event data; in response to the multimodal sensor data containing synchronous sensing data, pulse encoding is performed on the synchronous sensing data to obtain a second pulse sequence; and pulse sequence data is determined based on the first pulse sequence and the second pulse sequence, wherein the second pulse sequence is used to represent the pulse sequence corresponding to the synchronous sensing data.

[0111] Furthermore, the processing module 702 is also configured to: determine pulse sequence data based on the first pulse sequence in response to the multimodal sensor data not containing synchronous sensing data.

[0112] Furthermore, the processing module 702 is also configured to: in response to the synchronous sensing data being image frame data, perform differential coding on the image frame data, or perform frequency coding on the image frame data to obtain a second pulse sequence; in response to the synchronous sensing data being radar point cloud data, perform Poisson coding on the radar point cloud data to obtain a second pulse sequence.

[0113] Furthermore, the target spiking neural network model includes an input layer, a hidden layer, and an output layer. The processing module 702 is also used to: receive pulse sequence data using the input layer; perform spatiotemporal pulse integration on the pulse sequence data using the hidden layer to obtain a pulse spatial temporal representation, wherein the pulse spatial temporal representation is used to represent the spatiotemporal data set of pulse events; and output decision pulse data based on the pulse spatial temporal representation using the output layer.

[0114] Furthermore, the hidden layer includes multiple spiking neurons. The processing module 702 is also used to: integrate the pulse sequence data in the hidden layer using multiple spiking neurons to obtain a pulse spatial temporal representation, wherein the multiple spiking neurons are constructed according to the leak integral firing model.

[0115] Furthermore, the control module 703 is also used to: generate vehicle decision commands based on the target perception results, wherein the vehicle decision commands are used to control multiple vehicle actuators, the multiple vehicle actuators including at least one of the following: braking mechanism, steering mechanism and throttle mechanism; and perform control operations on the vehicle based on the vehicle decision commands.

[0116] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0117] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0118] S1, acquire multimodal sensor data of the vehicle, wherein the multimodal sensor data includes: asynchronous event data and / or synchronous sensing data;

[0119] S2, the target spiking neural network model is used to predict and process the multimodal sensor data to obtain the target perception result. The target perception result is used to represent the perception attribute information of the perceived object. The perception attribute information includes at least one of the following: bounding box information, category information and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data.

[0120] S3 performs control operations on the vehicle based on the target perception results.

[0121] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0122] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0123] S1, acquire multimodal sensor data of the vehicle, wherein the multimodal sensor data includes: asynchronous event data and / or synchronous sensing data;

[0124] S2, the target spiking neural network model is used to predict and process the multimodal sensor data to obtain the target perception result. The target perception result is used to represent the perception attribute information of the perceived object. The perception attribute information includes at least one of the following: bounding box information, category information and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data.

[0125] S3 performs control operations on the vehicle based on the target perception results.

[0126] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0127] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0128] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0129] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0134] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A vehicle control method, characterized in that, include: Acquire multimodal sensor data of the vehicle, wherein the multimodal sensor data includes: asynchronous event data and / or synchronous sensing data; The multimodal sensor data is predicted using a target spiking neural network model to obtain target perception results. The target perception results are used to represent the perception attribute information of the perceived object. The perception attribute information includes at least one of the following: bounding box information, category information, and predicted trajectory information. The target spiking neural network model is obtained by iterative training using multiple sets of sample labeled data. The vehicle is controlled based on the target perception results.

2. The vehicle control method according to claim 1, characterized in that, The target perception result is obtained by using the target spiking neural network model to predict the multimodal sensor data, including: Based on the multimodal sensor data, pulse sequence data is determined, wherein the pulse sequence data is used to determine the spatiotemporal semantic features of the sensed object; The target spiking neural network model is used to predict the pulse sequence data to obtain decision pulse data, wherein the decision pulse data includes: pulse spatial distribution data and pulse frequency. The pulse spatial distribution data is used to determine the detection location and detection category of the sensing object, and the pulse frequency is used to determine the detection confidence of the sensing object. The decision pulse data is decoded to obtain the target perception result.

3. The vehicle control method according to claim 2, characterized in that, Determining the pulse sequence data based on the multimodal sensor data includes: The asynchronous event data is preprocessed to obtain a first pulse sequence, wherein the first pulse sequence is used to represent the pulse sequence after time-domain alignment and noise filtering of the asynchronous event data; In response to the multimodal sensor data including the synchronous sensing data, pulse coding is performed on the synchronous sensing data to obtain a second pulse sequence, and the pulse sequence data is determined based on the first pulse sequence and the second pulse sequence, wherein the second pulse sequence is used to represent the pulse sequence corresponding to the synchronous sensing data.

4. The vehicle control method according to claim 3, characterized in that, Determining the pulse sequence data based on the multimodal sensor data includes: In response to the fact that the multimodal sensor data does not include the synchronous sensing data, the pulse sequence data is determined based on the first pulse sequence.

5. The vehicle control method according to claim 3, characterized in that, The synchronous sensing data is pulse-coded to obtain the second pulse sequence, which includes: In response to the synchronous sensing data being image frame data, differential encoding is performed on the image frame data, or frequency encoding is performed on the image frame data to obtain the second pulse sequence; In response to the synchronous sensing data being radar point cloud data, the radar point cloud data is Poisson encoded to obtain the second pulse sequence.

6. The vehicle control method according to claim 2, characterized in that, The target spiking neural network model includes an input layer, a hidden layer, and an output layer. The target spiking neural network model is used to predict the pulse sequence data to obtain the decision pulse data, which includes: The pulse sequence data is received using the input layer; The hidden layer is used to perform spatiotemporal pulse integration on the pulse sequence data to obtain a pulse spatial temporal representation, wherein the pulse spatial temporal representation is used to represent the spatiotemporal data set of pulse events; The decision pulse data is output using the output layer based on the pulse spatial temporal representation.

7. The vehicle control method according to claim 6, characterized in that, The hidden layer includes multiple spiking neurons. The hidden layer is used to perform spatiotemporal pulse integration on the pulse sequence data to obtain the pulse spatial temporal representation, which includes: In the hidden layer, the multiple spiking neurons are used to perform spatiotemporal pulse integration on the pulse sequence data to obtain the pulse spatial temporal representation, wherein the multiple spiking neurons are constructed according to the leak integral firing model.

8. The vehicle control method according to any one of claims 1 to 7, characterized in that, Performing control operations on the vehicle based on the target perception results includes: Based on the target perception result, a vehicle decision command is generated, wherein the vehicle decision command is used to control multiple vehicle actuators, and the multiple vehicle actuators include at least one of the following: a braking mechanism, a steering mechanism, and a throttle mechanism; The vehicle is controlled based on the vehicle decision command.

9. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

Images