Aerial view vehicle instance segmentation and prediction method and system for autonomous driving

Abstract

The application relates to the technical field of automatic driving environment perception, and particularly discloses a bird's-eye view vehicle instance segmentation and prediction method and system for automatic driving, which comprises the following steps: extracting bird's-eye view features at each moment; based on the motion pose of the ego vehicle, performing spatio-temporal alignment on the bird's-eye view features at historical moments to obtain a historical bird's-eye view feature sequence; inputting the historical bird's-eye view feature sequence into a time series prediction network to output a bird's-eye view semantic segmentation map and a reverse motion flow map; for each future moment, determining vehicle foreground pixels according to the bird's-eye view semantic segmentation map of the moment, associating the foreground pixels with vehicle instances at the previous moment based on the reverse motion flow map of the moment, and generating vehicle instance segmentation results; and extracting the positions of the same vehicle instance at different future moments according to the vehicle instance segmentation results to generate future trajectory prediction of the vehicle instance. The application significantly improves the real-time processing capability and deployment feasibility of the automatic driving system.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology for autonomous driving, and specifically to a method and system for segmenting and predicting vehicle instances from a bird's-eye view for autonomous driving. Background Technology

[0002] With the development of autonomous driving technology, environmental perception systems typically adopt a modular design, breaking down the overall task into multiple independent modules such as target detection, map building, target tracking, and trajectory prediction. While this division of labor facilitates system development, it also leads to the problem of perception errors being transmitted and accumulated between modules. Furthermore, the trajectory prediction module heavily relies on high-precision maps for prior information, which imposes high costs and limitations on scenario adaptability in practical applications.

[0003] To overcome the aforementioned shortcomings, joint perception and prediction methods based on bird's-eye view have emerged. These methods, through shared feature computation and joint learning, directly output vehicle instance segments and their future motion states from sensor data end-to-end, effectively reducing error accumulation and decreasing reliance on high-precision maps. However, existing joint perception and prediction models generally suffer from large numbers of model parameters, high computational complexity, and slow inference speed, making it difficult to meet the stringent requirements of lightweight deployment and real-time response in practical autonomous driving systems. Therefore, developing a lightweight joint perception and prediction model that significantly improves efficiency while maintaining accuracy has become a crucial problem urgently needing to be solved in this field. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to provide a method and system for segmenting and predicting vehicle instances from a bird's-eye view for autonomous driving, thereby solving the technical challenges of large model size and slow inference speed in existing joint perception and prediction methods.

[0005] This invention provides a method for segmenting and predicting vehicle instances from bird's-eye view images for autonomous driving, comprising: Acquire vehicle perception data at the current moment and at least one historical moment; The vehicle perception data is input into a semantic segmentation network to extract bird's-eye view features at each time point; Based on the vehicle's motion pose, the bird's-eye view features from historical moments are spatiotemporally aligned and unified to the bird's-eye view coordinate system of the current moment, resulting in an aligned sequence of historical bird's-eye view features. The historical bird's-eye view feature sequence is input into a temporal prediction network, which outputs in parallel semantic segmentation maps and reverse motion flow maps of bird's-eye views at multiple future time points; wherein, the reverse motion flow map represents the displacement vector of each foreground pixel in the bird's-eye view at a future time point to the position of its corresponding vehicle instance at the previous time point. For each future time step, the vehicle foreground pixels are determined based on the bird's-eye view semantic segmentation map at that time step, and the foreground pixels are associated with the vehicle instance at the previous time step based on the reverse motion flow graph at that time step, generating a vehicle instance segmentation result with temporal consistency. Based on the vehicle instance segmentation results, the positions of the same vehicle instance at different future times are extracted, and the future trajectory prediction of the vehicle instance is generated.

[0006] In one possible implementation, the vehicle perception data is a sequence of images captured by a monocular camera.

[0007] In one possible implementation, the semantic segmentation network is a pyramid occupancy network; the step of inputting the vehicle perception data into the semantic segmentation network to extract the bird's-eye view features at each time moment includes: Extract multi-scale image features from the vehicle perception data; Based on camera geometry, the multi-scale image features are transformed into a bird's-eye view space to obtain initial bird's-eye view features; The initial bird's-eye view features are encoded to obtain bird's-eye view features containing semantic information.

[0008] In one possible implementation, the step of aligning the bird's-eye view features from historical moments based on the vehicle's motion pose, and unifying them to the current bird's-eye view coordinate system, results in an aligned sequence of historical bird's-eye view features, including: Based on the 6-DOF pose vector sequence in the vehicle's motion pose, calculate the coordinate transformation matrix from each historical moment to the current moment; Using the coordinate transformation matrix, the bird's-eye view features at each historical moment are resampled and mapped to the bird's-eye view coordinate system at the current moment to obtain an aligned sequence of historical bird's-eye view features.

[0009] In one possible implementation, the temporal prediction network is a convolutional neural network based on an encoder-predictor-decoder structure; the step of inputting the historical bird's-eye view feature sequence into the temporal prediction network, and having the temporal prediction network output in parallel semantic segmentation maps and reverse motion flow maps of bird's-eye views at multiple future time points, includes: The historical bird's-eye view feature sequence is input into a temporal prediction network, and the encoder performs spatial downsampling on the historical bird's-eye view feature sequence to extract multi-scale spatiotemporal features. The predictor maps the multi-scale spatiotemporal features from the historical time dimension to the future time dimension to obtain the mapped multi-scale spatiotemporal features. The decoder performs spatial upsampling on the mapped multi-scale spatiotemporal features to reconstruct the spatial resolution of the original input, and outputs the bird's-eye view semantic segmentation map and the reverse motion flow map respectively.

[0010] In one possible implementation, the encoder, predictor, and decoder together constitute an improved U-Net architecture, and the bird's-eye view semantic segmentation graph and the inverse motion flow graph are output in parallel by two independent branches.

[0011] In one possible implementation, associating the foreground pixel with the vehicle instance from the previous time step to generate a vehicle instance segmentation result with temporal consistency includes: The foreground pixel is associated with the vehicle instance from the previous time step according to the following formula: in, express Every foreground pixel in the bird's-eye view at any time Time-corresponding vehicle instance The displacement vector at the average center position, Represents a vehicle instance At any moment The average center position, express Forward pixels in the bird's-eye view at any moment coordinates Indicates time Forward pixels in bird's-eye view coordinates Indicates in Vehicle instance segmentation results at time point. Represents the coordinates of the foreground pixel.

[0012] In one possible implementation, the step of extracting the positions of the same vehicle instance at different future times based on the vehicle instance segmentation result and generating a future trajectory prediction for the vehicle instance includes: Calculate the centroid positions of the instance mask of the same vehicle instance at various future times, and connect the centroid positions in chronological order to form the future trajectory prediction of the vehicle instance.

[0013] In one possible implementation, the semantic segmentation network is trained based on a semantic segmentation task; With the parameters of the pre-trained semantic segmentation network fixed, the temporal prediction network is trained based on the bird's-eye view feature sequences of historical and future times.

[0014] This invention also provides a high-efficiency, lightweight bird's-eye view vehicle instance segmentation and prediction system for autonomous driving, applicable to any of the methods described, including: The acquisition module is used to acquire vehicle perception data at the current time and at least one historical time. The extraction module is used to input the vehicle perception data into the semantic segmentation network and extract the bird's-eye view features at each time point; The alignment module is used to perform spatiotemporal alignment of the bird's-eye view features from historical moments based on the vehicle's motion pose, unifying them to the bird's-eye view coordinate system of the current moment, and obtaining the aligned historical bird's-eye view feature sequence. The prediction module is used to input the historical bird's-eye view feature sequence into the temporal prediction network, and the temporal prediction network outputs in parallel the semantic segmentation map and the reverse motion flow map of the bird's-eye view at multiple future times; wherein, the reverse motion flow map represents the displacement vector of each foreground pixel in the bird's-eye view at a future time to the position of its corresponding vehicle instance at the previous time. The association module is used to determine the vehicle foreground pixels based on the bird's-eye view semantic segmentation map at each future time, and associate the foreground pixels with the vehicle instance at the previous time based on the reverse motion flow graph at that time, so as to generate a vehicle instance segmentation result with temporal consistency. The generation module is used to extract the positions of the same vehicle instance at different future times based on the vehicle instance segmentation results, and generate the future trajectory prediction of the vehicle instance.

[0015] The method and system for bird's-eye view vehicle instance segmentation and prediction for autonomous driving provided by this invention achieves approximately twice the inference speed and half the number of parameters on the nuScenes dataset by optimizing the network structure and training strategy. At the same time, it maintains good performance in indicators such as vehicle segmentation intersection-union ratio and instance tracking quality, significantly improving the real-time processing capability and deployment feasibility of autonomous driving systems. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a bird's-eye view vehicle instance segmentation and prediction method for autonomous driving provided as an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of the joint sensing and prediction model provided in an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of the present invention by way of example, but should not be used to limit the scope of the present invention. That is, the present invention is not limited to the described preferred embodiments, and the scope of the present invention is defined by the claims.

[0018] In the description of this invention, it should be noted that, unless otherwise stated, "a plurality of" means two or more; the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance; those skilled in the art can understand the specific meaning of the above terms in this invention as appropriate.

[0019] Figure 1 A flowchart illustrating a bird's-eye view vehicle instance segmentation and prediction method for autonomous driving, provided as an embodiment of the present invention, is shown below. Figure 1 As shown, this invention provides a method for bird's-eye view vehicle instance segmentation and prediction for autonomous driving. This method mainly includes core steps such as bird's-eye view (BEV) feature extraction, spatiotemporal feature alignment, multi-scale temporal prediction, and instance association and trajectory generation, including: Step S1: Obtain vehicle perception data at the current time and at least one historical time. In one possible implementation, the vehicle perception data is a sequence of images captured by a monocular camera.

[0020] It is understood that in other embodiments, the sensing data may also come from multi-modal sensors such as multi-view cameras, lidar, and millimeter-wave radar, or their fused data.

[0021] Step S2: Input the vehicle perception data into the semantic segmentation network to extract the bird's-eye view features at each time point; In one possible implementation, the semantic segmentation network is a pyramid occupancy network; multi-scale image features are extracted from vehicle perception data; based on camera geometry, the multi-scale image features are transformed into a bird's-eye view space to obtain initial bird's-eye view features; the initial bird's-eye view features are encoded to obtain bird's-eye view features containing semantic information. Here, the bird's-eye view is an abstract representation of a scene viewed from above, unifying perceptual information from different perspectives into a common coordinate system conducive to planning and decision-making.

[0022] In one possible implementation, a semantic segmentation network based on a bird's-eye view is built using a Pyramid Occupancy Network (PON) to obtain semantic segmentation results for bird's-eye views with multiple labels, preparing for subsequent temporal prediction. The Pyramid Occupancy Network is used to predict the occupancy probability of spatial grids in a bird's-eye view from monocular images.

[0023] In the PON model, the bird's-eye view is presented in the form of a semantically occupied grid map, occupying each spatial location in the grid map. They all have a related state. , This indicates that the space is occupied. Empty. In reality, the true state of space is unknown, therefore... Treat it as a random variable, and take the observation at the current moment as the basis. To conditionally estimate the occupancy probability Extending further to the semantic occupancy grid, then... The state indicates whether a cell exists or does not exist in a given grid cell. Objects of a class. In this setting, the same grid cell can have multiple types of occupancy, such as roads, intersections, and vehicle classes coexisting at a given location. The PON model uses the imaging relationships of camera geometry and a fully connected network for inference, mapping features from the image to the bird's-eye view space, predicting the occupancy probability of the raster in the bird's-eye view space from the monocular image. The model's input is a monocular camera image, and the output is a 2D raster representation with multi-class labels (predicting a set of multi-class binary labels for each location in the bird's-eye view). The PON model can be represented as... Its structure mainly consists of a multi-scale image feature extraction module, a multi-scale feature-to-bird's-eye view spatial transformation module, and a bird's-eye view feature processing module, as detailed below: (1) The multi-scale image feature extraction module is implemented using a pre-trained FPN-ResNet50 network. The pre-trained ResNet-50 network is used to extract features from the image at multiple scales. Then, using a feature pyramid network structure, starting with the feature map at the highest level (smallest scale), it is fused with feature maps of the corresponding scales through upsampling and lateral connections. This ensures that each feature map contains both high-level semantic information and low-level detailed information, ultimately resulting in a set of fused multi-scale features with strong representational capabilities. .

[0024] (2) The transformation of multi-scale features to the bird's-eye view space is achieved through a dense feature transformation network, Dense-Transform. Due to occlusion, lack of depth information, and unknown ground topology, a large amount of vertical contextual information is needed to map features to the bird's-eye view space. In the horizontal direction, the relationship between the bird's-eye view position and the image position can be determined by camera geometry (camera intrinsics). In the vertical direction, the network needs to learn complex context to infer depth. Therefore, in the dense feature transformation network, the image feature size is first compressed along the vertical direction while retaining the horizontal dimension. Then, 1D convolution is applied along the horizontal axis to reshape the feature mapping. Finally, based on perspective, a set of features is predicted along the depth axis in the polar coordinate system using known camera intrinsics (focal length). and optical axis The projected horizontal coordinates are resampled to a Cartesian coordinate system to obtain the bird's-eye view features.

[0025] In this network, image features at different scales correspond to different distances in the bird's-eye view space. This approach is based on the relationship between receptive fields at different scales and object sizes, and also stems from geometric intuition. The resampling step specifically involves generating a regular [database] on the bird's-eye view plane using camera intrinsics. The coordinate grid is generated, and then the coordinates of the grid are mapped to the corresponding image coordinates on the camera imaging plane through perspective projection transformation. Finally, PyTorch's built-in grid sampling operation is used to sample and rearrange pixels from the camera feature map according to these image coordinates, thereby obtaining the feature representation of the bird's-eye view space.

[0026] (3) The bird's-eye view feature processing module is a top-down feature extraction network composed of multiple ResNet-Bottleneck blocks. It extracts more abstract and semantic feature representations by gradually reducing the spatial resolution of the feature map and increasing the number of channels, thereby obtaining the required bird's-eye view features. .

[0027] Step S3: Based on the vehicle's motion pose, the bird's-eye view features from historical moments are spatiotemporally aligned and unified to the bird's-eye view coordinate system of the current moment to obtain the aligned historical bird's-eye view feature sequence. In one possible implementation, since each moment acquires bird's-eye view features based on the current location... Therefore, before inputting the spatiotemporal prediction model, it should be uniformly aligned to the current bird's-eye view space. The predicted future bird's-eye view segmentation and motion flow results are also based on the current bird's-eye view space. In this invention, based on the 6-DOF pose vector sequence in the vehicle's motion pose, the coordinate transformation matrix from each historical moment to the current moment is calculated. Using the coordinate transformation matrix, the bird's-eye view features of each historical moment are resampled and mapped to the current bird's-eye view coordinate system to obtain the aligned historical bird's-eye view feature sequence. This step eliminates the scene appearance offset caused by the vehicle's motion, providing a stable and consistent spatiotemporal context for the subsequent temporal prediction module, which is key to improving prediction accuracy.

[0028] Specifically, the correspondence between different bird's-eye view spatial positions is obtained through coordinate transformation of the vehicle's own motion (a 6-DOF pose vector sequence), mapping the spatial features of the bird's-eye view at historical moments to the bird's-eye view space at the current moment, represented as... For each historical moment, the rotation and translation information of the bird's-eye view space from that historical moment to the current moment is first extracted from the coordinate transformation of the vehicle's own motion, forming an affine transformation matrix. Then, the transformed grid space is obtained using the built-in PyTorch affine_grid function, and the bird's-eye view features are sampled into the transformed grid space using the grid_sample function to obtain the transformed bird's-eye view features.

[0029] Step S4: Input the historical bird's-eye view feature sequence into the temporal prediction network, and the temporal prediction network outputs the semantic segmentation map and reverse motion flow map of the bird's-eye view at multiple future times in parallel. The reverse motion flow graph represents the displacement vector of each foreground pixel in the bird's-eye view at a future moment to the position of its corresponding vehicle instance at the previous moment.

[0030] In one possible implementation, a sequence of aerial view features from multiple historical moments within a unified space is obtained. Subsequently, a multi-scale temporal prediction module predicts future bird's-eye view vehicle semantic segmentation maps and reverse motion flows, respectively. Historical bird's-eye view feature sequences are input into the temporal prediction network, and an encoder performs spatial downsampling on these sequences to extract multi-scale spatiotemporal features. The predictor then maps these multi-scale spatiotemporal features from the historical time dimension to the future time dimension, that is, it maps the features of each scale from... Mapped to The system obtains the mapped multi-scale spatiotemporal features, enabling reasoning about future states. The decoder then spatially upsamples these features to reconstruct the spatial resolution of the original input, outputting semantic segmentation maps of bird's-eye views at multiple future time points. (Representing the probability that each position is a vehicle) and reverse motion flow graph (A two-dimensional vector field).

[0031] In one possible implementation, the temporal prediction network is a convolutional neural network based on an encoder-predictor-decoder structure. The encoder, predictor, and decoder together form an improved U-Net architecture, which takes the bird's-eye view feature sequence as input and consists only of 2D convolutional networks. This architecture fully utilizes the skip connections of the encoder-decoder structure to fuse low-level details and high-level semantic information. Furthermore, the bird's-eye view semantic segmentation map and the inverse motion flow map are output in parallel by two independent branches, and the network weights of the two branches are not shared, so as to specialize the learning of their respective tasks.

[0032] Step S5: For each future time, determine the vehicle foreground pixels based on the bird's-eye view semantic segmentation map at that time, and associate the foreground pixels with the vehicle instance at the previous time based on the reverse motion flow graph at that time to generate a vehicle instance segmentation result with temporal consistency. This is a key step in the invention, realizing the transformation from frame-by-frame semantic segmentation to cross-frame instance trajectories. After obtaining the semantic segmentation map and inverse motion flow graph for future timeframes, the semantic segmentation map is first post-processed, such as thresholding, to obtain a binarized vehicle foreground mask. Then, using the inverse motion flow graph, a vehicle instance ID is assigned to each foreground pixel in the future frame through a mechanism called "instance inverse motion flow association," ensuring that this ID is the same as the vehicle instance ID it belonged to in the historical timeframe, thereby achieving cross-frame vehicle instance tracking.

[0033] The foreground pixel is associated with the vehicle instance from the previous time step according to the following formula: in, express Every foreground pixel in the bird's-eye view at any time Time-corresponding vehicle instance The displacement vector at the average center position, Represents a vehicle instance At any moment The average center position, express Forward pixels in the bird's-eye view at any moment coordinates Indicates time Forward pixels in bird's-eye view coordinates Indicates in Vehicle instance segmentation results at time point. Represents the coordinates of the foreground pixel.

[0034] By using the reverse motion flow of vehicle instances, the vehicle instance is directly propagated from the pixel at the target position of the flow vector in the previous frame to the current frame. Using this method, the vehicle instance for each pixel is assigned individually, generating pixel-level ID associations. This reverse motion flow association mechanism can more effectively accommodate prediction errors and generate continuous trajectories. Because adjacent grid cells around the center tend to share the same ID, errors often occur in peripheral single pixels.

[0035] Step S6: Based on the vehicle instance segmentation results, extract the positions of the same vehicle instance at different future times and generate the future trajectory prediction of the vehicle instance.

[0036] In one possible implementation, such as Figure 2As shown, by using reverse flow deformation, multiple future positions can be associated with the same instance in the previous frame. For successfully matched instances, the centroid positions of the instance mask of the same vehicle instance at various future times are calculated, and the centroid positions are connected in chronological order to form the future trajectory prediction of that vehicle instance. In this way, the system ultimately outputs the future motion trajectory of each detected vehicle, rather than isolated frame-by-frame detection boxes.

[0037] In one possible implementation, the training process of this invention employs a two-stage training strategy. In the first stage, a semantic segmentation network is trained based on a semantic segmentation task, enabling it to acquire strong feature extraction capabilities from images to bird's-eye view space. In the second stage, with the parameters of the trained semantic segmentation network fixed, a temporal prediction network is specifically trained using supervised learning, based on sample pairs constructed from bird's-eye view feature sequences from historical and future times. This decoupled training strategy allows for separate optimization of complex feature learning and temporal relationship learning, reducing the difficulty of joint training and making the final model easier to converge and deploy in a lightweight manner.

[0038] In one possible implementation, instance segmentation and tracking quality can be measured by Video Panoptic Quality (VPQ), and segmentation prediction accuracy can be measured by Intersection over Union (IoU).

[0039] This invention also provides a high-efficiency, lightweight bird's-eye view vehicle instance segmentation and prediction system for autonomous driving, applicable to any of the methods described, including: The acquisition module is used to acquire vehicle perception data at the current time and at least one historical time. The extraction module is used to input vehicle perception data into the semantic segmentation network and extract the bird's-eye view features at each time point; The alignment module is used to perform spatiotemporal alignment of the bird's-eye view features from historical moments based on the vehicle's motion pose, unifying them to the bird's-eye view coordinate system of the current moment, and obtaining the aligned historical bird's-eye view feature sequence. The prediction module is used to input the feature sequence of historical bird's-eye view into the temporal prediction network, which outputs the semantic segmentation map and the reverse motion flow map of the bird's-eye view at multiple future time points in parallel. The reverse motion flow map represents the displacement vector of each foreground pixel in the bird's-eye view at a future time point to the position of the corresponding vehicle instance at the previous time point. The association module is used to determine the vehicle foreground pixels based on the bird's-eye view semantic segmentation map at each future time, and associate the foreground pixels with the vehicle instances at the previous time based on the reverse motion flow graph at that time, so as to generate vehicle instance segmentation results with temporal consistency. The generation module is used to extract the positions of the same vehicle instance at different future times based on the vehicle instance segmentation results, and generate the future trajectory prediction of the vehicle instance.

[0040] Combination Figure 2 The method of this invention is described below. First, a monocular image is input, and pre-trained bird's-eye view features are obtained through a bird's-eye view semantic segmentation model. Then, the bird's-eye view features from different historical moments are aligned to the current bird's-eye view space and input into a multi-scale temporal prediction module to predict the semantic segmentation and vehicle motion flow of future bird's-eye view vehicles. The semantic segmentation map provides the cognition of "where the car is," while the reverse motion flow provides the association clue of "where the car came from." Finally, the two results are processed to obtain the trajectory prediction of vehicle instance segmentation over a future period. The multi-scale temporal prediction module is designed with two branches, using the same prediction network architecture, to predict the semantic segmentation results and motion flow of future bird's-eye view vehicles, respectively. The representation of the bird's-eye view vehicle semantic segmentation is the same as that obtained during the first stage of training, while the motion flow refers to the displacement vector of dynamic vehicles. Here, the reverse flow from the next frame to the previous frame is used, that is, the displacement vector of each foreground pixel (predicted to be occupied by a vehicle in the bird's-eye view raster) at a certain time point to the same instance vehicle at the previous time point. This approach allows each occupied grid cell to be directly associated with a vehicle instance in the previous frame, thereby extracting future motion trajectories.

[0041] The core of this invention lies in constructing an end-to-end joint perception and prediction model using a two-stage training strategy. The first stage uses a pyramid occupancy network based on semi-supervised learning to extract semantic features from the bird's-eye view. Through a student-teacher network architecture and multiple loss functions, a bird's-eye view feature representation containing rich scene information is obtained. The second stage employs an improved U-Net architecture to construct a temporal prediction network. This network first aligns the bird's-eye view features from historical moments spatiotemporally based on vehicle motion. Then, through an encoder-predictor-decoder structure, it predicts future vehicle semantic segmentation maps and reverse motion flows, respectively. Finally, it achieves pixel-level ID association through the instance reverse motion flow, generating temporally consistent instance trajectory predictions.

[0042] The efficient and lightweight bird's-eye view vehicle instance segmentation and prediction method and system for autonomous driving provided by this invention have the following beneficial effects: 1) Significantly Improved Efficiency and Lightweight Design: This invention employs a two-stage training strategy (pre-training for bird's-eye view feature extraction followed by training a lightweight temporal prediction network) and an optimized network architecture (such as the improved U-Net). On public datasets, this achieves a substantial reduction in model parameters (approximately half) and a significant increase in inference speed (approximately 2x), effectively addressing the problems of heavy computational burden and poor real-time performance in existing joint models. The two-stage strategy decouples complex feature learning from dynamic prediction, allowing the prediction network to be designed to be more lightweight without having to learn visual feature extraction from scratch.

[0043] 2) Guarantee and improve prediction accuracy: By introducing a spatiotemporal alignment module based on vehicle motion, historical BEV features are unified to the current coordinate system, eliminating the perspective shift caused by vehicle motion, providing more accurate and consistent contextual information for time series prediction, thereby significantly improving the accuracy of future motion prediction.

[0044] 3) Enhanced trajectory continuity and stability: An innovative reverse motion flow approach is used for vehicle instance association. This method achieves pixel matching by predicting the displacement "backtracking" from the current frame to the previous frame. This mechanism better accommodates prediction errors (since errors often occur around instances), ensuring that the same vehicle instance is stably and continuously tracked across consecutive frames, thus generating smoother and more reliable future trajectories.

[0045] 4) Achieving end-to-end lightweight joint learning: This invention solves vehicle instance segmentation and trajectory prediction tasks in a unified framework. By using a shared lightweight BEV feature extraction and prediction network, it avoids error accumulation in traditional modular pipelines, reduces dependence on high-precision maps, and is more suitable for deployment in actual autonomous driving systems.

[0046] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0047] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0048] The methods and apparatus provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0049] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "a specific embodiment" or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for segmenting and predicting vehicle instances from bird's-eye view images for autonomous driving, characterized in that, include: Acquire vehicle perception data at the current moment and at least one historical moment; The vehicle perception data is input into a semantic segmentation network to extract bird's-eye view features at each time point; Based on the vehicle's motion pose, the bird's-eye view features from historical moments are spatiotemporally aligned and unified to the bird's-eye view coordinate system of the current moment, resulting in an aligned sequence of historical bird's-eye view features. The historical bird's-eye view feature sequence is input into a temporal prediction network, which outputs in parallel semantic segmentation maps and reverse motion flow maps of bird's-eye views at multiple future time points; wherein, the reverse motion flow map represents the displacement vector of each foreground pixel in the bird's-eye view at a future time point to the position of its corresponding vehicle instance at the previous time point. For each future time step, the vehicle foreground pixels are determined based on the bird's-eye view semantic segmentation map at that time step, and the foreground pixels are associated with the vehicle instance at the previous time step based on the reverse motion flow graph at that time step, generating a vehicle instance segmentation result with temporal consistency. Based on the vehicle instance segmentation results, the positions of the same vehicle instance at different future times are extracted, and the future trajectory prediction of the vehicle instance is generated.

2. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, The vehicle perception data is an image sequence captured by a monocular camera.

3. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, The semantic segmentation network is a pyramid occupancy network; the process of inputting the vehicle perception data into the semantic segmentation network to extract the bird's-eye view features at each time step includes: Extract multi-scale image features from the vehicle perception data; Based on camera geometry, the multi-scale image features are transformed into a bird's-eye view space to obtain initial bird's-eye view features; The initial bird's-eye view features are encoded to obtain bird's-eye view features containing semantic information.

4. The method for segmenting and predicting vehicle instances from a bird's-eye view for autonomous driving according to claim 1, characterized in that, The process of aligning historical bird's-eye view features based on the vehicle's motion pose to the current bird's-eye view coordinate system, resulting in an aligned sequence of historical bird's-eye view features, includes: Based on the 6-DOF pose vector sequence in the vehicle's motion pose, calculate the coordinate transformation matrix from each historical moment to the current moment; Using the coordinate transformation matrix, the bird's-eye view features at each historical moment are resampled and mapped to the bird's-eye view coordinate system at the current moment to obtain an aligned sequence of historical bird's-eye view features.

5. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, The temporal prediction network is a convolutional neural network based on an encoder-predictor-decoder structure; the step of inputting the historical bird's-eye view feature sequence into the temporal prediction network, and having the temporal prediction network output bird's-eye view semantic segmentation maps and reverse motion flow maps for multiple future time moments in parallel includes: The historical bird's-eye view feature sequence is input into a temporal prediction network, and the encoder performs spatial downsampling on the historical bird's-eye view feature sequence to extract multi-scale spatiotemporal features. The predictor maps the multi-scale spatiotemporal features from the historical time dimension to the future time dimension to obtain the mapped multi-scale spatiotemporal features. The decoder performs spatial upsampling on the mapped multi-scale spatiotemporal features to reconstruct the spatial resolution of the original input, and outputs the bird's-eye view semantic segmentation map and the reverse motion flow map respectively.

6. The method for segmenting and predicting vehicle instances from a bird's-eye view for autonomous driving according to claim 5, characterized in that, The encoder, predictor, and decoder together constitute an improved U-Net architecture, and the bird's-eye view semantic segmentation graph and the inverse motion flow graph are output in parallel by two independent branches.

7. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, The step of associating the foreground pixels with the vehicle instance from the previous time step to generate a vehicle instance segmentation result with temporal consistency includes: The foreground pixel is associated with the vehicle instance from the previous time step according to the following formula: in, express Every foreground pixel in the bird's-eye view at any time Time-corresponding vehicle instance The displacement vector at the average center position, Represents a vehicle instance At any moment The average center position, express Forward pixels in the bird's-eye view at any moment coordinates Indicates time Forward pixels in bird's-eye view coordinates Indicates in Vehicle instance segmentation results at time point. Represents the coordinates of the foreground pixel.

8. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, The step of extracting the positions of the same vehicle instance at different future times based on the vehicle instance segmentation result and generating a future trajectory prediction for the vehicle instance includes: Calculate the centroid positions of the instance mask of the same vehicle instance at various future times, and connect the centroid positions in chronological order to form the future trajectory prediction of the vehicle instance.

9. The method for segmenting and predicting vehicle instances from bird's-eye view for autonomous driving according to claim 1, characterized in that, Also includes: The semantic segmentation network is trained based on the semantic segmentation task; With the parameters of the pre-trained semantic segmentation network fixed, the temporal prediction network is trained based on the bird's-eye view feature sequences of historical and future times.

10. A bird's-eye view vehicle instance segmentation and prediction system for autonomous driving, applied to the method described in any one of claims 1-9, characterized in that, include: The acquisition module is used to acquire vehicle perception data at the current time and at least one historical time. The extraction module is used to input the vehicle perception data into the semantic segmentation network and extract the bird's-eye view features at each time point; The alignment module is used to perform spatiotemporal alignment of the bird's-eye view features from historical moments based on the vehicle's motion pose, unifying them to the bird's-eye view coordinate system of the current moment, and obtaining the aligned historical bird's-eye view feature sequence. The prediction module is used to input the historical bird's-eye view feature sequence into the temporal prediction network, and the temporal prediction network outputs in parallel the semantic segmentation map and the reverse motion flow map of the bird's-eye view at multiple future times; wherein, the reverse motion flow map represents the displacement vector of each foreground pixel in the bird's-eye view at a future time to the position of its corresponding vehicle instance at the previous time. The association module is used to determine the vehicle foreground pixels based on the bird's-eye view semantic segmentation map at each future time, and associate the foreground pixels with the vehicle instance at the previous time based on the reverse motion flow graph at that time, so as to generate a vehicle instance segmentation result with temporal consistency. The generation module is used to extract the positions of the same vehicle instance at different future times based on the vehicle instance segmentation results, and generate the future trajectory prediction of the vehicle instance.

Images