A multi-frame spatio-temporal correlation-based surround-view fisheye image bird's-eye view segmentation method

Abstract

The application discloses a kind of based on multi-frame space-time association's look-around fisheye image bird's eye view segmentation method, comprising: obtaining look-around RGB image;Image features are extracted using feature extraction network, and image feature set is constructed;Multi-time image features are aggregated into unified bird's eye view space using space-time synchronous encoder, and the BEV feature of current time is generated;Through area-level contrast learning with ground truth, the semantic consistency and space-time robustness of BEV feature are optimized;The optimized BEV feature is input into double-branch stage supervision decoder, respectively through attention branch and convolution branch for semantic mapping and feature reconstruction, and high-precision BEV segmentation map is fused and generated.The method can effectively solve the problems of serious fisheye image distortion, time sequence alignment difficulty and inconsistent feature semantics, and realize high-precision, space-time consistent bird's eye view semantic segmentation in complex scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a kind of bird's eye view segmentation methods for surround-view fisheye images based on multi-frame space-time correlation. BACKGROUND

[0002] In the field of automation and intelligent assembly, especially in the context of large aircraft assembly, the accuracy of the environmental perception system has a significant impact on the safety and efficiency of the operation. With the increasing complexity of the airport and large aircraft assembly process, intelligent transportation systems and automated equipment, such as baggage carts, tow trucks, aircraft tow trucks, automated swing arms, and assembly robots, are gradually becoming popular. These devices need to rely on surround-view camera systems to perceive the surrounding environment in real-time during ground transportation and assembly operations to ensure accurate navigation and position control, thereby achieving efficient and safe collaborative operations.

[0003] Bird's eye view (BEV) segmentation plays a crucial role in environmental perception, providing a unified spatial representation for downstream tasks such as path planning, navigation, collision detection, and operation scheduling. In practical applications, surround-view fisheye cameras have become an important sensing unit for environmental perception systems due to their wide field of view, low cost, and ease of installation. However, due to the inherent field of view distortion and non-linear mapping characteristics of fisheye lenses, the images captured by them exhibit significant geometric distortion, which poses a significant challenge for accurate BEV segmentation.

[0004] In the prior art, CN119723090A discloses a multi-view joint semantic segmentation method for surround-view fisheye images, the construction and training process of the multi-view joint segmentation model, including: preparing a training data set, taking fisheye images from multiple views at the same time as a set of surround-view images, the training data set includes multiple sets of surround-view images and corresponding pixel-level semantic annotations; establishing a multi-view joint segmentation model based on real-time semantic segmentation, including a feature extraction module, a distortion detail supplement module, a multi-view context fusion module, and a decoder module; training the multi-view joint segmentation model based on the training data set and cross-entropy loss.

[0005] However, this method has the following limitations: 1. During the projection process, due to the simplification of the camera model or the existence of calibration errors, the target object often appears misaligned or deformed in the BEV space, affecting the accuracy of downstream tasks; 2. With the movement of the device in the environment, frequent self-motion causes significant temporal and perspective differences between historical frames and the current frame, making it difficult for single-frame methods to utilize cross-temporal information to ensure the consistency of BEV features; 3. Most existing BEV segmentation methods rely only on the supervision signal of the final output (such as cross-entropy loss on the segmentation map), lacking effective guidance for intermediate BEV features, resulting in inconsistent feature semantics and limited prediction accuracy.

[0006] Therefore, how to obtain high-precision, geometrically consistent BEV segmentation features under the conditions of considering fisheye distortion, equipment self-motion, and spatiotemporal feature consistency has become a technical problem that urgently needs to be solved in the field of intelligent assembly and environmental perception. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation.

[0008] The objective of this invention can be achieved through the following technical solutions:

[0009] This invention provides a method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation, comprising the following steps:

[0010] The system uses a surround-view camera system to acquire multi-time, multi-view surround-view RGB images of the aircraft's ground environment.

[0011] A feature extraction network is used to extract image features from the panoramic RGB image to generate a multi-time-and-view image feature set.

[0012] The image feature set is aggregated into a unified bird's-eye view space using a spatiotemporal synchronous encoder to obtain the BEV features at the current moment;

[0013] Based on the current BEV features and ground truth, a regional-level comparative learning is performed to optimize the current BEV features.

[0014] The optimized BEV features at the current time step are input into a two-branch stage-supervised decoder, and semantic mapping and feature reconstruction are performed through the attention branch and the convolution branch respectively. The results are then fused to generate a BEV segmentation map.

[0015] Furthermore, the surround-view camera system includes four wide-angle fisheye cameras located at the front, rear, left, and right of the aircraft, respectively.

[0016] Furthermore, the feature extraction network is a convolutional neural network (CNN).

[0017] Furthermore, the image feature set includes panoramic RGB images captured by each wide-angle fisheye camera at various times.

[0018] Furthermore, the step of aggregating the image feature set into a unified bird's-eye view space using a spatiotemporal synchronous encoder to obtain the BEV features at the current moment specifically includes:

[0019] Initialize BEV query vector set Each query Corresponding bird's-eye view space One of the BEV mesh cells, Represents bird's-eye view space BEV mesh unit The corresponding query vector;

[0020] Bird's-eye view space Each BEV mesh cell in the graph is mapped to multiple preset heights, forming a 3D reference point at the current moment. The formula is:

[0021]

[0022]

[0023] in, Indicates the current time At the preset height 3D reference point below, , This indicates the number of BEV grid cells in the ground coordinate system, i.e., the resolution of the BEV grid in the horizontal and vertical directions. This represents the side length of the actual region represented by each BEV mesh cell. , For a preset set of height values; , Represents BEV mesh element The corresponding coordinates in the real-world coordinate system; Represents BEV mesh element Two-dimensional coordinates;

[0024] Based on camera pose and self-motion compensation, the current moment 3D reference point below Transformation into the historical coordinate system through relative self-motion In the process, obtain the anchor point in the historical coordinate system. :

[0025]

[0026] in, Representing historical moments Anchor point in the self-coordinate system; Representing historical moments With the current moment The coordinate transformation matrix between them Representing historical moments The homogeneous transformation matrix from the world coordinate system to the self coordinate system. Indicates the current time The homogeneous transformation matrix from the world coordinate system to the self coordinate system;

[0027] Anchor point in the historical coordinate system Mapping to historical moments t The next i In the coordinate system of a wide-angle fisheye camera, obtain the 3D anchor point coordinates in the camera coordinate system. :

[0028]

[0029] in, Indicating a historical moment t Next i 3D anchor point coordinates in the coordinate system of a wide-angle fisheye camera Indicates the first i The extrinsic parameter matrix of a wide-angle fisheye camera For rotation matrix, It is a translation vector;

[0030] Using a distortion-sensing model The coordinates of each anchor point in the wide-angle fisheye camera coordinate system are obtained. Projection in pixel coordinates:

[0031]

[0032] in, Indicates the first i A wide-angle fisheye camera at a historical moment t The pixel coordinates below; For the first i A distortion perception model for a wide-angle fisheye camera; Indicates the first i Intrinsic parameter matrix of a wide-angle fisheye camera Indicates the first i The mirror parameters of a distortion-perceiving model for a wide-angle fisheye camera. Indicates the first i Distortion parameters of a distortion perception model for a wide-angle fisheye camera;

[0033] Using a spatiotemporally synchronized multi-view cross-temporal attention mechanism, the current time step is generated based on the image feature set, query vector set, and pixel coordinates. M Geometrically consistent BEV features .

[0034] Furthermore, the multi-view cross-temporal attention mechanism using spatiotemporal synchronization generates the current time based on the image feature set, query vector set, and pixel coordinates. M Geometrically consistent BEV features The formula is:

[0035]

[0036] in, Indicates the current time M In the bird's-eye view space BEV features corresponding to each BEV mesh cell This represents the set of valid image regions, i.e., invalid regions that have been removed from self-occluded areas or areas outside the camera's field of view; Indicates the first At each BEV grid cell location, there is a set of camera viewpoints corresponding to the effective image area. express The number of effective camera angles; This represents the time index of the current moment. Represents the preset height value set Z The total height in; Indicating a historical moment t Next, by the first i A wide-angle fisheye camera captures three-dimensional points in the BEV space. Pixel coordinates projected onto the two-dimensional image plane; Indicates by the first i A wide-angle fisheye camera at a historical moment Image features are obtained from the acquired panoramic RGB images through a feature extraction network; This represents a multi-scale deformable cross-attention mechanism.

[0037] Furthermore, the step of performing region-level comparative learning based on the BEV features at the current moment and the ground truth to optimize the BEV features at the current moment specifically includes:

[0038] BEV features obtained at the current time Calculate each semantic category c Region feature embedding The formula is:

[0039]

[0040] in, Indicates the current time M Next semantic category c Regional feature embedding, Indicates based on truth mask The marked belongs to a semantic category c A set of grid cells; Indicates the current time M In the bird's-eye view space The BEV features corresponding to each BEV grid cell; the semantic categories include vehicles, buildings, and ground; the truth mask. A mask generated from ground truth image labels, used to represent the semantic category to which each pixel belongs; Indicates belonging to a semantic category c The total number of cells in the grid cell set;

[0041] A lightweight multilayer perceptron (MLP) is used to map the class indices in the truth mask to the truth embeddings. The formula is:

[0042]

[0043] in, Represents the semantic category after mapping by MLP c Truth embedding, This represents a multilayer perceptron network. This represents the category index obtained based on the truth mask;

[0044] Embedding of regional features and truth embedding use Normalization processing yields normalized region feature embeddings. and truth embedding The formula is:

[0045]

[0046] in, Represents the normalized semantic category c Regional feature embedding, This represents the normalized truth embedding; express Norm;

[0047] Embedded based on normalized region features and truth embedding Constructing a cosine similarity matrix The formula is:

[0048]

[0049] in, Represents the cosine similarity matrix The Middle i row and number j Column elements; Represents the normalized region feature embedding and truth embedding The inner product; These are learnable parameters;

[0050] Based on the constructed cosine similarity matrix Constructing a symmetric contrast loss Through symmetrical contrast loss Embedding of regional features Optimization is performed by extracting the BEV features corresponding to each grid cell from the optimized region feature embedding to obtain the optimized BEV features for the current time step. .

[0051] Furthermore, the symmetric contrast loss The formula is:

[0052]

[0053]

[0054] in, Indicates symmetric contrast loss. Represents cross-entropy loss; It is the identity matrix; The total number of semantic categories; Represents the first in the identity matrix i row and number j Column elements; This is a transpose.

[0055] Furthermore, the optimized BEV features at the current time step are input into a dual-branch stage-supervised decoder, where semantic mapping and feature reconstruction are performed via the attention branch and convolution branch, respectively, and then fused to generate a BEV segmentation map. Specifically, this includes:

[0056] The optimized BEV features at the current time are input into the dual-branch stage-supervised decoder;

[0057] For the attention branch, in each stage s Based on a set of learnable query vectors, feature interaction is performed between the optimized BEV features and a multi-head cross-attention mechanism to obtain staged fine-grained embedding features;

[0058] A lightweight classification module performs semantic mapping on the obtained staged fine-grained embedding features to generate attention-branch staged BEV semantic prediction results at the corresponding resolution. The formula is:

[0059]

[0060] in, This indicates the attention branch in the stage. Next Each BEV grid cell corresponds to a semantic category. Attention branch stage BEV semantic prediction results; Indicates the stage Next Staged fine-grained embedding features of each BEV mesh cell; The learnable classification weight matrix representing the attention branch; This represents the normalized activation function, used to transform classification scores into a probability distribution;

[0061] For the convolutional branch, the optimized BEV features are sequentially reconstructed in stages through several upsampling modules to gradually restore the spatial resolution and obtain a staged convolutional feature representation. ;

[0062] The classification module of the convolutional branch represents the stage convolutional features. Perform semantic mapping to generate BEV semantic prediction results for the convolutional branches at the corresponding resolution. The formula is:

[0063]

[0064] in, Indicates the convolution branch at the th Phase 1 Each BEV grid cell corresponds to a semantic category. The predicted probability; This indicates that the upsampling module of the convolution branch is at the 1st... The first stage generated The staged convolutional feature representation of each BEV grid cell, This represents the optimized BEV feature at the current time. Indicates the convolution branch at the th The classification weight matrix for each stage;

[0065] The semantic prediction results of the attention branch and the convolution branch at the highest resolution stage are fused to obtain the final BEV segmentation prediction result, as shown in the formula:

[0066]

[0067] in, Indicates that the final time is in semantic category at each BEV grid cell The predicted probability, i.e., the BEV segmentation prediction result; , Preset weighting coefficients; Represents the normalization function;

[0068] Based on BEV segmentation prediction results Generate BEV segmentation map.

[0069] Furthermore, the BEV segmentation prediction results Generating a BEV segmentation map specifically includes:

[0070] BEV segmentation prediction results Each BEV mesh cell For all semantic categories The predicted probabilities are used to select categories, and the final semantic label corresponding to each grid cell is determined. The formula is:

[0071]

[0072] in, Indicates the first The final semantic label at each BEV mesh cell; Indicates the total number of semantic categories;

[0073] Final semantic labels for all BEV mesh cells According to its aerial view space The spatial positions in the image are arranged and reconstructed to generate the BEV segmentation map at the current moment.

[0074] Compared with the prior art, the present invention has the following advantages:

[0075] (1) This invention integrates the cross-temporal information fusion module and the cross-view spatial fusion module into a tightly integrated spatiotemporal synchronous fusion structure, reducing geometric distortion and cumulative error. Secondly, this invention performs region-level contrastive learning and multi-stage semantic guidance on BEV features, enhancing the spatial and semantic consistency of cross-view temporal features. Furthermore, the dual-branch stage-supervised decoder proposed in this invention provides two strategies: global optimization or convolutional refinement, achieving stable optimization without increasing inference cost and improving the decoding quality of BEV features.

[0076] (2) When projecting fisheye images onto a BEV, the existing technology often uses simplified geometric projection or single-frame distortion pre-correction, which makes it difficult to accurately handle fisheye distortion and camera calibration errors, resulting in deformation or misalignment of the target in the BEV space, affecting the downstream planning and positioning accuracy. This invention introduces distortion-aware projection and distortion-reducing sampling into the spatiotemporal synchronous encoder, performs explicit distortion correction on each 3D anchor point and performs accurate sampling in the pixel domain. Therefore, it effectively corrects the influence of fisheye nonlinear distortion and calibration error, making the geometric position projected onto the BEV more consistent with the real world, thereby significantly reducing target misalignment and shape distortion in the BEV, and improving the geometric accuracy and positioning availability of ground objects in the BEV.

[0077] (3) Existing technologies often ignore or fail to eliminate temporal misalignment caused by device self-motion in multi-temporal scenarios. Direct splicing or simple fusion of historical frame features will result in obvious viewpoint / time inconsistency. This invention introduces self-motion compensation based on relative pose in the spatiotemporal synchronous encoder and combines it with a spatiotemporal synchronous multi-view cross-temporal attention mechanism. The current 3D reference point is first transformed to the historical frame coordinate system and the attitude synchronization and distortion correction are performed before attention sampling fusion. This explicitly eliminates the geometric misalignment caused by self-motion between frames, so that the historical frame information can reliably supplement the viewpoint blind spot and weak texture area after being aligned with the current frame. This improves the temporal consistency and robustness of the BEV features after multi-frame fusion and enhances the recovery capability for moving objects and cross-frame sparse observations.

[0078] (4) Existing BEV methods typically only use pixel-level supervision of the final segmentation map, resulting in a lack of semantic guidance for intermediate BEV features, which manifests as inconsistent feature semantics and poor class separability. This invention proposes a semantic consistency supervision module, which adopts region-level contrastive learning. By comparing the feature embeddings of each semantic region on BEV with the ground value embeddings obtained by mapping the ground value mask in the embedding space and minimizing the symmetric cross-entropy loss, semantic constraints are directly imposed on the intermediate BEV representations at the class level. This mechanism enhances the class separability and semantic consistency of BEV features, making it easier for subsequent decoders to generate high-accuracy segmentation results from semantically distinguishable BEV features, especially improving the semantic confusion problem at small targets and inter-class boundaries.

[0079] (5) Existing methods often ignore the effectiveness of the field of view and occlusion factors when performing multi-view fusion, resulting in noise introduced by invalid viewpoints; the present invention adopts an effective viewpoint set in the BEV sampling and fusion process. The filtering process only samples and fuses views that are within the camera's field of view and are not self-occluded for each BEV unit. Therefore, in multi-view and multi-temporal fusion, it can eliminate observations outside the field of view and those that are occluded, reduce the interference of erroneous observations, and improve the stability and anti-occlusion capability of fusion. In particular, it is more robust to scenes with overlapping views and close-range occlusion in surround view systems.

[0080] (6) Existing decoders with a single strategy have difficulty in balancing global semantic consistency and local spatial details. They either rely solely on convolution to obtain global context or rely solely on attention to preserve spatial details. This invention designs attention branches and convolution branches simultaneously in a dual-branch stage supervised decoder. The attention branch provides a global semantic optimization path through multi-head cross attention, while the convolution branch refines local space through a stage-by-stage upsampling module. The two predictions are then fused at the highest resolution stage. This parallel structure can balance global consistency and local details without changing the inference complexity constraint, significantly improving boundary fineness and semantic consistency, thereby achieving more robust segmentation results in complex scenes. Attached Figure Description

[0081] Figure 1 This is a flowchart of the bird's-eye view segmentation method according to an embodiment of the present invention;

[0082] Figure 2 This is a flowchart of a spatiotemporal synchronization encoder according to an embodiment of the present invention. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0084] Example 1:

[0085] This embodiment provides a method for bird's-eye view segmentation based on multi-frame spatiotemporal correlation of panoramic fisheye images, such as... Figure 1 As shown, it includes the following steps:

[0086] Step S1: Acquire multi-moment, multi-view panoramic RGB images of the aircraft's ground environment using a panoramic camera system; the panoramic camera system includes four wide-angle fisheye cameras located at the front, rear, left, and right of the aircraft.

[0087] Step S2: Use a feature extraction network to extract image features from the panoramic RGB image to generate a multi-time and multi-view image feature set; the feature extraction network is a convolutional neural network (CNN).

[0088] Step S2 specifically includes:

[0089] The RGB images captured at each moment by four surround-view fisheye cameras (located in front, behind, left, and right of the aircraft, respectively) are input into the feature extraction network;

[0090] Normalize, scale, or perform necessary color corrections on the input RGB image to adapt it to the input requirements of the Convolutional Neural Network (CNN).

[0091] The RGB images at each time point and from each viewpoint are processed by a convolutional neural network (CNN) to generate corresponding image feature representations. This forms a multi-time, multi-view image feature set, which is used for subsequent spatiotemporal synchronous encoder processing.

[0092] Step S3: Use a spatiotemporal synchronous encoder to aggregate the image feature set into a unified bird's-eye view space to obtain the BEV features at the current moment, such as... Figure 2 As shown, it specifically includes:

[0093] Step S301: Initialize the BEV query vector set Each query Corresponding bird's-eye view space One of the BEV mesh cells, Represents bird's-eye view space BEV mesh unit The corresponding query vector;

[0094] Step S302: Space the bird's-eye view Each BEV mesh cell in the graph is mapped to multiple preset heights, forming a 3D reference point at the current moment. The specific content includes:

[0095] For each BEV grid cell Map them to the world coordinate system and obtain the world coordinates, i.e.:

[0096]

[0097] Where X and Y represent the number of grid cells in the ground coordinate system, l This represents the side length of the actual region represented by each grid cell. , This indicates the number of BEV grid cells in the ground coordinate system, i.e., the resolution of the BEV grid in the horizontal and vertical directions. This represents the side length of the actual region represented by each BEV mesh cell. , Represents BEV mesh element The corresponding coordinates in the real-world coordinate system; Represents BEV mesh element Two-dimensional coordinates;

[0098] Through a set of preset heights Self-framework in the present moment Generate a set of 3D reference points ,Right now:

[0099]

[0100] in, Indicates the current time At the preset height 3D reference point below; , For a preset set of height values;

[0101] Step S303: Based on camera pose and self-motion compensation, the current moment... 3D reference point below Transformation into the historical coordinate system through relative self-motion In the process, the anchor points of the historical coordinate system are obtained, including:

[0102] To obtain data from historical images, each anchor point at the current moment... Transformation into the historical coordinate system through relative self-motion middle:

[0103]

[0104] in, Representing historical moments Anchor point in the self-coordinate system; Representing historical moments With the current moment The coordinate transformation matrix between them Representing historical moments The homogeneous transformation matrix from the world coordinate system to the self coordinate system. Indicates the current time The homogeneous transformation matrix from the world coordinate system to the self coordinate system;

[0105] Anchor point of the historical coordinate system Mapped to t The moment i Obtain the anchor point coordinates in the camera coordinate system. :

[0106]

[0107] in, Indicating a historical moment t Next i 3D anchor point coordinates in a wide-angle fisheye camera coordinate system Indicates the first i The extrinsic parameter matrix of a wide-angle fisheye camera For rotation matrix, It is a translation vector;

[0108] Step S304: Using a distortion-aware model The coordinates of each anchor point in the wide-angle fisheye camera coordinate system are obtained. Projection in pixel coordinates:

[0109]

[0110] in, This refers to the camera's internal parameters. Indicates the mirror parameters of the model. Represents the distortion parameters of the model;

[0111] S305. Using a spatiotemporally synchronized multi-view cross-temporal attention mechanism, sample and fuse image features from multiple perspectives and time sequences to generate geometrically consistent BEV features for the current time M. ,Right now:

[0112]

[0113] in, Indicates the current time M In the bird's-eye view space BEV features corresponding to each BEV mesh cell This represents the set of valid image regions, i.e., invalid regions that have been removed from self-occluded areas or areas outside the camera's field of view; Indicates the first At each BEV grid cell location, there is a set of camera viewpoints corresponding to the effective image area. express The number of effective camera angles; This represents the time index of the current moment. Represents the preset height value set Z The total height in; Indicating a historical moment t Next, by the first i A wide-angle fisheye camera captures three-dimensional points in the BEV space. Pixel coordinates projected onto the two-dimensional image plane; Indicates by the first i A wide-angle fisheye camera at a historical moment Image features are obtained from the acquired panoramic RGB images through a feature extraction network; This represents a multi-scale deformable cross-attention mechanism.

[0114] Step S3 is to address issues such as geometric distortion, calibration errors, and temporal misalignment caused by self-motion in the process of projecting fisheye images onto BEV, thereby obtaining geometrically consistent and semantically reliable BEV features to ensure the accuracy of subsequent segmentation and localization. Specifically, firstly, a query center is established for each grid cell by initializing the BEV query vector set, which facilitates the orderly aggregation of pixel information from multiple viewpoints and times in the BEV space. Then, each BEV grid cell is mapped to multiple preset heights and 3D reference points are generated. Height sampling compensates for the loss of vertical dimension information caused by single height projection, enhancing the ability to distinguish targets at different heights (such as ground equipment and tall structures). To address the temporal misalignment caused by equipment self-motion, relative pose transformation is used to convert the reference point at the current moment to the historical frame coordinate system, and the reference point is mapped to the camera coordinate system through camera extrinsic parameters, ensuring that the historical frame observation is spatially aligned with the current moment, thereby eliminating the geometric offset caused by motion. To address the nonlinear distortion and calibration error of the fisheye lens, a distortion perception model is applied to project 3D points in the camera coordinate system into pixel coordinates, and distortion correction is performed in the pixel domain to reduce the one-time impact of projection error on BEV geometry. Finally, a spatiotemporally synchronized multi-view cross-temporal attention mechanism (PS-CMA) is adopted, and features from each camera at each time and height are fused using multi-scale deformable cross-attention sampling, combined with an effective set of viewpoints. This process not only eliminates invalid observations that are occluded or outside the field of view, but also dynamically focuses on geometrically relevant pixel positions through deformable sampling, thereby achieving reliable information convergence in both spatial and temporal dimensions. Through the above steps, S3 can effectively reduce deformation and misalignment caused by fisheye distortion and calibration errors, eliminate viewpoint inconsistencies caused by inter-frame self-motion, enhance the recognition ability of the height dimension, and suppress invalid viewpoint noise. The resulting BEV features show significant improvements in geometric consistency, temporal stability, and semantic expression, providing more accurate and robust input for subsequent semantic consistency supervision and dual-branch decoders, thereby improving the overall accuracy of BEV segmentation and the reliability of engineering deployment.

[0115] Step S4: Perform regional-level comparative learning based on the current BEV features and ground truth to optimize the current BEV features, specifically including:

[0116] BEV features obtained at the current time Calculate each semantic category c Region feature embedding The formula is:

[0117]

[0118] in, Indicates the current time M Next semantic category c Regional feature embedding, Indicates based on truth mask The marked belongs to a semantic category c A set of grid cells; Indicates the current time M In the bird's-eye view space BEV features corresponding to each BEV grid cell; semantic categories include vehicles, buildings, and ground; ground truth mask. A mask generated from ground truth image labels, used to represent the semantic category to which each pixel belongs; Indicates belonging to a semantic category c The total number of cells in the grid cell set;

[0119] A lightweight multilayer perceptron (MLP) is used to map the class indices in the truth mask to the truth embeddings. The formula is:

[0120]

[0121] in, Represents the semantic category after mapping by MLP c Truth embedding, This represents a multilayer perceptron network. This represents the category index obtained based on the truth mask;

[0122] Embedding of regional features and truth embedding use Normalization processing yields normalized region feature embeddings. and truth embedding The formula is:

[0123]

[0124] in, Represents the normalized semantic category c Regional feature embedding, This represents the normalized truth embedding; express Norm;

[0125] Embedded based on normalized region features and truth embedding Constructing a cosine similarity matrix The formula is:

[0126]

[0127] in, Represents the cosine similarity matrix The Middlei row and number j Column elements; Represents the normalized region feature embedding and truth embedding The inner product; These are learnable parameters;

[0128] Based on the constructed cosine similarity matrix Constructing a symmetric contrast loss Through symmetrical contrast loss Embedding of regional features Optimization is performed by extracting the BEV features corresponding to each grid cell from the optimized region feature embedding to obtain the optimized BEV features for the current time step. .

[0129] Symmetrical contrast loss The formula is:

[0130]

[0131]

[0132] in, Indicates symmetric contrast loss. Represents cross-entropy loss; It is the identity matrix; The total number of semantic categories; Represents the first in the identity matrix i row and number j Column elements; This is a transpose.

[0133] Step S5: Input the optimized BEV features of the current time step into the dual-branch stage-supervised decoder, perform semantic mapping and feature reconstruction through the attention branch and convolution branch respectively, and fuse them to generate a BEV segmentation map, specifically including:

[0134] The optimized BEV features are input into the two-branch stage-supervised decoder;

[0135] For the attention branch, in each stage s Based on a set of learnable query vectors, feature interaction is performed between the optimized BEV features and a multi-head cross-attention mechanism to obtain staged fine-grained embedding features;

[0136] A lightweight classification module performs semantic mapping on the obtained staged fine-grained embedding features to generate attention-branch staged BEV semantic prediction results at the corresponding resolution. The formula is:

[0137]

[0138] in, This indicates the attention branch in the stage. Next Each BEV grid cell corresponds to a semantic category. Attention branch stage BEV semantic prediction results; Indicates the stage Next Staged fine-grained embedding features of each BEV mesh cell; The learnable classification weight matrix representing the attention branch; This represents the normalized activation function, used to transform classification scores into a probability distribution;

[0139] For the convolutional branch, the optimized BEV features are sequentially reconstructed in stages through several upsampling modules to gradually restore the spatial resolution and obtain a staged convolutional feature representation. ;

[0140] The classification module of the convolutional branch represents the stage convolutional features. Perform semantic mapping to generate BEV semantic prediction results for the convolutional branches at the corresponding resolution. The formula is:

[0141]

[0142] in, Indicates the convolution branch at the th Phase 1 Each BEV grid cell corresponds to a semantic category. The predicted probability; This indicates that the upsampling module of the convolution branch is at the 1st... The first stage generated The staged convolutional feature representation of each BEV grid cell, This represents the optimized BEV features; Indicates the convolution branch at the th The classification weight matrix for each stage;

[0143] The semantic prediction results of the attention branch and the convolution branch at the highest resolution stage are fused to obtain the final BEV segmentation prediction result, as shown in the formula:

[0144]

[0145] in, Indicates that the final time is in semantic category at each BEV grid cell The predicted probability, i.e., the BEV segmentation prediction result; , Preset weighting coefficients; Represents the normalization function;

[0146] Based on BEV segmentation prediction results Generate the final BEV segmentation map, specifically including:

[0147] BEV segmentation prediction results Each BEV mesh cell For all semantic categories The predicted probabilities are used to select categories, and the final semantic label corresponding to each grid cell is determined. The formula is:

[0148]

[0149] in, Indicates the first The final semantic label at each BEV mesh cell; Indicates the total number of semantic categories;

[0150] Final semantic labels for all BEV mesh cells According to its aerial view space The spatial positions are arranged and reconstructed to generate the final BEV segmentation map at the current moment.

[0151] The loss function for the attention branch is:

[0152]

[0153]

[0154] in, Let be the loss function for the entire attention branch, and let represent the sum of weighted losses across all stages. These are the weighting coefficients. This refers to the number of stages involved in the training process. It is an exponential function, when the unit Category: When it is 1, unit The category is not The time is 0; For the current stage The set of all BEV mesh cells, representing all mesh cells to be predicted; For the first The stage loss is represented by the difference between the attention branch prediction result and the true label in the current stage, calculated using cross-entropy loss.

[0155] The loss function for the convolution branch is:

[0156]

[0157]

[0158] in, Let be the loss function for the entire convolutional branch, and let be the sum of the weighted losses for all convolutional stages. K This refers to the number of convolutional stages involved in the training process; These are the weighting coefficients; Loss for each convolutional stage; For the current convolution stage The set of all BEV mesh cells represents all mesh cells to be predicted.

[0159] Example 2:

[0160] This experimental example uses the model of the present invention and evaluates the difference in semantic segmentation performance when using and not using the multi-frame spatiotemporal correlation-based panoramic fisheye image bird's-eye view segmentation method described in the present invention.

[0161] Among them, STSyn-BEV represents a bird's-eye view segmentation model based on multi-frame spatiotemporal correlation of surround-view fisheye images. ∅ Represents the baseline model.

[0162] This experiment uses the FB-SSEM dataset, which contains 20 daytime self-driving car sequences in a parking lot environment (at a speed of 0.35 m / s). Each sequence represents a different parking lot setting, i.e., all vehicles at different locations on the ground. Each sequence consists of 1000 samples recorded at a frequency of 2 Hz; each sample provides RGB images from four fisheye cameras (front, rear, left, and right) and synchronized pixel-level BEV semantic labels.

[0163] This experimental example uses 70 / 15 / 15% training / validation / test segmentation to partition each sequence. The BEV segmentation task includes five semantic classes: vehicle (self + parked car / truck), bus, electric vehicle charger, ground, and inaccessible area.

[0164] STSyn-BEV and the baseline model STSyn-BEV ∅ Implemented in PyTorch and trained on a single NVIDIA RTX 3090 GPU (24GB VRAM). The BEV plane covers an area of ​​16.67m × 16.67m, discretized into a grid with a resolution of 0.33m (2500 query vectors). The model's height anchors were chosen at 0m, 0.25m, and 1.8m.

[0165] Qualitative results of the experiment:

[0166] The BEV segmentation task includes five semantic classes: vehicle (self + parked car / truck), bus, electric vehicle charger, ground, and inaccessible area. By comparing whether the multi-frame spatiotemporal correlation-based surround-view fisheye image bird's-eye view segmentation method described in this invention is used, the following conclusions are drawn:

[0167] 1. When the multi-frame spatiotemporal correlation-based panoramic fisheye image bird's-eye view segmentation method described in this invention is not used, although STSyn-BEV ∅ Distortion correction was performed before feature extraction, but it exhibits obvious structural discontinuities and jagged boundaries in BEV space.

[0168] 2. When using the multi-frame spatiotemporal correlation-based panoramic fisheye image bird's-eye view segmentation method described in this invention, long-term temporal information alignment and fusion are introduced into the spatiotemporal synchronous encoder, combined with stage supervision that is progressively refined by the decoder. This design achieves more stable geometric consistency and more accurate detail recovery, resulting in a smoother, more continuous, and morphologically complete BEV output.

[0169] 3. When the bird's-eye view segmentation method based on multi-frame spatiotemporal correlation of the panoramic fisheye image described in this invention is used, the predicted bird's-eye view basically coincides with the ground truth, and the accuracy is high, which proves the effectiveness of this method.

[0170] Quantitative results of the experiment:

[0171] To quantitatively evaluate the difference between the semantic segmentation map obtained by the multi-frame spatiotemporal correlation-based panoramic fisheye image bird's-eye view segmentation method described in this invention and the ground truth, the mean Intersection-over-Union (mIoU) and frequency-weighted IoU (fwIoU) indices are defined as follows:

[0172] For each category The Intersection-over-Union (IoU) ratio is defined as:

[0173]

[0174] in, , and They represent The number of true positive, false positive, and false negative pixels for each class. mIoU is a standard metric for semantic segmentation, calculated by comparing the number of true positive, false positive, and false negative pixels across all classes. C The average is obtained as follows:

[0175]

[0176] fwIoU further considers a frequency-weighted approach for each category to address the class imbalance problem:

[0177]

[0178] in, This represents the total number of pixels belonging to category c among all ground truth labels in the entire dataset.

[0179] Based on the calculation formulas for the average cross-union ratio and the frequency-weighted cross-union ratio, the results of calculating the average cross-union ratio and the frequency-weighted cross-union ratio of the panoramic fisheye image bird's-eye view segmentation method based on multi-frame spatiotemporal correlation described in this invention are shown in Table 1.

[0180] Table 1. Average Crossover Union Ratio and Frequency-Weighted Crossover Union Ratio Results

[0181]

[0182] Among them, STSyn-BEV ∅ The baseline model, which does not employ the method described in this invention, has an average cross-union ratio of 0.8142 and a frequency-weighted cross-union ratio of 0.8624.

[0183] STSyn-BEV represents the model using the present invention, with an average cross-union ratio of 0.8767 and a frequency-weighted cross-union ratio of 0.9154.

[0184] By comparing the average crossover ratio and frequency-weighted crossover ratio of the two, the method of the present invention can significantly improve the accuracy of BEV semantic segmentation graphs, by an increase of 7.67% (mIoU) and 6.15% (fwIoU).

[0185] To further evaluate the effect of the method described in this invention on each semantic category, and to assess the robustness of the method under class imbalance, the class segmentation results are shown in Table 2:

[0186] Table 2 Class segmentation results

[0187]

[0188] Frequent categories such as cars and ground appeared in 100% of the frames, while rare categories such as electric vehicle chargers and non-drivable areas appeared in only 52.84% and 64.30% of the frames, respectively.

[0189] In the rare categories of electric vehicle chargers and non-drivable areas, STSyn-BEV achieved an average score of 0.797, compared to STSyn-BEV... ∅The improvement from 0.663 is 20.2%. This clearly demonstrates that our method improves the representation of underrepresented semantic classes.

[0190] In the ground and automotive high-frequency categories, the STSyn-BEV achieved an average score of 0.932, compared to the STSyn-BEV... ∅ The score of 0.929 remains competitive. This indicates that our improvements to the rare class did not substantially affect the overall performance of the frequent class.

[0191] To further evaluate the effectiveness of each module in the method described in this invention, ablation experiments were conducted, and the results are shown in Table 3:

[0192] Table 3 Ablation Experiment Results

[0193]

[0194] in, The representative does not use the spatiotemporal synchronization encoder module described in this invention, and its average cross-connection-to-union ratio is 0.8481, and its frequency-weighted cross-connection-to-union ratio is 0.8866.

[0195] in, The representative does not use the semantic consistency supervision module in the method described in this invention, and its average crossover and union ratio is 0.8617, and its frequency-weighted crossover and union ratio is 0.8960.

[0196] in, The representative of the dual-branch stage decoder module that does not employ the method described in this invention has an average cross-connection-to-union ratio of 0.8679 and a frequency-weighted cross-connection-to-union ratio of 0.9032.

[0197] STSyn-BEV represents all modules using the method described in this invention, with an average cross-union ratio of 0.8767 and a frequency-weighted cross-union ratio of 0.9154.

[0198] Among them, compared with STSyn-BEV, , and The average crossover-union ratios decreased by 0.0286, 0.015 and 0.0088, respectively, demonstrating the effectiveness of the spatiotemporal synchronous encoder, the semantic consistency supervision module and the dual-branch stage supervised decoder.

[0199] Therefore, the experimental examples demonstrate the effectiveness of the panoramic fisheye image bird's-eye view segmentation method based on multi-frame spatiotemporal correlation described in this invention.

[0200] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0201] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation, characterized in that, Includes the following steps: The system uses a surround-view camera system to acquire multi-time, multi-view surround-view RGB images of the aircraft's ground environment. A feature extraction network is used to extract image features from the panoramic RGB image to generate a multi-time-and-view image feature set. The image feature set is aggregated into a unified bird's-eye view space using a spatiotemporal synchronous encoder to obtain the BEV features at the current moment; Based on the current BEV features and ground truth, a regional-level comparative learning is performed to optimize the current BEV features. The optimized BEV features at the current time are input into a dual-branch stage-supervised decoder, and semantic mapping and feature reconstruction are performed through the attention branch and the convolution branch, respectively, and then fused to generate a BEV segmentation map. The step of aggregating the image feature set into a unified bird's-eye view space using a spatiotemporal synchronous encoder to obtain the BEV features at the current moment specifically includes: Initialize BEV query vector set Each query Corresponding bird's-eye view space One of the BEV mesh cells, Represents bird's-eye view space BEV mesh unit The corresponding query vector; Bird's-eye view space Each BEV mesh cell in the graph is mapped to multiple preset heights, forming a 3D reference point at the current moment. ; Based on camera pose and self-motion compensation, the current moment 3D reference point below Transformation into the historical coordinate system through relative self-motion In the process, obtain the anchor point in the historical coordinate system. ; Anchor point in the historical coordinate system Mapping to historical moments t The next i In the coordinate system of a wide-angle fisheye camera, obtain the 3D anchor point coordinates in the camera coordinate system. ; Using a distortion-sensing model The coordinates of each anchor point in the wide-angle fisheye camera coordinate system are obtained. Projection in pixel coordinates: in, Indicates the first i A wide-angle fisheye camera at a historical moment t The pixel coordinates below; For the first i A distortion perception model for a wide-angle fisheye camera; Indicates the first i Intrinsic parameter matrix of a wide-angle fisheye camera Indicates the first i The mirror parameters of a distortion-perceiving model for a wide-angle fisheye camera. Indicates the first i Distortion parameters of a distortion perception model for a wide-angle fisheye camera; Using a spatiotemporally synchronized multi-view cross-temporal attention mechanism, the current time step is generated based on the image feature set, query vector set, and pixel coordinates. M Geometrically consistent BEV features .

2. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The surround-view camera system includes four wide-angle fisheye cameras located at the front, rear, left, and right of the aircraft, respectively.

3. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The feature extraction network is a convolutional neural network (CNN).

4. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The image feature set includes panoramic RGB images captured by each wide-angle fisheye camera at various times.

5. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The 3D reference point The formula is: in, Indicates the current time At the preset height 3D reference point below, , This indicates the number of BEV grid cells in the ground coordinate system, i.e., the resolution of the BEV grid in the horizontal and vertical directions; This represents the side length of the actual region represented by each BEV mesh cell. , For a preset set of height values; , Represents BEV mesh element The corresponding coordinates in the real-world coordinate system; Represents BEV mesh element Two-dimensional coordinates; Anchor points in the historical coordinate system : in, Representing historical moments Anchor point in the self-coordinate system; Representing historical moments With the current moment The coordinate transformation matrix between them Representing historical moments The homogeneous transformation matrix from the world coordinate system to the self coordinate system. Indicates the current time The homogeneous transformation matrix from the world coordinate system to the self coordinate system; 3D anchor point coordinates in the camera coordinate system : in, Indicating a historical moment t Next i 3D anchor point coordinates in the coordinate system of a wide-angle fisheye camera Indicates the first i The extrinsic parameter matrix of a wide-angle fisheye camera Let be a rotation matrix. It is a translation vector.

6. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The multi-view cross-temporal attention mechanism using spatiotemporal synchronization generates the current time based on the image feature set, query vector set, and pixel coordinates. M Geometrically consistent BEV features The formula is: in, Indicates the current time M In the bird's-eye view space BEV features corresponding to each BEV mesh cell This represents the set of valid image regions, i.e., invalid regions that have been removed from self-occluded areas or areas outside the camera's field of view; Indicates the first At each BEV grid cell location, there is a set of camera viewpoints corresponding to the effective image area. express The number of effective camera angles; This represents the time index of the current moment. Represents the preset height value set Z The total height in; Indicating a historical moment t Next, by the first i A wide-angle fisheye camera captures three-dimensional points in the BEV space. Pixel coordinates projected onto the two-dimensional image plane; Indicates by the first i A wide-angle fisheye camera at a historical moment Image features are obtained from the acquired panoramic RGB images through a feature extraction network; This represents a multi-scale deformable cross-attention mechanism.

7. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The step of performing regional-level comparative learning based on the BEV features at the current moment and the ground truth to optimize the BEV features at the current moment specifically includes: BEV features obtained at the current time Calculate each semantic category c Region feature embedding The formula is: in, Indicates the current time M Next semantic category c Regional feature embedding, Indicates based on truth mask The marker belongs to a semantic category c A set of grid cells; Indicates the current time M In the bird's-eye view space The BEV features corresponding to each BEV grid cell; the semantic categories include vehicles, buildings, and ground; the truth mask. A mask generated from ground truth image labels, used to represent the semantic category to which each pixel belongs; Indicates belonging to a semantic category c The total number of cells in the grid cell set; A lightweight multilayer perceptron (MLP) is used to map the class indices in the truth mask to the truth embeddings. The formula is: in, Represents the semantic category after mapping by MLP c Truth embedding, This represents a multilayer perceptron network. This represents the category index obtained based on the truth mask; Embedding of regional features and truth embedding use Normalization processing yields normalized region feature embeddings. and truth embedding The formula is: in, Represents the normalized semantic category c Regional feature embedding, This represents the normalized truth embedding; express Norm; Embedded based on normalized region features and truth embedding Constructing a cosine similarity matrix The formula is: in, Represents the cosine similarity matrix The Middle i row and number j Column elements; Represents the normalized region feature embedding and truth embedding The inner product; These are learnable parameters; Based on the constructed cosine similarity matrix Constructing a symmetric contrast loss Through symmetrical contrast loss Embedding of regional features Optimization is performed by extracting the BEV features corresponding to each grid cell from the optimized region feature embedding to obtain the optimized BEV features for the current time step. .

8. The method for bird's-eye view segmentation of panoramic fisheye images based on multi-frame spatiotemporal correlation according to claim 7, characterized in that, The symmetric contrast loss The formula is: in, Indicates symmetric contrast loss. Represents cross-entropy loss; It is the identity matrix; The total number of semantic categories; Represents the first in the identity matrix i row and number j Column elements; This is a transpose.

9. The method for bird's-eye view segmentation of a panoramic fisheye image based on multi-frame spatiotemporal correlation according to claim 1, characterized in that, The process of inputting the optimized BEV features at the current time step into a dual-branch stage-supervised decoder, performing semantic mapping and feature reconstruction via the attention branch and convolution branch respectively, and fusing them to generate a BEV segmentation map specifically includes: The optimized BEV features at the current time are input into the dual-branch stage-supervised decoder; For the attention branch, in each stage s Based on a set of learnable query vectors, feature interaction is performed between the optimized BEV features and a multi-head cross-attention mechanism to obtain staged fine-grained embedding features; A lightweight classification module performs semantic mapping on the obtained staged fine-grained embedding features to generate attention-branch staged BEV semantic prediction results at the corresponding resolution. The formula is: in, This indicates the attention branch in the stage. Next Each BEV grid cell corresponds to a semantic category. Attention branch stage BEV semantic prediction results; Indicates the stage Next Staged fine-grained embedding features of each BEV mesh cell; The learnable classification weight matrix representing the attention branch; This represents the normalized activation function, used to transform classification scores into a probability distribution; For the convolutional branch, the optimized BEV features are sequentially reconstructed in stages through several upsampling modules to gradually restore the spatial resolution and obtain a staged convolutional feature representation. ; The classification module of the convolutional branch represents the stage convolutional features. Perform semantic mapping to generate BEV semantic prediction results for the convolutional branches at the corresponding resolution. The formula is: in, Indicates the convolution branch at the th Phase 1 Each BEV grid cell corresponds to a semantic category. The predicted probability; This indicates that the upsampling module of the convolution branch is at the 1st... The first stage generated The staged convolutional feature representation of each BEV grid cell, This represents the optimized BEV feature at the current time. Indicates the convolution branch at the th The classification weight matrix for each stage; The semantic prediction results of the attention branch and the convolution branch at the highest resolution stage are fused to obtain the final BEV segmentation prediction result, as shown in the formula: in, Indicates the final time in semantic category at each BEV grid cell The predicted probability, i.e., the BEV segmentation prediction result; , Preset weighting coefficients; Represents the normalization function; Based on BEV segmentation prediction results Generate BEV segmentation map.

10. A method for bird's-eye view segmentation of a panoramic fisheye image based on multi-frame spatiotemporal correlation according to claim 9, characterized in that, The BEV segmentation prediction results Generating a BEV segmentation map specifically includes: BEV segmentation prediction results Each BEV mesh cell For all semantic categories The predicted probabilities are used to select categories, and the final semantic label corresponding to each grid cell is determined. The formula is: in, Indicates the first The final semantic label at each BEV mesh cell; Indicates the total number of semantic categories; Final semantic labels for all BEV mesh cells According to its location in the bird's-eye view space The spatial positions in the image are arranged and reconstructed to generate the BEV segmentation map at the current moment.

Images