Aerial view semantic segmentation method and device based on geometric prior, equipment and medium

Abstract

The application provides a bird's-eye view semantic segmentation method and device based on geometric prior, equipment and medium, wherein the method comprises: acquiring an image to be processed and bird's-eye view query features; inputting the image to be processed into a feature extractor to obtain image features output by the feature extractor; inputting the image features and the bird's-eye view query features into a feature extraction model to obtain bird's-eye view features output by the feature extraction model; based on the bird's-eye view features, performing semantic segmentation on the image to be processed; the feature extraction model comprises a self-attention module, and the self-attention module is used for cross-view self-attention conversion of tokens of the image to be processed and the bird's-eye view query features. The method, device, electronic equipment and storage medium provided by the application can reduce the computational complexity of the self-attention mechanism, further reduce the complexity of image feature extraction, and improve the accuracy and reliability of subsequent semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, and in particular to a method, apparatus, device, and medium for semantic segmentation of bird's-eye view based on geometric priors. Background Technology

[0002] The ability to perceive the surrounding environment is a crucial capability for autonomous driving. Specifically, autonomous vehicles need to have capabilities such as 3D object detection and map segmentation. In recent years, bird's-eye view (BEV) maps, as a type of map that simultaneously facilitates vehicle perception of the surrounding environment and planning downstream tasks, have attracted the attention of many researchers, and much work has been dedicated to perceiving objects within bird's-eye view maps.

[0003] In the existing technology, the semantic segmentation methods based on bird's-eye view mainly include the semantic segmentation method based on geometric projection and the semantic segmentation method based on Transformer. The semantic segmentation method based on geometric projection first generates a series of pseudo point clouds based on the multi-view image, and then projects the features of the image onto the bird's-eye view plane based on the pseudo point clouds and the parameter matrix of the camera.

[0004] In Transformer-based bird's-eye view semantic segmentation methods, the attention mechanism used is global, and the computational complexity is directly proportional to the number of input viewpoints, the resolution of the feature maps, and the resolution of the query features in the bird's-eye view. For scenarios requiring high-resolution bird's-eye views, the computational overhead increases significantly, exceeding the computational load limit of many devices. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for semantic segmentation of bird's-eye view based on geometric priors, in order to solve the problem that in the prior art, the computational cost of semantic segmentation increases significantly for scenarios requiring high-resolution bird's-eye view images, exceeding the computational load limit of many devices.

[0006] This invention provides a semantic segmentation method for bird's-eye view images based on geometric priors, comprising:

[0007] Obtain query features from the image to be processed and the bird's-eye view;

[0008] The image to be processed is input into a feature extractor to obtain the image features output by the feature extractor;

[0009] The image features and the bird's-eye view query features are input into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model;

[0010] Based on the bird's-eye view features, semantic segmentation is performed on the image to be processed;

[0011] The feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

[0012] According to the present invention, a semantic segmentation method for bird's-eye view based on geometric prior is provided, wherein the feature extraction model includes multiple cascaded feature extraction modules;

[0013] The step of inputting the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model includes:

[0014] The previous token and the previous bird's-eye view query feature of the image to be processed are input into the current feature extraction module to obtain the current bird's-eye view query feature output by the current feature extraction module. The previous bird's-eye view query feature is the feature extraction module output before the current feature extraction module.

[0015] The bird's-eye view query features output by the last feature extraction module are used as the bird's-eye view features.

[0016] According to a geometric prior-based bird's-eye view semantic segmentation method provided by the present invention, the step of inputting the previous token and the previous bird's-eye view query features of the image to be processed into the current feature extraction module to obtain the current bird's-eye view query features output by the current feature extraction module includes:

[0017] The previous token and the previous bird's-eye view query features of the image to be processed are input into the self-attention module of the current feature extraction module. The self-attention module samples the bird's-eye view query features to obtain sampled bird's-eye view query features. Then, the sampled bird's-eye view query features are inversely sampled to obtain the inversely sampled bird's-eye view query features output by the self-attention module.

[0018] The inverse-sampled bird's-eye view query features are input into the feedforward propagation network of the current feature extraction module to obtain the current bird's-eye view query features output by the feedforward propagation network.

[0019] According to the present invention, a bird's-eye view semantic segmentation method based on geometric prior is provided. The self-attention module includes a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. The image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query features to obtain a first bird's-eye view query feature. The image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query features to obtain a second bird's-eye view query feature. The scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query features.

[0020] According to the present invention, a bird's-eye view semantic segmentation method based on geometric prior is provided, wherein the feature extractor includes a first convolutional layer, a first activation layer and a first normalization layer connected in sequence, and the feature extractor is used to output multi-scale image features.

[0021] According to the present invention, a semantic segmentation method for bird's-eye view images based on geometric priors is provided, wherein semantic segmentation of the image to be processed based on the bird's-eye view features includes:

[0022] The bird's-eye view features are input into the decoder, which outputs the semantic segmentation result of the image to be processed. The decoder includes a second convolutional layer, a second normalization layer, and a second activation layer connected in sequence.

[0023] According to the present invention, a bird's-eye view semantic segmentation method based on geometric prior is provided, wherein the step of inputting the image features and the bird's-eye view query features into a feature extraction model to obtain the bird's-eye view features output by the feature extraction model includes:

[0024] Obtain the intrinsic and extrinsic parameter matrices of the camera, which are used for geometric priors;

[0025] Based on the camera's intrinsic and extrinsic parameter matrices, as well as the image features and the bird's-eye view query features, the bird's-eye view features output by the feature extraction model are obtained.

[0026] The present invention also provides a bird's-eye view semantic segmentation device based on geometric priors, comprising:

[0027] The acquisition unit is used to acquire query features of the image to be processed and the bird's-eye view;

[0028] An image feature extraction unit is used to input the image to be processed into a feature extractor and obtain the image features output by the feature extractor.

[0029] A bird's-eye view feature extraction unit is used to input the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model;

[0030] A semantic segmentation unit is used to perform semantic segmentation on the image to be processed based on the bird's-eye view features;

[0031] The feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the geometric prior-based bird's-eye view semantic segmentation method as described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the geometric prior-based bird's-eye view semantic segmentation method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the geometric prior-based bird's-eye view semantic segmentation method as described above.

[0035] The present invention provides a method, apparatus, device, and medium for semantic segmentation of bird's-eye view based on geometric priors. The self-attention module in the feature extraction model performs cross-view self-attention transformation on the token of the image to be processed and the query features of the bird's-eye view, which can reduce the computational complexity of the self-attention mechanism, further reduce the complexity of image feature extraction, and improve the accuracy and reliability of the obtained bird's-eye view features, thereby further improving the accuracy and reliability of subsequent semantic segmentation. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 This is one of the flowcharts of the bird's-eye view semantic segmentation method based on geometric priors provided by the present invention;

[0038] Figure 2 This is a schematic diagram of the process for sampling features for bird's-eye view queries provided by the present invention;

[0039] Figure 3 This is a schematic diagram of the process for inverse sampling of query features from a sampled bird's-eye view, provided by the present invention.

[0040] Figure 4 This is a schematic diagram of the structure of the self-attention module provided by the present invention;

[0041] Figure 5 This is the second flowchart of the bird's-eye view semantic segmentation method based on geometric prior provided by the present invention;

[0042] Figure 6This is a schematic diagram of the structure of the bird's-eye view semantic segmentation device based on geometric prior provided by the present invention;

[0043] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0046] In recent years, bird's-eye view (BEV) has attracted the attention of many researchers as a type of map that facilitates vehicle perception of the surrounding environment and planning of downstream tasks. There has been a lot of work dedicated to perceiving objects in bird's-eye view.

[0047] Given the high cost of radar equipment and the low resolution of point cloud data itself, many recent works utilize inexpensive and high-resolution vehicle-mounted multi-view cameras to obtain bird's-eye views. The most straightforward method is to directly project the image obtained from the multi-view camera onto the bird's-eye view plane. While this method is simple and effective, objects above the bird's-eye view plane suffer severe distortion due to the planar assumptions it makes. Therefore, many recent works have shifted towards first obtaining feature maps of the bird's-eye view and then using these feature maps to obtain the bird's-eye view.

[0048] The main semantic segmentation methods based on bird's-eye view images include those based on geometric projection and those based on Transformer. Transformer-based methods can model long-distance dependencies, but the global attention mechanism brings huge computational costs, and these methods do not effectively utilize the geometric priors provided by the camera's parameter matrix.

[0049] To address the aforementioned problems, this invention provides a semantic segmentation method for bird's-eye view images based on geometric priors. Figure 1This is one of the flowcharts illustrating the bird's-eye view semantic segmentation method based on geometric priors provided by this invention, such as... Figure 1 As shown, the method includes:

[0050] Step 110: Obtain the query features of the image to be processed and the bird's-eye view.

[0051] Specifically, the features of the image to be processed and the bird's-eye view can be obtained. The image to be processed here is the image that needs to be semantically segmented later. The image to be processed here can be an image acquired by an image acquisition device such as an in-vehicle camera, mobile phone, camera, or tablet. For example, the image to be processed can be multiple images taken from multiple perspectives at the same vehicle position. The multiple perspectives here can be front, rear, left front, left rear, right front, and right rear perspectives. Correspondingly, the image to be processed can include a front perspective image, a rear perspective image, a left front perspective image, a left rear perspective image, a right front perspective image, and a right rear perspective image. This embodiment of the invention does not specifically limit this.

[0052] The size of the image to be processed here can be 224×480, etc., and the bird's-eye view query feature here can be a pre-defined feature, which can be represented as Q∈R. d×H×W .

[0053] Step 120: Input the image to be processed into the feature extractor to obtain the image features output by the feature extractor.

[0054] Specifically, after acquiring the image to be processed, the image can be input into a feature extractor to obtain the image features output by the feature extractor. The feature extractor may include a first convolutional layer, a first activation layer, and a first normalization layer connected in sequence. The first convolutional layer can be a cascaded multilayer convolutional neural network (CNN), a deep neural network (DNN), or a combination of CNN and DNN, etc. This embodiment of the invention does not specifically limit this.

[0055] The first activation layer here can use the GELU (Gaussian Error Linear Unit) activation function, the Sigmoid activation function, or the ReLU (Rectified Linear Units) activation function. The first normalization layer here can be LN (Layer Normalization), BN (Batch Normalization), or IN (Instance Normalization), etc. The embodiments of the present invention do not specifically limit this.

[0056] The feature extractor here can also be the EfficientNet-B4 model, etc., and the embodiments of the present invention do not specifically limit it.

[0057] Here, the image features output by the feature extractor can be multi-scale, for example, 1 / 4, 1 / 8, 1 / 16, 1 / 32, etc. The embodiments of the present invention do not specifically limit this.

[0058] Step 130: Input the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model; the feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

[0059] Specifically, after obtaining the image features, the image features and the bird's-eye view query features can be input into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model.

[0060] The feature extraction model here may include multiple cascaded feature extraction modules, which may include self-attention modules. The feature extraction modules may be Transformer models, LSTM (Long Short Term Memory networks), RNN (Recurrent Neural Networks), etc. The embodiments of the present invention do not specifically limit them.

[0061] The self-attention module here is used to perform cross-view self-attention transformation between the token of the image to be processed and the bird's-eye view query features. The self-attention module here can be a cross-attention module. The token of the image to be processed here refers to the image to be processed being divided into different image blocks in the manner of a sliding window, and each image block is encoded.

[0062] The cross-view self-attention transformation here refers to calculating the similarity between the token of the image to be processed and the bird's-eye view query features, constructing an attention matrix, and transforming the bird's-eye view query features to obtain features optimized by the self-attention mechanism.

[0063] The bird's-eye view features here reflect the characteristic information at the bird's-eye view level.

[0064] Here, when outputting the bird's-eye view features, the camera's intrinsic and extrinsic parameter matrices are also taken into account. These intrinsic and extrinsic parameter matrices can be used for geometric priors.

[0065] Understandably, in Transformer-based bird's-eye view semantic segmentation methods, the attention mechanism used is global, and its computational complexity is directly proportional to the number of input viewpoints, the resolution of the feature maps, and the resolution of the bird's-eye view query features. However, performing cross-view self-attention transformation on the tokens of the image to be processed and the bird's-eye view query features can reduce the computational complexity of the self-attention mechanism, further reducing the complexity of image feature extraction. Furthermore, it improves the accuracy and reliability of the obtained bird's-eye view features, thereby further enhancing the accuracy and reliability of subsequent semantic segmentation.

[0066] Step 140: Based on the bird's-eye view features, perform semantic segmentation on the image to be processed.

[0067] Specifically, after obtaining the bird's-eye view features, semantic segmentation can be performed on the image to be processed based on the bird's-eye view features.

[0068] It is understandable that the bird's-eye view features output by the feature extraction model are features after cross-view self-attention transformation. Semantic segmentation is then performed based on these bird's-eye view features, which further improves the accuracy and reliability of semantic segmentation.

[0069] The method provided in this invention, with its self-attention module in the feature extraction model, performs cross-view self-attention transformation on the tokens and bird's-eye view query features of the image to be processed. This reduces the computational complexity of the self-attention mechanism, further reduces the complexity of image feature extraction, and improves the accuracy and reliability of the obtained bird's-eye view features, thereby further improving the accuracy and reliability of subsequent semantic segmentation.

[0070] Based on the above embodiments, the feature extraction model includes multiple cascaded feature extraction modules;

[0071] Step 130 includes:

[0072] Step 131: Input the previous token and the previous bird's-eye view query feature of the image to be processed into the current feature extraction module to obtain the current bird's-eye view query feature output by the current feature extraction module. The previous bird's-eye view query feature is the feature extraction module output before the current feature extraction module.

[0073] Step 132: Use the bird's-eye view query features output by the last feature extraction module as the bird's-eye view features.

[0074] Specifically, the feature extraction model includes multiple cascaded feature extraction modules. These feature extraction modules can be Transformer models, LSTM models, RNN models, etc., and the embodiments of the present invention do not specifically limit them.

[0075] Here, the spatial resolution of the cascaded feature extraction modules can be arranged from high to low.

[0076] In the process of extracting features from a bird's-eye view, the first feature extraction module in the feature extraction model can be selected as the current feature extraction module, and the process of extracting bird's-eye view features can be executed:

[0077] The previous token and the previous bird's-eye view query features of the image to be processed can be input into the current feature extraction module to obtain the current bird's-eye view query features output by the current feature extraction module. The previous bird's-eye view query features here are the outputs of the feature extraction modules before the current feature extraction module. The previous token here can be the image features output by the convolutional encoder, for example, it can be the 1 / 4 scale image features output by the convolutional encoder.

[0078] After obtaining the current bird's-eye view query features output by the current feature extraction module, the next feature extraction module (i.e., the second-ranked feature extraction module) can be used as the current feature extraction module, and the process of extracting bird's-eye view features can be returned.

[0079] That is, after obtaining the current bird's-eye view query features output by the first feature extraction module, the current bird's-eye view query features and the current token of the image to be processed can be input into the current feature extraction module to obtain the current bird's-eye view query features output by the current feature extraction module. Here, the current token can be the image features output by the convolutional encoder, for example, it can be the 1 / 8 scale image features output by the convolutional encoder.

[0080] The process of extracting bird's-eye view features by using the third-ranked feature extraction module as the current feature extraction module is similar to the process of extracting bird's-eye view features by using the second-ranked feature extraction module as the current feature extraction module, and will not be repeated here.

[0081] Following this pattern, we know that the current feature extraction module is the last feature extraction module in the feature extraction model. Here, the last feature extraction module is the last feature extraction module in the feature extraction model.

[0082] After the current bird's-eye view query features are extracted by the feature extraction module at the very end, the current bird's-eye view query features extracted by the feature extraction module at the very end can be used as the bird's-eye view features.

[0083] The method provided in this invention includes a feature extraction model comprising multiple cascaded feature extraction modules. The resulting bird's-eye view features are the result of feature extraction operations executed sequentially by the multiple feature extraction modules, thereby improving the accuracy and reliability of the bird's-eye view features and further enhancing the accuracy and reliability of subsequent semantic segmentation of the image to be processed based on the bird's-eye view features.

[0084] Based on the above embodiments, step 131 includes:

[0085] Step 1311: Input the previous token and the previous bird's-eye view query feature of the image to be processed into the self-attention module of the current feature extraction module. The self-attention module samples the bird's-eye view query feature to obtain the sampled bird's-eye view query feature. Then, the sampled bird's-eye view query feature is inversely sampled to obtain the inverse sampled bird's-eye view query feature output by the self-attention module.

[0086] Step 1312: Input the inverse sampled bird's-eye view query features into the feedforward propagation network of the current feature extraction module to obtain the current bird's-eye view query features output by the feedforward propagation network.

[0087] Specifically, Figure 2 This is a schematic diagram of the process for sampling features for bird's-eye view queries provided by the present invention, such as... Figure 2 As shown, the previous token and the previous bird's-eye view query feature of the image to be processed can be input into the self-attention module of the current feature extraction module. The self-attention module samples the bird's-eye view query feature to obtain the sampled bird's-eye view query feature. Q11 represents the first token of the bird's-eye view query feature Q1 corresponding to the first view, and so on. Q14 represents the fourth token of the bird's-eye view query feature Q1 corresponding to the first view, and Qi1 represents the first token of the bird's-eye view query feature Qi corresponding to the i-th view, and so on.

[0088] The self-attention module here can be a patch attention module (PA), an image attention module (IA), or a scene attention module (SA). This embodiment of the invention does not specifically limit it in this way.

[0089] For example, based on the intrinsic parameter matrix, extrinsic parameter matrix, and the coordinates of feature anchor points queried from the bird's-eye view, the features queried from the bird's-eye view can be projected onto the camera plane, as shown in the following formula:

[0090]

[0091] Where I represents the image coordinate system, W represents the world coordinate system, x represents the coordinate itself, and K represents the image coordinate system. i R represents the intrinsic parameter matrix. i This represents the extrinsic parameter matrix. Based on the coordinate values ​​obtained from the projection, the coordinates closest to the center point of the projection plane are sampled, and the sampled bird's-eye view query features are obtained based on these coordinates, as shown in the following formula:

[0092]

[0093] Q i ={q j |j∈Idx i}

[0094] Among them, Q i This indicates that the attention module is sampling the bird's-eye view query features, where c represents the bird's-eye view query features corresponding to the i-th view of the image to be processed. i This represents the coordinates of the center point of the i-th view of the image to be processed.

[0095] In addition, the query features of the sampled bird's-eye view can be sampled once, as shown in the following formula:

[0096]

[0097] Q ip ={q j |j∈Idx ip}

[0098] Among them, Q ip This indicates that the attention module is sampling the bird's-eye view query features. The p-th image patch of the i-th view of the image to be processed represents the sampled bird's-eye view query feature corresponding to the previous token of the image to be processed. c ip This represents the coordinates of the center point of the p-th image block in the i-th view of the image to be processed, which is the coordinates of the center point of the previous token of the image to be processed.

[0099] Figure 3 This is a schematic diagram of the process for inverse sampling of query features from a sampled bird's-eye view, as provided by the present invention. Figure 3 As shown, after obtaining the sampled bird's-eye view query features, the sampled bird's-eye view query features can be inversely sampled to obtain the inversely sampled bird's-eye view query features output by the self-attention module. The inversely sampled bird's-eye view query features here reflect the feature information at the bird's-eye view level.

[0100] The self-attention module here can be an image self-attention module, a scene self-attention module, or both an image self-attention module and a scene self-attention module. This embodiment of the invention does not specifically limit this.

[0101] That is, the image self-attention module and scene self-attention module in the self-attention module can be used to propagate the updated sampled bird's-eye view query features to the entire bird's-eye view query features, and obtain the inverse sampled bird's-eye view query features output by the self-attention module.

[0102] After obtaining the inverse-sampled bird's-eye view query features, these features can be input into the feed forward network (FFN) of the current feature extraction module to obtain the current bird's-eye view query features output by the feed forward network.

[0103] Based on the above embodiments, Figure 4 This is a schematic diagram of the self-attention module provided by the present invention, as shown below. Figure 4 As shown, the self-attention module includes a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. The image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query feature to obtain a first bird's-eye view query feature. The image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query feature to obtain a second bird's-eye view query feature. The scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query feature.

[0104] Specifically, the self-attention module may include a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. Here, the image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query features to obtain the first bird's-eye view query features. Here, the image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query features to obtain the second bird's-eye view query features. Here, the scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query features to obtain the bird's-eye view features.

[0105] Here, after the image patch self-attention module, a feedforward propagation network is connected, which outputs the first bird's-eye view query features. Similarly, after the image self-attention module, a feedforward propagation network is connected, which outputs the second bird's-eye view query features. After the scene self-attention module, a feedforward propagation network is also connected, which outputs the bird's-eye view features.

[0106] The image patch self-attention module, image self-attention module, and scene self-attention module mentioned here can all be cross-attention modules, etc., and the embodiments of the present invention do not make specific limitations on this.

[0107] Based on the above embodiments, the feature extractor includes a first convolutional layer, a first activation layer, and a first normalization layer connected in sequence, and the feature extractor is used to output multi-scale image features.

[0108] Specifically, the feature extractor here includes a first convolutional layer, a first activation layer, and a first normalization layer connected in sequence. The first convolutional layer here can be a deep fully convolutional network (Deep-ConvNet) or a fully convolutional network (FCN), etc. The embodiments of the present invention do not specifically limit this.

[0109] The first activation layer here can use the GELU activation function, the Sigmoid activation function, or the ReLU activation function. The first normalization layer here can be LN, BN, or IN, etc. The embodiments of the present invention do not make specific limitations on this.

[0110] The feature extractor here is used to output multi-scale image features, such as 1 / 4, 1 / 8, 1 / 16, 1 / 32, etc. The embodiments of the present invention do not specifically limit this.

[0111] For example, let the image to be processed be x∈R 6×3×H×W The input is fed into the feature extractor, and the image features z at four scales are obtained from the output of the feature extractor. i ∈R 6×c×h×w i = 1, 2, 3, 4.

[0112] Based on the above embodiments, step 140 includes:

[0113] The bird's-eye view features are input into the decoder, which outputs the semantic segmentation result of the image to be processed. The decoder includes a second convolutional layer, a second normalization layer, and a second activation layer connected in sequence.

[0114] Specifically, the bird's-eye view features can be input into the decoder, and the decoder outputs the semantic segmentation result of the image to be processed. The decoder here can be a convolutional decoder, which can include a second convolutional layer, a second normalization layer and a second activation layer connected in sequence. The second convolutional layer here can be a deep fully convolutional network or a fully convolutional network, etc. The embodiments of the present invention do not specifically limit this.

[0115] The second activation layer here can use the GELU activation function, the Sigmoid activation function, or the ReLU activation function. The second normalization layer here can be LN, BN, or IN, etc. The embodiments of the present invention do not make specific limitations on this.

[0116] Here, the second convolutional layer may be the same as or different from the first convolutional layer; the second activation layer may be the same as or different from the first activation layer; and the second normalization layer may be the same as or different from the first normalization layer. This embodiment of the invention does not impose specific limitations on these aspects.

[0117] Based on the above embodiments, step 130 includes:

[0118] Step 310: Obtain the intrinsic and extrinsic parameter matrices of the camera, which are used for geometric priors;

[0119] Step 320: Based on the intrinsic and extrinsic parameter matrices of the camera, as well as the image features and the bird's-eye view query features, the bird's-eye view features output by the feature extraction model are obtained.

[0120] Specifically, the intrinsic and extrinsic parameter matrices of the camera can be obtained, and these matrices are used for geometric priors.

[0121] In autonomous driving perception applications, the camera mentioned here can be an in-vehicle camera or the like, and this embodiment of the invention does not specifically limit it.

[0122] The intrinsic parameter matrix of the camera here reflects the camera's own properties. The intrinsic parameter matrix of each camera is different, and calibration is required to know these parameters. The intrinsic parameter matrix of the camera describes the relationship between the points of the object and the points of the image.

[0123] The extrinsic parameter matrix of the camera here is the transformation from the world coordinate system to the camera coordinate system.

[0124] After obtaining the camera's intrinsic and extrinsic parameter matrices, the bird's-eye view features output by the feature extraction model can be obtained based on the camera's intrinsic and extrinsic parameter matrices, as well as image features and bird's-eye view query features.

[0125] For example, based on the intrinsic parameter matrix, extrinsic parameter matrix, and the coordinates of feature anchor points queried from the bird's-eye view, the features queried from the bird's-eye view can be projected onto the camera plane, as shown in the following formula:

[0126]

[0127] Where I represents the image coordinate system, W represents the world coordinate system, x represents the coordinate itself, and K represents the image coordinate system. i R represents the intrinsic parameter matrix. i This represents the extrinsic parameter matrix. Based on the coordinate values ​​obtained from the projection, the coordinates closest to the center point of the projection plane are sampled, and the bird's-eye view features are obtained based on these coordinates.

[0128] Based on any of the above embodiments Figure 5This is the second flowchart of the bird's-eye view semantic segmentation method based on geometric priors provided by this invention, as shown below. Figure 5 As shown, the method includes:

[0129] The first step is to obtain the query features of the image to be processed and the bird's-eye view.

[0130] The second step is to input the image to be processed into the feature extractor to obtain the image features output by the feature extractor. The feature extractor here may include a first convolutional layer, a first activation layer and a first normalization layer connected in sequence. The feature extractor here is used to output multi-scale image features.

[0131] The third step involves inputting the previous token and the previous bird's-eye view query features of the image to be processed into the self-attention module of the current feature extraction module in the feature extraction model. The self-attention module samples the bird's-eye view query features to obtain sampled bird's-eye view query features. Then, the sampled bird's-eye view query features are inversely sampled to obtain the inversely sampled bird's-eye view query features output by the self-attention module.

[0132] The inverse-sampled bird's-eye view query features are input into the feedforward propagation network of the current feature extraction module, resulting in the current bird's-eye view query features output by the feedforward propagation network. The previous bird's-eye view query features here are the outputs of the feature extraction modules preceding the current feature extraction module.

[0133] The bird's-eye view query features output by the last feature extraction module are used as the bird's-eye view features.

[0134] The fourth step is to perform semantic segmentation on the image to be processed based on the features of the bird's-eye view.

[0135] The feature extraction model here includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

[0136] The self-attention module here may include a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. The image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query features to obtain the first bird's-eye view query features. The image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query features to obtain the second bird's-eye view query features. The scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query features.

[0137] The following describes the geometric prior-based bird's-eye view semantic segmentation device provided by the present invention. The geometric prior-based bird's-eye view semantic segmentation device described below and the geometric prior-based bird's-eye view semantic segmentation method described above can be referred to in correspondence.

[0138] Based on any of the above embodiments, the present invention provides a bird's-eye view semantic segmentation device based on geometric prior. Figure 6 This is a schematic diagram of the structure of the bird's-eye view semantic segmentation device based on geometric prior provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0139] The acquisition unit 610 is used to acquire query features of the image to be processed and the bird's-eye view;

[0140] Image feature extraction unit 620 is used to input the image to be processed into the feature extractor to obtain the image features output by the feature extractor;

[0141] The bird's-eye view feature extraction unit 630 is used to input the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model;

[0142] Semantic segmentation unit 640 is used to perform semantic segmentation on the image to be processed based on the bird's-eye view features;

[0143] The feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

[0144] The apparatus provided in this invention includes a self-attention module in the feature extraction model that performs cross-view self-attention transformation on the tokens and bird's-eye view query features of the image to be processed. This reduces the computational complexity of the self-attention mechanism, further reduces the complexity of image feature extraction, and improves the accuracy and reliability of the obtained bird's-eye view features, thereby further improving the accuracy and reliability of subsequent semantic segmentation.

[0145] Based on any of the above embodiments, the feature extraction model includes multiple cascaded feature extraction modules;

[0146] The extraction of bird's-eye view feature units is specifically used for:

[0147] The current feature extraction unit is used to input the previous token and the previous bird's-eye view query features of the image to be processed into the current feature extraction module to obtain the current bird's-eye view query features output by the current feature extraction module. The previous bird's-eye view query features are output by the feature extraction modules before the current feature extraction module.

[0148] The bird's-eye view feature subunit is used to take the bird's-eye view query features output by the last feature extraction module as the bird's-eye view features.

[0149] Based on any of the above embodiments, the current feature extraction unit is specifically used for:

[0150] The previous token and the previous bird's-eye view query features of the image to be processed are input into the self-attention module of the current feature extraction module. The self-attention module samples the bird's-eye view query features to obtain sampled bird's-eye view query features. Then, the sampled bird's-eye view query features are inversely sampled to obtain the inversely sampled bird's-eye view query features output by the self-attention module.

[0151] The inverse-sampled bird's-eye view query features are input into the feedforward propagation network of the current feature extraction module to obtain the current bird's-eye view query features output by the feedforward propagation network.

[0152] Based on any of the above embodiments, the self-attention module includes a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. The image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query feature to obtain a first bird's-eye view query feature. The image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query feature to obtain a second bird's-eye view query feature. The scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query feature.

[0153] Based on any of the above embodiments, the feature extractor includes a first convolutional layer, a first activation layer, and a first normalization layer connected in sequence, and the feature extractor is used to output multi-scale image features.

[0154] Based on any of the above embodiments, the step of performing semantic segmentation on the image to be processed based on the bird's-eye view features includes:

[0155] The bird's-eye view features are input into the decoder, which outputs the semantic segmentation result of the image to be processed. The decoder includes a second convolutional layer, a second normalization layer, and a second activation layer connected in sequence.

[0156] Based on any of the above embodiments, the step of inputting the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model includes:

[0157] Obtain the intrinsic and extrinsic parameter matrices of the camera, which are used for geometric priors;

[0158] Based on the camera's intrinsic and extrinsic parameter matrices, as well as the image features and the bird's-eye view query features, the bird's-eye view features output by the feature extraction model are obtained.

[0159] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a bird's-eye view semantic segmentation method based on geometric priors. This method includes: acquiring an image to be processed and bird's-eye view query features; inputting the image to be processed into a feature extractor to obtain image features output by the feature extractor; inputting the image features and the bird's-eye view query features into a feature extraction model to obtain bird's-eye view features output by the feature extraction model; and performing semantic segmentation on the image to be processed based on the bird's-eye view features. The feature extraction model includes a self-attention module, which performs cross-view self-attention transformation on the tokens of the image to be processed and the bird's-eye view query features.

[0160] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the geometric prior-based bird's-eye view semantic segmentation method provided by the above methods. The method includes: acquiring an image to be processed and bird's-eye view query features; inputting the image to be processed into a feature extractor to obtain image features output by the feature extractor; inputting the image features and the bird's-eye view query features into a feature extraction model to obtain bird's-eye view features output by the feature extraction model; performing semantic segmentation on the image to be processed based on the bird's-eye view features; the feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the tokens of the image to be processed and the bird's-eye view query features.

[0162] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the geometric prior-based bird's-eye view semantic segmentation method provided by the above methods. The method includes: acquiring an image to be processed and bird's-eye view query features; inputting the image to be processed into a feature extractor to obtain image features output by the feature extractor; inputting the image features and the bird's-eye view query features into a feature extraction model to obtain bird's-eye view features output by the feature extraction model; performing semantic segmentation on the image to be processed based on the bird's-eye view features; the feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the tokens of the image to be processed and the bird's-eye view query features.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic segmentation method for bird's-eye view images based on geometric priors, characterized in that, include: Obtain query features from the image to be processed and the bird's-eye view; The image to be processed is input into a feature extractor to obtain the image features output by the feature extractor; The image features and the bird's-eye view query features are input into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model; Based on the bird's-eye view features, semantic segmentation is performed on the image to be processed; The feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

2. The semantic segmentation method for bird's-eye view based on geometric priors according to claim 1, characterized in that, The feature extraction model includes multiple cascaded feature extraction modules; The step of inputting the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model includes: The previous token and the previous bird's-eye view query feature of the image to be processed are input into the current feature extraction module to obtain the current bird's-eye view query feature output by the current feature extraction module. The previous bird's-eye view query feature is the feature extraction module output before the current feature extraction module. The bird's-eye view query features output by the last feature extraction module are used as the bird's-eye view features.

3. The semantic segmentation method for bird's-eye view based on geometric priors according to claim 2, characterized in that, The step of inputting the previous token and the previous bird's-eye view query features of the image to be processed into the current feature extraction module to obtain the current bird's-eye view query features output by the current feature extraction module includes: The previous token and the previous bird's-eye view query features of the image to be processed are input into the self-attention module of the current feature extraction module. The self-attention module samples the bird's-eye view query features to obtain sampled bird's-eye view query features. Then, the sampled bird's-eye view query features are inversely sampled to obtain the inversely sampled bird's-eye view query features output by the self-attention module. The inverse-sampled bird's-eye view query features are input into the feedforward propagation network of the current feature extraction module to obtain the current bird's-eye view query features output by the feedforward propagation network.

4. The bird's-eye view semantic segmentation method based on geometric priors according to any one of claims 1 to 3, characterized in that, The self-attention module includes a cascaded image patch self-attention module, an image self-attention module, and a scene self-attention module. The image patch self-attention module is used to perform cross-view self-attention transformation on the image patches of the bird's-eye view query feature to obtain a first bird's-eye view query feature. The image self-attention module is used to perform cross-view self-attention transformation on the first bird's-eye view query feature to obtain a second bird's-eye view query feature. The scene self-attention module is used to perform cross-view self-attention transformation on the second bird's-eye view query feature.

5. The semantic segmentation method for bird's-eye view based on geometric priors according to claim 1, characterized in that, The feature extractor includes a first convolutional layer, a first activation layer, and a first normalization layer connected in sequence, and is used to output multi-scale image features.

6. The semantic segmentation method for bird's-eye view based on geometric priors according to claim 1, characterized in that, The semantic segmentation of the image to be processed based on the bird's-eye view features includes: The bird's-eye view features are input into the decoder, which outputs the semantic segmentation result of the image to be processed. The decoder includes a second convolutional layer, a second normalization layer, and a second activation layer connected in sequence.

7. The semantic segmentation method for bird's-eye view based on geometric priors according to claim 1, characterized in that, The step of inputting the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model includes: Obtain the intrinsic and extrinsic parameter matrices of the camera, which are used for geometric priors; Based on the camera's intrinsic and extrinsic parameter matrices, as well as the image features and the bird's-eye view query features, the bird's-eye view features output by the feature extraction model are obtained.

8. A semantic segmentation device for bird's-eye view based on geometric priors, characterized in that, include: The acquisition unit is used to acquire query features of the image to be processed and the bird's-eye view; An image feature extraction unit is used to input the image to be processed into a feature extractor and obtain the image features output by the feature extractor. A bird's-eye view feature extraction unit is used to input the image features and the bird's-eye view query features into the feature extraction model to obtain the bird's-eye view features output by the feature extraction model; A semantic segmentation unit is used to perform semantic segmentation on the image to be processed based on the bird's-eye view features; The feature extraction model includes a self-attention module, which is used to perform cross-view self-attention transformation on the token of the image to be processed and the bird's-eye view query features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the bird's-eye view semantic segmentation method based on geometric priors as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the bird's-eye view semantic segmentation method based on geometric priors as described in any one of claims 1 to 7.

Images