Online map construction method and system based on inverse perspective mapping and synchronous learning

Abstract

The invention relates to an online map construction method and system based on inverse perspective mapping and synchronous learning. A perspective image is obtained and preprocessed, a multi-scale inverse perspective mapping image is obtained through transformation, and global aerial view features are obtained through multi-scale strip convolution decoding, strip convolution residual connection in different directions, layer-by-layer decoding and feature fusion; obtaining perspective features of the preprocessed perspective image through a synchronous learning sequence and a semantic segmentation network; and converting the perspective features into a bird's-eye view coordinate system, aligning the perspective features with the global bird's-eye view features in space to obtain enhanced perspective features, fusing the global bird's-eye view features and the enhanced perspective features through convolution operation, and outputting the fused features to a current frame of the constructed map through a decoder. Inverse perspective mapping and synchronous learning are combined, multi-frame aerial view features and perspective features are fused, a road structure under an aerial view angle is reserved, and environmental details in a perspective image are supplemented in a perspective mode.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction, and in particular to an online map construction method and system based on reverse perspective mapping and synchronous learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The map data that needs to be updated in real time during the realization of vehicle autonomous driving is called online map, which contains detailed road information, traffic signs, lane divisions, obstacle locations, etc. These maps are usually created and updated using sensor data (such as lidar, cameras, etc.). In addition to being applied in autonomous driving scenarios, the relevant online map construction methods can also be applied in fields such as intelligent transportation.

[0004] PV-BEV feature conversion, i.e., the conversion of perspective view (PV) features to bird's-eye view (BEV) features, is a core problem in online map construction. Since the camera is usually placed on a vehicle parallel to the ground and facing the outside environment, the image is captured in a perspective view, which is orthogonal to the BEV, making the conversion between the two views difficult. Since points in space can be converted to image space by perspective mapping, the inverse problem of projecting image pixels into 3D space is undecidable.

[0005] Inverse Perspective Mapping (IPM) is a method proposed to solve the mathematically impossible mapping problem under the additional constraint that the inverse mapped points are located on the horizontal plane. The transformation applies camera rotation homography and anisotropic scaling. The homography matrix can be physically derived from the intrinsic and extrinsic parameters of the camera. Some methods use convolutional neural networks to extract semantic features of perspective images and estimate the vertical vanishing point and horizon (ground plane vanishing line) in the image to determine the homography matrix. However, this method is usually unable to accurately detect targets above the ground plane, such as 3D objects such as buildings, vehicles and pedestrians, which will produce severe geometric distortions. At the same time, since the IPM transformation can only capture information on the road plane and lacks interaction with dynamic or static objects above the road surface, it leads to the sparsity of feature information, which limits the comprehensive understanding of the environment. Summary of the invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides an online map construction method and system based on inverse perspective mapping and synchronous learning, which applies multi-scale strip convolution to extract global and local information under the BEV perspective, and synchronously learns the sequence to generate perspective features under the perspective perspective, which are collaboratively integrated through the dual feature fusion module, and the perspective complementary map learning is used to improve the joint learning ability and enhance the model generalization ability.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides online map construction based on reverse perspective mapping and synchronous learning, comprising the following steps:

[0009] Acquire and preprocess the perspective image, and obtain a multi-scale inverse perspective mapping image through transformation;

[0010] The obtained inverse perspective mapping image is decoded by multi-scale strip convolution, and the strip convolution residual connection in different directions is used. After layer-by-layer decoding and feature fusion, the global bird's-eye view feature is obtained;

[0011] After preprocessing, the perspective image is subjected to synchronous learning sequence and semantic segmentation network to obtain perspective features;

[0012] The perspective feature is converted to a bird's-eye view coordinate system and spatially aligned with the global bird's-eye view feature to obtain an enhanced perspective feature. The global bird's-eye view feature and the enhanced perspective feature are fused through a convolution operation. The fused feature is output as the current frame of the constructed map through a decoder.

[0013] Among them, layer-by-layer decoding and feature fusion are specifically as follows: the features of the reverse perspective mapping image of the current layer are channel-adjusted, and features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution results in four different directions, which are then connected in series in the channel dimension to form a fusion feature map in the corresponding direction, which is then fused with the features of the reverse perspective mapping image of the next layer to obtain the decoding features of the current layer and serve as the input of the next layer, until the features of the reverse perspective mapping image of the last layer are fused with the decoding features of the previous layer, and features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution results in four different directions, which are then connected in series in the channel dimension to obtain the global bird's-eye view features.

[0014] Furthermore, the preprocessed perspective image is transformed to obtain a multi-scale inverse perspective mapping image. Specifically, the preprocessed perspective image is transformed into a multi-scale inverse perspective mapping image through image scale transformation and then inverse perspective transformation.

[0015] Furthermore, the preprocessed perspective image is transformed to obtain a multi-scale inverse perspective mapping image. Specifically, the preprocessed perspective image is a multi-view perspective image I n And the camera's internal parameters and external reference Inverse perspective mapping image It is calculated by assuming the height h, as shown below:

[0016]

[0017] Proj(X i ,Y j )=∑ n I n (u,v), if (u <imH,v<imW);

[0018]

[0019] Among them, Plane represents the collection of all grids, Z c is the depth value in the camera coordinate system, u and v, u0 and v0 are the values ​​in the pixel coordinate system, n is the number of cameras, imH and imW are the perspective images I n The size of the image, H and W are the sizes of the original image.

[0020] Furthermore, the multi-scale inverse perspective mapping image has multiple layers, and the features of the multi-layer inverse perspective mapping image are obtained by using an encoder, and the global bird's-eye view features are obtained through layer-by-layer decoding and feature fusion.

[0021] Further, layer-by-layer decoding and feature fusion, including the fourth layer decoding, are as follows:

[0022] Input feature map By reducing the number of channels through the convolution layer, we get the adjusted feature map F4:

[0023] F4 is used to extract features in the vertical, horizontal and diagonal directions to obtain strip convolution results in four different directions. Fuse in the channel dimension to form a directional fusion feature map d4:

[0024] The processed feature map d 4f The feature map of the third layer encoder Fusion forms the decoding feature d4 of the current layer.

[0025] Furthermore, layer-by-layer decoding and feature fusion also include third-layer decoding, specifically:

[0026] Channel adjustment: Upsample the decoded feature d4 of the previous layer to match the encoder feature of the third layer The resolution of the adjusted feature d 3up ;

[0027] D 3upFeatures are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and then fused on the channel to obtain the fused feature d 3f ;

[0028] D 3f With the input features of the next layer Fusion forms the decoding feature d3 of the current layer.

[0029] Furthermore, layer-by-layer decoding and feature fusion also include second-layer decoding, specifically:

[0030] Channel adjustment: Upsample the previous layer decoding feature d3 to obtain the adjusted feature d 2up ;

[0031] D 2up Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and then fused on the channel to obtain the fused feature d 2f ;

[0032] D 2f With the input features of the next layer Fusion forms the current layer decoding feature d2.

[0033] Furthermore, layer-by-layer decoding and feature fusion also include first-layer decoding, specifically:

[0034] After upsampling the previous layer feature d2, features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, which are fused on the channel to obtain the fused feature F with improved resolution. o , as the global bird's-eye view feature of the output.

[0035] Furthermore, the perspective feature is transformed into a bird's-eye view coordinate system and spatially aligned with the global bird's-eye view feature to obtain an enhanced perspective feature, specifically:

[0036] The perspective feature F is transformed into pv Features converted to bird's-eye view coordinates

[0037] Features in bird's eye view space Perform geometric operations to ensure that the feature F o The convolutional blocks are used to process the connected features and obtain the enhanced features F. me .

[0038] Furthermore, the global bird's-eye view features and the enhanced perspective features are fused through convolution operations, specifically:

[0039] The enhanced features of the first three frames are introduced, and the fused features of the first three frames are converted to the current frame perspective through the vehicle motion matrix. The features of different time steps are aligned, and the multi-frame features are temporally fused. The obtained fused features are output by the decoder to the current frame of the constructed map.

[0040] A second aspect of the present invention provides a system for implementing the above method, comprising:

[0041] The data acquisition and preprocessing module is configured to: acquire and preprocess the perspective image, and obtain a multi-scale inverse perspective mapping image through transformation;

[0042] The bird's-eye view feature module is configured as follows: the obtained inverse perspective mapping image is decoded by multi-scale strip convolution, and strip convolution residual connections in different directions are used to obtain global bird's-eye view features through layer-by-layer decoding and feature fusion;

[0043] The perspective feature module is configured as follows: the perspective image after preprocessing is subjected to synchronous learning sequence and semantic segmentation network to obtain perspective features;

[0044] The fusion output module is configured to: convert the perspective feature into a bird's-eye view coordinate system, and align it spatially with the global bird's-eye view feature to obtain an enhanced perspective feature, fuse the global bird's-eye view feature and the enhanced perspective feature through a convolution operation, and output the fused feature to the current frame of the constructed map through a decoder.

[0045] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in online map construction based on inverse perspective mapping and synchronous learning as described above.

[0046] A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in online map construction based on inverse perspective mapping and synchronous learning as described above are implemented.

[0047] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the above-mentioned online map construction based on reverse perspective mapping and synchronous learning.

[0048] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects:

[0049] 1. The BEV (bird's eye view) feature can provide global and local information on the road plane, and is suitable for capturing the overall road structure and static elements such as lane boundaries, pedestrians crossing the street and obstacles. The perspective (PV) feature captures the relationship between static and dynamic objects from the camera's perspective, and contains semantic information about objects above and the surrounding environment. It can make up for the lack of BEV features in terms of height and object context. By fusing the two types of features, more comprehensive scene information can be obtained. This combination improves the accuracy of overall map construction and effectively reduces the information loss that may be caused by a single perspective. At the same time, perspective features help to more accurately detect dynamic objects (such as vehicles and pedestrians) around the road, and maintain the comprehensiveness and dynamism of environmental understanding in real-time map updates.

[0050] 2. Apply multi-scale strip convolution to extract global and local information from the BEV perspective, synchronously learn the sequence to generate perspective features from the perspective, synergistically integrate through the dual feature fusion module, and improve the joint learning ability through perspective complementary map learning to enhance the model generalization ability.

[0051] 3. Considering that when only the features of a single frame are fused, the semantic relevance of multiple frames is insufficient, which may lead to the lack of persistence of some long-term stable static scene information (such as road signs, traffic lights, etc.), the BEV features and perspective features of multiple frames are introduced to fuse and weighted integration of similar semantic information in multiple frames is performed, thereby retaining some long-term stable static semantic features and reducing the impact of dynamic object interference on static scene understanding. The fused features not only retain the clarity of the road structure from the BEV perspective, but also supplement the environmental details in the perspective image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0053] Figure 1 It is a schematic diagram of an online map construction process based on reverse perspective mapping and synchronous learning provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0056] Embodiment 1:

[0057] Since the BEV (bird's eye view) feature can provide global and local information on the road plane, it is suitable for capturing the overall road structure and static elements such as lane boundaries, pedestrians crossing the street and obstacles. The perspective (PV) feature captures the relationship between static and dynamic objects from the camera's perspective, including semantic information of objects above and the surrounding environment, which can make up for the lack of BEV features in terms of height and object context. By fusing the two types of features, more comprehensive scene information can be obtained.

[0058] For example, perspective features can make up for the shortcomings of BEV features in identifying three-dimensional objects (such as buildings, trees, etc.) and provide rich spatial relationship information. This combination improves the accuracy of overall map construction and effectively reduces the information loss that may be caused by a single perspective. At the same time, perspective features help to more accurately detect dynamic objects (such as vehicles and pedestrians) around the road, and maintain the comprehensiveness and dynamism of environmental understanding in real-time map updates. Therefore, this embodiment proposes a synchronous learning strategy based on dual perspectives, which applies a multi-scale strip convolutional network to extract global and local information under the BEV perspective, and synchronously learns the sequence to generate perspective features under the perspective perspective, which are collaboratively integrated through the dual feature fusion module, and the perspective complementary map learning is used to improve the joint learning ability and enhance the model generalization ability. The specific process is as follows: Figure 1 shown.

[0059] Step S1: Data preprocessing: Two representative datasets, nuScenes and Argoverse, were selected, and the image pairs in each dataset were divided into training image pairs and test image pairs.

[0060] Step S2: Image IPM transformation: The initial multi-view perspective image is transformed into a multi-scale IPM image through image scale transformation and then IPM transformation.

[0061] Step S3: BEV feature extraction: The IPM image obtained in S2 is subjected to multi-scale strip convolution decoding. The multi-scale IPM image features are connected through strip convolution residuals in different directions to fuse multi-scale context information, fuse global and local features, and output feature F o .

[0062] Step S4: Perspective feature extraction: A synchronous learning sequence is designed to extract perspective features. The perspective images acquired by sensors around the vehicle are input, and semantic segmentation is performed through a pre-trained semantic segmentation network to generate perspective features and extract the correlation features between static and dynamic objects to make up for the missing context in the BEV image.

[0063] Step S5: Dual feature fusion: After synchronous learning with the perspective feature extraction branch, the perspective feature can be obtained from the dense perspective branch, and the acquired feature is input into the backbone network to be aligned and fused with the global feature through the dual feature fusion module to obtain enhanced features.

[0064] The perspective features are converted into the BEV coordinate system through the feature inverse perspective mapping (IPM) technology, so that the two can be aligned in space. The BEV features and perspective features are fused through convolution operations. At the same time, only the features of a single frame are fused, and the semantic correlation of multiple frames is not considered enough, which may lead to the lack of persistence of some long-term stable static scene information (such as road signs, traffic lights, etc.). Therefore, the BEV features and perspective features of multiple frames are fused, and the similar semantic information in multiple frames is weighted and integrated, so as to retain some long-term stable static semantic features and reduce the influence of dynamic object interference on static scene understanding. The fused features not only retain the clarity of the road structure from the BEV perspective, but also supplement the environmental details in the perspective image.

[0065] Step S6: Perspective complementary map learning: After feature fusion, the model's ability to understand the environment is improved by joint learning between the perspective map perspective and the BEV map perspective. The complementary information from different perspectives helps to make up for the details that may be missed by a single perspective.

[0066] Step S7: Decoding and loss supervision: The fused enhanced features F ms Passed to the decoder to generate the final BEV semantic map. At the same time, in order to supervise the proposed model, the entire training loss consists of four parts: semantic mapping loss, perspective mapping loss, joint learning loss, and additional task loss.

[0067] In step 2: given a multi-view perspective image I n And the camera's internal parameters and external reference IPM Image It can be calculated by assuming the height h:

[0068]

[0069] Proj(X i ,Y j )=∑ n I n (u,v), if (u <imH,v<imW);

[0070]

[0071] Among them, Plane represents the collection of all grids, Z cis the depth value in the camera coordinate system, u and v, u0 and v0 are the values ​​in the pixel coordinate system, n is the number of cameras, imH and imW are the perspective images I n The size of the image, H and W are the sizes of the original image.

[0072] In step 3: the multi-layer IPM image obtained by IPM transformation is input into the encoder. The encoder designs a multi-scale strip convolution mechanism and inputs the four-layer IPM image obtained by S2. These features represent different spatial resolutions. The deeper the layer, the more abstract the features and the lower the resolution.

[0073] Multi-scale strip convolution mechanism: Strip convolution refers to the extraction of features along a specific direction by designing longer and narrower convolution kernels in different directions. Specifically, strip convolution is performed in the longitudinal, transverse, and diagonal directions, using the elongated shape of the strip convolution kernel to capture global and local information in a specific direction. This multi-directional information helps to extract road structure and boundaries, and is particularly suitable for extracting linear road signs and lane lines, while maintaining local details and reducing the loss of long-distance feature information. Multi-scale strip convolution is decoded layer by layer. After the strip convolution operation, the feature map of each layer is fused with the decoded features of the previous layer to gradually construct a global bird's-eye view feature map.

[0074] Layer-by-layer decoding and feature fusion:

[0075] 1. Fourth layer decoding:

[0076] Channel adjustment: Input feature map The number of channels is reduced through the convolution layer to better integrate with other feature maps, and the adjusted feature map F4 is obtained:

[0077]

[0078] Extract features from F4 along the vertical, horizontal and diagonal directions. Assume that the direction of the strip convolution kernel is determined by D h,v Control, define the strip convolution formula:

[0079]

[0080] Among them, h, v∈{0,1} control the convolution direction (such as vertical or horizontal strip convolution), w i,j is the weight of the strip convolution kernel.

[0081] The strip convolution results in four different directions Concatenate in the channel dimension to form a directional fusion feature map d4:

[0082]

[0083] Each layer input feature encoder feature fusion: the processed feature map d 4f With feature map Fusion forms the decoding feature d4 of the current layer:

[0084]

[0085] 2. Third layer decoding:

[0086] Channel adjustment: Upsample the previous layer decoded feature d4 to match the third layer encoder feature Resolution:

[0087] d 3up =Unsample(d4);

[0088] D 3up Through multi-directional strip convolution processing, convolution features in four directions are obtained (features are extracted along the longitudinal, transverse and diagonal directions respectively, and strip convolution results in four different directions are obtained), and fused on the channel:

[0089]

[0090] D 3f With features Fusion forms the current layer decoding feature d3:

[0091]

[0092] 3. Second layer decoding

[0093] Channel adjustment: Upsample the previous layer of decoded feature d3:

[0094] d 2up =Unsample(d3);

[0095] D 2f Through multi-directional strip convolution processing, convolution features in four directions are obtained (features are extracted along the longitudinal, transverse and diagonal directions respectively, and strip convolution results in four different directions are obtained), and fused on the channel:

[0096]

[0097] D 2f With features Fusion forms the current layer decoding feature d2:

[0098]

[0099] 4. First layer decoding

[0100] Upsampling and strip convolution: After upsampling the previous layer feature d2, perform strip convolution (extract features along the vertical, horizontal and diagonal directions respectively, and obtain strip convolution results in four different directions) and fusion to obtain a high-resolution fusion feature F o :

[0101] d 1up =Unsample(d2);

[0102]

[0103] In step 4: design a synchronous learning sequence for perspective feature extraction to extract rich road features from perspective images: input the multi-view perspective images into a pre-trained lightweight semantic segmentation network ShuffleNet, which uses grouped convolution and channel shuffling techniques to reduce the amount of computation while maintaining high accuracy to capture rich road features:

[0104]

[0105] M pv =D p (F pv );

[0106] Among them, D p is the decoder, M pv refers to the road map on the perspective image, F pv Refers to the depth characteristics of the perspective image.

[0107] In step 5: First, the perspective feature F is transformed into pv Features converted to BEV coordinate system The conversion formula is:

[0108]

[0109] Proj(X i ,Y j )=∑ n F pv (u,v), if (u <imH,v<imW);

[0110]

[0111]

[0112] Where u and v are the pixel coordinates in the perspective image, imH and imW are the height and width of the image, respectively, and Z c is the depth value in the camera coordinate system, u0 and v0 are the pixel coordinates in the original perspective image, and is the camera’s intrinsic and extrinsic parameter matrix.

[0113] Then, the auxiliary features in the BEV space are Perform geometric operations, such as rotation and flipping, to ensure that the feature F o The consistent alignment of Perform pooling and convolution operations:

[0114]

[0115] Where L d is a downsampling function based on the convolution structure, C 421 represents a convolution operation with a kernel size of 4, a stride of 2, and a padding of 1. 311 It represents a convolution operation with a kernel of 3, a stride of 1, and a padding of 1. RELU is the activation function, and BN (BatchNormalization) is a batch normalization operation.

[0116] Again Perform a fill operation to further adjust the shape consistency:

[0117]

[0118] Finally, the convolution block CB is applied to process the concatenated features to obtain the enhanced features F me :

[0119]

[0120] Introducing enhanced features of the first three frames: The fused features of the first three frames are converted to the current frame perspective through the vehicle motion matrix to align the features of different time steps:

[0121]

[0122] in, Fuse features from the previous frame to align with the current frame’s viewpoint.

[0123] Use Concat to fuse multi-frame features in time series:

[0124]

[0125] As the output of the dual feature fusion module.

[0126] In step 6: In order to improve the generalization ability of the model, the road map M generated during perspective feature extraction is used pv and the semantic map M generated in the BEV space bevThe loss function is designed to achieve mutual supervision of global road structures under different semantics.

[0127] First, the BEV map M bev Perform binarization:

[0128]

[0129] The road map M generated by the dense perspective branch pv Map converted to BEV coordinate system through IPM transformation Converted by the following formula:

[0130]

[0131] Proj(X i ,Y j )=∑ n M pv (u,v), if (u <imH,v<imW);

[0132]

[0133] After obtaining the map in the same coordinate system, a loss function is designed and defined as follows:

[0134]

[0135] In step 7: The entire model training loss consists of three parts: semantic mapping loss, perspective mapping loss, and joint learning loss:

[0136] LOSS=α1×LOSS hd +α2×LOSS pv +α3×LOSS jl ;

[0137] Among them, LOSS hd is the BEV semantic mapping loss, LOSS pv is the loss of perspective semantic mapping, LOSS jl is the supervision constraint between BEV and perspective. The weight relationship is α3 = 0.1 × α1, LOSS hd and LOSS pv is the cross entropy loss function.

[0138] experiment.

[0139] All experiments were conducted using an NVIDIA RTX A6000 GPU. The experiments only used visual sensor data as input. The HD map range is (-30m, 30m) on the X axis and (-15m, 15m) on the Y axis, and the map resolution is 0.15m.

[0140] Three map elements are evaluated: lane dividers, pedestrian crossings, and road boundaries. For semantic mapping, the intersection over union (IoU) is used as the standard evaluation metric in the model. All class metrics are obtained by averaging the values ​​of the three classes. The nuScenes and Argoverse datasets are used as experimental datasets, respectively. The experimental results are shown in Tables 1 and 2.

[0141] Table 1: Semantic mapping results on the nuScenes dataset

[0142]

[0143] Table 2: Semantic mapping results on the Argoverse dataset

[0144]

[0145] As can be seen from Tables 1 and 2, this method significantly outperforms existing methods on different datasets. The mIoU of this method on the nuScenes dataset reaches 39.4%, and the mIoU on the Argoverse dataset reaches 47.9%. This method performs well in semantic mapping.

[0146] Embodiment 2:

[0147] A system for implementing the above method comprises:

[0148] The data acquisition and preprocessing module is configured to: acquire and preprocess the perspective image, and obtain a multi-scale inverse perspective mapping image through transformation;

[0149] The bird's-eye view feature module is configured as follows: the obtained inverse perspective mapping image is decoded by multi-scale strip convolution, and strip convolution residual connections in different directions are used to obtain global bird's-eye view features through layer-by-layer decoding and feature fusion;

[0150] The perspective feature module is configured as follows: the perspective image after preprocessing is subjected to synchronous learning sequence and semantic segmentation network to obtain perspective features;

[0151] The fusion output module is configured to: convert the perspective feature into a bird's-eye view coordinate system, and align it spatially with the global bird's-eye view feature to obtain an enhanced perspective feature, fuse the global bird's-eye view feature and the enhanced perspective feature through a convolution operation, and output the fused feature to the current frame of the constructed map through a decoder.

[0152] Embodiment three:

[0153] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in online map construction based on inverse perspective mapping and synchronous learning as described in the first embodiment above are implemented.

[0154] Embodiment 4:

[0155] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in online map construction based on inverse perspective mapping and synchronous learning as described in the first embodiment are implemented.

[0156] Embodiment five:

[0157] This embodiment provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of online map construction based on reverse perspective mapping and synchronous learning described in the first embodiment are implemented.

[0158] The steps involved in the above embodiments 2 to 5 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Online map construction based on inverse perspective mapping and synchronous learning, characterized by: The following steps are involved: Acquire and preprocess the perspective image, and obtain a multi-scale inverse perspective mapping image through transformation; The obtained inverse perspective mapping image is decoded by multi-scale strip convolution, and the strip convolution residual connection in different directions is used. After layer-by-layer decoding and feature fusion, the global bird's-eye view feature is obtained; After preprocessing, the perspective image is subjected to synchronous learning sequence and semantic segmentation network to obtain perspective features; The perspective feature is converted to a bird's-eye view coordinate system and spatially aligned with the global bird's-eye view feature to obtain an enhanced perspective feature. The global bird's-eye view feature and the enhanced perspective feature are fused through a convolution operation. The fused feature is output as the current frame of the constructed map through a decoder. Among them, layer-by-layer decoding and feature fusion are specifically as follows: the features of the reverse perspective mapping image of the current layer are channel-adjusted, and features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution results in four different directions, which are then connected in series in the channel dimension to form a fusion feature map in the corresponding direction, which is then fused with the features of the reverse perspective mapping image of the next layer to obtain the decoding features of the current layer and serve as the input of the next layer, until the features of the reverse perspective mapping image of the last layer are fused with the decoding features of the previous layer, and features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution results in four different directions, which are then connected in series in the channel dimension to obtain the global bird's-eye view features.

2. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: The preprocessed perspective image is a multi-view perspective image I n And the camera's internal parameters and external reference Inverse perspective mapping image It is calculated by assuming the height h, as shown below: Proj(X i ,Y j ) = ∑ n I n (u, v), if (u < imH, v < imW); Among them, Plane represents the collection of all grids, Z c is the depth value in the camera coordinate system, u and v, u0 and v0 are the values ​​in the pixel coordinate system, n is the number of cameras, imH and imW are the perspective images I n The size of the image, H and W are the sizes of the original image.

3. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: The multi-scale inverse perspective mapping image has multiple layers. The encoder is used to obtain the features of the multi-layer inverse perspective mapping image, and the global bird's-eye view features are obtained through layer-by-layer decoding and feature fusion.

4. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: Layer-by-layer decoding and feature fusion, including the fourth layer decoding, are as follows: Input feature map By reducing the number of channels through the convolution layer, we get the adjusted feature map F4: F4 is used to extract features along the longitudinal, transverse and diagonal directions respectively, and strip convolution results in four different directions are obtained. Fuse in the channel dimension to form a directional fusion feature map d4: The processed feature map d 4f The feature map of the third layer encoder Fusion forms the decoding feature d4 of the current layer.

5. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: Layer-by-layer decoding and feature fusion also include the third-layer decoding, specifically: Channel adjustment: Upsample the decoded feature d4 of the previous layer to match the encoder feature of the third layer The resolution of the adjusted feature d 3up ; D 3up Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and then fused on the channel to obtain the fused feature d 3f ; D 3f With the input features of the next layer Fusion forms the decoding feature d3 of the current layer.

6. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: Layer-by-layer decoding and feature fusion also include second-layer decoding, specifically: Channel adjustment: Upsample the previous layer decoding feature d3 to obtain the adjusted feature d 2up ; D 2up Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and then fused on the channel to obtain the fused feature d 2f ; D 2f With the input features of the next layer Fusion forms the current layer decoding feature d2.

7. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: Layer-by-layer decoding and feature fusion, including the first layer decoding, are as follows: After upsampling the previous layer feature d2, features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, which are fused on the channel to obtain the fused feature F with improved resolution. o , as the global bird's-eye view feature of the output.

8. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: The perspective feature is transformed into the bird's-eye view coordinate system and spatially aligned with the global bird's-eye view feature to obtain the enhanced perspective feature, specifically: The perspective feature F is transformed into pv Features converted to bird's-eye view coordinates Features in the bird's eye view space Perform geometric operations to ensure that the feature F o The convolutional blocks are used to process the connected features and obtain the enhanced features F. me .

9. The online map construction based on reverse perspective mapping and synchronous learning as claimed in claim 1, characterized in that: The global bird's-eye view features and enhanced perspective features are fused through convolution operations, specifically: The enhanced features of the first three frames are introduced, and the fused features of the first three frames are converted to the current frame perspective through the vehicle motion matrix. The features of different time steps are aligned, and the multi-frame features are temporally fused. The obtained fused features are output by the decoder to the current frame of the constructed map.

10. An online map construction system implementing the method according to any one of claims 1 to 9, characterized in that: include: The data acquisition and preprocessing module is configured to: acquire and preprocess the perspective image, and obtain a multi-scale inverse perspective mapping image through transformation; The bird's-eye view feature module is configured as follows: the obtained inverse perspective mapping image is decoded by multi-scale strip convolution, and strip convolution residual connections in different directions are used to obtain global bird's-eye view features through layer-by-layer decoding and feature fusion; The perspective feature module is configured as follows: the perspective image after preprocessing is subjected to synchronous learning sequence and semantic segmentation network to obtain perspective features; The fusion output module is configured to: convert the perspective feature into a bird's-eye view coordinate system, and align it spatially with the global bird's-eye view feature to obtain an enhanced perspective feature, fuse the global bird's-eye view feature and the enhanced perspective feature through a convolution operation, and output the fused feature to the current frame of the constructed map through a decoder.

Images