Online map construction method and system based on inverse perspective mapping and simultaneous learning

Abstract

The present application relates to an online map construction method and system based on inverse perspective mapping and synchronous learning, a perspective image is obtained and preprocessed, a multi-scale inverse perspective mapping image is obtained through transformation, multi-scale strip convolution decoding is performed, different direction strip convolution residual connections are used, layer-by-layer decoding and feature fusion are performed, and global bird's eye view features are obtained; the preprocessed perspective image is subjected to a synchronous learning sequence and a semantic segmentation network to obtain perspective features; the perspective features are converted to a bird's eye view coordinate system, and are spatially aligned with the global bird's eye view features to obtain enhanced perspective features; the global bird's eye view features and the enhanced perspective features are fused through convolution operation; and the fused features are output by a decoder to construct a current frame of a map. In combination with inverse perspective mapping and synchronous learning, the bird's eye view features and the perspective features of multiple frames are fused, the road structure under the bird's eye view is retained, and the perspective image is supplemented with environmental details.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of map construction, in particular to an online map construction method and system based on inverse perspective mapping and synchronous learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] During the implementation of automatic driving of a vehicle, map data that needs to be updated in real time, i.e., online map, is required, which contains detailed road information, traffic signs, lane division, obstacle position, etc. These maps are usually created and updated using sensor data (such as lidar, camera, etc.), and the related online map construction method can be applied in the field of intelligent transportation in addition to the automatic driving scene.

[0004] PV-BEV feature conversion, i.e., perspective view (PV) feature conversion to bird's eye view (BEV) feature, is a core problem in online map construction. Since the camera is usually placed on the vehicle parallel to the ground and facing the outside environment, the image is captured in the perspective view, which is orthogonal to the BEV, making the conversion between the two views difficult. Since a point in space can be converted to the image space through perspective mapping, the inverse problem of projecting an image pixel to a three-dimensional space is indeterminate.

[0005] Inverse Perspective Mapping (IPM) is a method proposed under additional constraints to solve the mathematically impossible mapping problem, i.e., the point of inverse mapping is located on the horizontal plane. This transformation applies camera rotation homography and anisotropic scaling. The homography matrix can be physically derived from the camera's internal and external parameters. Some methods use convolutional neural networks to extract semantic features of the perspective view image and estimate the vertical vanishing point and the horizon (ground plane vanishing line) in the image to determine the homography matrix. However, this method usually cannot accurately detect targets above the ground plane, such as buildings, vehicles, and pedestrians, and 3D objects will produce serious geometric distortion. At the same time, since the IPM transformation can only capture information on the road plane, it lacks interaction with dynamic or static objects above the road surface, thus leading to sparsity of feature information and limiting comprehensive understanding of the environment. SUMMARY

[0006] In order to solve the technical problems in the background art, the present application provides an online map construction method and system based on inverse perspective mapping and synchronous learning, which applies multi-scale strip convolution to extract global and local information in the BEV perspective, generates perspective features in the perspective perspective through synchronous learning sequence, cooperatively integrates through a double feature fusion module, and improves the joint learning ability through perspective complementary map learning to enhance the model generalization ability.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides an online map construction based on inverse perspective mapping and synchronous learning, comprising the following steps:

[0009] The perspective image is obtained and preprocessed, and a multi-scale inverse perspective mapping image is obtained through transformation;

[0010] The obtained inverse perspective mapping image is decoded through multi-scale strip convolution, different direction strip convolution residual connection is used, layer-by-layer decoding and feature fusion are performed, and global bird's eye view features are obtained;

[0011] The preprocessed perspective image is obtained through a synchronous learning sequence and a semantic segmentation network to obtain perspective features;

[0012] The perspective features are converted to the bird's eye view coordinate system and spatially aligned with the global bird's eye view features to obtain enhanced perspective features, the global bird's eye view features and the enhanced perspective features are fused through convolution operation, and the fused features are output by the decoder to construct the current frame of the map;

[0013] Wherein, the layer-by-layer decoding and feature fusion are as follows: the features of the current layer of the inverse perspective mapping image are adjusted in the channel, and the features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four strip convolution results in different directions, which are connected in the channel dimension to form the corresponding direction fusion feature map, then the fusion feature map is fused with the features of the next layer of the inverse perspective mapping image to obtain the decoding features of the current layer as the input of the next layer, until the features of the last layer of the inverse perspective mapping image are fused with the decoding features of the last layer, and the features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four strip convolution results in different directions, which are connected in the channel dimension to obtain the global bird's eye view features.

[0014] Further, the preprocessed perspective image is transformed into a multi-scale inverse perspective mapping image, specifically: the preprocessed perspective image is subjected to image scale transformation and then inverse perspective transformation to convert it into a multi-scale inverse perspective mapping image.

[0015] Further, the preprocessed perspective image is transformed into a multi-scale inverse perspective mapping image, specifically: the preprocessed perspective image is a multi-view perspective image I n and the intrinsic parameters of the camera and extrinsic parameters inverse perspective mapped image calculated by the assumed height h, as shown in the following formula:

[0016]

[0017] Proj(X i ,Y j )=∑ n I n (u,v),if(u<imH,v<imW);

[0018]

[0019] where Plane represents a set of all meshes, Z c is a depth value in the camera coordinate system, u and v, u0 and v0 are values in the pixel coordinate system, n is the number of cameras, imH and imW are the sizes of the perspective image I n , and H and W are the sizes of the original image.

[0020] Further, the multi-scale inverse perspective mapped image has multiple layers, and features of the multi-layer inverse perspective mapped image are obtained by using an encoder, and global bird's eye view features are obtained by layer-by-layer decoding and feature fusion.

[0021] Further, the layer-by-layer decoding and feature fusion include fourth layer decoding, specifically:

[0022] input feature map The channel number is reduced through a convolution layer to obtain an adjusted feature map F4:

[0023] Features are extracted along the longitudinal, transverse and diagonal directions of F4 respectively to obtain four strip-shaped convolution results in different directions Fusion in the channel dimension forms a direction fusion feature map d4:

[0024] The processed feature map d 4f is fused with the feature map of the third layer encoder to form the decoding feature d4 of the current layer.

[0025] Further, the layer-by-layer decoding and feature fusion also include third layer decoding, specifically:

[0026] Channel adjustment: the decoding feature d4 of the previous layer is up-sampled to match the resolution of the third layer encoder feature to obtain an adjusted feature d 3up ;

[0027] d 3upFeatures are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and fused on the channel to obtain the fused feature d 3f ;

[0028] d 3f and the input features of the next layer Fusion forms the decoding feature d3 of the current layer.

[0029] Furthermore, layer-by-layer decoding and feature fusion also include second-layer decoding, specifically:

[0030] Channel adjustment: Upsample the previous layer decoding feature d3 to obtain the adjusted feature d 2up ;

[0031] d 2up Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, and fused on the channel to obtain the fused feature d 2f ;

[0032] d 2f and the input features of the next layer Fusion forms the current layer decoding feature d2.

[0033] Furthermore, layer-by-layer decoding and feature fusion also include first-layer decoding, specifically:

[0034] After upsampling the previous layer feature d2, features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain strip convolution features in four different directions, which are fused on the channel to obtain the fused feature F with improved resolution. o , as the global bird's-eye view feature of the output.

[0035] Furthermore, the perspective feature is transformed into a bird's-eye view coordinate system and spatially aligned with the global bird's-eye view feature to obtain the enhanced perspective feature, specifically:

[0036] The method based on feature inverse perspective mapping transforms the perspective feature F pv Features converted to bird's-eye view coordinates

[0037] Features in bird's-eye view space Perform geometric operations to ensure that the feature F o The convolution block is used to process the connected features and obtain the enhanced features F. me .

[0038] Furthermore, the global bird's-eye view features and the enhanced perspective features are fused through convolution operations, specifically:

[0039] The enhanced features of the first three frames are introduced, the fusion features of the first three frames are converted to the perspective of the current frame by a vehicle motion matrix, the features of different time steps are aligned, and the multi-frame features are time-series fused to obtain the fusion features, which are output by the decoder to construct the current frame of the map.

[0040] The second aspect of the application provides a system required to implement the above method, comprising:

[0041] The data acquisition and preprocessing module is configured to obtain a perspective image and pre-process it to obtain a multi-scale inverse perspective mapping image through transformation;

[0042] The bird's eye view feature module is configured to decode the obtained inverse perspective mapping image through multi-scale strip convolution, utilize strip convolution residual connection in different directions, and decode and fuse features layer by layer to obtain global bird's eye view features;

[0043] The perspective feature module is configured to obtain perspective features from the pre-processed perspective image through a synchronous learning sequence and a semantic segmentation network;

[0044] The fusion output module is configured to convert the perspective features to the bird's eye view coordinate system, align them with the global bird's eye view features in space to obtain enhanced perspective features, fuse the global bird's eye view features and the enhanced perspective features through convolution operation, and output the fused features by the decoder to construct the current frame of the map.

[0045] The third aspect of the application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the online map construction based on inverse perspective mapping and synchronous learning as described above.

[0046] The fourth aspect of the application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the steps in the online map construction based on inverse perspective mapping and synchronous learning as described above.

[0047] The fifth aspect of the application provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in the online map construction based on inverse perspective mapping and synchronous learning as described above.

[0048] Compared with the prior art, the above one or more technical solutions have the following beneficial effects:

[0049] 1、BEV (bird's eye view) features can provide global and local information on the road plane, suitable for capturing the overall road structure and static elements such as lane boundaries, pedestrians crossing the street and obstacles. The perspective (PV) feature captures the relationship between static and dynamic objects in the camera view, including semantic information about the objects above and the surrounding environment, which can make up for the lack of BEV features in terms of height and object context. By combining the two types of features, more comprehensive scene information can be obtained, which improves the accuracy of overall map construction and effectively reduces the information loss that may be caused by a single perspective. At the same time, the perspective feature helps to more accurately detect dynamic objects (such as vehicles and pedestrians) around the road, maintaining the comprehensiveness and dynamics of environmental understanding in real-time map updating.

[0050] 2、In the BEV view, multi-scale strip convolution is applied to extract global and local information, and the sequence is simultaneously learned to generate perspective features in the perspective view. Through a dual feature fusion module, the two types of features are integrated, and the joint learning ability is improved through view complementary map learning, enhancing the model generalization ability.

[0051] 3、Considering that only single-frame features are fused, the semantic correlation of multiple frames is insufficient, which may lead to a lack of persistence of some long-term stable static scene information (such as road signs and traffic lights), multi-frame BEV features and perspective features are introduced to fuse similar semantic information within multiple frames, thereby retaining some long-term stable static semantic features and reducing the influence of dynamic object interference on static scene understanding. The fused features not only retain the clarity of the road structure in the BEV view, but also supplement the environmental details in the perspective image. BRIEF DESCRIPTION OF DRAWINGS

[0052] The accompanying drawings, which form a part of this description, are included to provide further understanding of the application, and are incorporated in and constitute a part of this application. The schematic embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application.

[0053] Figure 1 is a schematic diagram of an online map construction process based on inverse perspective mapping and synchronous learning provided by one or more embodiments of the application. DETAILED DESCRIPTION

[0054] The application will be further described below in conjunction with the drawings and embodiments.

[0055] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0056] Embodiment one:

[0057] Since the BEV (bird's eye view) feature can provide global and local information on the road plane, it is suitable for capturing the overall road structure as well as static elements such as lane boundaries, pedestrians crossing the street, and obstacles. The perspective (PV) feature captures the relationship between static and dynamic objects from the camera's perspective, including semantic information about objects above and the surrounding environment, which can make up for the lack of BEV features in terms of height and object context. By fusing the two types of features, more comprehensive scene information can be obtained.

[0058] For example, the perspective feature can complement the lack of BEV features in identifying three-dimensional objects such as buildings, trees, etc., providing rich spatial relationship information. This combination improves the accuracy of overall map construction and effectively reduces the information loss that may result from a single perspective. At the same time, the perspective feature helps to more accurately detect dynamic objects around the road, such as vehicles and pedestrians, maintaining the comprehensiveness and dynamism of environmental understanding in real-time map updating. Therefore, the embodiment proposes a synchronous learning strategy based on dual perspectives, which applies a multi-scale strip convolutional network in the BEV perspective to extract global and local information, synchronously learns a sequence to generate perspective features in the perspective perspective, and cooperatively integrates through a dual feature fusion module, and improves the joint learning ability through perspective complementary map learning, enhances the model generalization ability, and the specific process is as shown in Figure 1 .

[0059] Step S1: data preprocessing. Two representative datasets, nuScenes and Argoverse datasets, are selected, and the image pairs in each dataset are divided into training image pairs and test image pairs.

[0060] Step S2: image IPM transformation: the initial multi-view perspective image is subjected to image scale transformation and then IPM transformation to convert it into a multi-scale IPM image.

[0061] Step S3: BEV feature extraction: multi-scale strip convolution decoding is performed on the IPM image obtained in S2, and the multi-scale IPM image features are connected through strip convolution in different directions, fusing multi-scale context information, fusing global and local features, and outputting the feature F o .

[0062] Step S4: perspective feature extraction: a synchronous learning sequence is designed for perspective feature extraction, the perspective image obtained by the vehicle's surrounding sensors is input, a pre-trained semantic segmentation network is used for semantic segmentation, and perspective features are generated, the correlation features between static and dynamic objects are extracted, and the context loss in the BEV image is made up.

[0063] Step S5: Dual feature fusion: After synchronous learning with the perspective feature extraction branch, perspective features can be obtained from the dense perspective branch. The obtained features are input into the backbone network and aligned and fused with the global features through the dual feature fusion module to obtain enhanced features.

[0064] The perspective features are converted into the BEV coordinate system through the feature inverse perspective mapping (IPM) technology, so that the two can be aligned in space. The BEV features and perspective features are fused through convolution operations. At the same time, only the features of a single frame are fused, and the semantic correlation of multiple frames is not considered enough, which may lead to the lack of persistence of some long-term stable static scene information (such as road signs, traffic lights, etc.). Therefore, the fusion of BEV features and perspective features of multiple frames is introduced to weightedly integrate similar semantic information in multiple frames, thereby retaining some long-term stable static semantic features and reducing the impact of dynamic object interference on static scene understanding. The fused features not only retain the clarity of the road structure from the BEV perspective, but also supplement the environmental details in the perspective image.

[0065] Step S6: Perspective Complementary Map Learning: After feature fusion, the model's understanding of the environment is enhanced by joint learning between the perspective map and the BEV map. The complementary information from different perspectives helps to compensate for details that may be missed from a single perspective.

[0066] Step S7: Decoding and loss supervision: The fused enhanced features F ms The result is passed to the decoder to generate the final BEV semantic map. At the same time, to supervise the proposed model, the entire training loss consists of four parts: semantic mapping loss, perspective mapping loss, joint learning loss, and additional task loss.

[0067] In step 2: given a multi-view perspective image I n And the camera's internal parameters and external reference IPM Image It can be calculated by assuming the height h:

[0068]

[0069] Proj(X i ,Y j )=∑ n I n (u,v), if (u <imH,v<imW);

[0070]

[0071] Among them, Plane represents the collection of all grids, Z cis the depth value in the camera coordinate system, u and v, u0 and v0 are the values in the pixel coordinate system, n is the number of cameras, imH and imW are the size of the perspective image I n , H and W are the size of the original image.

[0072] In step three: input the multi-layer IPM image obtained by IPM transformation into the encoder, and the encoder designs a multi-scale strip convolution mechanism, inputting the four-layer IPM image obtained by S2 These features represent different spatial resolutions, and the deeper the layer, the more abstract the feature and the lower the resolution.

[0073] Multi-scale strip convolution mechanism: strip convolution refers to the design of a long and narrow convolution kernel in different directions to extract features in a specific direction. Specifically, strip convolution is performed in the longitudinal, transverse and diagonal directions, and the elongated shape of the strip convolution kernel captures global and local information in a specific direction. This multi-directional information helps to extract road structures and boundaries, and is particularly suitable for extracting linear road signs and lane lines, while maintaining local details and reducing the loss of long-distance feature information. Multi-scale strip convolution decodes layer by layer, and the feature map of each layer is fused with the decoded features of the previous layer after strip convolution operation, gradually constructing a global bird's eye view feature map.

[0074] Layer-by-layer decoding and feature fusion:

[0075] 1. Fourth layer decoding:

[0076] Channel adjustment: adjust the input feature map F4 by reducing the number of channels through a convolution layer, so as to better fuse with other feature maps, and obtain the adjusted feature map F4:

[0077]

[0078] Extract features along the longitudinal, transverse and diagonal directions respectively. Assuming that the direction of the strip convolution kernel is controlled by D h,v , define the strip convolution formula:

[0079]

[0080] where h, v ∈ {0, 1} control the convolution direction (such as longitudinal or transverse strip convolution), w i,j is the weight of the strip convolution kernel.

[0081] Concatenate the results of the four different direction strip convolutions in the channel dimension to form the direction fusion feature map d4:

[0082]

[0083] Each layer input feature encoder feature fusion: the processed feature map d 4f is fused with the feature map d to form the decoding feature d4 of the current layer:

[0084]

[0085] 2. Third layer decoding:

[0086] Channel adjustment: up-sampling the last layer decoding feature d4 to match the resolution of the third layer encoder feature d :

[0087] d 3up = Unsample(d4);

[0088] d 3up is processed by multi-directional strip convolution to obtain four directional convolution features (features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four different directional strip convolution results), and fused in the channel:

[0089]

[0090] d 3f is fused with the feature to form the decoding feature d3 of the current layer:

[0091]

[0092] 3. Second layer decoding

[0093] Channel adjustment: up-sampling the last layer decoding feature d3:

[0094] d 2up = Unsample(d3);

[0095] d 2f is processed by multi-directional strip convolution to obtain four directional convolution features (features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four different directional strip convolution results), and fused in the channel:

[0096]

[0097] d 2f is fused with the feature to form the decoding feature d2 of the current layer:

[0098]

[0099] 4. First layer decoding

[0100] Upsampling and strip convolution: the last layer of features d2 is upsampled and then subjected to strip convolution (features are extracted along the longitudinal, transverse and diagonal directions to obtain four different direction strip convolution results) and fusion to obtain high-resolution fusion features F o :

[0101] d 1up = Unsample(d2);

[0102]

[0103] In step four: design a synchronous learning sequence for perspective feature extraction, which is used to extract rich road features from perspective images: input multi-view perspective images into a pre-trained lightweight semantic segmentation network ShuffleNet, which reduces the amount of calculation by using group convolution and channel shuffling technology while maintaining high accuracy, to capture rich road features:

[0104]

[0105] M pv = D p (F pv );

[0106] Where D p is the decoder, M pv refers to the road map on the perspective image, and F pv refers to the perspective depth feature.

[0107] In step five: first, the perspective feature F pv is converted to the feature in the BEV coordinate system by the feature inverse perspective mapping (IPM) technology:

[0108]

[0109] Proj(X i ,Y j ) = ∑ n F pv (u,v), if (u<imH,v<imW);

[0110]

[0111]

[0112] Where u and v are pixel coordinates in the perspective image, imH and imW are the height and width of the image respectively, Z c is the depth value in the camera coordinate system, u0, v0 are pixel coordinates in the original perspective image, and are the intrinsic and extrinsic parameter matrices of the camera.

[0113] Then, auxiliary features in the BEV space are performed, such as rotation and flipping, to ensure consistent alignment with the features F o , and then are subjected to pooling and convolution operations:

[0114]

[0115] where L d is a down-sampling function based on the convolution structure, C 421 represents a convolution operation with a kernel of 4, a stride of 2, and padding of 1, C 311 represents a convolution operation with a kernel of 3, a stride of 1, and padding of 1, and RELU is an activation function, and BN (Batch Normalization) is a batch normalization operation.

[0116] Further adjustment of shape consistency is performed on F by padding operation:

[0117]

[0118] Finally, the connected features are processed by a convolution block CB to obtain enhanced features F me :

[0119]

[0120] The enhanced features of the first three frames are introduced: The fused features of the first three frames are converted to the current frame perspective by the vehicle motion matrix, so as to align the features at different time steps:

[0121]

[0122] wherein, is the fused features of the first frame aligned to the current frame perspective.

[0123] The multi-frame features are fused in time sequence by Concat:

[0124]

[0125] as the output of the dual feature fusion module.

[0126] In step six: in order to improve the generalization ability of the model, the road map M pv generated by perspective feature extraction and the semantic map M bev ​The loss function is designed to realize mutual supervision of global road structure under different semantics.

[0127] First, the BEV map M bev Binary processing is performed:

[0128]

[0129] The road map M pv is converted to the BEV coordinate system by IPM transformation by the following formula:

[0130]

[0131] Proj(X i ,Y j )=∑ n M pv (u,v), if (u<imH,v<imW);

[0132]

[0133] After obtaining the map in the same coordinate system, a loss function is designed as follows:

[0134]

[0135] In step seven: the whole model training loss is composed of three parts: semantic mapping loss, perspective mapping loss, and joint learning loss:

[0136] LOSS=α1×LOSS hd +α2×LOSS pv +α3×LOSS jl ;

[0137] where LOSS hd is the BEV semantic mapping loss, LOSS pv is the loss of perspective semantic mapping, and LOSS jl is the supervision constraint between BEV and perspective. The weight relationship is α3=0.1×α1, LOSS hd and LOSS pv are cross-entropy loss functions.

[0138] Experiment.

[0139] All experiments are performed using NVIDIA RTX A6000 GPU. The experiment only uses visual sensor data as input. The HD map range is X axis (-30m, 30m) and Y axis (-15m, 15m), and the map resolution is 0.15m.

[0140] Three map elements are evaluated: lane dividers, pedestrians crossing the street, and road boundaries. For semantic mapping, Intersection over Union (IoU) is used as the standard evaluation metric in the mIoU, and all class measurements are obtained by averaging the values of the three classes. The experimental results are shown in Table 1 and Table 2, using the nuScenes and Argoverse datasets as experimental datasets, respectively.

[0141] Table 1: Semantic mapping results on the nuScenes dataset

[0142]

[0143] Table 2: Semantic mapping results on the Argoverse dataset

[0144]

[0145] As can be seen from Table 1 and Table 2, the present method significantly outperforms existing methods on different datasets. The mIoU of the present method on the nuScenes dataset reaches 39.4%, and the mIoU on the Argoverse dataset reaches 47.9%. The present method performs well in semantic mapping.

[0146] Embodiment Two:

[0147] A system implementing the above method, comprising:

[0148] A data acquisition and preprocessing module configured to obtain a perspective image and preprocess it, and obtain a multi-scale inverse perspective mapping image through transformation;

[0149] An aerial view feature module configured to obtain the inverse perspective mapping image through multi-scale strip convolution decoding, utilize strip convolution residual connections in different directions, and obtain global aerial view features through layer-by-layer decoding and feature fusion;

[0150] A perspective feature module configured to obtain perspective features from the preprocessed perspective image through a synchronous learning sequence and a semantic segmentation network;

[0151] A fusion output module configured to convert the perspective features to an aerial view coordinate system, align them with the global aerial view features in space, obtain enhanced perspective features, and fuse the global aerial view features and the enhanced perspective features through convolution operation, and output the current frame of the constructed map through the decoder.

[0152] Embodiment Three:

[0153] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the online map construction based on inverse perspective mapping and synchronous learning as described in the embodiment one.

[0154] Embodiment four:

[0155] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps in the online map construction based on inverse perspective mapping and synchronous learning as described in the embodiment one when executing the program.

[0156] Embodiment five:

[0157] The embodiment provides a computer program product, which comprises a computer program, and the program is executed by a processor to realize the steps in the online map construction based on inverse perspective mapping and synchronous learning as described in the embodiment one.

[0158] The steps involved in the above embodiments two to five correspond to the embodiment one, and the specific implementation can be referred to the related description part of the embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; and should also be understood as including any medium capable of storing, encoding or carrying the instruction set for execution by the processor and making the processor execute any method in the present application.

[0159] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. Online map construction based on inverse perspective mapping and simultaneous learning, characterized in that: The method comprises the following steps: obtaining perspective images and preprocessing, and obtaining multi-scale inverse perspective mapping images through transformation; the obtained inverse perspective mapping images are decoded through multi-scale strip convolution, connected through strip convolution residual connection in different directions, and decoded and fused layer by layer to obtain global bird's eye view features; the perspective features are converted into the bird's eye view coordinate system and spatially aligned with the global bird's eye view features to obtain enhanced perspective features, and the global bird's eye view features and the enhanced perspective features are fused through convolution operation, and the fused features are output through a decoder to obtain a current frame of the constructed map; wherein the layer-by-layer decoding and feature fusion comprises the following steps: the fourth layer decoding, specifically: the third layer decoding, specifically: input feature map reduce the number of channels through a convolutional layer to obtain an adjusted feature map : Will Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four different direction strip convolution results Fusion in the channel dimension to form a direction fusion feature map : the processed feature map feature map of the third layer encoder fused to form a decoded feature of the current layer ; the second layer decoding, specifically: Channel adjustment: decode features from the previous layer are upsampled to match the resolution of the third layer encoder features , resulting in adjusted features ; Will Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four strip-shaped convolutional features in different directions, and the features are fused in the channel to obtain the fused features ; will be described below. input features of the next layer fusion, forming the decoding features of the current layer ; the first layer decoding, specifically: Channel adjustment: decode the features of the previous layer to get the adjusted features ; Will Features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four strip-shaped convolutional features in different directions, which are fused in the channel to obtain the fused features ; will be described below. input features of the next layer fusion, forming the current layer decoding features ; wherein the global bird's eye view features and the enhanced perspective features are fused through convolution operation, specifically: The last layer of features After upsampling, features are extracted along the longitudinal, transverse and diagonal directions respectively to obtain four strip-shaped convolutional features in different directions, which are fused in the channel to obtain a fused feature with improved resolution , as the output global bird's eye view feature; the multi-scale inverse perspective mapping images have multiple layers, and the features of the multi-layer inverse perspective mapping images are obtained through an encoder, and the global bird's eye view features are obtained through layer-by-layer decoding and feature fusion.

2. The online map building based on inverse perspective mapping and simultaneous learning of claim 1, wherein: The preprocessed perspective image is a multi-view perspective image and an intrinsic parameter of the camera and an extrinsic parameter , an inverse perspective mapping image by assuming a height is calculated as shown in the following formula: ； ； ； ； wherein, represents a set of all meshes, is a depth value in the camera coordinate system, and , and is a value in the pixel coordinate system, is the number of cameras, and is the size of the perspective image , and is the size of the original image.

3. The online map building based on inverse perspective mapping and simultaneous learning of claim 1, wherein: the perspective features are converted into the bird's eye view coordinate system and spatially aligned with the global bird's eye view features to obtain enhanced perspective features, specifically:

4. The online map building based on inverse perspective mapping and simultaneous learning of claim 1, wherein: comprising: Methods based on feature inverse perspective mapping to convert a perspective feature to an aerial view coordinate system ; Features in an aerial space Geometric operations are performed to ensure consistent alignment with the features , further pooling, convolution and padding operations are performed, and the connected features are processed using a convolution block to obtain enhanced features .

5. An online map building system implementing the method of any of claims 1-4, characterized in that, a data acquisition and preprocessing module configured to obtain perspective images and preprocess, and obtain multi-scale inverse perspective mapping images through transformation; a bird's eye view feature module configured to obtain the inverse perspective mapping images and decode them through multi-scale strip convolution, connect them through strip convolution residual connection in different directions, and obtain global bird's eye view features through layer-by-layer decoding and feature fusion; a perspective feature module configured to obtain perspective features through a synchronous learning sequence and a semantic segmentation network after preprocessing of the perspective images; a fusion output module configured to convert the perspective features into the bird's eye view coordinate system, spatially align them with the global bird's eye view features to obtain enhanced perspective features, fuse the global bird's eye view features and the enhanced perspective features through convolution operation, and output the fused features through a decoder to obtain a current frame of the constructed map. ​

Images