Visual positioning method, device and equipment based on two-dimensional plane map

Abstract

The invention provides a visual positioning method, device and equipment based on a two-dimensional plane map. The visual positioning method comprises the following steps: acquiring a front view image acquired by image acquisition equipment of a moving target; generating a semantic overlook segmentation image corresponding to the front view image by using the semantic segmentation view mapping model; matching the semantic overlook segmentation map with a two-dimensional plane map, and determining the position of the moving target when the front view image is collected; the semantic segmentation view mapping model comprises a coding network, a loop view projection network, a cross view projection network and a decoding network; the invention further discloses a visual positioning device based on the two-dimensional plane map, a computer program product and electronic equipment. The simple two-dimensional map and the front view image collected by the monocular camera are used for positioning, and positioning can be completed at low cost under the condition that a high-precision map does not need to be constructed.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning technology, and in particular to a visual positioning method, device and equipment based on a two-dimensional plane map. Background Art

[0002] With the rapid development of autonomous driving, the market application of traditional SLAM technology (Simultinous Localization and Mapping) has gradually expanded. SLAM technology relies on external sensors to complete positioning and map construction. The current mainstream SLAM technology can be divided into laser SLAM technology based on lidar and visual SLAM technology based on visual sensors.

[0003] Laser SLAM technology relies on laser radar sensors for positioning and mapping, uses 3D laser radar to obtain three-dimensional information, generates point cloud maps by integrating GPS, IMU, and laser radar, and completes the positioning task through the point cloud map during vehicle driving. It is not affected by light, but the cost of laser radar is very high and it is easily affected by small obstacles such as dust in the air (raindrops, haze). Visual SLAM technology uses monocular, binocular, and depth cameras, and is composed of multiple modules, namely sensor information reading, front-end visual odometer, back-end (non-linear) optimization, mapping, loop detection, and visual SLAM technology uses feature point extraction to generate point cloud maps. This method requires loop detection due to the accumulation of errors. It can be seen that both laser SLAM technology and visual SLAM technology need to rely on pre-established high-precision map data, which is costly. Summary of the invention

[0004] The present invention aims to at least solve the technical problems of the need for pre-established high-precision map data and high map construction costs in existing autonomous driving positioning technologies, and to provide a visual positioning method, device and equipment based on two-dimensional plane maps, which do not require the construction of high-precision maps and can complete positioning at low cost based on simple two-dimensional maps.

[0005] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a visual positioning method based on a two-dimensional plane map, including: obtaining a front view image collected by an image acquisition device of a mobile target; using a pre-trained semantic segmentation view mapping model to generate a semantic bird's-eye view segmentation map corresponding to the front view image; matching the semantic bird's-eye view segmentation map with the two-dimensional plane map to determine the position of the mobile target when the front view image is collected; wherein the semantic segmentation view mapping model includes: an encoding network, used to extract the front view features of the front view image; a recurrent view projection network, projecting the front view features to the bird's-eye view domain to obtain the bird's-eye view features, and projecting the bird's-eye view features back to the front view domain to obtain the front view projection features; a cross-view projection network, combining the front view features, the bird's-eye view features and the front view projection features to obtain the enhanced bird's-eye view features; a decoding network, decoding the enhanced bird's-eye view features to obtain a semantic bird's-eye view segmentation map, wherein the pixel values ​​of the pixels in the semantic bird's-eye view segmentation map are the first preset values ​​representing roads or the second preset values ​​representing non-roads, and the first preset value is not equal to the second preset value.

[0006] Preferably, the recurrent view projection network includes a first multilayer perceptron and a second multilayer perceptron with the same structure; the first multilayer perceptron projects the front view features to the top view domain to obtain the top view projection features; the second multilayer perceptron projects the top view projection features back to the front view domain to obtain the front view projection features.

[0007] Preferably, the cross-view projection network includes: a block division unit, which performs the same block division processing on the front view feature, the top view projection feature and the front view projection feature respectively; a correlation matrix generation unit, which calculates the correlation coefficient of each block in the top view projection feature with all the blocks in the front view feature, and constructs a correlation matrix using the correlation coefficient; a related information extraction unit, which extracts the maximum correlation coefficient of each row in the correlation matrix and the index corresponding to the maximum correlation coefficient, uses the extracted maximum correlation coefficient to form a maximum correlation coefficient vector, and uses the extracted index to form a maximum correlation coefficient index vector; a feature selection unit, which selects blocks corresponding to all maximum correlation coefficient indices in the maximum correlation coefficient index vector from the front view projection feature and forms a new mapping feature; a convolution layer, which performs convolution processing on the spliced ​​feature after the new mapping feature and the top view projection feature are spliced ​​to obtain a convolution feature; a multiplication unit, which performs element-wise multiplication on the convolution feature and the maximum correlation coefficient vector to obtain a first feature; a residual unit, which performs residual aggregation on the first feature and the top view projection feature to obtain an enhanced top view projection feature.

[0008] Preferably, the i-th block X′ in the top-view projection feature X′ i and the jth block X in the front view feature X j The correlation coefficient r i，j for:

[0009]

[0010] Among them, ||X i ′|| represents block X′ i The absolute value of ||X j || indicates block X j The absolute value of Representation calculation The inner product of ; i, j∈[1,...,hw], hw represents the number of blocks after the top view projection feature / front view feature / front view projection feature is divided.

[0011] Preferably, the loss function in the training process of the semantic segmentation view mapping model is:

[0012] L=L cross +L cycle

[0013] Among them, L cross represents the cross entropy loss; L cycle Represents cycle loss, L cycle =||X″-X||, ||X″-X|| represents the absolute value of the element difference between the front view projection feature X″ and the front view feature X.

[0014] Preferably, the matching of the semantic overhead segmentation map with the two-dimensional plane map to determine the position of the moving target when the front view image is captured includes: sliding the semantic overhead segmentation map in the two-dimensional map more than once, and executing after each sliding: extracting a regional image consisting of an area corresponding to the position of the semantic overhead segmentation map and a neighborhood of the area corresponding to the position from the two-dimensional map; extracting the contour line of the road in the semantic overhead segmentation map, recorded as the first road contour line; extracting the contour line of the road in the regional image, recorded as the second road contour line; if the first road contour line matches the second road contour line, determining the position of the moving target when the front view image is captured according to the position of the regional image on the two-dimensional map; if the first road contour line does not match the second road contour line, entering the next sliding.

[0015] Preferably, the method for determining whether the first road contour line matches the second road contour line includes: obtaining the moment feature of the first road contour line, recorded as the first moment feature; obtaining the moment feature of the second road contour line, recorded as the second moment feature; calculating the similarity between the first moment feature and the second moment feature, and determining whether the first road contour line matches the second road contour line based on the similarity.

[0016] In order to achieve the above-mentioned purpose of the present invention, according to the second aspect of the present invention, the present invention provides a visual positioning device based on a two-dimensional plane map, which is used to implement the visual positioning method based on a two-dimensional plane map described in the first aspect of the present invention, including: an image acquisition module, which acquires a front view image acquired by an image acquisition device of a mobile target; a bird's-eye view projection module, which uses a pre-trained semantic segmentation view mapping model to generate a semantic bird's-eye view segmentation map corresponding to the front view image; a matching positioning module, which matches the semantic bird's-eye view segmentation map with the two-dimensional plane map to determine the position of the mobile target when the front view image is acquired; wherein the semantic segmentation The view mapping model includes: an encoding network, which is used to extract the front view features of the front view image; a recurrent view projection network, which projects the front view features to the top view domain to obtain the top view projection features, and projects the top view projection features back to the front view domain to obtain the front view projection features; a cross view projection network, which combines the front view features, the top view projection features and the front view projection features to obtain the enhanced top view projection features; a decoding network, which decodes the enhanced top view projection features to obtain a semantic top view segmentation map, wherein the pixel value of the pixel point in the semantic top view segmentation map is a first preset value representing a road or a second preset value representing a non-road, and the first preset value is not equal to the second preset value.

[0017] In order to achieve the above-mentioned purpose of the present invention, according to the third aspect of the present invention, the present invention provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the visual positioning method based on a two-dimensional plane map described in the first aspect of the present invention.

[0018] In order to achieve the above-mentioned purpose of the present invention, according to the fourth aspect of the present invention, the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the visual positioning method based on a two-dimensional plane map as described in any one of claims 1 to 7.

[0019] The beneficial technical effects of the present invention include:

[0020] Based on a simple two-dimensional map, positioning is performed by using an image acquisition device on the mobile target to acquire an image in front of the mobile target, namely, a front view image. The front view image is mapped to a top view domain using a semantic segmentation view mapping model and semantic segmentation is performed to identify the road, and a binary image of the semantically segmented road in the top view domain, namely, a semantic top view segmentation map, is obtained. The semantic top view segmentation map is then matched with the simple two-dimensional map to obtain the position information of the mobile target when the front view image is acquired. Positioning can be completed at a relatively low cost and without the need for a high-precision map.

[0021] In order to avoid the traditional perspective transformation that directly projects the image into a top view, the semantic segmentation view mapping model uses a cyclic view projection network. First, the front view feature information is converted into a top view projection feature, and then the top view projection feature is converted into a front view projection feature in the front view domain. Through the cyclic projection process for training, the perspective transformation of different road planes can be realized, and the occluded part can be predicted to ensure the effective transmission of the front view information. The semantic segmentation view mapping model also uses a cross view projection network to associate the front view features before the view projection with the top view projection features after projection in the top view domain to enhance the features after the view projection. In addition, the periodic loss is introduced into the loss function during the training process of the semantic segmentation view mapping model to further improve the representativeness of the features, so that the obtained semantic top view segmentation map can accurately represent the road information in front of the moving target in the same or similar style of the two-dimensional map, thereby improving the positioning accuracy.

[0022] In the process of matching the semantic overhead segmentation map with the two-dimensional map, the semantic overhead segmentation map is slid on the two-dimensional map more than once. After each slide, the corresponding region image is extracted from the two-dimensional map. The region image and the semantic overhead segmentation map are matched based on the moment features of the contour line to find the best matching point. The relative relationship between the two can be solved, thereby achieving accurate matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of a visual positioning method based on a two-dimensional plane map in a preferred embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of the network structure of a semantic segmentation view mapping model in a preferred embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of the structure of a cyclic view projection network in a preferred embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of the structure of a cross-view projection network in a preferred embodiment of the present invention;

[0027] Figure 5 It is a schematic diagram of the matching principle between the semantic overhead segmentation map and the two-dimensional map in a preferred embodiment of the present invention;

[0028] Figure 6 It is the processing effect of the semantic segmentation view mapping model of the present invention on the first example image under the KITTI dataset;

[0029] Figure 7 It is the processing effect of the semantic segmentation view mapping model of the present invention on the second example image under the KITTI dataset;

[0030] Figure 8 It is the processing effect of the semantic segmentation view mapping model of the present invention on the third example image under the KITTI dataset;

[0031] Fig. 9 is a road matching test result of the visual positioning method based on the two-dimensional plane map provided by the present invention;

[0032] Fig.10 is a crossroad matching test result of the visual positioning method based on the two-dimensional plane map provided by the present invention;

[0033] Fig.11 It is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0035] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0036] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0037] The execution subject of the visual positioning method based on a two-dimensional plane map includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the visual positioning method based on a two-dimensional plane map can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms.

[0038] On actual roads, humans can rely on simple two-dimensional maps and signs inside buildings to complete positioning and navigation in unfamiliar environments. They use their eyes to obtain the current environment and extract relevant route information, and finally match it with the simple two-dimensional map or relevant road signs. Based on this principle, if the mobile target (vehicle, etc.) can also estimate the information of the current map from the monocular visual image, and match it with the provided simple two-dimensional map to achieve positioning, the equipment requirements can be greatly reduced compared to laser radar. Based on this, the present invention provides a visual positioning method, device, program product and electronic device based on a two-dimensional plane map.

[0039] The present invention discloses a visual positioning method based on a two-dimensional plane map. In a preferred embodiment, referring to Figure 1 As shown, the method includes:

[0040] Step S1, obtaining a front view image captured by an image acquisition device of a mobile target. The mobile target is a vehicle or device that can travel on a road, preferably but not limited to a car, a motorcycle, a bicycle, and a mobile robot. The image acquisition device is preferably but not limited to a monocular camera. The image acquisition device is installed on the mobile target, and it captures images in the direction in front of the mobile target or in the direction of travel, and takes this direction as the front view direction. The captured image is a front view image, such as Figure 6 Neutron diagram (a), Figure 7 Neutron graph (a) and Figure 8 As shown in the neutron diagram (a).

[0041] Further preferably, after acquiring the front view image, a correction process is performed to obtain a corrected image, and the corrected image is input into the subsequent step S2 for processing. The correction process mainly corrects the influence of camera distortion on the image, such as pincushion distortion, barrel distortion, tangential distortion, thin prism distortion, etc.

[0042] Step S2, using the pre-trained semantic segmentation view mapping model to generate a semantic top view segmentation map corresponding to the front view image. The pixel value of the pixel point in the semantic top view segmentation map is a first preset value representing a road or a second preset value representing a non-road, and the first preset value is not equal to the second preset value. The semantic top view segmentation map is a binary map, and the value range of the pixel point in the map includes a first preset value and a second preset value, wherein the first preset value represents a road and the second preset value represents a non-road, such as Figure 6 Neutron diagram (b), Figure 7 Neutron graph (b) and Figure 8 In the middle image (b), the white part represents the road and the black part represents the non-road.

[0043] Step S3, matching the semantic overhead segmentation map with the two-dimensional plane map to determine the position of the mobile target when the front view image is captured. Specifically, the position of the mobile target in the two-dimensional map, i.e., the map position, is obtained according to the matching result. Each map position in the two-dimensional map corresponds to a world coordinate position, thereby obtaining the world coordinate position of the mobile target.

[0044] The matching method of the semantic overhead segmentation map and the two-dimensional plane map can adopt the existing template matching technology. Specifically, the semantic overhead segmentation map is slid on the simple two-dimensional map to be searched, and the semantic overhead segmentation map and the simple two-dimensional map are compared each time the map is slid, and finally the best match is made through the matching similarity. The existing SIFT (scale-invariant feature transformation matching algorithm) method can also be used for feature matching, extracting key nodes on the semantic overhead segmentation map and the two-dimensional map that are independent of scale and rotation, generating key node descriptors, and matching the key points of the two images based on the descriptors. The existing contour matching algorithm can also be used to extract the contours of the semantic overhead segmentation map and the two-dimensional plane map, and match them based on the area features, perimeter features and moment features of the two contours. The above matching methods are all existing technologies and will not be repeated here.

[0045] In this embodiment, refer to Figure 2 As shown, the semantic segmentation view mapping model includes:

[0046] The encoding network is used to extract the front view features of the front view image. The encoding network is preferably but not limited to a ResNet network, which can avoid degenerate linearity.

[0047] The cyclic view projection network projects the front view features into the top view domain to obtain the top view projection features, and then projects the top view projection features back into the front view domain to obtain the front view projection features.

[0048] The cross-view projection network combines the front view features, the top view projection features and the front view projection features to obtain the enhanced top view projection features;

[0049] The decoding network decodes the enhanced top-view projection features to obtain a semantic top-view segmentation map. The decoding network is preferably a CNN network.

[0050] In this embodiment, the purpose of the semantic segmentation view mapping model is to project the front view image of the front view domain to the top view domain to obtain the top view, and to perform semantic segmentation on the top view after the projection transformation, identify roads and non-roads, and perform binary transformation according to the recognition result to obtain a semantic top view segmentation map in the form of a binary map. Since there is a projection transformation between the front view and the top view, direct view projection will inevitably lead to the loss of a large amount of image information. Traditional perspective transformation cannot solve this problem. In order to improve the significance of the correlation between the front view image and the top view image in the deep network and enhance the extracted visual features, a cyclic view projection network and a cross view projection network are introduced.

[0051] In this embodiment, preferably, referring to Figure 3 As shown, the recurrent view projection network includes a first multilayer perceptron and a second multilayer perceptron with the same structure; the first multilayer perceptron projects the front view feature X to the top view domain to obtain the top view projection feature X′, which is expressed as X′=F MLP (X); The second multi-layer perceptron projects the top view projection feature X′ back to the front view domain to obtain the front view projection feature X″, expressed as X″=F′ MLP (X′). The first multi-layer perceptron and the second multi-layer perceptron can be two-layer fully connected MLPs. The recurrent view projection network introduces a recurrent self-supervision scheme to strengthen the view projection, projecting the top view projection feature X′ back to the front view domain, so that the front view projection feature X″ contains the most relevant features in the view projection of the front view.

[0052] In this embodiment, it is observed that the front view feature X is similar to the front view projection feature X″, but is very different from the top view projection feature X′ due to the different domains. X, X′ and X″ will be sent to the cross view projection network for further processing. The main purpose of the cross view projection network is to associate the front view feature X before view projection with the top view projection feature X′ after view projection, and enhance the features after view projection. Since the front view projection feature X″ contains the substantial information of the front view, view projection can be performed, and the features can also be further enhanced. Figure 4 As shown, the cross-view projection network includes:

[0053] The block division unit performs the same block division processing on the front view features, the top view projection features and the front view projection features respectively. For example, the front view features, the top view projection features and the front view projection features are all divided into 16*16 blocks, with 16 blocks in the horizontal direction and 16 blocks in the vertical direction, and a total of 16*16=256 blocks.

[0054] The correlation matrix generation unit calculates the correlation coefficients of each block in the top view feature and all blocks in the front view feature, and uses the correlation coefficients to construct the correlation matrix R. The i-th block X′ in the top view feature X′ i and the jth block X in the front view feature X j The correlation coefficient r i，j for:

[0055]

[0056] Among them, ||X i ′|| represents block X′ i The absolute value of block X′ i The absolute value of the eigenvalue matrix of ||X j || indicates block X j The absolute value of block X j The absolute value of the eigenvalue matrix of ; Representation calculation and The inner product of ; i, j∈[1,...,hw], hw represents the number of blocks after the top view projection feature / front view feature / front view projection feature is divided. The size of the correlation matrix R is hw×hw.

[0057] The relevant information extraction unit extracts the maximum correlation coefficient of each row in the correlation matrix R and the index corresponding to the maximum correlation coefficient, and uses the extracted maximum correlation coefficient to form a maximum correlation coefficient vector W, and uses the extracted index to form a maximum correlation coefficient index vector H.

[0058]

[0059]

[0060] argmax means finding the index corresponding to the maximum correlation coefficient in the correlation matrix R.

[0061] The feature selection unit selects blocks corresponding to all maximum correlation coefficient indexes in the maximum correlation coefficient index vector from the orthographic projection feature X″ and forms a new mapping feature T.

[0062] Both the front view feature X and the front view projection feature X″ are features of the front view, but the front view feature X contains the complete information of the front view, while the front view projection feature X″ retains the relevant information of the view projection. Assuming that the correlation between the front view feature X and the top view projection feature X′ is similar to the correlation between the front view projection feature X″ and the top view projection feature X′, then the correlation between the front view feature X and the top view projection feature X′ can be used to extract the most important information from the front view projection feature X″. The processing process of the feature selection unit can be expressed as function F fs It means that the function F fs Indicates h from X″ i The feature vector t is extracted from i :

[0063]

[0064] Convolutional layer, convolution processing is performed on the concatenated features of the new mapping feature T and the top-view projection feature X′ to obtain the convolution feature F conv (X′, T). The new mapping feature T stores the most relevant information of each block of the top-view projection feature X′ and the front-view projection feature X″. T can be reshaped to the same size as the top-view projection feature X′ and then spliced ​​(connected) with the top-view projection feature X′. The spliced ​​spliced ​​features are convolved to obtain the convolution feature. The convolution kernel size of the convolution layer can be 3×3.

[0065] The multiplication unit performs element-wise multiplication of the convolution feature and the maximum correlation coefficient vector to obtain the first feature, i.e., F conv (X′, T))⊙W;

[0066] Residual unit, which performs residual aggregation on the first feature and the top view projection feature to obtain the enhanced top view projection feature X out :

[0067] X out =X′+F conv (X′, T))⊙W.

[0068] In this embodiment, the training process of the semantic segmentation view mapping model is:

[0069] Step 1: Obtain a sample image set. The sample image set can come from existing datasets (such as KITTI, Argoverse) or autonomous images. When each sample image is taken, the laser radar collects point cloud data to obtain a point cloud map, which is converted into a binary road view, i.e., real information, such as Figure 6 Subgraph (c) and Figure 7 Sub-image (c). The real information is used as the supervision data for each sample image.

[0070] Step 2: Build the network structure of the above semantic segmentation view mapping model and initialize the network parameters.

[0071] Step 3, divide the sample image set into a training set, a validation set, and a test set;

[0072] Step 4: Use the training set to train the semantic segmentation view mapping model, and take the minimum value of the loss function as the goal during the training process. Use the gradient descent method to continuously update the network parameters of the semantic segmentation view mapping model until the training stop condition is reached, such as reaching the maximum number of training times or the loss function value is minimized. After the training stop condition is reached, the final network parameters of the semantic segmentation view mapping model are output.

[0073] Step 5: Use the validation set to validate the trained semantic segmentation view mapping model. If the validation passes, the training ends and the process goes to step 6. If the validation fails, the hyperparameters (learning rate, number of network layers, etc.) are adjusted and the process goes back to step 4.

[0074] Step 6: Use the test set to train the semantic segmentation view mapping model, and output the network parameters of the semantic segmentation view mapping model obtained in step 4.

[0075] In a preferred embodiment, in order to improve the representativeness of features, the loss function in the training process of the semantic segmentation view mapping model is:

[0076] L=L cross +L cycle

[0077] Among them, L cross represents the cross entropy loss; L cycle Indicates cycle loss;

[0078] L cycl e=||X″-X||;

[0079] ||X″-X|| represents the absolute value of the element-level difference between the front projection feature X″ and the front view feature X. First, X″-X is calculated, that is, the two features are element-wise subtracted to obtain the difference feature, and then the absolute value of the difference feature is calculated.

[0080]

[0081] Where N represents the number of feature points of the semantic top-view segmentation map of the sample image or the corresponding real information, n represents the feature point index; y n Represents the true value of the nth feature point in the real information of the sample image. When the nth feature point belongs to the road, y n =1, when the nth feature point belongs to non-road, y n =0; p nIndicates whether the nth feature point in the semantic overhead segmentation map of the sample image belongs to the road. If the nth feature point belongs to the road, p n =1, when the nth feature point belongs to non-road, p n =0.

[0082] In this embodiment, a verification experiment is performed on the semantic segmentation view mapping model obtained by the above training. In the experiment, the KITTI dataset and the Argoverse dataset are used. The Raw part of the KITTI dataset is used, which contains 10156 training images and 5074 verification images. The Argoverse dataset includes 6723 training images and 2418 verification images. mIOU (Intersection-over-Union) and mAP (Average Precision) are used as evaluation indicators. IOU refers to the ratio of the intersection and union of the true value and the predicted value. The evaluation results are shown in Table 1 below:

[0083] Table 1 Evaluation results on KITTI and Argoverse datasets

[0084]

[0085] It can be seen that the method in this paper has achieved good success on both datasets, with the intersection-over-union ratio exceeding 75% and the average accuracy maintained at around 87%. It can be seen that the accuracy of the model on both datasets is high and quite close.

[0086] Ablation experiments were conducted on the cyclic view projection module under the KITTI dataset. The module is mainly divided into the MLP part and the cyclic structure. First, the base model is defined as the encoder-decoder ordinary semantic segmentation structure with the same overall network structure, and then the MLP is added. As shown in Table 2, the MLP significantly enhances the effectiveness of the view projection. Next, the attention mechanism is used to enhance the top view projection by measuring the correlation between the network input and the top view projection. As shown in Table 2, after adding the correlation, the model performance is significantly improved. Then the cyclic view projection and cycle loss are introduced into the network. Finally, the feature extraction module is added to further improve the performance.

[0087] Table 2 Ablation experiment of cyclic view projection module on KITTI dataset

[0088]

[0089]

[0090] To test some harsh conditions, we selected scenes with more vehicles for testing, such as Figure 6 Public example 1: Figure 7 Public Example 2, and Figure 8 Public example three. It can be seen that there are more vehicles in example one, and most of the vehicles are located on both sides of the road, and half of the road is covered by shadows. The vehicle partially blocks the road information in front. In the end, the model's prediction successfully restored the information of the part blocked by the vehicle. Since the actual road information is collected from the lidar, there will be errors in the actual road area drawing. It can be seen that the radar data on the right has obvious noise information on the left and right sides of the road. The actual road is a parallel straight line, and the road collected by the lidar is obviously missing a part. However, through the training of the network, the missing part was successfully predicted, which made up for the error of the original data to a certain extent.

[0091] In a preferred embodiment, in order to improve the accuracy of positioning matching, in step S3, the semantic top view segmentation map is matched with the two-dimensional plane map to determine the position of the moving target when the front view image is collected. Figure 5 As shown, including:

[0092] Step S31, sliding the semantic bird's-eye view segmentation map on the two-dimensional map more than once, the semantic bird's-eye view segmentation map can be slid sequentially or in a jumpy manner on the two-dimensional map, and after each sliding, the following operations are executed:

[0093] Step S32, extracting a region image consisting of a region corresponding to the semantic overhead segmentation map and a neighborhood of the region corresponding to the position from the two-dimensional map. The neighborhood of the region corresponding to the position refers to a region formed by expanding outward from one or more boundary edges of the region corresponding to the position with a width d, where d is greater than or equal to one pixel.

[0094] Step S33, extracting the contour line of the road in the semantic bird's-eye view segmentation image, and recording it as the first road contour line. The contour line can be directly extracted according to the pixel value change characteristics.

[0095] The contour line of the road in the regional image is extracted and recorded as the second road contour line. The contour line can be manually drawn.

[0096] Step S34, if the first road contour line matches the second road contour line, determining the position of the moving target when the front view image is captured according to the position of the area image on the two-dimensional map;

[0097] If the first road contour line does not match the second road contour line, the next sliding is performed.

[0098] In this embodiment, the traditional template matching method and the contour matching method are combined to improve the matching accuracy.

[0099] Further preferably, the method of determining whether the first road contour line matches the second road contour line comprises:

[0100] Obtaining a moment feature of a first road contour line, recorded as a first moment feature;

[0101] Obtaining a moment feature of the second road contour line, recorded as a second moment feature;

[0102] The similarity between the first moment feature and the second moment feature is calculated, and whether the first road contour line matches the second road contour line is determined according to the similarity. If the similarity is greater than a preset similarity threshold, the first road contour line is considered to match the second road contour line, otherwise, it is considered not to match.

[0103] In this embodiment, matching is performed through the moment features of the contour line, which solves the problem that template matching cannot handle image scaling, translation, and rotation. The moment features of the contour line are invariant to scaling, translation, and rotation. Therefore, in addition to completing the matching task, the relative relationship between the two matching parts can be solved, thereby achieving accurate matching.

[0104] Fig. 9 and Fig.10 The road matching test results in the visual positioning method based on two-dimensional plane map provided by the present invention are demonstrated. Fig. 9 In the Fig. 9 (a) is a part of a simple two-dimensional map. The road contours are extracted from the semantic bird's-eye view segmentation map using a contour extraction algorithm. The extracted contours are compared with Fig. 9 (a) matches and finally successfully matches the upper right corner, such as Fig. 9 (d) Different from template matching, using contour matching can obtain translation, rotation, and scaling information between matching positions. Fig.10 Match the intersection with the map and finally match it successfully.

[0105] The present invention also provides a visual positioning device based on a two-dimensional plane map, which is used to implement the above-mentioned visual positioning method based on a two-dimensional plane map. In a preferred embodiment, the visual positioning device includes:

[0106] An image acquisition module, which acquires a front view image acquired by an image acquisition device of a moving target;

[0107] A bird's-eye view projection module generates a semantic bird's-eye view segmentation map corresponding to the front view image using a pre-trained semantic segmentation view mapping model;

[0108] A matching and positioning module matches the semantic overhead segmentation map with the two-dimensional plane map to determine the position of the moving target when the front view image is collected;

[0109] Wherein, the semantic segmentation view mapping model includes:

[0110] An encoding network for extracting front view features of the front view image;

[0111] The cyclic view projection network projects the front view features into the top view domain to obtain the top view projection features, and then projects the top view projection features back into the front view domain to obtain the front view projection features.

[0112] The cross-view projection network combines the front view features, the top view projection features and the front view projection features to obtain the enhanced top view projection features;

[0113] A decoding network is used to decode the enhanced bird's-eye view projection features to obtain a semantic bird's-eye view segmentation map, wherein the pixel value of the pixel point in the semantic bird's-eye view segmentation map is a first preset value representing a road or a second preset value representing a non-road, and the first preset value is not equal to the second preset value.

[0114] In this embodiment, the image acquisition module, the overhead projection module and the matching and positioning module correspond one to one with steps S1, S2 and S3 of the above method, and are not described in detail here.

[0115] The present invention also discloses a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-mentioned visual positioning method based on a two-dimensional plane map provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0116] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein:

[0117] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a visual positioning method based on a two-dimensional plane map provided by the present invention.

[0118] like Fig.11 , which is a schematic diagram of the structure of an electronic device for a visual positioning method based on a two-dimensional plane map provided by an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a visual positioning method program based on a two-dimensional plane map.

[0119] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a visual positioning method based on a two-dimensional plane map, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0120] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as a code of a visual positioning method program based on a two-dimensional plane map, but also can be used to temporarily store data that has been output or is to be output.

[0121] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0122] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Disp1ay), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0123] Fig.11 Only an electronic device with components is shown, and those skilled in the art will understand that Fig.11 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0124] For example, although not shown, the electronic device may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0125] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0126] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0128] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A visual positioning method based on a two-dimensional plane map, characterized in that: include: Acquire a front view image acquired by an image acquisition device of a moving target; Generate a semantic top-view segmentation map corresponding to the front view image using a pre-trained semantic segmentation view mapping model; Matching the semantic top-view segmentation map with the two-dimensional plane map to determine the position of the moving target when the front-view image is captured; Wherein, the semantic segmentation view mapping model includes: An encoding network for extracting front view features of the front view image; The cyclic view projection network projects the front view features into the top view domain to obtain the top view projection features, and then projects the top view projection features back into the front view domain to obtain the front view projection features. The cross-view projection network combines the front view features, the top view projection features and the front view projection features to obtain the enhanced top view projection features; A decoding network is used to decode the enhanced bird's-eye view projection features to obtain a semantic bird's-eye view segmentation map, wherein the pixel value of the pixel point in the semantic bird's-eye view segmentation map is a first preset value representing a road or a second preset value representing a non-road, and the first preset value is not equal to the second preset value.

2. A visual positioning method based on a two-dimensional plane map as claimed in claim 1, characterized in that: The recurrent view projection network includes a first multi-layer perceptron and a second multi-layer perceptron having the same structure; The first multi-layer perceptron projects the front view features into the top view domain to obtain the top view projection features; The second multi-layer perceptron projects the top-view projection features back to the front-view domain to obtain the front-view projection features.

3. A visual positioning method based on a two-dimensional plane map as described in claim 1 or 2, characterized in that: The cross-view projection network comprises: A block division unit performs the same block division processing on the front view feature, the top view projection feature and the front view projection feature respectively; A correlation matrix generating unit calculates the correlation coefficients of each block in the top view projection feature and all blocks in the front view feature, and constructs a correlation matrix using the correlation coefficients; A related information extraction unit extracts the maximum correlation coefficient of each row in the correlation matrix and the index corresponding to the maximum correlation coefficient, forms a maximum correlation coefficient vector using the extracted maximum correlation coefficient, and forms a maximum correlation coefficient index vector using the extracted index; A feature selection unit selects blocks corresponding to all maximum correlation coefficient indexes in the maximum correlation coefficient index vector from the orthographic projection features and forms a new mapping feature; Convolution layer, convolution processing is performed on the concatenated features of the new mapping features and the top-view projection features to obtain the convolution features; A multiplication unit, performing element-wise multiplication of the convolution feature and the maximum correlation coefficient vector to obtain a first feature; The residual unit performs residual aggregation on the first feature and the top-view projection feature to obtain an enhanced top-view projection feature.

4. A visual positioning method based on a two-dimensional plane map as claimed in claim 3, characterized in that: The i-th block X′ in the top-view projection feature X′ i and the jth block X in the front view feature X j The correlation coefficient r i，j for: Among them, ||X i ′|| represents block X′ i The absolute value of ||X j || indicates block X j The absolute value of Representation calculation and The inner product of ; i, j∈[1,...,hw], hw represents the number of blocks after the top view projection feature / front view feature / front view projection feature is divided.

5. A visual positioning method based on a two-dimensional plane map as claimed in claim 1, characterized in that: The loss function in the training process of the semantic segmentation view mapping model is: L=L cross +L cycle Among them, L cross represents the cross entropy loss; L cycle Represents the cycle loss, L cycle =||X″-X||, ||X″-X|| represents the absolute value of the element difference between the front view projection feature X″ and the front view feature X.

6. A visual positioning method based on a two-dimensional plane map as described in claim 1, 2, 4 or 5, characterized in that: The matching of the semantic top-view segmentation map with the two-dimensional plane map to determine the position of the moving target when the front-view image is captured includes: Slide the semantic bird's-eye view segmentation map more than once in the 2D map, and execute after each slide: Extracting a regional image consisting of a region corresponding to a position in the semantic bird's-eye view segmentation map and a neighborhood of the region corresponding to the position from a two-dimensional map; Extract the contour line of the road in the semantic bird's-eye view segmentation image, and record it as the first road contour line; Extracting the contour line of the road in the regional image, recorded as the second road contour line; If the first road contour line matches the second road contour line, determining the position of the moving target when the front view image is captured according to the position of the area image on the two-dimensional map; If the first road contour line does not match the second road contour line, the next sliding is performed.

7. A visual positioning method based on a two-dimensional plane map as claimed in claim 6, characterized in that: The method for determining whether the first road contour line matches the second road contour line includes: Obtaining a moment feature of a first road contour line, recorded as a first moment feature; Obtaining a moment feature of the second road contour line, recorded as a second moment feature; The similarity between the first moment feature and the second moment feature is calculated, and whether the first road contour line matches the second road contour line is determined according to the similarity.

8. A visual positioning device based on a two-dimensional plane map, characterized in that: A visual positioning method based on a two-dimensional plane map for implementing any one of claims 1 to 7, comprising: An image acquisition module, which acquires a front view image acquired by an image acquisition device of a moving target; A bird's-eye view projection module generates a semantic bird's-eye view segmentation map corresponding to the front view image using a pre-trained semantic segmentation view mapping model; A matching and positioning module matches the semantic overhead segmentation map with the two-dimensional plane map to determine the position of the moving target when the front view image is captured; Wherein, the semantic segmentation view mapping model includes: An encoding network for extracting front view features of the front view image; The cyclic view projection network projects the front view features into the top view domain to obtain the top view projection features, and then projects the top view projection features back into the front view domain to obtain the front view projection features. The cross-view projection network combines the front view features, the top view projection features and the front view projection features to obtain the enhanced top view projection features; A decoding network is used to decode the enhanced bird's-eye view projection features to obtain a semantic bird's-eye view segmentation map, wherein the pixel value of the pixel point in the semantic bird's-eye view segmentation map is a first preset value representing a road or a second preset value representing a non-road, and the first preset value is not equal to the second preset value.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of a visual positioning method based on a two-dimensional plane map described in one of claims 1-7 are implemented.

10. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a visual positioning method based on a two-dimensional plane map as described in any one of claims 1 to 7.

Images