Method and device for calculating three-dimensional position of signboard based on street view image

Abstract

The present application relates to a method and device for calculating the three-dimensional position of a sign based on street view images. The method efficiently realizes the recognition and alignment of signs in crowd-source open street view data through computer vision, three-dimensional reconstruction and other methods, provides accurate estimation of the geographical location of the sign, and significantly improves the accuracy and robustness of geographic entity positioning, which is conducive to the further application of street view resources in the fields of automatic generation of POIs, construction of urban semantic three-dimensional models, smart cities, etc.

Description

Technical Field

[0001] The present application relates to the field of geographic information and computer science and technology, and in particular to a method and device for calculating the three-dimensional position of a sign based on a street view image. Background Art

[0002] In recent years, with the rapid development of street view databases and computer vision technology, street view images have become an important data source for urban research. They are widely used in the evaluation of urban built environments, the creation and maintenance of spatial data infrastructure, and the enhancement of geographic information systems. However, when processing and analyzing these open street view images obtained from the Internet, due to the complex street environment, the images usually lack detailed shooting camera parameters, resulting in the difficulty of accurately locating the target geographic entity. This challenge needs to be urgently addressed in urban research and applications.

[0003] At present, similar studies have attempted to calculate the geographic location of the target object from street view images, but there are some shortcomings. Prior art 1: A three-dimensional measurement method for the geographic location and height of the target object based on street view images (CN106908043B). This method selects two suitable viewpoints according to the target object in the street view map, and obtains three street view images with different perspectives under the two viewpoints; the azimuth of the target object under each viewpoint is calculated based on the three street view images, and two spatial straight lines are determined in combination with the viewpoint coordinates, and a method for the intersection of spatial straight lines is provided to calculate the actual height and geographic location of the target object. This method requires a lot of manual assistance. For a target object, multiple street view images with good field of view and no obstruction are required as references for solution, and the processing efficiency is low. Existing technology 2: Noorian et al. (Shahin Sharifi Noorian; Sihang Qiu; Achilleas Psyllidis; Alessandro Bozzon; Geert-Jan Houben. Detecting, Classifying, and Mapping Retail Storefronts Using Street-level Imagery [A]. ICMR '20: Proceedings of the 2020 International Conference on Multimedia Retrieval [C], 2020) proposed a store location estimation algorithm that combines building outline data and street view image metadata. The algorithm obtains the outline data of the building facade projected onto the ground from a bird's-eye view from the OSM (OpenStreetMap) crowd-source open platform, and finds the intersection of the spatial ray from the street view shooting point to the center point of the target store and the building outline, and calculates the longitude and latitude coordinates of the intersection to determine the store's geographic location. The main disadvantage of this method is that it relies on auxiliary data from a third-party source, which may have inconsistent data accuracy and difficulty in ensuring data reliability. In summary, the current geographic entity positioning technology mainly faces two challenges. First, existing methods have deficiencies in positioning accuracy; second, the characteristics of street view data sources bring about alignment challenges. In street view images, the same sign may be photographed from multiple angles and positions, which significantly increases the complexity of mapping multiple detected objects to real-world entities. Summary of the invention

[0004] Based on this, it is necessary to provide a method and device for calculating the three-dimensional position of a sign based on street view images, which can accurately detect and align the sign entity from the street view image and calculate the precise geographical location of the sign entity in order to solve the above technical problems.

[0005] A method for calculating a three-dimensional position of a sign based on a street view image, the method comprising:

[0006] Collecting a street view panoramic image dataset along the road network, projecting the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and dividing the three-dimensional reconstruction sub-scenes according to the road network buffer zone;

[0007] identifying the signboard target in the street view image by using a pre-trained target detection network;

[0008] Performing scene text recognition on the signboard target to obtain text content and text line position of the signboard target;

[0009] Performing image feature extraction and feature point matching on the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and constructing a signboard entity according to the feature point set and the text content;

[0010] Perform similarity analysis on signboard entities in street view images from different perspectives to obtain a set of signboard entities representing the same signboard;

[0011] Based on the multi-view street view images, the street is reconstructed in three dimensions to generate a sparse three-dimensional point cloud of the scene, a consistency algorithm is used to fit the two-dimensional plane corresponding to the sign entity, and the distance between the two-dimensional plane and the three-dimensional point cloud is calculated to obtain the interior point of the two-dimensional plane;

[0012] The central coordinates of the inner points are calculated to obtain the three-dimensional position of the signboard.

[0013] In one embodiment, the target detection network is a Cascade R-CNN target detection model; the backbone network of the Cascade R-CNN target detection model is Swin-Transformer; and further comprising: identifying the coordinates of a rectangular detection frame of a sign target in the street view image through a pre-trained Cascade R-CNN target detection model. and confidence.

[0014] In one embodiment, the method further includes: performing scene text recognition on the sign target based on the PaddleOCR framework to obtain the text content of the sign target. and text line positions.

[0015] In one embodiment, the method further includes: using a SuperGlue deep learning network to extract image features and match feature points of the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the sign target area. .

[0016] In one embodiment, the method further includes: constructing a signboard entity according to the feature point set and the text content:

[0017] ;

[0018] in, Represents the coordinates of the rectangular detection box of the sign target, Represents text content. Represents the set of feature points within the target area of ​​the signboard.

[0019] In one embodiment, the method further includes: performing feature matching on the signboard entities in the street view images from different perspectives to obtain matching relationships. , if the matching relationship is the image feature matching relationship A subset of , then the normalized value of the matching relationship is calculated to obtain the image feature similarity:

[0020] ;

[0021] ;

[0022] in, , the feature set extracted from street view image A and street view image B is and , are all feature points, and are the feature point sets of the signboard entity in street view image A and the signboard entity in street view image B, respectively. Indicates the similarity of image features;

[0023] The text similarity between the text content of the signboard entity in street view image A and the text content of the signboard entity in street view image B is determined by normalized Levenshtein distance:

[0024] ;

[0025] in, Indicates the text similarity, and Represents the text content of the signboard entity in the street view image A and the signboard entity in the street view image B;

[0026] According to the image feature similarity and the text similarity, the signboard entity similarity between the signboard entity in the street view image A and the signboard entity in the street view image B is obtained as follows:

[0027] ;

[0028] in, represents the similarity of the signboard entity, represents the weighted value;

[0029] If the signboard entity similarity If the similarity threshold is exceeded, the signboard entity in the street view image A and the signboard entity in the street view image B are judged to be the same signboard, thereby obtaining the signboard entity set of the signboard object. .

[0030] In one embodiment, the method further includes: selecting the signboard entity set The five signboard entities with the most inliers are merged to obtain the inlier set ;

[0031] Compute the interior point set The coordinates of the center point of all the interior points in are:

[0032] ;

[0033] in, The coordinates of the inner point are , is the three-dimensional position of the sign.

[0034] A device for calculating a three-dimensional position of a signboard based on a street view image, the device comprising:

[0035] A data acquisition module is used to collect a street view panoramic image dataset along the road network, project the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and divide the three-dimensional reconstruction sub-scenes according to the road network buffer;

[0036] A signboard target extraction module, used to identify the signboard target in the street view image through a pre-trained target detection network;

[0037] A text extraction module, used to perform scene text recognition on the signboard target to obtain the text content and text line position of the signboard target;

[0038] A signboard entity construction module is used to extract image features and match feature points on the street view image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and to construct a signboard entity based on the feature point set and the text content;

[0039] A similarity analysis module is used to perform similarity analysis on signboard entities in street view images of different viewing angles to obtain a set of signboard entities representing the same signboard;

[0040] A feature point matching module is used to perform three-dimensional reconstruction of the street based on multi-view street view images, generate a sparse three-dimensional point cloud of the scene, use a consistency algorithm to fit the two-dimensional plane corresponding to the sign entity, calculate the distance between the two-dimensional plane and the three-dimensional point cloud, and thereby obtain the interior point of the two-dimensional plane;

[0041] The position calculation module is used to calculate the central coordinates of the inner point to obtain the three-dimensional position of the signboard.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Collecting a street view panoramic image dataset along the road network, projecting the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and dividing the three-dimensional reconstruction sub-scenes according to the road network buffer zone;

[0044] identifying the signboard target in the street view image by using a pre-trained target detection network;

[0045] Performing scene text recognition on the signboard target to obtain text content and text line position of the signboard target;

[0046] Performing image feature extraction and feature point matching on the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and constructing a signboard entity according to the feature point set and the text content;

[0047] Perform similarity analysis on signboard entities in street view images from different perspectives to obtain a set of signboard entities representing the same signboard;

[0048] Based on the multi-view street view images, the street is reconstructed in three dimensions to generate a sparse three-dimensional point cloud of the scene, a consistency algorithm is used to fit the two-dimensional plane corresponding to the sign entity, and the distance between the two-dimensional plane and the three-dimensional point cloud is calculated to obtain the interior point of the two-dimensional plane;

[0049] The central coordinates of the inner points are calculated to obtain the three-dimensional position of the signboard.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0051] Collecting a street view panoramic image dataset along the road network, projecting the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and dividing the three-dimensional reconstruction sub-scenes according to the road network buffer zone;

[0052] identifying the signboard target in the street view image by using a pre-trained target detection network;

[0053] Performing scene text recognition on the signboard target to obtain text content and text line position of the signboard target;

[0054] Performing image feature extraction and feature point matching on the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and constructing a signboard entity according to the feature point set and the text content;

[0055] Perform similarity analysis on signboard entities in street view images from different perspectives to obtain a set of signboard entities representing the same signboard;

[0056] Based on the multi-view street view images, the street is reconstructed in three dimensions to generate a sparse three-dimensional point cloud of the scene, a consistency algorithm is used to fit the two-dimensional plane corresponding to the sign entity, and the distance between the two-dimensional plane and the three-dimensional point cloud is calculated to obtain the interior point of the two-dimensional plane;

[0057] The central coordinates of the inner points are calculated to obtain the three-dimensional position of the signboard.

[0058] The above-mentioned method and device for calculating the three-dimensional position of a sign based on street view images obtains a crowd-sourced open street view image dataset from the Internet, and combines deep learning technologies such as target detection and scene text recognition to automatically extract the sign target and its multimodal features in the street view image. By integrating image features with text features, the sign entities across perspectives are accurately aligned to ensure the uniqueness of the extracted sign entities. By reconstructing the sparse point cloud of the street scene in three dimensions, the mapping relationship between the two-dimensional feature points of the sign entity and the three-dimensional point cloud is established; and the plane equation and the set of internal points where the sign entity is located are fitted to effectively eliminate noise interference in the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of a flow chart of a method for calculating a three-dimensional position of a sign based on a street view image in one embodiment;

[0060] Figure 2 A schematic diagram of dividing a street view panoramic image into images of different natural perspectives in one embodiment;

[0061] Figure 3 An example diagram of a 3D reconstruction sub-scene divided along a road network buffer in one embodiment;

[0062] Figure 4 A diagram showing the structure of a Cascade R-CNN target detection network in one embodiment;

[0063] Figure 5 A pseudo code flow chart of fitting a signboard plane equation based on a RANSAC method in one embodiment;

[0064] Figure 6 is a structural block diagram of a device for calculating a three-dimensional position of a sign based on a street view image in one embodiment;

[0065] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] In one embodiment, Figure 1 As shown, a method for calculating the three-dimensional position of a sign based on a street view image is provided, comprising the following steps:

[0068] Step 102: collect a street view panoramic image dataset along the road network, project the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and divide the three-dimensional reconstruction sub-scenes according to the road network buffer.

[0069] In this step, the street view panoramic images in the study area can be crawled along the road network with a sampling interval of 10m, and the sub-scenes can be divided at intervals of 100 meters to facilitate 3D reconstruction. Figure 2 As shown in FIG. 3 , each street view panoramic image can be projected into 8 natural perspective images with a horizontal interval of 45°, a field of view of 60°, and an elevation angle of 5°. The resolution of each natural perspective image is 1024 × 1024. The 3D reconstruction sub-scenes divided by the road network buffer are shown in FIG. 3 .

[0070] Step 104: Identify the signboard target in the street view image through the pre-trained target detection network.

[0071] The target detection network may be an R-CNN network, which is used to identify signboard targets in street view images.

[0072] Step 106: Perform scene text recognition on the signboard target to obtain the text content and text line position of the signboard target.

[0073] Step 108, performing image feature extraction and feature point matching on the street view image in the 3D reconstruction sub-scene to obtain a feature point set in the sign target area, and constructing a sign entity based on the feature point set and the text content.

[0074] Step 110 , performing similarity analysis on the signboard entities in the street view images from different viewing angles to obtain a set of signboard entities representing the same signboard.

[0075] In this step, when the street view collection vehicle collects street view images along the street, the same sign is often photographed from multiple angles and positions, so a sign in the real scene may be detected in multiple images. In order to associate these sign entities with their corresponding unique locations, the image and text features are comprehensively considered to evaluate the similarity of the sign entities across viewpoints, thereby achieving accurate alignment of the sign entities.

[0076] Step 112, based on the multi-view street view images, the street is reconstructed in three dimensions to generate a sparse three-dimensional point cloud of the scene, a consistency algorithm is used to fit the two-dimensional plane corresponding to the sign entity, and the distance between the two-dimensional plane and the three-dimensional point cloud is calculated to obtain the inner point of the two-dimensional plane.

[0077] In this step, for the three-dimensional reconstruction step, first, local features are extracted and matched from multi-view street view images based on the SuperGlue deep learning network, and then a sparse three-dimensional point cloud of the scene is constructed through the Structure From Motion (SFM) algorithm. Due to the lack of camera focal length parameters in the street view image metadata, it is assumed that all pictures are taken by the same fixed-focal-length camera during the street view shooting process. Therefore, it is reasonable to assume that the shooting focal length of all pictures is 50mm. This assumption causes the reconstructed point cloud to have scale uncertainty, and the coordinate system needs to be aligned with the camera's geographic coordinates in the street view metadata as a reference. Coordinate system alignment is achieved by solving a similarity transformation, which includes a rotation matrix , scaling factor and translation vector Three basic components, similarity transformation equation is expressed as:

[0078] ;

[0079] in, represents the camera coordinates in the local coordinate system, Represents the camera geographic coordinates in the street view metadata. Based on at least three known camera coordinate points, the parameters can be solved , and , and transform the 3D point cloud into the geographic coordinate system.

[0080] Step 114, calculate the center coordinates of the inner point to obtain the three-dimensional position of the signboard.

[0081] In the above-mentioned method for calculating the three-dimensional position of signs based on street view images, a crowd-sourced open street view image dataset is obtained from the Internet, and deep learning technologies such as target detection and scene text recognition are combined to automatically extract sign targets and their multimodal features in street view images. By integrating image features with text features, the sign entities across perspectives are accurately aligned to ensure the uniqueness of the extracted sign entities. By reconstructing the sparse point cloud of the street scene in three dimensions, the mapping relationship between the two-dimensional feature points of the sign entity and the three-dimensional point cloud is established; and the plane equation and the set of internal points where the sign entity is located are fitted to effectively eliminate noise interference in the point cloud.

[0082] In one embodiment, the target detection network is a Cascade R-CNN target detection model; the backbone network of the Cascade R-CNN target detection model is Swin-Transformer; the rectangular detection frame coordinates of the sign target in the street view image are identified by the pre-trained Cascade R-CNN target detection model. and confidence.

[0083] Specifically, the Cascade R-CNN network is a two-stage target detection algorithm based on the Region-Convolution Neural Network (R-CNN) combined with a cascade structure. The R-CNN detection method mainly includes two stages. The first stage generates several candidate boxes with different aspect ratios on the feature map through the Region Proposals Network (RPN). The second stage classifies the selected candidate boxes and adjusts the coordinate offset of the candidate boxes through regression to make the predicted area of ​​the candidate boxes more accurate. The network structure of Cascade R-CNN is as follows: Figure 4 As shown in the figure, RPN is a probabilistic neural network, ROI Align represents ROI alignment, FC represents the fully connected layer, Cls represents the hidden layer output, Box represents the candidate box, and Output represents the final output. It is multi-level cascaded on the basis of R-CNN, that is, the bounding box outputted by the previous level is used as the candidate box of the next level, and the intersection over union (IoU) threshold of the candidate box is gradually increased. Specifically, the IoU thresholds in the H1, H2, and H3 stages are set to 0.5, 0.6, and 0.7 respectively. In the initial stage, the model uses a lower IoU threshold to better learn the features of difficult-to-detect targets, and in the later stage, the IoU threshold is increased to ensure the detection accuracy of the model. The model is trained on the Amap sign detection open source dataset for 100 rounds, and finally outputs the coordinates and confidence of the rectangular detection box of the sign target.

[0084] In one embodiment, scene text recognition is performed on the sign target based on the PaddleOCR framework to obtain the text content of the sign target. and text line positions.

[0085] In another embodiment, a SuperGlue deep learning network is used to perform image feature extraction and feature point matching on the street view image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the sign target area.

[0086] Specifically, the feature set extracted from street view image A and street view image B is: and , All are feature points. From the two-dimensional feature point coordinates and feature descriptors The feature matching relationship set can be expressed as .

[0087] In one embodiment, the signboard entity is constructed according to the feature point set and the text content as follows:

[0088] ;

[0089] in, Represents the coordinates of the rectangular detection box of the sign target, Represents text content. Represents the set of feature points within the target area of ​​the signboard.

[0090] In one embodiment, feature matching is performed on signboard entities in street view images of different viewing angles to obtain matching relationships. , if the matching relationship is the image feature matching relationship A subset of , then the normalized value of the matching relationship is calculated to obtain the image feature similarity:

[0091] ;

[0092] ;

[0093] in, and are the feature point sets of the signboard entity in street view image A and the signboard entity in street view image B, respectively. Represents the image feature similarity; the text similarity between the text content of the signboard entity in the street view image A and the text content of the signboard entity in the street view image B is determined by normalized Levenshtein distance:

[0094] ;

[0095] in, Indicates the text similarity, and represents the text content of the signboard entity in the street view image A and the signboard entity in the street view image B; according to the image feature similarity and the text similarity, the signboard entity similarity between the signboard entity in the street view image A and the signboard entity in the street view image B is obtained as follows:

[0096] ;

[0097] in, represents the similarity of the signboard entity, Represents a weighted value; if the sign entity similarity If the similarity threshold is exceeded, the signboard entity in the street view image A and the signboard entity in the street view image B are judged to be the same signboard, thereby obtaining the signboard entity set of the signboard object. . It can be represented by the set of all the signboard entities assigned to it in the scene, that is, .

[0098] In one embodiment, two-dimensional feature points of the signboard entity are established. The mapping relationship with the three-dimensional point cloud is The corresponding three-dimensional object point set is . Use the RANSAC method to fit the plane equation where the signboard entity is located , and calculate the distance between each 3D point and the plane equation to distinguish the internal point set belonging to the plane With outlier set The specific algorithm flow is as follows: Figure 5 As shown. Considering that a signboard often corresponds to multiple signboard entities, and in evaluating the quality of point cloud reconstruction, the number of 3D points is often positively correlated with the reconstruction quality. Therefore, this method selects the signboard entity set The five signboard entities with the largest number of inliers are merged into their inliers:

[0099] ;

[0100] According to the merged interior point set , calculate the center coordinates of all inner points as the three-dimensional position coordinates of the signboard. The coordinates of the inner point are , then the three-dimensional position of the sign in the ECEF coordinate system is:

[0101] ;

[0102] is the three-dimensional position of the sign.

[0103] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0104] In one embodiment, Figure 6 As shown, a device for calculating the three-dimensional position of a signboard based on a street view image is provided, comprising: a data acquisition module 602, a signboard target extraction module 604, a text extraction module 606, a signboard entity construction module 608, a similarity analysis module 610, a feature point matching module 612 and a position calculation module 614, wherein:

[0105] The data acquisition module 602 is used to collect a street view panoramic image dataset along the road network, project the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and divide the three-dimensional reconstruction sub-scenes according to the road network buffer;

[0106] A signboard target extraction module 604, configured to identify the signboard target in the street view image by using a pre-trained target detection network;

[0107] A text extraction module 606 is used to perform scene text recognition on the signboard target to obtain the text content and text line position of the signboard target;

[0108] A signboard entity construction module 608 is used to perform image feature extraction and feature point matching on the street view image in the 3D reconstruction sub-scene to obtain a feature point set in the signboard target area, and construct a signboard entity according to the feature point set and the text content;

[0109] A similarity analysis module 610 is used to perform similarity analysis on the signboard entities in the street view images from different viewing angles to obtain a set of signboard entities representing the same signboard;

[0110] The feature point matching module 612 is used to perform three-dimensional reconstruction of the street based on the multi-view street view images, generate a sparse three-dimensional point cloud of the scene, use a consistency algorithm to fit the two-dimensional plane corresponding to the sign entity, calculate the distance between the two-dimensional plane and the three-dimensional point cloud, and thereby obtain the interior point of the two-dimensional plane;

[0111] The position calculation module 614 is used to calculate the central coordinates of the inner point to obtain the three-dimensional position of the signboard.

[0112] In one embodiment, the target detection network 604 is a Cascade R-CNN target detection model; the backbone network of the Cascade R-CNN target detection model is Swin-Transformer; the sign target extraction module 604 is also used to identify the rectangular detection frame coordinates of the sign target in the street view image through the pre-trained Cascade R-CNN target detection model. and confidence.

[0113] In one embodiment, the text extraction module 606 is further used to perform scene text recognition on the sign target based on the PaddleOCR framework to obtain the text content of the sign target. and text line positions.

[0114] In one embodiment, the signboard entity construction module 608 is further configured to use the SuperGlue deep learning network to perform image feature extraction and feature point matching on the street view image in the 3D reconstruction sub-scene to obtain a feature point set within the signboard target area. .

[0115] In one embodiment, the signboard entity construction module 608 is further configured to construct the signboard entity according to the feature point set and the text content:

[0116] ;

[0117] in, Represents the coordinates of the rectangular detection box of the sign target, Represents text content. Represents the set of feature points within the target area of ​​the signboard.

[0118] In one embodiment, the similarity analysis module 610 is also used to perform feature matching on the signboard entities in the street view images from different perspectives to obtain matching relationships. , if the matching relationship is the image feature matching relationship A subset of , then the normalized value of the matching relationship is calculated to obtain the image feature similarity:

[0119] ;

[0120] ;

[0121] in, , the feature set extracted from street view image A and street view image B is and , are all feature points, and are the feature point sets of the signboard entity in street view image A and the signboard entity in street view image B, respectively. Indicates the similarity of image features;

[0122] The text similarity between the text content of the signboard entity in street view image A and the text content of the signboard entity in street view image B is determined by normalized Levenshtein distance:

[0123] ;

[0124] in, Indicates the text similarity, and Represents the text content of the signboard entity in the street view image A and the signboard entity in the street view image B;

[0125] According to the image feature similarity and the text similarity, the signboard entity similarity between the signboard entity in the street view image A and the signboard entity in the street view image B is obtained as follows:

[0126] ;

[0127] in, represents the similarity of the signboard entity, represents the weighted value;

[0128] If the signboard entity similarity If the similarity threshold is exceeded, the signboard entity in the street view image A and the signboard entity in the street view image B are judged to be the same signboard, thereby obtaining the signboard entity set of the signboard object. .

[0129] In one embodiment, the location calculation module 614 is further configured to select the signboard entity set The five signboard entities with the most inliers are merged to obtain the inlier set ;

[0130] Compute the interior point set The coordinates of the center point of all the interior points in are:

[0131] ;

[0132] in, The coordinates of the inner point are , is the three-dimensional position of the sign.

[0133] For the specific definition of the three-dimensional position calculation device of the sign based on the street view image, please refer to the definition of the three-dimensional position calculation method of the sign based on the street view image above, which will not be repeated here. Each module in the above-mentioned three-dimensional position calculation device of the sign based on the street view image can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0134] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for calculating the three-dimensional position of a sign based on a street view image is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0135] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0136] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0138] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for calculating the three-dimensional position of a sign based on a street view image, characterized in that: The method comprises: Collecting a street view panoramic image dataset along the road network, projecting the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and dividing the three-dimensional reconstruction sub-scenes according to the road network buffer zone; identifying the signboard target in the street view image by using a pre-trained target detection network; Performing scene text recognition on the signboard target to obtain text content and text line position of the signboard target; Performing image feature extraction and feature point matching on the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and constructing a signboard entity according to the feature point set and the text content; Perform similarity analysis on signboard entities in street view images from different perspectives to obtain a set of signboard entities representing the same signboard; Based on the multi-view street view images, the street is reconstructed in three dimensions to generate a sparse three-dimensional point cloud of the scene, a consistency algorithm is used to fit the two-dimensional plane corresponding to the sign entity, and the distance between the two-dimensional plane and the three-dimensional point cloud is calculated to obtain the interior point of the two-dimensional plane; Calculating the center coordinates of the inner point to obtain the three-dimensional position of the signboard; Constructing a signboard entity according to the feature point set and the text content, including: According to the feature point set and the text content, the signboard entity is constructed as follows: ； in, Represents the coordinates of the rectangular detection box of the sign target, Represents text content. Represents a set of feature points within the target area of ​​the signboard; Similarity analysis is performed on signboard entities in street view images from different perspectives to obtain a set of signboard entities representing the same signboard, including: Perform feature matching on signboard entities in street view images from different perspectives to obtain matching relationships , if the matching relationship is the image feature matching relationship A subset of , then the normalized value of the matching relationship is calculated to obtain the image feature similarity: ； ； in, , the feature set extracted from street view image A and street view image B is and , are all feature points, and are the feature point sets of the signboard entity in the street view image A and the signboard entity in the street view image B, respectively, representing the image feature similarity; The text similarity between the text content of the signboard entity in street view image A and the text content of the signboard entity in street view image B is determined by normalized Levenshtein distance: ; in, Indicates the text similarity, and Represents the text content of the signboard entity in the street view image A and the signboard entity in the street view image B; According to the image feature similarity and the text similarity, the signboard entity similarity between the signboard entity in the street view image A and the signboard entity in the street view image B is obtained as follows: ; in, represents the signboard entity similarity, represents the weighted value; If the signboard entity similarity If the similarity threshold is exceeded, the signboard entity in the street view image A and the signboard entity in the street view image B are judged to be the same signboard, thereby obtaining the signboard entity set of the signboard object. ; Calculate the center coordinates of the inner point to obtain the three-dimensional position of the signboard, including: Select the signboard entity collection The five signboard entities with the most inliers are merged to obtain the inlier set ; Compute the interior point set The coordinates of the center point of all the interior points in are: ; in, The coordinates of the inner point are , is the three-dimensional position of the sign.

2. The method according to claim 1, characterized in that The target detection network is a Cascade R-CNN target detection model; the backbone network of the Cascade R-CNN target detection model is Swin-Transformer; Using a pre-trained target detection network, identifying a sign target in the street view image includes: The coordinates of the rectangular detection box of the sign target in the street view image are identified by using the pre-trained Cascade R-CNN target detection model. and confidence.

3. The method according to claim 1, characterized in that: Performing scene text recognition on the signboard target to obtain text content and text line position of the signboard target includes: Based on the PaddleOCR framework, scene text recognition is performed on the signboard target to obtain the text content of the signboard target and text line positions.

4. The method according to claim 1, characterized in that: Performing image feature extraction and feature point matching on the street scene image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area includes: The SuperGlue deep learning network is used to extract image features and match feature points of the street view image in the 3D reconstruction sub-scene to obtain a feature point set in the sign target area. .

5. A device for calculating the three-dimensional position of a sign based on a street view image, used to implement the method for calculating the three-dimensional position of a sign based on a street view image as claimed in any one of claims 1 to 4, characterized in that: The device comprises: A data acquisition module is used to collect a street view panoramic image dataset along the road network, project the street view panoramic images in the street view panoramic image dataset into street view images of various perspectives, and divide the three-dimensional reconstruction sub-scenes according to the road network buffer; A signboard target extraction module, used for identifying the signboard target in the street view image through a pre-trained target detection network; A text extraction module, used to perform scene text recognition on the signboard target to obtain the text content and text line position of the signboard target; A signboard entity construction module is used to extract image features and match feature points on the street view image in the three-dimensional reconstruction sub-scene to obtain a feature point set in the signboard target area, and to construct a signboard entity based on the feature point set and the text content; A similarity analysis module is used to perform similarity analysis on signboard entities in street view images of different viewing angles to obtain a set of signboard entities representing the same signboard; A feature point matching module is used to perform three-dimensional reconstruction of the street based on multi-view street view images, generate a sparse three-dimensional point cloud of the scene, use a consistency algorithm to fit the two-dimensional plane corresponding to the sign entity, calculate the distance between the two-dimensional plane and the three-dimensional point cloud, and thereby obtain the interior point of the two-dimensional plane; The position calculation module is used to calculate the central coordinates of the inner point to obtain the three-dimensional position of the signboard.

Images