A method for obtaining urban bridge lifeline information

Through the Mask R-CNN model and camera internal and external parameters, combined with the function of the opencv library, the error problem in the measurement of the pier is solved, and the accurate acquisition of the pier height and position coordinates is achieved, and the measurement efficiency and accuracy are improved.

CN118644544BActive Publication Date: 2024-11-26HEFEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411124889.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-26
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

In the prior art, there are errors in measuring the height and position coordinates of the bridge pier, making it difficult to achieve accurate measurements, and manual measurements are time-consuming and labor-intensive.

Method used

The Mask R-CNN model is used to combine the camera's internal and external parameters, and the height and longitude coordinates of the bridge pier are calculated through the shooting point coordinates of the panoramic image, the internal parameters and angle of the shooting camera, and the contour information of the mask image is extracted using the opencv library for accurate results. information acquisition.

Benefits of technology

Accurate acquisition of the height and position coordinates of the bridge pier is achieved, the accuracy and efficiency of measurement are improved, and the error of manual measurement is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644544B_ABST
    Figure CN118644544B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing technology, and specifically to a method for obtaining lifeline information of urban bridges. After completing the training of the Mask R-CNN model, the present invention obtains a corresponding mask image through the Mask R-CNN model to display the bridge piers to be identified in the mask image. Then, the mask of the bridge piers to be identified is processed and extracted into data for standby use through the function in the opencv library. Then, the exif file generated when the commercial equipment is photographed is accessed to obtain the shooting coordinates, focal length, shooting azimuth and other information corresponding to the picture, and the data extracted from the mask are used as input and input into the photo information conversion module to accurately obtain the height and position coordinates of the bridge piers.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a method for acquiring lifeline information of a city bridge. Background Art

[0002] Urban infrastructure lifeline safety engineering is an important part of urban renewal and new urban infrastructure construction. As an important part of urban infrastructure construction, how to carry out bridge lifeline safety engineering related work efficiently and accurately has become the current mainstream research direction.

[0003] With the continuous improvement of current scientific and technological levels, computer image recognition technology has flourished, and its application in current bridge engineering is constantly expanding, and is developing towards a more extensive trend. At the same time, in bridge engineering practice, it is necessary to strengthen the in-depth analysis of computer image recognition technology, and to achieve good monitoring according to the characteristics of bridge engineering construction to prevent the occurrence of sudden problems, so that the project can proceed more smoothly and provide important support for subsequent engineering management.

[0004] Today, as research in the field of bridge groups flourishes, computer image recognition technology is increasingly favored in the field of bridge group research by conducting research on various aspects of bridge groups based on computer image recognition technology to reduce research costs and improve the versatility of research results. Bridge construction monitoring and control is a crucial part of bridge construction technology, and its goal is to ensure that the bridge reaches the designed state. During the entire construction process, by real-time monitoring of the actual state of the bridge structure and the surrounding environment, analyzing the error between the actual state and the ideal state, and applying modern control theory, the error is accurately identified, adjusted and predicted. Through this process, the bridge construction state can be as close to the ideal state as possible, ensuring that the bridge structure remains safe during the construction process, and ultimately meeting the requirements of the design and construction specifications to achieve the completed bridge state.

[0005] As an indispensable part of the bridge, the bridge pier plays an important role in supporting and positioning. In the process of bridge construction monitoring and control, the position and height of the bridge pier are pre-set, that is, the position coordinates and height of each completed bridge pier are fixed. In the process of detecting the status of the bridge pier, the height of the bridge pier is usually measured manually using a ruler on-site measurement, and the position coordinates of the bridge pier are measured by standing under the bridge pier with a handheld positioning device. Although this manual measurement method can measure the height and position coordinates of the bridge pier, it is time-consuming and labor-intensive; and because of the manual measurement method, there are certain errors in the measured height and position coordinates of the bridge pier, which makes it difficult to achieve the requirements of accurate measurement. Summary of the invention

[0006] In order to avoid and overcome the technical problems existing in the prior art, the present invention provides a method for obtaining lifeline information of urban bridges. The present invention can accurately obtain the height and position coordinates of bridge piers.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for obtaining urban bridge lifeline information comprises the following steps:

[0009] S1. Input two panoramic images containing the target bridge pier into the Mask R-CNN model to obtain the corresponding mask images, and extract the contour information of the bridge pier in the two mask images respectively;

[0010] S2, respectively obtaining the coordinates of the shooting points of the two panoramas, the internal parameters of the shooting camera and the corresponding angles during shooting, and storing them in the shooting database;

[0011] S3. Input the data in the shooting database and the profile information of the bridge pier into the photo information conversion module to calculate the height and longitude and latitude coordinates of the target bridge pier.

[0012] As a further solution of the present invention: the photo information conversion module is described as follows:

[0013] ;

[0014] ;

[0015] ;

[0016] In the formula, h cen Indicates the height of the bridge pier; lng a Indicates the shooting point a The longitude value of lng b Indicates the shooting point b The longitude value of f a,l Indicates the shooting points along the mask image a When pointing to the direction of the bridge pier, the converted focal length in the direction of the straight line where the leftmost side of the bridge pier outline is located; s a,l Indicates the shooting points along the mask image a When pointing in the direction of the pier, the pixel length of the leftmost side of the pier outline; Indicates the shooting points along the mask image a When pointing to the bridge pier, the shooting point a The leftmost side of the bridge pier is at the shooting point a The connecting line and shooting point in the horizontal planea The center line of the bridge pier is at the shooting point a The horizontal angle between the lines in the horizontal plane; f a,r Indicates the shooting points along the mask image a When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the rightmost side of the bridge pier outline is located; s a,r Indicates the shooting points along the mask image a When pointing in the direction of the pier, the pixel length of the rightmost side of the pier outline; Indicates the shooting points along the mask image a When pointing to the bridge pier, the shooting point a The rightmost side of the bridge pier is at the shooting point a The connecting line and shooting point in the horizontal plane a The center line of the bridge pier is at the shooting point a The horizontal angle between the lines in the horizontal plane;

[0017] f b,l Indicates the shooting points along the mask image b When pointing to the direction of the bridge pier, the converted focal length in the direction of the straight line where the leftmost side of the bridge pier outline is located; s b,l Indicates the shooting points along the mask image b When pointing in the direction of the pier, the pixel length of the leftmost side of the pier outline; Indicates the shooting points along the mask image b When pointing to the bridge pier, the shooting point b The leftmost side of the bridge pier is at the shooting point b The connecting line and shooting point in the horizontal plane b The center line of the bridge pier is at the shooting point b The horizontal angle between the lines in the horizontal plane; f b,r Indicates the shooting points along the mask image b When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the rightmost side of the bridge pier outline is located; s b,r Indicates the shooting points along the mask image b When pointing in the direction of the pier, the pixel length of the rightmost side of the pier outline; Indicates the shooting points along the mask image b When pointing to the bridge pier, the shooting point b The rightmost side of the bridge pier is at the shooting point b The connecting line and shooting point in the horizontal plane b The center line of the bridge pier is at the shooting point bThe horizontal angle between the lines in the horizontal plane; lng cen Indicates the longitude value of the bridge pier; lat cen Indicates the latitude value of the bridge pier.

[0018] As a further solution of the present invention: when using the Mask R-CNN model, the Mask R-CNN model needs to be trained. The training process of the Mask R-CNN model is as follows:

[0019] S11. Obtain the Mars coordinates of each bridge pier in the set area through the Amap geographic inverse coding system;

[0020] S12, converting the Martian coordinates of each bridge pier in the set area into longitude and latitude coordinates in the Baidu map coordinate system;

[0021] S13, obtaining street view images of each bridge pier in a set area from Baidu geographic database, and converting the street view images of each bridge pier into corresponding panoramic images composed of multiple street view distribution tile images;

[0022] S14, according to the panoramic view of the bridge pier and the longitude and latitude coordinates of the bridge pier, obtaining the longitude and latitude coordinates of the shooting point for shooting the panoramic view of the bridge pier, and storing the longitude and latitude coordinates in a shooting point coordinate set;

[0023] S15, using the panoramic JSAPI to process the longitude and latitude coordinates of each shooting point in the shooting point coordinate set, so as to obtain a panoramic image of the bridge pier observed at the shooting point through the longitude and latitude coordinates of the shooting point, and store each panoramic image in the training set;

[0024] S16. Input each panorama in the training set into the Mask R-CNN model to complete the training of the Mask R-CNN model.

[0025] As a further solution of the present invention: the specific process of obtaining the longitude and latitude coordinates of the shooting point in step S14 is as follows:

[0026] S141, numbering each street view distribution tile image in each panorama according to the conversion procedure between the street view distribution tile image and the coordinate information; combining the number with the pixel coordinates of the panorama to obtain the longitude and latitude coordinates corresponding to each pixel point in the panorama;

[0027] S142, taking the corresponding longitude and latitude coordinates as the center, draw a circle with a set radius, and the intersection of the circle and the road in the corresponding panoramic image is the shooting point;

[0028] S143, inputting the pixel coordinates of the shooting point in the panoramic image and the serial number in the panoramic image as input quantities into a conversion program, so as to obtain the longitude and latitude coordinates of the shooting point.

[0029] As a further solution of the present invention: performing a deletion operation on the training set: deleting the panoramic images of the bridge piers in the training set whose resolution is lower than a set value;

[0030] Then, data expansion operation is performed on the training set: the panoramas that are not deleted in the training set are rotated, translated, scaled or flipped to obtain corresponding new panoramas, and the new panoramas are stored in the training set to expand the number of panoramas in the training set.

[0031] As a further solution of the present invention: the two panoramic images input into the Mask R-CNN model in step S2 are both exif files; the exif files are input into the exifread function to extract the shooting coordinates of the corresponding shooting points, the internal parameters of the shooting camera, and the azimuth angle during shooting.

[0032] As a further solution of the present invention: after obtaining the two mask images of the target bridge pier, the approxPolyDP function in the opencv library is used to perform polygon fitting on the mask images to extract the contour information of the two mask images.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] After completing the training of the Mask R-CNN model, the present invention obtains the corresponding mask image through the Mask R-CNN model to display the bridge piers to be identified in the mask image. Then, the mask of the bridge piers to be identified is processed and extracted into data for standby use through the function in the opencv library. Then, the exif file generated when the commercial equipment is photographed is accessed to obtain the shooting coordinates, focal length, shooting azimuth and other information corresponding to the picture, and the data extracted from the mask are used as input and input into the photo information conversion module to accurately obtain the height and position coordinates of the bridge piers. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the main flow chart of the present invention.

[0036] Figure 2 This is the overall framework diagram of the Mask R-CNN model in the present invention.

[0037] Figure 3 A flow chart is established for the training set in the present invention.

[0038] Figure 4 This is a diagram demonstrating the mask effect in the present invention.

[0039] Figure 5 This is a schematic diagram of the imaging principle of the camera in the present invention.

[0040] Figure 6 The shooting point in the present invention a Schematic diagram of imaging.

[0041] Figure 7 It is a schematic diagram of a horizontal cross section of a rectangular bridge pier in the present invention.

[0042] Figure 8 This is a simplified three-dimensional schematic diagram of the bridge pier in the present invention.

[0043] Fig. 9 It is a vertical cross-sectional view of the bridge pier in the present invention.

[0044] Fig.10 It is a schematic diagram of horizontal observation in the present invention.

[0045] Fig.11 It is a bridge pier in the present invention.

[0046] Fig.12 It is the mask map generated by Mask R-CNN in the present invention.

[0047] Fig.13 It is the mask image of the bridge pier to be identified in the present invention.

[0048] Fig.14 This is a schematic diagram of the bridge pier mask after processing in the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1~Figure 3 In order to quickly extract the position and shape information of the bridge, a mathematical model based on Mask R-CNN and internal and external parameters of the camera is proposed to quickly obtain the height of the bridge pier and the coordinate information of the center point of the bridge pier in the picture. The present invention first obtains a Mask R-CNN model dedicated to bridge pier mask recognition by using Baidu panoramic picture training, and obtains a mask image of the rectangular bridge pier in the provided photo through the Mask R-CNN model. Then, the mask image is processed using the image processing function to obtain a smoother mask of the bridge pier and its contour information, and the processing of the photographed photo is preliminarily completed. After that, the exif file generated when the photo was taken is retrieved to view the key camera parameters such as the latitude and longitude information of the photo, the camera focal length and the shooting direction. Finally, the above information is mapped to the height and coordinate information of the selected bridge pier through a model built based on the principle of stereo vision.

[0051] The Mask R-CNN model is a more general and better instance segmentation model developed based on the object detection network Faster R-CNN and the fully convolutional network FCN. This instance segmentation model can perform pixel-level segmentation on instances in photos. Its main improvement is to add a mask prediction module of the FCN architecture on the basis of Faster R-CNN, and to add the advanced backbone network deep residual network Resnet and feature pyramid FPN. The overall framework of Mask R-CNN is as follows: Figure 2 As shown in the figure, the feature extraction network Resnet50-FPN is first used to extract the image to be processed to obtain the feature map of the image; then the RoI region is generated on the feature map with the help of the region proposal network RPN, and the pooling of the features of the corresponding position of the RoI region is completed in the RoIAlign layer. Finally, the classification and regression of the target frame are completed through the detection frame classification and detection frame regression and prediction modules, and the pixel-level segmentation of the target is completed in the mask prediction module. The masking process does not change the size of the photo, but only hides the non-bridge pier part of the photo.

[0052] To obtain the information corresponding to the bridge pier selected in the photo, it is first necessary to obtain the contour information of the corresponding area in the photo. Compared with the traditional target recognition algorithm, the mask generated by the Mask R-CNN model has high accuracy and high recognition efficiency; at the same time, by constructing a dedicated training data set, a mask that meets the usage requirements can be generated in a targeted manner. The recognition object of the present invention is the bridge pier, which belongs to a building and has a large volume, but the details still need to be labeled for personalized model training.

[0053] After completing the training of the Mask R-CNN model, the Mask R-CNN model can be used to obtain the mask of the bridge piers in the photo, and then the user can specify the bridge piers to be identified. The bridge piers to be identified are then masked using the functions of the opencv library, and then extracted into data for later use. Then access the exif file generated when the commercial equipment is used to obtain the shooting coordinates, camera focal length, shooting azimuth and other information corresponding to the photo; then, the data extracted together with the mask are used as input and input into the photo information conversion module, which processes the data to obtain the height, coordinates and other information of the bridge piers. The overall flow chart is as follows: Figure 1 shown.

[0054] 1. Data acquisition.

[0055] 1.1. Construction of Mask R-CNN training set.

[0056] It is intended to obtain bridge coordinate information, bridge street view images, street view related parameters, and street view distribution tile maps in Baidu geographic database.

[0057] Baidu Maps and Amap both store bridge coordinate information in the form of geocoding system point of interest (POI) coordinate records. After comparing the similarities and differences between the bridge POI information databases in the geocoding system of Baidu Maps and Amap, Amap, which records a larger amount of data, was selected to obtain bridge coordinate information.

[0058] The present invention takes a certain city as an example, uses the Amap geographic inverse coding system to obtain the response results of POI types of bridges and overpasses within the city, and designs a program to store the id, name, and location in the obtained json file into a project excel table.

[0059] It was observed that the acquired bridge coordinate information was repeated, so a program was designed to delete the repeated cells in the Excel table.

[0060] In order to use the bridge coordinate information from Amap to obtain the bridge street view in Baidu Map, the stored bridge coordinate information must be converted first. Using the coordinate conversion service provided by Baidu Map Open Platform, the source coordinate type is set to the Mars coordinates (gcj02) used by Amap, and the target coordinate type is set to the longitude and latitude coordinates (bd09ll) used by Baidu Map. The design program converts the bridge coordinate information recorded in the Excel table into the bridge coordinate information in the Baidu Map coordinate system.

[0061] There are four types of panoramic map related interfaces in Baidu Map Open Platform, namely Android panoramic SDK, iOS panoramic SDK, panoramic JSAPI and panoramic static image service API. Considering the background of the present invention, the interface of the panoramic static image service API can obtain limited information and cannot meet the needs well, so the panoramic JSAPI is selected as the service interface of the present invention.

[0062] Use the Panorama class of JavaScript API v3.0 to obtain the corresponding panorama according to the coordinates and obtain its detailed information.

[0063] Obviously, it can be found that the bridge panoramic view obtained only through the bridge coordinate information in the Amap geographic inverse coding information system cannot fully meet the needs of the present invention. Therefore, it is necessary to design a program to obtain more reasonable coordinates of the corresponding shooting points of the bridge, so as to obtain a good bridge panoramic view including the bridge piers.

[0064] Considering the various limitations of street view image collection, it is necessary to first obtain the distribution information of Baidu map panorama. Through the analysis of Baidu map website, the level 18 tile map of the panorama map distribution map within the urban area is obtained.

[0065] Based on the conversion program of tile map and coordinate information, the geographic coordinates corresponding to the pixel point can be obtained through the number and pixel coordinates of each tile map, that is, the geographic coordinates of the bridge shooting point can be converted into the tile map pixel coordinates of the bridge shooting point. Through the bridge geographic coordinate information obtained above, the tile map number of the bridge location can be obtained.

[0066] By processing the tile map using the opencv library, we can get the intersection of the main roads in the map. Clustering the processed results can get the location of the main road intersection in the tile map where the bridge is located. A circle with a geographic diameter of 20m is made with the coordinates as the center. The intersection of the circle and the main road in the map is the shooting point. The pixel coordinates of the shooting point in the map and the tile map number are input into the conversion program as input to obtain the geographic coordinates of the shooting point.

[0067] The panoramic JSAPI is used to obtain the corresponding panoramic image according to the geographic coordinates of the shooting point, and its detailed information is obtained and stored. The specific process of establishing the training set is as follows: Figure 3 shown.

[0068] 1.2. Obtain the internal and external parameters of the camera corresponding to the photo.

[0069] According to the principle of light propagation and camera photography, in order to complete the 3D reconstruction of a specific object in the photo through two photos, the necessary internal and external parameters of the camera include the image length of the two panoramic pictures ( img col )、Image Width( img row )、camera coordinates( lng a , lng b , lat a and lat b )、Camera azimuth( ), photo shooting focal length ( f ). Currently, commercial devices (including digital cameras, smart phones, etc.) will generate corresponding exif files when taking photos. Exif files are used to record the attribute information and shooting data of digital photos, including the various internal and external parameters of the camera required above.

[0070] In Python language, you can extract the internal and external parameter data of a photo by calling the exifread library.

[0071] 2. Data processing.

[0072] 2.1. Acquisition of pier photo contour data.

[0073] Through the above process, several photos (panoramic images) containing the target bridge piers were collected, and then these photos were processed using EISeg software to remove photos with too much noise, and a training set that meets the training requirements of the Mask R-CNN model was initially obtained. Finally, the photos were cropped, rotated, and other operations were performed to enhance the training set data, complete the expansion of the training set, and force the model to learn more robust features, thereby effectively improving the generalization ability of the model.

[0074] The general shape of the bridge pier is a rectangular parallelepiped, and it is arranged vertically on the corresponding ground. The bridge pier has four side faces and four side edges, that is, the cross section of the bridge pier is a rectangle.

[0075] After completing the training and adjustment of the Mask R-CNN model, a Mask R-CNN model that can identify bridge piers in the input panoramic image was obtained.

[0076] By using the prediction function interface of the Mask R-CNN model, the bridge pier image to be processed can be accurately processed and converted into the corresponding mask image. This process makes full use of the powerful prediction function of the Mask R-CNN model to ensure accurate identification and segmentation of the bridge pier, so that a mask image with detailed mask information can be generated. Then, the approxPolyDP function of the opencv library is used to fit the mask into a polygon to obtain a more detailed bridge pier contour map. The fitting effect is shown below. Figure 4 shown. Figure 4 The black part is the mask part, and the yellow part is the outline of the bridge pier.

[0077] Finally, use the findContours function in the opencv library to process the mask image selected by the user to obtain the contour information data corresponding to the mask.

[0078] 2.2. 3D reconstruction of photos.

[0079] According to the length of the two panorama images obtained above ( img col )、Image Width( img row )、camera coordinates( lng a , lng b , lat a and lat b )、Camera azimuth( ), photo shooting focal length ( f ) and the pier mask contour information, the following derivation can be completed.

[0080] Figure 5 This is the camera imaging principle diagram. The bridge pier represented by the blue line is inverted and presented on the camera imaging plane after passing through the lens in the camera. According to the imaging principle, the following results can be obtained:

[0081] ;

[0082] In the formula, h l Indicates the height of the leftmost side of the pier outline when the shooting point points to the pier in the mask image. s l It indicates the pixel length of the leftmost side of the pier outline when the shooting point points to the pier in the mask image. f l It indicates the converted focal length of the straight line along the leftmost side of the pier outline when the shooting point points to the pier in the mask image. d l Indicates the horizontal distance from the leftmost side of the pier outline to the shooting point in the mask image when the shooting point points to the pier. lng l Indicates the longitude value of the leftmost side of the pier outline when the shooting point points to the pier in the mask image. lat l Indicates the latitude value of the leftmost side of the pier outline when the shooting point points to the pier in the mask image. lng a For shooting point a The longitude value of . lat a For shooting point a In the process of taking photos, the shooting point is usually located in the diagonal direction of the cross section of the bridge pier, so the leftmost side and the rightmost side of the bridge pier are usually the two opposite sides of the bridge pier.

[0083] ;

[0084] In the formula, h r Indicates the height of the rightmost side of the pier outline when the shooting point points to the pier in the mask image. s r It indicates the pixel length of the rightmost side of the pier outline when the shooting point points to the pier in the mask image. f r It indicates the converted focal length of the straight line along the rightmost side of the pier outline when the shooting point points to the pier in the mask image. d r It indicates the horizontal distance from the rightmost side of the pier outline to the shooting point when the mask image points toward the pier. lngr Indicates the longitude value of the rightmost side of the pier outline when the shooting point points to the pier in the mask image; lat r Indicates the latitude value of the rightmost side of the pier outline when the shooting point points to the pier in the mask image.

[0085] When taking a photo, the imaging rule at the horizontal viewing angle also follows the above principle. Obviously, when the projection distance of the object from the camera in the direction of camera focus remains unchanged, the number of horizontal pixels occupied by the projection on the photo remains unchanged, and the left and right boundary angles change accordingly.

[0086] Based on the above analysis, Figure 6 The shooting point shown a ( lng a , lat a ) in the horizontal direction, the following formula can be listed:

[0087] ;

[0088] ;

[0089] At this time, Indicates the shooting points along the mask image a When pointing in the direction of the bridge pier, the horizontal angle between the leftmost side of the pier outline and the camera focus direction. img a,col For shooting point a The horizontal length of the mask image taken at . For shooting point a The real angle when the image is taken at Figure 6 Twice the angle between the line between the shooting point and point A and the vertical axis. Indicates the shooting points along the mask image a When pointing to the direction of the bridge pier, the horizontal angle between the rightmost side of the bridge pier outline and the camera focus direction. Figure 6 As shown, Figure 6 The imaging plane is the horizontal plane where the shooting point is located. With the center point O of the mask image as the origin, an image coordinate system O-XY is established in the mask image, and the horizontal axis X intersects with the right edge of the mask image at point A, and the vertical axis Y intersects with the lower edge of the mask image at point B. Shooting point a The line connecting the rightmost side of the pier outline in the mask image intersects the abscissa axis X at point D. The horizontal distance between point D and point O is the a l . Shooting point aThe line connecting the rightmost side of the pier outline in the mask image intersects the abscissa axis X at point C. The horizontal distance between point C and point O is the a r .

[0090] The following formula can be obtained by analyzing the imaging rules in each direction of the mask image:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] In the formula, f a,l Indicates the shooting points along the mask image a When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the leftmost side of the pier outline lies. f a,r Indicates the shooting points along the mask image a When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the rightmost side of the pier outline lies. h a,l Indicates the shooting points along the mask image a The height of the leftmost side of the pier profile when pointing towards the pier. h a,r Indicates the shooting points along the mask image a The height of the rightmost side of the pier profile when pointing towards the pier.

[0096] Similarly, for the shooting point b ( lng b , lat b ), shooting point c ( lng c , lat c ) and shooting points d ( lng d , lat d ), we will also get the corresponding calculation formula. Among them, lng b For shooting point b The longitude value of lat b For shooting point b The latitude value of . lngc For shooting point c The longitude value of lat c For shooting point c The latitude value of . lng d For shooting point d The longitude value of lat d For shooting point d The latitude value. The above method can be used to obtain the calculated height values ​​of the four sets of opposite side lines of a quadrilateral. In a small area, considering that the panorama of Baidu Map, the source of data, is framed by a framing car, it is assumed that the focal length of the camera is f Remain unchanged.

[0097] like Figure 7 As shown, the four sides of the same pier are marked as 1, 2, 3 and 4, and the longitude and latitude of these four sides are obtained at the same time. lng 1 is the longitude value of the first side, lat 1 is the latitude value of the first side. Ln 2 is the longitude value of the second side, lat 2 is the latitude value of the second side. Ln 3 is the longitude value of the third side, lat 3 is the latitude value of the third side. Ln 4 is the longitude value of the 4th side, lat 4 is the latitude value of the 4th side.

[0098] Based on the above division of sides, the longitude and latitude of the projection point of the center line of the pier cross section rectangle in the vertical direction can be represented by the longitude and latitude of any pair of oppositely arranged sides. That is:

[0099] ;

[0100] or

[0101] ;

[0102] in, lng cen for Figure 8 Center point of the cross section rectangle of the middle pier center The longitude value of lat cen for Figure 8 Center point of the cross section rectangle of the middle pier center The latitude value of .

[0103] For a bridge pier with a rectangular cross section, its above-ground part is simplified to a column with a rectangular cross section, and its three-dimensional diagram is as follows: Figure 8 shown.

[0104] like Fig. 9 As shown, for any pair of opposite sides, the height of the midline is h cen The calculation is as follows:

[0105] ;

[0106] Based on the formula We can get:

[0107] ;

[0108] Then we have:

[0109] ;

[0110] In the formula, It represents the angle between the line between the observation point and the outermost side and the line between the observation point and the midpoint of the rectangle. The type is azimuth.

[0111] from Fig.10 It can be observed that the shooting point a The straight line formed by connecting the leftmost side (i.e. the first side) of the bridge pier field of view forms the following straight line formula:

[0112] ;

[0113] Substituting into the linear form, we get:

[0114] ;

[0115] ;

[0116] ;

[0117] Similarly:

[0118] ;

[0119] ;

[0120] Then synthesize the following formula:

[0121] ;

[0122] and:

[0123] ;

[0124] We can get:

[0125] ;

[0126] Similarly, for the shooting point b have:

[0127] ;

[0128] In the above formula lng cen , lat cen and h cen All are to be evaluated, and the remaining values ​​are known values. The above formulas can be combined to form the solution of the three-variable linear equation system.

[0129] The solution is:

[0130] ;

[0131] ;

[0132] ;

[0133] The Mask R-CNN model can be used to convert the image lengths of the two panoramas ( img col )、Image Width( img row )、camera coordinates( lng a , lng b , lat a , lat b )、Camera azimuth( ), photo shooting focal length ( f ) and the pier mask contour information are mapped to the height and coordinate information of the selected pier.

[0134] 3. Practical application.

[0135] like Fig.11 As shown, two panoramic images, i.e., photos, of the bridge pier are obtained using mainstream commercial equipment, including digital cameras, smart phones and other devices or databases such as Baidu Map Panorama.

[0136] For photos taken with mainstream commercial equipment, the image lengths of the two panoramas can be obtained by reading the attached exif files ( img col )、Image Width( img row )、camera coordinates( lng a , lng b , lat a , lat b )、Camera azimuth( ), photo shooting focal length ( f ) and pier mask outline information. For Baidu Map panorama, you can use the image's corresponding id in Baidu Map to access the Panorama class to obtain key camera parameter data. Fig.11 For example, the camera related parameters obtained by accessing the Panorama class are shown in Table 1.

[0137] Table 1 Camera related parameters

[0138] ;

[0139] The image length, image width and camera focal length must be determined based on the parameters provided when the photo is acquired.

[0140] Then use the Mask R-CNN model to perform semantic segmentation on the image to complete the recognition of bridge piers and the generation of masks. Fig.12 shown.

[0141] from Fig.12 As can be seen in the figure, the trained Mask R-CNN model can complete the segmentation and recognition task of the bridge pier object in the picture, but the mask edge is messy. Therefore, the mask needs to be further processed. Fig.13 A mask image is shown, and then processed using the opencv library to obtain the following Fig.14 The more regular mask image shown in Figure 1 is then projected onto the horizontal plane and the corresponding shooting points are added to obtain Figure 6 The complete mask image required.

[0142] The contour information in the processed pier mask image is extracted and input into the information conversion module together with the internal and external parameter information corresponding to the photo. The height and coordinate results of the pier are shown in Table 2.

[0143] Table 2 Calculation results of bridge pier information

[0144] ;

[0145] The measured data of the bridge pier are shown in Table 3 below:

[0146] Table 3 Measured results of bridge pier information

[0147] ;

[0148] By comparing Table 2 and Table 3, it can be seen that the calculation results of the present invention are very close to the actual data, which means that the present invention accurately obtains the height and position coordinates of the bridge pier.

[0149] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for obtaining urban bridge lifeline information, characterized in that: The following steps are involved: S1. Input two panoramic images containing the target bridge pier into the Mask R-CNN model to obtain the corresponding mask images, and extract the contour information of the bridge pier in the two mask images respectively; S2, respectively obtaining the coordinates of the shooting points of the two panoramas, the internal parameters of the shooting camera and the corresponding angles during shooting, and storing them in the shooting database; S3, input the data in the shooting database and the profile information of the bridge pier into the photo information conversion module to calculate the height and longitude and latitude coordinates of the target bridge pier; the photo information conversion module is expressed as follows: ; ; ; In the formula, h cen Indicates the height of the bridge pier; lng a Indicates the shooting point a The longitude value of lng b Indicates the shooting point b The longitude value of lat a For shooting point a The latitude value of f a,l Indicates the shooting points along the mask image a When pointing to the direction of the bridge pier, the converted focal length in the direction of the straight line where the leftmost side of the bridge pier outline is located; s a,l Indicates the shooting points along the mask image a When pointing in the direction of the pier, the pixel length of the leftmost side of the pier outline; Indicates the shooting points along the mask image a When pointing to the bridge pier, the shooting point a The leftmost side of the bridge pier is at the shooting point a The connecting line and shooting point in the horizontal plane a The center line of the bridge pier is at the shooting point a The horizontal angle between the lines in the horizontal plane; f a,r Indicates the shooting points along the mask image a When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the rightmost side of the bridge pier outline is located; s a,r Indicates the shooting points along the mask image a When pointing in the direction of the pier, the pixel length of the rightmost side of the pier outline; Indicates the shooting points along the mask image a When pointing to the bridge pier, the shooting point a The rightmost side of the bridge pier is at the shooting point a The connecting line and shooting point in the horizontal plane a The center line of the bridge pier is at the shooting point a The horizontal angle between the lines in the horizontal plane; f b,l Indicates the shooting points along the mask image b When pointing to the direction of the bridge pier, the converted focal length in the direction of the straight line where the leftmost side of the bridge pier outline is located; s b,l Indicates the shooting points along the mask image b When pointing in the direction of the pier, the pixel length of the leftmost side of the pier outline; Indicates the shooting points along the mask image b When pointing to the bridge pier, the shooting point b The leftmost side of the bridge pier is at the shooting point b The connecting line and shooting point in the horizontal plane b The center line of the bridge pier is at the shooting point b The horizontal angle between the lines in the horizontal plane; f b,r Indicates the shooting points along the mask image b When pointing in the direction of the bridge pier, the converted focal length in the direction of the straight line where the rightmost side of the bridge pier outline is located; s b,r Indicates the shooting points along the mask image b When pointing in the direction of the pier, the pixel length of the rightmost side of the pier outline; Indicates the shooting points along the mask image b When pointing to the bridge pier, the shooting point b The rightmost side of the bridge pier is at the shooting point b The connecting line and shooting point in the horizontal plane b The center line of the bridge pier is at the shooting point b The horizontal angle between the lines in the horizontal plane; lng cen Indicates the longitude value of the bridge pier; lat cen Indicates the latitude value of the bridge pier.

2. A method for obtaining urban bridge lifeline information according to claim 1, characterized in that: When using the Mask R-CNN model, the Mask R-CNN model needs to be trained. The training process of the Mask R-CNN model is as follows: S11. Obtain the Mars coordinates of each bridge pier in the set area through the Amap geographic inverse coding system; S12, converting the Martian coordinates of each bridge pier in the set area into longitude and latitude coordinates in the Baidu map coordinate system; S13, obtaining street view images of each bridge pier in a set area from Baidu geographic database, and converting the street view images of each bridge pier into corresponding panoramic images composed of multiple street view distribution tile images; S14, according to the panoramic view of the bridge pier and the longitude and latitude coordinates of the bridge pier, obtaining the longitude and latitude coordinates of the shooting point for shooting the panoramic view of the bridge pier, and storing the longitude and latitude coordinates in a shooting point coordinate set; S15, using the panoramic JSAPI to process the longitude and latitude coordinates of each shooting point in the shooting point coordinate set, so as to obtain a panoramic image of the bridge pier observed at the shooting point through the longitude and latitude coordinates of the shooting point, and store each panoramic image in the training set; S16. Input each panorama in the training set into the Mask R-CNN model to complete the training of the Mask R-CNN model.

3. A method for obtaining urban bridge lifeline information according to claim 2, characterized in that: The specific process of obtaining the longitude and latitude coordinates of the shooting point in step S14 is as follows: S141, numbering each street view distribution tile image in each panorama according to the conversion procedure between the street view distribution tile image and the coordinate information; combining the number with the pixel coordinates of the panorama to obtain the longitude and latitude coordinates corresponding to each pixel point in the panorama; S142, taking the corresponding longitude and latitude coordinates as the center, draw a circle with a set radius, and the intersection of the circle and the road in the corresponding panoramic image is the shooting point; S143, inputting the pixel coordinates of the shooting point in the panoramic image and the serial number in the panoramic image as input quantities into a conversion program, so as to obtain the longitude and latitude coordinates of the shooting point.

4. A method for obtaining urban bridge lifeline information according to claim 3, characterized in that: Delete the training set: delete the panoramic images in the training set whose pier resolution is lower than the set value; Then, data expansion operation is performed on the training set: the panoramas that are not deleted in the training set are rotated, translated, scaled or flipped to obtain corresponding new panoramas, and the new panoramas are stored in the training set to expand the number of panoramas in the training set.

5. A method for obtaining urban bridge lifeline information according to claim 4, characterized in that: The two panoramic images input into the Mask R-CNN model in step S2 are both exif files. By inputting the exif files into the exifread function, the shooting coordinates of the corresponding shooting points, the intrinsic parameters of the shooting camera, and the azimuth angle at the time of shooting can be extracted.

6. A method for obtaining urban bridge lifeline information according to claim 5, characterized in that: After obtaining the two mask images of the target bridge piers, the approxPolyDP function in the opencv library is used to perform polygon fitting on the mask images to extract the contour information of the two mask images.