A real-time acquisition and processing method and system for surveying and mapping operation data

Through the building extraction neural network model and image matching technology, the extraction problem under the height offset of the building during satellite shooting is solved, the building extraction accuracy is improved, and the acquisition cost is reduced through point cloud data reuse, realizing the Efficient building color.

CN118334056BActive Publication Date: 2024-11-22YICHANG SURVEYING & MAPPING INST CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410502708.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-11-22
Estimated Expiration
2044-04-25

AI Technical Summary

Technical Problem

The prior art is difficult to accurately extract building information in high-resolution remote sensing images when building height is offset during satellite shooting, and the three-dimensional color allocation technology relies on panoramic camera acquisition, resulting in low point cloud data reuse and high acquisition cost. .

Method used

The building extraction neural network model is adopted, and the feature extraction, decoding and location inference modules are combined with image matching and color assignment technology to improve the accuracy of building extraction and realize the multiplexing of point cloud data.

Benefits of technology

It improves the accuracy and generalization of building extraction, reduces the cost of point cloud data acquisition, and achieves efficient color allocation for built or unconstructed point cloud models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118334056B_ABST
    Figure CN118334056B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of surveying and mapping technology, and provides a real-time acquisition and processing method and system for surveying and mapping operation data, the method comprising: obtaining urban building images and interactive information of urban building images; inputting urban building images and interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; performing image matching on the urban building images and their corresponding segmented images to obtain matching feature points; the matching feature points include a first feature point and a second feature point; and coloring the urban building images based on the first feature point and the second feature point. The present invention extracts buildings based on urban building images and interactive information, has a higher accuracy for offset and roof segmentation extraction, improves the accuracy of building extraction, and can color buildings without re-collecting point cloud data and point cloud modeling, thereby realizing the reuse of point cloud data and reducing acquisition costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of surveying and mapping technology, and in particular to a method and system for real-time acquisition and processing of surveying and mapping operation data. Background Art

[0002] The distribution of urban buildings is one of the important research indicators in many fields such as urban planning, disaster management, and geographic information systems. Building extraction is a key step in obtaining urban building distribution information, which is of great significance for understanding urban structure, guiding urban development, and responding to emergencies.

[0003] In the prior art, buildings are mainly extracted through high-resolution remote sensing images. First, high-resolution remote sensing images of the target area are obtained using satellites, and then building information is further extracted using statistics or neural network models. However, for situations where the satellite is completely perpendicular to the building when shooting, or almost perpendicular to the building when shooting, and there is a large overlap between the roof and the seat, the neural network model can achieve a good extraction effect by directly performing semantic segmentation of the building roof; however, for the case of high offset, that is, the side, roof and corner of a large area of ​​the building captured by the satellite, the effect of the existing technical solution is not ideal, and it is difficult to accurately extract the building information in the high-resolution remote sensing image. At the same time, when it is necessary to color the building, it is colored by three-dimensional coloring technology. However, this method is too dependent on the images collected by the panoramic camera during shooting, and the reusability of the point cloud data is not high. For the same area at different time periods, multiple acquisitions and modeling are required, which increases the acquisition cost. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a real-time acquisition and processing method and system for surveying and mapping operation data, so as to improve the accuracy of building extraction and reduce the cost of point cloud data acquisition.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A real-time acquisition and processing method for surveying and mapping operation data, comprising:

[0007] Acquire city building images and interactive information of the city building images;

[0008] The urban building image and the interactive information are input into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; the building extraction result includes a building roof extraction result, a building offset extraction result and a building location extraction result;

[0009] Performing image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly;

[0010] The city building image is colored based on the first feature point and the second feature point.

[0011] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, the urban building image and the interactive information are input into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model, including:

[0012] Inputting the city building image and the interaction information into the feature extraction module, obtaining the previous phase image block feature map and the interaction feature output by the feature extraction module;

[0013] Inputting the previous phase image block feature map and the interactive feature into the feature decoding module, and obtaining the building roof extraction result and the building offset extraction result output by the feature decoding module;

[0014] The building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module.

[0015] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, the urban building image and the interactive information are input into the feature extraction module to obtain the previous phase image block feature map and interactive features output by the feature extraction module, including:

[0016] Extracting features from the city building image based on an image feature extraction network to obtain a city building image feature map;

[0017] Encoding the interaction information based on an interaction information encoder to obtain interaction features;

[0018] According to the city building image and the interaction information, the city building image feature map and the interaction feature are paired, and the previous phase image block feature map and the interaction feature are output.

[0019] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, the preceding phase image block feature map and the interactive feature are input into the feature decoding module to obtain the building roof extraction result and the building offset extraction result output by the feature decoding module, including:

[0020] Extracting information from the previous phase image block feature map and the interaction feature based on a bidirectional attention network to obtain an offset token and a semantic mask token;

[0021] Inputting the offset token into a first fully connected network to obtain a building offset extraction result output by the first fully connected network;

[0022] The semantic mask token is input into a second fully connected network to obtain a building roof extraction result output by the second fully connected network.

[0023] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, the building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module, including:

[0024] According to the building offset extraction result, an interpretation head is selected to perform regression reasoning to obtain a roof offset;

[0025] Performing offset correction on the building roof extraction result according to the roof offset to obtain the building location extraction result;

[0026] The specific steps of selecting an interpretation head for regression reasoning according to the building offset extraction result include:

[0027] The interpretation head for regression reasoning is selected according to the building offset extraction result, and its expression is:

[0028]

[0029] Among them, ω represents the selector, which is a vector composed of 0 and 1, indicating whether to let Participate in the regression of i offset; is the prediction of the offset of i by the Aug head, is the prediction of the offset of i by the Ref header, O i is the actual prediction result for the i offset.

[0030] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, coloring the urban building image based on the first feature point and the second feature point includes:

[0031] Determine a first conversion matrix between the urban building image and the segmented image based on the first feature point and the second feature point;

[0032] The point cloud model is colored based on the segmented image, the first transformation matrix and the second transformation matrix; the second transformation matrix is ​​the transformation matrix between the urban building image and the point cloud data; the point cloud model is determined based on the point cloud data; each point in the point cloud model corresponds to each point in the point cloud data.

[0033] According to a real-time acquisition and processing method for surveying and mapping operation data provided by the present invention, the second conversion matrix is ​​determined by calibration of the urban building image and the point cloud data;

[0034] The coloring of the point cloud model based on the segmented image, the first transformation matrix and the second transformation matrix includes:

[0035] Determining a conversion relationship between the segmented image and the point cloud data based on the first conversion matrix and the second conversion matrix;

[0036] Based on the conversion relationship, the RGB value of each point on the segmented image is assigned to the corresponding point of the point cloud model.

[0037] The present invention also provides a real-time acquisition and processing system for surveying and mapping operation data, comprising:

[0038] An acquisition module, used to acquire city building images and interaction information of the city building images;

[0039] A building extraction module, used for inputting the urban building image and the interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; the building extraction result includes a building roof extraction result, a building offset extraction result and a building location extraction result;

[0040] An image matching module is used to perform image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly;

[0041] The building coloring module is used to color the city building image based on the first feature point and the second feature point.

[0042] The present invention also provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby realizing any of the above-mentioned methods for real-time acquisition and processing of surveying and mapping operation data.

[0043] The present invention also provides a non-transitory computer-readable storage medium, characterized in that a computer software program is stored in the storage medium, and when the computer software program is executed by a processor, it implements the real-time acquisition and processing method of surveying and mapping operation data as described in any of the above-mentioned methods.

[0044] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for real-time acquisition and processing of surveying and mapping operation data.

[0045] The beneficial effects of the present invention are: building extraction based on urban building images and interactive information has higher accuracy in offset and roof segmentation extraction and has stronger generalization. The performance in unknown data sets is still more accurate, which improves the accuracy of building extraction. For elements such as buildings and terrain, their point cloud models are not frequently changed. Therefore, the present invention only needs to collect images of the target without re-collecting point cloud data and point cloud modeling, and can color the constructed or unconstructed point cloud models, that is, color the buildings, thereby realizing the reuse of point cloud data and reducing the acquisition cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of the real-time acquisition and processing method of surveying and mapping operation data provided by the present invention;

[0047] Figure 2 It is a structural schematic diagram of the real-time acquisition and processing system for surveying and mapping operation data provided by the present invention;

[0048] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0052] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0053] Reference Figure 1 , Figure 1 The figure is a flow chart of the real-time acquisition and processing method for surveying and mapping operation data provided by the present invention. Taking the real-time acquisition and processing system as the execution subject, one of the manifestations of the real-time acquisition and processing system is a surveying and mapping system. The real-time acquisition and processing method for surveying and mapping operation data provided by the present invention includes:

[0054] Step 10: Acquire city building images and interactive information of city building images.

[0055] Optionally, the surveying and mapping system obtains an image of a city building, which is a high-resolution remote sensing image acquired from a satellite. The image may be an original satellite remote sensing image with the same resolution and size, or may be an image of a city building obtained by preprocessing the original image of the building.

[0056] Optionally, the original building image is preprocessed to obtain the urban building image as follows: a preprocessed original building image Img is cropped using a sliding window of uniform size and no overlap to obtain image blocks of the same size; a remote sensing image of size 1024×1024×3 is obtained, wherein pixels whose height of the last row of the sliding window image block is less than 1024 and whose width of the last column of the sliding window image block is less than 1024 are filled with 0 values; and before inputting the model, the original image is compressed to 256×256×3 using four times downsampling.

[0057] It can be understood that the urban building images used as input for the building extraction neural network model obtained through the above steps in the embodiment of the present invention are image blocks with the same size, and the building extraction neural network model is used to extract buildings from the image blocks, so that the building extraction neural network model pays more attention to local features in the high-resolution remote sensing image during extraction, reducing the influence of the large-scale distribution of buildings and other environmental features in the remote sensing image on the building extraction results.

[0058] Optionally, the surveying and mapping system obtains interactive information of urban building images, wherein the interactive information of urban building images may include the type and purpose of the building, the status of the building, the scale and height of the building, the spatial relationship between the buildings, the design style of the building and the functional services of the building.

[0059] Step 20, inputting the urban building images and interaction information into the building extraction neural network model to obtain the building extraction results output by the building extraction neural network model.

[0060] Optionally, the surveying and mapping system inputs the urban building images and interaction information into the building extraction neural network model to obtain the building extraction results output by the building extraction neural network model, wherein the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module, and the building extraction results include building roof extraction results, building offset extraction results and building location extraction results.

[0061] Optionally, the surveying and mapping system inputs the urban building images and interaction information into the building extraction neural network model to obtain the building extraction results output by the building extraction neural network model, including: inputting the urban building images and interaction information into the feature extraction module to obtain the previous phase image block feature map and interaction features output by the feature extraction module, inputting the previous phase image block feature map and interaction features into the feature decoding module to obtain the building roof extraction results and building offset extraction results output by the feature decoding module, and inputting the building roof extraction results and building offset extraction results into the location reasoning module to obtain the building location extraction results output by the location reasoning module.

[0062] Step 30, performing image matching on the urban building image and its corresponding segmented image to obtain matching feature points.

[0063] Optionally, the surveying and mapping system segments the urban building image to obtain a segmented image. For example, a threshold-based segmentation method, an edge-based segmentation method, or a region-based segmentation method may be used to segment the urban building image to obtain a segmented image.

[0064] Optionally, the surveying and mapping system can first determine the resolution of the urban building image. The image resolution determines the number and quality of the extracted feature points. The segmented image should also meet the resolution requirements to maintain the stability of feature point extraction after the image is compressed, stretched, and transformed during the image feature extraction process. In addition, before performing image matching, the surveying and mapping system can adjust the image brightness by calculating the grayscale mean in the urban building image and the grayscale mean in the segmented image, thereby increasing the robustness and robustness of feature point extraction. Afterwards, the surveying and mapping system can perform image matching between the urban building image and the segmented image (for example, by using a brute force method, a nearest neighbor search algorithm, etc.) to obtain matching feature points. Among them, the matching feature points include a first feature point and a second feature point. The first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image, and the first feature point is matched one-to-one with the second feature point.

[0065] Step 40: coloring the urban building image based on the first feature point and the second feature point.

[0066] Optionally, the surveying and mapping system colors the image of the urban building according to the first feature point and the second feature point.

[0067] The embodiment of the present invention extracts buildings based on urban building images and interactive information, and has higher accuracy in offset and roof segmentation extraction and stronger generalization. The performance in unknown data sets is still more accurate, which improves the accuracy of building extraction. For elements such as buildings and terrain, their point cloud models are not frequently changed. Therefore, the present invention only needs to collect images of the target without re-collecting point cloud data and point cloud modeling, and can color the constructed or unconstructed point cloud models, that is, color the buildings, thereby realizing the reuse of point cloud data and reducing the acquisition cost.

[0068] Optionally, coloring the urban building image based on the first feature point and the second feature point includes:

[0069] Determine a first conversion matrix between the urban building image and the segmented image based on the first feature point and the second feature point;

[0070] The point cloud model is colored based on the segmented image, the first transformation matrix, and the second transformation matrix.

[0071] Optionally, the surveying and mapping system may calculate the transformation relationship between the urban building image and the segmented image by matching the first feature point and the second feature point, and record it as the first transformation matrix X.

[0072] Optionally, the surveying and mapping system can color the point cloud model based on the segmented image, the first transformation matrix X and the second transformation matrix Y, the second transformation matrix Y is the transformation matrix between the urban building image and the point cloud data, the point cloud model is determined based on the point cloud data, and each point in the point cloud model corresponds to each point in the point cloud data.

[0073] Specifically, the surveying and mapping system can obtain the color attributes of the segmented image and assign color to the point cloud model through the first transformation matrix and the second transformation matrix.

[0074] Optionally, the second transformation matrix is ​​determined by calibrating the city building image and point cloud data.

[0075] Coloring the point cloud model based on the segmented image, the first transformation matrix and the second transformation matrix includes:

[0076] Determine a conversion relationship between the segmented image and the point cloud data based on the first conversion matrix and the second conversion matrix;

[0077] Based on the conversion relationship, the RGB value of each point on the segmented image is assigned to each point of the corresponding point cloud model.

[0078] In the present invention, the second conversion matrix Y of the urban building image and the point cloud data can be obtained by calibration. The surveying and mapping system can determine the conversion relationship between the segmented image and the point cloud data based on the first conversion matrix X and the second conversion matrix Y. For example, the point cloud data coordinates corresponding to any point P (x, y) on the segmented image are (XP*Y). Each point of the point cloud model corresponds to each point of the point cloud data. The surveying and mapping system can use the above conversion relationship to assign the RGB values ​​of each point on the segmented image to each point of the corresponding point cloud model, thereby completing the update of the point cloud model.

[0079] The embodiment of the present invention colors the point cloud model based on the segmented image, the first transformation matrix and the second transformation matrix. It only needs to collect the image of the target, and there is no need to re-collect the point cloud data and point cloud modeling. That is, the constructed or unconstructed point cloud model can be colored, thereby realizing the reuse of the point cloud data and reducing the acquisition cost.

[0080] In one embodiment, the city building image and the segmented image are matched to obtain matching feature points, including:

[0081] Based on the feature extraction algorithm, the image features of the urban building image and the segmented image are extracted respectively to obtain the image feature points;

[0082] Perform feature matching on the image feature points to obtain matching feature points.

[0083] Optionally, the surveying and mapping system can use a feature extraction algorithm to extract the image features of the urban building image and the segmented image respectively to perform image feature matching. Specifically, the feature extraction algorithm may include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Harris corner detection, etc. After image feature extraction, image feature points of the urban building image and the segmented image can be obtained. The surveying and mapping system can obtain matching feature points by performing feature matching on the image feature points. Specifically, the surveying and mapping system can obtain matching feature points by performing spatial nearest neighbor value calculation on the multi-dimensional feature vector.

[0084] The embodiment of the present invention matches the urban building image with the segmented image through feature extraction and feature matching to obtain matching feature points, so as to determine the first transformation matrix, and then color the point cloud model, providing data preparation for point cloud coloring, thereby realizing the reuse of point cloud data and reducing the acquisition cost.

[0085] In one embodiment, determining a segmented image based on a city building image includes:

[0086] Based on the imaging principle of urban building images, the urban building images are projected and segmented to obtain segmented images.

[0087] Specifically, the mapping system uses the image coordinates ((0, 0)) as the starting point, the y-axis direction as the 0° longitude direction, and the x-axis direction as the equator for projection, and divides the projected urban building image into multiple segmented images with different longitudes and latitudes as boundaries based on longitude and latitude. The image segmented in this way can support image segmentation of any size (i.e., the size of the image segmentation is changed according to the projection range).

[0088] The embodiment of the present invention obtains a segmented image by projecting and segmenting the image of the urban building, thereby ensuring the image matching accuracy and the accuracy of the point cloud coloring.

[0089] In one embodiment, the real-time acquisition and processing method for surveying and mapping operation data further includes:

[0090] Determine the projection area of ​​the segmented image through boundary points;

[0091] The positional relationship between the first feature point and the projection area is determined by the maximum area method to obtain a determination result;

[0092] Based on the judgment result, the quality of image matching is determined.

[0093] After image matching, the surveying and mapping system can also check the matching quality of image feature points. The surveying and mapping system can determine the projection area of ​​the urban building image and the segmented image respectively through the image boundary points, and then perform projection calculations on the feature points. Assuming that the projection coordinates of the feature point of image A on image B are (x, y), the surveying and mapping system can use the maximum area method (any point falls into the plane, and the area of ​​the figure composed of the four points on the plane and the point is calculated) to determine whether the feature point of image A is outside the projection area of ​​image B, thereby calculating the proportion of points within the projection area. The higher the proportion, the higher the image matching quality.

[0094] Specifically, the surveying and mapping system can determine the projection area of ​​the segmented image through the boundary points; determine the positional relationship between the first feature point and the projection area of ​​the segmented image through the maximum area method to obtain a determination result; and determine the quality of image matching based on the determination result. It can also determine the projection area of ​​the urban building image through the boundary points; determine the positional relationship between the second feature point and the projection area of ​​the urban building image through the maximum area method to obtain a determination result; and determine the quality of image matching based on the determination result.

[0095] The embodiment of the present invention can further ensure the accuracy of point cloud coloring by checking the quality of image matching, directly performing point cloud coloring when the image matching quality is qualified, and re-matching until the image matching quality is qualified and then performing point cloud coloring when the image matching quality is unqualified.

[0096] Optionally, the urban building image and the interaction information are input into a feature extraction module to obtain a feature map of the previous phase image block and interaction features output by the feature extraction module, including:

[0097] Based on the image feature extraction network, the image of the urban building is subjected to feature extraction to obtain the image feature map of the urban building;

[0098] Encode the interactive information based on the interactive information encoder to obtain interactive features;

[0099] According to the urban building images and interaction information, the urban building image feature map and the interaction features are paired, and the previous phase image block feature map and the interaction features are output.

[0100] The image feature extraction network in the embodiment of the present invention may be a ViT network obtained through unsupervised training; the front-phase image block feature map and the interaction feature in the embodiment of the present invention correspond to the input token.

[0101] In one embodiment of the present invention, unsupervised training ViT is used to extract features from the input image, and Prompt Encoder is used to encode the input interactive information to obtain the front-phase image block feature map Fea of ​​two groups of corresponding tokens. img , interactive features Feapro After multiple attention and self-attention mechanisms of the image feature extraction network, the final output feature map Fea img The size is 256×256×32.

[0102] Optionally, the feature map of the previous phase image block and the interactive features are input into the feature decoding module to obtain the building roof extraction result and the building offset extraction result output by the feature decoding module, including:

[0103] Based on the bidirectional attention network, information is extracted from the feature map of the previous phase image block and the interactive features to obtain the offset token and semantic mask token;

[0104] Inputting the offset token into the first fully connected network to obtain a building offset extraction result output by the first fully connected network;

[0105] The semantic mask token is input into the second fully connected network to obtain the building roof extraction result output by the second fully connected network.

[0106] In an embodiment of the present invention, a bidirectional attention network is used to perform multi-layer self-attention and cross-attention processing on the feature map of the previous phase image block and the interaction feature to extract the offset token and the semantic mask token. During the information extraction process, the size of the offset token and the semantic mask token remains unchanged.

[0107] In an embodiment of the present invention, the offset token is input to the first fully connected network, specifically, first through a forward connection layer, and activated by ReLU, and then through a fully connected layer, the number of nodes of the channel is regressed to 2 channels. The two channels respectively represent the coordinate position of the offset in the plane rectangular coordinate system. The semantic mask token is input to the second fully connected network, specifically, first through a forward connection layer, the forward connection layer of the second fully connected network has the same or similar structure as the forward connection layer of the first fully connected network, but the parameters are not shared, and the matrix operation in the forward connection layer is performed to obtain different semantic layer information under the same shared prompt; and then after the disambiguation operation, the model will fix the final roof representative token to obtain the building roof extraction result.

[0108] In one embodiment of the present invention, based on the feature map Fea of ​​the input image block img , and interactive features Fea pro , use a two-way transformer to extract information and output the token Off corresponding to the offset corresponding to the instance to , the token Mas corresponding to the semantic mask to Token Off to Through a set of fully connected networks, the regression offset Off; the token Mas toAfter passing through the fully connected layer, it is combined with the feature map Fea img Perform a product operation and then perform an upsampling to obtain the final instance segmentation mask Mas. to It will first pass through a feed forward network (FFN), the number of nodes in the hidden layer of the forward connection layer is 1024, and ReLU is used for activation, and the number of hidden layers is 4. Then it passes through a fully connected layer to regress the number of nodes in the channel to 2 channels. The two channels represent the coordinate positions of the offset in the plane rectangular coordinate system. to It will pass through a non-parametric shared FFN with a similar structure to the above, and then undergo matrix operations in the last layer, with the number of channels becoming 4, representing different semantic layers of information under the same shared prompt. After the disambiguation operation, the model will fix the final roof representative token.

[0109] Optionally, the building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module, including:

[0110] According to the building offset extraction results, an interpretation head is selected for regression reasoning to obtain the roof offset;

[0111] The building roof extraction result is offset corrected according to the roof offset to obtain the building location extraction result.

[0112] When performing location inference in the embodiment of the present invention, multiple different offset decoders will perform joint prediction. In view of the loss caused by the uncertainty of the building length during the prediction process, the model will autonomously select the appropriate decoding head through a ROAM system (Reference Offset Augmentation Module) to perform the final inference prediction. The Distance Non-Max Suppression algorithm (DNMS algorithm) used subsequently will use the offset with better prediction results to suppress the fuzzy offset direction based on prior statistical knowledge, thereby obtaining a better overall prediction result.

[0113] The interpretation head for regression inference is selected according to the building offset extraction result, and its expression is:

[0114]

[0115] Among them, ω represents the selector, which is a vector composed of 0 and 1, indicating whether to let Participate in the regression of this offset; is the prediction of the offset of i by the Aug head, is the prediction of the offset of i by the Ref header, O i is the actual prediction result for the i offset;

[0116] According to the building offset extraction results, after selecting the interpretation head for regression reasoning, the following steps are also included: directional suppression of the fuzzy offset, the expression is:

[0117]

[0118] Among them, K represents the Gaussian kernel function, x is the input sample, l represents the entire sample set, σ is the standard deviation of the sample set, and e is a natural constant. After counting the lengths and directions of all offsets within the range, the Gaussian distance between the length of each offset and the mean of all offsets is calculated as the influence weight on the direction to correct the direction. The algorithm can be used iteratively to reach the final balance point, which is the regression inference result.

[0119] The real-time acquisition and processing system for surveying and mapping operation data provided by the present invention is described below. The real-time acquisition and processing system for surveying and mapping operation data described below and the real-time acquisition and processing method for surveying and mapping operation data described above can be referred to each other. Figure 2 As shown, the system includes:

[0120] An acquisition module 201 is used to acquire city building images and interaction information of the city building images;

[0121] The building extraction module 202 is used to input the urban building image and the interactive information into the building extraction neural network model to obtain the building extraction result output by the building extraction neural network model; the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; the building extraction result includes a building roof extraction result, a building offset extraction result and a building location extraction result;

[0122] The image matching module 203 is used to perform image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly;

[0123] The building coloring module 204 is configured to color the city building image based on the first feature point and the second feature point.

[0124] The embodiment of the present invention extracts buildings based on urban building images and interactive information, and has higher accuracy in offset and roof segmentation extraction and stronger generalization. The performance in unknown data sets is still more accurate, which improves the accuracy of building extraction. For elements such as buildings and terrain, their point cloud models are not frequently changed. Therefore, the present invention only needs to collect images of the target without re-collecting point cloud data and point cloud modeling, and can color the constructed or unconstructed point cloud models, that is, color the buildings, thereby realizing the reuse of point cloud data and reducing the acquisition cost.

[0125] In one embodiment, the building extraction module 202 is further configured to:

[0126] Inputting the city building image and the interaction information into the feature extraction module, obtaining the previous phase image block feature map and the interaction feature output by the feature extraction module;

[0127] Inputting the previous phase image block feature map and the interactive feature into the feature decoding module, and obtaining the building roof extraction result and the building offset extraction result output by the feature decoding module;

[0128] The building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module.

[0129] In one embodiment, the building extraction module 202 is further configured to:

[0130] Extracting features from the city building image based on an image feature extraction network to obtain a city building image feature map;

[0131] Encoding the interaction information based on an interaction information encoder to obtain interaction features;

[0132] According to the city building image and the interaction information, the city building image feature map and the interaction feature are paired, and the previous phase image block feature map and the interaction feature are output.

[0133] In one embodiment, the building extraction module 202 is further configured to:

[0134] Extracting information from the previous phase image block feature map and the interaction feature based on a bidirectional attention network to obtain an offset token and a semantic mask token;

[0135] Inputting the offset token into a first fully connected network to obtain a building offset extraction result output by the first fully connected network;

[0136] The semantic mask token is input into a second fully connected network to obtain a building roof extraction result output by the second fully connected network.

[0137] In one embodiment, the building extraction module 202 is further configured to:

[0138] According to the building offset extraction result, an interpretation head is selected to perform regression reasoning to obtain a roof offset;

[0139] Performing offset correction on the building roof extraction result according to the roof offset to obtain the building location extraction result;

[0140] The specific steps of selecting an interpretation head for regression reasoning according to the building offset extraction result include:

[0141] The interpretation head for regression reasoning is selected according to the building offset extraction result, and its expression is:

[0142]

[0143] Among them, ω represents the selector, which is a vector composed of 0 and 1, indicating whether to let Participate in the regression of i offset; is the prediction of the offset of i by the Aug head, is the prediction of the offset of i by the Ref header, O i is the actual prediction result for the i offset.

[0144] In one embodiment, the building coloring module 204 is further used to:

[0145] Determine a first conversion matrix between the urban building image and the segmented image based on the first feature point and the second feature point;

[0146] The point cloud model is colored based on the segmented image, the first transformation matrix and the second transformation matrix; the second transformation matrix is ​​the transformation matrix between the urban building image and the point cloud data; the point cloud model is determined based on the point cloud data; each point in the point cloud model corresponds to each point in the point cloud data.

[0147] In one embodiment, the building coloring module 204 is further used to:

[0148] Determining a conversion relationship between the segmented image and the point cloud data based on the first conversion matrix and the second conversion matrix;

[0149] Based on the conversion relationship, the RGB value of each point on the segmented image is assigned to the corresponding point of the point cloud model.

[0150] See also Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0151] Acquire city building images and interactive information of the city building images;

[0152] The urban building image and the interactive information are input into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; the building extraction result includes a building roof extraction result, a building offset extraction result and a building location extraction result;

[0153] Performing image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly;

[0154] The city building image is colored based on the first feature point and the second feature point.

[0155] See also Figure 4 , Figure 4 Schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400, on which a computer program 411 is stored. When the computer program 411 is executed by a processor, the following steps are implemented:

[0156] Acquire city building images and interactive information of the city building images;

[0157] The urban building image and the interactive information are input into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; the building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; the building extraction result includes a building roof extraction result, a building offset extraction result and a building location extraction result;

[0158] Performing image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly;

[0159] The city building image is colored based on the first feature point and the second feature point.

[0160] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0166] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A real-time acquisition and processing method for surveying and mapping operation data, characterized in that: include Acquire urban building images and interactive information of the urban building images; the interactive information of the urban building images includes the type and purpose of the building, the state of the building, the scale and height of the building, the spatial relationship between the buildings, the design style of the building and the functional services of the building; Inputting the urban building image and the interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; The building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; The building extraction results include building roof extraction results, building offset extraction results and building location extraction results; Performing image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly; Coloring the urban building image based on the first feature point and the second feature point; The step of inputting the urban building image and the interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model includes: Inputting the city building image and the interaction information into the feature extraction module, obtaining the previous phase image block feature map and the interaction feature output by the feature extraction module; Inputting the previous phase image block feature map and the interactive feature into the feature decoding module, and obtaining the building roof extraction result and the building offset extraction result output by the feature decoding module; The building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module.

2. The real-time acquisition and processing method for surveying and mapping operation data according to claim 1 is characterized in that: The step of inputting the city building image and the interaction information into the feature extraction module to obtain the previous phase image block feature map and the interaction feature output by the feature extraction module includes: Extracting features from the city building image based on an image feature extraction network to obtain a city building image feature map; Encoding the interaction information based on an interaction information encoder to obtain interaction features; According to the city building image and the interaction information, the city building image feature map and the interaction feature are paired, and the previous phase image block feature map and the interaction feature are output.

3. The real-time acquisition and processing method for surveying and mapping operation data according to claim 2 is characterized in that: The step of inputting the previous phase image block feature map and the interactive feature into the feature decoding module to obtain the building roof extraction result and the building offset extraction result output by the feature decoding module includes: Extracting information from the previous phase image block feature map and the interaction feature based on a bidirectional attention network to obtain an offset token and a semantic mask token; Inputting the offset token into a first fully connected network to obtain a building offset extraction result output by the first fully connected network; The semantic mask token is input into a second fully connected network to obtain a building roof extraction result output by the second fully connected network.

4. The real-time acquisition and processing method for surveying and mapping operation data according to claim 3 is characterized in that: The step of inputting the building roof extraction result and the building offset extraction result into the location reasoning module to obtain the building location extraction result output by the location reasoning module includes: According to the building offset extraction result, an interpretation head is selected to perform regression reasoning to obtain a roof offset; Performing offset correction on the building roof extraction result according to the roof offset to obtain the building location extraction result; The specific steps of selecting an interpretation head for regression reasoning according to the building offset extraction result include: The interpretation head for regression reasoning is selected according to the building offset extraction result, and its expression is: Among them, ω represents the selector, which is a vector composed of 0 and 1, indicating whether to let Participate in the regression of i offset; is the prediction of the offset of i by the Aug head, is the prediction of the offset of i by the Ref header, is the actual prediction result for the i offset.

5. The real-time acquisition and processing method for surveying and mapping operation data according to claim 1 is characterized in that: The coloring of the urban building image based on the first feature point and the second feature point includes: Determine a first conversion matrix between the urban building image and the segmented image based on the first feature point and the second feature point; The point cloud model is colored based on the segmented image, the first transformation matrix and the second transformation matrix; the second transformation matrix is ​​the transformation matrix between the urban building image and the point cloud data; the point cloud model is determined based on the point cloud data; each point in the point cloud model corresponds to each point in the point cloud data.

6. The real-time acquisition and processing method for surveying and mapping operation data according to claim 5 is characterized in that: The second transformation matrix is ​​determined by calibrating the urban building image and the point cloud data; The coloring of the point cloud model based on the segmented image, the first transformation matrix and the second transformation matrix includes: Determining a conversion relationship between the segmented image and the point cloud data based on the first conversion matrix and the second conversion matrix; Based on the conversion relationship, the RGB value of each point on the segmented image is assigned to the corresponding point of the point cloud model.

7. A real-time acquisition and processing system for surveying and mapping operation data, characterized in that: include: An acquisition module, used to acquire city building images and interaction information of the city building images; The interactive information of the urban building images includes the type and purpose of the building, the state of the building, the scale and height of the building, the spatial relationship between the buildings, the design style of the building and the functional services of the building; A building extraction module, used for inputting the urban building image and the interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model; The building extraction neural network model includes a feature extraction module, a feature decoding module and a location reasoning module; The building extraction results include building roof extraction results, building offset extraction results and building location extraction results; An image matching module is used to perform image matching on the urban building image and its corresponding segmented image to obtain matching feature points; the matching feature points include a first feature point and a second feature point; the first feature point is determined based on the urban building image; the second feature point is determined based on the segmented image; the first feature point and the second feature point are matched correspondingly; A building coloring module, used for coloring the urban building image based on the first feature point and the second feature point; The step of inputting the urban building image and the interactive information into a building extraction neural network model to obtain a building extraction result output by the building extraction neural network model includes: Inputting the city building image and the interaction information into the feature extraction module, obtaining the previous phase image block feature map and the interaction feature output by the feature extraction module; Inputting the previous phase image block feature map and the interactive feature into the feature decoding module, and obtaining the building roof extraction result and the building offset extraction result output by the feature decoding module; The building roof extraction result and the building offset extraction result are input into the location reasoning module to obtain the building location extraction result output by the location reasoning module.

8. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the real-time acquisition and processing method for surveying and mapping operation data as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the real-time acquisition and processing method for surveying and mapping operation data according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Remote sensing image urban area extracting method based on SIFT and multi-sub-graph matching

    CN104778463A

  • Method and device for generating three-dimensional live-action point cloud model

    CN110570466A

  • Three-dimensional point cloud semantic segmentation method and system based on multi-head two-stage attention

    CN114792372A

  • Convolutional neural network and Transform fused remote sensing image building extraction method

    CN116071650A