A Remote Sensing Image Segmentation Method, System, and Storage Medium Based on Point Cloud Guidance

Abstract

The present application relates to a remote sensing image segmentation method, system and storage medium based on point cloud guidance, belonging to the field of image segmentation, and includes the following steps: obtaining original point cloud data and remote sensing images; converting the original point cloud data into image coordinate data; screening out multiple image coordinates from the image coordinate data as point cloud prior coordinates; determining corresponding prior weights according to each point cloud prior coordinate, adding the prior weights to the point cloud prior coordinates to generate guidance coordinates, combining the guidance coordinates into guidance vectors, and generating guidance information according to the guidance vectors; obtaining feature information according to the remote sensing images; fusing the guidance information and the feature information to generate guidance feature information; and obtaining image segmentation information according to the guidance feature information. By generating guidance information based on the original point cloud data and fusing the guidance information with the feature information in traditional image segmentation, the point cloud data can guide image segmentation, thereby improving the accuracy of image segmentation.

Description

Technical Field

[0001] The present application relates to the field of image segmentation, and in particular, to a remote sensing image segmentation method, system and storage medium based on point cloud guidance. Background Art

[0002] As a new energy source, solar energy has the advantages of inexhaustible resources. To expand the scale of solar energy utilization, in addition to solar energy collection equipment, sufficient space is also required to install these devices. Currently, remote sensing image segmentation technology is mainly used to find areas such as rooftops, greenhouses, and waters that can be used to install solar energy collection equipment.

[0003] Currently, although CNN-based methods have achieved excellent results in the field of remote sensing image segmentation, existing segmentation methods usually rely on image quality, and remote sensing images are easily affected by weather. Once there is rainy or foggy weather, the remote sensing images are likely to be blurred, which results in the fact that CNN-based methods cannot meet the strict requirements for segmentation accuracy in some remote sensing applications. Summary of the Invention

[0004] In order to improve the accuracy of remote sensing image segmentation, the present application provides a remote sensing image segmentation method, system and storage medium based on point cloud guidance.

[0005] In a first aspect, the present application provides a remote sensing image segmentation method based on point cloud guidance, adopting the following technical solutions:

[0006] A remote sensing image segmentation method based on point cloud guidance includes the following steps:

[0007] Obtain original point cloud data and remote sensing images;

[0008] Convert the original point cloud data into image coordinate data, where the image coordinate data includes a plurality of image coordinates, and the image coordinates are coordinates in a longitude and latitude coordinate system;

[0009] Select a plurality of image coordinates from the image coordinate data as point cloud prior coordinates, and the point cloud prior coordinates are characterized as image coordinates that can guide image segmentation;

[0010] Determine corresponding prior weights according to each point cloud prior coordinate, and add the prior weights to the corresponding point cloud prior coordinates to generate guidance coordinates,

[0011] Combine the guidance coordinates into a guidance vector, and input the guidance vector into a transformer-based encoder to generate guidance information;

[0012] Input the remote sensing image into the Encoder module of the preset Swin-Unet network to obtain the output information of the Encoder module and use this output information as feature information;

[0013] Fuse the guidance information and the feature information to generate guidance feature information;

[0014] Input the guidance feature information into the Bottleneck module of the Swin-Unet network to enable the Decoder module of the Swin-Unet network to output image segmentation information.

[0015] By adopting the above technical solution, the guidance information is generated according to the original point cloud data, and the guidance information is fused with the feature information in traditional image segmentation, so that the point cloud data can guide image segmentation, thereby improving the accuracy of image segmentation.

[0016] Preferably, the conversion of the original point cloud data into image coordinate data includes the following steps:

[0017] Convert the original point cloud data into homogeneous point cloud data in homogeneous coordinates;

[0018] Calculate the image coordinate data in homogeneous coordinates according to the homogeneous point cloud data according to a preset formula;

[0019] The preset formula is:

[0020]

[0021] Where X is the point cloud coordinate (x0, y0, z0, r0, 1) of the homogeneous point cloud data, x0, y0, z0 are three-dimensional coordinates, r0 is the reflection coefficient, R represents the rotation matrix, T represents the translation matrix, is the internal parameter matrix of the scanning device used to generate the original point cloud data, is the identity matrix, and Y is the image coordinate (u0, v0, t0) in the image coordinate data.

[0022] Preferably, the screening of multiple point cloud prior coordinates from the image coordinate data includes the following steps:

[0023] Voxelize the image coordinate data to obtain a three-dimensional grid (h1*w1*z1), where z1 represents the height of the three-dimensional grid; segment the three-dimensional grid along the h1-w1 plane to obtain multiple sub-grids;

[0024] In each sub-grid, sequentially screen out a preset number of image coordinates from top to bottom in the z1 direction to form a point cloud set, and the image coordinates in the point cloud set are the point cloud prior coordinates.

[0025] Preferably, determining the corresponding prior weight according to each point cloud prior coordinate includes the following steps:

[0026] Determine the corresponding height value according to the point cloud prior coordinate, where the height value is used to characterize the relative position of the corresponding point cloud prior coordinate in the z1 direction in the same point cloud set. The higher the relative position of the point cloud prior coordinate, the larger the corresponding height value;

[0027] Judge whether the height value is a preset value,

[0028] If so, use the preset first weight as the corresponding prior weight;

[0029] If not, use the preset second weight as the corresponding prior weight, where the first weight is greater than the second weight.

[0030] Preferably, determining the corresponding height value according to the point cloud prior coordinate includes the following steps:

[0031] Obtain the maximum value and the minimum value in the z1 direction in the same point cloud set;

[0032] Calculate the difference between the maximum value and the minimum value;

[0033] Divide the coordinate value corresponding to the point cloud prior coordinate in the z1 direction by the difference, and the obtained quotient is the corresponding height value.

[0034] Preferably, combining the guidance coordinates into a guidance vector includes the following steps:

[0035] Stack the image coordinates in the same point cloud set to obtain a point cloud vector with a preset length;

[0036] Determine the position vector according to the sub-grid corresponding to the point cloud set, where the position vector is used to characterize the position of the sub-grid in the three-dimensional grid;

[0037] Calculate the guidance vector according to the point cloud vector and the position vector.

[0038] Preferably, fusing the guidance information and the feature information to generate the guidance feature information includes the following steps: performing multiple downsamplings on the guidance information to generate multiple guidance sub-informations, and arranging the guidance information and the multiple guidance sub-informations in order to form a first set;

[0039] Perform multiple downsamplings on the feature information to generate multiple feature sub-informations, and arrange the feature information and the multiple feature sub-informations in order to form a second set;

[0040] Interleavingly extract the first set and the second set in sequence and arrange them in a preset first direction to form a guidance vector;

[0041] Expand the guidance vector to generate guidance feature information.

[0042] Preferably, the expanding the guidance vector to generate guidance feature information includes the following steps:

[0043] Extract the data of the odd positions of the guidance vector in the first direction to form a first-dimensional vector;

[0044] Extract the data of the even positions of the guidance vector in the first direction to form a second-dimensional vector;

[0045] Arrange the first-dimensional vector and the second-dimensional vector in sequence along the second direction different from the first direction to form guidance feature information.

[0046] In a second aspect, the present application provides a remote sensing image segmentation system based on point cloud guidance, adopting the following technical solution:

[0047] A remote sensing image segmentation system based on point cloud guidance, characterized by comprising a guidance module, an image segmentation module and a fusion module, wherein the image segmentation module includes an encoding unit and a bottleneck-decoding unit;

[0048] The guidance module is used to obtain the original point cloud data and generate guidance information according to the original point cloud data;

[0049] The encoding unit is used to obtain a remote sensing image and encode the remote sensing image to generate corresponding feature information;

[0050] The fusion module is used to fuse the guidance information and the feature information to generate guidance feature information, and input the guidance feature information into the bottleneck-decoding unit;

[0051] The bottleneck-decoding unit is used to obtain image segmentation information according to the guidance feature information.

[0052] Preferably, the guidance module includes a coordinate intercepting unit, a prediction unit and an encoding generation unit,

[0053] The coordinate intercepting unit is used to obtain the original point cloud data and convert the original point cloud data into image coordinate data;

[0054] The prediction unit is used to screen out multiple image coordinates from the image coordinate data as point cloud prior coordinates, determine corresponding prior weights according to each point cloud prior coordinate, add the prior weights to the corresponding point cloud prior coordinates to generate guidance coordinates, and combine the guidance coordinates into a guidance vector;

[0055] The encoding generation unit is used to generate guidance information according to the guidance vector.

[0056] In a third aspect, a computer storage medium stores a program for implementing a remote sensing image segmentation method guided by point cloud. The program for implementing the interface call method is executed by a processor to implement the steps of the above-mentioned remote sensing image segmentation method guided by point cloud.

[0057] In summary, the present application includes the following beneficial technical effects: generating guidance information based on the original point cloud data, and fusing the guidance information with the feature information in traditional image segmentation, so that the point cloud data can guide the image segmentation, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the overall step block diagram of an embodiment of the present application;

[0059] Figure 2 is the model schematic diagram of the encoder based on transformer in the present application;

[0060] Figure 3 is the model schematic diagram of the conventional Swin-Unet network;

[0061] Figure 4 is the step block diagram of converting the original point cloud data into image coordinate data in an embodiment of the present application;

[0062] Figure 5 is the step block diagram of screening out multiple point cloud prior coordinates from the image coordinate data in an embodiment of the present application;

[0063] Figure 6 is the step block diagram of determining the corresponding prior weight according to each point cloud prior coordinate in an embodiment of the present application;

[0064] Figure 7 is the step block diagram of determining the corresponding height value according to the point cloud prior coordinate in an embodiment of the present application;

[0065] Figure 8 is the step block diagram of combining the guidance coordinates into a guidance vector in an embodiment of the present application;

[0066] Figure 9 is the step block diagram of fusing the guidance information and the feature information to generate guidance feature information in an embodiment of the present application;

[0067] Figure 10 is the step block diagram of expanding the guidance vector to generate guidance feature information in an embodiment of the present application;

[0068] Figure 11 is the structural schematic diagram of a remote sensing image segmentation system guided by point cloud in an embodiment of the present application;

[0069] Figure 12 It is a schematic structural diagram of the guiding module according to an embodiment of the present application;

[0070] Figure 13 It is a training flowchart of the system according to an embodiment of the present application;

[0071] Figure 14 It is a verification flowchart of the system according to an embodiment of the present application.

[0072] Explanation of reference numerals: 1, guiding module; 11, coordinate intercepting unit; 12, prediction unit; 13, encoding generation unit; 2, image segmentation module; 3, fusion module. Detailed implementation manners

[0073] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for illustration" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary", "for example" or "for illustration" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of words such as "exemplary", "for example" or "for illustration" is intended to present relevant concepts in a specific manner.

[0074] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the technical features indicated. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0075] The following will Figure 1 in conjunction with the attached Figure 14 make a further detailed description of the present application.

[0076] The embodiments of the present application disclose a remote sensing image segmentation method based on point cloud guidance.

[0077] As Figure 1 shown, the remote sensing image segmentation method based on point cloud guidance includes the following steps:

[0078] S101. Obtain the original point cloud data and the remote sensing image.

[0079] Among them, the original point cloud data is obtained by scanning the area to be measured in a way that a 3D scanning device is carried by vehicles such as satellites, airplanes, and drones. The remote sensing image is obtained after processing the remote sensing data of the area to be measured obtained by a remote sensing satellite.

[0080] It should be noted that the area to be measured corresponding to the original point cloud data and the area to be measured corresponding to the remote sensing image are the same area. It is precisely because both the original point cloud data and the remote sensing image are associated with the same area to be measured that the point cloud data can play a guiding role in the segmentation of the remote sensing image.

[0081] S102. Convert the original point cloud data into image coordinate data.

[0082] Among them, the original point cloud data includes multiple original point cloud coordinates, and the original point cloud coordinates are coordinates in a three-dimensional coordinate system constructed with a 3D scanning device as the origin. The image coordinate data includes multiple image coordinates, and the image coordinates are coordinates in a longitude and latitude coordinate system.

[0083] Since the coordinate system used for the original point cloud data is not unified with the coordinate system used for the remote sensing image, and non-unified coordinates will cause inconvenience in data processing in subsequent steps, it is necessary to first change the coordinate system of the original point cloud data to facilitate subsequent data processing.

[0084] S103. Select multiple image coordinates from the image coordinate data as point cloud prior coordinates.

[0085] Among them, the point cloud prior coordinates are characterized as image coordinates that can guide image segmentation.

[0086] The data volume of the original point cloud data is often extremely large. If the converted image coordinate data is directly input into subsequent operations, it is very likely that the amount is too large and the machine will stop. Therefore, it is necessary to screen the image coordinate data to obtain a small amount of image coordinates with prior knowledge, and use these image coordinates as point cloud prior coordinates, which can reduce the subsequent calculation amount and ensure the improvement of the accuracy of image segmentation.

[0087] S104. Determine the corresponding prior weight according to each point cloud prior coordinate, and add the prior weight to the corresponding point cloud prior coordinate to generate guiding coordinates.

[0088] The prior weight is used to represent the importance degree of the corresponding point cloud prior coordinate for guiding image segmentation. The larger the prior weight, the more important the corresponding point cloud prior coordinate is for guiding image segmentation.

[0089] The remote sensing image takes a top-down view to photograph the buildings, roads, and vegetation in the area to be measured. Then, in the remote sensing image, most of the non-transparent sub-high buildings / vegetation / roads will be blocked by the high-rise buildings / vegetation / roads. Therefore, the focus of image segmentation is the top-level buildings / vegetation. Correspondingly, the heights represented by each point cloud prior coordinate are often different, and the prior weight is actually the manifestation of the corresponding point cloud prior coordinate in terms of height. Theoretically, the higher the height represented by the point cloud prior coordinate, the larger the prior weight.

[0090] Further, a prior weight is added to the corresponding prior coordinates of the point cloud to generate guiding coordinates. Specifically, the prior coordinates of the point cloud are extended from three-dimensional coordinates to four-dimensional coordinates, and the prior weight is used as the attribute of the fourth dimension.

[0091] For example, if the prior coordinates of the point cloud are (x, y, z) and the corresponding prior weight is g, the converted guiding coordinates are (x, y, z, g).

[0092] S105. Combine the guiding coordinates into a guiding vector and input the guiding vector into a transformer-based encoder to generate guiding information.

[0093] The guiding vector meets the input requirements of the transformer-based encoder.

[0094] Since CNN depends on manually defined receptive fields, the region of interest of each convolutional kernel is fixed and there is no correlation between them; while self-attention considers the degree of correlation of different regions of the input image and automatically learns the shape and size of the receptive field, so a transformer-based encoder is used for guiding encoding.

[0095] By inputting the guiding vector, the transformer-based encoder outputs a row of vectors. The output vectors play a role in guiding segmentation, and this vector is the guiding information. Its model is as Figure 2 shown, where (x 1 , x 2 ,..) represents the guiding vector, and (h 1 , h 2 ,..) represents the guiding information.

[0096] S106. Input the remote sensing image into the Encoder module of the preset Swin-Unet network to obtain the output information of the Encoder module and use this output information as feature information.

[0097] As Figure 3 shown, the Swin-Unet network consists of an Encoder module, a Bottleneck module, a Decoder module, and skip connections. The Encoder module takes an image as input and outputs feature information to the Bottleneck module. The Bottleneck module uses two consecutive Swin Transformer blocks to process the feature information.

[0098] In this step, only the Encoder module of the Swin-Unet network is used to obtain feature information.

[0099] S107: Fusing the guidance information and the feature information to generate guidance feature information.

[0100] S108. Input the guided feature information into the Bottleneck module of the Swin-Unet network to enable the Decoder module of the Swin-Unet network to output image segmentation information.

[0101] The guided feature information is composed of the guidance information and feature information. It introduces information about the height and density of the point cloud information in the guidance information, and then fuses it with the RGB information of the remote sensing image. Compared with the single remote sensing image information, it has more learning objects and provides a basis for more accurate subsequent segmentation.

[0102] Please refer to Figure 4 As an optional embodiment, converting the original point cloud data into image coordinate data includes the following steps:

[0103] S201: Convert the original point cloud data into homogeneous point cloud data in homogeneous coordinates.

[0104] S202 : Calculate image coordinate data under homogeneous coordinates based on the homogeneous point cloud data according to a preset formula.

[0105] The default formula is:

[0106]

[0107] Among them, X is the point cloud coordinate table of homogeneous point cloud data (x0, y0, z0, r0, 1), x0, y0, z0 are three-dimensional coordinates, r0 is the reflection coefficient, R represents the rotation matrix, T represents the translation matrix, is the intrinsic parameter matrix of the scanning device used to generate the original point cloud data, is the unit matrix, and Y is the image coordinate (u0, v0, t0) in the image coordinate data.

[0108] The original point cloud data in the original point cloud data is three-dimensional coordinates (x0, y0, z0), which are first expanded into homogeneous point cloud data in order to facilitate conversion into image coordinate data in step S202.

[0109] In addition, when acquiring the original point cloud data, it is easy to scan the non-detection area around the area to be tested, resulting in some irrelevant point cloud data in the original point cloud data. After substituting it into the above preset formula for calculation, these irrelevant point cloud data can also be eliminated to reduce the subsequent amount of calculation.

[0110] Please refer to Figure 5 As an optional embodiment, filtering out a plurality of point cloud prior coordinates from the image coordinate data includes the following steps:

[0111] S301. Voxelize the image coordinate data to obtain a three-dimensional grid (h1*w1*z1).

[0112] The three-dimensional grid is formed by stacking multiple planar images in the vertical direction. Here, h1 of the three-dimensional grid is equal to the length of the image, w1 is equal to the width of the image, and z1 represents the height of the three-dimensional grid.

[0113] S302. Divide the three-dimensional grid along the h1-w1 plane to obtain multiple sub-grids.

[0114] Dividing the three-dimensional grid along the h1-w1 plane means not dividing in the z1 direction. That is, the sub-grid is (h2*w2*z1). The division method adopts equal-ratio division in terms of length and width, i.e., mean division.

[0115] S303. In each sub-grid, sequentially select a preset number of image coordinates from top to bottom in the z1 direction to form a point cloud set. The image coordinates in the point cloud set are the point cloud prior coordinates.

[0116] As mentioned above, the remote sensing image is a top view and pays more attention to the objects at higher levels. Therefore, when selecting the point cloud prior coordinates, it is also necessary to preferentially select the image coordinates at higher positions in the height direction (i.e., the z1 direction).

[0117] The preset number is set by the staff. In this embodiment, the preset number is 4.

[0118] In one embodiment, due to the diverse distribution of the point cloud, it is possible that the number of all image coordinates in a certain sub-grid does not reach the preset number, and thus it is impossible to normally select the preset number of point cloud prior coordinates. Then, randomly initialize the generation of image coordinates to enable the current step to continue, but the image coordinates need to be marked as invalid to avoid affecting the accuracy of the result.

[0119] Please refer to Figure 6 , as an alternative embodiment, to determine the corresponding prior weight according to each point cloud prior coordinate, the following steps are included:

[0120] S401. Determine the corresponding height value according to the point cloud prior coordinate.

[0121] Among them, the height value is used to represent the relative position of the corresponding point cloud prior coordinate in the z1 direction in the same point cloud set. The higher the relative position of the point cloud prior coordinate, the larger the corresponding height value.

[0122] It should be noted that the height values corresponding to the prior coordinates of the point clouds between different point cloud sets do not have any comparative significance. This is because different point cloud sets correspond to different sub-grids, and different sub-grids are mapped to different map regions in the remote sensing map. In reality, even if the map regions are adjacent, it is very likely that one map region is a high-rise building and the adjacent map region is flat ground. In each map region, there will be corresponding highest points, second-highest points, third-highest points, etc. The height value only further differentiates the height differences existing in the prior coordinates of different point clouds in the same map region.

[0123] Please refer to Figure 7 , as an alternative embodiment, determining the corresponding height value according to the prior coordinates of the point cloud includes the following steps:

[0124] S501. Obtain the maximum value and the minimum value in the z1 direction in the same point cloud set.

[0125] S502. Calculate the difference between the maximum value and the minimum value.

[0126] S503. Divide the coordinate value corresponding to the prior coordinate of the point cloud in the z1 direction by the difference, and the obtained quotient is the corresponding height value.

[0127] Since the height value is a relative value, it is necessary to consider the maximum value and the minimum value in the z1 direction in the same point cloud set, so as to obtain the maximum relative distance in the z1 direction. Then, by comparing the coordinate values corresponding to different prior coordinates of the point cloud in the z1 direction with the relative distance, the positions of each prior coordinate of the point cloud in the point cloud set are determined.

[0128] Taking the prior coordinate (x3, y3, z3) of the point cloud in the point cloud set as an example, the formula for obtaining its height value h is:

[0129]

[0130] Among them, max(z i ) is the largest coordinate value in the z1 direction in the same point cloud set, and min(z i ) is the smallest coordinate value in the z1 direction in the same point cloud set.

[0131] S402. Determine whether the height value is a preset value. If the height value is a preset value, use the preset first weight as the corresponding prior weight; if the height value is not a preset value, use the preset second weight as the corresponding prior weight.

[0132] Among them, the preset value is 1, which represents the point cloud coordinates at the absolute high position in the sub-grid.

[0133] The first weight is greater than the second weight. In this embodiment, the first weight is 1 / 2 and the second weight is 1 / 6.

[0134] In one embodiment, before adding the prior weight to the corresponding prior coordinates of the point cloud to generate the guiding coordinates, it is necessary to first normalize the image coordinates in the point cloud set.

[0135] The normalization process is as follows:

[0136]

[0137]

[0138]

[0139] Among them, x0, y0, z0 are the three-dimensional coordinates of the image coordinates, and max(x i ), max(y i ), max(z i ) are the maximum coordinate values in the three coordinate directions of the same point cloud set respectively, and min(x i ), min(y i ), min(z i ) are the minimum coordinate values in the three coordinate directions of the same point cloud set.

[0140] It can be seen that the calculation method of z norm is the same as the calculation method of h in the above step S503. In order to reduce the operation, the image coordinates can be normalized first, and then z norm can be directly extracted as the height value h, and then through normalization, weighting, and then form the guiding coordinates mentioned in step S104.

[0141] Please refer to Figure 8 , as an alternative embodiment, combining the guiding coordinates into a guiding vector includes the following steps:

[0142] S601. Stack the image coordinates in the same point cloud set to obtain a point cloud vector with a preset length.

[0143] The preset length is set by the staff. Generally, in order to match the input format requirements of the encoder, the preset length is generally 20.

[0144] S602. Determine the position vector according to the sub-grid corresponding to the point cloud set.

[0145] Among them, the position vector is used to represent the position of the sub-grid in the three-dimensional grid.

[0146] The position vector is obtained by using the sine coding method, specifically using the following formula:

[0147]

[0148] Among them, pos is the row number of the corresponding sub-grid, i is the column number of the corresponding sub-grid, and dim is equal to the total number of columns of the three-dimensional grid.

[0149] S603. Calculate the guidance vector based on the point cloud vector and the position vector.

[0150] The guidance vector is obtained by multiplying the point cloud vector and the position vector. In order to enable the multiplication calculation, taking a preset length of 20 as an example, the point cloud vector with a size of (h1 * w1 * 20) needs to be deformed by dividing the picture into 32x32 grids to form the size.

[0151] Please refer to Figure 9 , as an alternative embodiment, fuse the guidance information and the feature information to generate the guidance feature information, including the following steps:

[0152] S701. Perform multiple downsamplings on the guidance information to generate multiple guidance sub-informations, and arrange the guidance information and the multiple guidance sub-informations in sequence to form a first set.

[0153] S702. Perform multiple downsamplings on the feature information to generate multiple feature sub-informations, and arrange the feature information and the multiple feature sub-informations in sequence to form a second set.

[0154] S703. Interleave and extract the first set and the second set in sequence and arrange them along a preset first direction to form a guidance vector.

[0155] S704. Expand the guidance vector to generate the guidance feature information.

[0156] In order to obtain multi-scale information, the pyramid sampling method is used to process the guidance information and the feature information respectively to generate multiple guidance sub-informations and feature sub-informations. Then, in order to ensure the full fusion of the two types of information, the feature sub-information and the guidance sub-information are cross-fused.

[0157] Taking four downsamplings as an example, the ratios of the four downsamplings are 0.5, 0.25, 0.125, and 0.125 respectively. Then, four guidance sub-informations V2 - V5 are formed from the guidance information V1, and the first set includes V1 - V5; four feature sub-informations T2 - T5 are formed from the feature information T1, and the second set includes T1 - T5.

[0158] After interleaving and extracting, they are sorted in the order of V1, T1, V2, T2, V3, T3, V4, T4, V5, T5 to form a guidance vector.

[0159] Of course, the number of sampling times and the proportion of each sampling are not limited to the above examples.

[0160] Please refer to Figure 10 , as an alternative embodiment, expand the guiding vector to generate guiding feature information, including the following steps:

[0161] S801. Extract the data at the odd positions of the guiding vector in the first direction to form a first-dimensional vector.

[0162] S802. Extract the data at the even positions of the guiding vector in the first direction to form a second-dimensional vector.

[0163] S803. Arrange the first-dimensional vector and the second-dimensional vector in sequence along the second direction to form guiding feature information.

[0164] Among them, the second direction is different from the first direction. The first direction is equivalent to the length of the guiding vector, and the second direction is equivalent to expanding the guiding vector by one dimension.

[0165] Since cross-fusion changes the length of the information, and at the same time to expand the spatial scale of the information, the cross-fused guiding vector is reduced to 1 / 4 in the length direction and increased to 4 times in the depth direction. Through the above steps, it is ensured that the guiding feature information meets the input requirements of the Swin-Unet network in terms of length.

[0166] Steps S801 and S802 can be carried out synchronously, and there is no requirement for the order of execution.

[0167] In addition, in order to ensure the same model input and improve the speed of subsequent odd-even interpolation, when using the pyramid sampling method, the sum of the sampling ratios needs to be an integer multiple of 1. In this way, regardless of whether the guiding information V1 and / or the feature information T1 is odd or even, the data volume of the finally formed guiding vector must be even. Therefore, there is no need to judge the odd-even of the guiding vector and how to process the odd numbers subsequently.

[0168] Then, to form the first set from the guiding information, instead of performing four downsamplings, the guiding information is downsampled twice, using the bilinear interpolation method to generate guiding sub-information of the original sizes 1 / 2 and 1 / 4 respectively. Repeat the guiding sub-information of the last sampling layer and add the original guiding information to generate four-scale information of the original sizes 1, 1 / 2, 1 / 4, and 1 / 4 times. The method for forming the second set is the same.

[0169] The embodiment of the present application also discloses a remote sensing image segmentation system based on point cloud guidance, as Figure 11 shown, including a guiding module 1, an image segmentation module 2, and a fusion module 3.

[0170] Among them, the image segmentation module 2 includes an encoding unit and a bottleneck-decoding unit.

[0171] As Figure 11 shown, the image segmentation module 2 is formed based on the Swin-Unet network, and the image segmentation module 2 does not make any changes to the main structure of the Swin-Unet network. Only to cooperate with the fusion module 3, the output path of the Encoder module and the input path of the Bottleneck module in the Swin-Unet network are adjusted. That is, the output of the Encoder module in the Swin-Unet network is given to the fusion module 3, and the input of the Bottleneck module in the Swin-Unet network is provided by the fusion module 3.

[0172] Correspondingly, the encoding unit is the Encoder module in the Swin-Unet network, and the bottleneck-decoding unit is the combination of the Bottleneck module and the Decoder module in the Swin-Unet network.

[0173] The guiding module 1 is used to obtain the original point cloud data and generate guiding information according to the original point cloud data.

[0174] The encoding unit is used to obtain a remote sensing image and encode the remote sensing image to generate corresponding feature information.

[0175] The fusion module 3 is used to fuse the guiding information and the feature information to generate guiding feature information, and input the guiding feature information into the bottleneck-decoding unit.

[0176] The bottleneck-decoding unit is used to obtain image segmentation information according to the guiding feature information.

[0177] Please refer to Figure 12 , the guiding module 1 includes a coordinate intercepting unit 11, a prediction unit 12, and an encoding generation unit 13.

[0178] The coordinate intercepting unit 11 is used to obtain the original point cloud data and convert the original point cloud data into image coordinate data.

[0179] The prediction unit 12 is used to screen out multiple image coordinates from the image coordinate data as point cloud prior coordinates, determine corresponding prior weights according to each point cloud prior coordinate, add the prior weights to the corresponding point cloud prior coordinates to generate guiding coordinates, and combine the guiding coordinates into a guiding vector.

[0180] The encoding generation unit 13 is used to generate guiding information according to the guiding vector.

[0181] Among them, the encoding generation unit is an encoder based on a transformer.

[0182] It should be noted that the above-mentioned point cloud-guided remote sensing image segmentation system can be regarded as a model that has been trained and verified, and its model parameters have been set accurately.

[0183] That is to say, when the above-mentioned point cloud-guided remote sensing image segmentation system is just set up, it is necessary to first train and verify the system through a large number of remote sensing images and corresponding point cloud maps. Among them, the training process is as follows Figure 13 , and the verification process is as follows Figure 14 .

[0184] The embodiment of the present application also discloses a computer storage medium, on which a program for implementing a point cloud-guided remote sensing image segmentation method is stored. The program for implementing a point cloud-guided remote sensing image segmentation method is executed by a processor to implement the steps of the above-mentioned point cloud-guided remote sensing image segmentation method.

[0185] The above are all the preferred embodiments of the present application. Without limiting the protection scope of the present application accordingly, therefore: Any equivalent changes made according to the structure, shape, and principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A remote sensing image segmentation method based on point cloud guidance, characterized in that, Including the following steps: Obtain the original point cloud data and remote sensing images; Convert the original point cloud data into image coordinate data, where the image coordinate data includes multiple image coordinates, and the image coordinates are coordinates in the longitude and latitude coordinate system; Select multiple image coordinates from the image coordinate data as prior coordinates of the point cloud, and the prior coordinates of the point cloud are characterized as image coordinates that can guide image segmentation; Determine the corresponding prior weight according to each prior coordinate of the point cloud, and add the prior weight to the corresponding prior coordinate of the point cloud to generate a guiding coordinate; Combine the guiding coordinates into a guiding vector, and input the guiding vector into an encoder based on a transformer to generate guiding information; Input the remote sensing image into the Encoder module of a preset Swin-Unet network to obtain the output information of the Encoder module and use this output information as feature information; Fuse the guiding information and the feature information to generate guiding feature information; Input the guiding feature information into the Bottleneck module of the Swin-Unet network to enable the Decoder module of the Swin-Unet network to output image segmentation information; Among them, the step of combining the guiding coordinates into a guiding vector includes the following steps: Stack the image coordinates in the same point cloud set to obtain a point cloud vector with a preset length; Determine a position vector according to the sub-grid corresponding to the point cloud set, and the position vector is used to represent the position of the sub-grid in the three-dimensional grid; Calculate the guiding vector according to the point cloud vector and the position vector.

2. The method for remotely sensed image segmentation based on point cloud guidance according to claim 1, wherein The step of converting the original point cloud data into image coordinate data includes the following steps: Convert the original point cloud data into homogeneous point cloud data in homogeneous coordinates; Calculate the image coordinate data in homogeneous coordinates according to the homogeneous point cloud data according to a preset formula; The preset formula is: Y = P rect00 R rect00 RTX Among them, X is the point cloud coordinate (x0, y0, z0, r0, 1) in the homogeneous point cloud number, x0, y0, and z0 are three-dimensional coordinates, r0 is the reflection coefficient, R represents the rotation matrix, T represents the translation matrix, and P rect00 is the internal parameter matrix of the scanning device used to generate the original point cloud data, and R rect00 is the identity matrix, and Y is the image coordinate (u0, v0, t0) in the image coordinate data.

3. A method for remote sensing image segmentation based on point cloud guidance according to claim 1, characterized in that, The step of selecting multiple prior coordinates of the point cloud from the image coordinate data includes the following steps: Voxelize the image coordinate data to obtain a three-dimensional grid (h1*w1*z1), where z1 represents the height of the three-dimensional grid; Segment the three-dimensional grid along the h1-w1 plane to obtain multiple sub-grids; Sequentially select a preset number of image coordinates from top to bottom in the z1 direction in each sub-grid to form a point cloud set, and the image coordinates in the point cloud set are the prior coordinates of the point cloud.

4. The method for remote sensing image segmentation based on point cloud guidance according to claim 3, characterized in that The step of determining the corresponding prior weight according to each prior coordinate of the point cloud includes the following steps: Determine the corresponding height value according to the prior coordinate of the point cloud, and the height value is used to represent the relative position of the corresponding prior coordinate of the point cloud in the z1 direction in the same point cloud set. The higher the relative position of the prior coordinate of the point cloud, the larger the corresponding height value; Judge whether the height value is a preset value; If so, use the preset first weight as the corresponding prior weight; If not, use the preset second weight as the corresponding prior weight, and the first weight is greater than the second weight.

5. A method for remote sensing image segmentation based on point cloud guidance according to claim 4, characterized in that, The step of determining the corresponding height value according to the prior coordinate of the point cloud includes the following steps: Obtain the maximum value and the minimum value in the z1 direction in the same point cloud set; Calculate the difference between the maximum value and the minimum value; Divide the coordinate value corresponding to the prior coordinate of the point cloud in the z1 direction by the difference, and the obtained quotient is the corresponding height value.

6. The method for remote sensing image segmentation based on point cloud guidance according to claim 1, wherein The fusion of the guidance information and the feature information to generate the guidance feature information includes the following steps: Perform multiple downsamplings on the guidance information to generate multiple guidance sub-information, and arrange the guidance information and the multiple guidance sub-information in order to form a first set; Perform multiple downsamplings on the feature information to generate multiple feature sub-information, and arrange the feature information and the multiple feature sub-information in order to form a second set; Interleavingly extract the first set and the second set in sequence and arrange them along a preset first direction to form a guidance vector; Expand the guidance vector to generate the guidance feature information.

7. A method for remote sensing image segmentation based on point cloud guidance according to claim 6, characterized in that, The expansion of the guidance vector to generate the guidance feature information includes the following steps: Extract the data at odd positions of the guidance vector in the first direction to form a first-dimensional vector; Extract the data at even positions of the guidance vector in the first direction to form a second-dimensional vector; Arrange the first-dimensional vector and the second-dimensional vector in sequence along a second direction to form the guidance feature information, where the second direction is different from the first direction.

8. A remote sensing image segmentation system based on point cloud guidance, characterized in that, Execute a remote sensing image segmentation method based on point cloud guidance according to any one of claims 1 to 7.

9. A computer storage medium, characterized in that: A program for implementing a remote sensing image segmentation method based on point cloud guidance is stored on the storage medium, and the program for implementing a remote sensing image segmentation method based on point cloud guidance is executed by a processor to implement the steps of a remote sensing image segmentation method based on point cloud guidance according to any one of claims 1 to 7.

Images