An adaptive instance segmentation graph patch cutting gap repairing method

Abstract

The application discloses a kind of self-adapting instance segmentation graph spot cutting gap repair methods, comprising the following steps: S1: input remote sensing image data, and read original image;S2: image overlap cutting;By image overlap cutting method, the original image input in step S1 is cut into repeatable prediction image tile;S3: model inference;Import instance segmentation model, the image tile cut in step S2 is input into pre-trained instance segmentation model to carry out instance ground object graph spot prediction, and generate predicted instance ground object graph spot;S4: graph spot screening;S5: coordinate conversion;S6: result drawing, it is solved that the inference result of current instance segmentation method when using, there is image tile, instance graph spot will appear more obvious cutting gap and cutting trace problem when being spliced and restored to original image range size.

Description

Technical Field

[0001] This invention relates to the field of geographic information technology, and in particular to an adaptive instance segmentation patch cutting gap repair method. Background Technology

[0002] In the field of geographic information technology, the analysis and computation of remote sensing images currently typically employ graphics processing units (GPUs) that excel at array operations. However, commercially available GPUs often cannot directly process large-scale remote sensing images. Therefore, when processing large-scale remote sensing images, algorithm designers often need to segment the images into individual image tiles using a specific instance segmentation method before placing them on the GPU for computation.

[0003] Currently, during instance segmentation, the instance segmentation model is limited by the tile boundaries, which causes the instance object patches generated by inference to be truncated by the tile boundaries. When the inference results of all image tiles are stitched together and restored to the original image size, the instance patches often show obvious cutting gaps and cutting marks.

[0004] In the patent CN114357062B, titled "A Method for Detecting Overlapping or Gap in Vector Data Based on Spatial Location Segmentation," the original data is cropped, and the new data is distributed to storage nodes according to spatial location relationships. Although this greatly improves the efficiency of checking for overlap or gap in massive vector data, avoids time-consuming data broadcasting during calculation, and significantly reduces the number of other elements that need to be judged and calculated for each element, it still has the problem that when the inference results of image tiles are stitched together and restored to the original image range, the instance patches will have relatively obvious cutting gaps and cutting marks. Summary of the Invention

[0005] Based on this, and in response to the above problems, this invention proposes an adaptive method for repairing cut gaps in instance segmentation patches. This method solves the problem that when current instance segmentation methods are used, instance patches will have obvious cut gaps and cut marks when the inference results of image tiles are stitched together and restored to the original image size.

[0006] The technical solution of this invention is:

[0007] An adaptive instance segmentation patch cutting gap repair method, characterized by comprising the following steps:

[0008] S1: Input remote sensing image data and read the raw image;

[0009] S2: Image overlap and cutting;

[0010] The original image input in step S1 is cut into repeatable and predictable image tiles using the image overlay cutting method.

[0011] S3: Model Inference;

[0012] Import the instance segmentation model, input the image tiles cut in step S2 into the pre-trained instance segmentation model to predict instance feature patches, and generate predicted instance feature patches;

[0013] S4: Image patch filtering;

[0014] In the instance feature patches generated in step S3, complete feature patch objects and uncut feature patch objects are filtered out, and a JSON predicted patch data file that can store multiple objects is generated;

[0015] S5: Coordinate transformation;

[0016] S6: Result plotting;

[0017] The results were drawn, and the gaps in the cut pattern were repaired.

[0018] Preferably, step S1 specifically includes:

[0019] Input remote sensing image data and read the original image. The remote sensing image data includes the original image and the corresponding geographic information.

[0020] Preferably, when performing step S2, which involves image overlap cropping of the original image, only the original remote sensing image needs to be provided, and the size and overlap of the image tiles to be cropped can be specified to achieve batch automated cropping.

[0021] Preferably, step S3 specifically includes:

[0022] The image tiles cut from S2 are input into a pre-trained instance segmentation model. After the instance segmentation model predicts the image tiles, it outputs instance feature patches. Then, the patch coordinate transformation algorithm restores the patches to the corresponding positions in the original image.

[0023] Preferably, in step S4, the patch selection steps are as follows:

[0024] A: Record the relative coordinates of the top left corner of each cutting window in the original image, as well as the cutting width and height, and record the cutting situation corresponding to each cutting window based on the cutting window coordinates, width, and height information;

[0025] B: Record the relative position information of the top left corner point of each predicted patch, and save it in point-to-point format;

[0026] C: Determine whether the top left corner of each image patch is located within the image tile selection area according to the order of the image patches;

[0027] If so, the feature is considered to be a complete feature within the cut window, and the feature is retained.

[0028] If not, the image patch is considered to be truncated by the image tile and is discarded;

[0029] D: Merge all filtered patches into a single list and convert it into a JSON file that can store multiple objects.

[0030] Preferably, step S5 specifically includes:

[0031] Import the geographic information corresponding to the original image in the remote sensing image data input in step S1, convert the image coordinates of the predicted patches filtered in step S4 into geographic coordinates, and then generate geojson and shapefile predicted patch data files or vector layers with geographic information.

[0032] Preferably, the image overlap and cutting method is as follows:

[0033] Define the image tile to be cut as a square with a fixed width and height, determine the width and height of the image tile, set the overlap parameter to 0.2, and the image overlap cutting method will automatically calculate the cutting step length;

[0034] Starting from the top left point of the original image, the image is overlapped and cut from left to right and from top to bottom according to the set overlap degree.

[0035] When the cutting window reaches the right and lower boundaries of the original image respectively, the right boundary of the original image is taken as the right boundary of the tile, and the lower boundary of the original image is taken as the lower boundary of the tile.

[0036] When the cutting window reaches the lower right corner of the original image, the right and lower boundaries of the original image are used as the right and lower boundaries of the tile.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] This invention, through image overlap cutting and overlapping sampling of the original image, can adaptively repair the cutting gaps of the image patches predicted by the deep learning model, demonstrating strong practicality. The overlap cutting method in this invention can effectively avoid cutting gaps and traces caused by secondary cutting of the predicted image patches. It can repair cutting gaps in the original image and can be applied to images of any size. The image patch cutting gap repair method proposed in this invention can realize a personalized customization scheme with adjustable cutting size, cutting method, and cutting overlap, which has strong compatibility and practical application value. It solves the problem that when current instance segmentation methods are used, when the inference results of image tiles are stitched together and restored to the original image range, instance image patches will have obvious cutting gaps and cutting traces. Attached Figure Description

[0039] Figure 1 This is a flowchart of an adaptive instance segmentation patch cutting gap repair method described in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the overlapping cutting method described in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the model reasoning process described in the embodiments of the present invention;

[0042] Figure 4 This is a schematic diagram of the image patch screening method described in the embodiments of the present invention;

[0043] Figure 5 This is a schematic diagram of the patch before repair in Embodiment 5 of the present invention;

[0044] Figure 6 This is a schematic diagram of the original image superimposed on the patch after repair using the method of the present invention in Embodiment 5 of the present invention;

[0045] Figure 7 This is a schematic diagram of the patch before repair in Embodiment 6 of the present invention;

[0046] Figure 8 This is a schematic diagram of the original image superimposed on the patch after repair using the method of the present invention in Embodiment 6 of the present invention. Detailed Implementation

[0047] Instance segmentation based on deep learning is one of the hottest directions in the field of computer vision. Since deep learning requires a lot of array operations on images, GPUs (Graphics Processing Units), which are good at array operations, are widely used in deep learning. In the field of remote sensing applications, due to the large amount of data, GPUs on the market often cannot directly process large remote sensing images. Therefore, in the field of geographic information technology, when processing such ultra-large images, algorithm designers often need to cut large images into independent image tiles, such as 1024*1024, according to a certain segmentation method, and then place them on the GPU for processing.

[0048] This results in the instance segmentation model's inference being constrained by tile boundaries during instance segmentation, causing the inferred instance object patches to be truncated by the tile boundaries. Therefore, when the inference results of all image tiles are stitched together and restored to the original image size, the instance patches often show obvious cutting gaps and cutting marks.

[0049] The paper "Research on Gap Repair Algorithm Based on Convolutional Neural Network Prediction Results" introduces a method for repairing gaps in image patch cutting by expanding the tile cutting range. This method uses the Overlapsize algorithm to improve the matching degree between the prediction results and the actual results, and then fills the gaps through relevant functions in the PostGIS database, so that the image tiles can be truly stitched together and restored into a complete large image.

[0050] In the aforementioned paper, "Research on Gap Repair Algorithm Based on Prediction Results of Convolutional Neural Network", the solution for repairing cut gaps is to expand the cutting range of the cut image tiles and perform overlapping cuts, put them into the model for inference and prediction, and after obtaining the model's prediction results for the expanded image, the prediction results are cropped to restore the original cutting size.

[0051] This method solves the problem of cut gaps in image tiles to some extent, but the predicted results of the enlarged image are cropped during the final processing. When encountering large feature patches or feature patches that cross image tiles, the feature patches will still be truncated. Therefore, the method in the paper cannot completely solve the problem of image tile splicing gaps.

[0052] Patent CN114357062B mentions a method for checking overlapping or gaps in vector data based on spatial location segmentation. It utilizes a spatial grid to trim the original data, allowing new data to be distributed across storage nodes according to spatial location relationships. During overall spatial inspection, overlapping or gap checks can be performed using spatial vector data within the spatial grid.

[0053] In the aforementioned patent CN114357062B, the original data is truncated, allowing the new data to be distributed across storage nodes according to spatial relationships. This significantly improves the efficiency of overlapping or gap detection in massive vector data, avoids time-consuming data broadcasting during computation, and greatly reduces the number of other elements that each element needs to judge and calculate. Each element only needs to be judged or calculated with other elements within the same grid, instead of being judged or calculated with all other elements at once. However, this method only discloses a gap detection method for vector data and does not solve the problem of gap cutting in vector patterns.

[0054] In addition, a method for processing gaps in vector graphics patches based on topology tools in ArcGIS software has been proposed in the geographic information technology industry. This method is based on the spatial topological relationship of vector graphics patches and uses topology tools in ArcGIS software to perform spatial analysis on the vector graphics patches.

[0055] Topological relations refer to the relationships between spatial data that satisfy the principles of topological geometry, namely the adjacency, association, inclusion, and connectivity relationships between entities represented by nodes, arcs, and polygons.

[0056] This method first uses topology tools to set a filtering rule that there should be no gaps, identifying polygon objects with gaps in the vector map. Then, it uses ArcGIS spatial editing functions to manually modify the polygons with cutting gaps, thus completing the repair of the cutting gaps. This cutting gap repair method based on ArcGIS topology tools can effectively solve the problem of cutting gaps in polygons, and the accuracy is generally high after manual correction. However, this method is often limited by the operator's prior knowledge, and its automation level is low, resulting in low efficiency and time-consuming and labor-intensive processing when dealing with a large number of cutting gaps.

[0057] The method of this invention enables adaptive repair of cut gaps in predicted image tiles. This method can adaptively adjust for different image tile sizes and different land cover types, and solves the problem of cut gaps caused by image cutting during the model inference stage. This reduces the workload of manual correction and greatly ensures the integrity of the image tiles, thus having high application value.

[0058] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0059] Example 1:

[0060] like Figure 1 As shown, this embodiment discloses an adaptive instance segmentation patch cutting gap repair method, including the following steps:

[0061] S1: Input remote sensing image data and read the raw image;

[0062] Input remote sensing image data and read the original image. The remote sensing image data includes the original image and the corresponding geographic information.

[0063] S2: Image overlap and cutting;

[0064] The original image input in step S1 is cut into repeatable and predictable image tiles using the image overlay cutting method.

[0065] S3: Model Inference;

[0066] Import the instance segmentation model, input the image tiles cut in step S2 into the pre-trained instance segmentation model to predict instance feature patches, and generate predicted instance feature patches;

[0067] S4: Image patch filtering;

[0068] In the instance feature patches generated in step S3, complete feature patch objects and uncut feature patch objects are filtered out, and a JSON predicted patch data file that can store multiple objects is generated;

[0069] S5: Coordinate transformation;

[0070] Import the geographic information corresponding to the original image in the remote sensing image data input in step S1, convert the image coordinates of the predicted patches after filtering in step S4 into geographic coordinates, and then generate geojson and shapefile predicted patch data files or vector layers with geographic information.

[0071] S6: Result plotting;

[0072] The results were drawn, and the gaps in the cut pattern were repaired.

[0073] Preferably, when performing step S2, which involves image overlap cropping of the original image, only the original remote sensing image needs to be provided, and the size and overlap of the image tiles to be cropped can be specified to achieve batch automated cropping.

[0074] Preferably, step S3 specifically involves: inputting the image tiles cut in S2 into a pre-trained instance segmentation model; the instance segmentation model predicts the image tiles and outputs instance feature patches; and then using a patch coordinate transformation algorithm to restore the patches to the corresponding positions in the original image.

[0075] Preferably, in step S4, the patch selection steps are as follows:

[0076] A: Record the relative coordinates of the top left corner of each cutting window in the original image, as well as the cutting width and height, and record the cutting situation corresponding to each cutting window based on the cutting window coordinates, width, and height information;

[0077] B: Record the relative position information of the top left corner point of each predicted patch, and save it in point-to-point format;

[0078] C: Determine whether the top left corner of each image patch is located within the image tile selection area according to the order of the image patches;

[0079] If so, the feature is considered to be a complete feature within the cut window, and the feature is retained.

[0080] If not, the image patch is considered to be truncated by the image tile and is discarded;

[0081] D: Merge all filtered patches into a single list and convert it into a JSON file that can store multiple objects.

[0082] Preferably, the image overlap and cutting method is as follows:

[0083] Define the image tile to be cut as a square with a fixed width and height, determine the width and height of the image tile, set the overlap parameter to 0.2, and the image overlap cutting method will automatically calculate the cutting step length;

[0084] Starting from the top left point of the original image, the image is overlapped and cut from left to right and from top to bottom according to the set overlap degree.

[0085] When the cutting window reaches the right and lower boundaries of the original image respectively, the right boundary of the original image is taken as the right boundary of the tile, and the lower boundary of the original image is taken as the lower boundary of the tile.

[0086] When the cutting window reaches the lower right corner of the original image, the right and lower boundaries of the original image are used as the right and lower boundaries of the tile.

[0087] Example 2:

[0088] S2: Image overlap and cutting;

[0089] The original image input in step S1 is cut into repeatable and predictable image tiles using the image overlay cutting method.

[0090] Specific image overlay and cutting methods, such as Figure 2 As shown, we define a normally cut image tile as a square with a fixed width and height, for example: 1024px * 1024px, as... Figure 2 As shown at point ①, by specifying the overlap parameter, which is generally set to 0.2, the image overlap cutting method will automatically calculate the cutting step length.

[0091] Starting from the top left point of the original image, the image is overlapped and cut from left to right and from top to bottom according to the set overlap degree.

[0092] When the cutting window reaches the right and bottom boundaries of the original image respectively, the right boundary of the original image is used as the right boundary of the tile, such as... Figure 2 As shown at point ②, the lower boundary of the original image is used as the lower boundary of the tile, as follows: Figure 2 As shown at point ③ in the middle;

[0093] When the cutting window reaches the lower right corner of the original image, the right and lower boundaries of the original image are used as the right and lower boundaries of the tile, such as... Figure 2 As shown at point ④ in the middle.

[0094] Example 3:

[0095] S3: Model Inference;

[0096] Import the instance segmentation model, input the image tiles cut in step S2 into the pre-trained instance segmentation model to predict instance feature patches, and generate predicted instance feature patches;

[0097] A diagram illustrating the specific model inference process is shown below. Figure 3 As shown.

[0098] Example 4:

[0099] S4: Image patch filtering;

[0100] In the instance feature patches generated in step S3, complete feature patch objects and uncut feature patch objects are filtered out, and a JSON predicted patch data file that can store multiple objects is generated;

[0101] The steps for filtering image patches are as follows:

[0102] A: Record the relative coordinates of the top-left corner of each cutting window in the original image, as well as the cutting width and height. Based on the cutting window coordinates and width / height information, record the cutting pattern corresponding to each cutting window, such as... Figure 2 As shown;

[0103] B: Record the relative position information of the top left corner point of each predicted patch, and save it in point-to-point format;

[0104] C: Determine whether the top left corner of each image patch is located within the image tile selection area according to the order of the image patches;

[0105] If so, the feature is considered to be a complete feature within the cut window, and the feature is retained.

[0106] If not, then the image patch is considered to be truncated by the image tile and discarded. Figure 4 As shown;

[0107] D: Merge all filtered patches into a single list and convert it into a JSON file that can store multiple objects.

[0108] Example 5:

[0109] Figure 5 To repair the previous image patches, Figure 6 The original image is superimposed on the patch after restoration using the method of this invention;

[0110] Example 6:

[0111] Figure 7 To repair the previous image patches, Figure 8 The original image is superimposed on the patch after restoration using the method of this invention;

[0112] Examples 5 and 6 above are comparison images before and after repair using the method of the present invention. It can be clearly seen that... Figure 6 and Figure 8 The example pattern has no cutting gaps or cutting marks.

[0113] The method of this invention mainly includes four parts: image overlay and cutting, model inference, patch selection, and coordinate transformation. Image overlay and cutting primarily achieves repeated patch prediction at the image tile cutting locations; model inference calls a pre-trained instance segmentation model to predict instance feature patches; patch selection filters complete feature patches, discarding those truncated by image tile boundaries; coordinate transformation converts the processed patches into data files or vector layers with geographic information, facilitating data transmission and visualization in actual production work.

[0114] Principle of this invention:

[0115] This invention, through image overlay cutting and overlay sampling of the original image, can adaptively repair the cutting gaps of the patches predicted by the deep learning model, and has strong practicality. The overlay cutting method in this invention can effectively avoid the cutting gaps and traces caused by secondary cutting of the predicted patches. It can repair the cutting gaps of the original image and can be applied to images of any size. The patch cutting gap repair method proposed in this invention can realize a personalized customization scheme with adjustable cutting size, cutting method and cutting overlap, and has strong compatibility and practical application value.

[0116] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An adaptive instance segmentation graph cut gap filling method, characterized in that, Includes the following steps: S1: Input remote sensing image data and read the raw image; S2: Image overlap and cutting; The original image input in step S1 is cut into repeatable and predictable image tiles using the image overlay cutting method. S3: Model Inference; Import the instance segmentation model, input the image tiles cut in step S2 into the pre-trained instance segmentation model to predict instance feature patches, and generate predicted instance feature patches; S4: Image patch filtering; In the instance feature patches generated in step S3, complete feature patch objects and uncut feature patch objects are filtered out, and a JSON predicted patch data file that can store multiple objects is generated; S5: Coordinate transformation; S6: Result plotting; The results were drawn, and the gaps in the cut pattern were repaired.

2. The adaptive instance segmentation patch cutting gap repair method according to claim 1, characterized in that, Step S1 is as follows: Input remote sensing image data and read the original image. The remote sensing image data includes the original image and the corresponding geographic information.

3. The adaptive instance segmentation patch cutting gap repair method according to claim 1 or 2, characterized in that, In step S2, when performing image overlap cropping on the original image, you only need to provide the original remote sensing image and specify the size and overlap of the image tiles to be cropped to achieve batch automated cropping.

4. The adaptive instance segmentation patch cutting gap repair method according to claim 3, characterized in that, Step S3 is as follows: The image tiles cut from S2 are input into a pre-trained instance segmentation model. After the instance segmentation model predicts the image tiles, it outputs instance feature patches. Then, the patch coordinate transformation algorithm restores the patches to the corresponding positions in the original image.

5. The adaptive instance segmentation patch cutting gap repair method according to claim 4, characterized in that, In step S4, the image patch selection steps are as follows: A: Record the relative coordinates of the top left corner of each cutting window in the original image, as well as the cutting width and height, and record the cutting situation corresponding to each cutting window based on the cutting window coordinates, width, and height information; B: Record the relative position information of the top left corner point of each predicted patch, and save it in point-to-point format; C: Determine whether the top left corner of each image patch is located within the image tile selection area according to the order of the image patches; If so, the feature is considered to be a complete feature within the cut window, and the feature is retained. If not, the image patch is considered to be truncated by the image tile and is discarded; D: Merge all filtered patches into a single list and convert it into a JSON file that can store multiple objects.

6. The adaptive instance segmentation patch cutting gap repair method according to claim 3, characterized in that, Step S5 is as follows: Import the geographic information corresponding to the original image in the remote sensing image data input in step S1, convert the image coordinates of the predicted patches filtered in step S4 into geographic coordinates, and then generate geojson and shapefile predicted patch data files or vector layers with geographic information.

7. The adaptive instance segmentation patch cutting gap repair method according to claim 1, characterized in that, The image overlap and cutting method is as follows: Define the image tile to be cut as a square with a fixed width and height, determine the width and height of the image tile, set the overlap parameter to 0.2, and the image overlap cutting method will automatically calculate the cutting step length; Starting from the top left point of the original image, the image is overlapped and cut from left to right and from top to bottom according to the set overlap degree. When the cutting window reaches the right and lower boundaries of the original image respectively, the right boundary of the original image is taken as the right boundary of the tile, and the lower boundary of the original image is taken as the lower boundary of the tile. When the cutting window reaches the lower right corner of the original image, the right and lower boundaries of the original image are used as the right and lower boundaries of the tile.

Images