A remote sensing image labeling method, device and equipment

Abstract

This application provides a method, apparatus, and device for labeling remote sensing images. The method includes: acquiring seed pixels and their corresponding target categories; using the seed pixels as starting pixels, searching for foreground pixels corresponding to the target category in each remote sensing image, and determining closed regions of the remote sensing image based on the seed pixels and foreground pixels; selecting K remote sensing images to be labeled from T remote sensing images based on the similarity between the closed regions of the remote sensing images; determining fusion regions based on the closed regions of the K remote sensing images to be labeled; and performing the following synchronous labeling operation in parallel on the K remote sensing images to be labeled based on the target categories: determining target regions based on the fusion regions and the labeled regions of the remote sensing images to be labeled, and labeling the target regions in the remote sensing images to be labeled with target categories; the labeled regions are regions in the remote sensing images to be labeled that have already been labeled with categories. The technical solution of this application can reduce the workload of labeling remote sensing images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus and device for annotating remote sensing images. Background Technology

[0002] Machine learning is a pathway to achieving artificial intelligence. It is an interdisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. Machine learning studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning places great emphasis on algorithm design, enabling computers to automatically learn patterns from data and use these patterns to predict unknown data.

[0003] Machine learning has been widely used in fields such as deep learning, data mining, computer vision, natural language processing, biometrics, search engines, medical diagnosis, speech and handwriting recognition, etc.

[0004] To implement artificial intelligence processing using machine learning, a large number of sample images are needed. A machine learning model is then trained based on these images, and this model is used to perform artificial intelligence processing. However, training this model requires users to annotate each image, which is a labor-intensive and time-consuming process, resulting in a poor user experience.

[0005] For example, for pixel-level prediction tasks, such as semantic segmentation, users need to label the category of each pixel in the sample image. In particular, in the field of remote sensing, users need to label the category of each pixel in remote sensing images from multiple time periods, which is a huge amount of work. Summary of the Invention

[0006] This application provides a remote sensing image annotation method. The sample data set includes T remote sensing images from different time phases, where T is a positive integer greater than 1. The method includes:

[0007] Obtain the seed pixel and the target category corresponding to the seed pixel;

[0008] Starting with a seed pixel, search for foreground pixels in each remote sensing image that correspond to the target category, and determine the closed region of the remote sensing image based on the seed pixel and the foreground pixels.

[0009] Based on the similarity between closed regions in remote sensing images, K remote sensing images to be labeled are selected from T remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T.

[0010] The fusion region is determined based on the closed regions of the K unannotated remote sensing images;

[0011] Based on the target category, the following synchronous annotation operation is performed in parallel on the K remote sensing images to be annotated: the target region is determined based on the fused region and the already labeled region of the remote sensing image to be annotated, and the target category is labeled for the target region in the remote sensing image to be annotated; wherein, the already labeled region is the region in the remote sensing image to be annotated that has already been labeled with a category.

[0012] This application provides a remote sensing image annotation device. The sample data set includes T remote sensing images from different time phases, where T is a positive integer greater than 1. The device includes:

[0013] The acquisition module is used to acquire seed pixels and the target category corresponding to the seed pixels;

[0014] The processing module is used to search for foreground pixels corresponding to the target category in each remote sensing image, starting from the seed pixel; determine the closed region of the remote sensing image based on the seed pixel and the foreground pixel; and select K remote sensing images to be labeled from T remote sensing images based on the similarity between the closed regions of the remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T.

[0015] The determination module is used to determine the fusion region based on the closed regions of the K remote sensing images to be labeled;

[0016] The annotation module is used to perform the following synchronous annotation operation in parallel on the K remote sensing images to be annotated based on the target category: determine the target region based on the fused region and the already annotated region of the remote sensing image to be annotated, and annotate the target region in the remote sensing image to be annotated with the target category; wherein, the already annotated region is the region in the remote sensing image to be annotated that has already been annotated with a category.

[0017] This application provides a remote sensing image annotation device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the remote sensing image annotation method of the above example.

[0018] As can be seen from the above technical solutions, in this embodiment, the device can automatically label the category of each pixel in remote sensing images from multiple time phases, eliminating the need for users to manually label the category of each pixel in remote sensing images from multiple time phases. This reduces the workload of labeling remote sensing images, shortens labeling time, speeds up labeling efficiency, reduces labor costs, and improves user experience. The device can automatically extract the fused region of the remote sensing image (i.e., label each pixel within the fused region of the remote sensing image), and the device can perform synchronous labeling operations on remote sensing images from multiple time phases in parallel, thereby achieving parallel synchronous labeling of remote sensing images, improving labeling efficiency, and reducing labeling time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0020] Figure 1 This is a flowchart illustrating a remote sensing image annotation method according to one embodiment of this application;

[0021] Figure 2 This is a schematic diagram of remote sensing images at different time phases in one embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating a remote sensing image annotation method according to one embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating the synchronous annotation process in one embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a reference label diagram in one embodiment of this application;

[0025] Figure 6A and Figure 6B This is a schematic diagram of a reference label diagram in one embodiment of this application;

[0026] Figure 7 This is a schematic diagram illustrating the final effect of the transition value in one embodiment of this application;

[0027] Figure 8 This is a flowchart illustrating the edge finishing process in one embodiment of this application;

[0028] Figure 9 This is a schematic diagram illustrating the effect of edge refinement in one embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the structure of a remote sensing image annotation device according to one embodiment of this application;

[0030] Figure 11 This is a hardware structure diagram of a remote sensing image annotation device according to one embodiment of this application. Detailed Implementation

[0031] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0033] This application proposes a remote sensing image annotation method. The sample data set may include multiple sets of remote sensing images, and each set of remote sensing images may include T remote sensing images from different time phases, where T is a positive integer greater than 1. This remote sensing image annotation method is used to automatically annotate multiple sets of remote sensing images. For ease of description, the automatic annotation process of a set of remote sensing images will be used as an example, that is, it is necessary to automatically annotate T remote sensing images from different time phases. The T remote sensing images from different time phases are remote sensing images of the same scene.

[0034] See Figure 1 The diagram shown is a flowchart of the remote sensing image annotation method, which may include:

[0035] Step 101: Obtain the seed pixel and the target category corresponding to the seed pixel.

[0036] Step 102: Starting with the seed pixel, search for the foreground pixel corresponding to the target category in each remote sensing image, and determine the closed region of the remote sensing image based on the seed pixel and the foreground pixel.

[0037] In one possible implementation, for each remote sensing image, a seed pixel can be used as a starting pixel, and it can be determined whether there are any unpredictable pixels among the surrounding pixels of the starting pixel. If so, the similarity between the starting pixel and the unpredictable pixel can be determined; if the similarity is greater than a preset threshold, the unpredictable pixel can be identified as a foreground pixel, and the unpredictable pixel can be used as the starting pixel, and the operation of determining whether there are any unpredictable pixels among the surrounding pixels of the starting pixel can be returned; otherwise, the unpredictable pixel can be identified as an edge pixel. If not, the seed pixel and all foreground pixels can form a closed region of the remote sensing image.

[0038] For example, determining the similarity between a starting pixel and an unpredicted pixel may include: determining a first mean and a first variance of all pixels in a first region, and determining a second mean and a second variance of all pixels in a second region; the first region is centered on the starting pixel, and the second region is centered on the unpredicted pixel, with the first region having the same size as the second region; and determining the similarity between the starting pixel and the unpredicted pixel based on the first mean, the first variance, the second mean, and the second variance. Here, similarity characterizes whether there is significant change in image content between different time phases. For example, if the similarity is high (e.g., greater than a threshold), then there is no significant change in image content between different time phases; conversely, if the similarity is low (e.g., not greater than a threshold), then there is significant change in image content between different time phases.

[0039] Step 103: Based on the similarity between closed regions in the remote sensing images, select K remote sensing images to be labeled from the T remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T.

[0040] For example, if T remote sensing images include one reference remote sensing image and T-1 non-reference remote sensing images, then the reference remote sensing image is selected as the remote sensing image to be labeled. For each non-reference remote sensing image: determine the similarity between the non-reference remote sensing image and the reference remote sensing image; wherein, based on the union region between the closed regions of the non-reference remote sensing image and the closed regions of the reference remote sensing image, the similarity is determined by the ratio of the number of pixels with inconsistent prediction results in the union region to the total number of pixels in the union region, and the similarity is inversely proportional to the ratio. If the similarity is greater than a preset threshold, the non-reference remote sensing image is selected as the remote sensing image to be labeled; otherwise, the selection of the non-reference remote sensing image as the remote sensing image to be labeled is prohibited.

[0041] Step 104: Determine the fusion region based on the closed regions of the K unannotated remote sensing images.

[0042] For example, for each pixel in a closed region of a remote sensing image to be labeled, a transition value for the pixel is determined based on the minimum Euclidean distance between the pixel and the unclosed region and a configured value; a union region is determined among the closed regions of K remote sensing images to be labeled; for each pixel in the union region, it is determined whether the pixel is a target pixel based on the transition value corresponding to the pixel in the K remote sensing images to be labeled; and a fusion region is determined based on all target pixels in the union region.

[0043] Step 105: Perform the following synchronous annotation operation on the K remote sensing images to be labeled in parallel based on the target category: Determine the target region based on the fused region and the labeled region of the remote sensing image to be labeled, and label the target region in the remote sensing image to be labeled with the target category; wherein, the labeled region is the region in the remote sensing image to be labeled that has already been labeled with a category. Parallel synchronous annotation operation on the K remote sensing images to be labeled means that the annotation operation on the K remote sensing images to be labeled can be performed in parallel. For example, while performing the annotation operation on the first remote sensing image to be labeled based on the target category (i.e., labeling the target region in the remote sensing image to be labeled with the target category), the annotation operation on the second remote sensing image to be labeled will also be performed based on the target category, and the annotation operation on the third remote sensing image to be labeled will also be performed based on the target category, and so on.

[0044] For example, a non-overlapping region can be determined between the fused region and the labeled region. For each edge pixel in the non-overlapping region, the nearest matching pixel in the labeled region is determined, and the distance between the edge pixel and the matching pixel is determined. If the distance is less than a preset threshold, all pixels on the line connecting the edge pixel and the matching pixel are added to the non-overlapping region. Based on this, the target region of the remote sensing image to be labeled can be determined based on the non-overlapping region. For example, the non-overlapping region can be determined as the target region of the remote sensing image to be labeled.

[0045] For example, determining the non-overlapping area between the merged area and the labeled area may include, but is not limited to: if the merged area and the labeled area overlap, the overlapping area can be removed from the merged area to obtain the non-overlapping area; or, if the merged area and the labeled area do not overlap, the merged area can be determined as the non-overlapping area.

[0046] As can be seen from the above technical solutions, in this embodiment, the device can automatically label the category of each pixel in remote sensing images from multiple time phases, eliminating the need for users to manually label the category of each pixel in remote sensing images from multiple time phases. This reduces the workload of labeling remote sensing images, shortens labeling time, speeds up labeling efficiency, reduces labor costs, and improves user experience. The device can automatically extract the fused region of the remote sensing image (i.e., label each pixel within the fused region of the remote sensing image), and the device can perform synchronous labeling operations on remote sensing images from multiple time phases in parallel, thereby achieving parallel synchronous labeling of remote sensing images, improving labeling efficiency, and reducing labeling time.

[0047] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0048] In machine learning tasks (such as deep learning), each sample image needs to be labeled. For example, for pixel-level prediction tasks, such as semantic segmentation, each pixel in the sample image needs to be labeled with a category. Different categories of pixels need to be labeled with different tags to represent different categories, until all pixels are labeled. In the field of remote sensing, there are remote sensing images of the same scene at multiple time points. This means that each pixel in remote sensing images at multiple time points needs to be labeled with a category. The workload of labeling these remote sensing images is huge, requiring a large number of annotations.

[0049] See Figure 2 The image shown is a schematic diagram of remote sensing images from different time phases (only two phases are shown; different time phases can be understood as different times). The left side shows a remote sensing image from one time phase, and the right side shows a remote sensing image from another time phase. For example, the remote sensing field has the following characteristics: complex environment, multiple land cover categories (such as farmland, roads, water bodies, buildings, greenhouses, etc.), and... Figure 2 (Not fully shown in the image), color differences are inconsistent, with areas of variation or no variation. Figure 2 The most obvious example is the farmland area near the highway.

[0050] Due to the aforementioned characteristics, the land cover categories at the same location may change in remote sensing images of the same scene at different time phases. Therefore, even at the same location, the category of a remote sensing image at one time phase cannot be directly transferred to a remote sensing image at another time phase. In other words, it is necessary to label the category of each pixel in the remote sensing image at each time phase, which results in a large amount of labeling work.

[0051] In response to the above findings, this application proposes a remote sensing image annotation method that can automatically annotate multiple remote sensing images from different time periods, thereby accelerating annotation efficiency and reducing manual costs.

[0052] For example, the sample dataset may include multiple sets of remote sensing images. A single set of remote sensing images may include multiple remote sensing images, and these multiple images are for the same scene (e.g., the same latitude and longitude). Different sets of remote sensing images may be for different scenes (e.g., different latitudes and longitudes). Since the annotation method for each set of remote sensing images is the same, the automatic annotation process of a set of remote sensing images will be used as an example in subsequent embodiments. For a set of remote sensing images, the set may include T registered remote sensing images from different time phases, i.e., T remote sensing images at different times, where T can be a positive integer greater than 1, and the T remote sensing images from different time phases are for the same scene.

[0053] See Figure 3 The diagram shows a flowchart of a remote sensing image annotation method. The input data consists of T registered remote sensing images I1, I2, I3, ..., I... T These T remote sensing images correspond to T time phases, and the categories to be labeled are 1, 2, ..., C-1, C, meaning there are a total of C categories, such as farmland, roads, water bodies, buildings, greenhouses, etc. The output data (i.e., the labeling results) is the category of each pixel in the T remote sensing images.

[0054] For example, a remote sensing image annotation method can involve two processes: a synchronous annotation process and an edge refinement process. In the synchronous annotation process, a seed pixel (x, y) and its corresponding category c can be specified. Starting with the seed pixel (x, y), surrounding pixels are automatically predicted. If the predicted category is the same as the category c of the seed pixel (x, y), the surrounding pixels are labeled. After the automatic labeling of surrounding pixels is completed, the prediction results of T remote sensing images can be compared and aligned. In the edge refinement process, automated refinement can be performed based on the results of the synchronous annotation. For example, it can automatically analyze whether there is overlap or proximity with other labeled areas. If so, the two are merged or matched according to their categories to complete the refinement.

[0055] For example, in the first segmentation round, the user can specify a seed pixel (x, y) and its corresponding category c, and perform a synchronous annotation process and an edge refinement process to complete the annotation of some pixels in the remote sensing image. Then, it is determined whether all pixels in all remote sensing images have been annotated. If not, the second segmentation round is executed. In the second segmentation round, the user can specify a seed pixel (x', y') and its corresponding category c'. The seed pixel (x', y') represents the unannotated pixel. A synchronous annotation process and an edge refinement process are performed to complete the annotation of some pixels in the remote sensing image. Then, it is determined whether all pixels in all remote sensing images have been annotated. If not, the third segmentation round is executed.

[0056] This process of simultaneous annotation and edge refinement continues until all pixels in all remote sensing images are annotated, and the category of each pixel in T remote sensing images is output.

[0057] In summary, the annotation process involves continuously specifying seed pixels and categories for annotation until all pixels in all remote sensing images are annotated. Each time a seed pixel is specified, unannotated pixels are selected from all pixels in the remote sensing images and designated as the seed pixel. Each specification of the seed pixel and category triggers a new segmentation round. In this new round, simultaneous annotation and edge refinement processes are performed to annotate a subset of pixels in the remote sensing images, and this process continues indefinitely.

[0058] Based on the above process, the final effect is as follows: each time a seed pixel and category are specified, the labels of the surrounding pixels of the seed pixel are assigned. If there are no obvious changes between remote sensing images of different time phases, the labels are aligned. If there are obvious changes between remote sensing images of different time phases, no labels are made. When it overlaps or is close to an area that has already been labeled (i.e., the labeled area), it will be merged or aligned with it to complete the refinement.

[0059] The following describes the synchronous annotation process and the edge refinement process with reference to specific embodiments.

[0060] See Figure 4 The diagram shown illustrates the flowchart of the synchronous annotation process, which may include:

[0061] Step 401: Obtain the seed pixel and the target category corresponding to the seed pixel.

[0062] For example, T remote sensing images can be divided into one reference remote sensing image and T-1 non-reference remote sensing images. A seed pixel point P(x,y) in the reference remote sensing image is specified, where (x,y) represents the pixel coordinates in the reference remote sensing image. The category c corresponding to the seed pixel point P(x,y) is specified. For ease of distinction, the category c corresponding to the seed pixel point P(x,y) is called the target category. The seed pixel point P(x,y) is a pixel point that has not been fully labeled.

[0063] Step 402: Starting with the seed pixel, search for the foreground pixel corresponding to the target category in the reference remote sensing image, and determine the closed region of the reference remote sensing image based on the seed pixel and the foreground pixel.

[0064] For example, establishing a baseline label map M for a baseline remote sensing image. c，tThe resolution of the baseline label map is the same as that of the baseline remote sensing image. The baseline label map is used to store the prediction results of the baseline remote sensing image. c is the target category corresponding to the seed pixel, and t is the time sequence number corresponding to the baseline remote sensing image. In the initial state, all pixels in the baseline label map can be set to -1, indicating that all pixels are unpredicted pixels.

[0065] Based on this, starting with a seed pixel, we can predict whether surrounding pixels belong to the same category as the seed pixel. Then, we recursively predict all pixels outwards until pixels of different categories are predicted as the current feature edge. According to the characteristics of remote sensing imagery, ultimately, pixels of different categories will surround pixels of the same category to form a closed region A. c，t Enclosed area A c，t All pixels within the target pixel are of the same category as the seed pixel, meaning they all belong to the target category.

[0066] For example, regarding step 402, the closed region of the reference remote sensing image can be determined using the following steps:

[0067] Step 4021: Use the seed pixel as the starting pixel and determine whether there are any unpredicted pixels among the surrounding pixels of the starting pixel (e.g., the 8 surrounding pixels of the starting pixel).

[0068] For example, the reference remote sensing image corresponds to the reference label map M. c，t Find the seed pixel from the baseline label map, set the seed pixel to "1" to indicate that the seed pixel is a foreground pixel, and then determine whether there are any unpredicted pixels with a value of "-1" among the 8 surrounding pixels of the seed pixel.

[0069] If yes, then proceed to step 4022; otherwise, proceed to step 4024.

[0070] Step 4022: For each unpredicted pixel, determine the similarity between the starting pixel and the unpredicted pixel. If the similarity is greater than a preset threshold (this preset threshold can be configured based on experience and is not restricted), then the unpredicted pixel is determined as a foreground pixel; otherwise, if the similarity is not greater than the preset threshold, then the unpredicted pixel is determined as an edge pixel.

[0071] For example, based on the baseline label map M c，tFor each unpredictable pixel with a value of "-1", if the similarity between the starting pixel and the unpredictable pixel is greater than a preset threshold, the unpredictable pixel is set to "1", indicating that the unpredictable pixel is a foreground pixel. If the similarity is not greater than the preset threshold, the unpredictable pixel is set to "0", indicating that the unpredictable pixel is an edge pixel.

[0072] Step 4023: For each unpredicted pixel, if the unpredicted pixel is a foreground pixel, it can be used as a starting pixel; if the unpredicted pixel is an edge pixel, it can not be used as a starting pixel. At this point, at least one starting pixel can be obtained. For each starting pixel, return to the step of determining whether there are any unpredicted pixels among the surrounding pixels (e.g., the 8 surrounding pixels of the starting pixel).

[0073] In summary, steps 4021-4023 can be repeated until there are no unpredicted pixels in the surrounding pixels of the starting pixel (which can be multiple pixels after iteration), at which point step 4024 is executed.

[0074] Step 4024: Combine the seed pixels and all foreground pixels to form a closed region of the reference remote sensing image. For example, based on the reference label image M... c，t It can form a closed region by combining all pixels with a value of "1".

[0075] For example, in step 402, a search can be performed on the foreground pixels. The process of searching for foreground pixels can be called edge expansion. The process of searching for foreground pixels (i.e., predicting the category of the pixels surrounding the seed pixel) can be as follows: taking the seed pixel P(x,y) as the starting pixel, iteratively predicting whether the surrounding pixels from the starting pixel P(x,y) have the same category as the starting pixel P(x,y).

[0076] In each iteration, the starting pixel is set to P(x,y), and its eight surrounding pixels are set to Pi(x',y'). The similarity between the starting pixel P(x,y) and each of its eight surrounding pixels Pi(x',y') is calculated. After calculating the similarity, if the similarity is greater than a threshold thS, it indicates that the surrounding pixels Pi(x',y') are similar to the starting pixel P(x,y), and they can be considered to belong to the same category. The baseline label image M is then used. c，t The Pi point is set to 1, representing a foreground pixel. If the similarity is not greater than the threshold thS, it means that the surrounding pixel Pi(x',y') has extended to the edge of the current feature, and the baseline label map M is then set. c，t Setting Pi to 0 indicates an edge pixel.

[0077] Traverse all similar Pi points surrounding the starting pixel P(x,y), i.e., Pi points with a value of 1, and recursively expand the region as described above. The assignment method for the next round of recursion is: P <= Pi, Pi <= Pij (all surrounding pixels with a value of -1; <= means assigning the value to the right of the sign to the value to the left of the sign). Finally, the baseline label map M... c，t See the form Figure 5 As shown. Pixels with a value of 1 are foreground pixels, and pixels with a value of 0 are edge pixels. Obviously, edge pixels surround the foreground pixels, forming a closed area. The blank area has a value of -1, indicating an unpredicted pixel. In practical applications, an unpredicted pixel could be either a foreground pixel or an edge pixel.

[0078] For example, the set of foreground pixels can be designated as region A. c，t A represents the region consisting of all pixels with a value of 1. Clearly, the set of foreground pixels is the closed region of the baseline remote sensing image, denoted as A. c，t .

[0079] For example, considering the complexity of remote sensing image environments, methods such as opening operations can also be used to eliminate noise or micro-holes in the prediction results and reduce the impact of prediction instability. There are no restrictions on this.

[0080] In the above embodiments, it is necessary to determine the similarity between the starting pixel and the unpredicted pixel. For example, the similarity between the starting pixel and the unpredicted pixel can be determined as follows: Determine a first region centered on the starting pixel, and determine a second region centered on the unpredicted pixel. The size of the first region and the size of the second region can be the same, such as the size of the first region being 3*3 and the size of the second region being 3*3. Then, determine the first mean and the first variance of all pixels in the first region, and determine the second mean and the second variance of all pixels in the second region; based on the first mean, the first variance, the second mean, and the second variance, determine the similarity between the starting pixel and the unpredicted pixel. For example, the similarity between the starting pixel and the unpredicted pixel can be determined using the following formula (1).

[0081]

[0082] In formula (1), Simi(P, Pi) represents the similarity between the starting pixel P and the unpredicted pixel Pi, μ P μ represents the first mean value of all pixels in the first region. Pi σ represents the second mean of all pixels in the second region. P σ represents the first variance of all pixels in the first region. Pic1 and c2 represent the second variance of all pixels in the second region. Both c1 and c2 are preset constants, and there are no restrictions on these preset constants.

[0083] Step 403: For each non-reference remote sensing image (i.e., T-1 non-reference remote sensing images), starting with the seed pixel, search for the foreground pixel corresponding to the target category in the non-reference remote sensing image, and determine the closed region of the reference remote sensing image based on the seed pixel and the foreground pixel.

[0084] For example, step 403 is similar to step 402, and will not be repeated here. Seed pixels refer to pixels in the reference remote sensing image and pixels in non-reference remote sensing images.

[0085] After steps 402 and 403, the closed regions corresponding to T remote sensing images can be obtained. These closed regions can be denoted as A. c，1 A c，2 A c，T .

[0086] Step 404: For each non-reference remote sensing image, determine the similarity between the non-reference remote sensing image and the reference remote sensing image. For example, denote the closed region of the non-reference remote sensing image as A. c，ti The closed region of the reference remote sensing image is denoted as A. c，t The similarity between a non-reference remote sensing image and a reference remote sensing image can be determined based on the closed regions of the non-reference remote sensing image and the closed regions of the reference remote sensing image.

[0087] For example, a union region between a closed region of a non-reference remote sensing image and a closed region of a reference remote sensing image can be determined, and the similarity can be determined by the ratio of the number of pixels with inconsistent prediction results in the union region to the total number of pixels in the union region, and the similarity is inversely proportional to the ratio.

[0088] For example, see Figure 6A and Figure 6B As shown, Figure 6A All pixels with a value of 1 form a closed region A in the non-reference remote sensing image. c，ti , Figure 6B All pixels with a value of 1 form a closed region A in the baseline remote sensing image. c，t The closed region A can be determined. c，ti With closed area A c，tThe union region between the values ​​is determined, that is, the union region of all regions with a value of 1. Then, the total number of pixels in the union region is counted, and the number of pixels with inconsistent prediction results in the union region is counted (for example, if a pixel in the union region corresponds to 1 in one closed region and 0 in another closed region, it means that the prediction result of the pixel is inconsistent; otherwise, it means that the prediction result of the pixel is consistent).

[0089] Then, the ratio of the number of pixels with inconsistent predictions to the total number of pixels is calculated, and the similarity between the non-reference remote sensing image and the reference remote sensing image is determined based on this ratio. Clearly, a larger ratio indicates a higher proportion of pixels with inconsistent predictions, and a lower similarity. Conversely, a smaller ratio indicates a lower proportion of pixels with inconsistent predictions, and a higher similarity.

[0090] In one possible implementation, the similarity between a non-reference remote sensing image and a reference remote sensing image can be determined using the following formula (2). Of course, formula (2) is just an example and is not intended to limit the scope of the application.

[0091]

[0092] In formula (2), R(A) ti A t ) represents the closed region A c，ti With closed area A c，t The similarity between the non-reference remote sensing image and the reference remote sensing image. A ∪ Indicates closed region A c，ti With closed area A c，t The union region between A and B. p represents the union region A. ∪ For any pixel in M, c，t (p) indicates that pixel p is located in the closed region A. c，t The corresponding value in M ​​is either 1 or 0. c，ti (p) indicates that pixel p is located in the closed region A. c，ti The corresponding value is 1 or 0. This is an indicator function; it is 1 if the internal condition is true, and 0 otherwise.

[0093] In summary, in formula (2), the numerator represents the number of pixels with inconsistent prediction results (i.e., the area of ​​inconsistent prediction results), and the denominator represents the total number of pixels (i.e., the total area of ​​the union).

[0094] Step 405: Select the reference remote sensing image as the remote sensing image to be labeled. For each non-reference remote sensing image, if the similarity between the non-reference remote sensing image and the reference remote sensing image is greater than a preset threshold, then the non-reference remote sensing image is selected as the remote sensing image to be labeled; if the similarity is not greater than the preset threshold, then the selection of the non-reference remote sensing image as the remote sensing image to be labeled is prohibited. Thus, K remote sensing images to be labeled can be selected from T remote sensing images, where K is a positive integer greater than 1, and K is less than or equal to T.

[0095] For example, if R(A) ti A t If the value is greater than the preset threshold thR, it indicates that the similarity between the non-reference remote sensing image and the reference remote sensing image is relatively high, meaning that there is no change or only a small change between the remote sensing images of the two time periods. Therefore, the non-reference remote sensing image can be selected as the remote sensing image to be labeled. However, since A cannot be guaranteed... c，t With A c，ti The segmentation results are exactly the same, so they need to be aligned. See the following steps.

[0096] For example, if R(A) ti A t If the value is not greater than the preset threshold thR, it indicates that the similarity between the non-reference remote sensing image and the reference remote sensing image is relatively small, meaning that there is a significant change between the remote sensing images from the two time periods. Therefore, the non-reference remote sensing image is not selected as the remote sensing image to be labeled. In other words, in the current segmentation round, the closed region A of the non-reference remote sensing image will no longer be included. c，ti Mark the closed area A. c，ti The pixels in the image are unlabeled pixels, and in subsequent segmentation rounds, the closed region A is... c，ti The pixels in the image are labeled.

[0097] Step 406: Determine the fusion region based on the closed regions of the K remote sensing images to be labeled. That is, label and align the closed regions of the K remote sensing images to be labeled to obtain the fusion region.

[0098] In one possible implementation, the closed region of the reference remote sensing image can be used as the fusion region. That is to say, the fusion region of the K remote sensing images to be labeled is the closed region of the reference remote sensing image.

[0099] In another possible implementation, the fusion region can be determined using the following steps:

[0100] Step 4061: For each remote sensing image to be labeled (i.e., process the K remote sensing images to be labeled separately), for each pixel in the closed region of the remote sensing image to be labeled, determine the transition value of the pixel based on the minimum Euclidean distance between the pixel and the unclosed region and the configured value.

[0101] For example, for each pixel in a closed region of a remote sensing image to be labeled, the transition value of that pixel can be determined by formula (3). Of course, formula (3) is just an example and is not a limitation.

[0102]

[0103] In formula (3), p represents a pixel in a closed region of the remote sensing image to be labeled, M is any pixel in the closed region, and M c，ti (p) represents the transition value of pixel p. d_edge(p) represents the minimum Euclidean distance between pixel p and the non-closed region of the remote sensing image to be labeled, that is, the minimum Euclidean distance between pixel p and the edge pixel of the closed region (value 0). thE is a pre-configured value, configured based on experience.

[0104] In summary, for each pixel in a closed region of the remote sensing image to be labeled, the transition value of that pixel can be determined by formula (3), thereby obtaining the transition value of each pixel in the closed region.

[0105] Obviously, after performing the above processing on the K remote sensing images to be labeled, the transition value of each pixel in the closed region of the K remote sensing images to be labeled can be obtained. This process will not be repeated.

[0106] For example, determining the transition value for each pixel using the above method can reduce the value at the mask edges, so that the value gradually increases from 0 to 1 as it changes from the edge to the interior. See the final effect below. Figure 7 As shown.

[0107] Step 4062: Determine the union region among the closed regions of the K remote sensing images to be labeled. For each pixel in this union region, determine whether the pixel is a target pixel based on the corresponding transition value in the K remote sensing images to be labeled. For example, if the weighted average of the transition values ​​corresponding to the pixel in the K remote sensing images to be labeled is greater than a preset threshold, then the pixel is determined to be a target pixel; otherwise, if the weighted average is not greater than the preset threshold, then the pixel is determined not to be a target pixel.

[0108] For example, the union region among closed regions of K remote sensing images to be labeled can be determined, and this union region is denoted as A. ∪ Then, iterate through the union region A in sequence. ∪ For each pixel in the image, the weighted average value of the pixel can be determined by the following formula (4) based on the transition value of the pixel in each remote sensing image to be labeled. Of course, formula (4) is just an example and is not a limitation.

[0109]

[0110] In formula (4), p can represent the union region A. ∪ For any pixel in the image, W(p) can represent the weighted average value of pixel p, K can represent the total number of remote sensing images to be labeled, and M... c，j (p) can represent the transition value of pixel p in the j-th remote sensing image to be labeled, where j ranges from 1 to K.

[0111] After obtaining the weighted average value of pixel p, if the weighted average value is greater than the preset threshold (which can be configured based on experience and is not limited, such as 0.5, 0.6, etc.), then pixel p is determined to be the target pixel. Otherwise, if the weighted average value is not greater than the preset threshold, then pixel p is determined not to be the target pixel.

[0112] Clearly, for the union region A ∪ After performing the above processing on each pixel in the set, it can be determined whether the pixel is the target pixel, and then the result can be obtained from the union region A. ∪ Find all target pixels.

[0113] Step 4063: Determine the fusion region based on all target pixels in the union region. That is, the region composed of all target pixels can be used as the fusion region, and this fusion region can be used as the fusion region of each remote sensing image to be labeled. Thus, the fusion region of each remote sensing image to be labeled can be obtained.

[0114] Clearly, the fused region obtained through the above method can fully combine the information of the closed regions of all the remote sensing images to be labeled, making the labeling results more accurate, that is, the fused region is more accurate and reliable.

[0115] At this point, the synchronous annotation process is complete, and the fused region of each remote sensing image to be annotated is obtained. Based on the fused region of each remote sensing image to be annotated, edge refinement can be performed on the fused region of each remote sensing image to be annotated. It should be noted that the edge refinement process of different remote sensing images to be annotated can be completed in parallel and synchronously, that is, the edge refinement process of each remote sensing image to be annotated is performed at the same time. Of course, the edge refinement process of each remote sensing image to be annotated can also be performed at different times, and the edge refinement process of one remote sensing image to be annotated can be performed only after the edge refinement process of one remote sensing image to be annotated is completed.

[0116] Since the edge refinement process for the fusion region of each unannotated remote sensing image is the same, for ease of description, the following explanation will take the edge refinement of the fusion region of an unannotated remote sensing image as an example.

[0117] During edge refinement, the input data includes the fusion region of the remote sensing image to be labeled, denoted as fusion region A. c，t The input data also includes the labeled regions of the remote sensing image to be labeled. The labeled regions are those regions in the remote sensing image that have already been categorized. For example, in the first segmentation round, the labeled regions are empty. In the second segmentation round, the labeled regions are those regions in the first segmentation round that have already been categorized. In the third segmentation round, the labeled regions are those regions in both the first and second segmentation rounds that have already been categorized, and so on.

[0118] For example, the labeled area is denoted as A. else A else This includes all regions that have already been categorized, including other regions A′ of the same category. c，t And other areas A of different categories c′，t .

[0119] For example, since region A has been labeled else The edge finishing process has been completed, therefore, in the fusion region A c，t During the edge refinement process, the already labeled area A will not be affected. else The category needs to be modified.

[0120] For example, during edge refinement, the edges in the blending region A can be refined. c，t and the already labeled area A else Compare and refine the fusion area A. c，t The label. For example, if the merged region A c，t and the already labeled area A else If adjacent features are too close or overlap, they are considered to be different types of land features (such as a road and farmland on both sides) or the same type of land feature (such as two plots of farmland). Therefore, it is necessary to merge region A. c，t The category was refined to match the already labeled area A. else To fit together (referring to different categories) or to blend together (referring to the same category).

[0121] See Figure 8 The diagram shown illustrates the edge finishing process, which may include:

[0122] Step 801: Determine the non-overlapping areas between the merged area and the labeled area.

[0123] For example, if the merged region overlaps with the labeled region, the overlapping region can be removed from the merged region to obtain a non-overlapping region. For instance, if merged region A... c，t The pixels in the image are already located in the labeled area A.else If it is inside, it indicates that the fusion region A is... c，t With the already labeled area A else There are overlapping areas between them, therefore, it is necessary to merge from region A. c，t Remove the overlapping region from the merged region A to obtain the non-overlapping region. c，t Subtract the integration area A from the middle c，t With the already labeled area A else The intersection of these points can be represented by the following notation. CuP is the symbol for complement. If the merged region does not overlap with the labeled region, then the merged region can be defined as a non-overlapping region. For example, if merged region A... c，t None of the pixels in the diagram are located in the labeled area A. else If it is inside, it indicates that the fusion region A is... c，t With the already labeled area A else There is no overlapping area between them, therefore, region A will be merged. c，t As a non-overlapping region.

[0124] In summary, we can base our analysis on the fusion region A. c，t and the already labeled area A else Identify the non-overlapping region between the merged region and the labeled region. For ease of description, we will denot the non-overlapping region as A. c，t .

[0125] In one possible implementation, after removing the overlapping region from the fused region, the overlapping region in the remote sensing image to be labeled can also be labeled with the category corresponding to the labeled region. For example, if fused region A c，t With the already labeled area A else If there are overlapping areas, then it is necessary to merge from region A. c，t Remove the overlapping region and label the overlapping region as labeled region A. else The corresponding category, that is, labeling each pixel in the overlapping region with the labeled region A. else The corresponding category.

[0126] Step 802: For each edge pixel in the non-overlapping region, determine the nearest matching pixel in the labeled region, and determine the distance between the edge pixel and the matching pixel.

[0127] For example, located in non-overlapping region A c，t All pixels at the edge form a closed curve E. All pixels on the closed curve E are called edge pixels, hereinafter denoted as edge pixel p. Clearly, all pixels on the closed curve E belong to the non-overlapping region A. c，tAll pixels within the closed curve E belong to the non-overlapping region A. c，t All pixels outside the closed curve E do not belong to the non-overlapping region A. c，t .

[0128] For example, for each edge pixel p on the closed curve E (i.e., each edge pixel in the non-overlapping region), it can be obtained from the labeled region A. else In the process, determine the nearest matching pixel q to the edge pixel p, and determine the distance d between the edge pixel p and the matching pixel q.

[0129] Step 803: If the distance is less than a preset threshold, then all pixels on the line connecting the edge pixel and the matching pixel are added to the non-overlapping region. If the distance is not less than the preset threshold, then all pixels on the line connecting the edge pixel and the matching pixel are not added to the non-overlapping region.

[0130] For example, regarding the non-overlapping region A c，t For each edge pixel p, after determining the distance d between edge pixel p and matching pixel q, if the distance d is less than a preset threshold thD (which can be configured empirically), then for the non-overlapping region A... c，t To expand, all pixels on the line connecting edge pixel p and matching pixel q are added to the non-overlapping region A. c，t That is, complete the non-overlapping region A. c，t With the already labeled area A else The fitting or fusion. If the distance d is not less than the preset threshold thD, then the non-overlapping region A is not fitted or merged. c，t Expand upon.

[0131] Obviously, for the non-overlapping region A c，t After performing the above processing on each edge pixel p, the expanded non-overlapping region A can be obtained. c，t Based on the extended non-overlapping region A c，t Proceed to the next steps.

[0132] Step 804: Determine the target region of the remote sensing image to be labeled based on the non-overlapping region. For example, the expanded non-overlapping region A can be used as the target region. c，t The target area of ​​the remote sensing image to be labeled was determined.

[0133] Step 805: Label the target category for the target region in the remote sensing image to be labeled.

[0134] For example, after obtaining the target region of the remote sensing image to be labeled, it means that the edge refinement process is completed. This target region is the region corresponding to the target category. Therefore, the target category is labeled for the target region in the remote sensing image to be labeled. In subsequent segmentation rounds, the target region is updated to the labeled region.

[0135] For example, see Figure 9 As shown, for Example 1, the fusion region A c，t With the already labeled area A else There are overlapping areas between them, which can be merged from region A. c，t Remove the overlapping region and label the overlapping region as labeled region A. else The corresponding category. For Example 2, when the distance d between edge pixel p and matching pixel q is less than the preset threshold thD, all pixels on the line connecting edge pixel p and matching pixel q are added to the non-overlapping region A. c，t Label the target category for all pixels on the connecting line.

[0136] As can be seen from the above technical solutions, in this embodiment, the device can automatically label each pixel in remote sensing images across multiple time phases, eliminating the need for users to manually label each pixel. This reduces the workload of labeling remote sensing images, shortens labeling time, accelerates labeling efficiency, reduces labor costs, and improves user experience. Simultaneous labeling of remote sensing images across multiple time phases, followed by unified alignment and refinement, yields greater benefits. Furthermore, the alignment process globally considers the labeling results of all remote sensing images, resulting in greater accuracy. The device can automatically extract the fusion region of the remote sensing images (i.e., label each pixel within the fusion region) and perform simultaneous labeling operations on multiple time phases in parallel, enabling parallel synchronous labeling of remote sensing images, improving labeling efficiency, and reducing labeling time. Labellers do not need to focus on all pixels in each remote sensing image; each execution only requires inputting a seed pixel from a specific remote sensing image for automatic multi-time phase synchronous labeling and edge refinement, further enhancing labeling efficiency. Annotators only need to input a seed pixel from a remote sensing image, and the system automatically searches for and labels surrounding pixels of the same type, with all time phases processed synchronously. If there are no significant changes between time phases, label alignment is performed; otherwise, the changing time phase is not labeled, requiring no manual intervention. The annotation results are analyzed, and when pixels overlap or are close to already labeled areas, they are merged or aligned to complete refinement. Through multi-time phase synchronous annotation and edge refinement, the final effect is that with each click, labels for surrounding pixels are assigned. The synchronous annotation and edge refinement processes can be cyclically repeated until annotation is complete.

[0137] Based on the same concept as the above method, this application proposes a remote sensing image annotation device. The sample data set includes T remote sensing images from different time phases, where T is a positive integer greater than 1. See [link to relevant documentation]. Figure 10 The diagram shown is a structural schematic of the remote sensing image annotation device, which includes:

[0138] The acquisition module 1001 is used to acquire seed pixels and the target category corresponding to the seed pixels;

[0139] The processing module 1002 is used to search for foreground pixels corresponding to the target category in each remote sensing image, starting from the seed pixel; determine the closed region of the remote sensing image based on the seed pixel and the foreground pixel; and select K remote sensing images to be labeled from T remote sensing images based on the similarity between the closed regions of the remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T.

[0140] The determination module 1003 is used to determine the fusion region based on the closed regions of K remote sensing images to be labeled;

[0141] The annotation module 1004 is used to perform the following synchronous annotation operation on the K remote sensing images to be annotated in parallel based on the target category: determine the target region based on the fused region and the already annotated region of the remote sensing image to be annotated, and annotate the target category for the target region in the remote sensing image to be annotated; wherein, the already annotated region is the region in the remote sensing image to be annotated that has already been annotated with a category.

[0142] For example, when the processing module 1002 searches for foreground pixels corresponding to the target category in each remote sensing image using a seed pixel as the starting pixel, and determines the closed region of the remote sensing image based on the seed pixel and the foreground pixels, it specifically performs the following steps: For each remote sensing image, the seed pixel is used as the starting pixel, and it is determined whether there are any unpredicted pixels among the surrounding pixels of the starting pixel; if so, the similarity between the starting pixel and the unpredicted pixel is determined; if the similarity is greater than a preset threshold, the unpredicted pixel is determined as a foreground pixel, the unpredicted pixel is used as the starting pixel, and the process returns to perform the operation of determining whether there are any unpredicted pixels among the surrounding pixels of the starting pixel; otherwise, the unpredicted pixel is determined as an edge pixel; if not, the seed pixel and all foreground pixels form the closed region of the remote sensing image.

[0143] For example, when the processing module 1002 determines the similarity between the starting pixel and the unpredicted pixel, it specifically performs the following steps: determining the first mean and first variance of all pixels in a first region, and determining the second mean and second variance of all pixels in a second region; wherein the first region is centered on the starting pixel, the second region is centered on the unpredicted pixel, and the first region and the second region are of the same size; and determining the similarity between the starting pixel and the unpredicted pixel based on the first mean, the first variance, the second mean, and the second variance.

[0144] For example, when the processing module 1002 selects K remote sensing images to be labeled from T remote sensing images based on the similarity between closed regions of remote sensing images, it specifically performs the following: If the T remote sensing images include one reference remote sensing image and T-1 non-reference remote sensing images, then the reference remote sensing image is selected as the remote sensing image to be labeled. For each non-reference remote sensing image: the similarity between the non-reference remote sensing image and the reference remote sensing image is determined; wherein, based on the union region between the closed regions of the non-reference remote sensing image and the closed regions of the reference remote sensing image, the similarity is determined by the ratio of the number of pixels with inconsistent prediction results in the union region to the total number of pixels in the union region, and the similarity is inversely proportional to the ratio; if the similarity is greater than a preset threshold, then the non-reference remote sensing image is selected as the remote sensing image to be labeled; otherwise, the selection of the non-reference remote sensing image as the remote sensing image to be labeled is prohibited.

[0145] For example, when determining the fusion region based on the closed regions of K remote sensing images to be labeled, the determining module 1003 is specifically used to: for each pixel in the closed region of the remote sensing images to be labeled, determine the transition value of the pixel based on the minimum Euclidean distance between the pixel and the non-closed region and the configured value; determine the union region between the closed regions of the K remote sensing images to be labeled; for each pixel in the union region, determine whether the pixel is a target pixel based on the transition value corresponding to the pixel in the K remote sensing images to be labeled; and determine the fusion region based on all target pixels in the union region.

[0146] For example, when the annotation module 1004 determines the target region based on the fused region and the labeled region of the remote sensing image to be annotated, it is specifically used to: determine the non-overlapping region between the fused region and the labeled region; for each edge pixel in the non-overlapping region, determine the nearest matching pixel in the labeled region, and determine the distance between the edge pixel and the matching pixel; if the distance is less than a preset threshold, add all pixels on the line connecting the edge pixel and the matching pixel to the non-overlapping region; and determine the target region of the remote sensing image to be annotated based on the non-overlapping region.

[0147] For example, when the annotation module 1004 determines the non-overlapping region between the fused region and the annotated region, it is specifically used to: if the fused region and the annotated region have an overlapping region, then remove the overlapping region from the fused region to obtain the non-overlapping region; or, if the fused region and the annotated region do not have an overlapping region, then determine the fused region as the non-overlapping region.

[0148] Based on the same concept as the methods described above, this application proposes a remote sensing image annotation device, see [link to relevant documentation]. Figure 11 As shown, the remote sensing image annotation device may include: a processor 1101 and a machine-readable storage medium 1102, wherein the machine-readable storage medium 1102 stores machine-executable instructions that can be executed by the processor 1101; wherein the processor 1101 is used to execute the machine-executable instructions to implement the remote sensing image annotation method disclosed in the above example of this application.

[0149] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the remote sensing image annotation method disclosed in the above examples of this application.

[0150] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0151] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0152] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A remote sensing image annotation method, characterized in that, The sample dataset includes T remote sensing images from different time phases, where T is a positive integer greater than 1. The method includes: Obtain the seed pixel and the target category corresponding to the seed pixel; Starting with a seed pixel, search for foreground pixels in each remote sensing image that correspond to the target category, and determine the closed region of the remote sensing image based on the seed pixel and the foreground pixels. Based on the similarity between closed regions in remote sensing images, K remote sensing images to be labeled are selected from T remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T. The fusion region is determined based on the closed regions of all the K unlabeled remote sensing images; Based on the target category, the following synchronous annotation operation is performed in parallel on the K remote sensing images to be annotated: the target region is determined based on the fused region and the already annotated region of the remote sensing image to be annotated, and the target category is annotated for the target region in the remote sensing image to be annotated; wherein, the already annotated region is the region in the remote sensing image to be annotated that has already been annotated with a category.

2. The method according to claim 1, characterized in that, The step of searching for foreground pixels corresponding to the target category in each remote sensing image, starting from a seed pixel, and determining the closed region of the remote sensing image based on the seed pixel and the foreground pixels, includes: For each remote sensing image, the seed pixel is used as the starting pixel, and it is determined whether there are any unpredicted pixels among the surrounding pixels of the starting pixel. If so, determine the similarity between the starting pixel and the unpredicted pixel; If the similarity is greater than a preset threshold, the unpredicted pixel is determined as a foreground pixel, the unpredicted pixel is taken as the starting pixel, and the operation of determining whether there are unpredicted pixels among the surrounding pixels of the starting pixel is returned; otherwise, the unpredicted pixel is determined as an edge pixel. If not, the seed pixels and all foreground pixels will form a closed region of the remote sensing image.

3. The method according to claim 2, characterized in that, Determining the similarity between the starting pixel and the unpredicted pixel includes: Determine the first mean and first variance of all pixels in the first region, and determine the second mean and second variance of all pixels in the second region; wherein the first region is centered on the starting pixel, the second region is centered on the unpredicted pixel, and the first region and the second region are of the same size. Based on the first mean, the first variance, the second mean, and the second variance, the similarity between the starting pixel and the unpredicted pixel is determined.

4. The method according to claim 1, characterized in that, The similarity between closed regions based on remote sensing images involves selecting K remote sensing images to be labeled from T remote sensing images, including: If the T remote sensing images include one reference remote sensing image and T-1 non-reference remote sensing images, then the reference remote sensing image is selected as the remote sensing image to be labeled, and for each non-reference remote sensing image: Determine the similarity between the non-reference remote sensing image and the reference remote sensing image; wherein, based on the union region between the closed regions of the non-reference remote sensing image and the closed regions of the reference remote sensing image, the similarity is determined by the ratio of the number of pixels with inconsistent prediction results in the union region to the total number of pixels in the union region, and the similarity is inversely proportional to the ratio. If the similarity is greater than a preset threshold, the non-reference remote sensing image is selected as the remote sensing image to be labeled; otherwise, the selection of the non-reference remote sensing image as the remote sensing image to be labeled is prohibited.

5. The method according to claim 1, characterized in that, The step of determining the fusion region based on the closed regions of all the K unlabeled remote sensing images includes: For each pixel in a closed region of the remote sensing image to be labeled, a transition value for the pixel is determined based on the minimum Euclidean distance between the pixel and the unclosed region and the configured value; wherein, the minimum value between the minimum Euclidean distance and the configured value is determined, and the quotient of the minimum value and the configured value is determined as the transition value. Determine the union region among the closed regions of the K remote sensing images to be labeled; For each pixel in the union region, based on the transition value corresponding to that pixel in the K remote sensing images to be labeled, it is determined whether that pixel is a target pixel. The fusion region is determined based on all target pixels in the union region.

6. The method according to claim 1, characterized in that, The process of determining the target region based on the fused region and the labeled region of the remote sensing image to be labeled includes: Determine the non-overlapping region between the fused region and the labeled region; For each edge pixel in the non-overlapping region, the nearest matching pixel is determined from the labeled region, and the distance between the edge pixel and the matching pixel is determined; if the distance is less than a preset threshold, all pixels on the line connecting the edge pixel and the matching pixel are added to the non-overlapping region. The target region of the remote sensing image to be labeled is determined based on the non-overlapping region.

7. The method according to claim 6, characterized in that, Determining the non-overlapping region between the fused region and the labeled region includes: If the fused region overlaps with the labeled region, the overlapping region is removed from the fused region to obtain the non-overlapping region; or, if the fused region does not overlap with the labeled region, the fused region is determined as the non-overlapping region.

8. A remote sensing image annotation device, characterized in that, The sample dataset includes T remote sensing images from different time phases, where T is a positive integer greater than 1. The device includes: The acquisition module is used to acquire seed pixels and the target category corresponding to the seed pixels; The processing module is used to search for foreground pixels corresponding to the target category in each remote sensing image, starting from the seed pixel; determine the closed region of the remote sensing image based on the seed pixel and the foreground pixel; and select K remote sensing images to be labeled from T remote sensing images based on the similarity between the closed regions of the remote sensing images, where K is a positive integer greater than 1 and K is less than or equal to T. The determination module is used to determine the fusion region based on the closed regions of all the K unlabeled remote sensing images; The annotation module is used to perform the following synchronous annotation operation on the K remote sensing images to be annotated in parallel based on the target category: determine the target region based on the fused region and the already annotated region of the remote sensing image to be annotated, and annotate the target region in the remote sensing image to be annotated with the target category; wherein, the already annotated region is the region in the remote sensing image to be annotated that has already been annotated with a category.

9. The apparatus according to claim 8, characterized in that, in, The processing module uses a seed pixel as the starting pixel and searches for foreground pixels corresponding to the target category in each remote sensing image. Specifically, when determining the closed region of the remote sensing image based on the seed pixel and the foreground pixels, it performs the following steps: For each remote sensing image, it uses the seed pixel as the starting pixel and determines whether there are any unpredictable pixels among the surrounding pixels of the starting pixel; if so, it determines the similarity between the starting pixel and the unpredictable pixel; if the similarity is greater than a preset threshold, it determines the unpredictable pixel as a foreground pixel, uses the unpredictable pixel as the starting pixel, and returns to perform the operation of determining whether there are any unpredictable pixels among the surrounding pixels of the starting pixel; otherwise, it determines the unpredictable pixel as an edge pixel; if not, it combines the seed pixel and all foreground pixels to form the closed region of the remote sensing image. Specifically, when determining the similarity between the starting pixel and the unpredicted pixel, the processing module is used to: determine the first mean and first variance of all pixels in a first region, and determine the second mean and second variance of all pixels in a second region; wherein the first region is centered on the starting pixel, the second region is centered on the unpredicted pixel, and the first region and the second region are of the same size; and determine the similarity between the starting pixel and the unpredicted pixel based on the first mean, the first variance, the second mean, and the second variance. Specifically, when the processing module selects K remote sensing images to be labeled from T remote sensing images based on the similarity between closed regions of remote sensing images, it performs the following: If the T remote sensing images include one reference remote sensing image and T-1 non-reference remote sensing images, the reference remote sensing image is selected as the remote sensing image to be labeled. For each non-reference remote sensing image: the similarity between the non-reference remote sensing image and the reference remote sensing image is determined; wherein, based on the union region between the closed regions of the non-reference remote sensing image and the closed regions of the reference remote sensing image, the similarity is determined by the ratio of the number of pixels with inconsistent prediction results in the union region to the total number of pixels in the union region, and the similarity is inversely proportional to the ratio; if the similarity is greater than a preset threshold, the non-reference remote sensing image is selected as the remote sensing image to be labeled; otherwise, the selection of the non-reference remote sensing image as the remote sensing image to be labeled is prohibited. Specifically, when determining the fusion region based on the closed regions of all the K unlabeled remote sensing images, the determining module is used to: for each pixel in the closed region of the unlabeled remote sensing image, determine the transition value of the pixel based on the minimum Euclidean distance between the pixel and the non-closed region and a configured value; wherein, the minimum value of the minimum Euclidean distance and the configured value is determined, and the quotient of the minimum value and the configured value is determined as the transition value; and, determine the union region among the closed regions of the K unlabeled remote sensing images; for each pixel in the union region, determine whether the pixel is a target pixel based on the corresponding transition value of the pixel in the K unlabeled remote sensing images; and determine the fusion region based on all target pixels in the union region. Specifically, when the annotation module determines the target region based on the fused region and the already annotated region of the remote sensing image to be annotated, it is used to: determine the non-overlapping region between the fused region and the already annotated region; for each edge pixel in the non-overlapping region, determine the nearest matching pixel in the already annotated region, and determine the distance between the edge pixel and the matching pixel; if the distance is less than a preset threshold, add all pixels on the line connecting the edge pixel and the matching pixel to the non-overlapping region; and determine the target region of the remote sensing image to be annotated based on the non-overlapping region. Specifically, when the annotation module determines the non-overlapping region between the fused region and the annotated region, it is used to: if the fused region and the annotated region have an overlapping region, remove the overlapping region from the fused region to obtain the non-overlapping region; or, if the fused region and the annotated region do not have an overlapping region, determine the fused region as the non-overlapping region.

10. A remote sensing image annotation device, characterized in that, The method includes a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the steps of the method according to any one of claims 1-7.

Images