Remote sensing image interactive segmentation method, computer device and storage medium

Abstract

The present application relates to a remote sensing image interactive segmentation method, computer equipment, and storage medium. Dynamic radius coding is used to reasonably prejudge data information, allowing the device network to appropriately amplify the influence of the interaction point coordinates in the early stages, thereby enabling faster output of higher-precision images. In the middle and late stages, the influence of the interaction point coordinates is reduced, thereby enabling stable output of high-precision images. Simultaneously, an enhanced learning method matching the dynamic radius coding is designed to assist in image output. This method enables the image segmentation model to achieve rapid and accurate segmentation of remote sensing images while maintaining stability.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a remote sensing image interactive segmentation method, computer equipment and storage medium. Background Art

[0002] With the implementation of an increasing number of Earth observation projects, the field of remote sensing has experienced explosive growth, ushering in an era of remote sensing big data. Interactive segmentation of remote sensing images is a fundamental research topic in Earth observation, playing an important role in application areas such as traffic monitoring, road extraction, and land resource management.

[0003] Traditional intelligent interactive segmentation methods either focus on improving the convergence speed of the prediction device network, that is, improving the device's ability to quickly and accurately output images. Based on reinforcement learning theory, an additional prediction device network is designed to automatically generate interaction points to reduce the number of user interactions required; or they focus on improving the convergence of the prediction device network, that is, improving the stability of the device's image output. Combined with the inverse correction method of the L-BFGS optimization algorithm, they correct possible errors in the predicted image through repeated iterations during the inference phase.

[0004] However, in the process of implementing the present invention, the inventors found that the traditional intelligent interactive segmentation method cannot achieve rapid and accurate segmentation of remote sensing images while being stable. Summary of the Invention

[0005] Based on the above technical problems, a remote sensing image interactive segmentation method, a remote sensing image interactive segmentation device, a computer device and a computer-readable storage medium are provided to solve the above problems.

[0006] A remote sensing image interactive segmentation method, the method comprising:

[0007] Obtain the mask to be corrected, the coordinates of the interaction points, and the target image collected by the remote sensing equipment;

[0008] Dynamic radius encoding is performed according to the mask to be corrected, the coordinates of the interaction point and the target image to obtain the interaction feature map and the focused image; the focused image is obtained by focusing and cropping the target image;

[0009] Adjust the channel and size of the interaction feature map to obtain interaction information;

[0010] Utilize the first layer of the segmenter to extract semantic information of the focused image;

[0011] Fusing interactive information with semantic information to obtain comprehensive information;

[0012] Based on the interactive feature map and the focused image, the segmenter and the adjustment process are iteratively optimized using an incremental learning method. During the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain the initial predicted image. The other layers of the segmenter are the number of segmenter layers except the first layer.

[0013] The initial predicted image is subjected to an inverse focusing transform to obtain the final segmented image.

[0014] In one embodiment, the process of performing dynamic radius encoding to obtain an interactive feature map includes:

[0015] The coordinates of the interaction point are spliced ​​with the coordinates of the historical interaction points to obtain the coordinates of the current interaction point;

[0016] Focusing the mask to be corrected to obtain a focused mask to be corrected;

[0017] Focusing the coordinates of the current interaction point to obtain the focused interaction point coordinates; the focused interaction point coordinates include the positive focused interaction point coordinates and the negative focused interaction point coordinates;

[0018] The focused mask to be corrected is corrected according to the coordinates of the focused interaction points to obtain an interactive segmentation feature map; the interactive segmentation feature map includes a positive interactive segmentation feature map and a negative interactive segmentation feature map;

[0019] The focused mask to be corrected and the interactive segmentation feature map are spliced ​​to obtain the interactive feature map.

[0020] In one embodiment, the step of correcting the focus mask to be corrected according to the focus interaction point coordinates to obtain the interactive segmentation feature map includes:

[0021] Using a Laplace filter to extract the mask edge of the mask to be corrected;

[0022] Perform distance transformation on the coordinates of the focused interaction points to generate a distance feature map;

[0023] The minimum distance between the focus interaction point coordinates and the mask edge is calculated according to the distance feature map and the mask edge to obtain the coding radius; the focus interaction point coordinates include the positive focus interaction point coordinates and the negative focus interaction point coordinates;

[0024] The coordinates of the positive focus interaction point and the negative focus interaction point are encoded respectively according to the encoding radius to obtain the positive interaction segmentation feature map and the negative interaction segmentation feature map.

[0025] In one embodiment, the coding radius includes a first coding radius, a second coding radius, and a third coding radius, the distance mapping interval of the first coding radius is (0, 3], the distance mapping interval of the second coding radius is (3, 10], and the distance mapping interval of the third coding radius is (10, +∞).

[0026] In one embodiment, the process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method includes a convergence speed priority training phase, which is:

[0027] The coding radius is set to the third coding radius, and the minimum area of ​​the true value mask is Z pixels; Z is a positive integer not less than 1;

[0028] Extract the focused image and the mask to be corrected;

[0029] Generate M interaction point coordinates by random sampling and obtain an initialization mask based on the focused image and the mask to be corrected; M is a positive integer not less than 1;

[0030] After performing N corrected sampling segmentation on the initialized mask according to the true value mask, return to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches the set number of tasks, and calculate the loss function of the last corrected sampling segmentation; N is a positive integer not less than 1;

[0031] Update the parameters of the image segmentation model according to the loss function.

[0032] In one embodiment, the process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method further includes an incremental training phase, which includes:

[0033] After the convergence speed priority training phase is completed, the coding radius is set to the dynamic coding radius, and the minimum area of ​​the true value mask is S pixels; the dynamic coding radius includes the first coding radius, the second coding radius, or the third coding radius, and S is a positive integer greater than 1 and less than Z;

[0034] Extract the focused image and the mask to be corrected;

[0035] Generate M interaction point coordinates at one time through random sampling, obtain an initialization mask based on the focused image and the mask to be corrected, and perform M random sampling and segmentation based on the initialization mask to obtain the first loss function; M is a positive integer not less than 1;

[0036] After performing N corrective sampling segmentation on the initialized mask according to the true value mask, return to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches the set number of tasks, and calculate the second loss function during the last corrective sampling segmentation; N is a positive integer not less than 1;

[0037] Update the parameters of the image segmentation model according to the first loss function and the second loss function.

[0038] In one embodiment, the segmenter is HRNet32-OCR.

[0039] A remote sensing image interactive segmentation device, comprising:

[0040] Data acquisition module: used to obtain the mask to be corrected, the coordinates of the interaction points, and the target image collected by the remote sensing equipment;

[0041] Dynamic radius encoding module: used to perform dynamic radius encoding based on the mask to be corrected, the coordinates of the interaction point and the target image to obtain the interaction feature map and the focused image; the focused image is obtained by focusing and cropping the target image;

[0042] Preprocessing module: used to adjust the channel and size of the interaction feature map to obtain interaction information;

[0043] Semantic extraction module: used to extract semantic information of the focused image using the first layer of the segmentor;

[0044] Information fusion module: used to fuse interactive information with semantic information to obtain comprehensive information;

[0045] Incremental learning module: It is used to iteratively optimize the segmenter and the adjustment process using incremental learning methods based on the interactive feature map and the focused image. During the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain the initial predicted image. The other layers of the segmenter are the number of segmenter layers except the first layer.

[0046] Inverse focusing module: used to perform inverse focusing transformation on the initial predicted image to obtain the final segmented image.

[0047] On the other hand, a computer device is also provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the above-mentioned remote sensing image interactive segmentation method are implemented.

[0048] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned remote sensing image interactive segmentation method are implemented.

[0049] The above-mentioned interactive remote sensing image segmentation method, computer device, and storage medium first obtains the mask to be corrected, the coordinates of the interaction points, and the target image captured by the remote sensing device to provide data information for the current image segmentation. Then, dynamic radius encoding is performed based on the obtained data information to obtain an interaction feature map and a focused image. The focused image is obtained by focusing and cropping the target image. By focusing on the important features of the target image, the computational workload of the prediction device is reduced. Dynamic radius encoding performs reasonable pre-judgment encoding based on the data information, allowing the device network to appropriately amplify the influence of the interaction point coordinates in the early stage, allowing the device to output a more accurate image more quickly. The device network then reduces the influence of the interaction point coordinates in the middle and late stages, avoiding a decrease in the accuracy of the device output image due to overshoot and other reasons, thereby ensuring the accuracy of the device output image from a stability perspective.

[0050] After obtaining the interaction feature map, it is also necessary to adjust the channel and size of the interaction feature map, and use the first layer of the segmenter to extract the semantic information of the focused image to maintain the uniformity of the information form and ensure the smooth progress of the subsequent information fusion process, and then ensure the integrity of the data information by fusing the interaction information and semantic information.

[0051] In addition, a matching incremental learning method is designed to iteratively optimize the segmenter and adjustment process. The interactive information is used to ensure that the entire image segmentation model can conform to the dynamic radius encoding, so that the image segmentation model has a fast recognition and positioning capability in the early stage, and the coordinates of the interaction points are predicted and adjusted according to the actual situation in the middle and late stages to avoid excessive autonomy of the image segmentation model, which leads to a decrease in the output image accuracy, thereby making the image segmentation model have better stability.

[0052] After the above steps, the final inverse focusing is performed to restore the originally focused transformed image to the form of the target image, thereby quickly and accurately outputting the final segmented image consistent with the form of the target image while taking into account stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 1 is a flow chart of an interactive remote sensing image segmentation method according to an embodiment;

[0054] Figure 2 A schematic diagram of a process for performing dynamic radius encoding to obtain an interactive feature map in one embodiment;

[0055] Figure 3 1 is a schematic diagram of a process for correcting a focus mask to be corrected according to the coordinates of a positive focus interaction point and a negative focus interaction point, respectively, in one embodiment;

[0056] Figure 4A schematic diagram of a process for prioritizing convergence speed training of a segmenter and an adjustment process using an incremental learning method in one embodiment;

[0057] Figure 5 2 is a schematic diagram of the structure of modules of a remote sensing image interactive segmentation device in one embodiment;

[0058] Figure 6 A schematic diagram of the process of incremental training of an image segmentation model in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are merely used to explain the present application and are not intended to limit the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0060] It should be noted that the reference to "embodiment" in this document means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The presentation of this phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be understood by those skilled in the art that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0061] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0062] In one embodiment, Figure 1 As shown, a remote sensing image interactive segmentation method is provided, the method comprising:

[0063] Step 102: Obtain the mask to be corrected, the coordinates of the interaction points, and the target image captured by the remote sensing equipment.

[0064] In this step, the mask to be corrected, i.e., the segmented image output by the prediction device from the previous prediction, can be used to provide the raw data for the current prediction output. The prediction device can be a smartphone, tablet, or a remote control with a built-in screen on a drone. It is noteworthy that when the user performs the first click, the prediction device can initialize the mask to a zero matrix and set the result of the dynamic radius encoding to the maximum value, thereby ensuring that the prediction device's calculations and processing are fast and accurate during the first click. The interaction point coordinates can be manually input, such as when the user clicks on the target image near desired or undesired target information. Alternatively, the interaction point coordinates can be generated by the prediction device automatically generating click inputs on the target image according to a predefined program. The predefined program is designed based on actual needs. For example, if random verification is required, a random click program can be designed, while if fixed-point verification of a specific target is required, a fixed-point click program can be designed. In this step, the target image can be acquired using remote sensing equipment, providing the most original and reliable data sample for subsequent processing.

[0065] Step 104 , dynamic radius encoding is performed according to the mask to be corrected, the coordinates of the interaction points, and the target image to obtain an interaction feature map and a focused image; the focused image is obtained by focusing and cropping the target image.

[0066] In this step, dynamic radius encoding has the characteristic of only affecting a portion of the interaction feature map, reducing the affected area and thus preventing the loss of prediction accuracy due to interference between positive and negative clicks. The encoding object can be the minimum distance between the input interaction point coordinates and the mask to be corrected. This minimum distance varies with the input interaction point coordinates. The interaction feature map is generated by encoding the minimum distances between all interaction point coordinates and the mask to be corrected.

[0067] Through dynamic radius encoding, the position information of all interaction point coordinates is converted from the form of two-dimensional coordinates to the form of interaction feature maps, providing complete and reliable data for the subsequent judgment of the influence of each interaction point coordinate. In addition, the interaction feature map, as a collection of influences corresponding to the interaction point coordinates, can be combined with the target image to amplify the influence corresponding to the early interaction point coordinates, so that the network of the prediction device can output images with higher accuracy faster, and reduce the influence corresponding to the middle and late interaction point coordinates in combination with the target image, so as to avoid the decrease in the accuracy of the network output image of the prediction device due to overshoot and other reasons, and ensure the accuracy of the network output image of the prediction device from the perspective of stability.

[0068] Step 106: Adjust the channel and size of the interaction feature map to obtain interaction information.

[0069] In this step, since the size and channels of the interaction feature map will change during dynamic radius encoding due to the integration of the focused mask to be corrected, to ensure the smooth progress of the subsequent information fusion process, the channels and size of the interaction feature map must be adjusted to ensure that the resulting interaction information and subsequent semantic information are consistent in form. Interaction information is data based on the coordinates of the interaction point where the user clicks on the target image. It is based on human subjective intention and is used to indicate human-led information during the image segmentation process. Reasonable prejudgment of this interaction information in step 104 ensures that the user's clicks are accurate and effective during the human-computer interaction process, and can serve as reference data for correcting the mask to be corrected.

[0070] Step 108 : Extract semantic information of the focused image using the first layer of the segmenter.

[0071] In this step, the focused image can be understood as the image obtained by focusing and magnifying the location or target of interest. This can effectively improve the local details of the predicted output target image, and is particularly effective for the segmentation of small targets. Extraction usually involves inputting the focused image into the first layer of the segmenter for convolution, and outputting semantic information consistent with the above-mentioned interactive information format. Semantic information is data information based on the target image collected by the remote sensing device, which preserves the objective target features of the target image itself. It is used to indicate the information dominated by the prediction device in image segmentation, and serves as reference comparison data for correcting the mask to be corrected, so as to ensure that the predicted output image does not deviate from the actual target image.

[0072] Step 110: Fusing the interactive information with the semantic information to obtain comprehensive information.

[0073] In this step, the interactive information and semantic information obtained through steps 106 and 108 are identical in form. These two complement and constrain each other, ensuring that the final output image does not deviate from the actual target image while allowing the user to maintain their dominant position during human-computer interaction. This step integrates the interactive information and semantic information, ensuring data integrity and rigorous logic, thereby ensuring the accuracy and reliability of the combined information.

[0074] Step 112, based on the interactive feature map and the focused image, the segmenter and the adjustment process are iteratively optimized using an incremental learning method; during the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain an initial predicted image, and the other layers of the segmenter are the number of segmenter layers except the first layer of the segmenter.

[0075] It can be understood that the interactive segmentation process is an iterative one, meaning that the user will adjust the image position by continuously adding interaction point coordinates until the desired image is obtained. Therefore, this step utilizes an incremental learning method to optimize each iteration, thereby shortening the time it takes for the user to obtain the desired image by outputting a high-precision mask. In the first incremental learning stage, the dynamic radius encoding encoding result is set to the maximum value, enabling the image segmentation model in the prediction device to quickly identify and locate the image in the initial stage. Then, based on this image segmentation model's rapid identification and positioning, the second incremental learning stage is carried out. Dynamic radius encoding is used to set the encoding result to dynamically change, i.e., the interaction point coordinates are determined based on the actual situation. This avoids excessive autonomy of the image segmentation model, which may lead to reduced output image accuracy, thereby ensuring better stability of the image segmentation model. It is worth noting that during the optimization process, other layers of the segmenter perform segmentation based on the information features in the comprehensive information. In other words, the portion of interest to the user is accurately segmented through repeated calculations and marked for output. In addition, since the coordinates of the interaction points will change dynamically according to the needs each time the user adjusts them, the image segmentation model in the prediction device can automatically correct the encoding results corresponding to the interaction point coordinates in the interaction feature map that cannot truly reflect the user needs in the next iteration.

[0076] Step 114 : Perform an inverse focusing transformation on the initial predicted image to obtain a final segmented image.

[0077] In this step, the inverse focusing transformation is to restore the size and form of the initial predicted image to the size and form of the target image collected by the remote sensing equipment, that is, the final segmented image, so as to ensure the unity of the size and form of the output image and the input image.

[0078] The above-mentioned interactive segmentation method of remote sensing images initializes the mask to be corrected as a zero matrix and sets the result obtained by dynamic radius encoding to the maximum value, thereby ensuring that the network operation of the prediction device is fast and accurate at the first click; converts the position information of all interaction point coordinates from two-dimensional coordinates to interaction feature maps, and provides complete and reliable data for the subsequent prediction device network to judge the influence of each interaction point coordinate; by combining with the target image to amplify the influence corresponding to the early interaction point coordinates, the prediction device network can output a more accurate image faster, and by combining with the target image to reduce the influence corresponding to the middle and late interaction point coordinates, avoid the network of the prediction device due to overshoot and other reasons. The accuracy of the network output image decreases instead of increases, ensuring the accuracy of the network output image of the prediction device from the perspective of stability. By designing a two-stage incremental learning method, in the first stage of incremental learning, the encoding radius of the dynamic radius encoding is set to the maximum value, so that the image segmentation model has the ability to quickly identify and locate in the initial stage. Then, based on the image segmentation model's ability to quickly identify and locate, the second stage of incremental learning is carried out, and the encoding radius of the dynamic radius encoding is set to dynamically change. The coordinates of the interaction points are judged according to the actual situation to avoid excessive autonomy of the image segmentation model, which leads to a decrease in the accuracy of the output image, thereby ensuring better stability of the image segmentation model. After the above steps, inverse focusing is finally performed to restore the originally focused transformed image to the form of the target image, so as to quickly and accurately output the final segmented image consistent with the form of the target image while taking into account stability.

[0079] In one embodiment, Figure 2 As shown in FIG, the process of performing dynamic radius encoding to obtain the interactive feature map includes:

[0080] Step 202: Concatenate the interaction point coordinates with the historical interaction point coordinates to obtain the current interaction point coordinates.

[0081] In this step, the historical interaction point coordinates are the interaction point coordinates stored before the current image segmentation. The purpose of splicing the interaction point coordinates with the historical interaction point coordinates is to complete the historical interaction information, ensure the integrity of the interaction information, and prevent the image segmentation model from forgetting the historical user click information and causing the prediction accuracy to regress.

[0082] Step 204 : focusing the mask to be corrected to obtain a focused mask to be corrected.

[0083] In this step, consistent with step 108 , focusing the mask to be corrected can be understood as a mask obtained by focusing and magnifying the mask to be corrected, which is matched with the focused target image to jointly improve the local details of the predicted output target image.

[0084] Step 206 , focusing the current interaction point coordinates to obtain focused interaction point coordinates; the focused interaction point coordinates include positive focused interaction point coordinates and negative focused interaction point coordinates.

[0085] It is worth noting that the coordinates of the current interaction point will also change due to the enlargement of the image, so this step will also focus the coordinates of the current interaction point to obtain the focused interaction point coordinates, where the positive focused interaction point coordinates are used to mark the area of ​​interest to the user, and the negative focused interaction point coordinates are used to mark the area of ​​no interest to the user. This ensures that the interaction point coordinate information used in the subsequent correction process is comprehensive and scientific.

[0086] Step 208 , correcting the focus mask to be corrected according to the focus interaction point coordinates to obtain an interaction segmentation feature map; the interaction segmentation feature map includes a positive interaction segmentation feature map and a negative interaction segmentation feature map.

[0087] It can be understood that the correction process can be to correct or complete the to-be-corrected or missing parts of the mask to be corrected according to the coordinate position of the focus interaction point to obtain a complete mask. The complete mask can be used for image segmentation of the subsequent segmenter and contains the complete features of the target image, so it is called an interactive segmentation feature map, which provides complete and reliable data information for subsequent image segmentation. Since the focus interaction point coordinates include the positive focus interaction point coordinates and the negative focus interaction point coordinates, all generated interactive segmentation feature maps also include positive interactive segmentation feature maps and negative interactive segmentation feature maps, that is, the focus mask to be corrected is corrected according to the positive focus interaction point coordinates to obtain a positive interactive segmentation feature map, and the focus mask to be corrected is corrected according to the negative focus interaction point coordinates to obtain a negative interactive segmentation feature map.

[0088] In step 210 , the focused mask to be corrected and the interactive segmentation feature map are spliced ​​to obtain an interactive feature map.

[0089] In this step, stitching can be understood as overlapping the focused mask to be corrected, the positive interaction segmentation feature map, and the negative interaction segmentation feature map to ensure the integrity of the data.

[0090] In this embodiment, the integrity of the interactive information in the adjustment process is guaranteed by combining the historical interaction point coordinates, so as to ensure that the segmentation model does not forget before the user obtains the desired target; by focusing and magnifying the mask to be corrected and matching it with the focused target image, the local details of the predicted output target image are jointly improved; by classifying the focused interaction point coordinates into positive focused interaction point coordinates and negative focused interaction point coordinates, which are used to represent the areas of interest and areas of no interest to the user respectively, it is guaranteed that the interaction point coordinate information used in the subsequent correction process is comprehensive and scientific; by correcting the focused mask to be corrected according to the focused interaction point coordinates, an interactive segmentation feature map is obtained, which provides complete and reliable data information for the subsequent segmenter to perform image segmentation.

[0091] In one embodiment, Figure 3 As shown, the process of correcting the focus mask to be corrected according to the positive focus interaction point coordinates and the negative focus interaction point coordinates respectively includes:

[0092] Step 302: Using a Laplacian filter, extract the mask edge of the mask to be corrected.

[0093] In this step, the Laplacian filter is used because it is simple, efficient, and computationally inexpensive, allowing it to efficiently extract the mask edges of the mask to be corrected. The extraction process is accomplished using the following algorithm: Specifically, the h×w binary mask is convolved with the Laplacian kernel to obtain a binary double-edge image of the mask to be corrected. The outer edge of the double edge is then extracted to complete the mask edge extraction. The Laplacian kernel h can be expressed as:

[0094]

[0095] Edge_map=Conv(binary mask,kernel=h)<0

[0096] Conv represents the convolution operation, and the convolution kernel is the value of kernel. The "<0" operation sets all values ​​in the matrix less than 0 to 1, and all others to 0.

[0097] Step 304: Perform distance transformation on the coordinates of the focused interaction point to generate a distance feature map.

[0098] In this step, the distance transformation process can be to convert the coordinates of the focused interaction point from the form of two-dimensional coordinates into the form of distance information between the point and the edge of the image feature. All distance information generates a distance feature map, which can fully represent the difference between the user's required image and the mask to be corrected.

[0099] Step 306, calculating the minimum distance between the focus interaction point coordinates and the mask edge according to the distance feature map and the mask edge to obtain the coding radius; the focus interaction point coordinates include positive focus interaction point coordinates and negative focus interaction point coordinates.

[0100] In this step, each minimum distance is mapped to the encoding radius of the corresponding focus interaction point coordinates, and the mapping is performed according to a manually designed radius mapping function. When the minimum distance is small, it means that it is closer to the image required by the user, that is, the user wants to make fine adjustments. At this time, the influence of the interaction point coordinates is usually further reduced to ensure the stability of the output data of the image segmentation model; when the minimum distance is large, it means that it is far away from the image required by the user, that is, the user wants to make larger adjustments. At this time, the influence of the interaction point coordinates is usually further expanded to improve the speed and accuracy of the output data of the image segmentation model. The radius mapping function is a piecewise function, which is divided into three levels according to the energy distribution characteristics, that is, three levels of encoding radius.

[0101] In step 308 , the coordinates of the positive focus interaction point and the negative focus interaction point are encoded according to the encoding radius to obtain a positive interaction segmentation feature map and a negative interaction segmentation feature map.

[0102] In this step, each focus interaction point coordinate is encoded according to different coding radii. It is worth noting that the encoding methods of the positive / negative focus interaction point coordinates are not exactly the same. The reason is that the purpose of the positive focus interaction point coordinates is to expand the area range of the mask to be corrected, so they generally only appear on the outside of the mask. However, the purpose of the negative focus interaction point coordinates can be located on the inside of the mask to remove the erroneous segmentation area, or on the outside of the mask to indicate the area of ​​no interest. When the negative focus interaction point coordinates are located on the inside, the minimum distance calculated by the mask to be corrected cannot truly reflect the influence of the current negative focus interaction point coordinates. At this time, the current negative focus interaction point coordinates will automatically adjust their radius in the next iteration because they are located outside the mask, thereby effectively solving the errors that occur in the process and ensuring the validity of the image segmentation model prediction.

[0103] In this embodiment, a Laplace filter is used to efficiently extract the edges of the required mask to be corrected; a distance feature map is generated by performing distance transformation on the coordinates of the focused interaction point to ensure smooth subsequent processing; a coding radius is obtained by calculating the minimum distance between the coordinates of the focused interaction point and the edge, and the influence of the coordinate point is adjusted according to its size to ensure that the image segmentation model can quickly output high-precision images while taking into account stability; finally, the special case of the negative focused interaction point coordinates is fully considered and corrected to ensure the effectiveness of the image segmentation model prediction.

[0104] In one embodiment, the coding radius includes a first coding radius, a second coding radius, and a third coding radius, the distance mapping interval of the first coding radius is (0, 3], the distance mapping interval of the second coding radius is (3, 10], and the distance mapping interval of the third coding radius is (10, +∞).

[0105] In this embodiment, the first and second coding radii are relatively small. The influence of coding radii within these radii is relatively small, and the influence of coding radii within these radii is typically reduced to prevent overshoot. The third coding radius is relatively large, and the influence of coding radii within this radius is relatively large. The influence of coding radii within this radius is typically amplified to enable the prediction device network to output high-precision images more quickly. The three levels of coding radius correspond to respective mapping intervals, which are calculated based on the radius mapping function. Note that the unit of the coding radius is pixels (px).

[0106] The process of incremental training of image segmentation model is as follows Figure 6 As shown, the coding radius is first set to the third coding radius, and the pre-trained segmentation model is trained with convergence speed priority to obtain a transition model. This process ensures that the image segmentation model obtained by the final training can quickly and accurately complete the image segmentation. Then the coding radius is set to the dynamic coding radius, and the transition model is incrementally trained. This process ensures that the image segmentation model obtained by the final training can stably complete the image segmentation. Through two training stages, the image segmentation model can quickly and stably output high-precision images.

[0107] In one embodiment, Figure 4 The process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method includes a convergence speed priority training phase, which is:

[0108] Step 402 : Set the coding radius to a third coding radius, and the minimum area of ​​the true value mask to Z pixels; Z is a positive integer not less than 1.

[0109] In this step, first, one or more labeled masks of the target image are merged to obtain a true value mask, and the minimum area of ​​the true value mask is set to Z pixels, usually 8000px. Then, random sampling is performed based on the true value mask to generate M positive / negative interaction points, and the initial mask to be corrected is obtained through the image segmentation model. Since the threshold value of the control sampling point M is too random, the result is difficult to control. In order to improve the accuracy and convergence speed of the network judgment of the prediction device, this step adopts the method of setting the coding radius to the third coding radius, and the minimum area of ​​the true value mask is set to Z pixels. The larger the minimum area of ​​the true value mask, the wider the target area, and the network of the prediction device can more boldly amplify the influence of the subsequent randomly sampled interaction point coordinates, and Z is a positive integer not less than 1.

[0110] Step 404: extract the focused image and the mask to be corrected.

[0111] The focused image and mask to be corrected here are identical in nature to those in step 102. The target image is also acquired by remote sensing equipment and provides the most original and reliable data sample for subsequent processing. The mask to be corrected is also the segmented image predicted and output by the prediction device.

[0112] Step 406 , generating M interaction point coordinates by random sampling and obtaining an initialized mask according to the focused image and the mask to be corrected; M is a positive integer not less than 1.

[0113] In this step, random sampling is used to generate M interaction point coordinates. It is important to note that the M interaction point coordinates generated by random sampling in this step are usually generated at once. M cannot be less than 1 and must be a positive integer. The randomly sampled interaction point coordinates are generated at random locations in the focused image based on the true value mask.

[0114] Step 408: After performing N corrective sampling segmentation on the initialized mask according to the true value mask, return to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches the set number of tasks, and calculate the loss function of the last corrective sampling segmentation; N is a positive integer not less than 1.

[0115] In this step, the correction sampling usually places the click near the center of the largest incorrectly segmented area to correct the initial mask, thereby allowing the trained image segmentation model to quickly and accurately complete the image segmentation in the early stage. The number N cannot be less than 1 and must be a positive integer. In addition, the loss function value of the last correction sampling segmentation is calculated to provide reliable data support for subsequent iterations of the image segmentation model. The expression for the loss function calculation is:

[0116]

[0117] where p i,j Represents the prediction confidence of the interaction point coordinate position (i, j).

[0118] Step 410: Update the parameters of the image segmentation model according to the loss function.

[0119] In this embodiment, by setting the coding radius to the third coding radius and the minimum area of ​​the true value mask to Z pixels, it is ensured that the network of the prediction device can more boldly amplify the influence of the coordinates of the subsequent randomly sampled interaction points, thereby achieving the image segmentation model to quickly output high-precision images.

[0120] In one embodiment, the process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method further includes an incremental training phase, which includes:

[0121] After the convergence speed priority training phase is completed, the coding radius is set to the dynamic coding radius, and the minimum area of ​​the true value mask is S pixels; the dynamic coding radius includes the first coding radius, the second coding radius, or the third coding radius, and S is a positive integer greater than 1 and less than Z;

[0122] Extract the focused image and the mask to be corrected;

[0123] Generate M interaction point coordinates at one time through random sampling, obtain an initialization mask based on the focused image and the mask to be corrected, and perform M random sampling and segmentation based on the initialization mask to obtain the first loss function; M is a positive integer not less than 1;

[0124] After performing N corrective sampling segmentation on the initialized mask according to the true value mask, return to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches the set number of tasks, and calculate the second loss function during the last corrective sampling segmentation; N is a positive integer not less than 1;

[0125] Update the parameters of the image segmentation model according to the first loss function and the second loss function.

[0126] The difference between this embodiment and the previous embodiment is that, first, the implementation process of this embodiment has completed the convergence speed priority training stage, which can ensure that the image segmentation model can quickly and accurately complete the image segmentation; secondly, in this embodiment, the coding radius is set to a dynamic coding radius, and the minimum area of ​​the true value mask is set to S pixels, where S is a positive integer greater than 1 and less than Z. This can ensure the user's subjectivity and will not allow the image segmentation model to excessively dominate the image segmentation, thereby ensuring the stability of the image output by the image segmentation model. It is worth noting that since each iteration of the image segmentation model only obtains one interaction point coordinate, it can only generate one coding radius, which is any one of the first coding radius, the second coding radius, and the third coding radius. In addition, in this embodiment, the first loss function Loss will be obtained after random sampling and segmentation. RoC , and the first loss function Loss RoC Combined with the second loss function Loss CG , and update the parameters of the image segmentation model together. The first loss function and the second loss function are used to optimize the convergence speed and stability of the image segmentation model respectively, so as to ensure that the trained image segmentation model can quickly and accurately complete the image segmentation. The specific loss function calculation expression is:

[0127] Loss total =Loss RoC +λ·Loss CG

[0128] Among them, Loss total is the total loss function, and λ is a hyperparameter used to balance the scaling factor between accuracy and stability.

[0129] In one embodiment, the segmenter is HRNet32-OCR.

[0130] It is known that there are many types of segmenters for implementing image segmentation, such as the Transformer-type segformer, the CNN-type U-Net++, and the HRNet32-OCR used in the present invention. This embodiment uses HRNet32-OCR as the segmenter of the image segmentation model of the present invention because HRNet32-OCR has the advantages of being mature and stable and can efficiently and stably complete the image segmentation task.

[0131] It should be understood that although Figure 1 、 Figure 2 、 Figure 3 as well as Figure 4The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0132] In one embodiment, Figure 5 As shown, a remote sensing image interactive segmentation device 100 is provided, comprising a data acquisition module 11, a dynamic radius encoding module 12, a preprocessing module 13, a semantic extraction module 14, an information fusion module 15, an incremental learning module 16 and an inverse focusing module 17, wherein: the data acquisition module 11 is used to obtain the mask to be corrected, the coordinates of the interaction points and the target image collected by the remote sensing device; the dynamic radius encoding module 12 is used to perform dynamic radius encoding according to the mask to be corrected, the coordinates of the interaction points and the target image to obtain the interactive feature map and the focused image; the focused image is obtained by focusing and cropping the target image; the preprocessing module 13 is used to adjust the interactive feature map The channels and sizes of the feature map are used to obtain the interactive information; the semantic extraction module 14 is used to extract the semantic information of the focused image using the first layer of the segmenter; the information fusion module 15 is used to fuse the interactive information with the semantic information to obtain the comprehensive information; the incremental learning module 16 is used to iteratively optimize the segmenter and the adjustment process based on the interactive feature map and the focused image using the incremental learning method; during the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain the initial predicted image, and the other layers of the segmenter are the number of segmenter layers except the first layer of the segmenter; the inverse focusing module 17 is used to perform an inverse focusing transformation on the initial predicted image to obtain the final segmented image.

[0133] The aforementioned audio classification device 100 based on semi-supervised incremental learning, through the collaboration of various modules, first obtains the mask to be corrected, the coordinates of the interaction points, and the target image captured by the remote sensing device to provide data information for the current image segmentation. Then, dynamic radius encoding is performed based on the obtained data information to obtain an interaction feature map and a focused image. The focused image is obtained by focusing and cropping the target image. By focusing on the important features of the target image, the computational workload of the prediction device is reduced. Dynamic radius encoding makes reasonable predictions based on the data information, allowing the device to appropriately amplify the influence of the interaction point coordinates in the early stages, enabling the device to output a more accurate image more quickly. In the middle and later stages, the influence of the interaction point coordinates is reduced to avoid a decrease in the device output image accuracy due to overshoot, thereby ensuring the stability of the device output image accuracy. After obtaining the interaction feature map, it is necessary to adjust the channels and size of the interaction feature map and extract semantic information of the focused image using the first layer of the segmenter to maintain the uniformity of the information format and ensure the smoothness of the subsequent information fusion process. Finally, the integration of the interaction information and semantic information ensures the integrity of the data information. In addition, a special incremental learning method is designed to iteratively optimize the segmenter and adjustment process. Interaction information is used to ensure that the entire image segmentation model is compatible with dynamic radius encoding, allowing the image segmentation model to quickly identify and locate objects in the early stages. In the middle and late stages, the coordinates of the interaction points are pre-judged and adjusted based on actual conditions to avoid excessive autonomy in the image segmentation model, which can lead to reduced output image accuracy. This ensures better stability for the image segmentation model. After completing the above steps, inverse focusing is performed to restore the originally focused transformed image to the target image format, thereby quickly and accurately outputting the final segmented image consistent with the target image format while maintaining stability.

[0134] In one embodiment, each module of the above-mentioned remote sensing image interactive segmentation device 100 can also be used to implement corresponding processing steps of other embodiments of the above-mentioned remote sensing image interactive segmentation method.

[0135] For the specific limitations of the interactive remote sensing image segmentation device 100, please refer to the corresponding limitations of the interactive remote sensing image segmentation method above and will not be repeated here. The various modules in the interactive remote sensing image segmentation device 100 can be implemented in whole or in part through software, hardware, or a combination thereof. The aforementioned modules can be embedded in or independent of a device with specific data processing functions in hardware form, or can be stored in the memory of the aforementioned device in software form, so that the processor can call and execute the corresponding operations of the aforementioned modules. The aforementioned device can be, but is not limited to, various types of portable data analysis and processing devices available in the art.

[0136] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following processing steps are implemented:

[0137] Obtain the mask to be corrected, the coordinates of the interaction points and the target image collected by the remote sensing equipment; perform dynamic radius encoding according to the mask to be corrected, the coordinates of the interaction points and the target image to obtain the interaction feature map and the focused image; the focused image is obtained by focusing and cropping the target image; adjust the channel and size of the interaction feature map to obtain the interaction information; use the first layer of the segmenter to extract the semantic information of the focused image; fuse the interaction information with the semantic information to obtain the comprehensive information; based on the interaction feature map and the focused image, use the incremental learning method to iteratively optimize the segmenter and the adjustment process; in the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain the initial predicted image, and the other layers of the segmenter are the number of segmenter layers except the first layer of the segmenter; perform an inverse focusing transformation on the initial predicted image to obtain the final segmented image.

[0138] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer device also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of the specific data processing equipment in different application scenarios. This specification will not list them one by one in detail.

[0139] In one embodiment, when the processor executes the computer program, it can also implement the additional steps or sub-steps in each embodiment of the above-mentioned remote sensing image interactive segmentation method.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following processing steps are implemented:

[0141] Obtain the mask to be corrected, the coordinates of the interaction points and the target image collected by the remote sensing equipment; perform dynamic radius encoding according to the mask to be corrected, the coordinates of the interaction points and the target image to obtain the interaction feature map and the focused image; the focused image is obtained by focusing and cropping the target image; adjust the channel and size of the interaction feature map to obtain the interaction information; use the first layer of the segmenter to extract the semantic information of the focused image; fuse the interaction information with the semantic information to obtain the comprehensive information; based on the interaction feature map and the focused image, use the incremental learning method to iteratively optimize the segmenter and the adjustment process; in the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain the initial predicted image, and the other layers of the segmenter are the number of segmenter layers except the first layer of the segmenter; perform an inverse focusing transformation on the initial predicted image to obtain the final segmented image.

[0142] In one embodiment, when the computer program is executed by a processor, it can also implement the additional steps or sub-steps in each embodiment of the above-mentioned remote sensing image interactive segmentation method.

[0143] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, referred to as RDRAM) and interface dynamic random access memory (DRDRAM).

[0144] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A remote sensing image interactive segmentation method, characterized in that: The method comprises: Obtain the mask to be corrected, the coordinates of the interaction points, and the target image collected by the remote sensing equipment; Dynamic radius encoding is performed according to the mask to be corrected, the coordinates of the interaction point and the target image to obtain an interaction feature map and a focused image; the focused image is obtained by focusing and cropping the target image; Adjusting the channel and size of the interaction feature map to obtain interaction information; extracting semantic information of the focused image using a first layer of a segmenter; Fusing the interactive information with the semantic information to obtain comprehensive information; Iteratively optimizing the segmenter and the adjustment process based on the interaction feature map and the focused image using an incremental learning method; during the iterative optimization process, other layers of the segmenter cut the comprehensive information to obtain an initial predicted image, the other layers of the segmenter being the number of segments of the segmenter excluding the first layer of the segmenter; Performing an inverse focusing transformation on the initial predicted image to obtain a final segmented image; The process of performing dynamic radius encoding to obtain an interactive feature map includes: The coordinates of the interaction point are combined with the coordinates of the historical interaction points to obtain the coordinates of the current interaction point; focusing the mask to be corrected to obtain a focused mask to be corrected; Focusing the current interaction point coordinates to obtain focused interaction point coordinates; the focused interaction point coordinates include positive focused interaction point coordinates and negative focused interaction point coordinates; Correcting the focus mask to be corrected according to the focus interaction point coordinates to obtain an interactive segmentation feature map; the interactive segmentation feature map includes a positive interactive segmentation feature map and a negative interactive segmentation feature map; The focused mask to be corrected and the interactive segmentation feature map are spliced ​​to obtain an interactive feature map.

2. The interactive segmentation method for remote sensing images according to claim 1, characterized in that: The step of correcting the focus mask to be corrected according to the focus interaction point coordinates to obtain an interactive segmentation feature map includes: extracting a mask edge of the focus mask to be corrected using a Laplace filter; Performing distance transformation on the coordinates of the focused interaction point to generate a distance feature map; Calculating the minimum distance between the focus interaction point coordinates and the mask edge according to the distance feature map and the mask edge to obtain a coding radius; the focus interaction point coordinates include positive focus interaction point coordinates and negative focus interaction point coordinates; The positive focus interaction point coordinates and the negative focus interaction point coordinates are encoded respectively according to the encoding radius to obtain the positive interaction segmentation feature map and the negative interaction segmentation feature map.

3. The interactive segmentation method for remote sensing images according to claim 2, characterized in that: The coding radius includes a first coding radius, a second coding radius and a third coding radius. The distance mapping interval of the first coding radius is (0, 3], the distance mapping interval of the second coding radius is (3, 10], and the distance mapping interval of the third coding radius is (10, +∞).

4. The interactive segmentation method for remote sensing images according to claim 3, characterized in that: The process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method includes a convergence speed priority training phase, which is: The coding radius is set to the third coding radius, and the minimum area of ​​the true value mask is Z pixels; Z is a positive integer not less than 1; extracting the focused image and the mask to be corrected; Generate M interaction point coordinates by random sampling and obtain an initialization mask according to the focused image and the mask to be corrected; M is a positive integer not less than 1; After performing N corrected sampling segmentation on the initialized mask according to the true value mask, returning to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches the set number of tasks, and calculating the loss function of the last corrected sampling segmentation; N is a positive integer not less than 1; Update the parameters of the image segmentation model according to the loss function.

5. The interactive segmentation method for remote sensing images according to claim 4, characterized in that: The process of iteratively optimizing the segmenter and the adjustment process using the incremental learning method also includes an incremental training phase, and the incremental training phase includes: After the convergence speed priority training phase is completed, the coding radius is set to a dynamic coding radius, and the minimum area of ​​the true value mask is S pixels; the dynamic coding radius includes the first coding radius, the second coding radius, or the third coding radius, and S is a positive integer greater than 1 and less than Z; extracting the focused image and the mask to be corrected; Generate M interaction point coordinates at one time by random sampling, obtain an initialization mask based on the focused image and the mask to be corrected, and perform M random sampling and segmentation based on the initialization mask to obtain a first loss function; M is a positive integer not less than 1; After performing N corrective sampling segmentation on the initialized mask according to the true value mask, returning to the step of extracting the focused image and the mask to be corrected until the number of segmentation tasks reaches a set number of tasks, and obtaining a second loss function by calculation during the last corrective sampling segmentation; N is a positive integer not less than 1; Update the parameters of the image segmentation model according to the first loss function and the second loss function.

6. The interactive segmentation method for remote sensing images according to claim 1, characterized in that: The segmenter is HRNet32-OCR.

7. A remote sensing image interactive segmentation device, characterized in that: include: Data acquisition module: used to obtain the mask to be corrected, the coordinates of the interaction points, and the target image collected by the remote sensing equipment; Dynamic radius encoding module: used to perform dynamic radius encoding according to the mask to be corrected, the coordinates of the interaction point and the target image to obtain the interaction feature map and the focused image; The focused image is obtained by focusing and cropping the target image; Preprocessing module: used to adjust the channel and size of the interaction feature map to obtain interaction information; Semantic extraction module: for extracting semantic information of the focused image using the first layer of the segmentor; Information fusion module: used to fuse the interactive information with the semantic information to obtain comprehensive information; Incremental learning module: configured to iteratively optimize the segmenter and the adjustment process using an incremental learning method based on the interactive feature map and the focused image; During the iterative optimization process, the other layers of the segmenter cut the comprehensive information to obtain an initial predicted image, and the other layers of the segmenter are the number of segments except the first layer of the segmenter; Inverse focusing module: used for performing inverse focusing transformation on the initial predicted image to obtain the final segmented image; The process of performing dynamic radius encoding to obtain an interactive feature map includes: The coordinates of the interaction point are combined with the coordinates of the historical interaction points to obtain the coordinates of the current interaction point; focusing the mask to be corrected to obtain a focused mask to be corrected; Focusing the current interaction point coordinates to obtain focused interaction point coordinates; the focused interaction point coordinates include positive focused interaction point coordinates and negative focused interaction point coordinates; Correcting the focus mask to be corrected according to the focus interaction point coordinates to obtain an interactive segmentation feature map; the interactive segmentation feature map includes a positive interactive segmentation feature map and a negative interactive segmentation feature map; The focused mask to be corrected and the interactive segmentation feature map are spliced ​​to obtain an interactive feature map.

8. A computer device comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the steps of the remote sensing image interactive segmentation method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the remote sensing image interactive segmentation method according to any one of claims 1 to 6 are implemented.

Images