Model training method, image segmentation method, terminal device and computer medium

Abstract

The present application discloses a model training method, an image segmentation method, a terminal device and a computer medium. The model training method performs filtering processing on an original training image set to amplify the contrast of the original training image set, and increases the number of small category samples by resampling the original training image set, thereby improving the training effect of the segmentation model; the image segmentation method uses a first segmentation model and a second segmentation model to predict and sum the remote sensing image to be detected, thereby improving the accuracy of image segmentation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, an image segmentation method, a terminal device, and a computer medium. Background Art

[0002] Remote sensing refers to the use of sensors carried by aircraft to survey targets from a distance, primarily through long-range imaging. With the continuous improvement and advancement of remote sensing and deep learning technologies, as well as the continuous development of hardware equipment, the application of remote sensing images has become inextricably linked to our daily lives. Remote sensing image interpretation is widely used in fields such as geographic surveys, land and resources investigations, urban development, agricultural production, and forest protection, and has gradually become a crucial means of understanding the Earth's surface.

[0003] Remote sensing images are widely used in agriculture, enabling crop yield estimation, crop area statistics, and crop health monitoring. Existing methods for calculating crop area based on remote sensing images typically use segmentation models to segment pixels from different crop regions and then perform crop area statistics. However, existing segmentation methods typically use RGB three-channel remote sensing images for segmentation, resulting in low segmentation accuracy and inaccurate segmented crop areas. Summary of the Invention

[0004] To solve the above technical problems, the present application provides a model training method, an image segmentation method, a terminal device and a computer medium.

[0005] To solve the above problems, the present application provides a first technical solution: providing a model training method, applied to a segmentation model, including: acquiring a remote sensing image, cutting the remote sensing image into several preset-size images, and using the preset-size images as an original training image set; filtering the original training image set; resampling the filtered original training image set, and adding the resampled images to the training image set; training the segmentation model based on the training image set.

[0006] The original training image set includes a first near-infrared channel image and a three-channel image; the step of filtering the original training image set includes: performing Fourier transform on the first near-infrared channel image to obtain a frequency domain image of the near-infrared channel; filtering the frequency domain image to filter out low-frequency information of the frequency domain image; performing inverse Fourier transform on the filtered frequency domain image to obtain a second near-infrared channel image; superimposing the second near-infrared channel image with the three-channel image, and adding the superimposed image to the original training image set.

[0007] Among them, the step of resampling the original training image set after filtering and adding the resampled image to the training image set includes: cropping a first image from the remote sensing image, the size of the first image being within a preset pixel area; scaling the first image to a preset size, and adding the scaled first image to the training image set.

[0008] Among them, the step of resampling the original training image set after filtering and adding the resampled images to the training image set includes: cropping a number of first images from the remote sensing image, the size of the first images being within a preset pixel area, and the first images serving as the first training image set; obtaining a number of second images from the original training image set as the second training image set; selecting a preset number of images from the first training image set and the second training image set, and flipping and / or cropping the selected images; splicing the processed preset number of images, and scaling the spliced ​​images to the preset size, so as to add the scaled images to the training image set.

[0009] The preset number includes 4 and / or 6, and the ratio of the first training image set to the second training image set is 4:6.

[0010] Wherein, the preset number includes 4 and 6; after the step of splicing the processed preset number of images and scaling the spliced ​​images to the preset size to add the scaled images to the training image set, the model training method also includes: when the preset number is 4, scaling the spliced ​​images to the preset size and serving as the third training image set; when the preset number is 6, scaling the spliced ​​images to the preset size and serving as the fourth training image set; the ratio of the first training image set, the third training image set and the fourth training image set is 4:2:1.

[0011] Among them, the step of training the segmentation model based on the training image set includes: obtaining the segmentation label of the training image set; obtaining a smoothing weight based on the ratio of the number of training images of each segmentation label to the total number of training image sets; calculating the loss function of each segmentation label based on the smoothing weight, so as to train the segmentation model using the loss function.

[0012] To solve the above problem, the present application provides a second technical solution: providing an image segmentation method, applied to a segmentation model, the segmentation model is obtained using the training method as described above, the segmentation model includes a first segmentation model and a second segmentation model, the pixel size of the training image set of the first segmentation model is a first size, the pixel size of the training image set of the second segmentation model is a second size, and the second size is at least twice the first size; the image segmentation method includes: inputting the remote sensing image to be detected into the first segmentation model to obtain a first prediction result of the remote sensing image to be detected; inputting the remote sensing image to be detected into the second segmentation model to obtain a second prediction result of the remote sensing image to be detected, wherein the first prediction result and the second prediction result are prediction results relative to the same segmented area of ​​the remote sensing image to be detected; and summing the first prediction result and the second prediction result according to preset weights to obtain a classification result of the remote sensing image to be detected.

[0013] To solve the above problems, the present application provides a third technical solution: a terminal device is provided, comprising a processor and a memory connected to the processor, wherein a program image is stored in the memory, and the processor calls the program image stored in the memory to execute any of the methods described above.

[0014] To solve the above problem, the present application provides a fourth technical solution: providing a computer-readable storage medium storing program instructions, wherein the program instructions are executed to implement the above method.

[0015] The present application provides a model training method, an image segmentation method, a terminal device and a computer medium. The model training method performs filtering processing on the original training image set to amplify the contrast of the original training image set, and increases the number of small category samples by resampling the original training image set, thereby improving the training effect of the segmentation model; the image segmentation method uses a first segmentation model and a second segmentation model to predict and add the remote sensing image to be detected, thereby improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0017] Figure 1 This is a flowchart of the first embodiment of the model training method provided by this application;

[0018] Figure 2 This is a schematic diagram of an operation of an embodiment of a remote sensing image provided by this application;

[0019] Figure 3 This is a flow chart of the second embodiment of the model training method provided by this application;

[0020] Figure 4 This is a flowchart of the third embodiment of the model training method provided by this application;

[0021] Figure 5 This is an operational diagram of an embodiment of the resampling process provided by the present application;

[0022] Figure 6 This is a flowchart of the first embodiment of the image segmentation method provided by the present application;

[0023] Figure 7 This is an operational diagram of an embodiment of the model integration provided by this application;

[0024] Figure 8 is an operational schematic diagram of an embodiment of morphological processing provided by this application;

[0025] Figure 9 is a flowchart of a second embodiment of the image segmentation method provided by this application;

[0026] Figure 10 This is a schematic diagram of an operation of an embodiment of image cutting provided by the present application;

[0027] Figure 11 This is a schematic diagram of a framework of an embodiment of a terminal device provided by the present application;

[0028] Figure 12 It is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0032] In agriculture, remote sensing imagery possesses unique characteristics, enabling applications such as crop yield estimation, crop area statistics, and crop health monitoring. The combination of remote sensing imagery and artificial intelligence (AI) can significantly empower agriculture and elevate its development to a new level.

[0033] Existing methods for calculating crop area based on remote sensing images often construct an instance segmentation network model. Using the segmentation network, they can segment various irregular crop regions, segmenting pixels from different crop regions and then calculating crop area statistics. Furthermore, these methods often utilize RGB (red, green, blue) three-channel remote sensing images. Existing methods for calculating crop area using near-infrared remote sensing images also treat near-infrared images as simple channel images. Due to the inconsistency of crop areas in the agricultural sector, data imbalance across different types is severe. Existing methods for calculating crop area using near-infrared remote sensing images rely on models and labeled data without carefully analyzing the characteristics of crop remote sensing images. Consequently, these methods result in segmented crop areas that are generally ineffective.

[0034] In view of this, the present application proposes a model training method, an image segmentation method, a terminal device, and a computer medium. The model training method and the image segmentation method are applied to a segmentation model, the model training method is used to train the segmentation model, and the image segmentation method is used to segment remote sensing images and calculate the area of ​​crop regions through the segmentation model. The image segmentation method of the present application fully utilizes the data of the near-infrared channel at the data layer to improve the accuracy of segmentation, and uses resampling to reduce data imbalance in the data processing stage. In the model stage, multi-model fusion and TTA (test time augmentation) are used to improve the model prediction accuracy, thereby greatly improving the accuracy of statistical crop area statistics in agricultural remote sensing images.

[0035] The model training method and image segmentation method of the present application can be applied to terminal devices and the field of computer technology. Specifically, the segmentation model can be used to perform segmentation prediction on remote sensing images and calculate the area of ​​crop regions based on the prediction results. The calculated crop region area can be used in business scenarios such as e-commerce, electronic payment, securities, electronic banking, tax transactions, credit cards, online shopping, and insurance. For example, it can be used to assist in processing bank agricultural loans and other businesses. The purpose of the terminal device is not specifically limited here.

[0036] The terminal device of the present application may be a server or a system comprising a server and a local terminal. Accordingly, the various parts of the electronic terminal, such as the various units, subunits, modules, and submodules, may be all disposed in the server or separately disposed in the server and the local terminal.

[0037] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the model training method and / or image segmentation method of the embodiments of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0038] See Figure 1-2 , Figure 1 This is a flow chart of the first embodiment of the model training method provided by this application. Figure 2 This is a schematic diagram of an operation of an embodiment of the remote sensing image provided by this application. Figure 1 As shown, the model training method proposed in this embodiment is applied to the segmentation model, and the specific steps are as follows:

[0039] Step S11: Acquire a remote sensing image, and cut the remote sensing image into a plurality of images of preset sizes, so as to use the images of preset sizes as an original training image set.

[0040] Specifically, a segmentation model is a model for segmenting an image. The segmentation model is used to predict what object or type the pixels in each frame of the input image belong to. The segmentation model includes but is not limited to the Transformer segmentation network model produced under the encoding-decoding framework, attention mechanism and other architectures. The segmentation model can also be a detection model based on a visual recognition algorithm. The application framework and principle of the segmentation model are not specifically limited here.

[0041] Remote sensing images are images of targets detected by various sensors. Segmentation models can analyze, infer, and judge the various target identification features provided by remote sensing images to achieve the purpose of identifying targets or phenomena. Remote sensing images can be acquired through remote sensing photography, data entry, and / or online queries. Remote sensing images can be multi-channel images. For example, remote sensing technology can be used to capture images of multiple bands or spectral segments. Remote sensing images can include RGB three-channel images and near-infrared channel images.

[0042] Because remote sensing images are often used for long-distance surveys and have large areas, this embodiment segments the remote sensing images into multiple original training images of preset sizes to form an original training image set. Semantic segmentation of each pixel in each frame is then performed to predict the type of the original training images. Preset sizes include, but are not limited to, regular sizes such as 512*512, 1024*1024, 1536*1536, 2048*2048, and 2248*2248. The choice of preset size can be determined based on the original size of the remote sensing image, the intended segmentation purpose, the type of segmentation model, and other factors, and is not specifically limited here.

[0043] Among them, Figure 2 As shown, when cutting a remote sensing image, a dotted frame of a preset size can be generated on the remote sensing image, and the remote sensing image can be cut according to the position of the dotted frame. Specifically, the cutting method can start from any direction, such as the upper left corner, the upper right corner, the lower left corner, or the lower right corner. When cutting in the opposite direction of the starting direction, if the edge area of ​​the remote sensing image does not meet the preset size, the edge area of ​​the remote sensing image can be padded with pixels to ensure that the cut image of the edge area meets the preset size and that the original training image set includes images of all areas of the remote sensing image, thereby ensuring the accuracy of the segmentation model results.

[0044] Step S12: Filter the original training image set.

[0045] The original training image set includes multiple original training images of preset sizes. In order to improve the segmentation model's ability to capture the gradient transformation information of the original training images, the original training image set can be filtered before segmentation prediction is performed on the original training images. The filtering process includes but is not limited to low-frequency filtering, high-pass filtering, etc.

[0046] Step S13: resampling the filtered original training image set, and adding the resampled images to the training image set.

[0047] After filtering the images in the original training image set, the low-frequency information of the images is filtered out, making the segmentation model more sensitive to the gradient transformation information of the original training images. The purpose of the segmentation model is different. When there are many objects or types in the remote sensing image, and the sample size difference between each object or type is large, in order to improve the training effect of the segmentation model on a small number of samples, the original training image set after filtering can be resampled. Specifically, the resampling process includes random angle flipping, random size cropping, Gaussian blurring, normalization processing, etc. Through resampling, the number of training images of small categories of targets can be increased to reduce the category imbalance of remote sensing images.

[0048] Step S14: training the segmentation model based on the training image set.

[0049] After filtering and resampling, a training image set is obtained, and the segmentation model is trained based on the training image set, wherein the training image set includes the original training image and the training image added after the resampling process.

[0050] Therefore, in an embodiment of the present application, a model training method includes acquiring a remote sensing image, cutting the remote sensing image into a plurality of images of preset sizes, and using the images of preset sizes as an original training image set; filtering the original training image set; resampling the filtered original training image set and adding the resampled images to the training image set; and training a segmentation model based on the training image set. This model training method improves the training effect of the segmentation model by filtering the original training image set to amplify the contrast of the original training image set and by resampling the original training image set to increase the number of small category samples.

[0051] In one embodiment, the original training image set includes a first near-infrared channel image and a three-channel image, so that the original training image set has the advantages of both near-infrared and visible light band information, the image is rich in color and has relatively rich geological information and surface environment information, so that the segmentation model can use the rich image information to detect objects and types, thereby improving the accuracy of model training.

[0052] For details, see Figure 3 , Figure 3 This is a flow chart of the second embodiment of the model training method provided by this application. Figure 3 As shown, in this embodiment, step S12 further includes the following steps:

[0053] Step S21: performing Fourier transform on the first near-infrared channel image to obtain a frequency domain image of the near-infrared channel.

[0054] Specifically, the first near-infrared channel image is a remote sensing image taken in the near-infrared band of 0.76-0.90 microns, and the three-channel image is a remote sensing image loaded and rendered through the red, green, and blue channels. The original training image set can be understood as including channel images of the near-infrared band and synthetic images of the three bands rendered through red, green, and blue, that is, the original training image set can be understood as including four remote sensing image bands.

[0055] The near-infrared band is located in the high-reflection area of ​​plants. Using the near-infrared band for remote sensing image acquisition can reflect a large amount of plant information, so the near-infrared channel map can be used for plant identification and classification. In the agricultural field, for example, when calculating the area of ​​agricultural crops in remote sensing data through a segmentation model, the detection accuracy of the segmentation model can be increased by using the near-infrared channel image. Furthermore, when filtering the original training image set, in order to further amplify the effective area of ​​the near-infrared channel image, the first near-infrared channel image of the original training image set can be subjected to a fast Fourier transform (FFT) to convert the first near-infrared channel image into a Fourier frequency domain image. The specific formula is as follows:

[0056]

[0057] Among them, F(ω) is the image function of f(t), and f(t) is the original image function of F(ω).

[0058] Step S22: performing filtering processing on the frequency domain image to filter out low-frequency information of the frequency domain image.

[0059] After obtaining the frequency domain image of the first near-infrared channel image, the frequency domain image is subjected to high-pass filtering to filter out the low-frequency information of the frequency domain image. Specifically, the frequency domain image can be filtered using a first-order high-pass filter, and the specific formula is as follows:

[0060] Y(n)=αX(n)+(1-α)Y(n-1);

[0061] Where α is the filter coefficient; X(n) is the current sample value; Y(n-1) is the previous filter output value; and Y(n) is the current filter output value. The first-order high-pass filter method uses a weighted sum of the current sample value and the previous filter output value to obtain an effective filter value, ensuring that the output has a feedback effect on the input.

[0062] Furthermore, in order to ensure that the parameters of the first-order high-pass filter are applicable to crops, when using remote sensing images to train the segmentation model, the filter parameters can be trained simultaneously using a support vector machine (SVM) with a Gaussian kernel function. Alternatively, the segmentation model can be fused with the support vector machine with a Gaussian kernel function so that the filter parameters can be updated simultaneously when the segmentation model is trained and updated according to the loss function, thereby improving the reliability of the filtering process.

[0063] Step S23: performing an inverse Fourier transform on the filtered frequency domain image to obtain a second near-infrared channel image.

[0064] After the frequency domain image filtering process is completed, the filtered frequency domain image is subjected to an inverse Fourier transform to convert the filtered frequency domain image into a near-infrared channel image to obtain a second near-infrared channel image. It can be understood that the first near-infrared channel image is an image that has not been filtered, and the second near-infrared channel image is an image that has been filtered. The formula for the inverse Fourier transform is as follows:

[0065]

[0066] Among them, F(ω) is the image function of f(t), and f(t) is the original image function of F(ω).

[0067] Step S24: superimposing the second near-infrared channel image and the three-channel image, and adding the superimposed image to the original training image set.

[0068] After an inverse Fourier transform, the frequency domain image of the near-infrared channel is converted back into a spatial domain image, the second near-infrared channel image. This second near-infrared channel image is filtered to remove low-frequency information while retaining information with more pronounced gradient transformations. This increases the contrast of the second near-infrared channel image, making it easier for the segmentation model to analyze the plant information contained in the second near-infrared channel image.

[0069] After acquiring the second near-infrared channel image, the second near-infrared channel image is superimposed on the original three-channel image, and the superimposed image is added to the original training image set. Specifically, the superposition method can be sequential superposition, that is, the second near-infrared channel image is superimposed on the channels of the three-channel image. For example, the channels of the three-channel image are in the order of R, G, and B channels, and the near-infrared channel is denoted as N. That is, the superimposed image is a four-channel image of RGBN.

[0070] Therefore, in an embodiment of the present application, low-frequency filtering is performed on the near-infrared channel image, so that the effective information and effective area of ​​the near-infrared image are further amplified, thereby improving the accuracy of model segmentation.

[0071] In one embodiment, step S13 further includes the following steps: cropping a first image from the remote sensing image, where the size of the first image is within a preset pixel area; scaling the first image to a preset size, and adding the scaled first image to the training image set.

[0072] Specifically, in this embodiment, the resampling process is performed by randomly cropping any region of the remote sensing image to crop a first image from the remote sensing image. The size of the first image is within a preset pixel region. The specific range of the preset pixel region is related to factors such as the preset size of the original training image set, the training duration, and the training requirements. For example, when the size of the original training image set is 512*512, the preset pixel region can be set within a range of 384 to 1536 pixels. The range of the preset pixel region can also be set to a larger range, which is not specifically limited here.

[0073] After obtaining the randomly cropped first image, the first image is scaled to a preset size, and the scaled first image is added to the training image set to increase the number of training images.

[0074] In one embodiment, see Figure 4-5 , Figure 4 This is a flow chart of the third embodiment of the model training method provided by this application. Figure 5 This is an operational diagram of an embodiment of the resampling process provided by this application. Figure 4 As shown, in this embodiment, step S13 further includes the following steps:

[0075] Step S31: a plurality of first images are cropped from the remote sensing image. The sizes of the first images are within a preset pixel area. The first images are used as a first training image set.

[0076] Similarly, similar to the resampling process in the previous embodiment, random cropping is performed on any region of the remote sensing image to crop a first image from the remote sensing image. The size of the first image is within the preset pixel region. The first image cropped from the remote sensing image serves as the first training image set.

[0077] Step S32: Acquire a number of second images from the original training image set as a second training image set.

[0078] A plurality of second images are obtained from the images in the original training image set, where the second images are remote sensing image regions cropped according to a preset size, and the plurality of second images are used as a second training image set.

[0079] Step S33: selecting a preset number of images from the first training image set and the second training image set, and performing a flipping process and / or a cropping process on the selected images.

[0080] A preset number of images are selected from the first training image set and the second training image set. It can be understood that the selected images may all be first images, all be second images, or at least one first image and at least one second image.

[0081] The selected image is flipped and / or cropped. For example, in one embodiment, the selected image can be flipped at a random angle, and the flipping angle includes 0°-360°; in another embodiment, the selected image can also be cropped at a random size. Specifically, the cropping size should be greater than 0, the cropping size should be less than or equal to the maximum value of the preset pixel area of ​​the first image, or the cropping size should be less than or equal to the preset size; in other embodiments, the selected image can be flipped at a random angle and then the flipped image can be cropped at a random size, or the selected image can be cropped at a random size and then the cropped image can be flipped at a random angle.

[0082] Flipping at random angles and / or cropping at random sizes can enhance the training images to help the segmentation model extract and learn relevant features of the training images in a manner that is not affected by position, lighting, etc., thereby improving the accuracy of the segmentation model.

[0083] Step S34: stitching a preset number of processed images, and scaling the stitched images to a preset size, so as to add the scaled images to the training image set.

[0084] After flipping and / or cropping the selected images, a preset number of processed images are stitched together. Multiple stitching combinations are possible. For example, when the preset number is four, the stitched images may include the top left, top right, bottom left, and bottom right images, resulting in 24 possible stitching combinations. Therefore, any stitching method can be randomly selected to stitch together the preset number of processed images. After stitching, the scaled images are added to the training image set.

[0085] Furthermore, after the selected images are cropped and / or flipped, and before the preset number of images are spliced ​​together, the processed images may be scaled to a preset size. In this case, after the preset number of images are spliced ​​together, the size of the spliced ​​image is the product of the preset size and the preset number. To ensure that the size of the training image set is fixed, the spliced ​​images need to be scaled again to the preset size. For example, if the preset size is 512*512 and the preset number is 4, after the processed images are scaled to 512*512, the size of the spliced ​​images will be 1024*1024. In this case, the size needs to be scaled again to 512*512, and the scaled images will be added to the training image set.

[0086] As will be appreciated, in this embodiment, the resampling process includes cropping any region of the remote sensing image and splicing a preset number of first and second images. Both the first training image set and the spliced ​​and scaled images can serve as training images for the segmentation model. Resampling the original training image set significantly increases the number of samples in the minority category, thereby improving the training effectiveness of the segmentation model.

[0087] Optionally, the preset number includes 4 and / or 6; the ratio of the first training image set to the second training image set is 4:6.

[0088] Specifically, when the preset number is 4, 4 images are randomly selected from the first training image set and the second training image set, and the 4 images are spliced ​​to form a new training image; when the preset number is 6, 6 images are randomly selected from the first training image set and the second training image set, and the 6 images are spliced ​​to form a new training image.

[0089] In order to control the degree of training of the segmentation model for simple samples and complex samples, the ratio of the number of the first training image set and the second training image set can be restricted, that is, the weight ratio of the first image and the second image construction is set to 4:6.

[0090] Furthermore, the preset number includes 4 and 6. After step S34, the model training method further includes: when the preset number is 4, scaling the spliced ​​images to a preset size and serving as a third training image set; when the preset number is 6, scaling the spliced ​​images to a preset size and serving as a fourth training image set; and the ratio of the first training image set, the third training image set, and the fourth training image set is 4:2:1.

[0091] Specifically, when the preset numbers are 4 and 6, resampling of the preset number of 4 and resampling of the preset number of 6 are performed simultaneously. Since the training images obtained after resampling are composed of multiple images, the sample features are relatively complex. In order to ensure the authenticity of the samples and control the training degree of the segmentation model for simple samples and complex samples, the ratio of the numbers of the first training image set, the third training image set, and the fourth training image set can be restricted. In this embodiment, the ratio of the numbers of the first training image set, the third training image set, and the fourth training image set can be 4:2:1.

[0092] It can be understood that the training image set used for training in the segmentation model includes the original training image set cropped from the remote sensing image according to a preset size, the first training image set arbitrarily cropped from the remote sensing image, the third training image set that has been resampled a preset number of 4, and the fourth training image set that has been resampled a preset number of 6.

[0093] In one embodiment, step S14 further includes the following steps: obtaining segmentation labels of the training image set; obtaining smoothing weights based on the ratio of the number of training images for each segmentation label to the total number of training image sets; and calculating the loss function of each segmentation label based on the smoothing weights to train the segmentation model using the loss function.

[0094] Specifically, because remote sensing images are generally large in size, they contain many objects or types, and the sample sizes of each object or type vary greatly. For example, when using a segmentation model to calculate the area of ​​a crop area, the number of soybean crop images accounts for 0.9 of the total number of images, and the number of wheat crop images accounts for 0.01 of the total number of images. The difference between the number of soybean crops and the number of wheat crops is 90 times, resulting in sample imbalance. Therefore, to improve the training effect of the segmentation model on a small number of samples, smoothing weights can be set to correct for the multiple difference.

[0095] Specifically, the smoothing weight is calculated based on the ratio of the number of training images for each segmentation label to the total number of training images in the set. The formula is as follows:

[0096]

[0097] Among them, m represents the category of crops that need to be divided (such as wheat, corn, soybeans, rice, etc.), e is the natural base number, S i It is expressed as the ratio of the number of training images including the i-th crop to the total number of training images.

[0098] In one embodiment, the original training image set includes a first near-infrared channel image and a three-channel image. In this embodiment, the training image set is also normalized. For example, the three-channel image is normalized using the Imagenet standard, and its mean and variance are [111.45638763, 113.8965259, 112.22587782] and [25.93080005, 25.29681979, 26.92847348], respectively. The first near-infrared channel image is normalized by calculating the statistical results of the full data, and its mean and variance are 118.30 and 30.93, respectively.

[0099] In one embodiment, the above-mentioned model training method can use the AdamW optimizer or the two optimization algorithms of AdaGrad and RMSProp to obtain the model parameters of the segmentation network model, and the learning rate of the AdamW optimizer can be set to 0.00006.

[0100] The present application also proposes an image segmentation method, which is applied to a segmentation model trained using any of the above-mentioned embodiments. The segmentation model includes a first segmentation model and a second segmentation model. The pixel size of the training image set of the first segmentation model is a first size, and the pixel size of the training image set of the second segmentation model is a second size, and the second size is at least twice the first size.

[0101] See Figure 6-7 , Figure 6 is a flowchart of the first embodiment of the image segmentation method provided by this application, Figure 7 This is an operational diagram of an embodiment of the model integration provided by this application. Figure 6 As shown, in an embodiment of the present application, the image segmentation method includes:

[0102] Step S41: inputting the remote sensing image to be detected into a first segmentation model to obtain a first prediction result of the remote sensing image to be detected.

[0103] The first segmentation model and the second segmentation model are both segmentation models trained using any of the above-mentioned implementation methods. The pixel size of the training image set of the second segmentation model is at least twice that of the training image set of the first segmentation model. The second size can be a multiple of an integer such as 3 or 4 times the first size. For example, the first size is 512*512 and the second size is 1536*1536, or the first size is 1024*1024 and the second size is 2048*2048. The first size, the second size, and their multiples are not specifically limited here. It can be understood that the first segmentation model is a small-size segmentation model and the second segmentation model is a large-size segmentation model. The first segmentation model and the second segmentation model can use a segmentation network model with the same architecture.

[0104] Step S42: inputting the remote sensing image to be detected into the second segmentation model to obtain a second prediction result of the remote sensing image to be detected, wherein the first prediction result and the second prediction result are prediction results relative to the same segmentation area of ​​the remote sensing image to be detected.

[0105] Specifically, such as Figure 7As shown, the first prediction result is the prediction result obtained by the first segmentation model when analyzing several first cut images when the remote sensing image to be detected is cut according to the first size. The second prediction result is the prediction result obtained by the second segmentation model when analyzing several second cut images when the remote sensing image to be detected is cut according to the second size. When the second size is twice the first size, relative to the same segmented area of ​​the remote sensing image to be detected, when the second segmentation model analyzes the second cut images located in the same segmented area, the first segmentation model needs to analyze four first cut images located in the same segmented area in order to ensure that the first and second prediction results are prediction results for the same segmented area.

[0106] Step S43: summing the first prediction result and the second prediction result according to preset weights to obtain a classification result of the remote sensing image to be detected.

[0107] After obtaining the first prediction result and the second prediction result for the same segmented area of ​​the remote sensing image to be detected, the first prediction result and the second prediction result are summed according to the preset weight to obtain the classification result of the remote sensing image to be detected. For example, the summation formula is as follows:

[0108] result=0.6*B5+0.4*B2

[0109] Where B5 is the second prediction from the second segmentation model, B2 is the first prediction from the first segmentation model, and result is the classification result. As shown in the formula, the second segmentation model's result has a 60% influence on the classification result, while the first segmentation model has a 40% influence on the classification result.

[0110] It can be understood that when the first segmentation model and the second segmentation model perform analysis and prediction on the same segmentation area, the prediction result is the confidence that the crops in the segmentation area belong to a certain object or type. Therefore, after the first prediction result and the second prediction result are added according to the preset weights, the prediction confidence that the crops in the segmentation area belong to a certain object or type can be obtained, so as to obtain the classification result of the remote sensing image to be detected based on the prediction confidence.

[0111] In this embodiment, a large-scale segmentation model and a small-scale segmentation model are used to analyze the remote sensing image to be detected respectively, and the analysis results are weighted to obtain the final fusion result, thereby ensuring the image segmentation speed while further improving the image segmentation accuracy.

[0112] In one embodiment, after obtaining the classification result of a certain segmented region of the remote sensing image to be detected, the classification results of all segmented regions of the entire remote sensing image to be detected may be counted to obtain a final remote sensing classification map.

[0113] In one embodiment, after step S43, the remote sensing image segmentation method further includes: performing morphological processing on the remote sensing image to be detected based on the classification result to obtain a remote sensing classification map.

[0114] See Figure 8 , Figure 8 This is an operational diagram of an embodiment of morphological processing provided by this application. Figure 8 As shown in the figure, a segmentation prediction map can be obtained based on the classification results of all segmented areas. Since the segmentation prediction map is obtained by combining the prediction results of multiple segmented areas on the remote sensing image to be detected, there will be multiple holes or small connected areas in the segmentation prediction map, which will affect the visual effect of the segmentation prediction map.

[0115] Therefore, the holes can be filled and small connected areas can be removed by performing morphological processing on the segmentation prediction map. Specifically, the morphological processing methods include but are not limited to corrosion, expansion, opening and closing operations, etc. Exemplarily, in one embodiment, the segmentation prediction map can be expanded to increase the size of brighter objects in the image and reduce the size of darker objects, so that the closed area on the segmentation prediction map that is smaller than the specified pixel size can be filled. For example, the closed area can be filled with an area smaller than 3*3 pixels. In other embodiments, the segmentation prediction map can be corroded to use the minimum value of the area at each position in the image as the output grayscale value of the position to erase the connected area smaller than the specified pixel size. For example, the connected area smaller than 4*4 pixels can be erased.

[0116] like Figure 8 As shown in the figure, after morphological processing of the segmentation prediction map, the holes and small connected areas in the segmentation prediction map can be removed, so that the areas of the same type in the remote sensing classification map are coherent, the visual effect of the remote sensing classification map is better, and the user experience is improved.

[0117] See Figure 9-10 , Figure 9 is a flow chart of the second embodiment of the image segmentation method provided by this application, Figure 10 This is an operational diagram of an embodiment of image cutting provided by this application.

[0118] like Figure 9 As shown, in one embodiment, step S41 further includes the following steps:

[0119] Step S51: inputting the remote sensing image to be detected into a first segmentation model to cut the remote sensing image to be detected into a plurality of images to be detected of a first size.

[0120] Specifically, when cutting the remote sensing image to be detected, a dotted frame of a first size can be generated on the remote sensing image, and the remote sensing image can be cut according to the position of the dotted frame. Cutting can be started from any direction, such as the upper left corner, the upper right corner, the lower left corner, or the lower right corner. When cutting the edge area of ​​the remote sensing image to be detected, if it is found that the area to be cut does not meet the size of the first size, the area to be cut can be padded with pixels so that the image to be detected cut from the area to be cut meets the first size, thereby ensuring the accuracy of the segmentation model result.

[0121] Step S52: Analyze the image to be detected, the first flipped image of the image to be detected, and the second flipped image of the image to be detected respectively.

[0122] After acquiring a number of images to be detected of a first size, each image to be detected is analyzed separately, and at the same time, the upside-down flipped image of the image to be detected (i.e., the first flipped image) and the left-right flipped image of the image to be detected (i.e., the second flipped image) are analyzed, so that each image to be detected is analyzed three times.

[0123] It can be understood that the first flipped image and the second flipped image are images obtained by flipping the image to be detected in different directions. By transforming the image to be detected, the effect of test data augmentation (TTA) is achieved.

[0124] Step S53: performing arithmetic averaging on the analysis result of the image to be detected, the analysis result of the first flipped image, and the analysis result of the second flipped image, and taking the arithmetic average result as the first prediction result of the image to be detected.

[0125] Obtain the analysis result of the image to be detected, the analysis result of the first flipped image, and the analysis result of the second flipped image, perform arithmetic averaging on the results of the three analyses, and use the mean of the three analyses as the final first prediction result of the image to be detected.

[0126] In this embodiment, since the image to be detected is transformed and enhanced twice, and the average of the two transformation results and the original result is used as the final prediction result, the effect of smoothing the result and improving the generalization ability is achieved, thereby further improving the accuracy of the first segmentation model.

[0127] In one embodiment, similarly, the processing process of the remote sensing image to be detected by the second segmentation model is similar to steps S51-S53, and the cutting size (second size) of the second segmentation model is at least twice the cutting size (first size) of the first segmentation model. When the second size is twice the first size, Figure 10 shown.

[0128] See Figure 11 , Figure 11This is a schematic diagram of a framework of an embodiment of the terminal device provided by this application. Figure 11 As shown, the terminal device 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein the memory 102 stores program data, and the processor 101 calls the program data stored in the memory 102 to execute all the above methods.

[0129] Optionally, in one embodiment, the processor 101 is used to execute program data to implement the following model training method: acquiring a remote sensing image, cutting the remote sensing image into several preset-size images, and using the preset-size images as an original training image set; filtering the original training image set; resampling the filtered original training image set, and adding the resampled images to the training image set; training the segmentation model based on the training image set.

[0130] In another embodiment, the processor 101 is used to execute program data to implement the following image segmentation method: input the remote sensing image to be detected into a first segmentation model to obtain a first prediction result of the remote sensing image to be detected; input the remote sensing image to be detected into a second segmentation model to obtain a second prediction result of the remote sensing image to be detected, wherein the first prediction result and the second prediction result are prediction results relative to the same segmented area of ​​the remote sensing image to be detected; and sum the first prediction result and the second prediction result according to preset weights to obtain a classification result of the remote sensing image to be detected.

[0131] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip having signal processing capabilities. The processor 101 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0132] The memory 102 can be a memory stick, a TF card, etc., and can store all the information in the terminal device 100, including the input raw data, computer programs, intermediate operation results and final operation results, which are all stored in the memory 102. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the terminal device 100 has a memory function and can ensure normal operation. The memory 102 of the terminal device 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0133] See Figure 12 , Figure 12 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided by this application. Figure 12 As shown, the computer-readable storage medium 110 stores program instructions 111 capable of implementing all the above methods.

[0134] If the integrated units of the functional units in the various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium 110. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer-readable storage medium 110 includes a number of instructions in a program instruction 111 to enable a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile terminal such as a mobile phone, a tablet computer, a wearable device, or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media 110 (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by a computer-readable storage medium 110. These computer-readable storage media 110 can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the program instructions 111 executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer-readable storage media 110 may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the program instructions 111 stored in the computer-readable storage media 110 produce an article of manufacture including an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer-readable storage media 110 can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the program instructions 111 executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0139] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (which can be a personal computer, server, network device or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus or device), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0141] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that: Applied to segmentation models, including: Acquire a remote sensing image, and cut the remote sensing image into a plurality of images of preset sizes, so as to use the images of preset sizes as an original training image set; Performing filtering on the original training image set; Resampling the filtered original training image set, and adding the resampled images to the training image set; Training the segmentation model based on the training image set; The original training image set includes a first near-infrared channel image and a three-channel image; the step of filtering the original training image set includes: Performing Fourier transform on the first near-infrared channel image to obtain a frequency domain image of the near-infrared channel; Performing filtering on the frequency domain image to filter low-frequency information of the frequency domain image; Performing an inverse Fourier transform on the filtered frequency domain image to obtain a second near-infrared channel image; Superimposing the second near-infrared channel image with the three-channel image, and adding the superimposed image to the original training image set; The second near-infrared channel image and the three-channel image are superimposed in a sequential manner, and the second near-infrared channel image is superimposed on the channel of the three-channel image, so that the superimposed image is a four-channel image.

2. The model training method according to claim 1, characterized in that The step of resampling the filtered original training image set and adding the resampled image to the training image set comprises: cropping a first image from the remote sensing image, wherein the size of the first image is within a preset pixel area; The first image is scaled to a preset size, so as to add the scaled first image to the training image set.

3. The model training method according to claim 1, characterized in that The step of resampling the filtered original training image set and adding the resampled image to the training image set comprises: cropping a plurality of first images from the remote sensing image, wherein the sizes of the first images are within a preset pixel area, and the first images are used as a first training image set; Acquire a plurality of second images from the original training image set as a second training image set; selecting a preset number of images from the first training image set and the second training image set, and performing a flipping process and / or a cropping process on the selected images; The processed preset number of images are spliced ​​together, and the spliced ​​images are scaled to the preset size, so as to add the scaled images to the training image set.

4. The model training method according to claim 3, characterized in that The preset number includes 4 and / or 6, and the ratio of the first training image set to the second training image set is 4:

6.

5. The model training method according to claim 4, characterized in that The preset number includes 4 and 6; after the step of splicing the processed preset number of images and scaling the spliced ​​images to the preset size to add the scaled images to the training image set, the model training method further includes: When the preset number is 4, scaling the spliced ​​images to the preset size and serving as the third training image set; When the preset number is 6, scaling the spliced ​​images to the preset size and serving as the fourth training image set; The ratio of the first training image set, the third training image set, and the fourth training image set is 4:2:

1.

6. The model training method according to claim 1, characterized in that The step of training the segmentation model based on the training image set comprises: Obtaining segmentation labels for the training image set; Obtaining a smoothing weight based on a ratio of the number of training images of each segmentation label to the total number of the training image set; A loss function of each of the segmentation labels is calculated based on the smoothed weights to train the segmentation model using the loss function.

7. An image segmentation method, characterized in that: Applied to a segmentation model, the segmentation model is obtained using the model training method according to any one of claims 1 to 6, the segmentation model includes a first segmentation model and a second segmentation model, the pixel size of the training image set of the first segmentation model is a first size, the pixel size of the training image set of the second segmentation model is a second size, and the second size is at least twice the first size; The image segmentation method comprises: Inputting the remote sensing image to be detected into the first segmentation model to obtain a first prediction result of the remote sensing image to be detected; Inputting the remote sensing image to be detected into the second segmentation model to obtain a second prediction result of the remote sensing image to be detected, wherein the first prediction result and the second prediction result are prediction results relative to the same segmented area of ​​the remote sensing image to be detected; The first prediction result and the second prediction result are added according to a preset weight to obtain a classification result of the remote sensing image to be detected.

8. The image segmentation method according to claim 7, characterized in that: After the step of adding the first prediction result and the second prediction result according to a preset weight to obtain a classification result of the remote sensing image, the remote sensing image segmentation method further includes: Based on the classification result, morphological processing is performed on the remote sensing image to be detected to obtain a remote sensing classification map.

9. The image segmentation method according to claim 7, characterized in that: The step of inputting the remote sensing image to be detected into the first segmentation model to obtain a first prediction result of the remote sensing image to be detected includes: Inputting the remote sensing image to be detected into the first segmentation model to cut the remote sensing image to be detected into a plurality of images to be detected of a first size; respectively analyzing the image to be detected, the first flipped image of the image to be detected, and the second flipped image of the image to be detected; Performing arithmetic averaging on the analysis result of the image to be detected, the analysis result of the first flipped image, and the analysis result of the second flipped image, and using the arithmetic average result as the first prediction result of the image to be detected.

10. A terminal device, characterized in that: The method comprises a processor and a memory connected to the processor, wherein a program image is stored in the memory, and the processor calls the program image stored in the memory to execute the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that Program instructions are stored, and the program instructions are executed to implement the method according to any one of claims 1 to 9.

Images