Remote sensing ground feature element classification model training method and device based on pixel semantic relationship modeling, electronic equipment and medium

Abstract

The present disclosure provides a remote sensing ground feature element classification model training method and device based on pixel semantic relationship modeling, electronic equipment and medium, which can be applied to the technical field of remote sensing image processing. The method comprises: determining the semantic class frequency of each semantic class according to the number of pixel points corresponding to different semantic classes in the semantic true value map; for each pixel point, determining the class relationship value of the pixel point in the corresponding search neighborhood based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood; determining the class value of the pixel point in the corresponding search neighborhood based on the semantic class of each pixel point in the search neighborhood; inputting the ground feature sample map into the ground feature classification model to output ground feature prediction data; inputting the semantic class frequency, class relationship value, class value and ground feature prediction data corresponding to each pixel point into the remote sensing ground feature element classification loss function of the ground feature classification model to output the model loss value; and training the ground feature classification model based on the model loss value.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of remote sensing image processing, and more particularly, to a remote sensing feature element classification model training method and device based on pixel semantic relationship modeling, a storage medium and a program product. BACKGROUND

[0002] The development of deep learning and convolutional neural networks promotes the research of remote sensing semantic segmentation technology. In the training process of a remote sensing feature element classification model, the model gradient is generally realized by back propagation based on a standard cross-entropy loss function. However, any pixel in the input image contributes equally to the loss, which can cause the gradient of a semantically homogeneous region to exceed that of a semantically heterogeneous region, thus dominating the entire training process and resulting in slow model convergence.

[0003] In related technologies, a weight factor can be added for each remote sensing feature element category. However, in the process of implementing the concept of the present disclosure, the inventors have found that at least the following problem exists in related technologies. Since the number of remote sensing feature element categories is small, if a weight factor is added for each category, data set bias can affect the accuracy of category weight calculation, resulting in inaccurate feature element classification. SUMMARY

[0004] Therefore, the present disclosure provides a remote sensing feature element classification model training method and device based on pixel semantic relationship modeling, a storage medium and a program product.

[0005] One aspect of the present disclosure provides a remote sensing feature element classification model training method based on pixel semantic relationship modeling. The method includes: determining a semantic category frequency of each semantic category according to a number of pixel points corresponding to the semantic category in a semantic ground truth map, the semantic ground truth map being a ground truth map corresponding to a ground sample map used for model training; determining, for each pixel point, a category relationship value of the pixel point in a corresponding search neighborhood based on the semantic category frequency and a semantic category relationship between the pixel point and a center pixel in the search neighborhood; determining a category number value of the pixel point in the corresponding search neighborhood based on a semantic category of each pixel point in the search neighborhood; inputting the ground sample map into a ground classification model to output ground prediction data; inputting the semantic category frequency, the category relationship value, the category number value and the ground prediction data corresponding to each pixel point into a remote sensing feature element classification loss function of the ground classification model to output a model loss value of the ground classification model; and training the ground classification model based on the model loss value.

[0006] According to an embodiment of the present disclosure, the method further includes: performing inverse processing on each of the semantic category frequencies to obtain a plurality of semantic category weights; and performing logarithmic normalization processing on the semantic category weights based on a first preset experience value to obtain a normalized category weight corresponding to the pixel point.

[0007] According to an embodiment of the present disclosure, for each of the pixel points, the category relationship value of the pixel point in the corresponding search neighborhood is determined based on the semantic category frequency and the semantic category relationship between the pixel point and a center pixel in the search neighborhood, and includes: determining a search neighborhood corresponding to the pixel point based on a preset length; determining a classification difficulty coefficient of the pixel point in the search neighborhood based on the semantic category frequency; and determining the category relationship value of the pixel point in the search neighborhood based on the classification difficulty coefficient and the semantic category relationship between the pixel point and the center pixel in the search neighborhood.

[0008] According to an embodiment of the present disclosure, the category number value of each of the pixel points in the corresponding search neighborhood is determined based on the semantic category of each of the pixel points in the search neighborhood, and includes: determining a contrast pixel point in the search neighborhood that is different from the semantic category of the pixel point; and determining a category number value based on the semantic category of the contrast pixel point, the category number value being a number of different semantic categories in the contrast pixel point.

[0009] According to an embodiment of the present disclosure, the inputting of the semantic category frequency, the category relationship value, the category number value, and the ground feature prediction data corresponding to each of the pixel points into the remote sensing ground feature element classification loss function of the ground feature classification model includes: performing logarithmic normalization processing on a plurality of the category relationship values based on a second preset experience value to obtain a normalized relationship value corresponding to the pixel point; performing logarithmic normalization processing on a plurality of the category number values based on a third preset experience value to obtain a normalized category number value corresponding to the pixel point; and inputting the normalized category weight, the normalized relationship value, the normalized category number value, and the ground feature prediction data corresponding to each of the pixel points into the remote sensing ground feature element classification loss function of the ground feature classification model.

[0010] According to an embodiment of the present disclosure, the inputting of the normalized category weight, the normalized relationship value, the normalized category number value, and the ground feature prediction data corresponding to each of the pixel points into the remote sensing ground feature element classification loss function of the ground feature classification model includes: determining a loss weight of the pixel point based on the normalized category weight, the normalized relationship value, and the normalized category number value; and inputting the loss weight corresponding to each of the pixel points and the ground feature prediction data into the remote sensing ground feature element classification loss function of the ground feature classification model.

[0011] Another aspect of the embodiments of the present disclosure provides a remote sensing ground feature element classification method, comprising: obtaining a remote sensing ground feature image to be classified; inputting the remote sensing ground feature image to be classified into a classification model trained by the remote sensing ground feature element classification model training method based on pixel semantic relationship modeling, and outputting a semantic ground truth corresponding to the remote sensing ground feature image to be classified.

[0012] Another aspect of the embodiments of the present disclosure provides a remote sensing ground feature element classification model training device based on pixel semantic relationship modeling, comprising: a frequency determination module configured to determine a semantic class frequency of each semantic class according to a number of pixel points corresponding to different semantic classes in a semantic ground truth, the semantic ground truth being a ground truth corresponding to a ground sample image used for model training; a relationship determination module configured to determine, for each pixel point, a class relationship value of the pixel point in a corresponding search neighborhood based on the semantic class frequency and a semantic class relationship between the pixel point and a center pixel in the search neighborhood; a class determination module configured to determine a class number value of each pixel point in the search neighborhood based on a semantic class of the pixel point; a prediction module configured to input the ground sample image into a ground classification model and output ground prediction data; a loss calculation module configured to input the semantic class frequency, the class relationship value, the class number value and the ground prediction data corresponding to each pixel point into a remote sensing ground feature element classification loss function of the ground classification model, and output a model loss value of the ground classification model; and a training module configured to train the ground classification model based on the model loss value.

[0013] Another aspect of the embodiments of the present disclosure provides a remote sensing ground feature element classification device, comprising: an acquisition module configured to acquire a remote sensing ground feature image to be classified; and a classification module configured to input the remote sensing ground feature image to be classified into a classification model trained by the remote sensing ground feature element classification model training method based on pixel semantic relationship modeling, and output a semantic ground truth corresponding to the remote sensing ground feature image to be classified.

[0014] Another aspect of the embodiments of the present disclosure provides an electronic device, comprising: one or more processors; and a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described above.

[0015] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, the instructions, when executed, being configured to implement the method described above.

[0016] Another aspect of the embodiments of the present disclosure provides a computer program product, the computer program product comprising computer executable instructions for implementing the method as described above when executed.

[0017] According to the embodiments of the present disclosure, because the semantic category frequency of each semantic category is determined, the category relationship value of the pixel point in the corresponding search neighborhood is determined, and the category number value of the pixel point in the corresponding search neighborhood is determined, and the technical means of inputting the semantic category frequency, the category relationship value, the category number value and the feature prediction data corresponding to each pixel point into the remote sensing feature element classification loss function of the feature classification model, the pixel weight factor constructed by the feature classification model in the training process is more reasonable. And based on the semantic category frequency and the semantic category relationship between the pixel point and the center pixel in the search neighborhood, the category relationship value of the pixel point in the corresponding search neighborhood is determined, which more comprehensively measures the classification difficulty of the remote sensing feature element, and effectively improves the convergence speed and performance upper limit of the remote sensing feature element classification model training, so that the feature element classification is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture to which the remote sensing feature element classification model training method and device based on pixel semantic relationship modeling according to embodiments of the present disclosure can be applied is schematically shown;

[0020] Figure 2 A flowchart of the remote sensing feature element classification model training method based on pixel semantic relationship modeling according to embodiments of the present disclosure is schematically shown;

[0021] Figure 3 A flowchart of the remote sensing feature element classification method according to embodiments of the present disclosure is schematically shown;

[0022] Figure 4 A block diagram of the remote sensing feature element classification model training device based on pixel semantic relationship modeling according to embodiments of the present disclosure is schematically shown;

[0023] Figure 5 A block diagram of the remote sensing feature element classification device according to embodiments of the present disclosure is schematically shown;

[0024] Figure 6 A block diagram of an electronic device suitable for implementing the method described above according to embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have been omitted to avoid unnecessarily complicating the present disclosure with details that will be readily understood by those skilled in the art.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall not be taken to exclude

[0027] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined herein. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and not be interpreted in an idealized or overly formal way.

[0028] In the case where an expression similar to "at least one of A, B, and C, and the like" is used, it should generally be interpreted to include any of one, at least one, or more than one of A, B, and C, unless otherwise specifically defined herein. In the case where an expression similar to "at least one of A, B, or C, and the like" is used, it should generally be interpreted to include any of one, at least one, or more than one of A, B, and C, unless otherwise specifically defined herein.

[0029] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (such as including but not limited to user personal information) involved in the technical solutions comply with the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0030] The development of deep learning and convolutional neural networks promotes the research of remote sensing semantic segmentation technology. In recent years, the full convolutional network based on encoder-decoder structure is proposed, and the model performance is improved obviously. In the related technology, in the training process of the remote sensing feature classification model, the standard cross-entropy loss function is generally used to realize the back propagation of the model gradient. However, the contribution of any pixel in the input image to the loss is equal, which will lead to the gradient of the semantic homogeneous region (easy-to-classify pixel) exceeding the gradient of the semantic heterogeneous region (for example, target boundary and small sparse target), thus dominating the entire training process, resulting in slow model convergence speed and difficult model performance improvement.

[0031] To solve the above problems, a common strategy is to add a weight factor to each class, and by calculating the number of classes in the entire data set, different weights are given to each class, the weight value of the class with less number of classes is increased, and the weight value of the class with more number of classes is reduced. However, the number of remote sensing feature classes is generally small, and the data set deviation will affect the accuracy of class weight calculation, resulting in that the class weighting method does not have obvious performance improvement, and it is difficult to solve the problems of irregular shape of remote sensing feature target and large size difference of target.

[0032] In one embodiment, Focal loss (a loss function for solving the imbalance between positive and negative samples) proposes to weight based on the prediction confidence of each pixel, reduce the weight of the pixel with high prediction confidence, and strengthen the weight of the pixel with low prediction confidence. This method has good effect in the classification detection task of positive and negative samples, but the effect is not obvious when it is migrated to the feature classification task.

[0033] In another embodiment, Npa loss (neighborhood pixel similarity loss for semantic segmentation in high-resolution aerial images) proposes to use the number of different classes of center pixels in the neighborhood to measure the classification difficulty of the center pixels, but it treats each class of pixels equally and ignores the influence of pixel class on classification difficulty. In addition, weight standardization processing is also needed, and different standardization methods will seriously affect the model training effect.

[0034] In view of this, the embodiments of the present disclosure provide a pixel semantic relationship modeling based remote sensing ground feature element classification model training method, a pixel semantic relationship modeling based remote sensing ground feature element classification model training device, an electronic device, a readable storage medium and a computer program product. The method comprises: determining the semantic class frequency of each semantic class according to the number of pixel points corresponding to each semantic class in a semantic ground truth map, the semantic ground truth map being a ground truth map corresponding to a ground feature sample map used for model training; for each pixel point, determining the class relationship value of the pixel point in the corresponding search neighborhood based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood; determining the class number value of the pixel point in the corresponding search neighborhood based on the semantic class of each pixel point in the search neighborhood; inputting the ground feature sample map into a ground feature classification model to output ground feature prediction data; inputting the semantic class frequency, the class relationship value, the class number value and the ground feature prediction data corresponding to each pixel point into a remote sensing ground feature element classification loss function of the ground feature classification model to output a model loss value of the ground feature classification model; and training the ground feature classification model based on the model loss value.

[0035] Figure 1 An exemplary system architecture to which the pixel semantic relationship modeling based remote sensing ground feature element classification model training method and device according to the embodiments of the present disclosure can be applied is schematically shown. It should be noted that, Figure 1 The system architecture shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] As Figure 1 shown, the system architecture 100 according to the embodiment can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used as a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0037] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0038] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers and desktop computers, etc.

[0039] The server 105 can be a server providing various services, for example, a background management server providing support for a website browsed by a user using the terminal device 101, 102, or 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing result (for example, a web page, information, or data, or the like obtained or generated according to the user request) to the terminal device.

[0040] It should be noted that the pixel semantic relationship modeling-based remote sensing ground feature element classification model training method provided in the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the pixel semantic relationship modeling-based remote sensing ground feature element classification model training apparatus provided in the embodiments of the present disclosure can generally be arranged in the server 105. The pixel semantic relationship modeling-based remote sensing ground feature element classification model training method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, or 103 and / or the server 105. Correspondingly, the pixel semantic relationship modeling-based remote sensing ground feature element classification model training apparatus provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, or 103 and / or the server 105. Alternatively, the pixel semantic relationship modeling-based remote sensing ground feature element classification model training method provided in the embodiments of the present disclosure can also be executed by the terminal device 101, 102, or 103, or by another terminal device different from the terminal device 101, 102, or 103. Correspondingly, the pixel semantic relationship modeling-based remote sensing ground feature element classification model training apparatus provided in the embodiments of the present disclosure can also be arranged in the terminal device 101, 102, or 103, or in another terminal device different from the terminal device 101, 102, or 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system 100 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0042] Figure 2 A flowchart of the pixel semantic relationship modeling-based remote sensing ground feature element classification model training method according to the embodiments of the present disclosure is schematically shown.

[0043] As shown in Figure 2 The method includes operations S201-S206.

[0044] In operation S201, according to the number of pixel points corresponding to different semantic categories in a semantic ground truth map, the semantic category frequency of each semantic category is determined, and the semantic ground truth map is a ground truth map corresponding to a ground feature sample map used for model training.

[0045] In operation S202, for each pixel point, a class relationship value of the pixel point in the corresponding search neighborhood is determined based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood.

[0046] In operation S203, a class number value of the pixel point in the corresponding search neighborhood is determined based on the semantic class of each pixel point in the search neighborhood.

[0047] In operation S204, the ground object sample image is input into the ground object classification model, and ground object prediction data is output.

[0048] In operation S205, the semantic class frequency, the class relationship value, the class number value corresponding to each pixel point, and the ground object prediction data are input into a remote sensing ground object element classification loss function of the ground object classification model, and a model loss value of the ground object classification model is output.

[0049] In operation S206, the ground object classification model is trained based on the model loss value.

[0050] According to an embodiment of the present disclosure, a satellite cloud image or a aerial photograph can be obtained as a ground object sample image for model training, and a semantic ground truth image corresponding to the ground object sample image is obtained. The length and width of the ground object sample image and the semantic ground truth image are respectively and The semantic ground truth image includes different semantic classes, such as sea water, farmland, green forest land, house, and breeding farm, etc.

[0051] According to an embodiment of the present disclosure, the semantic class frequency of each semantic class is determined according to the number of pixel points corresponding to different semantic classes in the semantic ground truth image. The semantic class frequency can be represented by

[0052] represents the semantic class of the pixel. The semantic class frequency is determined based on the frequency of the pixel points corresponding to the semantic class and the total number of pixel points in the semantic ground truth image. For example, assuming that the total frequency of the pixel points corresponding to the sea water in the semantic class in the semantic ground truth image is 50, and the total number of pixel points in the semantic ground truth image is 200, then the semantic class frequency of the semantic class sea water is 1 / 4. It should be noted that this is only one method for calculating the semantic class frequency in this embodiment, and other ways for calculating the semantic class frequency can also be used, which are not limited herein.

[0053] According to an embodiment of the present disclosure, for each pixel point in the semantic ground truth image, a class relationship value of the pixel point in the corresponding search neighborhood is determined. Each pixel point in the semantic ground truth image has a corresponding search neighborhood, and the radius of the search neighborhood can be defined as ​The search neighborhood interval range can be represented as a rectangular region with a length and a width Each search neighborhood has a center pixel point and other pixel points. Based on the semantic category frequency and the semantic category relationship between the other pixel points and the center pixel, a category relationship value of the pixel point in the corresponding search neighborhood is determined.

[0054] According to an embodiment of the present disclosure, based on the semantic category of the center pixel point of the search neighborhood, the types of semantic categories appearing in the search neighborhood except for the semantic category of the center pixel point are counted, and then the category value of the pixel point in the corresponding search neighborhood is determined.

[0055] According to an embodiment of the present disclosure, the semantic category frequency, the category relationship value, the category value, and the ground feature prediction data corresponding to each pixel point are input into the remote sensing ground feature element classification loss function of the ground feature classification model. By considering the category relationship of the pixel point and the category value of the pixel point, the modeling of the pixel semantic relationship is realized, the difficulty of remote sensing ground feature target classification is more comprehensively measured, and a pixel weight factor is further constructed, and finally the remote sensing ground feature element classification loss function is established.

[0056] According to an embodiment of the present disclosure, because the semantic category frequency of each semantic category is determined, the category relationship value of the pixel point in the corresponding search neighborhood is determined, and the category value of the pixel point in the corresponding search neighborhood is determined, and the technical means of inputting the semantic category frequency, the category relationship value, the category value, and the ground feature prediction data corresponding to each pixel point into the remote sensing ground feature element classification loss function of the ground feature classification model, the pixel weight factor constructed by the ground feature classification model in the training process is more reasonable. Based on the semantic category frequency and the semantic category relationship between the pixel point and the center pixel in the search neighborhood, the category relationship value of the pixel point in the corresponding search neighborhood is determined, the difficulty of remote sensing ground feature element classification is more comprehensively measured, and the convergence speed and performance upper limit of the remote sensing ground feature element classification model training are effectively improved, so that the ground feature element classification is more accurate.

[0057] According to an embodiment of the present disclosure, the method can further include the following operations.

[0058] The reciprocal of each semantic category frequency is processed to obtain a plurality of semantic category weights; and the semantic category weights are logarithmically normalized based on a first preset experience value to obtain a normalized category weight corresponding to the pixel point.

[0059] According to an embodiment of the present disclosure, the semantic category weight is calculated based on the semantic category frequency, and each pixel point corresponds to a semantic category weight. The semantic category weight can be represented as , and the calculation formula is as follows:

[0060]

[0061] The semantic category weight is logarithmically normalized based on a first preset experience value, which can be 2, 10 or e. For a pixel point , the normalized category weight calculation formula is as follows:

[0062]

[0063] wherein, represents the first preset experience value, represents the semantic category weight corresponding to the pixel point with coordinates , max represents the maximum value of the semantic category weight in all pixel points, represents the normalized category weight corresponding to the pixel point with coordinates .

[0064] According to an embodiment of the present disclosure, for each pixel point, based on the semantic category frequency and the semantic category relationship between the pixel point and the center pixel in the search neighborhood, the category relationship value of the pixel point in the corresponding search neighborhood can include the following operations.

[0065] Based on the preset length, the search neighborhood corresponding to the pixel point is determined; based on the semantic category frequency, the classification difficulty coefficient of the pixel point in the search neighborhood is determined; based on the classification difficulty coefficient and the semantic category relationship between the pixel point and the center pixel in the search neighborhood, the category relationship value of the pixel point in the search neighborhood is determined.

[0066] According to an embodiment of the present disclosure, the preset length represents the radius of the search neighborhood, and the search neighborhood corresponding to the pixel point is determined based on the preset length, wherein the search neighborhood interval range can be represented as a rectangular area with a length and a width of . For the center pixel point , the number of different semantic categories in the search neighborhood is counted, and the semantic category frequency of different semantic categories is integrated to calculate the category relationship value. The category relationship value calculation formula is as follows:

[0067]

[0068] wherein, represents the category relationship value corresponding to the pixel point with coordinates , semantic category represents a semantic category, XOR represents an exclusive OR operation, and m and n are integers, represents the classification difficulty coefficient corresponding to the pixel point with coordinates .

[0069] ​​According to embodiments of this disclosure, determining the category value of a pixel in the corresponding search neighborhood based on the semantic category of each pixel in the search neighborhood may include the following operations.

[0070] Identify the contrasting pixels in the search neighborhood that have a different semantic category from the original pixel; determine the category value based on the semantic category of the contrasting pixels, where the category value is the value of the different semantic categories among the contrasting pixels.

[0071] According to embodiments of this disclosure, the search neighborhood is determined excluding the center pixel ( For comparison pixels other than the center pixel, the semantic categories of the comparison pixels and the center pixel are statistically analyzed. For example, suppose there are 8 comparison pixels and 1 center pixel in the search neighborhood. The semantic categories of the 8 comparison pixels are farmland, seawater, seawater, green forest, house, farm, seawater, and farmland. The semantic category of the center pixel is farmland, so the category value is determined to be 4.

[0072] According to embodiments of this disclosure, inputting the semantic category frequency, category relationship value, category value, and land cover prediction data corresponding to each pixel into the remote sensing land cover element classification loss function of the land cover classification model may include the following operations.

[0073] Log-normalization is performed on multiple category relation values ​​based on the second preset empirical value to obtain the normalized relation value corresponding to the pixel; log-normalization is performed on multiple category values ​​based on the third preset empirical value to obtain the normalized category value corresponding to the pixel; the normalized category weight, normalized relation value, normalized category value, and land cover prediction data corresponding to each pixel are input into the remote sensing land cover element classification loss function of the land cover classification model.

[0074] According to embodiments of this disclosure, the second preset empirical value can be selected as 2, 10, or e. For pixel points ( The formula for calculating the normalized relation value is as follows:

[0075]

[0076] in, This represents the second preset experience value. The coordinates are ( The category relationship value corresponding to the pixel of ) This represents the maximum value of the category relationship among all pixels. The coordinates are ( The normalization relation value corresponding to the pixel point.

[0077] According to embodiments of this disclosure, the third preset empirical value can be selected as 2, 10, or e. For pixel points ( The formula for calculating the normalized category numerical value is as follows:

[0078]

[0079] in, This represents the third preset experience value. The coordinates are ( The category value corresponding to the pixel. This represents the maximum value of the category value among all pixels. The coordinates are ( The normalized category value corresponding to the pixel.

[0080] According to embodiments of this disclosure, inputting the normalized category weight, normalized relation value, normalized category value, and land cover prediction data corresponding to each pixel into the remote sensing land cover element classification loss function of the land cover classification model may include the following operations.

[0081] The loss weight of each pixel is determined based on the normalized category weight, normalized relation value, and normalized category value; the loss weight corresponding to each pixel and the predicted land cover data are input into the remote sensing land cover element classification loss function of the land cover classification model.

[0082] According to embodiments of this disclosure, the loss weight corresponding to each pixel is calculated by combining the normalized category weight, the normalized relation value, and the normalized category value. The coordinates are ( Loss weights of pixels The calculation formula is as follows:

[0083]

[0084] in, This represents the weighting factor.

[0085] According to embodiments of this disclosure, a loss function for classifying remote sensing ground features is established based on the loss weights of pixels. As shown below:

[0086]

[0087] in, These represent semantic truth graphs, This represents the predicted data for ground features.

[0088] According to embodiments of this disclosure, pixel semantic relationships are modeled by comprehensively considering both the category relationship values ​​and the category numerical values ​​of pixels, thus providing a more comprehensive measure of the difficulty of classifying remotely sensed ground objects. The resulting loss function for remotely sensed ground object classification improves the convergence speed and performance ceiling of the training model.

[0089] Figure 3A flowchart of a remote sensing feature classification method according to an embodiment of the present disclosure is shown schematically.

[0090] As shown in the method includes operation S301 and operation S302. Figure 3

[0091] In operation S301, a remote sensing feature image to be classified is acquired.

[0092] In operation S302, the remote sensing feature image to be classified is input into a classification model trained by a remote sensing feature element classification model training method, and a semantic ground truth map corresponding to the remote sensing feature image to be classified is output.

[0093] According to embodiments of the present disclosure, the remote sensing feature image to be classified can be acquired from a cloud server or a database, and the classification model trained by the remote sensing feature element classification model training method is input into the classification model, and then the feature elements in the remote sensing feature image to be classified are classified to obtain a semantic ground truth map. The classification model is used to classify the remote sensing feature image to be classified, so that the feature element classification is more accurate.

[0094] Figure 4 A block diagram of a remote sensing feature element classification model training device based on pixel semantic relationship modeling according to an embodiment of the present disclosure is shown schematically.

[0095] As shown in the method includes operation S301 and operation S302. Figure 4

[0096] The frequency determination module 410 is configured to determine the semantic class frequency of each semantic class according to the number of pixel points corresponding to different semantic classes in the semantic ground truth map, the semantic ground truth map being a ground truth map corresponding to a feature sample image used for model training.

[0097] The relationship determination module 420 is configured to determine, for each pixel point, a class relationship value of the pixel point in the corresponding search neighborhood based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood.

[0098] The class determination module 430 is configured to determine the class value of the pixel point in the corresponding search neighborhood based on the semantic class of each pixel point in the search neighborhood.

[0099] The prediction module 440 is configured to input the feature sample image into the feature classification model and output feature prediction data.

[0100] ​​The loss calculation module 450 is configured to input the semantic category frequency corresponding to each pixel point, the category relationship value, the category numerical value, and the ground object prediction data into a remote sensing ground object element classification loss function of the ground object classification model, and output a model loss value of the ground object classification model.

[0101] The training module 460 is configured to train the ground object classification model based on the model loss value.

[0102] According to the embodiments of the present disclosure, because the semantic category frequency of each semantic category is determined, the category relationship value of the pixel point in the corresponding search neighborhood is determined, and the category numerical value of the pixel point in the corresponding search neighborhood is determined, and the technical means of inputting the semantic category frequency corresponding to each pixel point, the category relationship value, the category numerical value, and the ground object prediction data into the remote sensing ground object element classification loss function of the ground object classification model, the pixel weight factor constructed by the ground object classification model in the training process is more reasonable. And based on the semantic category frequency and the semantic category relationship between the pixel point and the center pixel in the search neighborhood, the category relationship value of the pixel point in the corresponding search neighborhood is determined, which more comprehensively measures the classification difficulty of the remote sensing ground object element, and thus effectively improves the convergence speed and performance upper limit of the remote sensing ground object element classification model training, so that the ground object element classification is more accurate.

[0103] According to the embodiments of the present disclosure, the remote sensing ground object element classification model training device 400 based on pixel semantic relationship modeling further comprises a reciprocal processing module and a normalization processing module.

[0104] The reciprocal processing module is configured to perform reciprocal processing on each semantic category frequency to obtain a plurality of semantic category weights.

[0105] The normalization processing module is configured to perform logarithmic normalization processing on the semantic category weights based on a first preset experience value to obtain a normalized category weight corresponding to the pixel point.

[0106] According to the embodiments of the present disclosure, the relationship determination module 420 comprises a neighborhood determination sub-module, a difficulty determination sub-module, and a relationship determination sub-module.

[0107] The neighborhood determination sub-module is configured to determine the search neighborhood corresponding to the pixel point based on a preset length.

[0108] The difficulty determination sub-module is configured to determine a classification difficulty coefficient of the pixel point in the search neighborhood based on the semantic category frequency.

[0109] The relationship determination sub-module is configured to determine the category relationship value of the pixel point in the search neighborhood based on the classification difficulty coefficient and the semantic category relationship between the pixel point and the center pixel in the search neighborhood.

[0110] According to the embodiments of the present disclosure, the category determination module 430 comprises a pixel determination sub-module and a numerical value determination sub-module.

[0111] The pixel determining sub-module is configured to determine a contrast pixel point different from the semantic category of the pixel point in the search neighborhood.

[0112] The numerical value determining sub-module is configured to determine a category numerical value based on the semantic category of the contrast pixel point, the category numerical value being a numerical value of different semantic categories in the contrast pixel point.

[0113] According to an embodiment of the present disclosure, the loss calculating module 450 comprises a relationship normalization sub-module, a category normalization sub-module, and an input sub-module.

[0114] The relationship normalization sub-module is configured to perform logarithmic normalization on the plurality of category relationship values based on a second preset experience value to obtain a normalized relationship value corresponding to the pixel point.

[0115] The category normalization sub-module is configured to perform logarithmic normalization on the plurality of category numerical values based on a third preset experience value to obtain a normalized category numerical value corresponding to the pixel point.

[0116] The input sub-module is configured to input the normalized category weight, the normalized relationship value, the normalized category numerical value, and the ground feature prediction data corresponding to each pixel point into a remote sensing ground feature element classification loss function of the ground classification model.

[0117] According to an embodiment of the present disclosure, the input sub-module comprises a weight determining unit and an input unit.

[0118] The weight determining unit is configured to determine a loss weight of the pixel point based on the normalized category weight, the normalized relationship value, and the normalized category numerical value.

[0119] The input unit is configured to input the loss weight corresponding to each pixel point and the ground feature prediction data into the remote sensing ground feature element classification loss function of the ground classification model.

[0120] Figure 5 A block diagram of a remote sensing ground feature element classification device according to an embodiment of the present disclosure is schematically shown.

[0121] As shown in Figure 5 The remote sensing ground feature element classification device 500 comprises an acquisition module 510 and a classification module 520.

[0122] The acquisition module 510 is configured to acquire a remote sensing ground image to be classified;

[0123] The classification module 520 is configured to input the remote sensing ground image to be classified into a classification model trained by the remote sensing ground feature element classification model training method, and output a semantic ground truth corresponding to the remote sensing ground image to be classified.

[0124] Any of the modules, sub-modules, units according to the embodiments of the present disclosure, or at least part of any of them, can be implemented in one module. Any of the modules, sub-modules, units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any of the modules, sub-modules, units according to the embodiments of the present disclosure can be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System on Chip, a System on Board, a System on Package, an Application Specific Integrated Circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, or in hardware or firmware, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, one or more of the modules, sub-modules, units according to the embodiments of the present disclosure can be implemented at least in part as computer program modules, which can perform corresponding functions when the computer program modules are run.

[0125] For example, any of the frequency determining module 410, the relationship determining module 420, the category determining module 430, the prediction module 440, the loss calculation module 450 and the training module 460 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of the modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units, and implemented in one module / unit / sub-unit. According to the embodiments of the present disclosure, at least one of the frequency determining module 410, the relationship determining module 420, the category determining module 430, the prediction module 440, the loss calculation module 450 and the training module 460 can be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System on Chip, a System on Board, a System on Package, an Application Specific Integrated Circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, or in hardware or firmware, or in any one of software, hardware and firmware, or in a proper combination of any of them. Alternatively, at least one of the frequency determining module 410, the relationship determining module 420, the category determining module 430, the prediction module 440, the loss calculation module 450 and the training module 460 can be implemented at least in part as computer program modules, which can perform corresponding functions when the computer program modules are run.

[0126] It should be noted that the pixel semantic relationship modeling based remote sensing ground feature element classification model training apparatus part in the embodiments of the present disclosure corresponds to the pixel semantic relationship modeling based remote sensing ground feature element classification model training method part in the embodiments of the present disclosure, the remote sensing ground feature element classification apparatus part in the embodiments of the present disclosure corresponds to the remote sensing ground feature element classification method part in the embodiments of the present disclosure, and the description of the pixel semantic relationship modeling based remote sensing ground feature element classification model training apparatus part specifically refers to the pixel semantic relationship modeling based remote sensing ground feature element classification model training method part, and the description of the remote sensing ground feature element classification apparatus part specifically refers to the remote sensing ground feature element classification method part, which will not be repeated here.

[0127] Figure 6 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0128] As shown in Figure 6 The electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to programs stored in a Read-Only Memory (ROM) 602 or loaded from a storage portion 608 into a Random Access Memory (RAM) 603. The processor 601 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an Application-Specific Integrated Circuit (ASIC)), etc. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0129] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the present disclosure by executing programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the present disclosure by executing programs stored in the one or more memories.

[0130] According to an embodiment of the present disclosure, the electronic device 600 can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read therefrom is installed in the storage part 608 as necessary.

[0131] According to an embodiment of the present disclosure, the method flow according to the embodiment of the present disclosure can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product including a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system implementing the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0132] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0133] According to an embodiment of the disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, a portable compact disc read-only memory (CD-ROM), an optical memory device, a magnetic memory device, or any suitable combination of the foregoing. In this disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0134] For example, according to an embodiment of the disclosure, the computer readable storage medium can include one or more memories described above as the ROM 602 and / or the RAM 603 and / or one or more memories other than the ROM 602 and the RAM 603.

[0135] Embodiments of the disclosure also include a computer program product, which includes a computer program containing program codes for executing the method provided by the embodiments of the disclosure, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the pixel semantic relationship modeling based remote sensing feature element classification model training method or the remote sensing feature element classification method provided by the embodiments of the disclosure.

[0136] When the computer program is executed by the processor 601, the above-mentioned functions defined in the system / apparatus of the embodiments of the disclosure are performed. According to an embodiment of the disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0137] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of a signal on a network medium, and downloaded and installed through the communication part 609 and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0138] According to embodiments of the present disclosure, program code of a computer program provided by embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, can be implemented using a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, "C" language, or similar programming languages. Program code can execute entirely on a user's computing device, partly on a user device, partly on a remote computing device, or entirely on a remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0139] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0140] Embodiments of the present disclosure have been described. However, these embodiments are merely intended to illustrate the present disclosure, and are not intended to limit the scope of the present disclosure. Although each of the embodiments is described above separately, this does not mean that the measures in each of the embodiments cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A remote sensing feature element classification model training method based on pixel semantic relationship modeling, comprising: determining the semantic class frequency of each semantic class according to the number of pixel points corresponding to different semantic classes in a semantic ground truth map, the semantic ground truth map being a ground truth map corresponding to a ground feature sample map used for model training; for each pixel point, determining the class relationship value of the pixel point in the corresponding search neighborhood based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood; determining the class numerical value of the pixel point in the corresponding search neighborhood based on the semantic class of each pixel point in the search neighborhood; inputting the ground feature sample map into a ground feature classification model to output ground feature prediction data; inputting the semantic class frequency, the class relationship value, the class numerical value and the ground feature prediction data corresponding to each pixel point into a remote sensing feature element classification loss function of the ground feature classification model to output a model loss value of the ground feature classification model; training the ground feature classification model based on the model loss value; the method further comprises: performing reciprocal processing on each semantic class frequency to obtain a plurality of semantic class weights; performing logarithmic normalization processing on the semantic class weights based on a first preset empirical value to obtain a normalized class weight corresponding to the pixel point; the inputting the semantic class frequency, the class relationship value, the class numerical value and the ground feature prediction data corresponding to each pixel point into the remote sensing feature element classification loss function of the ground feature classification model comprises: performing logarithmic normalization processing on a plurality of the class relationship values based on a second preset empirical value to obtain a normalized relationship value corresponding to the pixel point; performing logarithmic normalization processing on a plurality of the class numerical values based on a third preset empirical value to obtain a normalized class numerical value corresponding to the pixel point; inputting the normalized class weight, the normalized relationship value, the normalized class numerical value and the ground feature prediction data corresponding to each pixel point into the remote sensing feature element classification loss function of the ground feature classification model.

2. The method of claim 1, wherein, the determining the class relationship value of the pixel point in the corresponding search neighborhood based on the semantic class frequency and the semantic class relationship between the pixel point and the center pixel in the search neighborhood comprises: determining the search neighborhood corresponding to the pixel point based on a preset length; determining a classification difficulty coefficient of the pixel point in the search neighborhood based on the semantic class frequency; determining the class relationship value of the pixel point in the search neighborhood based on the classification difficulty coefficient and the semantic class relationship between the pixel point and the center pixel in the search neighborhood.

3. The method of claim 2, wherein, the determining the class numerical value of the pixel point in the corresponding search neighborhood based on the semantic class of each pixel point in the search neighborhood comprises: determining a contrast pixel point different from the semantic class of the pixel point in the search neighborhood; determining a class numerical value based on the semantic class of the contrast pixel point, the class numerical value being the numerical value of different semantic classes in the contrast pixel point.

4. The method of claim 1, wherein, The inputting of the normalized category weight, the normalized relationship value, the normalized category value and the ground object prediction data corresponding to each pixel point into the remote sensing ground object element classification loss function of the ground object classification model comprises: determining the loss weight of the pixel point based on the normalized category weight, the normalized relationship value and the normalized category value; inputting the loss weight corresponding to each pixel point and the ground object prediction data into the remote sensing ground object element classification loss function of the ground object classification model.

5. A remote sensing ground object element classification method, comprising: obtaining a remote sensing ground object image to be classified; inputting the remote sensing ground object image to be classified into the classification model trained by the remote sensing ground object element classification model training method based on pixel semantic relationship modeling in any one of claims 1-4, and outputting a semantic true value graph corresponding to the remote sensing ground object image to be classified.

6. A remote sensing ground object element classification model training device based on pixel semantic relationship modeling, comprising: a frequency determination module configured to determine a semantic category frequency of each semantic category according to the number of pixel points corresponding to different semantic categories in a semantic true value graph, the semantic true value graph being a true value graph corresponding to a ground object sample graph used for model training; a relationship determination module configured to determine, for each pixel point, a category relationship value of the pixel point in a corresponding search neighborhood based on the semantic category frequency and the semantic category relationship between the pixel point and a center pixel in the search neighborhood; a category determination module configured to determine a category value of the pixel point in the corresponding search neighborhood based on the semantic category of each pixel point in the search neighborhood; a prediction module configured to input the ground object sample graph into a ground object classification model and output ground object prediction data; a loss calculation module configured to input the semantic category frequency, the category relationship value, the category value and the ground object prediction data corresponding to each pixel point into a remote sensing ground object element classification loss function of the ground object classification model, and output a model loss value of the ground object classification model; a training module configured to train the ground object classification model based on the model loss value; an inverse processing module configured to perform inverse processing on each of the semantic category frequencies to obtain a plurality of semantic category weights; a normalization processing module configured to perform logarithmic normalization processing on the semantic category weights based on a first preset experience value to obtain a normalized category weight corresponding to the pixel point; a relationship normalization submodule configured to perform logarithmic normalization processing on a plurality of the category relationship values based on a second preset experience value to obtain a normalized relationship value corresponding to the pixel point; a category normalization submodule configured to perform logarithmic normalization processing on a plurality of the category values based on a third preset experience value to obtain a normalized category value corresponding to the pixel point; an input submodule configured to input the normalized category weight, the normalized relationship value, the normalized category value and the ground object prediction data corresponding to each pixel point into the remote sensing ground object element classification loss function of the ground object classification model.

7. A remote sensing ground object element classification device, comprising: an acquisition module configured to obtain a remote sensing ground object image to be classified; The classification module is configured to input the remote sensing ground object image to be classified into the classification model trained by the remote sensing ground object element classification model training method based on pixel semantic relationship modeling, and output a semantic ground truth map corresponding to the remote sensing ground object image to be classified.

8. An electronic device, comprising: one or more processors; memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform any of the methods of claims 1-4, or as claimed in claim 5.

Images