Semantic recognition model training method, semantic recognition method, and related devices

Abstract

The application provides a semantic recognition model training method, a semantic recognition method and related devices. The semantic recognition model training method comprises: obtaining a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image; wherein the first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region, and the second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region; superimposing the first image and the second image to obtain a combined image; and training an initial model using the combined image, the first annotation map and the second annotation map to obtain a semantic recognition model. Only the changed region and the attribute are annotated, which reduces manual intervention and reduces time consumption. Furthermore, the multi-channel advantage can be fully utilized, so that the semantic recognition model obtained by training has good performance and the recognition result is accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a semantic recognition model training method, a semantic recognition method and related devices. BACKGROUND

[0002] With the continuous development of electronic information technology, satellite remote sensing imaging technology has also made rapid progress. Remote sensing technology uses artificial satellites, space laboratories and other detection instruments to obtain electromagnetic wave information of target objects from a distance (4000km-6000km), and forms a multi-level, multi-angle, multi-field observation system. Multi-temporal remote sensing image semantic change detection technology is a technology that uses the information presented by multiple remote sensing images of the same ground area taken at different time points to determine the ground cover change characteristics and processes through mutual comparison and multiple processing methods, and assigns the changed pixels in the remote sensing image with corresponding semantic information before and after the change.

[0003] Remote sensing image semantic change detection technology can be divided into traditional methods and deep learning methods. Traditional methods such as classification, GIS, principal component analysis, etc. will cause time-consuming, low precision and other problems due to the complexity and diversity of land cover types, and require a lot of manual intervention. With the development of artificial intelligence technologies such as deep learning, multi-temporal remote sensing image semantic change detection based on deep learning overcomes the above shortcomings and becomes the mainstream method.

[0004] Remote sensing image semantic change detection has a wide range of applications in many fields, such as urban planning and management, resource exploration, environmental monitoring, public safety disaster prediction and battlefield intelligence analysis. Therefore, the research on remote sensing image semantic change detection has important value. Therefore, a method is needed that can accurately identify the semantic changes of remote sensing images. SUMMARY

[0005] The present application provides a semantic recognition model training method, a semantic recognition method and related devices, which can reduce manual intervention, reduce labeling time-consuming, and the semantic recognition model trained has good performance and accurate recognition results.

[0006] To solve the above technical problems, the first technical solution provided by the present application is to provide a training method of a semantic recognition model, comprising: acquiring a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image; wherein the first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region, and the second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region; superimposing the first image and the second image to obtain a combined image; training an initial model using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model.

[0007] To solve the above technical problems, the second technical solution provided by the present application is to provide a training device of a semantic recognition model, comprising: an acquisition module configured to acquire a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image; wherein the first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region, and the second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region; a superimposition module configured to superimpose the first image and the second image to obtain a combined image; and a training module configured to train an initial model using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model.

[0008] To solve the above technical problems, the third technical solution provided by the present application is to provide a semantic recognition method, comprising: acquiring a to-be-recognized image, wherein the to-be-recognized image comprises a first to-be-recognized image and a second to-be-recognized image; and recognizing the to-be-recognized image using a semantic recognition model, wherein the semantic recognition model is trained by any one of the above methods.

[0009] To solve the above technical problems, the fourth technical solution provided by the present application is to provide a semantic recognition device, comprising: an acquisition module configured to acquire a to-be-recognized image, wherein the to-be-recognized image comprises a first to-be-recognized image and a second to-be-recognized image; and a recognition module configured to recognize the to-be-recognized image using a semantic recognition model, wherein the semantic recognition model is trained by any one of the above methods.

[0010] To solve the above technical problems, the fifth technical solution provided by the present application is to provide an electronic device, comprising: a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute any one of the above methods.

[0011] To solve the above technical problems, the sixth technical solution of the present application provides a computer readable storage medium storing a program file, the program file being executable to implement any of the above methods.

[0012] The beneficial effects of the present application are that, different from the prior art, the first annotation icon of the present application is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region, and the second annotation icon is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region. Only the changed region and the attribute are annotated, which reduces manual intervention and time consumption. The first image and the second image are superimposed to obtain a combined image; the initial model is trained using the combined image, the first annotation icon, and the second annotation icon to obtain the semantic recognition model. In this way, the multi-channel advantage can be fully utilized, and the semantic recognition model obtained by training has good performance and accurate recognition results. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0014] Figure 1 The flowchart of an embodiment of the training method of the semantic recognition model of the present application;

[0015] Figure 2 The flowchart of an embodiment of the training method of the semantic recognition model of the present application; Figure 1 The flowchart of an embodiment of the training method of the semantic recognition model of the present application;

[0016] Figure 3 The structural diagram of an embodiment of the training device of the semantic recognition model of the present application;

[0017] Figure 4 The flowchart of an embodiment of the training method of the semantic recognition model of the present application;

[0018] Figures 5a-5b The flowchart of an embodiment of the training method of the semantic recognition model of the present application; Figure 4 The flowchart of an embodiment of the training method of the semantic recognition model of the present application;

[0019] Figure 6 The structural diagram of an embodiment of the training device of the semantic recognition model of the present application;

[0020] Figure 7 The structural diagram of an embodiment of the electronic device of the present application;

[0021] Figure 8 Structure diagram of an embodiment of the computer readable storage medium of the present application. DETAILED DESCRIPTION

[0022] The prior art proposes a semantic change detection method and system based on improved fuzzy C-means clustering. The method integrates remote sensing image preprocessing, bottom feature extraction, improved fuzzy C-means algorithm, high-level visual dictionary mapping, and semantic change information extraction, taking into account the rich regional structure information of high-resolution remote sensing images, making up for the limitations of pixel-level spectral statistical features in scanning complex targets, automatically extracting high-level semantic change information of the image, and effectively improving the timeliness and accuracy of the processing. The main shortcomings of this method are: a feature vector extraction module is used to extract bottom feature vector combinations for multi-temporal remote sensing image overlapping blocks, but the images between different time phases have diverse feature differences due to time, space, and seasonality, and the operation of extracting feature vectors for different time phase images separately will introduce a large amount of uncertainty error; fuzzy clustering is used to divide the feature vectors to realize the mapping of bottom features to high-level visual words, but the above method belongs to traditional algorithms and requires pixel pre-classification and labeling for the entire image, and the fuzzy C-means clustering algorithm needs to consume a large amount of computing resources and time to obtain the pixel change set of a specific feature vector.

[0023] The prior art also provides a dual-time-phase remote sensing image semantic change detection method based on multi-modal feature fusion. This method performs multi-class semantic segmentation on different time phase images, then performs change detection, can capture the class change relationship between multiple time phases, detects the categories before and after the pixel change at the same time, has a wider application range, meets end-to-end processing, does not require manual processing, and is convenient for engineering application. The main shortcomings of this method are: the method for obtaining the binary classification change map is to separately input the images of the previous and subsequent time phases into the ResNet network, subtract the corresponding positions of each layer of down-sampled feature maps, and then up-sample and fuse to obtain the binary classification change map, and the direct pixel value subtraction operation will cause precision loss; the method multiplies the predicted binary change area and the semantic segmentation result to obtain the final detection result, which introduces two errors in the process, first, the error caused by the multi-class semantic segmentation itself, and second, the semantic segmentation result corresponding to the predicted binary change area is not unique, and the unique change type corresponding to the change area cannot be determined; the design needs two predictions and two manual annotations of the binary and multi-class results, increasing the time and labor costs.

[0024] Based on the deficiencies of the prior art, the present application proposes a training method of a semantic recognition model, which does not need to perform time-consuming full-size image semantic segmentation annotation, only needs to label the pixels of the changed area, and only needs to perform one prediction to obtain the changed semantic graph, which can save a lot of manpower and time cost and improve the efficiency. In the following, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0025] Please refer to Figure 1 The flowchart of the first embodiment of the training method of the semantic recognition model of the present application is shown in the figure, which specifically includes:

[0026] Step S11: obtaining a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image.

[0027] In an embodiment, the first image and the second image of the present application are multi-temporal remote sensing images. The multi-temporal remote sensing images reflect the situation of the same area at different times. That is, the first image and the second image are images of the same area at different times. For example, the first image is an image of city C in 2018, and the second image is an image of city C in 2021.

[0028] The first annotation map is annotated with a first changed area of the first image relative to the second image and a first semantic attribute corresponding to the first changed area, and the second annotation map is annotated with a second changed area of the second image relative to the first image and a second semantic attribute corresponding to the second changed area. It should be noted that the first image is not limited to one, but can be multiple. Similarly, the second image is not limited to one, but can be multiple. It can be understood that when the first image and the second image are multiple, the multiple first images and the multiple second images correspond one by one. Assuming that the first image is an image of city C in 2018, and the second image is an image of city C in 2021. Then the first annotation map represents the changed area and the semantic attribute of the image of city C in 2018 relative to the image of city C in 2021. The second annotation map represents the changed area and the semantic attribute of the image of city C in 2021 relative to the image of city C in 2018.

[0029] In a specific embodiment, the semantic attribute categories are divided into 7 categories in total, including changed area, water surface, ground, low vegetation, tree, building and sports field. The pixels of the changed area corresponding to the first image and the second image are labeled using 1, 2, 3, 4, 5, and 6. For example, the pixels of the area changed to the water surface are labeled with the number 1, the pixels of the area changed to the ground are labeled with the number 2, the pixels of the area changed to the low vegetation are labeled with the number 3, the pixels of the area changed to the tree are labeled with the number 4, the pixels of the area changed to the building are labeled with the number 5, and the pixels of the area changed to the sports field are labeled with the number 6. After labeling all the changed areas, the remaining areas are defaulted as unchanged areas, and the pixels are defaulted as 0. After labeling, the first labeled image and the second labeled image are obtained.

[0030] It should be noted that the size of the first labeled image and the second labeled image obtained after labeling is the same as the size of the original first image and the second image, and the first labeled image and the second labeled image are both single-channel label images.

[0031] In this embodiment, time-consuming full-size image semantic segmentation labeling is not required, only the pixels of the changed area and the semantic attribute corresponding to the changed area need to be labeled, which is beneficial to improve the efficiency of sample labeling and reduce labor costs.

[0032] Further, at least one of image registration, channel stretching, image value normalization, and data enhancement is performed on the first image and the second image. Specifically, the 16-bit depth channel values of the first image and the second image are expanded according to RGB three different channels and distributed to the first range, for example, between 0 and 2300. Since a small amount of data fragments in the first image and the second image are seriously distributed, the values between the second range in the first range, for example, 0 to 450, and the third range in the first range, for example, 1000 to 2300, are considered as a very small part of image noise, which is filtered to stretch the channel. After channel stretching, the black edges caused by light and offset can be eliminated to affect the original image. After stretching the four channels in the first image and the second image, the different channel value distributions are difficult for the model to adapt to the four distributions. In order to make the model converge faster, the channel values are normalized to make the channel values as consistent as possible. Since the overall stretching data operation may lose some detailed information, which is not conducive to model generalization, a method of randomly changing the mean and variance is used for data enhancement to enhance the generalization of the model under different distributions as much as possible.

[0033] Step S12: superimposing the first image and the second image to obtain a combined image.

[0034] Specifically, the first image and the second image are four-channel remote sensing images, and the four channels are R channel, G channel, B channel and near-infrared channel. The first image and the second image are superimposed to form an eight-channel combined image. In this way, the advantages of multi-channel can be fully utilized.

[0035] Step S13: training an initial model by using the combined image, the first annotation image and the second annotation image to obtain the semantic recognition model.

[0036] The initial model is trained by using the combined image, the first annotation image and the second annotation image to obtain the semantic recognition model.

[0037] In an embodiment, the size of the multi-temporal remote sensing image is generally large, that is, the size of the first image and the second image and the combined image is large. In order to reduce the amount of calculation, the combined image, the first annotation image and the second annotation image are further cut to obtain a plurality of images with a preset size. For example, the combined image, the first annotation image and the second annotation image are cut into small images with a size of 512*512. Then, the initial model is trained by using the cut combined image, the first annotation image and the second annotation image to obtain the semantic recognition model.

[0038] Further, in an embodiment, the initial model is a neural network model, in order to avoid overfitting phenomenon of the neural network model on a small data set, the combined image, the first annotation image and the second annotation image are preprocessed. The preprocessing operation includes at least one of random rotation, horizontal / vertical flip, adding image distortion, scale zoom, adding noise interference and color change. In this way, the training sample set can be enriched, and the generalization ability of the network model can be improved.

[0039] The training method of the semantic recognition model of the embodiment does not need time-consuming full-size image semantic segmentation annotation, only needs to label the pixels of the change area and the semantic attribute corresponding to the change area, which is beneficial to improve the efficiency of sample labeling and reduce the labor cost. And the first image and the second image are superimposed to form an eight-channel combined image. In this way, the advantages of multi-channel can be fully utilized. The semantic recognition ability of the semantic recognition model obtained by training is better.

[0040] Please refer to Figure 2 , in order to Figure 1The flowchart of an embodiment of step S13 is shown in FIG. 6. In this application, the initial model is SCDU-Net network. The SCDU-Net convolutional network is based on U-Net model, and is divided into an expansive path and a contracting path. The contracting path includes a repeated structure. Each structure includes two 3x3 convolutional layers, a nonlinear ReLU layer, and a 2x2 max pooling layer. Four layers of down-sampling are performed, and the number of feature channels is doubled after each down-sampling. In the expansive path, up-convolution is used at each step, and a BN layer is added. After each up-sampling, a convolutional layer is added. The number of feature channels is halved, and the size of the feature map is doubled. Each layer in the contracting path is up-sampled.

[0041] Step S13 specifically includes:

[0042] Step S21: performing down-sampling processing on the combined image, the first annotation image, and the second annotation image to obtain a first feature map.

[0043] Specifically, the combined image, the first annotation image, and the second annotation image are sequentially processed by a convolutional layer such as a 3x3 convolutional layer, a nonlinear activation function such as ReLU, and a max pooling layer 2x2 max pooling layer (stride 2) to obtain a first feature map. In a specific embodiment, the combined image, the first annotation image, and the second annotation image are sequentially processed by a convolutional layer such as a 3x3 convolutional layer, a nonlinear activation function such as ReLU, and a max pooling layer 2x2 max pooling layer (stride 2) four times.

[0044] Step S22: performing up-sampling processing on the first feature map to obtain a second feature map.

[0045] Specifically, the first feature map is sequentially processed by an up-sampling convolutional layer such as up-convolution, a normalization layer such as BN, and a convolutional layer to obtain the second feature map. Specifically, the processing result of each layer of down-sampling is sequentially up-sampled by an up-sampling convolutional layer such as up-convolution, a normalization layer such as BN, and a convolutional layer to obtain the second feature map. The U-Net model is improved in this application, and an improved multi-scale skip connection is used. Each layer of down-sampling is up-sampled, and each layer of up-sampling is fused with feature information through a skip connection, so that the features of each layer are fully utilized.

[0046] Step S23: splicing the first feature map and the second feature map to obtain a feature map.

[0047] The first feature map and the second feature map are spliced by using a skip connection to obtain a feature map.

[0048] It should be noted that the first feature map obtained after downsampling has a large size, and the first feature map can be pruned before splicing.

[0049] In another embodiment, the feature map is processed by using a 3x3 convolution layer, a ReLU activation function and a 1x1 convolution layer in sequence to convert the 64-channel feature map into a specific number of categories, i.e., six classification results in semantic change detection.

[0050] Step S24: obtaining the semantic recognition model based on the feature map by using an intersection over union-based semantic segmentation loss function.

[0051] Specifically, the loss function in the initial network is replaced by an intersection over union-based semantic segmentation loss function, such as an IOU-based Lovasz-Softmax loss. The intersection over union-based semantic segmentation loss function is as follows:

[0052]

[0053]

[0054] wherein y* represents a label result, represents a network prediction result.

[0055] wherein the intersection over union evaluation index is also called a Jaccard coefficient, the difference between the prediction value and the true value is calculated pixel by pixel, the labels are sorted according to the error ranking, and the Lovasz-Softmax loss is obtained by combining the error and the Jaccard coefficient, which can effectively avoid the influence of class imbalance. Specifically, the loss function lovaszSoftmax based on IOU used in the present application alleviates the problem of uneven distribution of training sample categories, thereby improving the effect of semantic change detection.

[0056] Specifically, the combined image, the first annotation image and the second annotation image are input to an initial model for training. The initial model performs the above processing on the combined image, the first annotation image and the second annotation image, adopts a stochastic gradient descent (SGD) method and a back propagation method, and finally trains a semantic recognition model. It should be noted that, in the training, Batch size = 1 and epoch = 20. One epoch refers to a process of feeding all data into the network to complete a forward calculation and a back propagation. The so-called Batch is a part of data fed into the network for training each time, and the Batch Size is the number of training samples in each Batch.

[0057] Further, in an embodiment, the training process of the present application adopts a MultiStepLR dynamic learning rate adjustment strategy to obtain the semantic recognition model based on the feature map based on a semantic segmentation loss function based on the intersection over union. The learning rate learning rate starts at 1e -2 , and becomes 1e -3 when epoch is 8, and becomes 1e -4 when epoch is 15, and becomes 1e -5 .

[0058] The training method of the semantic recognition model proposed in the present application can realize end-to-end direct output of the change area and semantic information corresponding to the image by the SCDU-Net model, superimposing different channel inputs of multi-temporal images, without the need for artificial intervention, time-consuming pixel pre-classification and marking work on the whole image. At the same time, according to the advantages of deep learning, errors caused by feature differences will not be introduced; in view of the characteristics that the proportion of different categories of change areas is very different in remote sensing image semantic change detection, the loss of network model training is modified and replaced by a semantic segmentation loss function based on the intersection over union, solving the problem of unbalanced sample categories and solving the problem of too low recognition rate in small data samples.

[0059] In the present application, time-consuming full-size image semantic segmentation annotation is not required, only the pixels of the change area need to be labeled, and the change semantic map can be obtained by only one prediction, which can save a lot of manpower and time cost and improve the efficiency. Through the improvement of the U-Net network, the improved multi-scale skip connection and the modification of the last layer activation function to ReLU are added, and the training data channel stretching, normalization, histogram equalization, data enhancement and other operations are combined, which can make the network more easily converge and improve the training speed and accuracy of the network.

[0060] Please refer to Figure 3It is a structural schematic view of an embodiment of the training device of the semantic recognition model of the application, and specifically includes an acquisition module 31, a superposition module 32, and a training module 33.

[0061] The acquisition module 31 is configured to acquire a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image. The first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region. The second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region. The first image and the second image are images of the same region at different times.

[0062] The acquisition module 31 is further configured to perform at least one of image registration, channel stretching, image numerical normalization, and data enhancement on the first image and the second image, and to perform a preprocessing operation on the combined image, the first annotation map, and the second annotation map. The preprocessing operation includes at least one of random rotation, horizontal / vertical flipping, adding image distortion, scaling, adding noise interference, and color change.

[0063] The superposition module 32 is configured to superimpose the first image and the second image to obtain a combined image.

[0064] The training module 33 is configured to train an initial model using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model. Specifically, the training module 33 is configured to perform down-sampling processing on the combined image, the first annotation map, and the second annotation map to obtain a first feature map, perform up-sampling processing on the first feature map to obtain a second feature map, and concatenate the first feature map and the second feature map to obtain a feature map. The training module 33 is configured to obtain the semantic recognition model based on the feature map using a semantic segmentation loss function based on the intersection over union.

[0065] Specifically, the training module 33 sequentially processes the combined image, the first annotation map, and the second annotation map using a convolutional layer, a nonlinear activation function, and a maximum pooling layer to obtain the first feature map.

[0066] Specifically, the training module 33 sequentially processes the first feature map using an up-sampling convolutional layer, a normalization layer, and a convolutional layer to obtain the second feature map.

[0067] Specifically, the training module 33 sequentially processes the feature map using a 3x3 convolutional layer, a ReLU activation function, and a 1x1 convolutional layer.

[0068] Specifically, the training module 33 adopts a MultiStepLR dynamic learning rate adjustment strategy to obtain the semantic recognition model based on the feature map based on a semantic segmentation loss function based on an intersection over union.

[0069] The training device of the semantic recognition model provided in the application can realize direct end-to-end output of the change area and semantic information corresponding to the image by stacking different channel inputs of multi-temporal images through the SCDU-Net model, without the need for artificial intervention, time-consuming pixel pre-classification and marking work on the whole image. Meanwhile, according to the advantages of deep learning, errors caused by feature differences are not introduced; in view of the characteristics that the proportions of different categories of change areas are quite different in remote sensing image semantic change detection, the loss of network model training is modified and replaced by a semantic segmentation loss function based on an intersection over union, so that the problem of sample class imbalance is solved, and the problem of low recognition rate can also be well solved with small data samples.

[0070] In the application, time-consuming full-size image semantic segmentation labeling is not required, only the pixels of the change area need to be labeled, and the change semantic map can be obtained only once, which can save a lot of manpower and time cost and improve the efficiency. Through improvement of the U-Net network, improved multi-scale skip connection and modification of the last layer activation function to ReLU are added, and channel stretching, normalization, histogram equalization and data enhancement operations of training data are combined, so that the network can converge more easily, and the training speed and accuracy of the network can be improved.

[0071] Please refer to Figure 4 The flowchart of an embodiment of the semantic recognition method of the application specifically includes the following steps:

[0072] Step S41: acquiring an image to be recognized.

[0073] The image to be recognized includes a first image to be recognized and a second image to be recognized. The first image to be recognized and the second image to be recognized are images of the same area at different times.

[0074] Further, in an embodiment, the first image to be recognized is segmented to obtain a plurality of first sub-images, and the second image to be recognized is segmented to obtain a plurality of second sub-images. The plurality of first sub-images are overlapped according to a preset proportion, and the part of each first sub-image that is not overlapped is taken as a first recognition image. The plurality of second sub-images are overlapped according to a preset proportion, and the part of each second sub-image that is not overlapped is taken as a second recognition image. The plurality of first recognition images are spliced, and the plurality of second recognition images are spliced. Specifically, as shown in FIG. 4, the first image to be recognized is segmented to obtain a plurality of first sub-images 41, and the second image to be recognized is segmented to obtain a plurality of second sub-images 42. The plurality of first sub-images 41 are overlapped according to a preset proportion, and the part of each first sub-image 41 that is not overlapped is taken as a first recognition image 43. The plurality of second sub-images 42 are overlapped according to a preset proportion, and the part of each second sub-image 42 that is not overlapped is taken as a second recognition image 44. The plurality of first recognition images 43 are spliced to obtain a first recognition image 45, and the plurality of second recognition images 44 are spliced to obtain a second recognition image 46. Figure 5aAs shown, it is assumed that the first to-be-identified image is divided into 1-9 first sub-images. The 1-9 first sub-images are overlapped according to a preset ratio, for example, 1:5, and the specific process is as follows Figure 5b As shown, the first to-be-identified image is divided into 1-9 first sub-images. The 1-9 first sub-images are overlapped according to a preset ratio, for example, 1:5, and the specific process is as follows

[0075] Step S42: identifying the to-be-identified image by using the semantic identification model.

[0076] Specifically, the to-be-identified image is input into the semantic identification model, and the to-be-identified image includes the first to-be-identified image and the second to-be-identified image. The semantic identification model can identify the difference between the first to-be-identified image and the second to-be-identified image and the semantic attribute corresponding to the difference, and identify the difference between the second to-be-identified image and the first to-be-identified image and the semantic attribute corresponding to the difference.

[0077] Further, the first identification image after splicing and the second identification image after splicing are identified by using the semantic identification model.

[0078] The semantic identification method of the present application can realize an end-to-end multi-temporal remote sensing image semantic change detection method, which can detect the change region and identify the semantic change information at the same time. This method can improve the efficiency of sample labeling and reduce the labor cost.

[0079] Please refer to Figure 6 , which is a structural schematic diagram of an embodiment of the semantic identification device of the present application, and specifically includes an acquisition module 61 and an identification module 62.

[0080] The acquisition module 61 is used to acquire a to-be-identified image, and the to-be-identified image includes a first to-be-identified image and a second to-be-identified image.

[0081] Further, the acquisition module 61 is also used to divide the first to-be-identified image to obtain a plurality of first sub-images, divide the second to-be-identified image to obtain a plurality of second sub-images, overlap the plurality of first sub-images according to a preset ratio, and cut the part of each first sub-image that is not overlapped as a first identification image; and overlap the plurality of second sub-images according to a preset ratio, and cut the part of each second sub-image that is not overlapped as a second identification image; splice the plurality of first identification images, and splice the plurality of second identification images.

[0082] The recognition module 62 is configured to recognize the to-be-recognized image by using a semantic recognition model, which is trained by the method in any of the above embodiments.

[0083] In another embodiment, the recognition module 62 is further configured to recognize the first recognized image after splicing and the second recognized image after splicing by using a semantic recognition model.

[0084] The semantic recognition device of the present application can realize an end-to-end multi-temporal remote sensing image semantic change detection method, which can recognize semantic change information while detecting the change region. This helps to improve the efficiency of sample labeling and reduce labor costs.

[0085] Please refer to Figure 7 The structure of an embodiment of the electronic device of the present application is shown in the figure. The electronic device comprises a memory 202 and a processor 201 connected to each other.

[0086] The memory 202 is configured to store program instructions for implementing the method of the device of any of the above embodiments.

[0087] The processor 201 is configured to execute the program instructions stored in the memory 202.

[0088] The processor 201 can also be referred to as a CPU (Central Processing Unit). The processor 201 can be an integrated circuit chip with signal processing capability. The processor 201 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application-Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0089] The memory 202 can be a memory stick, a TF card, etc. It can store all information in the electronic device of the device, including input raw data, computer programs, intermediate running results and final running results. It is saved in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the electronic device has memory function and can work normally. The memory of the electronic device can be divided into main memory (memory) and auxiliary memory (external memory) according to its use. There are also classification methods of external memory and internal memory. External storage is usually a magnetic medium or an optical disc, which can store information for a long time. The memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but only for temporary storage of programs and data. When the power is off or the power is off, the data will be lost.

[0090] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. The division of the modules or units is merely logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0091] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0092] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or partly, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods in the embodiments of the present application.

[0094] Please refer to Figure 8A structural schematic diagram of the computer readable storage medium of the present application. The storage medium of the present application stores a program file 203 capable of realizing all the methods described above, wherein the program file 203 can be stored in the storage medium in the form of a software product, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The storage device described above includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes, or a computer, a server, a mobile phone, a tablet, and other terminal devices.

[0095] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for training a semantic recognition model, the method comprising: The method comprises the following steps: obtaining a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image; wherein the first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region, and the second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region; wherein the number of the first image and the second image is one or more; when the first image and the second image are both multiple, the multiple first images and the multiple second images correspond to each other in one-to-one manner; superimposing the first image and the second image to obtain a combined image; training an initial model using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model; wherein the step of training the initial model using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model comprises: performing down-sampling processing on the combined image, the first annotation map, and the second annotation map to obtain a first feature map; performing up-sampling processing on the first feature map to obtain a second feature map; splicing the first feature map and the second feature map to obtain a feature map; obtaining the semantic recognition model based on the feature map using a semantic segmentation loss function based on intersection over union.

2. The method of claim 1, wherein, The first image and the second image are images of the same region at different times.

3. The method of claim 1, wherein, The step of obtaining the first image, the second image, the first annotation map corresponding to the first image, and the second annotation map corresponding to the second image comprises: performing at least one of image registration, channel stretching, image value normalization, and data enhancement on the first image and the second image; and performing a preprocessing operation on the combined image, the first annotation map, and the second annotation map, the preprocessing operation comprising at least one of random rotation, horizontal / vertical flipping, adding image distortion, scale scaling, adding noise interference, and color change.

4. The method of claim 1, wherein, The step of performing down-sampling processing on the combined image, the first annotation map, and the second annotation map to obtain a first feature map comprises: sequentially processing the combined image, the first annotation map, and the second annotation map using a convolution layer, a nonlinear activation function, and a maximum value pooling layer to obtain the first feature map.

5. The method of claim 1, wherein, The step of performing up-sampling processing on the first feature map to obtain a second feature map comprises: sequentially processing the first feature map using an up-sampling convolution layer, a normalization layer, and a convolution layer to obtain the second feature map.

6. The method of claim 1, wherein, The step of obtaining the semantic recognition model based on the feature map using a semantic segmentation loss function based on intersection over union further comprises: sequentially processing the feature map using a 3x3 convolution layer, a ReLU activation function, and a 1x1 convolution layer.

7. The method of claim 1, wherein, The step of obtaining the semantic recognition model based on the feature map using a semantic segmentation loss function based on intersection over union further comprises: The semantic recognition model is obtained based on the feature map by using a MultiStepLR dynamic learning rate adjustment strategy and a semantic segmentation loss function based on an intersection over union. 8.A device for training a semantic recognition model, comprising: The method comprises the steps of: The acquisition module is configured to acquire a first image, a second image, a first annotation map corresponding to the first image, and a second annotation map corresponding to the second image. The first annotation map is annotated with a first change region of the first image relative to the second image and a first semantic attribute corresponding to the first change region. The second annotation map is annotated with a second change region of the second image relative to the first image and a second semantic attribute corresponding to the second change region. The first image and the second image are one or more in number.

9. A method of semantic recognition, characterized in that, When the first image and the second image are multiple, the multiple first images and the multiple second images are in one-to-one correspondence. The superposition module is configured to superimpose the first image and the second image to obtain a combined image. The training module is configured to train an initial model by using the combined image, the first annotation map, and the second annotation map to obtain the semantic recognition model.

10. The method of claim 9, wherein, The training module is further configured to perform down-sampling processing on the combined image, the first annotation map, and the second annotation map to obtain a first feature map. The first feature map and the second feature map are spliced to obtain a feature map. The semantic recognition model is obtained based on the feature map by using a semantic segmentation loss function based on an intersection over union. The method comprises the steps of: Acquiring an image to be recognized, wherein the image to be recognized comprises a first image to be recognized and a second image to be recognized. The image to be recognized is recognized by using a semantic recognition model.

11. A semantic recognition apparatus characterized by comprising: The step of acquiring the image to be recognized comprises the steps of: The first image to be recognized is segmented to obtain a plurality of first sub-images, and the second image to be recognized is segmented to obtain a plurality of second sub-images. The plurality of first sub-images are overlapped according to a preset proportion, and the non-overlapped part of each first sub-image is taken as a first recognition image.

12. An electronic device, comprising: The plurality of second sub-images are overlapped according to a preset proportion, and the non-overlapped part of each second sub-image is taken as a second recognition image. The plurality of first recognition images are spliced, and the plurality of second recognition images are spliced. The step of recognizing the image to be recognized by using the semantic recognition model comprises the steps of: The first recognition image after splicing and the second recognition image after splicing are recognized by using the semantic recognition model. The method comprises the steps of: An acquisition module is configured to acquire an image to be recognized, wherein the image to be recognized comprises a first image to be recognized and a second image to be recognized. An identification module is configured to recognize the image to be recognized by using a semantic recognition model. The method comprises the steps of: A memory and a processor, wherein the memory stores program instructions, and the processor fetches the program instructions from the memory to execute the training method of the semantic recognition model according to any one of claims 1-7; or The semantic recognition method according to any one of claims 9-10 is executed.

13. A computer-readable storage medium, characterized in that, A program file is stored, and the program file can be executed to implement the training method of the semantic recognition model according to any one of claims 1-7; or The semantic recognition method according to any one of claims 9-10 is executed. The semantic recognition method according to any one of claims 9-10 is executed.

Images