Optical and SAR image comparison decoupling fusion land cover classification method and device

Abstract

The invention discloses an optical and SAR image comparison decoupling fusion land cover classification method and device. Comprising the following steps: acquiring an original optical image and an original SAR image, and preprocessing the original optical image and the original SAR image to form a data set; respectively carrying out image enhancement on the optical image and the SAR image in the data set; respectively inputting the enhanced optical image and the enhanced SAR image into a land cover classification network model for classification to obtain a land cover classification result; calculating a joint loss function based on the land cover classification result and the land cover classification label, and training a land cover classification network model based on the joint function to obtain a trained land cover classification network model; and inputting a to-be-detected optical image and SAR influence and processing into the trained land cover classification network model to obtain a corresponding land cover classification result. The land cover classification accuracy and classification speed are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence security technology, and in particular to a surface cover classification method, device, storage medium and electronic device for optical and SAR image contrast decoupling fusion. Background Art

[0002] Global climate change is closely related to human activities. Land cover, a key indicator of human activity intensity, can be extracted using modern remote sensing technology, effectively revealing patterns in human activity and holding significant significance for deepening understanding of the Earth system and environmental research. With the rapid development of Earth observation technology, multimodal remote sensing data is becoming increasingly abundant, with optical imagery and synthetic aperture radar (SAR) imagery being the most widely used. Optical imagery, acquired by passive sensors, primarily uses visible and near-infrared wavelengths to capture the reflection of solar radiation from objects, revealing rich texture and color features. SAR imagery, on the other hand, is acquired by active microwave sensors, using the backscattering properties of objects to create images, making it more sensitive to the three-dimensional structure of targets. Due to their different imaging principles, optical and SAR imagery exhibit significant differences in their radiometric signatures, resulting in distinct representations of the same objects in both images. In recent years, the combined application of optical and SAR imagery has achieved some success in areas such as change monitoring, cloud processing, and agricultural mapping, but it also faces numerous challenges. The significant differences in the features of the two imagery types can easily lead to information interference during the fusion process, affecting the accuracy of object information extraction and hindering the precise classification of surface targets in combined applications.

[0003] Currently, one of the traditional strategies for the joint interpretation of optical and SAR imagery is to directly fuse them at the image level, then extract features using a deep neural network encoder, and then output the interpretation results through a decoder deconvolution operation. However, this method has several technical bottlenecks: First, the large amount of raw optical and SAR imagery data leads to low algorithm efficiency and significantly increased processing time. Second, the direct fusion of raw data without preprocessing can cause the advantages and disadvantages of different sensor data to overlap, making it difficult to achieve effective information complementarity, thereby affecting the fusion quality. In addition, pixel-based calculations are susceptible to interference from factors such as noise and data outliers, resulting in insufficient stability of the interpretation results, making it difficult to meet the needs of practical applications. Summary of the Invention

[0004] The embodiments of the present application provide a surface cover classification method, device, storage medium and electronic device for decoupling and fusion of optical and SAR images, which can effectively bridge the feature heterogeneity between optical and SAR images, reduce feature interference in the multimodal data fusion process, and improve the accuracy and speed of surface cover classification.

[0005] The present invention provides a surface cover classification method by contrasting and decoupling optical and SAR images, including: Acquiring original optical images and original SAR images, and preprocessing the original optical images and original SAR images to form a data set; Performing image enhancement on the optical image and the SAR image in the data set respectively; The enhanced optical image and SAR image are input into the surface cover classification network model for classification, and the surface cover classification results are obtained; Calculating a joint loss function based on the land cover classification result and the land cover classification label, and training the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The optical image and SAR impact to be detected are input into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

[0006] Furthermore, the above-mentioned surface cover classification method for optical and SAR image contrast decoupling fusion, wherein the original optical image and the original SAR image are preprocessed to form a data set, includes: Aligning the original optical image and the original SAR image to obtain an overlapping area between the original optical image and the original SAR image; Based on the sliding window method, the optical image and SAR image in the overlapping area are segmented and cropped into multiple optical image blocks and SAR image blocks of the same size. The optical image blocks and the SAR image blocks are combined into image units to form a data set.

[0007] Furthermore, in the above-mentioned optical and SAR image contrast decoupling fusion surface cover classification method, the surface cover classification model includes a parallel optical branch and a SAR branch, and a shared feature fusion module; The optical branch includes an optical encoder, a first specific feature decoupling module, a first shared feature decoupling module, a first feature parallel module and an optical decoder; the SAR branch includes a SAR encoder, a second specific feature decoupling module, a second shared feature decoupling module, a second feature parallel module and a SAR decoder.

[0008] Furthermore, in the above-mentioned optical and SAR image contrast decoupling fusion surface cover classification method, the processing process of the surface cover classification network model includes: The optical image is input into the optical encoder for feature encoding to obtain optical semantic features; the SAR image is input into the SAR encoder for feature encoding to obtain SAR semantic features; Inputting the optical semantic feature into a first specific feature decoupling module to obtain an optical nonlinear radiation feature; inputting the SAR semantic feature into a second specific feature decoupling module to obtain a SAR nonlinear radiation feature; Inputting the optical specific feature into a first shared feature decoupling module to obtain an optical shared feature; inputting the SAR specific feature into a second shared feature decoupling module to obtain a SAR shared feature; Inputting the optical shared features and the SAR shared features into the shared feature fusion module to obtain fused features; Inputting the optical nonlinear radiation feature and the fusion feature into a first feature parallel connection module for channel parallel connection to obtain an optical parallel connection feature; inputting the SAR nonlinear radiation feature and the fusion feature into a second feature parallel connection module for channel parallel connection to obtain a SAR parallel connection feature; Inputting the optical parallel features into an optical decoder to obtain a surface coverage classification result of the optical image; inputting the SAR parallel features into a SAR decoder to obtain a surface coverage classification result of the SAR image; The surface cover classification result of the optical image and the surface cover classification result of the SAR image are averaged and fused to obtain a final surface cover classification result.

[0009] Furthermore, in the above-mentioned optical and SAR image contrast decoupling fusion surface cover classification method, the surface cover classification network model further includes an optical image reconstruction module and a SAR image reconstruction module; After the step of obtaining the optical shared feature and the SAR shared feature, the method includes: The optical nonlinear radiation characteristics and the optical shared characteristics are input into the optical image reconstruction module to obtain the reconstruction result of the optical image; the SAR nonlinear radiation characteristics and the SAR shared characteristics are input into the SAR image reconstruction module to obtain the reconstruction result of the SAR image.

[0010] Furthermore, in the above-mentioned optical and SAR image contrast decoupling fusion surface cover classification method, the joint loss function is:

[0011] in, is the joint loss function, is the segmentation loss function, To decouple the contrast loss function, To decouple the weight of the contrast loss, it is a variable hyperparameter; The decoupled contrast loss function is:

[0012] in,

[0013] in, is the optical image decoupling contrast loss function, is the SAR image decoupling contrast loss function, Inactive function, is the mean square error, For optical imaging, is a SAR image, is the reconstruction result of the optical image, is the reconstruction result of SAR, To reconstruct the results using only optical radiation features, is the reconstruction result using only SAR radiation characteristics; The segmentation loss function is:

[0014] in, is the cross entropy loss function, is the dice loss, Labels for the land cover classification, is the surface cover classification result of the optical image, The surface cover classification result of SAR image.

[0015] Furthermore, in the above-mentioned surface cover classification method for contrast decoupling and fusion of optical and SAR images, the image enhancement operations performed on the optical image and the SAR image respectively include at least: random cropping, flip transformation, color distortion, grayscale conversion and noise addition operations.

[0016] The present application also provides a surface cover classification device for optical and SAR image comparison, decoupling and fusion, including: An acquisition module is used to acquire original optical images and original SAR images, and pre-process the original optical images and original SAR images to form a data set; An image enhancement module, configured to perform image enhancement on the optical image and the SAR image in the data set respectively; The surface cover classification module is used to input the enhanced optical image and SAR image into the surface cover classification network model for classification, and obtain the surface cover classification results; A training module is used to calculate a joint loss function based on the land cover classification result and the land cover classification label, and train the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The surface cover classification module is also used to input the optical image and SAR impact to be detected into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

[0017] An embodiment of the present application also provides a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned optical and SAR image comparison and decoupling fusion surface cover classification methods.

[0018] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the optical and SAR image comparison, decoupling and fusion surface cover classification method described in any one of the above items.

[0019] The optical and SAR image contrast decoupling fusion surface cover classification method, device, storage medium and electronic device provided in this application, in the process of joint interpretation of optical and SAR images, by constructing a feature decoupling mechanism, decomposes the two modal images into shared features and specific features, which can effectively bridge the feature heterogeneity between optical and SAR images, reduce feature interference in the multimodal data fusion process, and provide an optimization basis for subsequent information integration. The optical and SAR joint surface cover classification network model based on feature decoupling in this application adopts a modular design. Its technical path is intuitive and clear. While achieving collaborative analysis of multi-source data, it retains good interpretability. This design not only facilitates model performance analysis, but also provides a clear logical framework for subsequent algorithm optimization and field application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and other beneficial effects of the present application apparent.

[0021] Figure 1 Flowchart of the surface cover classification method for optical and SAR image comparison and decoupling fusion provided in an embodiment of the present application.

[0022] Figure 2 Schematic diagram of optical and SAR images of the experimental dataset provided in the embodiments of this application.

[0023] Figure 3 Schematic diagram of the combined surface cover classification results of optical and SAR images provided in the embodiment of the present application.

[0024] Figure 4 Schematic diagram of the structure of the optical and SAR image comparison and decoupling fusion surface cover classification device provided in an embodiment of the present application.

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0027] Currently, one of the traditional strategies for the joint interpretation of optical and SAR imagery is to directly fuse them at the image level, then extract features using a deep neural network encoder, and then output the interpretation results through a decoder deconvolution operation. However, this method has several technical bottlenecks: First, the large amount of raw optical and SAR imagery data leads to low algorithm efficiency and significantly increased processing time. Second, the direct fusion of raw data without preprocessing can cause the advantages and disadvantages of different sensor data to overlap, making it difficult to achieve effective information complementarity, thereby affecting the fusion quality. In addition, pixel-based calculations are susceptible to interference from factors such as noise and data outliers, resulting in insufficient stability of the interpretation results, making it difficult to meet the needs of practical applications.

[0028] Another common approach to joint interpretation of optical and SAR imagery is to use independent encoders to extract features from each image type, followed by feature fusion using methods such as feature addition, channel parallelization, attention mechanisms, or gating mechanisms. Finally, decoder deconvolution completes the joint interpretation of the images. However, this technical approach has significant limitations: on the one hand, before feature fusion, the independent encoders may not fully exploit the potential features of optical and SAR imagery, resulting in inadequate information utilization; on the other hand, complex fusion strategies are often used in the feature fusion stage, leading to problems such as bloated model structures and large parameter sizes. This increases the difficulty of model training and optimization, making it difficult to achieve efficient deployment in practical applications.

[0029] To address the above issues, embodiments of the present application provide a method, device, storage medium, and electronic device for ground cover classification using optical and SAR image comparison and decoupling fusion. The ground cover classification device provided in embodiments of the present application can be integrated into an electronic device, such as a terminal or server. The terminal may include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other device.

[0030] See also Figure 1 , Figure 1This is a flowchart of a method for ground cover classification using optical and SAR image contrast decoupling fusion provided in an embodiment of the present application, which is applied to electronic devices. The method for ground cover classification using optical and SAR image contrast decoupling fusion includes the following steps: S1, obtain the original optical image and the original SAR image, pre-process the original optical image and the original SAR image to form a data set.

[0031] In one embodiment, step S1 includes the following steps: S11, aligning the original optical image and the original SAR image to obtain an overlapping area of ​​the original optical image and the original SAR image; S12, based on a sliding window method, segmenting the optical image and the SAR image in the overlapping area and cropping them into multiple optical image blocks and SAR image blocks of the same size; S13: Combining the optical image blocks and the SAR image blocks into image units to form a data set.

[0032] Specifically, an optical remote sensing image A and a SAR remote sensing image B are collected within the same geographic area and timeframe. Using spatial registration techniques, pixels at corresponding geographic locations in images A and B are precisely aligned to obtain the overlapping region. Using a sliding window approach, the optical and SAR images A and B within the overlapping region are segmented and cropped into multiple image blocks of uniform size. The matching optical image blocks a and SAR image blocks b for each geographic region are combined into an image unit, thereby constructing a surface cover classification dataset for optical and SAR remote sensing images containing multiple groups of image units. Finally, the dataset is rationally divided into training and validation sets for subsequent model training and performance evaluation.

[0033] S2, perform image enhancement on the optical images and SAR images in the dataset respectively.

[0034] Specifically, random cropping, flipping, color distortion, grayscale conversion, and noise addition are used to enhance the optical images in the dataset. Random cropping, angle flipping, color distortion, grayscale conversion, and noise injection are used to enhance the SAR images in the dataset.

[0035] S3, the enhanced optical image and SAR image are input into the surface cover classification network model for classification, and the surface cover classification results are obtained.

[0036] In one embodiment, a land cover classification model includes a parallel optical branch and a SAR branch, as well as a shared feature fusion module. The optical branch includes an optical encoder, a first specific feature decoupling module, a first shared feature decoupling module, a first feature parallel connection module, and an optical decoder. The SAR branch includes a SAR encoder, a second specific feature decoupling module, a second shared feature decoupling module, a second feature parallel connection module, and a SAR decoder. The processing of the two branches is performed simultaneously.

[0037] The processing of the land cover classification network model includes: S31, inputting the optical image into an optical encoder for feature encoding to obtain optical semantic features; inputting the SAR image into a SAR encoder for feature encoding to obtain SAR semantic features.

[0038] It can be expressed by the following formula:

[0039]

[0040] in, For optical imaging, For optical encoders, is the optical semantic feature, is a SAR image, is the SAR encoder, It is the SAR semantic feature.

[0041] S32, inputting the optical semantic feature into the first specific feature decoupling module to obtain the optical nonlinear radiation feature; inputting the SAR semantic feature into the second specific feature decoupling module to obtain the SAR nonlinear radiation feature.

[0042] It can be expressed by the following formula:

[0043]

[0044] in, is the optical nonlinear radiation characteristic, Decoupling module for the first specific feature, is the SAR nonlinear radiation characteristic, Decoupling module for the first specific feature.

[0045] S33, inputting the optical specific feature into the first shared feature decoupling module to obtain the optical shared feature; inputting the SAR specific feature into the second shared feature decoupling module to obtain the SAR shared feature.

[0046]

[0047]

[0048] in, is the first shared feature decoupling module, is the optical sharing feature, It is a SAR shared feature decoupling module. It is the SAR shared feature.

[0049] S34, inputting the optical shared features and the SAR shared features into a shared feature fusion module to obtain fused features.

[0050] Using the shared feature fusion module for fusion can increase the amount of shared information. The above process can be expressed as follows:

[0051] in, For shared fusion modules, For fusion features.

[0052] Furthermore, the surface cover classification network model also includes an optical image reconstruction module and a SAR image reconstruction module; After the steps of obtaining the optical shared features and the SAR shared features, the following steps are included: The optical nonlinear radiation characteristics and the optical shared characteristics are input into the optical image reconstruction module to obtain the reconstruction result of the optical image; the SAR nonlinear radiation characteristics and the SAR shared characteristics are input into the SAR image reconstruction module to obtain the reconstruction result of the SAR image.

[0053] It can be expressed by the following formula:

[0054] in, is the optical image reconstruction module, is the reconstruction result of the optical image, is the SAR image reconstruction module, is the reconstruction result of SAR image.

[0055] S35, inputting the optical nonlinear radiation feature and the fusion feature into the first feature parallel module for channel parallel connection to obtain the optical parallel feature; inputting the SAR nonlinear radiation feature and the fusion feature into the second feature parallel module for channel parallel connection to obtain the SAR parallel feature.

[0056] Connecting channels in parallel can increase the amount of information in each branch. The above process can be expressed as follows:

[0057] in, For channel parallel operation, is the optical parallel characteristic, It is the SAR parallel characteristic.

[0058] S36, inputting the optical parallel features into an optical decoder to obtain a surface coverage classification result of the optical image; inputting the SAR parallel features into a SAR decoder to obtain a surface coverage classification result of the SAR image.

[0059] It can be expressed by the following formula:

[0060] in, For optical decoders, is the surface cover classification result of the optical image, For SAR decoder, The surface cover classification result of SAR image.

[0061] S37, performing average fusion on the surface cover classification result of the optical image and the surface cover classification result of the SAR image to obtain a final surface cover classification result.

[0062] S4, calculating a joint loss function based on the surface cover classification results and the surface cover classification labels, and training the surface cover classification network model based on the joint function to obtain a trained surface cover classification network model.

[0063] Among them, the joint loss function is:

[0064] in, is the joint loss function, is the segmentation loss function, To decouple the contrast loss function, The weight for decoupling the contrast loss is a variable hyperparameter.

[0065] The decoupled contrast loss function is:

[0066] in,

[0067] in, is the optical image decoupling contrast loss function, is the SAR image decoupling contrast loss function, Inactive function, is the mean square error loss, For optical imaging, is a SAR image, is the reconstruction result of the optical image, is the reconstruction result of SAR, To reconstruct the results using only optical radiation features, is the reconstruction result using only SAR radiation characteristics; The segmentation loss function is:

[0068] in, is the cross entropy loss function, is the dice loss, Labels for the land cover classification, is the surface cover classification result of the optical image, The surface cover classification result of SAR image.

[0069] Furthermore, taking the joint loss function as the optimization target, the stochastic gradient descent algorithm is used to iteratively optimize the parameters of the contrast decoupled fusion optical and SAR image joint surface cover classification network model, and the model performance is improved by continuously reducing the loss function value.

[0070] Furthermore, the memory warehouse is updated in real time based on the global features dynamically generated by optical images and SAR images to ensure the timeliness and accuracy of the stored information.

[0071] The above model training operations are executed cyclically, and each set of optical image and SAR image data pairs in the training set are completely processed until all training samples are traversed.

[0072] The optimized interpretation model was applied to the test set data to carry out joint interpretation tasks, and the interpretation accuracy and reliability of the surface cover classification network model were verified by evaluating relevant indicators.

[0073] In this embodiment, the optical-SAR image sample set is the WHU-OPT-SAR dataset. After the images in the WHU-OPT-SAR dataset are cropped into small images of 256×256 pixels without overlap, the training set contains 23,520 images and the test set contains 5,880 images. Data enhancement methods such as overlapping cropping, rotation and color transformation are adopted. In addition to color transformation, the corresponding training labels, near-infrared images and SAR images are also processed accordingly. Unless otherwise specified, the optical images in the WHU-OPT-SAR dataset used in all experiments in this application contain 4 channels of RGB and near-infrared by default, and the SAR images are single-channel intensity maps, and the WHU-OPT-SAR dataset is randomly divided into training and test sets in a ratio of 7:3. Figure 2 Schematic diagram of optical and SAR images of the experimental data set provided in the embodiment of this application, wherein: Figure 2 (a) is an optical image, Figure 2 (b) is the SAR image.

[0074] In the SAR image semantic segmentation task of this embodiment, the size of all input images is scaled to 256×256.

[0075] In some embodiments, the decoder uses a ResNet50 backbone network to extract features of the optical image and output a feature map.

[0076] In some embodiments, the interpretation metrics include the intersection over union (IoU) of each category of objects, recall, precision, and the harmonic mean of recall and precision.

[0077] During the testing phase, attention fusion methods RGBX, MCANet, SaCNet, gating mechanism fusion methods CMGFNet, CEGFNet, and the comparative decoupled fusion optical and SAR image joint interpretation method (ISDFNet) of this embodiment were compared. Table 1 shows the harmonic mean of recall and precision (F1 score) for each object category, the mean harmonic mean of all object categories (mF1 score), and the overall accuracy (OA) for all categories on the WHU-OPT-SAR dataset. Compared to the comparison methods, the ISDFNet method achieved the highest mF1 score (0.730) and the highest OA (0.844), with consistent growth trends across most metrics. The CMGFNet method, which utilizes a gating mechanism to fuse multimodal features, is more suitable for objects with clear boundaries. Forests are relatively vast and lack distinct boundaries in remote sensing images. This is because the proposed method achieves an F1 score of 0.850 for forests, which is 6.59% lower than the proposed method (0.906 for forests). The F1 score of the ISDFNet method increased by 8.96% and 0.97% compared to the RGBX and SaCNet methods, respectively; OA increased by 5.11%, 1.69%, and 1.44%. These results demonstrate that the contrast-decoupled fusion optical and SAR image joint interpretation method (ISDFNet) of this embodiment effectively handles the heterogeneous characteristics of optical and SAR images and achieves a deep fusion of the complementary information of the two modalities. This method significantly enhances the feature expression capability of the semantic segmentation network. Figure 3 This is a schematic diagram of the combined surface cover classification results of optical and SAR images provided in the embodiment of the present application. Figure 3 (a) is an optical image, Figure 3 (b) is the SAR image, Figure 3 (c) is the label image, Figure 3 (d) is the image obtained by the CMGFNet method. Figure 3 (e) is the image obtained by the CEGFNet method. Figure 3(f) is the image obtained by the RGBX method, Figure 3 (g) is the image obtained by SaCNet method, Figure 3 (h) is the image obtained by the ISDFNet method. The relevant visual analysis results are shown in Figure 3, which intuitively demonstrates its effectiveness in feature extraction.

[0078] Table 1 Comparison of mF1 score and OA of this method and other methods

[0079] S5, the optical image to be detected and the SAR impact are input into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification result.

[0080] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a surface cover classification device for optical and SAR image comparison, decoupling and fusion. The surface cover classification device for optical and SAR image comparison, decoupling and fusion can be implemented as an independent entity or integrated into an electronic device, which can be a terminal, server or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0081] See also Figure 4 , Figure 4 The present invention specifically describes an apparatus for ground cover classification by optical and SAR image comparison, decoupling and fusion, and application in electronic devices. The apparatus may include: An acquisition module is used to acquire original optical images and original SAR images, and pre-process the original optical images and original SAR images to form a data set; An image enhancement module, configured to perform image enhancement on the optical image and the SAR image in the data set respectively; The surface cover classification module is used to input the enhanced optical image and SAR image into the surface cover classification network model for classification, and obtain the surface cover classification results; A training module is used to calculate a joint loss function based on the land cover classification result and the land cover classification label, and train the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The surface cover classification module is also used to input the optical image and SAR impact to be detected into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

[0082] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also be found in the beneficial effects in the previous method embodiments, which will not be repeated here.

[0083] In addition, embodiments of the present application further provide an electronic device, which may be a computer, tablet computer, or other device. This electronic device can implement the steps of any embodiment of the optical and SAR image contrast decoupling fusion surface cover classification method provided in the embodiments of the present application, and thus can achieve the beneficial effects achieved by any optical and SAR image contrast decoupling fusion surface cover classification method provided in the embodiments of the present application. For details, please refer to the previous embodiments and will not be repeated here.

[0084] Figure 5 The following figure shows a block diagram of the specific structure of an electronic device provided in an embodiment of the present invention. This electronic device can be used to implement the surface cover classification method for optical and SAR image comparison and decoupling fusion provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other device.

[0085] RF circuit 510 is used to receive and transmit electromagnetic waves, converting them into electrical signals, thereby enabling communication with a communications network or other devices. RF circuit 510 may include various existing circuit components for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, memory, and the like. RF circuit 510 can communicate with various networks, such as the Internet, an intranet, or a wireless network, or with other devices via a wireless network. These wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The wireless networks may utilize various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g, and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messaging, and any other suitable communication protocols, including those currently undeveloped.

[0086] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-mentioned embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing functions such as taking pictures with the front camera, processing the captured images, and switching the display color of the displayed content on the display screen. The memory 520 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely located relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces. These graphical user interfaces can be composed of graphics, text, icons, videos, or any combination thereof. The display unit 540 may include a display panel 541. Optionally, the display panel 541 can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0088] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, which then converts them into sound signals for output. Microphone 562, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 560 and converted into audio data. The audio data is then processed by output processor 580 and transmitted via RF circuit 510 to, for example, another terminal. Alternatively, the audio data may be output to memory 520 for further processing. Audio circuit 560 may also include an earphone jack to allow communication between external headphones and electronic device 500.

[0089] Electronic device 500, through a transmission module 570 (e.g., a Wi-Fi module), can help users receive requests, send information, and so on, providing users with wireless broadband Internet access. Although the figure shows transmission module 570, it is understood that it is not a required component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0090] Processor 580 is the control center of electronic device 500. It connects all components of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 520 and accessing data stored in memory 520, it executes various functions of electronic device 500 and processes data, thereby providing overall monitoring of the electronic device. Optionally, processor 580 may include one or more processing cores. In some embodiments, processor 580 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 580.

[0091] Electronic device 500 also includes a power supply 590 (e.g., a battery) for powering various components. In some embodiments, the power supply can be logically connected to processor 580 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 590 can also include any components, such as one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0092] Although not shown, the electronic device 500 also includes a camera (such as a front camera and a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Acquiring original optical images and original SAR images, and preprocessing the original optical images and original SAR images to form a data set; Performing image enhancement on the optical image and the SAR image in the data set respectively; The enhanced optical image and SAR image are input into the surface cover classification network model for classification, and the surface cover classification results are obtained; Calculating a joint loss function based on the land cover classification result and the land cover classification label, and training the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The optical image and SAR impact to be detected are input into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

[0093] During specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments and will not be repeated here.

[0094] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished through instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any embodiment of the optical and SAR image contrast and decoupling fusion land cover classification method provided in an embodiment of the present invention.

[0095] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0096] Since the instructions stored in the storage medium can execute the steps in any embodiment of the optical and SAR image comparison decoupling fusion surface cover classification method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any optical and SAR image comparison decoupling fusion surface cover classification method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0097] The above is a detailed introduction to the optical and SAR image contrast decoupling fusion surface cover classification method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A surface cover classification method based on optical and SAR image contrast decoupling fusion, characterized by: The method comprises: Acquiring original optical images and original SAR images, and preprocessing the original optical images and original SAR images to form a data set; Performing image enhancement on the optical image and the SAR image in the data set respectively; The enhanced optical image and SAR image are input into the surface cover classification network model for classification, and the surface cover classification results are obtained; Calculating a joint loss function based on the land cover classification result and the land cover classification label, and training the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The optical image and SAR impact to be detected are input into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

2. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 1 is characterized in that: The original optical image and the original SAR image are preprocessed to form a data set, including: Aligning the original optical image and the original SAR image to obtain an overlapping area between the original optical image and the original SAR image; Based on the sliding window method, the optical image and SAR image in the overlapping area are segmented and cropped into multiple optical image blocks and SAR image blocks of the same size. The optical image blocks and the SAR image blocks are combined into image units to form a data set.

3. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 1 is characterized in that: The land cover classification model includes a parallel optical branch and a SAR branch, and a shared feature fusion module; The optical branch includes an optical encoder, a first specific feature decoupling module, a first shared feature decoupling module, a first feature parallel module and an optical decoder; the SAR branch includes a SAR encoder, a second specific feature decoupling module, a second shared feature decoupling module, a second feature parallel module and a SAR decoder.

4. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 3 is characterized in that: The processing process of the land cover classification network model includes: The optical image is input into the optical encoder for feature encoding to obtain optical semantic features; the SAR image is input into the SAR encoder for feature encoding to obtain SAR semantic features; Inputting the optical semantic feature into a first specific feature decoupling module to obtain an optical nonlinear radiation feature; inputting the SAR semantic feature into a second specific feature decoupling module to obtain a SAR nonlinear radiation feature; Inputting the optical specific feature into a first shared feature decoupling module to obtain an optical shared feature; inputting the SAR specific feature into a second shared feature decoupling module to obtain a SAR shared feature; Inputting the optical shared features and the SAR shared features into the shared feature fusion module to obtain fused features; Inputting the optical nonlinear radiation feature and the fusion feature into a first feature parallel connection module for channel parallel connection to obtain an optical parallel connection feature; inputting the SAR nonlinear radiation feature and the fusion feature into a second feature parallel connection module for channel parallel connection to obtain a SAR parallel connection feature; Inputting the optical parallel features into an optical decoder to obtain a surface coverage classification result of the optical image; inputting the SAR parallel features into a SAR decoder to obtain a surface coverage classification result of the SAR image; The surface cover classification result of the optical image and the surface cover classification result of the SAR image are averaged and fused to obtain a final surface cover classification result.

5. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 4 is characterized in that: The surface cover classification network model also includes an optical image reconstruction module and a SAR image reconstruction module; After the step of obtaining the optical shared feature and the SAR shared feature, the method includes: Inputting the optical nonlinear radiation feature and the optical shared feature into the optical image reconstruction module to obtain a reconstruction result of the optical image; The SAR nonlinear radiation feature and the SAR shared feature are input into the SAR image reconstruction module to obtain a reconstruction result of the SAR image.

6. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 1, characterized in that: The joint loss function is: in, is the joint loss function, is the segmentation loss function, To decouple the contrast loss function, To decouple the weight of the contrast loss, it is a variable hyperparameter; The decoupled contrast loss function is: in, in, is the optical image decoupling contrast loss function, is the SAR image decoupling contrast loss function, Inactive function, is the mean square error loss, For optical imaging, is a SAR image, is the reconstruction result of the optical image, is the reconstruction result of SAR, To reconstruct the results using only optical radiation features, is the reconstruction result using only SAR radiation characteristics; The segmentation loss function is: in, is the cross entropy loss function, is the dice loss, Labels for the land cover classification, is the surface cover classification result of the optical image, The surface cover classification result of SAR image.

7. The surface cover classification method based on optical and SAR image contrast decoupling fusion according to claim 1, characterized in that: The image enhancement operations performed on the optical image and the SAR image respectively include at least random cropping, flip transformation, color distortion, grayscale conversion and noise addition operations.

8. A surface cover classification device for optical and SAR image contrast decoupling fusion, characterized by: include: An acquisition module is used to acquire original optical images and original SAR images, and pre-process the original optical images and original SAR images to form a data set; An image enhancement module, configured to perform image enhancement on the optical image and the SAR image in the data set respectively; The surface cover classification module is used to input the enhanced optical image and SAR image into the surface cover classification network model for classification, and obtain the surface cover classification results; A training module is used to calculate a joint loss function based on the land cover classification result and the land cover classification label, and train the land cover classification network model based on the joint function to obtain a trained land cover classification network model; The surface cover classification module is also used to input the optical image and SAR impact to be detected into the trained surface cover classification network model after processing to obtain the corresponding surface cover classification results.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the optical and SAR image contrast decoupling fusion surface cover classification method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the optical and SAR image contrast decoupling fusion surface cover classification method according to any one of claims 1 to 7.

Images