Fine-grained representation remote sensing image interpretation method and system based on Shearlet priori knowledge

Abstract

The invention discloses a fine-grained representation remote sensing image interpretation method and system based on Shearlet priori knowledge. The method comprises the following steps: performing high-frequency sub-band information extraction on preprocessed remote sensing image data by adopting Shearlet transformation; a Laplacian pyramid filter and a shearlet filter are used for obtaining a high-frequency assembly and multidirectional high-frequency sub-bands obtained after the high-frequency assembly is split in the direction; sub-bands of the same level are adaptively fused through a space attention mechanism, and a detail-enhanced feature map is generated; carrying out multi-scale and multi-level weighted fusion on high-frequency component prior extracted through Shearlet transformation and a deep residual block obtained by a deep residual network; outputting an accurate remote sensing image analysis result by using the trained detail enhancement backbone and different detection heads of a plurality of tasks; through combination of Shearlet transformation and an advanced deep learning architecture, the accuracy and efficiency of remote sensing image interpretation are significantly improved, good model generalization ability and practicability are maintained, and it is predicted that remote sensing image processing will be promoted to develop towards a more intelligent and automatic direction.

Description

Background Art

[0002] Remote sensing technology has become an important tool for Earth observation since the mid-20th century, especially in the application of satellite and aerial images. These technologies can provide quantitative information about the Earth's surface and atmosphere, which is crucial for fields such as climate change research, environmental protection, agricultural monitoring and urban planning. With the development of remote sensing technology, the resolution and acquisition frequency of images have been significantly improved, but it also brings challenges such as large data volume and high information complexity.

[0003] Although the development of remote sensing technology has brought more data and higher image quality, existing image analysis methods often have difficulty in effectively processing this data. For example, classification methods based on traditional feature extraction rely on manually selected features, which is not only time-consuming and labor-intensive, but also limited in effectiveness by the representativeness and comprehensiveness of the selected features. In addition, when processing high-resolution remote sensing images, traditional methods often have low recognition accuracy because the algorithms are not sufficient to handle the complexity in the images.

[0004] In recent years, deep learning, especially convolutional neural networks (CNNs), has been shown to perform well in image recognition and classification tasks. However, when applied to remote sensing image analysis, these models are often limited by their lack of ability to capture high-frequency details. The unique properties of remote sensing images, such as large-scale scale variations, complex textures and structural features, make it difficult for standard CNN models to be directly applied.

[0005] The Shearlet transform provides an effective multi-scale and multi-directional analysis method, which is particularly suitable for processing image data with anisotropic characteristics. By accurately capturing high-frequency features such as edges and cracks in the image, the Shearlet transform can provide more detailed image analysis than the traditional wavelet transform. In addition, the nonlinear decomposition capability of the Shearlet transform enables it to reveal hidden and non-explicit information levels in the image, which is particularly valuable for parsing complex remote sensing data.

[0006] 5. Combining the high-frequency feature decomposition capability of the Shearlet transform with the powerful learning ability of the neural network can greatly improve the accuracy and efficiency of remote sensing image analysis. Through this fusion, not only can the network's perception of details be enhanced, but also the model's ability to parse complex scenes can be improved while maintaining the integrity of the deep semantics of the image. This new method is expected to promote the development of remote sensing image processing from traditional methods to a more intelligent and automated direction. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a remote sensing image interpretation method and system with fine-grained representation based on Shearlet prior knowledge in view of the deficiencies in the above-mentioned prior art, which is integrated into the learning process of the neural network in the form of spatial attention, optimizes the network's ability to parse complex remote sensing data, and is used to solve the technical problem that the fine-grained structural information in remote sensing images is omitted in traditional deep learning methods as the convolutional layer deepens.

[0008] The present invention adopts the following technical solutions: A fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge includes the following steps: S1. Preprocessing the original remote sensing image data; S2, using Shearlet transform to extract high-frequency sub-band information in multiple levels and multiple directions from the remote sensing image data pre-processed in step S1; using Laplacian pyramid and shearlet filter to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction; S3, construct high-frequency component prior based on Shearlet transform, adaptively fuse sub-bands at the same level through spatial attention mechanism, and generate feature maps with enhanced details; S4, construct a detail enhancement backbone network based on ResNet, and perform multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform in step S3 and the deep residual block obtained by the deep residual network; S5, using supervised learning to train the ResNet-based detail enhancement backbone network obtained in step S4, and repeatedly iterating using the remote sensing image data labeled in step S1 until the downstream task model performs stably on the validation set; S6. Use the detail enhancement backbone trained in step S5 and different detection heads for multiple tasks to output remote sensing image analysis results.

[0009] Preferably, in step S1, the preprocessing operation includes noise removal, contrast adjustment and image cropping.

[0010] Preferably, step S2 specifically includes: S201, input the pre-processed remote sensing image, and perform color space conversion on the image, from RGB to HSI; S202, applying the Laplace pyramid method to perform high- and low-frequency information stratification on the I component of the remote sensing image, and extracting the highest frequency component of the image; S203, for the extracted high-frequency components, use a Shearlet filter to obtain the highest frequency sub-bands in multiple directions; S204, down-sample the remaining low-frequency information to half of the previous one, apply the Laplacian pyramid method again to decompose the high- and low-frequency information, and obtain multi-directional sub-high-frequency sub-bands.

[0011] Preferably, in step S202, the Shearlet transformation details are as follows:

[0012] in, and Represent the space vector and the translation vector respectively, is the scaling parameter, represents the shear factor, is the expansion matrix, is the shear matrix, is the transformation domain.

[0013] Preferably, step S201 to step S204 are looped three times until the highest frequency sub-band, the second highest frequency sub-band and the higher frequency sub-band information are obtained.

[0014] Preferably, step S3 is specifically: S301, by rearranging the dimensions of the high-frequency subbands extracted three times recursively, preparing for the input format C×H×W of the deep learning model, where C is the number of channels, H is the height, and W is the width; each channel C receives all the directional subbands corresponding to one extraction, indicating the directional information at a specific scale; S302, using a spatial attention module based on Shearlet transform to fuse the multi-directional sub-bands; S303, the spatial attention module uses a convolutional layer and the ReLU activation function to enhance the model's learning ability of spatial texture information; then the second convolutional layer deepens the feature fusion through the Sigmoid activation function to generate a compressed feature map focusing on the key information of the image. The final output Shearlet attention feature map has a dimension of 1×H×W.

[0015] Preferably, step S4 is specifically: S401, build ResNet50 as the basic network. ResNet50 has five CNN residual blocks, namely the zeroth layer, the first layer, the second layer, the third layer and the fourth layer; S402, adjusting the size of the feature map obtained in step S3, so that the highest frequency feature map and the second layer CNN residual block have the same size, the second high frequency feature map and the third layer CNN residual block have the same size, and the higher frequency feature map and the fourth layer CNN residual block have the same size; S403, multiply the high-frequency prior feature map adjusted in step S402 by the CNN deep residual block features output by the ResNet backbone network on a pixel-by-pixel basis at multiple scales to achieve effective integration of the two types of features.

[0016] Preferably, step S5 is specifically: The neural network is trained through supervised learning and repeated iterations are performed using labeled remote sensing data; after the neural network training is completed, the performance of the neural network is evaluated.

[0017] Preferably, step S6 specifically includes: S601, using the trained detail enhancement backbone network to capture discriminative features in remote sensing images; S602, integrating multiple task detection heads with the detail enhancement backbone network to obtain a complete remote sensing multi-task processing model; S603, the model integrates the multi-task remote sensing image interpretation model obtained in step S602, and outputs the remote sensing image analysis results.

[0018] In a second aspect, the embodiments of the present invention provide Preprocessing module, which performs preprocessing operations on the original remote sensing image data; The Shearlet transform module uses Shearlet transform to extract high-frequency sub-band information in multiple levels and multiple directions from the pre-processed remote sensing image data. The Laplacian pyramid and shearlet filter are used to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction. The spatial attention module constructs a high-frequency component prior based on Shearlet transform, adaptively fuses sub-bands at the same level through the spatial attention mechanism, and generates a feature map with enhanced details. The detail enhancement backbone network module builds a detail enhancement backbone network based on ResNet, and performs multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform and the deep residual block obtained by the deep residual network; The training and output module uses the ResNet-based detail enhancement backbone network trained by supervised learning, and uses labeled remote sensing image data for repeated iterations until the downstream task model performs stably on the validation set. The trained detail enhancement backbone and different detection heads for multiple tasks are used to output remote sensing image analysis results.

[0019] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge are implemented.

[0020] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: A fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge. As a multi-scale and multi-directional geometric analysis tool, hearlet transform can effectively capture the detailed scale and directional features in the image, especially the edge details and other high-frequency parts. The output of this transform is input into the neural network as multi-scale and multi-level prior knowledge, which greatly improves the performance of the network in remote sensing image analysis tasks, especially in applications such as semantic segmentation, target detection and tracking; by adopting Shearlet prior knowledge, high-frequency detail features in the image can be effectively captured, which is often impossible to achieve in traditional wavelet transform methods. The multi-scale and multi-directional capabilities of Shearlet transform are particularly suitable for processing remote sensing images with complex textures and edges, thereby significantly improving the accuracy and efficiency of image interpretation.

[0022] Further, preprocessing operations including noise removal, contrast adjustment and image cropping are aimed at improving the quality of the original image data and providing clearer and more consistent input data for subsequent high-frequency feature extraction and analysis. This step reduces the error and complexity in subsequent processing and ensures the stability and reliability of the system.

[0023] Furthermore, the high-frequency subband information of the preprocessed image is extracted by Shearlet transform, which not only improves the image resolution and the visibility of edge information, but also reveals the tiny and detailed features in the image. This method is particularly suitable for analyzing high-resolution remote sensing data, thereby improving the accuracy of classification and recognition.

[0024] Furthermore, by constructing a high-frequency component prior based on Shearlet transform and using the spatial attention mechanism to adaptively fuse sub-bands at the same level, we can more effectively focus on the key areas in the image and enhance the model's response to important parts of the image, thereby improving the interpretation quality of remote sensing images.

[0025] Furthermore, the construction of the detail enhancement backbone network, combined with the deep residual network, achieves the effective integration of high-frequency prior features and deep features. This strategy optimizes the learning ability of the network, especially when dealing with complex remote sensing scenes, and can maintain the integrity of the deep semantics of the image while emphasizing the detail features.

[0026] Furthermore, supervised learning training details enhance the backbone network so that the model can perform stably on the validation set after iterative training, which ensures the reliability and generalization ability of the model in practical applications.

[0027] Furthermore, using the trained detail enhancement backbone and multi-task detection head, the model is able to output accurate remote sensing image analysis results, which is suitable for a variety of complex application scenarios, such as land cover classification, target detection, etc., demonstrating its wide application potential and practical value.

[0028] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0029] In summary, the present invention significantly improves the accuracy and efficiency of remote sensing image interpretation by combining Shearlet transform and advanced deep learning architecture, while maintaining good model generalization ability and practicality, and is expected to promote remote sensing image processing towards a more intelligent and automated direction.

[0030] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic flow chart of a method for fine-grained characterization of remote sensing images based on Shearlet prior knowledge provided in an embodiment of the present invention; Figure 2 A schematic diagram of using a Laplacian pyramid to split high- and low-frequency information and obtain multi-directional high-frequency sub-bands in a Shearlet transform provided in an embodiment of the present invention; Figure 3 A flowchart of adaptively fusing high-frequency multi-directional sub-bands at the same scale in Shearlet transform provided in an embodiment of the present invention; Figure 4 A schematic diagram of the fine-grained enhancement backbone ResNet structure provided in an embodiment of the present invention.

[0032] Figure 5 A schematic diagram of a computer device provided by an embodiment of the present invention; Figure 6 The present invention is a block diagram of an electronic device provided according to an embodiment. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0036] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0037] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0039] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0040] The present invention provides a fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge, aiming to improve the accuracy of remote sensing image analysis; by applying Shearlet transform to extract high-frequency features from remote sensing images, the present invention can highlight the fine-grained structural information in the image, which is usually omitted in traditional deep learning methods as the convolution layer deepens; the high-frequency feature emphasis mechanism of the present method is integrated into the learning process of the neural network in the form of spatial attention, thereby optimizing the network's ability to parse complex remote sensing data; in terms of technical implementation, Shearlet transform, as a multi-scale and multi-directional geometric analysis tool, can effectively capture the detailed scale and directional features in the image, especially the edge details and other high-frequency parts. The output of this transform is input into the neural network as multi-scale and multi-level prior knowledge, which greatly improves the performance of the network in remote sensing image analysis tasks, especially in applications such as semantic segmentation, target detection and tracking.

[0041] The present invention provides a remote sensing image interpretation method based on shearlet prior knowledge, comprising the following steps: S1, receiving raw remote sensing image data and performing a series of preprocessing operations on it, including noise removal, contrast adjustment and image cropping, to ensure data quality and consistency; S2, using Shearlet transform to extract multi-level high-frequency sub-band information from the pre-processed remote sensing image; The Laplacian pyramid and shearlet filter are used to obtain the high-frequency components and the multi-directional high-frequency sub-bands after splitting them by direction.

[0042] S201, input the pre-processed remote sensing image, and perform color space conversion on the image, from RGB to HSI; S202, applying the Laplace pyramid method to perform high- and low-frequency information stratification on the I component of the remote sensing image, and extracting the highest frequency component of the image; Shearlet transformation details are as follows:

[0043] in, and Represent the space vector and the translation vector respectively, is the scaling parameter, represents the shear factor. The shear transformation is done by the dilation matrix defined as and the shear matrix To adjust the scale and orientation:

[0044] S203, for the extracted high-frequency components, use a Shearlet filter to obtain the highest frequency sub-bands in multiple directions; The specific process is as follows: The Shearlet filter in the spatial domain is composed of scale , Shear , and cone area Parameterization; shear parameters for each cone region In scale The range from Expand to ,in =0.5; this configuration effectively divides the frequency domain into different directional segments for processing.

[0045] In the frequency domain, the Shearlet filter supports similar functions as the Trapezoid filter.

[0046] Cone Parameters is 0, 1, or 2, and determines the directionality of the filter in the frequency domain: For low frequency filters, (horizontally) emphasizes horizontal features, while (Vertical) Focus on vertical structures.

[0047] when and When the value is {-1, 0, 1}, the configurations from 1 to 7 are (0, 0, 0) (i.e., low-pass component), (0, -1, h), (0, 0, h), (0, 1, h), (0, 1, v), (0, 0, v) and (0, -1, v), respectively. Figure 3 This structured and detailed approach enhances the ability of Shearlet transform to optimize feature extraction and noise management in complex image processing tasks by effectively utilizing scale, shear, and cone parameters.

[0048] S204, down-sample the remaining low-frequency information to half of the previous one, apply the Laplacian pyramid method again to decompose the high- and low-frequency information, and obtain multi-directional sub-high-frequency sub-bands.

[0049] The above process is performed three times in total until the highest frequency sub-band, the second highest frequency sub-band and the higher frequency sub-band information are obtained.

[0050] S3, construct a high-frequency prior based on Shearlet transform, complete the multi-level sub-band extraction, and use the spatial attention mechanism to adaptively fuse the sub-bands of the same level; S301, by rearranging the dimensions of the high-frequency subbands extracted three times recursively, preparing for the input format C×H×W of the deep learning model, where C is the number of channels, H is the height, and W is the width; each channel C receives all the directional subbands extracted once, thereby representing the directional information at a specific scale; S302, using a spatial attention module based on Shearlet transform to fuse the multi-directional sub-bands; Through this method, the model is able to generate a single-channel attention feature map that highlights the most important feature areas in the image, which is discriminative for analyzing remote sensing images.

[0051] S303, the spatial attention module uses a convolutional layer (Conv1) to enhance the model's learning ability of spatial texture information using the ReLU activation function; then the second convolutional layer (Conv2) uses the Sigmoid activation function to deepen feature fusion and generate a compressed feature map that focuses on the key information of the image. The final output Shearlet attention feature map has a dimension of 1×H×W.

[0052] S4. Construct a detail enhancement backbone based on ResNet, and perform multi-scale and multi-level weighted fusion of the high-frequency prior extracted by Shearlet transform and the deep residual block obtained by the network, so as to enhance the perceptual depth and complexity of the neural network without sacrificing the deep semantics of the image; S401. Build ResNet50 as the basic network, perform multi-scale feature extraction, and obtain multi-scale information of different receptive fields; ResNet50 has a total of five CNN residual blocks, namely: layer 0, layer 1, layer 2, layer 3, and layer 4.

[0053] S402, adjusting the size of the feature map obtained in step S303, so that the highest frequency feature map and the second layer CNN residual block have the same size, the second high frequency feature map and the third layer CNN residual block have the same size, and the higher frequency feature map and the fourth layer CNN residual block have the same size; S403, multiply the high-frequency prior feature map adjusted in step S402 by the CNN deep residual block features output by the ResNet backbone network on a pixel-by-pixel basis at multiple scales to achieve effective integration of the two types of features.

[0054] S5. The combined network is trained through supervised learning, using labeled remote sensing data for repeated iterations until the model performs stably on the validation set. After the model is trained, a comprehensive performance evaluation is performed on it, including the calculation of key indicators in various downstream tasks. S501, training a neural network, the network is trained by supervised learning, and the labeled remote sensing data is used for repeated iterations; This process continues until the model performance on the validation set reaches a predetermined stability criterion.

[0055] S502, once the neural network training is completed, enter the next stage; At this stage, the model stops iterating and is ready for the next step of performance evaluation; S503: After completing the model training, a comprehensive performance evaluation of the network is performed.

[0056] This includes calculating the key performance indicators of the model in various downstream tasks, such as accuracy, recall, and F1 score, to ensure that the model meets the needs of practical applications.

[0057] S6. Using the trained detail enhancement backbone and different detection heads for multiple tasks, the model outputs accurate remote sensing image analysis results and supports specific application scenarios such as land cover classification and target detection.

[0058] S601, using a trained detail enhancement backbone network, which is specially designed to capture discriminative features in remote sensing images; This enhanced feature capture capability is achieved by combining the Shearlet transform with deep residual learning, ensuring that the model can effectively handle highly complex remote sensing data.

[0059] S602, integrating a plurality of specially designed task detection heads with a detail enhancement backbone network; These detectors are optimized for different application scenarios, such as land cover classification, target detection, etc., allowing the model to demonstrate its powerful analytical capabilities in multiple fields.

[0060] S603. The model integrates all the above functions and outputs accurate remote sensing image analysis results.

[0061] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0062] In another embodiment of the present invention, a fine-grained representation remote sensing image interpretation system based on Shearlet prior knowledge is provided. The system can be used to implement the above-mentioned fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge. Specifically, the fine-grained representation remote sensing image interpretation system based on Shearlet prior knowledge includes a preprocessing module, a Shearlet transformation module, a spatial attention module, a detail enhancement backbone network module, and a training and output module.

[0063] Among them, the preprocessing module performs preprocessing operations on the original remote sensing image data; The Shearlet transform module uses Shearlet transform to extract high-frequency sub-band information in multiple levels and multiple directions from the pre-processed remote sensing image data. The Laplacian pyramid and shearlet filter are used to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction. The spatial attention module constructs a high-frequency component prior based on Shearlet transform, adaptively fuses sub-bands at the same level through the spatial attention mechanism, and generates a feature map with enhanced details. The detail enhancement backbone network module builds a detail enhancement backbone network based on ResNet, and performs multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform and the deep residual block obtained by the deep residual network; The training and output module uses the ResNet-based detail enhancement backbone network trained by supervised learning, and uses labeled remote sensing image data for repeated iterations until the downstream task model performs stably on the validation set. The trained detail enhancement backbone and different detection heads for multiple tasks are used to output remote sensing image analysis results.

[0064] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the fine-grained characterization remote sensing image interpretation method based on Shearlet prior knowledge, including: The original remote sensing image data is preprocessed; Shearlet transform is used to extract multi-level and multi-directional high-frequency sub-band information from the preprocessed remote sensing image data; Laplacian pyramid and shearlet filter are used to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction respectively; a high-frequency component prior based on Shearlet transform is constructed, and sub-bands at the same level are adaptively fused through the spatial attention mechanism to generate a feature map with detail enhancement; a ResNet-based detail enhancement backbone network is constructed, and multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform and the deep residual block obtained by the deep residual network is performed; the ResNet-based detail enhancement backbone network trained by supervised learning is repeatedly iterated using labeled remote sensing image data until the downstream task model performs stably on the validation set; the trained detail enhancement backbone and different detection heads for multiple tasks are used to output remote sensing image analysis results.

[0065] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device, and can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that more specific examples (non-exhaustive list) of the computer-readable storage medium here include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0066] Computer readable storage media also include data signals propagated in baseband or as part of a carrier wave, which carry readable program codes. Such propagated data signals can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0067] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0068] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for interpreting remote sensing images with fine-grained representation based on Shearlet prior knowledge in the above embodiment; the processor may load and execute the following steps: The original remote sensing image data is preprocessed; Shearlet transform is used to extract multi-level and multi-directional high-frequency sub-band information from the preprocessed remote sensing image data; Laplacian pyramid and shearlet filter are used to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction respectively; a high-frequency component prior based on Shearlet transform is constructed, and sub-bands at the same level are adaptively fused through the spatial attention mechanism to generate a feature map with detail enhancement; a ResNet-based detail enhancement backbone network is constructed, and multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform and the deep residual block obtained by the deep residual network is performed; the ResNet-based detail enhancement backbone network trained by supervised learning is repeatedly iterated using labeled remote sensing image data until the downstream task model performs stably on the validation set; the trained detail enhancement backbone and different detection heads for multiple tasks are used to output remote sensing image analysis results.

[0069] See also Figure 5 , the terminal device is a computer device. The computer device 60 of this embodiment includes: a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When the computer program 63 is executed by the processor 61, the fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge in the embodiment is implemented. To avoid repetition, it is not described here one by one. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the fine-grained representation remote sensing image interpretation system based on Shearlet prior knowledge in the embodiment are implemented. To avoid repetition, it is not described here one by one.

[0070] The computer device 60 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will appreciate that Figure 5 This is only an example of the computer device 60 and does not constitute a limitation of the computer device 60. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0071] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, central processing units, graphics processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, data processing logic based on quantum computing, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0072] The memory 62 may be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 may also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device 60.

[0073] Furthermore, the memory 62 may include both an internal storage unit of the computer device 60 and an external storage device. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 may also be used to temporarily store data that has been output or is to be output.

[0074] Any reference to a memory, database or other medium used in the embodiments provided in the present application may include at least one of a non-volatile and a volatile memory. Non-volatile memory may include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical memory, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. Volatile memory may include a random access memory (RAM) or an external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0075] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0076] See also Figure 6 The terminal device 600 is an electronic device, and the electronic device is in the form of a general computing device. The components of the electronic device may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0077] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0078] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0079] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0080] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0081] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0082] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0083] We performed semantic segmentation, target detection, and small target tracking tasks. The baseline experiment used resnet-50. The downstream task processing heads were deeplabv3, faster-rcnn, and siamrpn++. We passed the shearlet layer three times. The experimental results are as follows:

[0084]

[0085]

[0086] In summary, the present invention provides a fine-grained representation remote sensing image interpretation method and system based on Shearlet prior knowledge, which embeds prior knowledge in a layered manner into a deep residual network, and can prove the effectiveness of this knowledge in multiple downstream tasks in the remote sensing field; in multiple tasks and multiple indicators, there is a relatively obvious improvement in the effect.

[0087] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0090] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0094] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge, characterized in that: The following steps are involved: S1. Preprocessing the original remote sensing image data; S2, using Shearlet transform to extract high-frequency sub-band information in multiple levels and multiple directions from the remote sensing image data pre-processed in step S1; using Laplacian pyramid and shearlet filter to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction; S3, construct high-frequency component prior based on Shearlet transform, adaptively fuse sub-bands at the same level through spatial attention mechanism, and generate feature maps with enhanced details; S4, construct a detail enhancement backbone network based on ResNet, and perform multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform in step S3 and the deep residual block obtained by the deep residual network; S5, using supervised learning to train the ResNet-based detail enhancement backbone network obtained in step S4, and repeatedly iterating using the remote sensing image data labeled in step S1 until the downstream task model performs stably on the validation set; S6. Use the detail enhancement backbone trained in step S5 and different detection heads for multiple tasks to output remote sensing image analysis results.

2. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 1 is characterized in that: In step S1, the preprocessing operations include noise removal, contrast adjustment and image cropping.

3. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 1 is characterized in that: Step S2 is specifically as follows: S201, input the pre-processed remote sensing image, and perform color space conversion on the image, from RGB to HSI; S202, applying the Laplace pyramid method to perform high- and low-frequency information stratification on the I component of the remote sensing image, and extracting the highest frequency component of the image; S203, for the extracted high-frequency components, use a Shearlet filter to obtain the highest frequency sub-bands in multiple directions; S204, down-sample the remaining low-frequency information to half of the previous one, apply the Laplacian pyramid method again to decompose the high- and low-frequency information, and obtain multi-directional sub-high-frequency sub-bands.

4. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 3 is characterized in that: In step S202, the Shearlet transformation details are as follows: in, and Represent the space vector and the translation vector respectively, is the scaling parameter, represents the shear factor, is the expansion matrix, is the shear matrix, is the transformation domain.

5. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 3 is characterized in that: Steps S201 to S204 are looped for three times until the highest frequency sub-band, the second highest frequency sub-band and the higher frequency sub-band information are obtained.

6. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 1 is characterized in that: Step S3 is specifically as follows: S301, by rearranging the dimensions of the high-frequency subbands extracted three times recursively, preparing for the input format C×H×W of the deep learning model, where C is the number of channels, H is the height, and W is the width; each channel C receives all the directional subbands corresponding to one extraction, indicating the directional information at a specific scale; S302, using a spatial attention module based on Shearlet transform to fuse the multi-directional sub-bands; S303, the spatial attention module uses a convolutional layer and the ReLU activation function to enhance the model's learning ability of spatial texture information; then the second convolutional layer deepens the feature fusion through the Sigmoid activation function to generate a compressed feature map focusing on the key information of the image. The final output Shearlet attention feature map has a dimension of 1×H×W.

7. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 1 is characterized in that: Step S4 is specifically as follows: S401, build ResNet50 as the basic network. ResNet50 has five CNN residual blocks, namely the zeroth layer, the first layer, the second layer, the third layer and the fourth layer; S402, adjusting the size of the feature map obtained in step S3, so that the highest frequency feature map and the second layer CNN residual block have the same size, the second high frequency feature map and the third layer CNN residual block have the same size, and the higher frequency feature map and the fourth layer CNN residual block have the same size; S403, multiply the high-frequency prior feature map adjusted in step S402 by the CNN deep residual block features output by the ResNet backbone network on a pixel-by-pixel basis at multiple scales to achieve effective integration of the two types of features.

8. The method for remote sensing image interpretation based on fine-grained representation of shearlet prior knowledge according to claim 1, characterized in that: Step S5 is specifically as follows: The neural network is trained through supervised learning and repeated iterations are performed using labeled remote sensing data; after the neural network training is completed, the performance of the neural network is evaluated.

9. The fine-grained representation remote sensing image interpretation method based on Shearlet prior knowledge according to claim 1 is characterized in that: Step S6 is specifically as follows: S601, using the trained detail enhancement backbone network to capture discriminative features in remote sensing images; S602, integrating multiple task detection heads with the detail enhancement backbone network to obtain a complete remote sensing multi-task processing model; S603, the model integrates the multi-task remote sensing image interpretation model obtained in step S602, and outputs the remote sensing image analysis results.

10. A fine-grained representation remote sensing image interpretation system based on Shearlet prior knowledge, characterized in that: include: Preprocessing module, which performs preprocessing operations on the original remote sensing image data; The Shearlet transform module uses Shearlet transform to extract high-frequency sub-band information in multiple levels and multiple directions from the pre-processed remote sensing image data. The Laplacian pyramid and shearlet filter are used to obtain high-frequency components and multi-directional high-frequency sub-bands after splitting them by direction. The spatial attention module constructs a high-frequency component prior based on Shearlet transform, adaptively fuses sub-bands at the same level through the spatial attention mechanism, and generates a feature map with enhanced details. The detail enhancement backbone network module builds a detail enhancement backbone network based on ResNet, and performs multi-scale and multi-level weighted fusion of the high-frequency component prior extracted by Shearlet transform and the deep residual block obtained by the deep residual network; The training and output module uses the ResNet-based detail enhancement backbone network trained by supervised learning, and uses labeled remote sensing image data for repeated iterations until the downstream task model performs stably on the validation set. The trained detail enhancement backbone and different detection heads for multiple tasks are used to output remote sensing image analysis results.

Images