A method, device, equipment and medium for extracting target from multi-source remote sensing image data

Abstract

The present disclosure provides a method, apparatus, device, and medium for extracting targets from multi-source remote sensing image data, which can be applied to the field of image processing technology. The multi-source remote sensing image data target extraction method includes: acquiring and preprocessing multi-source remote sensing image data; obtaining a target positioning frame based on the preprocessed data and a pre-trained network; obtaining boundary feature parameters based on the target positioning frame parameters; and inputting the target positioning frame and the boundary feature parameters into a SAM model to obtain a target extraction result. The present disclosure also provides a multi-source remote sensing image data target extraction apparatus, device, storage medium, and program product. The present disclosure has the advantages of enabling rapid, large-scale, automated extraction of offshore oil film targets, and achieving high accuracy in target extraction results.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically, to a method, apparatus, device, medium, and program product for extracting targets from multi-source remote sensing image data. Background Art

[0002] Multi-source remote sensing imagery provides rich information about surface features. By integrating and analyzing remote sensing images from different sources, we can more comprehensively and accurately understand various phenomena and changes on the Earth's surface. In recent years, marine oil spills caused by ship collisions and drilling platform leaks have become frequent, resulting in water pollution, harmful algal blooms, and severe economic losses. Therefore, it is necessary to promptly determine the location and extent of the oil spill, identify the oil type, and quantify the oil film thickness to mitigate the secondary damage caused by the accident.

[0003] Existing oil film extraction technologies can be categorized into two types. The first type is traditional oil film extraction methods, which mainly include the following steps: (1) dark spot detection, i.e., locating the possible location of the oil film; (2) feature extraction, i.e., manually or automatically screening the multi-dimensional features of the target, such as geometric features, physical features, texture features, and polarization features; (3) classifier classification, i.e., implementing pixel-level classification of the feature set, such as support vector machines, decision trees, and artificial neural networks. The second type is deep learning-based methods, such as CNN and VGG. Most of these models directly transfer models from the optical image field to the SAR image field, which loses phase information. In addition, a large number of samples need to be manually labeled, which is labor-intensive. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a multi-source remote sensing image data target extraction method, device, equipment, medium and program product with the ability to automatically predict target positioning parameters and achieve rapid extraction of offshore oil film targets under large-scale and automated conditions.

[0005] According to a first aspect of the present disclosure, a method for target extraction from multi-source remote sensing image data is provided, comprising acquiring and preprocessing multi-source remote sensing image data; acquiring a target positioning frame based on the preprocessed data and a pretrained network; acquiring boundary feature parameters based on the target positioning frame parameters; and inputting the target positioning frame and the boundary feature parameters into a SAM model to obtain a target extraction result.

[0006] According to an embodiment of the present disclosure, obtaining a target positioning frame based on the preprocessed data and the pretrained model includes: extracting basic features of the preprocessed data; outputting multi-scale features based on the basic features; obtaining a target candidate region based on the multi-scale features and a pretrained region candidate network; and obtaining a target positioning frame based on the target candidate region and a pretrained classification regression network.

[0007] According to an embodiment of the present disclosure, the training method of the region candidate network includes: taking multiple anchor frames of different sizes and proportions based on each feature map of the multi-scale features using training set data, and training the category probability and coordinate parameters of the anchor frames; the training loss function includes calculating the offset of the true value frame and the predicted frame relative to the anchor frame.

[0008] According to an embodiment of the present disclosure, the method includes: dividing the preprocessed image data into blocks, using a multi-head attention mechanism, through convolution and dimension adjustment to obtain image embedding information; the image embedding information is used to input the SAM model to obtain the extraction result.

[0009] According to an embodiment of the present disclosure, the extraction result is obtained by inputting the SAM model based on the target positioning frame and the boundary feature parameters, including: generating an extraction result after prediction of the image to be segmented based on the image embedding information, the target positioning frame and the boundary feature parameters through a multi-head attention mechanism and transposed convolution processing.

[0010] According to an embodiment of the present disclosure, the preprocessing includes polarization feature extraction, and the polarization feature extraction includes: fusing all channel scattering contribution values ​​of the image data to extract span features, and the polarization features are used for boundary feature parameter extraction.

[0011] Another aspect of an embodiment of the present disclosure provides a multi-source remote sensing image data target extraction device, including: a data set processing module, used to obtain and preprocess multi-source remote sensing image data; a target rapid positioning module, used to obtain a target positioning frame based on the preprocessed data and a pre-trained network; a SAM automatic segmentation module, used to obtain boundary feature parameters based on the parameters of the target positioning frame, and obtain a target extraction result based on the target positioning frame and the boundary feature parameters.

[0012] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0013] Another aspect of an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor is caused to perform the method described above.

[0014] Another aspect of an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0015] One or more of the above embodiments have the following beneficial effects:

[0016] The method disclosed herein acquires and preprocesses multi-source remote sensing image data; uses the preprocessed data and a pretrained network to obtain a target positioning frame; obtains boundary feature parameters based on the target positioning frame parameters; and finally, inputs the target positioning frame and boundary feature parameters into a SAM model to obtain target extraction results. The pretrained model automatically predicts target positioning parameters and uses boundary feature parameters to achieve rapid, automated, and high-accuracy extraction of offshore oil slick targets.

[0017] Using level 1 SLC data, including raw t-matrix data as input, integrated polarization feature extraction and boundary feature information acquisition enabled rapid target extraction of offshore oil slicks using multi-polarization and multi-temporal SAR. A target model was generated through transfer training based on a pre-made sample set. The target location parameters and boundary feature parameters predicted by the model were used to correct the SAM prediction results.

[0018] In addition, one or more embodiments of the present disclosure have the following beneficial effects:

[0019] (1) Meet the needs of large-scale, automated oil film extraction.

[0020] Existing technologies require manual or image processing methods to locate the target area in the acquired image before subsequent feature extraction and classification can be performed. However, the method proposed in the embodiments of the present disclosure, based on a pre-trained model, can automatically complete the entire process from data acquisition to target extraction, and can also extract targets from non-training source datasets.

[0021] (2) Solve the problem of inconsistent indicator verification among data.

[0022] Existing feature extraction methods, such as polarization, texture, and statistical features, often lack standardized methods. Alternatively, these features are filtered and combined using various methods before being fed into a classifier for classification. The proposed method automatically extracts high-level, multi-scale features and fuses these multiple scales, avoiding the phenomenon of feature changes caused by changes in the dataset.

[0023] (3) Reduce the manpower and energy consumption of large model labels.

[0024] Existing technologies that use large models to implement segmentation tasks require a large number of manually labeled samples, and the accuracy of the models when directly applied to downstream branch tasks is very poor. However, the method proposed in this disclosure extracts basic features using open-source models, and subsequent training only requires multi-scale high-level features, reducing the required sample size and training time. Furthermore, SAM technology is used to label a small number of samples, averaging one sample every 2 seconds, significantly improving speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0026] Figure 1 The application scenario diagram of a multi-source remote sensing image data target extraction method, apparatus, device, medium and program product according to an embodiment of the present disclosure is schematically shown.

[0027] Figure 2 A flowchart of a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0028] Figure 3 The flowchart of obtaining a data sample set according to an embodiment of the present disclosure is schematically shown.

[0029] Figure 4 A flowchart of a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0030] Figure 5 A flowchart of another method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0031] Figure 6 The figure schematically shows a BottleNeck structure diagram according to an embodiment of the present disclosure.

[0032] Figure 7 The figure schematically shows a multi-head intention mechanism diagram according to an embodiment of the present disclosure.

[0033] Figure 8 A diagram showing an implementation effect according to an embodiment of the present disclosure is shown.

[0034] Figure 9 Another implementation effect diagram according to an embodiment of the present disclosure is shown.

[0035] Figure 10 The following schematically shows a structural block diagram of a multi-source remote sensing image data target extraction device according to an embodiment of the present disclosure.

[0036] Figure 11 A block diagram of an electronic device suitable for implementing a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0037] It should be noted that, for the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the sizes of the overall / local structures or overall / local areas may be enlarged or reduced, that is, these drawings are not drawn according to the actual scale. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0039] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0040] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0041] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0042] Among existing oil film extraction technologies, traditional oil spill extraction methods suffer from high computational complexity, poor accuracy, and inconsistent indicator verification across different data sources. These methods are no longer able to meet the needs of large-scale, automated oil film extraction. While a large number of models have been successfully transplanted to oil film extraction using deep learning, the lack of sufficient standard datasets for downstream branch tasks still requires ground truth labels. Data labeling, in turn, often relies on manual annotation, which is time-consuming, labor-intensive, and unreliable. Furthermore, most of these two extraction methods involve cropping areas from the entire image for oil film extraction, lacking the ability to rapidly extract data from the entire process, from data acquisition to oil film segmentation.

[0043] In response to the aforementioned problems with existing technologies, the present invention proposes an oil film extraction method that at least partially addresses these technical issues. Based on the SAM universal segmentation model, this method automatically predicts target positioning model parameters through training, supplemented by boundary feature parameters. This method enables rapid, large-scale, automated extraction of offshore oil film targets. Multi-source data demonstrates the effectiveness of this method in practical applications, confirming the significant potential of the SAM model in oil film analysis.

[0044] An embodiment of the present disclosure provides a method for extracting targets from multi-source remote sensing image data, comprising the steps of: acquiring and preprocessing multi-source remote sensing image data; acquiring a target positioning frame based on the preprocessed data and a pre-trained network; acquiring boundary feature parameters based on the target positioning frame parameters; and inputting the target positioning frame and the boundary feature parameters into a SAM model to obtain a target extraction result.

[0045] Figure 1 The following diagram schematically illustrates an application scenario of the method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure. Figure 1 What is shown are merely examples to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0046] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0047] Users can use a first terminal device 101, a second terminal device 102, and a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only). The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers.

[0048] Server 105 can be a server that provides various services, such as a backend management server (for example only) that supports websites browsed by users using first terminal device 101, second terminal device 102, and third terminal device 103. The backend management server can analyze and process received user requests and other data, and provide feedback (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices. For example, server 105 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

[0049] In applications such as client applications and web applications (APPs), the client (i.e., front-end) and the server (i.e., back-end) can communicate data through network messages. For example, in the APP client, the server assembles the parameters that need to be obtained from the client and calls the network request method to send them to the server.

[0050] It should be noted that the method for extracting objects from multi-source remote sensing image data provided by the embodiments of the present disclosure can generally be executed by at least one of a terminal device or a server. Accordingly, the apparatus for extracting objects from multi-source remote sensing image data provided by the embodiments of the present disclosure can generally be provided in at least one of the terminal device or the server.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0052] The following will be based on Figure 1 The scene described by Figure 2-Figure 9 The target extraction method of multi-source remote sensing image data of the disclosed embodiment is described in detail.

[0053] Figure 2 A flowchart of a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0054] like Figure 2 As shown, the multi-source remote sensing image data target extraction method of this embodiment includes:

[0055] In operation S100 , multi-source remote sensing image data is acquired and pre-processed.

[0056] In operation S200 , a target positioning frame is obtained based on the preprocessed data and the pre-trained network.

[0057] In operation S300 , boundary feature parameters are obtained based on the parameters of the target positioning frame; and the target positioning frame and the boundary feature parameters are input into a SAM model to obtain an extraction result.

[0058] In this embodiment, the acquired multi-source remote sensing image data is first used to realize image preprocessing and generate a standard data set, the preprocessed data is input into a pre-trained target rapid positioning network, the predicted target positioning frame parameter information is output, and the boundary feature parameters are obtained based on the parameters of the target positioning frame. Finally, the target positioning frame parameter information and the boundary feature parameter information are fused and input into the SAM automatic segmentation module, and fine-tuned to obtain the predicted target extraction result.

[0059] In this embodiment, multi-source remote sensing data refers to remote sensing data acquired by different sensor types, platforms, or at different times, including optical remote sensing data, thermal infrared remote sensing data, microwave remote sensing data, and other types. In this embodiment, the multi-source remote sensing image data acquired can be single-look complex (SLC) image data from synthetic aperture radar (SAR). SAR SLC image data operates around the clock and in all weather conditions, unrestricted by factors such as cloud, rain, fog, and day or night. It also has a certain degree of penetration into vegetation and soil, enabling it to acquire information that optical remote sensing cannot. The two complement each other to provide more comprehensive ground feature information. This embodiment specifically uses level 1 SLC image data from the Sentinel-1 and Gaofen-3 satellites, which contain phase and amplitude information. SLC data refers to SAR data with a look number of 1 in both range and azimuth. It is raw, high-resolution data obtained by directly processing the radar's echo signals, preserving the signal's complete phase and amplitude information. It is stored in complex form, with each pixel consisting of a real and imaginary part. SLC data has the characteristics of high resolution, complete phase information, large data volume and obvious noise.

[0060] In this embodiment, preprocessing includes at least one processing technique, including orbit correction, radiometric calibration, polarization feature extraction, speckle filtering, and geocoding. Depending on the specific image data, one or more of these preprocessing techniques can be selected for use individually or in combination. Orbit correction utilizes precise orbit files to correct for differences between orbits. Radiometric calibration converts the brightness values ​​recorded by the sensor into backscatter coefficients. For example, in this embodiment, radiometric calibration of Gaofen-3 image data can be achieved using the following formula:

[0061]

[0062] In the above formula, is the radar backscatter coefficient (unit: dB); , I is the real part of the 1A-level product, Q is the imaginary part of the 1A-level product, and QV is the maximum value of the scene image before quantization, which can be obtained by parsing the metadata file.

[0063] Polarization feature extraction technology is mainly used to improve the accuracy of SAR image interpretation. This embodiment uses span features, but other multi-polarization features can also be used. The present invention fuses the scattering contribution values ​​of all channels to extract the span feature, as shown in the following formula:

[0064]

[0065] In the above formula, Represents the scattering amplitude value of different channels. Specifically, it also needs to be polarized. Let the original full polarization image before correction be M. Correct each group of pixels according to the following formula to obtain the relatively true ground object scattering matrix S after correction, where i and j represent the row and column numbers corresponding to the pixels. As shown in the following formula:

[0066]

[0067] Speckle filtering removes speckle noise from SAR signature images. This paper employs a polarimetric LEE method with a 5x5 window size to eliminate this effect. Specifically, edge template matching is first performed on the span signature image to select a directional window. Then, a local statistical filter is applied within this directional window to achieve speckle reduction.

[0068] Geocoding technology mainly converts SAR data of different phases from the slant range coordinate system to the geographic coordinate system, including image positioning and projection conversion, which will not be described in detail.

[0069] In some embodiments of the present application, the training method of the region candidate network includes: taking a plurality of anchor boxes of different sizes and proportions based on each feature map of the multi-scale feature using training set data, and training the category probability and coordinate parameters of the anchor boxes;

[0070] The training loss function involves calculating the offset of the true value box and the predicted box relative to the anchor box.

[0071] In this embodiment, the pre-trained network is trained based on a data sample set. The data sample set can be obtained in the following manner. Figure 3 The following schematically illustrates a flow chart for obtaining a data sample set according to an embodiment of the present disclosure. The flow chart for obtaining a data sample set specifically includes:

[0072] Operation S110 includes data acquisition, orbit correction, radiometric calibration, polarization feature extraction, speckle filtering, and geocoding preprocessing operations.

[0073] Operation S120 includes constructing a multi-scale pyramid model, implementing simulation, cropping and normalization, and enhancing data robustness by using rotation, folding and difference.

[0074] Operation S130 includes extracting embedded information of the data, implementing automatic labeling of training samples, and generating a data sample set in a standard format.

[0075] In this embodiment, the data set can be expanded by constructing a multi-scale pyramid model. By performing different degrees of smoothing filtering and downsampling operations on the original image, a series of images of different scales are obtained. Since images of different scales can capture the features of targets of different sizes, they can better cope with changes in target scale when performing feature matching, thereby improving the accuracy and robustness of matching. In this embodiment, the robustness of the data can also be enhanced by simulating, cropping and normalizing the image data, or by using methods such as rotation, folding, and interpolation. When the data diversity meets the conditions, the relevant image data expansion step can be omitted, or one or more combinations of the above expansion methods can be selected.

[0076] In this embodiment, data annotation tools can be used to extract embedded data information, automatically annotate training samples, and generate a data sample set in a standard format. The data sample set contains data, category, and coordinate information. The data sample set can be further divided into a training set, a validation set, and a test set, for example, in a ratio of 6:3:1.

[0077] In some embodiments of the present disclosure, obtaining a target positioning frame based on the preprocessed data and the pretrained model includes: extracting basic features of the preprocessed data; outputting multi-scale features based on the basic features; obtaining a target candidate region based on the multi-scale features and a pretrained region candidate network; and obtaining a target positioning frame based on the target candidate region and a pretrained classification regression network.

[0078] Figure 4 The flowchart of a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown. Figure 4 As shown, in this embodiment, the target extraction method of multi-source remote sensing image data includes:

[0079] In operation S210, ResNet is used as the backbone model to extract basic data features and output the last layer feature map of each stage.

[0080] In operation S220 , sampling and fusion processing of features at different levels are implemented based on the feature map of each stage, and a multi-scale feature map is output.

[0081] In operation S230 , anchor boxes of different sizes and scales are constructed based on the selected multi-scale feature map, and offsets are obtained through a pre-trained region proposal network to obtain suitable candidate boxes, i.e., target candidate regions.

[0082] In operation S240 , the candidate box is assigned to a suitable feature layer to obtain the region of interest, and then the final target parameters are obtained through a pre-trained classification regression network, that is, the target positioning box is obtained.

[0083] In this embodiment, the residual network (ResNet) is used to extract the basic features of the data set. The disclosed embodiment uses ResNet as the backbone model to solve the problem of gradient disappearance. In this embodiment, there are specifically five stages, each of which is composed of multiple BottleNeck structures stacked together, as shown in the schematic diagram. Figure 6 shown.

[0084] In this embodiment, multi-scale features are output based on the basic features. Specifically, the feature maps of different stages output by the network model initialization unit are further processed, that is, the features output by the last residual block layer are extracted respectively, recorded as {C2, C3, C4, C5, C6}, and the last four features are each subjected to 1*1 convolution to obtain {M3, M4, M5, M6}, and then M i The feature layers are upsampled and convolved as {N3, N4, N5, N6}, and then the N3, N4, and N5 feature layers are downsampled by 2x and 4x, fused, and convolved as {P3, P 41 ,P 42 ,P 51 ,P 52 ,P6}, P 41 and P 42 、P 51 and P 52 After weighting, {P2, P3, P4, P5, P6} multi-scale features are finally generated. This embodiment extracts multi-scale features and fuses them to avoid the phenomenon that features change with changes in the dataset.

[0085] In some embodiments of the present disclosure, the training method of the region candidate network includes: taking multiple anchor frames of different sizes and proportions based on each feature map of the multi-scale features using training set data, and training the category probability and coordinate parameters of the anchor frames respectively; the training loss function is to calculate the offset of the true value frame and the predicted frame relative to the anchor frame.

[0086] The region proposal network is used to recommend candidate target regions. The training of the region proposal network first involves taking multiple anchor boxes of different sizes and scales on each feature map output by the multi-scale feature extraction unit, and then training the class probabilities and coordinate parameters of the anchor boxes separately. In this embodiment, specifically, four anchor boxes of different sizes and scales are taken on each feature map output by the multi-scale feature extraction unit. The loss during training is calculated by calculating the offset of the ground truth box and the predicted box relative to the anchor box. The specific calculation formula is as follows:

[0087] , .

[0088] , .

[0089] , .

[0090] , .

[0091]

[0092] In the above formula, 、 and Represent the center coordinates, width and height of the predicted box, anchor box and true value box respectively.

[0093] In some embodiments of the present disclosure, a pre-trained classification and regression network is used to re-screen the prediction boxes to obtain the final parameters of the target. Specifically, the feature map output by the multi-scale feature extraction unit and the prediction boxes output by the region candidate unit are combined to obtain ROIs. The allocation of prediction boxes to feature layers is determined by the following formula.

[0094]

[0095] In the above formula, k is the feature layer of the multi-scale feature extraction unit, k0 is the feature layer of the initialization model unit mapped to the input image size, w and h represent the size of the prediction box, and the symbol Represents the lower bound of the internal value.

[0096] The basic features of the method proposed in the embodiment of the present disclosure are extracted using an open source model. Subsequently, only multi-scale high-level features need to be trained, reducing the required sample size and training time. SAM technology is used when labeling a small number of samples, with an average of one sample every 2 seconds, greatly improving the speed.

[0097] In some embodiments of the present disclosure, the preprocessed image data is divided into blocks, each depth vector of each block is assigned a position code, and a multi-head attention mechanism is used to obtain image embedding information through convolution and dimension adjustment; the image embedding information is used to input the SAM model to obtain the extraction result.

[0098] In some embodiments of the present disclosure, the extraction result is obtained by inputting the SAM model based on the target positioning frame and the boundary feature parameters, including: generating an extraction result after prediction of the image to be segmented based on the image embedding information, the target positioning frame and the boundary feature parameters through a multi-head attention mechanism and transposed convolution processing.

[0099] The SAM model is used to achieve pixel-level segmentation of objects in image data. The SAM model mainly consists of an image encoder, a hint encoder, and a mask decoder.

[0100] Figure 5 The flowchart of a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown. Figure 5 As shown, the multi-source remote sensing image data target extraction method of this embodiment includes:

[0101] In operation S310, the preprocessed image data is subjected to convolution and position encoding, and the multi-head attention mechanism is used to enhance the information, and the image embedding information is output after the dimension is adjusted.

[0102] In operation S320 , the image embedding information, ie, the object positioning box parameters and the boundary feature parameters are fused to form the input of the mask decoder.

[0103] In operation S330, after processing such as a multi-head attention mechanism and transposed convolution, a target extraction result after prediction of the image to be segmented is generated.

[0104] In this embodiment, the image encoder part divides the image into multiple small blocks through the convolution kernel, then assigns a position code to each depth vector of each block, enhances the feature vector information using a multi-head attention mechanism, and finally adjusts the dimension through convolution and batch processing to obtain image embedding information.

[0105] Figure 7 Schematically shows a schematic diagram of a multi-head intention mechanism according to an embodiment of the present disclosure. Figure 7 As shown in the figure, the multiple blocks on the left (labeled 1, 2, 3, ..., N) represent the image data of the input sequence. The multi-head attention mechanism inputs the input in parallel to multiple independent "heads" (represented as head1 to headN in the figure). Each head has its own independent weight matrix. For each head, the input elements are operated with multiple weight matrices separately. Here W q 、Wk Used to calculate the query (Query) and key (Key) related representation, W v Used to calculate the representation related to the value. The normalized (Softmax) attention score is weighted and summed with the value vector to obtain the attention output of each head. The output of all heads (Attention Map1 to Attention MapN) is spliced ​​(Concat operation) to obtain a vector that integrates the information of multiple heads. The spliced ​​vector is then passed through a weight matrix W o Perform linear transformation to obtain the final output.

[0106] In this embodiment, the hint encoder embeds hint information into the model using methods such as points, boxes, and masks. This embodiment uses a fusion approach as the input to the encoding unit, specifically consisting of two parts: the first part is the output of the classification and regression network, and the second part is the boundary feature parameters, which are intended to supervise the model's prediction results.

[0107] In this embodiment, the boundary feature parameter extraction process is as follows: First, the positioning frame parameters output by the classification regression network are used as a window, and the span feature map within the window is segmented and edge detected to define the boundary buffer area. Several points are randomly selected in the buffer area to form the boundary feature parameters {Q i 、T i The following formula is used to determine whether the feature sets Q and T are separable. If not, the boundary feature parameters are left blank. If they are, the boundary feature parameters are fused with the positioning frame and input into the SAM mask decoding unit, supplemented by a correction to the oil film extraction boundary. This embodiment uses boundary feature parameters to obtain points containing different targets. Both point information and frame information are input into the SAM model to jointly extract the target.

[0108]

[0109] In the above formula, and are the mean and variance of different sets respectively.

[0110] In this embodiment, the mask decoder uses a lightweight mask decoder to predict the mask of the image to be segmented. First, the output of the hint encoding unit and the output of the image encoding unit are combined through a multi-head attention mechanism and transposed convolution to generate the extraction result of the predicted image to be segmented.

[0111] In one embodiment of the present disclosure, a multi-source remote sensing image data target extraction method is used to quickly extract offshore oil slicks. The implementation process and results are as follows.

[0112] (1) Preprocessing stage: A total of more than 50 multi-polarization images of the accident area were obtained. After the image data were preprocessed with orbit correction, radiation calibration, polarization feature extraction, speckle filtering and geocoding, the subsequent process was divided into two branches. The first branch randomly extracted a small number of images from the dataset, and generated a standard dataset through dataset expansion and sample set generation operations to support subsequent training and verification of the model. The second branch served as a data pool for rapid prediction of the entire process after model training was completed, which was used to verify the automated extraction of marine targets and to verify that targets could also be extracted from non-training source datasets.

[0113] (2) Training phase: Obtain the first branch standard dataset of the preprocessing phase, extract the basic features and multi-scale features of the dataset, and then use the region candidate network and classification regression network to screen the prompt boxes of the target candidate regions respectively. The final target positioning box position parameters are obtained through training and parameter adjustment. Specifically, the loss during network training is as follows:

[0114]

[0115]

[0116]

[0117] In the above formula, is the category loss, is the regression loss; is the classification probability predicted by the anchor box, is the category label, which is 1 for positive samples and 0 for negative samples; Represents the offset of the prediction box in the region proposal network unit or classification regression unit, Indicates the offset of the two types of prediction boxes relative to the true value box; represents the number of samples in the mini-batch, Represents the number of prediction boxes, is the weight balancing parameter.

[0118] After model training, spot check data is used to verify the model. The verification accuracy is shown in the following table:

[0119] Table 1. Sea target extraction accuracy

[0120] Metric Oil Precision 0.960374 Recall 0.978448 F1 value 0.969326

[0121] (3) Prediction stage: The second branch data set in the preprocessing stage is input into the image encoder to obtain image embedding information, and the parameters obtained by training the prompt encoding unit and the boundary feature parameters together constitute the prompt information. The embedded information and the prompt information are input into the mask decoding unit to generate the predicted extraction result.

[0122] Figure 8 A diagram showing an implementation effect according to an embodiment of the present disclosure is shown. Figure 9 Another implementation effect diagram according to an embodiment of the present disclosure is shown. Figure 8-Figure 9 It can be seen that the oil film target positioning in the image is in line with expectations. This embodiment achieves rapid extraction of offshore oil film targets under large-scale and automated conditions.

[0123] Based on the above multi-source remote sensing image data target extraction method, the present disclosure also provides a multi-source remote sensing image data target extraction device. Figure 10 The device is described in detail.

[0124] like Figure 10 As shown, the multi-source remote sensing image data target extraction device 1200 of this embodiment includes a data set processing module 1201 , a target rapid positioning module 1202 , and a SAM automatic segmentation module 1203 .

[0125] The data set processing module 1201 is used to obtain and pre-process multi-source remote sensing image data;

[0126] A target rapid positioning module 1202 is configured to obtain a target positioning frame based on the preprocessed data and the pretrained network;

[0127] The SAM automatic segmentation module 1203 is configured to obtain boundary feature parameters based on the parameters of the target positioning frame, and obtain extraction results based on the target positioning frame and the boundary feature parameters.

[0128] In some embodiments of the present disclosure, the dataset processing module 1201 may include: a preprocessing unit for implementing technical processing such as data acquisition, orbit correction, radiometric calibration, polarization feature extraction, speckle filtering, and geocoding. The dataset processing module 1201 may also include: a dataset expansion unit for expanding the training dataset by constructing a multi-scale pyramid model, simulating, cropping, and normalizing the imagery, and enhancing the robustness of the data using methods such as rotation, folding, and interpolation; and a sample set generation unit for automatically labeling training samples by extracting embedded information from the data and generating a sample set in a standard format.

[0129] In some embodiments of the present disclosure, the preprocessing unit is used to extract polarization features of image data, including: fusing all channel scattering contribution values ​​of the image data to extract span features, which will not be repeated here.

[0130] In some embodiments of the present disclosure, the target rapid positioning module 1202 includes: a network model initialization unit for extracting basic features of the preprocessed data; a multi-scale feature extraction unit for outputting multi-scale features based on the basic features; a region candidate network unit for obtaining a target candidate region based on the multi-scale features and a pre-trained region candidate network; and a classification regression unit for obtaining a target positioning frame based on the target candidate region and a pre-trained classification regression network.

[0131] In some embodiments of the present disclosure, the SAM automatic segmentation module 1203 includes: an image encoding unit, which is used to obtain image embedding information based on preprocessed image data using a multi-head attention mechanism through convolution and dimension adjustment; a prompt encoding unit; which is used to embed prompt information into the SAM model using points, boxes, masks, etc., and obtain boundary feature parameters based on the parameters of the target positioning box; a mask decoding unit, which is used to generate an extraction result after prediction of the image to be segmented based on the image embedding information, the target positioning box and the boundary feature parameters after multi-head attention mechanism and transposed convolution processing.

[0132] For the parts not mentioned in the apparatus part, they can be understood with reference to the various embodiments of the above-mentioned method. That is, the apparatus part includes modules for executing the various steps of any one of the method embodiments described above. In addition, the implementation methods, technical problems solved, functions achieved, and technical effects achieved of each module / unit / subunit, etc. in the apparatus part embodiment are respectively the same or similar to the implementation methods, technical problems solved, functions achieved, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0133] According to an embodiment of the present disclosure, at least one of the dataset processing module 1201, the target rapid positioning module 1202, and the SAM automatic segmentation module 1203 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the dataset processing module 1201, the target rapid positioning module 1202, and the SAM automatic segmentation module 1203 can be at least partially implemented as a computer program module, which can perform the corresponding function when executed.

[0134] Figure 11 A block diagram of an electronic device suitable for implementing a method for extracting targets from multi-source remote sensing image data according to an embodiment of the present disclosure is schematically shown.

[0135] like Figure 11 As shown, the electronic device 1300 according to an embodiment of the present disclosure includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage unit 1308 into a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0136] Various programs and data required for the operation of the electronic device 1300 are stored in the RAM 1303. The processor 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. The processor 1301 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1302 and / or the RAM 1303. It should be noted that the programs may also be stored in one or more memories other than the ROM 1302 and the RAM 1303. The processor 1301 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0137] According to an embodiment of the present disclosure, electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to bus 1304. Electronic device 1300 may also include one or more of the following components connected to I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1308 including a hard disk; and a communication section 1309 including a network interface card such as a LAN card or modem. Communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to I / O interface 1305 as needed. Removable media 1311, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1310 as needed, so that computer programs read from the removable media can be installed into storage section 1308 as needed.

[0138] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiments of the present disclosure.

[0139] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1301. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0140] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal over a network medium, downloaded and installed via the communication portion 1309, and / or installed from removable media 1311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0141] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309 and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0142] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] Those skilled in the art will appreciate that the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of the present disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of the present disclosure. All such combinations and / or couplings fall within the scope of the present disclosure.

[0145] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for extracting targets from multi-source remote sensing image data, characterized in that: Including steps: Acquire multi-source remote sensing image data and preprocess it; Obtain the target positioning frame based on the preprocessed data and pre-trained network; Obtaining boundary feature parameters based on the target positioning frame parameters; Input the target positioning frame and the boundary feature parameters into the SAM model to obtain a target extraction result; The step of obtaining a target positioning frame based on the preprocessed data and the pretrained network includes: Extracting basic features of the preprocessed data; outputting multi-scale features based on the basic features; Obtaining a target candidate region based on the multi-scale features and a pre-trained region proposal network; Obtaining a target positioning frame based on the target candidate region and a pre-trained classification regression network; The obtaining of boundary feature parameters based on the target positioning frame parameters includes: Using the positioning frame parameters output by the classification regression network as a window, the span feature map in the window is segmented and edge detected to define the boundary buffer area; several points are randomly selected in the buffer area to form the boundary feature parameters {Q i 、T i }, if the feature sets Q and T are separable, the points are used as boundary feature parameters for input into the SAM model; The span feature is obtained by fusing and extracting the scattering contribution values ​​of all channels of the image data.

2. The method according to claim 1, characterized in that The training method of the region candidate network includes: Taking a plurality of anchor frames of different sizes and proportions based on each feature map of the multi-scale features using training set data, and training the category probability and coordinate parameters of the anchor frames; The training loss function involves calculating the offset of the true value box and the predicted box relative to the anchor box.

3. The method according to claim 1, characterized in that The method comprises: The preprocessed image data is divided into blocks, and the multi-head attention mechanism is used to obtain image embedding information through convolution and dimension adjustment; the image embedding information is used to input the SAM model to obtain the extraction result.

4. The method according to claim 1, wherein The step of inputting the target location frame and the boundary feature parameters into the SAM model to obtain a target extraction result includes: Based on the image embedding information, the target positioning frame and the boundary feature parameters, after multi-head attention mechanism and transposed convolution processing, an extraction result after the image to be segmented is generated.

5. The method according to claim 1, wherein The preprocessing includes polarization feature extraction, which includes: fusing all channel scattering contribution values ​​of image data to extract span features, and the polarization features are used for boundary feature parameter extraction.

6. A multi-source remote sensing image data target extraction device, characterized in that: The device comprises: Dataset processing module, used to acquire and preprocess multi-source remote sensing image data; The target fast positioning module is used to obtain the target positioning frame based on preprocessed data and pre-trained network; A SAM automatic segmentation module is used to obtain boundary feature parameters based on the parameters of the target positioning frame, and obtain target extraction results based on the target positioning frame and the boundary feature parameters; The step of obtaining a target positioning frame based on the preprocessed data and the pretrained network includes: Extracting basic features of the preprocessed data; outputting multi-scale features based on the basic features; Obtaining a target candidate region based on the multi-scale features and a pre-trained region proposal network; Obtaining a target positioning frame based on the target candidate region and a pre-trained classification regression network; The obtaining of boundary feature parameters based on the target positioning frame parameters includes: Using the positioning frame parameters output by the classification regression network as a window, the span feature map in the window is segmented and edge detected to define the boundary buffer area; several points are randomly selected in the buffer area to form the boundary feature parameters {Q i 、T i }, if the feature sets Q and T are separable, the points are used as boundary feature parameters for input into the SAM model; The span feature is obtained by fusing and extracting the scattering contribution values ​​of all channels of the image data.

7. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images