A red mud space extraction method combining prompt learning with a dual-branch network model

Abstract

The present invention discloses a red mud spatial extraction method combined with a prompt learning dual-branch network model, the method steps are as follows: collecting satellite remote sensing images and semantic visual information of land objects associated with the location of the study area, building a dual-branch network model including a scene classification branch and a semantic segmentation branch, the scene classification branch classifies the satellite remote sensing images and the semantic visual information of the land objects, extracts multi-layer features and identifies red mud to obtain red mud heat map information; the semantic segmentation branch uses the red mud heat map information to perform feature prompting, fine multi-layer feature fusion extraction, and segmentation processing on the satellite remote sensing images. The present invention uses POI semantic information and spectral index as prompt information for the scene classification branch to enhance the feature description capability of the red mud target, and at the same time uses the heat map generated by the scene classification branch as prompt information to assist the semantic segmentation branch processing, effectively solving the problem of difficult red mud identification and significantly improving the extraction efficiency and accuracy of red mud distribution information.

Description

Technical Field

[0001] The present invention relates to the field of aluminum waste image recognition, and in particular to a red mud space extraction method combining a prompt learning dual-branch network model. Background Art

[0002] In the aluminum industry, alumina is mostly extracted from bauxite. Common methods for extracting alumina include the Bayer process, the sintering process, and the combined process (a combination of the Bayer and sintering processes). The Bayer process uses a caustic soda solution to dissolve the alumina in bauxite, leaving red mud as the residual residue after filtration. The sintering process involves sintering bauxite with soda ash, lime, and other ingredients to leach the alumina, also producing red mud. Red mud is a solid waste generated during aluminum production. Because it often contains alkaline substances and heavy metals, it can pose a serious threat to the environment and human health. Therefore, regulating its distribution and inventory is crucial. Currently, in areas surrounding alumina production, individual red mud dumps can span hundreds to thousands of acres, with piles reaching heights exceeding 50 meters. Globally, approximately 120 to 150 million tons of red mud are generated annually, and the cumulative stockpile has exceeded 5 billion tons. Red mud dumps are widely distributed, and how to accurately monitor and effectively manage them has become a crucial part of environmental protection. Currently, the main way to obtain information on red mud distribution and storage is for aluminum production units to report it themselves (which can lead to problems such as concealment, misreporting, or delayed reporting, hindering effective supervision); or for regulatory authorities to arrange for personnel to conduct on-site surveys and estimates. However, some red mud dumps occupy small areas, while others are far away and scattered, and the range of red mud reserves is constantly changing. Field surveys and estimates are not only time-consuming and labor-intensive, but also difficult for regulatory authorities to grasp the actual distribution of red mud in a timely and accurate manner. With the continuous development of satellite remote sensing technology and object extraction methods, traditional object extraction methods are mainly used in areas where objects are relatively concentrated, such as buildings, rivers, vegetation, etc. in the study area (the study area selected is an area with relatively concentrated objects). Red mud is usually distributed in a point-like or small-scale scattered manner. Traditional object extraction methods are difficult to achieve high-precision distinction and also consume a lot of time. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems of accuracy and efficiency in red mud information recognition in remote sensing images, and to provide a red mud space extraction method combined with a prompt learning dual-branch network model. It adopts dual-branch parallel processing, corresponding layer jump bridging and information prompt fusion to achieve efficient and accurate recognition of the red mud spatial range.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A red mud spatial extraction method combined with a prompt learning dual-branch network model has the following steps: collecting satellite remote sensing images and semantic visual information of land objects associated with the study area, building a dual-branch network model including a scene classification branch and a semantic segmentation branch, the scene classification branch classifies the satellite remote sensing images and the semantic visual information of land objects, extracts multi-layer features and identifies red mud to obtain red mud heat map information; the semantic segmentation branch uses the red mud heat map information to perform feature prompting, fine multi-layer feature fusion extraction, segmentation processing on the satellite remote sensing images, and outputs the red mud segmentation result map.

[0006] In order to better realize the present invention, the satellite remote sensing images of the study area are associated with the semantic visual information of the land features according to the coordinate information and the multi-source data points are associated. The semantic visual information of the land features includes the POI information of red mud related factories and red mud storage sites.

[0007] Preferably, the classification multi-layer feature extraction method of the scene classification branch includes:

[0008] The Bert large language model is used to extract features from the semantic visual information of the ground objects, and then the local semantic gate and global semantic gate composed of perceptrons are input for feature extraction respectively;

[0009] The spectral index of satellite remote sensing images is calculated pixel by pixel to generate spectral index images. Feature extraction is performed using a local visual gate composed of an IRB neural network module and a global visual gate composed of a MiT neural network module.

[0010] The scene classification branch extracts several layers of classification features from satellite remote sensing images. The global features of each layer of classification features are fused with the output features of the global semantic gate and the output features of the global visual gate to indicate and enhance the corresponding position points. The local features of each layer of classification features are fused with the output features of the local semantic gate and the output features of the local visual gate to indicate and enhance the corresponding position points.

[0011] The scene classification branch outputs several layers of extracted and processed classification multi-layer features, and uses a classifier composed of a global maximum pooling module, a global average pooling module and a perceptron to identify red mud, make a judgment on whether it is red mud, and obtain red mud thermal map information.

[0012] Preferably, the semantic segmentation branch fine multi-layer feature extraction method includes:

[0013] Obtain prompt information from the red mud heat map information, including whether the pixel is red mud, whether the pixel combination area contains red mud, and the red mud density, intensity, and distribution. The semantic segmentation branch first relies on the prompt information of the red mud heat map information to quickly judge and process the satellite remote sensing image.

[0014] The semantic segmentation branch performs several layers of fine feature extraction on satellite remote sensing images. The global features of each layer of fine features are first globally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the global prompts, and the global feature map of the red mud heat map information is used to enhance the fusion of the corresponding position points; the local features of each layer of fine features are first locally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the local prompts, and the local feature map of the red mud heat map information is used to enhance the fusion of the corresponding position points.

[0015] Preferably, the classification multi-layer features of the scene classification branch include four layers of classification features, the fine multi-layer features of the semantic segmentation branch include four layers of fine features, and the red mud heat map information of the scene classification branch includes four layers of red mud heat map information; the four-layer structure of the scene classification branch and the four-layer structure of the semantic segmentation branch are jump-bridged in a one-to-one correspondence, the scene classification branch jump-bridges the corresponding red mud heat map information to the corresponding layer of the semantic segmentation branch and uses it for feature extraction prompts and enhanced fusion, and the scene classification branch jump-bridges the corresponding classification features to the corresponding layer of the semantic segmentation branch and uses it for enhanced fusion of fine features.

[0016] Preferably, the method for enhancing and fusing the local features of each layer of fine features using the local feature map of the red mud heat map information to the corresponding position points includes:

[0017] The local features of the fine features and the local feature map of the red mud heat map information are input into the first cross-attention module for attention weight fusion to obtain the local features of the enhanced fused fine features. Then the local features of the enhanced fused fine features and the local features of the fine features are input into the second cross-attention module for attention weight fusion to obtain the local features of the secondary enhanced fused fine features.

[0018] Preferably, each layer of classification features of the scene classification branch uses a local and global feature extractor to extract local features and global features. The local and global feature extractor includes an IRB neural network module, a local prompt module, a MiT neural network module, a global prompt module and a sparse fusion module. The IRB neural network module and the local prompt module constitute a local feature extraction branch, and the MiT neural network module and the global prompt module constitute a global feature extraction branch. The local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes:

[0019] The local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global prompt module to obtain the global prompt information of the global semantic gate and the global visual gate to perform global prompts of the red mud. The MiT neural network module extracts global features based on the global prompts and uses the global feature maps of the global semantic gate and the global visual gate to enhance the fusion of corresponding position points; the local feature extraction branch uses the local prompt module to obtain the local prompt information of the local semantic gate and the local visual gate to perform local prompts of the red mud. The IRB neural network module extracts local features based on the local prompts and uses the local feature maps of the local semantic gate and the local visual gate to enhance the fusion of corresponding position points; then the sparse fusion module performs sparse and fusion processing of the features, and outputs them after 1×1 convolution processing.

[0020] Preferably, the semantic segmentation branch uses a local and global feature extractor to extract fine features of each layer of satellite remote sensing images. The local and global feature extractor includes an IRB neural network module, a local prompt module, a MiT neural network module, a global prompt module and a sparse fusion module. The IRB neural network module and the local prompt module constitute a local feature extraction branch, and the MiT neural network module and the global prompt module constitute a global feature extraction branch. The local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes:

[0021] The local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global prompt module to obtain the global prompt information of the red mud heat map information for global prompting of the red mud. The MiT neural network module extracts global features based on the global prompt and uses the global feature map of the red mud heat map information to enhance the fusion of the corresponding position points; the local feature extraction branch uses the local prompt module to obtain the local prompt information of the red mud heat map information for local prompting of the red mud. The IRB neural network module extracts local features based on the local prompt and uses the local feature map of the red mud heat map information to enhance the fusion of the corresponding position points; then the sparse fusion module performs sparse and fusion processing of the features, and outputs them after 1×1 convolution processing.

[0022] Preferably, position-associated satellite remote sensing image samples and ground object semantic information samples are constructed and input into a dual-branch network model for model training. The scene classification branch classifies the satellite remote sensing image samples and the ground object semantic information samples, extracts multi-layer features and identifies red mud to obtain sample red mud thermal map information A; the semantic segmentation branch performs fine multi-layer feature extraction and segmentation processing on the satellite remote sensing image samples and outputs the red mud segmentation result map. The semantic segmentation branch obtains the sample red mud thermal map information B based on the red mud segmentation result map, and sets consistency parameter constraint loss for the sample red mud thermal map information A and B.

[0023] Preferably, the total loss of the dual-branch network model is obtained by weighted calculation of the scene classification branch loss, the semantic segmentation branch loss and the consistency constraint loss. The consistency constraint loss is the difference loss between the red mud heat map of the scene classification branch and the red mud heat map of the semantic segmentation branch. The scene classification branch loss and the semantic segmentation branch loss both use cross entropy loss.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] (1) The scene classification branch of the dual-branch network model of the present invention classifies satellite remote sensing images and semantic visual information of land objects, extracts multi-layer features and identifies red mud, and obtains red mud heat map information. The scene classification branch is used to quickly determine whether the image contains red mud and obtain multi-layer structure red mud heat map information; the semantic segmentation branch uses the red mud heat map information to perform feature prompts, fine multi-layer feature fusion extraction and recognition segmentation processing on the satellite remote sensing image, fully utilizes the red mud heat map information and assists in prompting fine feature extraction and feature fusion. The semantic segmentation branch contains richer semantics, visual prompts and feature information, which improves the accuracy of the semantic segmentation branch; the dual branches are used for parallel processing, corresponding layer jump bridging and information prompt fusion to achieve efficient and accurate recognition of the spatial range of red mud.

[0026] (2) The scene classification branch of the present invention utilizes POI semantic information and spectral index visual information to enhance the feature description of red mud targets in the dual-branch network model and improve the overall recognition accuracy; the scene classification branch quickly outputs red mud thermal map information to provide auxiliary prompts and enrich information for the semantic segmentation branch, thereby improving the red mud recognition accuracy and efficiency, and significantly improving the semantic segmentation branch's ability to recognize red mud targets and red mud spatial distribution.

[0027] (3) The scene classification branch of the present invention gives a macroscopic judgment of red mud, and the semantic segmentation branch performs in-depth fine-grained segmentation. The dual-branch network model effectively realizes fusion processing, which improves the accuracy and speed of red mud target spatial range extraction. By designing multiple loss functions to jointly supervise model training, the training limitations of small sample data are effectively alleviated, and the robustness of the model is improved.

[0028] (4) The present invention enhances the feature description capability of red mud targets by introducing POI semantic information and spectral index as prompt information of the scene classification branch. At the same time, the heat map generated by the scene classification branch is used as prompt information to assist the semantic segmentation branch processing, which effectively solves the problem of red mud identification difficulties. The present invention can significantly improve the extraction efficiency and accuracy of red mud distribution information, providing an efficient and reliable technical means for environmental protection departments and production enterprises, and providing important support for environmental monitoring and scientific decision-making.

[0029] (5) The present invention combines the collaborative dual-branch design of the scene classification branch and the semantic segmentation branch, which can determine the presence of red mud from a macroscopic scale and simultaneously extract its spatial range at the pixel level. If the scene classification branch indicates that red mud does not exist in the image area, the semantic segmentation branch does not need to be processed, which significantly improves the running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Flow chart of the method for spatial extraction of red mud in the embodiment;

[0031] Figure 2 Schematic diagram of the principle of the red mud spatial extraction method in the embodiment;

[0032] Figure 3 Schematic diagram of the principle of the sparse fusion module in the embodiment;

[0033] Figure 4 Schematic diagram of the principle of local and global feature extractors in the scene classification branch;

[0034] Figure 5 Schematic diagram of the principle of local and global semantic gates in the scene classification branch;

[0035] Figure 6 Schematic diagram of the principle of local and global vision gates in the scene classification branch;

[0036] Figure 7 This is a schematic diagram illustrating the principle of information prompting and fusion processing of local features in the embodiment;

[0037] Figure 8 Schematic diagram of the principles of local and global feature extractors in the semantic segmentation branch. DETAILED DESCRIPTION

[0038] Below in conjunction with embodiment, the present invention is described in further detail:

[0039] Example

[0040] like Figure 1 As shown in FIG, a red mud space extraction method combining a prompt learning dual-branch network model is proposed, and the method steps are as follows:

[0041] S1. Collect satellite remote sensing images and semantic visual information of the study area. The satellite remote sensing images of the study area and the semantic visual information of the study area are correlated based on coordinate information and multi-source data point association. The semantic visual information includes point of interest (POI) information (also known as semantic information) and visual information. In this embodiment, the visual information is a spectral index, calculated by acquiring the radiation values ​​of different bands in the electromagnetic spectrum using the formula: (red band − blue band) / (red band + blue band). In this embodiment, the POI information preferably includes points of interest (POIs) including red mud-related factories and red mud storage sites. This POI information is combined with survey data of the study area, information about the surrounding environment, and factory information (including information on alumina plants, bauxite processing plants, tailings ponds, etc.).

[0042] S2. Build a dual-branch network model including a scene classification branch and a semantic segmentation branch. The scene classification branch classifies satellite remote sensing images and semantic visual information of land objects, extracts multi-layer features, identifies red mud, and obtains red mud thermal map information. In this embodiment, ensure that each POI point is located at the center of the corresponding remote sensing image block, thereby forming effective input data (remote sensing image + POI + spectral index) to input the scene classification branch, where the remote sensing image size is a rectangular range with a side length of 1 km centered on the POI. Figure 2 As shown, in this embodiment, the classification multi-layer features of the scene classification branch include four layers of classification features , the semantic segmentation branch’s fine multi-layer features include four layers of fine features The red mud heat map information of the scene classification branch contains four layers of red mud heat map information. The four-layer structure of the scene classification branch and the four-layer structure of the semantic segmentation branch are jump-bridged one by one. The scene classification branch jumps and bridges the corresponding red mud heat map information to the corresponding layer of the semantic segmentation branch and uses it for feature extraction prompts and enhanced fusion. The scene classification branch jumps and bridges the corresponding classification features to the corresponding layer of the semantic segmentation branch and uses it for enhanced fusion of fine features. The main task of the scene classification branch is to determine whether the image block (a block area in the image) contains red mud. The main task of the semantic segmentation branch is to perform pixel-level red mud feature extraction and recognition in the image (including image blocks).

[0043] like Figure 2 As shown, this embodiment takes the scene classification branch's classification multi-layer features including four layers of classification features as an example, and the scene classification branch's classification multi-layer feature extraction method includes:

[0044] S21. Use the BERT language model (BERT is the abbreviation of Bidirectional Encoder Representations from Transformers) to extract features from the POI information (or semantic information) of the semantic visual information of the ground objects. Then input the features into the local semantic gate composed of perceptrons and the global semantic gate composed of perceptrons for feature extraction. Figure 5 As shown, the BERT large language model is used to perform feature extraction processing on POI information (or semantic information) (obtaining global and local semantic features). The extracted semantic features are input into the local semantic gate for processing to obtain local semantic features, which are used for prompting and enhanced fusion when extracting local features in the classification features of the scene classification branch; the extracted semantic features are input into the global semantic gate for processing to obtain global semantic features, which are used for prompting and enhanced fusion when extracting global features in the classification features of the scene classification branch.

[0045] S22, calculate the spectral index pixel by pixel for the satellite remote sensing image to generate a spectral index image, and use the local visual gate composed of the IRB neural network module and the global visual gate composed of the MiT neural network module to perform feature extraction processing respectively. Figure 6 As shown in the figure, the spectral index is calculated pixel by pixel to obtain the spectral index of each pixel. The spectral index calculation formula is: (red band − blue band) / (red band + blue band). The spectral index is first processed through a 3×3 convolution and input into the local vision gate and the global vision gate respectively. The spectral index after the 3×3 convolution is processed through the local vision gate to obtain local visual features, which are used for prompting and enhanced fusion when extracting local features from the classification features of the scene classification branch. The spectral index after the 3×3 convolution is processed through the global vision gate to obtain global visual features, which are used for prompting and enhanced fusion when extracting global features from the classification features of the scene classification branch.

[0046] S23, the scene classification branch extracts several layers of classification features from satellite remote sensing images, such as Figure 2 As shown, the classification multi-layer features of the scene classification branch in this embodiment include four layers of classification features. The global features of each layer of classification features use the global semantic gate output features (specifically Figure 5 The global semantic features of the output) and the global visual gate (specifically Figure 6 The output global visual features) output features for prompting and enhancing the corresponding position points. The local features of each layer of classification features use the local semantic gate to output features (specifically Figure 5 The local semantic features of the output) and the local visual gate output features (specifically Figure 6In some embodiments, each layer of classification features in the scene classification branch uses a local and global feature extractor to extract local features and global features. The local and global feature extractors in the scene classification branch include an IRB neural network module, a local prompt module (see Figure 4 , the local prompt module is Figure 4 The hint module corresponding to the semantic / visual information hint in the local feature extraction branch of Figure 4 , the global prompt module is Figure 4 The IRB neural network module and the local prompt module constitute a local feature extraction branch, the MiT neural network module and the global prompt module constitute a global feature extraction branch, and the local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes:

[0047] like Figure 4 As shown in the figure, the local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global hint module to obtain the global hint information of the global semantic gate and the global visual gate (see Figure 4 , the global hint module obtains the global semantic features output by the global semantic gate and the global visual features output by the global visual gate, and extracts information separately and combines them into global hint information. The global hint information can assist the global feature extraction branch to extract global features more quickly and accurately) to perform global hints for red mud. The MiT neural network module extracts global features based on global hints and uses the global feature maps of the global semantic gate and the global visual gate (including the global semantic features output by the global semantic gate and the global visual features output by the global visual gate) to perform enhanced fusion of corresponding position points (that is, the global semantic features output by the global semantic gate and the global visual features output by the global visual gate are used to fuse and enhance the global features in the classification features extracted by the local and global feature extractors). The local feature extraction branch uses the local hint module to obtain local hint information of the local semantic gate and the local visual gate (see Figure 4, the local prompt module obtains the local semantic features output by the local semantic gate and the local visual features output by the local visual gate, and extracts information separately to combine into local prompt information. The local prompt information can assist the local feature extraction branch to extract local features more quickly and accurately) to perform local prompts for red mud. The IRB neural network module extracts local features based on local prompts and uses the local feature maps of the local semantic gate and the local visual gate (including the local semantic features output by the local semantic gate and the local visual features output by the local visual gate) to perform enhanced fusion of corresponding position points (that is, the local semantic features output by the global semantic gate and the local visual features output by the global visual gate are used to fuse and enhance the local features in the classification features extracted by the local and global feature extractors). Then the sparse fusion module performs sparse and fusion processing of the features, and outputs them after 1×1 convolution processing. The technical principle of the sparse fusion module of this embodiment is as follows Figure 3 As shown, the perceptron is used to calculate the query matrix from the local features. , key matrix Sum Matrix , calculate the query matrix from the global features , key matrix Sum Matrix ,Will and Multiply and use the perceptron to calculate the relevance score matrix of attention weights and the sparse weight matrix ;Will and Multiply and use the perceptron to calculate the relevance score matrix of attention weights and the sparse weight matrix ; Finally, and and Multiply to get the sparse global features, and and Multiply to get the sparse local features.

[0048] S24, the scene classification branch outputs several layers (in this embodiment, a total of four layers) of extracted and processed classification multi-layer features, such as Figure 2 As shown, a classifier composed of a global maximum pooling module, a global average pooling module, and a perceptron is used to identify red mud, determine whether it is red mud, and obtain red mud heat map information. If the scene classification branch includes four layers, the classification multi-layer features of the scene classification branch include four layers of classification features. , the red mud heat map information also contains four layers of red mud heat map information, see Figure 2The scene classification branch jumps and bridges the corresponding red mud heat map information to the corresponding layer of the semantic segmentation branch and uses it for feature extraction prompts and enhanced fusion. The scene classification branch jumps and bridges the corresponding classification features to the corresponding layer of the semantic segmentation branch and uses it for enhanced fusion of fine features.

[0049] S3, semantic segmentation branch uses red mud heat map information to perform feature prompting, fine multi-layer feature fusion extraction, segmentation processing on satellite remote sensing images and output red mud segmentation result map, such as Figure 2 As shown, the fine multi-layer features of the semantic segmentation branch in this embodiment include four layers of fine features ,The fine multi-layer feature extraction method of the semantic segmentation branch includes:

[0050] S31, obtain prompt information from the red mud heat map information, the prompt information includes whether the pixel is red mud, whether the pixel combination area contains red mud, and the density, intensity and distribution of red mud; the semantic segmentation branch first relies on the prompt information of the red mud heat map information to quickly judge and process the satellite remote sensing image. Figure 2 As shown, the red mud heat map information also contains four layers of red mud heat map information. The scene classification branch jumps the red mud heat map information of the corresponding layer to the corresponding layer of the semantic segmentation branch (for the heat map information prompt of the corresponding layer of the semantic segmentation branch).

[0051] S32, the semantic segmentation branch performs several layers of fine feature extraction on the satellite remote sensing image. The fine multi-layer features of the semantic segmentation branch of this embodiment include four layers of fine features. The global features of each layer of fine features are first globally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the global prompts, and the corresponding position points are enhanced and fused using the global feature map of the red mud heat map information. The local features of each layer of fine features are first locally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the local prompts, and the corresponding position points are enhanced and fused using the local feature map of the red mud heat map information. The semantic segmentation branch of the present invention uses local and global feature extractors to extract the fine features of each layer of satellite remote sensing images, such as Figure 8 As shown, the local and global feature extractors of the semantic segmentation branch include the IRB neural network module, the local prompt module (see Figure 8 , the local hint module in the semantic segmentation branch is Figure 8 The prompt module corresponding to the heat map information prompt in the local feature extraction branch of the MiT neural network module and the global prompt module (see Figure 8 , the global hint module in the semantic segmentation branch is Figure 8The IRB neural network module and the local prompt module constitute a local feature extraction branch, the MiT neural network module and the global prompt module constitute a global feature extraction branch, and the local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes:

[0052] The local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global prompt module to obtain the global prompt information of the red mud heat map information to perform global prompts on the red mud. The global prompt module obtains the global prompt information of the red mud heat map information of the corresponding layer and uses the global prompt information to assist the local and global feature extractors in global prompts, so as to help the local and global feature extractors extract global features more quickly and accurately. The MiT neural network module extracts global features based on the global prompt and uses the global feature map of the red mud heat map information (i.e., the global feature information in the red mud heat map information) to enhance the fusion of corresponding position points. Preferably, as Figure 2 As shown, the four-layer structure of the scene classification branch of this embodiment corresponds to the four-layer structure of the semantic segmentation branch in a one-to-one jump bridge. The MiT neural network module also fuses all the features of the classification features in the scene classification branch when enhancing the fusion of the global feature map. The local feature extraction branch uses the local prompt module to obtain the local prompt information of the red mud heat map information to perform local prompts on the red mud. The local prompt module obtains the local prompt information of the red mud heat map information of the corresponding layer and uses the local prompt information to assist the local feature extraction branch in local prompts, so as to help the local feature extraction branch extract local features more quickly and accurately; the IRB neural network module extracts local features based on the local prompts and uses the local feature map of the red mud heat map information (i.e., the local feature information in the red mud heat map information) to perform enhanced fusion of the corresponding position points; preferably, as Figure 2 As shown, the four-layer structure of the scene classification branch in this embodiment is linked to the four-layer structure of the semantic segmentation branch in a one-to-one jump bridge. The MiT neural network module also integrates local features from the scene classification branch during local feature map enhancement and fusion. The sparse fusion module then performs feature sparsification and fusion processing, and the output is processed through 1×1 convolution.

[0053] In some embodiments, the local features of each layer of fine features are enhanced and fused at corresponding positions using the local feature map of the red mud heat map information (preferably, prompts are performed simultaneously during the enhanced fusion process) as follows: the local features of the fine features and the local feature map of the red mud heat map information (i.e., the local information of the red mud heat map information) are input into a first cross-attention module (the first cross-attention module uses the local prompts of the red mud heat map information to capture the weight relationship) for attention weight fusion to obtain the local features of the enhanced fused fine features, and then the local features of the enhanced fused fine features and the local features of the fine features are input into a second cross-attention module for attention weight fusion to obtain the local features of the fine features after secondary enhanced fusion. Similarly, the global features of each layer of fine features also adopt the above method and use the global feature map of the red mud heat map information (i.e., the global information of the red mud heat map information) to perform enhanced fusion of the corresponding position points. The specific method is: the global features of the fine features and the global feature map of the red mud heat map information are input into the first cross-attention module (the first cross-attention module uses the global prompts of the red mud heat map information to capture the weight relationship) for attention weight fusion to obtain the global features of the enhanced fused fine features, and then the global features of the enhanced fused fine features and the global features of the fine features are input into the second cross-attention module for attention weight fusion to obtain the global features of the fine features after secondary enhanced fusion.

[0054] The local features of each layer of classification features are enhanced and fused at the corresponding positions using the local feature maps of the local semantic gate and the local visual gate (including the local semantic features output by the local semantic gate and the local visual features output by the local visual gate) as follows: Figure 7As shown, the local features of the classification features and the local semantic features output by the local semantic gate are input into the first cross-attention module A (the first cross-attention module A uses local prompt information to capture the weight relationship) for attention weight fusion to obtain the local features of the classification features after semantic enhancement fusion, and then the local features of the classification features after semantic enhancement fusion and the local features of the classification features are input into the second cross-attention module A for attention weight fusion to obtain the local features of the classification features after secondary semantic enhancement fusion; at the same time, the local features of the classification features and the local visual features output by the local visual gate are input into the first cross-attention module B (the first cross-attention module B uses local prompt information to capture the weight relationship) for attention weight fusion to obtain the local features of the classification features after visual enhancement fusion, and then the local features of the classification features after visual enhancement fusion and the local features of the classification features are input into the second cross-attention module B for attention weight fusion to obtain the local features of the classification features after secondary visual enhancement fusion; then the local features of the classification features after secondary semantic enhancement fusion and the local features of the classification features after secondary visual enhancement fusion are feature fused. Similarly, the global features of each layer of classification features also adopt the above method and use the global feature maps of the global semantic gate and the global visual gate (including the global semantic features output by the global semantic gate and the global visual features output by the global visual gate) to enhance the fusion of the corresponding position points. The specific method is: the global features of the classification features and the global semantic features output by the global semantic gate are input into the first cross-attention module A (the first cross-attention module A uses global prompt information to capture the weight relationship) for attention weight fusion to obtain the global features of the classification features after semantic enhancement fusion, and then the global features of the classification features after semantic enhancement fusion and the global features of the classification features are input into the second cross-attention module A for attention weight fusion The global features of the classification features after secondary semantic enhancement fusion are obtained; at the same time, the global features of the classification features and the global visual features output by the global visual gate are input into the first cross-attention module B (the first cross-attention module B uses global prompt information to capture the weight relationship) for attention weight fusion to obtain the global features of the classification features after visual enhancement fusion, and then the global features of the classification features after visual enhancement fusion and the global features of the classification features are input into the second cross-attention module B for attention weight fusion to obtain the global features of the classification features after secondary visual enhancement fusion; then the global features of the classification features after secondary semantic enhancement fusion are fused with the global features of the classification features after secondary visual enhancement fusion for feature fusion.

[0055] In some embodiments, the present invention constructs a dual-branch network model that integrates position-associated satellite remote sensing image samples and ground object semantic information samples for model training. The spatial range of red mud distribution in the sample data is refined to generate pixel-level spatial range annotations. The sample dataset is divided into a training set, a validation set, and a test set in a 6:2:2 ratio, which are used for model training, validation, and performance evaluation, respectively. The training set is subjected to geometric and color transformations to expand data diversity and enhance the model's generalization capabilities. During data transformation, the positions of the POI points must be maintained, while the remote sensing images and their corresponding spectral indices and label files are synchronously subjected to geometric transformations (e.g., rotation, scaling, cropping, flipping) and color transformations (e.g., brightness adjustment, contrast enhancement, and hue changes) to improve the model's adaptability to diverse scenarios. The scene classification branch classifies the satellite remote sensing image samples and ground object semantic information samples, extracts multi-layer features, and identifies red mud, obtaining sample red mud heat map information A. The semantic segmentation branch performs fine multi-layer feature extraction and segmentation processing on the satellite remote sensing image samples and outputs the red mud segmentation result map. The semantic segmentation branch obtains the sample red mud heat map information B based on the red mud segmentation result map and sets the consistency parameter constraint loss for the sample red mud heat map information A and B. The consistency constraint loss formula is as follows: ,in, and Represent the heat map values ​​of the scene classification branch and the semantic segmentation branch at pixel i respectively.

[0056] In some embodiments, the total loss of the dual-branch network model is calculated by weighting the scene classification branch loss, the semantic segmentation branch loss, and the consistency constraint loss. The consistency constraint loss is the difference loss between the red mud heat map of the scene classification branch and the red mud heat map of the semantic segmentation branch. The scene classification branch loss and the semantic segmentation branch loss each use cross entropy loss. The consistency constraint loss formula is as follows: ,in, and They represent the heat map values ​​of the scene classification branch and the semantic segmentation branch at pixel i respectively. The loss function expression of the scene classification branch loss is as follows: ,in Is the label real (does it contain red mud)? is the model's predicted value, is the number of training samples. The loss function expression of semantic segmentation branch loss is: ,in, The true label of the pixel, is the model’s predicted probability, is the total number of pixels. The total loss expression of the two-branch network model is as follows: ,in, 、 and It is a hyperparameter that controls the contribution weight of the three parts of loss to the total loss.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A red mud space extraction method combining a prompt learning dual-branch network model, characterized by: The method steps are as follows: satellite remote sensing images and semantic visual information of ground objects associated with the study area are collected, and a two-branch network model consisting of a scene classification branch and a semantic segmentation branch is constructed. The scene classification branch classifies the satellite remote sensing images and ground object semantic visual information, extracts multi-layer features, identifies red mud, and obtains red mud heat map information; the semantic segmentation branch uses the red mud heat map information to perform feature prompting, fine multi-layer feature fusion extraction, segmentation processing on the satellite remote sensing images, and outputs the red mud segmentation result map; The semantic segmentation branch fine multi-layer feature extraction method includes: Obtain prompt information from the red mud heat map information, including whether the pixel is red mud, whether the pixel combination area contains red mud, and the red mud density, intensity, and distribution. The semantic segmentation branch first relies on the prompt information of the red mud heat map information to quickly judge and process the satellite remote sensing image. The semantic segmentation branch performs several layers of fine feature extraction on satellite remote sensing images. The global features of each layer of fine features are first globally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the global prompts, and the global feature map of the red mud heat map information is used to enhance the fusion of the corresponding position points; the local features of each layer of fine features are first locally prompted using the prompt information of the red mud heat map information, and then the fine features of the satellite remote sensing image are extracted in combination with the local prompts, and the local feature map of the red mud heat map information is used to enhance the fusion of the corresponding position points.

2. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: The satellite remote sensing images of the study area are associated with the semantic visual information of the land features according to the coordinate information and multi-source data points are associated. The semantic visual information of the land features includes the POI information of red mud related factories and red mud storage sites.

3. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: The classification multi-layer feature extraction method of the scene classification branch includes: The Bert large language model is used to extract features from the semantic visual information of the ground objects, and then the local semantic gate and global semantic gate composed of perceptrons are input for feature extraction respectively; The spectral index of satellite remote sensing images is calculated pixel by pixel to generate spectral index images. Feature extraction is performed using a local visual gate composed of an IRB neural network module and a global visual gate composed of a MiT neural network module. The scene classification branch extracts several layers of classification features from satellite remote sensing images. The global features of each layer of classification features are fused with the output features of the global semantic gate and the output features of the global visual gate to indicate and enhance the corresponding position points. The local features of each layer of classification features are fused with the output features of the local semantic gate and the output features of the local visual gate to indicate and enhance the corresponding position points. The scene classification branch outputs several layers of extracted and processed classification multi-layer features, and uses a classifier composed of a global maximum pooling module, a global average pooling module and a perceptron to identify red mud, make a judgment on whether it is red mud, and obtain red mud thermal map information.

4. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: The classification multi-layer features of the scene classification branch include four layers of classification features, the fine multi-layer features of the semantic segmentation branch include four layers of fine features, and the red mud heat map information of the scene classification branch includes four layers of red mud heat map information; the four-layer structure of the scene classification branch and the four-layer structure of the semantic segmentation branch are jump-bridged in a one-to-one correspondence, the scene classification branch jump-bridges the corresponding red mud heat map information to the corresponding layer of the semantic segmentation branch and uses it for feature extraction prompts and enhanced fusion, and the scene classification branch jump-bridges the corresponding classification features to the corresponding layer of the semantic segmentation branch and uses it for enhanced fusion of fine features.

5. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: The local features of each layer of fine features use the local information of the red mud heat map information to enhance the corresponding position point fusion method including: The local features of the fine features and the local feature map of the red mud heat map information are input into the first cross-attention module for attention weight fusion to obtain the local features of the enhanced fused fine features. Then the local features of the enhanced fused fine features and the local features of the fine features are input into the second cross-attention module for attention weight fusion to obtain the local features of the secondary enhanced fused fine features.

6. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 3 is characterized by: Each layer of classification features in the scene classification branch is extracted using a local and global feature extractor. The local and global feature extractor includes an IRB neural network module, a local prompt module, a MiT neural network module, a global prompt module, and a sparse fusion module. The IRB neural network module and the local prompt module constitute a local feature extraction branch, and the MiT neural network module and the global prompt module constitute a global feature extraction branch. The local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes: The local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global prompt module to obtain the global prompt information of the global semantic gate and the global visual gate to perform global prompts of the red mud. The MiT neural network module extracts global features based on the global prompts and uses the global feature maps of the global semantic gate and the global visual gate to enhance the fusion of corresponding position points; the local feature extraction branch uses the local prompt module to obtain the local prompt information of the local semantic gate and the local visual gate to perform local prompts of the red mud. The IRB neural network module extracts local features based on the local prompts and uses the local feature maps of the local semantic gate and the local visual gate to enhance the fusion of corresponding position points; then the sparse fusion module performs sparse and fusion processing of the features, and outputs them after 1×1 convolution processing.

7. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: The semantic segmentation branch extracts fine features of each layer of satellite remote sensing images using a local and global feature extractor. The local and global feature extractor includes an IRB neural network module, a local prompt module, a MiT neural network module, a global prompt module, and a sparse fusion module. The IRB neural network module and the local prompt module constitute a local feature extraction branch, and the MiT neural network module and the global prompt module constitute a global feature extraction branch. The local feature extraction branch and the global feature extraction branch are connected in parallel to form two branches. The method includes: The local and global feature extractors first use 1×1 convolution processing and then input the local feature extraction branch and the global feature extraction branch for parallel processing. The global feature extraction branch uses the global prompt module to obtain the global prompt information of the red mud heat map information for global prompting of the red mud. The MiT neural network module extracts global features based on the global prompt and uses the global feature map of the red mud heat map information to enhance the fusion of the corresponding position points; the local feature extraction branch uses the local prompt module to obtain the local prompt information of the red mud heat map information for local prompting of the red mud. The IRB neural network module extracts local features based on the local prompt and uses the local feature map of the red mud heat map information to enhance the fusion of the corresponding position points; then the sparse fusion module performs sparse and fusion processing of the features, and outputs them after 1×1 convolution processing.

8. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1 is characterized in that: A dual-branch network model is constructed to input position-associated satellite remote sensing image samples and ground object semantic information samples for model training. The scene classification branch classifies the satellite remote sensing image samples and ground object semantic information samples, extracts multi-layer features and identifies red mud to obtain the sample red mud heat map information A; the semantic segmentation branch performs fine multi-layer feature extraction and segmentation processing on the satellite remote sensing image samples and outputs the red mud segmentation result map. The semantic segmentation branch obtains the sample red mud heat map information B based on the red mud segmentation result map, and sets consistency parameter constraint loss for the sample red mud heat map information A and B.

9. The red mud space extraction method combining a prompt learning dual-branch network model according to claim 1, characterized in that: The total loss of the dual-branch network model is obtained by weighted calculation of the scene classification branch loss, the semantic segmentation branch loss and the consistency constraint loss. The consistency constraint loss is the difference loss between the red mud heat map of the scene classification branch and the red mud heat map of the semantic segmentation branch. The scene classification branch loss and the semantic segmentation branch loss both use cross entropy loss.

Images