Environmental pollution source intelligent identification system and method based on multi-modal target detection
By combining feature fusion and loss function optimization of image and text modalities, a multimodal target detection system is used to solve the problems of recognition accuracy and training efficiency of traditional single-modal models in environmental pollution source identification, and achieves high-precision identification and rapid adaptation of environmental pollution sources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RES INST FOR ENVIRONMENTAL INNOVATION SUZHOU TSINGHUA
- Filing Date
- 2025-12-25
- Publication Date
- 2026-05-01
AI Technical Summary
Traditional single-modal target detection models suffer from problems in environmental pollution source identification, such as homonymous objects, strong context dependence, huge differences in target morphology, and insufficient data for subdivided categories, resulting in low recognition accuracy and low training efficiency.
A multimodal object detection system is adopted, which combines image and text modalities. Multi-scale visual and semantic features are extracted through visual encoders and language encoders. Feature weighting is performed using a context-gated attention mechanism, and a category-aware hybrid loss function is used for training to improve detection accuracy and adaptability.
It effectively solves the problem of confusion between objects of the same spectrum and those of different species, improves the detection accuracy of targets with huge morphological differences, enhances context awareness, and has rich recognition categories with high training efficiency.
Smart Images
Figure CN121962574A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of environmental big data technology, and in particular to an intelligent identification system and method for environmental pollution sources based on multimodal target detection. Background Technology
[0002] With the rapid development of remote sensing technology, high-resolution Earth observation data has exploded, providing a valuable information foundation for large-scale environmental monitoring, especially for the identification and location of environmental pollution sources (such as industrial storage tanks, solid waste dumps, and flue gas emissions). Automated analysis of remote sensing images using computer vision technology, particularly deep learning target detection models, has become an important research direction in environmental regulation. Traditional pollution source identification methods rely on manual design and extensive prior knowledge for feature selection, making it difficult to automatically adapt to massive amounts of remote sensing image data, and thus have the following drawbacks.
[0003] 1. The problems of "different substances with the same spectrum" and "different spectra with the same substance" are serious.
[0004] Similar but different features: Different land features exhibit similar visual characteristics. For example, a "sewage sedimentation pond" is highly similar to a "fish pond," a "natural lake," or even a "large swimming pool" in images (all are bodies of water), and they can be easily confused based on image modality alone.
[0005] Different features of the same object: The same ground feature exhibits different characteristics under different lighting, seasons, and sensors.
[0006] 2. Strong context dependence, which single-modal models struggle to distinguish: Whether a target is a pollution source is highly dependent on its spatial context. For example, a "circular storage tank" appearing on a residential rooftop might be a water tower, but appearing in a chemical plant area is highly likely to be an "industrial storage tank." Traditional single-modal (pure image) target detection models struggle to effectively utilize this high-level semantic contextual information.
[0007] 3. Significant differences in target morphology, leading to high detection difficulty: Pollution source targets vary greatly in scale and morphology. For example, an "industrial fan" may occupy only a few pixels in remote sensing imagery, while a "farm" or "solid waste dump" may cover a large area; simultaneously, there exist non-rigid targets with blurred boundaries, such as "flue gas," and rigid targets with clear outlines, such as "chimneys." Existing models struggle to efficiently handle both simultaneously using a unified detection head and loss function.
[0008] 4. Insufficient data for subcategories: There is a lack of sufficient labeled data for some subcategories of pollution sources, which is insufficient to support the training of a large single-modal visual model from scratch. Summary of the Invention
[0009] The purpose of this invention is to provide an intelligent identification system and method for environmental pollution sources based on multimodal target detection, which solves the problem of confusion between objects of the same spectrum and those of different species. It has strong context awareness, improves the detection accuracy of targets with huge morphological differences, and has rich identification categories and high training efficiency.
[0010] The technical solution of this invention is: An intelligent environmental pollution source identification system based on multimodal target detection is deployed on the server side, including... An image preprocessing module is configured to receive input remote sensing images and perform preprocessing operations on the remote sensing images, including slicing, linear stretching, and image format conversion, to generate standardized image patches. An image enhancement module is communicatively connected to the image preprocessing module and is configured to perform one or more enhancement operations on the standardized image blocks. The enhancement operations include rotation, flipping, brightness adjustment, contrast adjustment, gamma transformation, affine transformation, or contrast-limited histogram equalization to generate an enhanced image. The multimodal target detection module, which is communicatively connected to the image enhancement module, specifically includes: A visual encoder configured to receive the enhanced image and extract its multi-scale visual feature map; A language encoder is configured to receive descriptive text phrases corresponding to the target pollution source category and extract their semantic feature vectors; the descriptive text phrases contain spatial context information and morphological feature information of the target. A context-gated attention unit, connected to the visual encoder and the language encoder respectively, is configured to perform the following operations: parse spatial context features from the semantic feature vector as a gating signal; use the gating signal to dynamically spatially weight the multi-scale visual feature map to generate a context-weighted visual feature map, thereby enhancing the feature response of the image region that matches the text description and suppressing irrelevant regions; The feature fusion and output unit dynamically fuses the gated and weighted multimodal features; it analyzes the fused features based on a multi-layer regression and classification network to complete the identification of pollution source categories and spatial location prediction. The output module is communicatively connected to the multimodal target detection module and is configured to analyze the prediction results, combine them with the geographic coordinate information of the original remote sensing image, calculate and output the precise latitude and longitude of the detected pollution source and its corresponding remote sensing image.
[0011] Preferably, the multimodal target detection module also includes a loss calculation unit configured during the training phase, which adopts a category-aware hybrid loss function and adaptively selects the calculation method of the bounding box regression loss based on the currently predicted pollution source category attributes.
[0012] Preferably, CIoU loss is used to calculate the bounding box regression loss for rigid boundary targets, while fuzzy boundary loss based on pixel affinity is automatically switched for non-rigid boundary targets.
[0013] Preferably, the feature fusion and output unit uses a Transformer structure to dynamically fuse the gated and weighted multimodal features.
[0014] An intelligent identification method for environmental pollution sources based on multimodal target detection includes the following steps: S1. Image preprocessing: The input remote sensing image is sliced, linearly stretched, and converted to a different image format. S2. Image Enhancement: Perform one or more operations on the pre-processed image, including rotation, flipping, brightness and contrast adjustment, gamma and affine transformation, and contrast-limited histogram equalization. S3. Multimodal Target Detection: Constructing a multimodal target detection model to identify pollution sources in images; including: Multi-scale visual features of the remote sensing image are extracted using a visual encoder; Semantic features of descriptive text corresponding to pollution source categories are extracted using a language encoder; the descriptive text contains spatial context information and / or morphological feature information of the target pollution source. Using a context-gated attention mechanism, spatial context features extracted from the descriptive text are used as gating signals to dynamically spatially weight the multi-scale visual features, resulting in weighted visual features. The weighted visual features and the semantic features are fused to obtain multimodal fusion features; Based on the multimodal fusion features, the target detection network outputs the pollution source category identification and spatial location prediction results; S4. Output detection results: Organize the predictions output by the model, and then combine them with the geographic information of remote sensing images to calculate the precise latitude and longitude of the target, and output the geographical location of the pollution source and the remote sensing images of its vicinity.
[0015] Preferably, the multimodal target detection model further includes a model training and optimization step, comprising: Data preparation: First, the enhanced remote sensing images were manually screened, and the coordinates of various target pollution sources were labeled; a descriptive text knowledge base was constructed for each type of pollution source, containing descriptive phrases such as its typical shape, color, and context. Model training: The model is input with an image containing the target pollution source, its corresponding coordinate labels, and its corresponding descriptive text. The parameters in the network are trained using the stochastic gradient descent algorithm. During training or inference, a category-aware hybrid loss function is used to adaptively select the calculation method of the bounding box regression loss based on the category attributes of the pollution source. Model optimization: The training results are quantitatively evaluated by combining metrics such as average accuracy, precision, and recall.
[0016] Preferably, the context-gated attention mechanism specifically includes: Extract spatial context feature vectors from the semantic features of the descriptive text; The spatial context feature vector is used as a gating signal to perform gating operations with the multi-scale visual features, thereby enhancing the feature response of image regions that match the spatial context features and suppressing the response of mismatched regions.
[0017] Preferably, the specific implementation of the category-aware hybrid loss function is as follows: When the pollution source category is a target with rigid boundaries, CIoU loss is used as the bounding box regression loss; When the pollution source category is a target with non-rigid or fuzzy boundaries, fuzzy boundary loss based on pixel affinity is used as the bounding box regression loss.
[0018] Preferably, the multimodal fusion features are fused using a Transformer structure.
[0019] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.
[0020] The advantages of this invention are: 1. Solving the problem of confusion between "similar objects with different spectra": This invention specifically introduces an "image-text" dual-modality. Its core innovation lies in the fact that the text modality is not a simple category label (such as "sedimentation tank"), but rather descriptive knowledge containing "spatial context" and "morphological features" (such as "rectangular turbid water body located in an industrial area"). Through multimodal fusion, the model can distinguish visually similar objects such as "sewage sedimentation tank" and "fish pond".
[0021] 2. Strong context awareness: Through a specific multimodal fusion mechanism, contextual information in the text (such as "located in an industrial area") is used as prior knowledge to guide the image modality to find the target in the correct spatial range (such as within an industrial plant area), effectively solving the problem of weak context awareness of single-modal models.
[0022] 3. Improve the detection accuracy of targets with huge morphological differences: The present invention designs a "category-aware hybrid loss function" that can adaptively switch the loss calculation method (e.g., switch to fuzzy boundary loss or rigid boundary loss) according to the category of the target to be detected (e.g., whether it is "smoke" or "storage tank"), which significantly improves the detection accuracy of non-rigid and small targets.
[0023] 4. Rich category recognition and high training efficiency: The introduced multimodal target detection model is usually pre-trained on massive amounts of data, with strong basic capabilities. This invention utilizes the flexibility of text modality, is not limited to predefined pollution source categories, and can quickly adapt to pollution source recognition scenarios (such as fans, chimneys, flue gas, cooling towers, industrial storage tanks, solid waste dumps, sedimentation tanks, farms, etc.) with only a small amount of data. Attached Figure Description
[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a structural diagram of an intelligent environmental pollution source identification system based on multimodal target detection; Figure 2 A schematic diagram illustrating visual and textual modal feature extraction; Figure 3 This is a schematic diagram of multimodal fusion. Figure 4 This is a schematic diagram of feature fusion and output; Figure 5 This is a schematic diagram of the category-aware hybrid loss. Detailed Implementation
[0025] like Figure 1 As shown, the present invention provides an intelligent environmental pollution source identification system based on multimodal target detection, comprising: an image preprocessing module, an image enhancement module, a multimodal target detection module, and an output module.
[0026] The image preprocessing module is deployed on the server side and is used to preprocess the input remote sensing images, including slicing, linear stretching, image format conversion, etc. The image enhancement module is deployed on the server side and is used to perform preliminary enhancement on the preprocessed image, including operations such as rotation, flipping, brightness and contrast adjustment, gamma and affine transformation, and contrast-limited histogram equalization. The multimodal target detection module runs on the server side and is the core part of this invention. Specifically, it includes: Visual encoder and language encoder: A pre-trained visual encoder is used to extract multi-scale visual features; a pre-trained language encoder is used to extract semantic features.
[0027] Descriptive text input: The input of the language encoder is not a single category label of the pollution source (such as "sedimentation pond"), but a set of descriptive text phrases containing the spatial context, morphological features or topological relationships of the pollution source (e.g., a rectangular turbid water body located in a chemical plant area). This descriptive text is used to assist the image modality in solving the problem of confusion between "same spectrum different objects".
[0028] Context-Gated Attention Mechanism: An adaptive improvement to the attention mechanism. This mechanism extracts additional spatial context features (such as semantic vectors extracted from "located in a chemical plant area") from the descriptive text phrase and uses them as a gating signal. This gating signal is used to dynamically weight the visual feature map output by the visual encoder, enabling the model to "focus" on regions in the visual features that match the textual context description (such as increasing the feature weight of the chemical plant area), while suppressing feature responses from irrelevant regions (such as residential areas).
[0029] Classification-Aware Hybrid Loss: In the training and inference of the model, this invention uses a hybrid loss function that adaptively switches the loss calculation method according to the identified pollution source category: when the detected target is a rigid boundary target such as an industrial storage tank or chimney, CIoU (CompleteIntersection over Union) loss is used to calculate its bounding box regression loss; when the detected target is a non-rigid target with ambiguous boundaries such as flue gas or solid waste dumps, it automatically switches to a fuzzy boundary loss (SoftDice Loss) based on pixel affinity.
[0030] Feature fusion and output: The Transformer structure is used to dynamically fuse the gated weighted multimodal features; the fused features are analyzed based on a multi-layer regression and classification network to complete the identification of pollution source categories and spatial location prediction.
[0031] The output module runs on the server side and is used to visualize the predictions generated by the pollution source identification module. It parses the pollution source category and location information based on the result vector output by the multimodal target detection module, and then calculates the geographic coordinates of the measured target by combining the positioning information of the current image, and outputs the accurate latitude and longitude of the detected pollution source and the remote sensing image of its vicinity.
[0032] Reference Figure 1 The present invention provides an intelligent identification method for environmental pollution sources based on multimodal target detection, comprising the following steps.
[0033] Step 1: Image preprocessing; First, a large amount of remote sensing imagery was collected, prioritizing images with a resolution accuracy better than 0.5m. Then, the remote sensing images were sliced and linearly stretched by 2% grayscale value to adapt the image format to the model.
[0034] Step 2: Image Enhancement; Enhancement processing is performed on the image, including brightness and contrast adjustment, gamma and affine transformation, and contrast-limited histogram equalization, to highlight key features in the image and weaken the interference of factors such as brightness and darkness, color, angle changes, and local cropping, thereby improving the generalization ability of the model.
[0035] Step 3: Construction of a multimodal target detection model; Construct a multimodal target detection network model to identify pollution sources in images.
[0036] Feature extraction: such as Figure 2 As shown, the image is input into the visual encoder; at the same time, the descriptive text of the pollution source (e.g., "a rectangular body of water with regular edges and turbid color located in an industrial park", rather than a simple "sedimentation pond" label) is input into the text encoder to extract semantic features.
[0037] Multimodal fusion: such as Figure 3 As shown, features from the image modality and the text modality are mapped to a unified feature space. These features are then fed into a context-gated attention module. This module specifically extracts contextual features from the text (such as industrial parks) and uses these as gating signals to spatially weight the image feature maps. This allows the model to "focus" on regions in the image that match the textual context description during fusion, effectively distinguishing visually similar objects such as "sedimentation pond" and "fishpond."
[0038] Feature fusion and output: such as Figure 4 As shown, a fusion layer based on a Transformer structure fuses feature vectors from different modalities (after gating and weighting) into a joint feature representation. The joint feature representation is then transformed into pollution source category and location information using a target detection task head (such as a multi-layer regression and classification network).
[0039] Loss Calculation: When calculating the loss, a category-aware hybrid loss function is used. Depending on whether the current target is rigid (e.g., a storage tank) or non-rigid (e.g., flue gas), the corresponding loss function is adaptively selected to improve positioning accuracy. For example... Figure 5 As shown, when the pollution source category is a target with rigid boundaries, CIoU loss is used as the bounding box regression loss; when the pollution source category is a target with non-rigid or fuzzy boundaries, fuzzy boundary loss based on pixel affinity is used as the bounding box regression loss.
[0040] Step 4: Model Training and Optimization The constructed multimodal target detection model is trained and optimized.
[0041] Data preparation: First, the enhanced remote sensing images are manually screened, and the coordinates of various target pollution sources are labeled. In addition, this invention also needs to build a descriptive text knowledge base for each type of pollution source, containing descriptive phrases such as its typical shape, color, and context (for example, "sedimentation pond" corresponds to "turbid water pond in industrial area", "rectangular water body", etc.; "fish pond" corresponds to "grid-like water body in suburbs", "near farmland", etc.).
[0042] Model training: Input the image containing the target pollution source, the corresponding coordinate labels, and the corresponding descriptive text into the model, and train the parameters in the network using the stochastic gradient descent algorithm.
[0043] Model optimization: The training results are quantitatively evaluated by combining metrics such as average accuracy, precision, and recall.
[0044] Step 5: Output the detection results The model output predictions are processed, and then combined with the geographic information from remote sensing imagery to calculate the precise latitude and longitude of the target, outputting the geographical location of the pollution source and the remote sensing imagery of its vicinity.
[0045] The present invention also protects a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method described.
[0046] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All modifications made according to the spirit and essence of the main technical solution of the present invention should be covered within the scope of protection of the present invention.
Claims
1. An intelligent identification system for environmental pollution sources based on multimodal target detection, characterized in that, Deployed on the server side, including An image preprocessing module is configured to receive input remote sensing images and perform preprocessing operations on the remote sensing images, including slicing, linear stretching, and image format conversion, to generate standardized image patches. An image enhancement module is communicatively connected to the image preprocessing module and is configured to perform one or more enhancement operations on the standardized image blocks. The enhancement operations include rotation, flipping, brightness adjustment, contrast adjustment, gamma transformation, affine transformation, or contrast-limited histogram equalization to generate an enhanced image. The multimodal target detection module, which is communicatively connected to the image enhancement module, specifically includes: A visual encoder configured to receive the enhanced image and extract its multi-scale visual feature map; A language encoder is configured to receive descriptive text phrases corresponding to the target pollution source category and extract their semantic feature vectors; the descriptive text phrases contain spatial context information and morphological feature information of the target. A context-gated attention unit, connected to the visual encoder and the language encoder respectively, is configured to perform the following operations: parse spatial context features from the semantic feature vector as a gating signal; use the gating signal to dynamically spatially weight the multi-scale visual feature map to generate a context-weighted visual feature map, thereby enhancing the feature response of the image region that matches the text description and suppressing irrelevant regions; The feature fusion and output unit dynamically fuses the gated and weighted multimodal features; it analyzes the fused features based on a multi-layer regression and classification network to complete the identification of pollution source categories and spatial location prediction. The output module is communicatively connected to the multimodal target detection module and is configured to analyze the prediction results, combine them with the geographic coordinate information of the original remote sensing image, calculate and output the precise latitude and longitude of the detected pollution source and its corresponding remote sensing image.
2. The intelligent environmental pollution source identification system based on multimodal target detection according to claim 1, characterized in that, The multimodal target detection module also includes a loss calculation unit, which is configured during the training phase and adopts a category-aware hybrid loss function. Based on the currently predicted pollution source category attributes, it adaptively selects the calculation method for the bounding box regression loss.
3. The intelligent environmental pollution source identification system based on multimodal target detection according to claim 2, characterized in that, For rigid boundary targets, CIoU loss is used to calculate their bounding box regression loss, while for non-rigid boundary targets, a fuzzy boundary loss based on pixel affinity is automatically switched.
4. The intelligent environmental pollution source identification system based on multimodal target detection according to claim 1, characterized in that, The feature fusion and output unit uses the Transformer structure to dynamically fuse the gated and weighted multimodal features.
5. An intelligent identification method for environmental pollution sources based on multimodal target detection, characterized in that, Includes the following steps: S1. Image preprocessing: The input remote sensing image is sliced, linearly stretched, and converted to a different image format. S2. Image Enhancement: Perform one or more operations on the pre-processed image, including rotation, flipping, brightness and contrast adjustment, gamma and affine transformation, and contrast-limited histogram equalization. S3. Multimodal Target Detection: Constructing a multimodal target detection model to identify pollution sources in images; including: Multi-scale visual features of the remote sensing image are extracted using a visual encoder; Semantic features of descriptive text corresponding to pollution source categories are extracted using a language encoder; the descriptive text contains spatial context information and / or morphological feature information of the target pollution source. Using a context-gated attention mechanism, spatial context features extracted from the descriptive text are used as gating signals to dynamically spatially weight the multi-scale visual features, resulting in weighted visual features. The weighted visual features and the semantic features are fused to obtain multimodal fusion features; Based on the multimodal fusion features, the target detection network outputs the pollution source category identification and spatial location prediction results; S4. Output detection results: Organize the predictions output by the model, and then combine them with the geographic information of remote sensing images to calculate the precise latitude and longitude of the target, and output the geographical location of the pollution source and the remote sensing images of its vicinity.
6. The intelligent identification method for environmental pollution sources based on multimodal target detection according to claim 5, characterized in that, The multimodal target detection model also includes model training and optimization steps, including: Data preparation: First, the enhanced remote sensing images were manually screened, and the coordinates of various target pollution sources were labeled; a descriptive text knowledge base was constructed for each type of pollution source, containing descriptive phrases such as its typical shape, color, and context. Model training: The model is input with an image containing the target pollution source, its corresponding coordinate labels, and its corresponding descriptive text. The parameters in the network are trained using the stochastic gradient descent algorithm. During training or inference, a category-aware hybrid loss function is used to adaptively select the calculation method of the bounding box regression loss based on the category attributes of the pollution source. Model optimization: The training results are quantitatively evaluated by combining metrics such as average accuracy, precision, and recall.
7. The intelligent identification method for environmental pollution sources based on multimodal target detection according to claim 5, characterized in that, The context-gated attention mechanism specifically includes: Extract spatial context feature vectors from the semantic features of the descriptive text; The spatial context feature vector is used as a gating signal to perform gating operations with the multi-scale visual features, thereby enhancing the feature response of image regions that match the spatial context features and suppressing the response of mismatched regions.
8. The intelligent identification method for environmental pollution sources based on multimodal target detection according to claim 6, characterized in that, The specific implementation of the category-aware hybrid loss function is as follows: When the pollution source category is a target with rigid boundaries, CIoU loss is used as the bounding box regression loss; When the pollution source category is a target with non-rigid or fuzzy boundaries, fuzzy boundary loss based on pixel affinity is used as the bounding box regression loss.
9. The intelligent identification method for environmental pollution sources based on multimodal target detection according to claim 5, characterized in that, The multimodal fusion features are fused using a Transformer structure.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 5 to 9.