Ecological security monitoring method and device for black soil basin based on multi-source data fusion

Abstract

The application discloses a black soil basin ecological safety monitoring method and device based on multi-source data fusion, and comprises the following steps: obtaining ecological environment report texts, environmental parameter time series data, structured attribute data and remote sensing image information of a target black soil basin; inputting the remote sensing image information into a pre-trained graph text generation model to obtain ecological environment description texts; performing feature extraction and data fusion on the ecological environment description texts, the ecological environment report texts, the environmental parameter time series data, the structured attribute data and the remote sensing image information to obtain multi-modal fusion data; inputting the multi-modal fusion data into a pre-trained environmental parameter prediction model to obtain predicted environmental parameters of the target black soil basin in a future period; and generating ecological safety early warning information when the predicted environmental parameters exceed a preset safety threshold range. The application improves the accuracy and timeliness of black soil basin ecological safety monitoring and early warning, and can be widely applied to the technical field of ecological environment monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of ecological environment monitoring, and in particular to a method and device for monitoring ecological security of a black soil watershed based on multi-source data fusion. Background Art

[0002] With global climate change and intensified human activities, ecological security issues in black soil basins are becoming increasingly prominent, profoundly impacting the regional economy, society, and even the global ecosystem. Black soil, one of Earth's most precious soil resources, boasts high yield, high quality, and high efficiency, earning it the nickname "the giant panda of cultivated land." However, due to long-term overexploitation, irrational use, and climate change, black soil basins face a range of ecological challenges, including soil erosion, declining fertility, water shortages, and reduced biodiversity. These challenges pose a serious threat to the sustainable use and ecological security of black soil resources.

[0003] To effectively address ecological security issues in the Black Soil Basin and achieve scientific management and protection, intelligent monitoring and early warning technologies have emerged. By accurately acquiring and analyzing ecological and environmental data in the Black Soil Basin in real time, these technologies provide decision-makers with a scientific basis for formulating appropriate protection and management measures. However, ecological and environmental data in the Black Soil Basin are multi-source, heterogeneous, and complex. Efficiently integrating this multi-source, heterogeneous data to improve the accuracy and timeliness of monitoring and early warning is a key and challenging area of ​​current research.

[0004] Explanation of terms:

[0005] Transformer: Proposed by Google in 2017, Transformer is a neural network model based on the self-attention mechanism, primarily used to process sequential data such as text and speech. This model has achieved remarkable success in the field of natural language processing (NLP) and has gradually been extended to other fields such as computer vision and audio processing.

[0006] CLIP: CLIP is a multimodal model that achieves cross-modal understanding by associating images and text through contrastive learning, supporting tasks such as zero-shot classification and image generation.

[0007] Cross-Attention: Cross-Attention is a cross-sequence attention mechanism in Transformer. By calculating the correlation between the query (target sequence) and the key value (source sequence), it dynamically integrates information from different sources to achieve contextual interaction and semantic alignment for tasks such as machine translation and text generation.

[0008] WGAN-GP: WGAN-GP is an improved version of the Wasserstein GAN, which introduces a gradient penalty term instead of weight clipping to force the discriminator to satisfy the Lipschitz continuity constraint, thereby improving the training stability and alleviating the mode collapse problem. SUMMARY

[0009] The present application aims to at least partially solve one of the technical problems existing in the prior art.

[0010] To this end, one object of the embodiments of the present application is to provide a black soil watershed ecological safety monitoring method based on multi-source data fusion, which improves the accuracy and timeliness of black soil watershed ecological safety monitoring and early warning.

[0011] Another object of the embodiments of the present application is to provide a black soil watershed ecological safety monitoring device based on multi-source data fusion.

[0012] In order to achieve the above technical purpose, the technical solutions adopted by the embodiments of the present application include:

[0013] On the one hand, the embodiments of the present application provide a black soil watershed ecological safety monitoring method based on multi-source data fusion, comprising the following steps:

[0014] Obtaining ecological environment report text, environmental parameter time series data, structured attribute data and remote sensing image information of the target black soil watershed;

[0015] Inputting the remote sensing image information into a pre-trained graph-to-text model to obtain an ecological environment description text;

[0016] Performing feature extraction and data fusion on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data and the remote sensing image information to obtain multi-modal fusion data;

[0017] Inputting the multi-modal fusion data into a pre-trained environmental parameter prediction model to obtain predicted environmental parameters of the target black soil watershed in a future period;

[0018] When the predicted environmental parameters exceed a preset safety threshold range, generating ecological safety early warning information.

[0019] Further, in an embodiment of the present application, the graph-to-text model is trained by the following steps:

[0020] Obtaining a preset remote sensing image sample and determining corresponding text description information through artificial labeling;

[0021] Normalizing and enhancing the remote sensing image samples to obtain training samples, and mapping the text description information into word sequences;

[0022] Inputting the training sample and the word sequence into the CLIP large language model for feature extraction to obtain an image embedding vector and a text embedding vector;

[0023] Constructing a generative adversarial network, inputting the image embedding vector and the text embedding vector into the generative adversarial network for training, and obtaining a trained generator subnetwork and a discriminator subnetwork;

[0024] The graph-to-text model is composed according to the CLIP large language model and the generator sub-network.

[0025] Furthermore, in one embodiment of the present invention, the inputting the image embedding vector and the text embedding vector into the generative adversarial network for training to obtain a trained generator subnetwork and a discriminator subnetwork specifically includes:

[0026] Initialize the generator subnetwork and the discriminator subnetwork;

[0027] Inputting the image embedding vector into the generator sub-network to generate a text sequence;

[0028] Inputting the generated text sequence and the text embedding vector into the discriminator subnetwork to obtain the authenticity probability of the generated text sequence and the semantic matching score with the image embedding vector;

[0029] The adversarial loss value is calculated based on the authenticity probability, the semantic alignment loss value is calculated based on the semantic matching score, and the gradient penalty term is determined based on the linear interpolation of the generated text sequence and the text embedding vector.

[0030] Determining a first loss value of the generator sub-network according to the adversarial loss value and the gradient penalty term, and determining a second loss value of the discriminator sub-network according to the adversarial loss value and the semantic alignment loss value;

[0031] The generator subnetwork and the discriminator subnetwork are jointly optimized according to the first loss value and the second loss value to obtain the trained generator subnetwork and the discriminator subnetwork.

[0032] Furthermore, in one embodiment of the present invention, the feature extraction and data fusion of the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain multimodal fusion data specifically includes:

[0033] Performing data cleaning and data alignment on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information;

[0034] Performing feature extraction on the aligned ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain first text features, second text features, environmental parameter features, attribute value features, and image features;

[0035] The first text feature, the second text feature, the environmental parameter feature, the attribute numerical feature and the image feature are normalized, and data fusion is performed through a pre-built Transformer model to obtain the multimodal fusion data.

[0036] Furthermore, in one embodiment of the present invention, the data fusion is performed using a pre-built Transformer model to obtain the multimodal fusion data, which specifically includes:

[0037] Performing feature splicing on the normalized first text feature, the second text feature, the environmental parameter feature, the attribute value feature, and the image feature to obtain a first spliced ​​feature;

[0038] Performing position encoding on the first splicing feature to obtain a second splicing feature;

[0039] A multi-head self-attention mechanism is added to the second splicing feature to obtain the multimodal fusion data.

[0040] Furthermore, in one embodiment of the present invention, the environmental parameter prediction model is trained by the following steps:

[0041] Acquiring historical multimodal fusion data of the target black soil watershed;

[0042] Sampling the historical multimodal fusion data according to a preset prediction time step to obtain multimodal fusion samples and corresponding environmental parameter labels;

[0043] Inputting the multimodal fusion samples into a pre-built radial basis function neural network to obtain environmental parameter prediction values;

[0044] Determining a third loss value according to a root mean square error between the environmental parameter prediction value and the environmental parameter label;

[0045] The parameters of the radial basis function neural network are updated according to the third loss value to obtain the trained environmental parameter prediction model.

[0046] Further, in an embodiment of the present application, when the predicted environment parameter exceeds the preset safety threshold range, an ecological safety warning information is generated, which specifically includes:

[0047] When the predicted environment parameter exceeds the preset safety threshold range, a target parameter index and a target time node exceeding the safety threshold range are determined;

[0048] The ecological safety warning information is generated according to the target parameter index and the target time node.

[0049] On the other hand, an embodiment of the present application provides a black soil watershed ecological safety monitoring device based on multi-source data fusion, comprising:

[0050] A multi-source data acquisition module is configured to acquire ecological environment report texts, environment parameter time series data, structured attribute data, and remote sensing image information of a target black soil watershed;

[0051] A graph text generation module is configured to input the remote sensing image information into a pre-trained graph text generation model to obtain ecological environment description texts;

[0052] A multi-source data fusion module is configured to perform feature extraction and data fusion on the ecological environment description texts, the ecological environment report texts, the environment parameter time series data, the structured attribute data, and the remote sensing image information to obtain multi-modal fusion data;

[0053] An environment parameter prediction module is configured to input the multi-modal fusion data into a pre-trained environment parameter prediction model to obtain predicted environment parameters of the target black soil watershed in a future period;

[0054] An ecological safety warning module is configured to generate an ecological safety warning information when the predicted environment parameter exceeds a preset safety threshold range.

[0055] On the other hand, an embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the black soil watershed ecological safety monitoring method based on multi-source data fusion as described above.

[0056] On the other hand, an embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the black soil basin ecological security monitoring method based on multi-source data fusion as described above.

[0057] The advantages and benefits of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention:

[0058] The embodiment of the present invention obtains an ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information for a target black soil watershed. The remote sensing image information is input into a pre-trained graph-based model to obtain an ecological environment description text. Feature extraction and data fusion are performed on the ecological environment description text, ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information to obtain multimodal fusion data. The multimodal fusion data is input into a pre-trained environmental parameter prediction model to obtain predicted environmental parameters for the target black soil watershed in the future. When the predicted environmental parameters exceed a preset safety threshold, an ecological safety warning message is generated. The embodiment of the present invention automatically generates an ecological environment description text based on the remote sensing image information of the target black soil watershed using a graph-based model. The collected ecological environment report text, environmental parameter time series data, and structured attribute data are combined to generate multimodal fusion data. The pre-trained environmental parameter prediction model is used to predict the environmental parameter change trend of the multimodal fusion data. When the predicted environmental parameters exceed the safety threshold, an early warning message is issued, thereby improving the accuracy and timeliness of ecological safety monitoring and early warning in the black soil watershed. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 A flowchart of a method for monitoring ecological security in a black soil watershed based on multi-source data fusion according to an embodiment of the present invention;

[0061] Figure 2 A flowchart of the steps for training a graph-to-text model provided in an embodiment of the present invention;

[0062] Figure 3 A flowchart of step S204 provided in an embodiment of the present invention;

[0063] Figure 4 A flowchart of step S103 provided in an embodiment of the present invention;

[0064] Figure 5 A flowchart of step S1033 provided in an embodiment of the present invention;

[0065] Figure 6 A flowchart of the steps of the training environment parameter prediction model provided by an embodiment of the present invention;

[0066] Figure 7 A flowchart of step S105 provided in an embodiment of the present invention;

[0067] Figure 8 A schematic diagram of the structure of a black soil watershed ecological security monitoring device based on multi-source data fusion provided by an embodiment of the present invention;

[0068] Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention;

[0069] Figure 10 A schematic diagram of the structure of a storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. It should be noted that, although the functional modules are divided in the system schematic and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic or the order in the flow chart. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and no limitation is placed on the order between the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0071] In the description of the present invention, the meaning of "a plurality" is two or more. If there is a description of "first" or "second", it is only used to distinguish technical features and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this document are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0072] The method for monitoring the ecological security of a black soil watershed based on multi-source data fusion provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for monitoring the ecological security of a black soil watershed based on multi-source data fusion, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Multi-source heterogeneous data fusion is a data processing technology that integrates and consolidates data from diverse sources, formats, and structures to extract valuable information and patterns. In the field of ecological and environmental monitoring, multi-source heterogeneous data fusion can integrate multiple data sources, such as satellite remote sensing data, ground observation data, meteorological data, soil data, and hydrological data, to form a comprehensive and accurate ecological and environmental monitoring network. Comprehensive analysis of this data can reveal the evolutionary patterns and potential risks of the ecological environment, providing a scientific basis for ecological security early warning.

[0076] like Figure 1 FIG2 is a flowchart of a method for monitoring ecological security of a black soil watershed based on multi-source data fusion according to an embodiment of the present invention. Figure 1 The embodiment of the present invention provides a method for monitoring ecological security of a black soil watershed based on multi-source data fusion, which specifically includes the following steps:

[0077] S101. Obtaining ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information of the target black soil watershed;

[0078] S102, inputting the remote sensing image information into a pre-trained image-to-text model to obtain a description text of the ecological environment;

[0079] S103, performing feature extraction and data fusion on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain multimodal fusion data;

[0080] S104, inputting the multimodal fusion data into a pre-trained environmental parameter prediction model to obtain the predicted environmental parameters of the target black soil watershed in the future period;

[0081] S105. When the predicted environmental parameters exceed the preset safety threshold range, an ecological safety warning information is generated.

[0082] The embodiment of the present invention automatically generates ecological environment description text based on the remote sensing image information of the target black soil watershed through a graph-to-text model, combines the collected ecological environment report text, environmental parameter time series data and structured attribute data to generate multimodal fusion data, predicts the environmental parameter change trend of the multimodal fusion data through a pre-trained environmental parameter prediction model, and issues an early warning when the predicted environmental parameters exceed the safety threshold range, thereby improving the accuracy and timeliness of ecological security monitoring and early warning in the black soil watershed.

[0083] The main reference data indicators for intelligent monitoring of ecological security in the Black Soil Basin are divided into four categories: ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information. Among them, remote sensing image information includes but is not limited to vegetation coverage, land use type, land degradation area, water eutrophication Chengdu and flood basin range, and is obtained by remote sensing images taken by high-resolution satellites and drones; ecological environment report text usually exists in unstructured or semi-structured form, including but not limited to expert experience descriptions of the ecological environment, research reports, and monitoring reports; environmental parameter time series data are data recorded in chronological order, including but not limited to precipitation, temperature changes, water quality pH value, dissolved oxygen content, and soil organic matter content, and are obtained by sampling in chronological order by corresponding high-precision sensors; structured attribute data are data stored in tabular form, including numerical and category information, specifically, including but not limited to soil pH value, soil heavy metal content, surface water and groundwater reserves, and soil erosion, and the data are obtained through manual laboratory measurement, monitoring station records, and sensor sampling.

[0084] like Figure 2 The figure shows a flow chart of steps for training a graph-to-text model according to an embodiment of the present invention. Figure 2 As an optional implementation, the graph-to-text model is trained by the following steps:

[0085] S201, obtaining a preset remote sensing image sample, and determining corresponding text description information through manual annotation;

[0086] S202, performing normalization and image enhancement on the remote sensing image samples to obtain training samples, and mapping the text description information into a word sequence;

[0087] S203, inputting the training sample and word sequence into the CLIP large language model for feature extraction to obtain image embedding vectors and text embedding vectors;

[0088] S204: Build a generative adversarial network, input the image embedding vector and the text embedding vector into the generative adversarial network for training, and obtain a trained generator sub-network and a discriminator sub-network;

[0089] S205. A graph-to-text model is constructed based on the CLIP large language model and the generator sub-network.

[0090] Specifically, the embodiments of the present invention use an intelligent image-to-text generation method, combined with artificial intelligence, generative AI, and multimodal data interaction technology, to intelligently generate accurate, reliable, and intuitive text descriptions and ecological environment evaluations based on remote sensing image data captured by satellites, drones, or monitoring stations, thereby assisting monitoring personnel in quickly making accurate judgments on ecological security.

[0091] The embodiment of the present invention combines the generative adversarial network and the pre-trained CLIP large language model to train an image-to-text model for generating text from images. The specific process is as follows:

[0092] 1) Data collection, mainly relying on low-orbit satellites and drones to obtain high-definition remote sensing images and infrared images I ;

[0093] 2) Build a data set, with each high-definition remote sensing and infrared image given a professional text description by professional ecological security monitoring experts As labels, images and text descriptions need to be strictly aligned. To ensure the semantic consistency between images and text, the CLIP Score indicator is used to quantitatively measure the semantic consistency between images and text descriptions given by experts. The specific calculation formula is:

[0094]

[0095] in, is the embedding vector of the image; is the embedding vector of the text, is the cosine similarity;

[0096] 3) Image preprocessing: First, standardize the image. Scaling the captured high-definition remote sensing images and infrared images to a fixed resolution and normalizing them. The normalization operation is based on:

[0097]

[0098] in, I is the pixel value matrix of the original input image; is the mean of each color channel, which is used to centralize the data to zero mean; is the standard deviation of each color channel, which is used to scale the data to unit variance;

[0099] Perform image enhancement on the normalized images, including but not limited to random cropping, horizontal flipping, and color jittering, to increase the image diversity of the dataset and prevent overfitting;

[0100] 4) Text preprocessing, using a pre-trained tokenizer to map the text into a sequence of tokens:

[0101]

[0102] Text length padding / truncation: unify the length of the mapped word sequence, fill in zeros if the length is insufficient, and truncate the excessive length into two sequences;

[0103] 5) Cross-modal feature extraction: using the pre-trained CLIP large language model to extract cross-modal features from images and text;

[0104] Extract image embedding vectors through CLIP’s Vision Transformer:

[0105]

[0106] Extract text embedding vectors through CLIP's text encoder for supervised training:

[0107]

[0108] 6) Construct a generative adversarial network, including a generator subnetwork and a discriminator subnetwork. Input the image embedding vector and text embedding vector into the generative adversarial network for training, and obtain the trained generator subnetwork and discriminator subnetwork.

[0109] 7) A graph-to-text model is constructed based on the CLIP large language model and the generator sub-network.

[0110] like Figure 3 FIG2 is a flowchart of step S204 provided in an embodiment of the present invention, referring to FIG2 Figure 3 As an optional implementation, the image embedding vector and the text embedding vector are input into a generative adversarial network for training to obtain a trained generator subnetwork and a discriminator subnetwork, which specifically includes:

[0111] S2041. Initialize the generator sub-network and the discriminator sub-network;

[0112] S2042. Input the image embedding vector into the generator sub-network to obtain a generated text sequence;

[0113] S2043, inputting the generated text sequence and the text embedding vector into the discriminator sub-network to obtain the authenticity probability of the generated text sequence and the semantic matching score with the image embedding vector;

[0114] S2044. Calculate the adversarial loss value based on the authenticity probability, calculate the semantic alignment loss value based on the semantic matching score, and determine the gradient penalty term based on the linear interpolation of the generated text sequence and the text embedding vector

[0115] S2045. Determine a first loss value of the generator sub-network based on the adversarial loss value and the gradient penalty term, and determine a second loss value of the discriminator sub-network based on the adversarial loss value and the semantic alignment loss value;

[0116] S2046. Jointly optimize the generator sub-network and the discriminator sub-network according to the first loss value and the second loss value to obtain trained generator sub-network and discriminator sub-network.

[0117] Specifically, the generator subnetwork input is the image embedding , the output is the probability distribution of generated text sequence ; The structure of the generator sub-network first includes a conditional injection module, Projection is the initial state of the decoder:

[0118]

[0119] in, is the image feature extracted by the pre-trained CLIP visual encoder; is the weight matrix, which is also the learnable parameter; It is a bias term and also a learnable parameter.

[0120] Then, we build the generator sub-network architecture by referring to the decoder structure of Transformer and generate text by autoregression:

[0121]

[0122] in, It is the word unit generated at the last moment, which is converted into a vector through the word embedding layer , as the input of the decoder at the current moment, guiding the generation of the next word; , is the hidden state of the previous moment, which stores the context information of the historical generated content; The cross-attention mechanism is injected into each layer of the decoder to ensure that each generated word is aligned with the image semantics. It is the hidden state at the current moment, which integrates the information of the previous word, historical state and image features; is the weight matrix, and the hidden state Mapped to a probability distribution over the vocabulary; Predict the generated word at the current moment The probability of , and the most likely word is selected after SoftMax normalization.

[0123] At the moment , input start character embedded , combined with generate ;calculate , sample to get word unit ; Repeat this process until a terminator is generated or the maximum length is reached .

[0124] In the Transformer Decoder, the image features and text generation are dynamically aligned through the cross-attention mechanism; the input of the discriminator subnetwork is the text sequence , the output is the probability of authenticity and semantic matching score; the discriminator sub-network architecture first generates text features extracted by the CLIP text encoder:

[0125]

[0126] in, Generate text features from the network for the generator.

[0127] Calculate the authenticity of the generated text features through the adversarial discriminative head:

[0128]

[0129] in, The embedding vector of the text generated by the generator after passing through the CLIP text encoder, the dimension is consistent with the output of CLIP; is the weight matrix of the linear layer, mapping the embedding vector to a scalar value; is the bias, which is a scalar value; is a sigmoid function that converts the output of the linear layer into a probability between 0 and 1, indicating the probability that the discriminator believes the text is real.

[0130] Calculate the cosine similarity with the image features to get the semantic alignment score as feedback and scoring for the generator subnetwork:

[0131]

[0132] in, Image features extracted by CLIP’s Vision Transformer.

[0133] Design the loss function and use joint adversarial loss and CLIP semantic alignment loss as the loss function for model training.

[0134] The goal of the adversarial loss is to make it difficult for the discriminator to identify the generated text as a fake sample. It consists of two parts: Wasserstein GAN loss and gradient penalty term (WGAN-GP). Wasserstein GAN loss is used to improve training stability and is expressed as:

[0135]

[0136] in, It is the adversarial loss value, reflecting the difference between the generated sample and the real sample from the perspective of the discriminator; To take the expectation of the real text distribution; Take the expectation of the text distribution generated by the generator sub-network;

[0137] The gradient penalty term expression is:

[0138]

[0139] in, is the linear interpolation of real and generated samples, Representation Discriminator D Interpolation samples The gradient, represents the L2 norm.

[0140] CLIP semantic alignment loss is used to maximize the cross-modal similarity between generated text and images, expressed as:

[0141]

[0142] in, N is the batch size, and the discrimination ability is enhanced by negative sample comparison;

[0143] The joint total loss function includes the total loss of the generator and the total loss of the discriminator. The total loss expression of the generator is:

[0144]

[0145] The total loss expression of the discriminator is:

[0146]

[0147] in, and These are typical hyperparameters.

[0148] Formulate a model training strategy and adopt a two-stage model training strategy, including generator pre-training and adversarial fine-tuning. In the generator pre-training stage, only the , use the teacher to force the injection of real words, initialize the reasonable text generation ability, and gradually reduce the teacher force ratio in the subsequent training process to improve the generation robustness; in the adversarial fine-tuning stage, fix the CLIP encoder and jointly optimize and ,Gumbel-SoftMax relaxation is used to solve the gradient break caused by the discreteness of text.

[0149] like Figure 4 FIG. 1 is a flowchart of step S103 provided in an embodiment of the present invention, referring to FIG. Figure 4 As an optional implementation method, feature extraction and data fusion are performed on the ecological environment description text, ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information to obtain multimodal fusion data, which specifically includes:

[0150] S1031, data cleaning and data alignment are performed on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data and the remote sensing image information;

[0151] S1032, feature extraction is respectively performed on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data and the remote sensing image information after data alignment, to obtain first text features, second text features, environmental parameter features, attribute numerical features and image features;

[0152] S1033, normalization processing is performed on the first text features, the second text features, the environmental parameter features, the attribute numerical features and the image features, and data fusion is performed through a pre-constructed Transformer model, to obtain multi-modal fusion data.

[0153] Specifically, the multi-source heterogeneous data intelligent fusion method of the embodiment of the application integrates data from different sources (sensors), formats and structures, including but not limited to text, image and time series, into a unified representation to support subsequent downstream tasks, and the specific process is as follows:

[0154] 1) Data cleaning, including three parts of missing value processing, noise processing and outlier processing; the missing value processing method adopts a mean value method, and the numerical values of each two data before and after the missing value are added and averaged to obtain the numerical value of the missing value;

[0155] The noise processing adopts a data denoising method based on wavelet transform, and the specific operation includes:

[0156] (1) Signal decomposition, the input signal data is decomposed by multi-scale wavelet to obtain approximation coefficients (low-frequency components) and detail coefficients (high-frequency components), and the decomposition is based on the calculation formula:

[0157]

[0158] wherein, is a wavelet coefficient, is a set of wavelet functions generated by a wavelet base function through translation and scaling, and satisfies , j is a scale parameter, k is a translation parameter;

[0159] (2) Threshold processing, the wavelet coefficient is subjected to hard threshold processing to remove noise components, and the hard threshold calculation expression is:

[0160]

[0161] wherein, is a threshold value, which is usually calculated by the formula Calculated; is the noise standard deviation, is the signal length;

[0162] (3) Signal reconstruction, using the processed wavelet coefficients Reconstruct the denoised data:

[0163]

[0164] The outlier processing adopts the multivariate linear regression method to draw the regression graph and remove the outlier values ​​that deviate significantly from the regression curve;

[0165] 2) Data alignment, which includes time alignment and spatial alignment. Time alignment interpolates or resamples data with different timestamps; spatial alignment aligns or interpolates data with different spatial resolutions.

[0166] 3) Feature extraction: For text data, word embedding operations are used to convert text into vectors; for image data, convolutional neural networks are used to extract potential image features; for time series data, long short-term memory networks are used to extract time series features; for structured data, numerical features are directly used or standardized;

[0167] 4) Data standardization: normalize or standardize all features to the same dimension;

[0168] 5) Perform data fusion through the pre-built Transformer model to obtain multimodal fusion data.

[0169] like Figure 5 FIG. 1 is a flowchart of step S1033 provided in an embodiment of the present invention, referring to FIG. Figure 5 As an optional implementation, data fusion is performed through a pre-built Transformer model to obtain multimodal fusion data, which specifically includes:

[0170] S10331, performing feature splicing on the normalized first text feature, the second text feature, the environmental parameter feature, the attribute value feature, and the image feature to obtain a first spliced ​​feature;

[0171] S10332. Position-encode the first splicing feature to obtain a second splicing feature;

[0172] S10333. Add a multi-head self-attention mechanism to the second splicing feature to obtain multimodal fusion data.

[0173] Specifically, for each data source, the extracted features are expressed as ,in is the number of samples, is the feature dimension of the first i data source; concatenate the features of all data sources as a unified input representation:

[0174]

[0175] where ;

[0176] Since the Transformer itself does not have position information, position encoding needs to be added:

[0177]

[0178] where, P is the position encoding matrix, which can be generated using sine and cosine functions; add position encoding to the input representation:

[0179]

[0180] Add multi-head self-attention mechanism, multi-head self-attention mechanism captures the dependency relationship in the data by calculating the relationship between query (Q), key (K) and value (V), and the calculation expression is:

[0181]

[0182]

[0183]

[0184] where, , , is a learnable weight matrix;

[0185] Calculate the attention score:

[0186]

[0187] Calculate the weighted value:

[0188]

[0189] Use multi-head attention mechanism to capture features in different subspaces:

[0190]

[0191] where,

[0192]

[0193] is the output weight matrix;

[0194] The output of the multi-head attention is nonlinearly transformed by a feedforward neural network:

[0195]

[0196] wherein, and is a bias;

[0197] The residual connection and layer normalization are added after each sublayer, and the Transformer model is obtained. The first text feature, the second text feature, the environmental parameter feature, the attribute numerical feature and the image feature after normalization are input into the Transformer model, and the multi-modal fusion data is obtained.

[0198] As Figure 6 shown is a step flowchart for training an environmental parameter prediction model according to an embodiment of the present application. Referring to Figure 6 , as a further optional implementation, the environmental parameter prediction model is trained by the following steps:

[0199] S301, obtaining historical multi-modal fusion data of a target black soil watershed;

[0200] S302, sampling the historical multi-modal fusion data according to a preset prediction time step to obtain multi-modal fusion samples and corresponding environmental parameter labels;

[0201] S303, inputting the multi-modal fusion samples into a pre-constructed radial basis function neural network to obtain an environmental parameter prediction value;

[0202] S304, determining a third loss value according to a root mean square error between the environmental parameter prediction value and the environmental parameter label;

[0203] S305, updating parameters of the radial basis function neural network according to the third loss value to obtain a trained environmental parameter prediction model.

[0204] Specifically, the embodiment of the present application predicts the change trend of the ecological safety index of the black soil watershed according to the current time series data by deeply mining and learning the distribution and change characteristics of the multi-source heterogeneous data, especially the time series data, and obtains an environmental parameter prediction value. The specific process is as follows:

[0205] 1) Obtain historical multi-modal fusion data of a target black soil watershed, and divide the training set, the validation set and the test set according to the ratio of 7:2:1;

[0206] 2) Determine a prediction time step, and sample the data at the time interval of the step to obtain multi-modal fusion samples and corresponding environmental parameter labels;

[0207] 3) Build a radial basis function neural network, train the neural network model on the training and validation datasets, save the trained weight model file, load the weight model file, complete the model performance test on the test dataset, and select the neural network model with excellent performance for deployment;

[0208] 4) The model performance test uses the root mean square error indicator, which is expressed as:

[0209]

[0210] in, is the number of samples, For the i True value, For the i predicted values.

[0211] like Figure 7 FIG. 1 is a flowchart of step S105 provided in an embodiment of the present invention, referring to FIG. Figure 7 As an optional implementation, when the predicted environmental parameters exceed the preset safety threshold range, an ecological safety warning message is generated, which specifically includes:

[0212] S1051. When the predicted environmental parameter exceeds a preset safety threshold range, determine the target parameter index and target time node that exceed the safety threshold range;

[0213] S1052. Generate ecological security early warning information based on target parameter indicators and target time nodes.

[0214] Specifically, when an embodiment of the present invention predicts that a certain parameter indicator may exceed a corresponding safety threshold, an early warning is given in a timely manner, and a specific time point at which the threshold may be exceeded is given to assist monitoring personnel in responding in advance.

[0215] The above describes the method flow of an embodiment of the present invention. It is understood that the embodiment of the present invention automatically generates ecological environment description text based on remote sensing image information of the target black soil watershed through a graph-to-text model, combines the collected ecological environment report text, environmental parameter time series data, and structured attribute data to generate multimodal fusion data, predicts the environmental parameter change trend of the multimodal fusion data through a pre-trained environmental parameter prediction model, and issues an early warning when the predicted environmental parameters exceed the safety threshold range, thereby improving the accuracy and timeliness of ecological security monitoring and early warning in the black soil watershed.

[0216] Compared with the prior art, the embodiments of the present invention also have the following advantages:

[0217] 1) This embodiment of the present invention proposes an image-to-text generation method based on a combination of a generative adversarial network (GAN) and a pre-trained CLIP large language model. This approach leverages the generative power of GAN and the cross-modal understanding capabilities of CLIP to achieve image-to-text conversion. The generator subnetwork generates text, while the discriminator subnetwork evaluates the authenticity of the generated text and its semantic consistency with the image. Adversarial training improves the quality and reliability of the generated text. Furthermore, the CLIP model extracts features from both the image and text, and calculates the CLIP score, which serves as a measure of semantic consistency, guiding the generator to produce text descriptions that align with the image content. A two-stage training strategy (generator pre-training and adversarial fine-tuning) improves model training efficiency and stability. In the generator pre-training phase, a teacher is used to inject real vocabulary to initialize reasonable text generation capabilities. In the adversarial fine-tuning phase, the generator and discriminator are jointly optimized to further improve the quality of the generated text and its semantic consistency with the image.

[0218] 2) This paper proposes a multi-source heterogeneous data fusion method based on the Transformer model architecture for processing the diverse data types (images, text, time series, and structured data) involved in ecological security monitoring in black soil watersheds. This innovative approach applies the Transformer model, which excels at capturing long-range dependencies and contextual information in sequence data. By leveraging the Transformer model's self-attention mechanism, this method can better mine potential connections between different data sources and achieve deep fusion. Different feature extraction methods (CNN, word embedding, LSTM, etc.) are employed for different types of data (images, text, time series, etc.) and integrated into the Transformer model input to achieve efficient feature fusion.

[0219] 3) This invention uses a radial basis function neural network (RBFNN) to predict the changing trends of ecological security indicators in the black soil watershed and issues early warning information based on the predicted results. RBFNN has strong nonlinear fitting and generalization capabilities, making it effective for predicting time series data. By training on time series data fused from multiple sources of heterogeneous data, the RBFNN learns the changing patterns of indicators and predicts future trends. Based on the predicted results, it sets appropriate early warning thresholds. When a predicted indicator is likely to exceed the threshold, a timely warning is issued, along with the specific time point at which the threshold is likely to be exceeded, providing decision support for monitoring personnel.

[0220] The embodiments of the present invention are of great significance for ensuring the sustainable utilization of black soil resources, promoting the green development of regional economies, and maintaining global ecological and environmental security:

[0221] 1) Improve the accuracy and timeliness of monitoring and early warning

[0222] Ecological and environmental data in black soil watersheds is multi-source, heterogeneous, and complex, making it difficult for traditional monitoring and early warning methods to comprehensively and accurately acquire and analyze this data. The embodiments of the present invention can improve the comprehensiveness and accuracy of data by integrating and consolidating multiple data sources. At the same time, by utilizing advanced data processing and analysis technologies, real-time monitoring and early warning of the ecological environment in black soil watersheds can be achieved, enabling timely identification of potential risks and improving the timeliness of early warnings. This is of great significance for preventing and reducing the occurrence of ecological disasters and ensuring the sustainable utilization of black soil resources.

[0223] 2) Promote the innovation and development of ecological environment monitoring technology

[0224] Research on intelligent monitoring and early warning methods based on multi-source heterogeneous data fusion will promote innovation and development in ecological and environmental monitoring technology. Traditional ecological and environmental monitoring technologies often rely on a single monitoring method and data source, making it difficult to achieve comprehensive and accurate monitoring. Embodiments of the present invention can integrate multiple data sources and monitoring methods to form a comprehensive and accurate ecological and environmental monitoring network. This will provide new ideas and methods for innovation in ecological and environmental monitoring technology, driving its development towards greater efficiency and intelligence.

[0225] 3) Promote green development of the regional economy

[0226] The Black Soil Basin is an important grain production base and agricultural economic zone, and its ecological security is directly related to the sustainable development of the regional economy. The embodiments of the present invention can achieve real-time monitoring and early warning of the ecological environment in the Black Soil Basin, providing decision-makers with a scientific basis for formulating reasonable protection and management measures. This will help reduce the occurrence of ecological disasters, ensure the stability of agricultural production, and promote the green development of the regional economy. Furthermore, the application of intelligent monitoring and early warning technology can also improve the intelligence level of agricultural production and promote the process of agricultural modernization.

[0227] 4) Improving global ecological and environmental security

[0228] As one of the planet's most precious soil resources, the ecological security of black soil basins is crucial not only to the sustainable development of regional economies but also to the stability and security of the global ecological environment. The embodiments of the present invention enable real-time monitoring and early warning of the ecological environment in black soil basins, enabling timely identification of potential risks and providing a scientific basis for global ecological and environmental security. This will help promote the protection and management of the global ecological environment and foster its sustainable development.

[0229] 5) Promote interdisciplinary cooperation and communication

[0230] The embodiments of this invention involve knowledge and technologies from multiple disciplines, including information technology, ecology, geography, and environmental science. This will promote interdisciplinary collaboration and exchange, driving the development and progress of related disciplines. Furthermore, interdisciplinary collaboration and exchange will contribute to more comprehensive and in-depth research results, providing strong support for the improvement and promotion of intelligent monitoring and early warning methods for ecological security in black soil watersheds.

[0231] 6) Provide scientific basis for policy making

[0232] When formulating policies to protect the ecological security of the Black Soil River Basin, relevant agencies require comprehensive and accurate data support. Embodiments of the present invention can provide real-time, accurate data and information, providing a scientific basis for policymakers. This will facilitate the formulation of more rational and effective policies and measures, promoting the protection and management of the ecological security of the Black Soil River Basin.

[0233] 7) Promoting the development of ecological civilization

[0234] By monitoring and providing early warnings of ecological and environmental changes in the Black Soil Basin in real time, ecological problems can be promptly identified and addressed, promoting ecosystem restoration and protection. This will contribute to building a more beautiful and livable ecological environment and promoting the development of an ecological civilization.

[0235] In summary, the embodiments of the present invention are of great significance for ensuring the sustainable utilization of black soil resources, promoting the green development of the regional economy, and maintaining global ecological and environmental security.

[0236] like Figure 8 The diagram shows the structure of the black soil basin ecological security monitoring device based on multi-source data fusion provided by the embodiment of the present invention. Figure 8 The embodiment of the present invention provides a black soil watershed ecological security monitoring device based on multi-source data fusion, comprising:

[0237] Multi-source data acquisition module, used to obtain ecological environment report text, environmental parameter time series data, structured attribute data and remote sensing image information of the target black soil watershed;

[0238] The image-to-text module is used to input remote sensing image information into a pre-trained image-to-text model to obtain a text description of the ecological environment;

[0239] Multi-source data fusion module, used to extract features and fuse data from ecological environment description text, ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information to obtain multimodal fusion data;

[0240] The environmental parameter prediction module is used to input the multimodal fusion data into the pre-trained environmental parameter prediction model to obtain the predicted environmental parameters of the target black soil watershed in the future period;

[0241] The ecological safety early warning module is configured to generate ecological safety early warning information when the predicted environmental parameter exceeds the preset safety threshold range.

[0242] The contents in the method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions same as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0243] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned black soil basin ecological safety monitoring method based on multi-source data fusion. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0244] As Figure 9 Fig. 1 shows a hardware structure schematic diagram of an electronic device provided by the embodiment of the present application, referring to Figure 9 The embodiment of the present application provides an electronic device, which comprises:

[0245] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute related programs to realize the technical solutions provided by the embodiment of the present application.

[0246] The memory 902 can be implemented in a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiment of the present application are implemented by software or firmware, the related program codes are saved in the memory 902 and are called and executed by the processor 901 to realize the black soil basin ecological safety monitoring method based on multi-source data fusion of the embodiment of the present application.

[0247] The input / output interface 903 is used to realize information input and output.

[0248] The communication interface 904 is used to realize communication interaction between the device and other devices, and can realize communication through a wired mode (for example, a USB, a network cable, etc.) or a wireless mode (for example, a mobile network, WIFI, Bluetooth, etc.).

[0249] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0250] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0251] like Figure 10 FIG2 is a schematic diagram of the structure of the storage medium provided by the embodiment of the present invention, referring to FIG2 Figure 10 An embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs 1001, and the one or more programs 1001 can be executed by one or more processors to implement the above-mentioned black soil basin ecological security monitoring method based on multi-source data fusion.

[0252] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0253] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0254] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0255] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features described above can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules, unless expressly stated otherwise. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the properties, functions and internal relationships of the various functional modules disclosed in the devices herein are considered to be within the ordinary skill in the engineer's art, given the benefit of this disclosure. Accordingly, the present application is not to be limited by the above description and examples, but is instead to be defined by the appended claims, in which changes can be made without departing from the scope and nature of the application disclosed herein.

[0256] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0257] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, device or apparatus) or in conjunction with these instructions execution system, device or apparatus. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by an instruction execution system, device or apparatus or in conjunction with these instruction execution system, device or apparatus.

[0258] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0259] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware which are stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0260] In the above description of the present specification, the description referring to the terms "one embodiment", "another embodiment", or "certain embodiments" or the like means that a specific feature, structure, material or characteristic described in connection with the embodiments or examples is included in at least one embodiment or example of the present application. The illustrative expressions of the above terms do not necessarily refer to the same embodiment or example in the present specification. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0261] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, alternatives and variations to these embodiments can be made without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

[0262] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A method for monitoring ecological security of black soil watershed based on multi-source data fusion, characterized in that: The following steps are involved: Obtain ecological environment report text, environmental parameter time series data, structured attribute data, and remote sensing image information for the target black soil watershed; Inputting the remote sensing image information into a pre-trained image-to-text model to obtain an ecological environment description text; Performing feature extraction and data fusion on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain multimodal fusion data; Inputting the multimodal fusion data into a pre-trained environmental parameter prediction model to obtain predicted environmental parameters of the target black soil watershed in the future period; When the predicted environmental parameters exceed the preset safety threshold, an ecological safety warning message is generated; The graph-to-text model is trained by the following steps: Obtain preset remote sensing image samples and determine the corresponding text description information through manual annotation; Normalizing and enhancing the remote sensing image samples to obtain training samples, and mapping the text description information into word sequences; Inputting the training sample and the word sequence into the CLIP large language model for feature extraction to obtain an image embedding vector and a text embedding vector; Constructing a generative adversarial network, inputting the image embedding vector and the text embedding vector into the generative adversarial network for training, and obtaining a trained generator subnetwork and a discriminator subnetwork; The graph-to-text model is formed according to the CLIP large language model and the generator sub-network; The environmental parameter prediction model is trained by the following steps: Acquiring historical multimodal fusion data of the target black soil watershed; Sampling the historical multimodal fusion data according to a preset prediction time step to obtain multimodal fusion samples and corresponding environmental parameter labels; Inputting the multimodal fusion samples into a pre-built radial basis function neural network to obtain environmental parameter prediction values; Determining a third loss value according to a root mean square error between the environmental parameter prediction value and the environmental parameter label; The parameters of the radial basis function neural network are updated according to the third loss value to obtain the trained environmental parameter prediction model.

2. The method for monitoring ecological security of a black soil watershed based on multi-source data fusion according to claim 1, characterized in that: The step of inputting the image embedding vector and the text embedding vector into the generative adversarial network for training to obtain a trained generator subnetwork and a discriminator subnetwork specifically includes: Initialize the generator subnetwork and the discriminator subnetwork; Inputting the image embedding vector into the generator sub-network to generate a text sequence; Inputting the generated text sequence and the text embedding vector into the discriminator subnetwork to obtain the authenticity probability of the generated text sequence and the semantic matching score with the image embedding vector; Calculating an adversarial loss value based on the authenticity probability, calculating a semantic alignment loss value based on the semantic matching score, and determining a gradient penalty term based on linear interpolation of the generated text sequence and the text embedding vector; Determining a first loss value of the generator sub-network according to the adversarial loss value and the semantic alignment loss value, and determining a second loss value of the discriminator sub-network according to the adversarial loss value and the gradient penalty term; The generator subnetwork and the discriminator subnetwork are jointly optimized according to the first loss value and the second loss value to obtain the trained generator subnetwork and the discriminator subnetwork.

3. The method for monitoring ecological security of a black soil watershed based on multi-source data fusion according to claim 1, characterized in that: The step of performing feature extraction and data fusion on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain multimodal fusion data specifically includes: Performing data cleaning and data alignment on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information; Performing feature extraction on the aligned ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data, and the remote sensing image information to obtain first text features, second text features, environmental parameter features, attribute value features, and image features; The first text feature, the second text feature, the environmental parameter feature, the attribute numerical feature and the image feature are normalized, and data fusion is performed through a pre-built Transformer model to obtain the multimodal fusion data.

4. The method for monitoring ecological security of a black soil watershed based on multi-source data fusion according to claim 3 is characterized in that: The data fusion is performed by using a pre-built Transformer model to obtain the multimodal fusion data, which specifically includes: Performing feature splicing on the normalized first text feature, the second text feature, the environmental parameter feature, the attribute value feature, and the image feature to obtain a first spliced ​​feature; Performing position encoding on the first splicing feature to obtain a second splicing feature; A multi-head self-attention mechanism is added to the second splicing feature to obtain the multimodal fusion data.

5. A method for monitoring ecological security of a black soil watershed based on multi-source data fusion according to any one of claims 1 to 4, characterized in that: When the predicted environmental parameters exceed the preset safety threshold range, generating ecological safety warning information, which specifically includes: When the predicted environmental parameter exceeds a preset safety threshold range, determining the target parameter index and target time node that exceed the safety threshold range; The ecological safety early warning information is generated according to the target parameter indicators and the target time node.

6. A black soil watershed ecological security monitoring device based on multi-source data fusion, characterized in that: include: Multi-source data acquisition module, used to obtain ecological environment report text, environmental parameter time series data, structured attribute data and remote sensing image information of the target black soil watershed; A picture-to-text module is used to input the remote sensing image information into a pre-trained picture-to-text model to obtain a text describing the ecological environment; A multi-source data fusion module is used to perform feature extraction and data fusion on the ecological environment description text, the ecological environment report text, the environmental parameter time series data, the structured attribute data and the remote sensing image information to obtain multimodal fusion data; An environmental parameter prediction module, configured to input the multimodal fusion data into a pre-trained environmental parameter prediction model to obtain predicted environmental parameters of the target black soil watershed in the future period; An ecological safety early warning module is used to generate ecological safety early warning information when the predicted environmental parameters exceed a preset safety threshold range; The graph-to-text model is trained by the following steps: Obtain preset remote sensing image samples and determine the corresponding text description information through manual annotation; Normalizing and enhancing the remote sensing image samples to obtain training samples, and mapping the text description information into word sequences; Inputting the training sample and the word sequence into the CLIP large language model for feature extraction to obtain an image embedding vector and a text embedding vector; Constructing a generative adversarial network, inputting the image embedding vector and the text embedding vector into the generative adversarial network for training, and obtaining a trained generator subnetwork and a discriminator subnetwork; The graph-to-text model is formed according to the CLIP large language model and the generator sub-network; The environmental parameter prediction model is trained by the following steps: Acquiring historical multimodal fusion data of the target black soil watershed; Sampling the historical multimodal fusion data according to a preset prediction time step to obtain multimodal fusion samples and corresponding environmental parameter labels; Inputting the multimodal fusion samples into a pre-built radial basis function neural network to obtain environmental parameter prediction values; Determining a third loss value according to a root mean square error between the environmental parameter prediction value and the environmental parameter label; The parameters of the radial basis function neural network are updated according to the third loss value to obtain the trained environmental parameter prediction model.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the black soil watershed ecological security monitoring method based on multi-source data fusion as described in any one of claims 1 to 5 are realized.

8. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the black soil basin ecological security monitoring method based on multi-source data fusion as described in any one of claims 1 to 5.

Images