Video and remote sensing fused spartina alterniflora intrusion detection method, equipment and medium

Abstract

The invention discloses a spartina alterniflora intrusion detection method and device fusing video and remote sensing, and a medium. The method comprises the following steps: obtaining video data and remote sensing data of a to-be-detected area towards the sea side; performing target detection on the spartina alterniflora in the seaside video data to obtain a first spartina alterniflora feature extraction result; vegetation extraction and classification are carried out on the remote sensing data, spartina alterniflora is distinguished from other vegetation, and a second spartina alterniflora feature extraction result is obtained; performing space-time consistency analysis on the seaside video data and the remote sensing data; estimating a motion vector in the video data facing the sea side by using an optical flow algorithm; and based on the motion vector, fusing the first spartina alterniflora feature extraction result and the second spartina alterniflora feature extraction result.

Description

Technical Field

[0001] The present invention belongs to the technical field of ecological monitoring, and in particular relates to a method, device and medium for detecting the invasion of Spartina alterniflora by integrating video and remote sensing. Background Art

[0002] Spartina alterniflora, as an alien invasive plant, has caused serious damage to the ecological environment worldwide in recent years, especially in coastal wetlands and estuaries. Its strong reproductive ability and adaptability to the environment enable Spartina alterniflora to quickly occupy and squeeze out local plant populations, leading to a series of ecological problems such as decreased biodiversity, increased soil salinization, and impaired intertidal zone ecological service functions. Therefore, timely and accurate monitoring of the invasion dynamics of Spartina alterniflora is of great significance for formulating effective prevention and control strategies and protecting the ecological environment.

[0003] Traditional methods for monitoring Spartina alterniflora rely on manual field surveys, which is not only time-consuming and labor-intensive, but also difficult to achieve real-time monitoring of large areas. With the rapid development of video monitoring technology, the use of drones, remote cameras and other equipment to collect video data has provided a new way to monitor the invasion of Spartina alterniflora. However, due to the limitations of the camera device, it is difficult to capture the dynamic changes in a large area, which poses a challenge to accurately extract information about the invasion of Spartina alterniflora from video data.

[0004] In order to solve these problems, methods for detecting the invasion of Spartina alterniflora based on video or remote sensing images have emerged. Among them, video monitoring technology has been widely used in the field of ecological monitoring due to its intuitive, real-time and continuous characteristics. By installing cameras, the growth and spread of Spartina alterniflora can be monitored in real time. However, traditional video monitoring methods have certain limitations in dealing with complex backgrounds, shadow interference and light changes, which affects the monitoring accuracy and efficiency to a certain extent.

[0005] Remote sensing monitoring technology obtains large-scale, high-resolution surface information through platforms such as satellites or drones, providing strong support for ecological monitoring. However, due to the influence of natural factors such as tides, clouds and fog, as well as surface cover, it is difficult to obtain depth information, the resolution is low, and the processing of remote sensing images faces certain challenges.

[0006] Video monitoring and remote sensing monitoring are two important technical means, each with its own unique advantages and limitations. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and provide a method, device and medium for detecting invasion of Spartina alterniflora based on video and remote sensing fusion.

[0008] In a first aspect, an embodiment of the present invention provides a method for detecting invasion of Spartina alterniflora by integrating video and remote sensing, the method comprising:

[0009] Obtain video data and remote sensing data of the seaward side of the area to be detected;

[0010] Target detection is performed on Spartina alterniflora in the video data of the Chaohai side, and the first Spartina alterniflora feature extraction result is obtained;

[0011] Extract and classify vegetation from remote sensing data, distinguish Spartina alterniflora from other vegetation, and obtain the second Spartina alterniflora feature extraction result;

[0012] The temporal and spatial consistency of the video data and remote sensing data on the sea side was analyzed; the motion vector in the video data on the sea side was estimated using the optical flow algorithm; based on the motion vector, the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result were fused.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned Spartina alterniflora invasion detection method that integrates video and remote sensing.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for detecting invasion of Spartina alterniflora by integrating video and remote sensing.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned video- and remote-sensing fusion Spartina alterniflora invasion detection method.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention provides a method for detecting invasion of Spartina alterniflora by integrating video and remote sensing. By integrating video data and remote sensing data, the global range of the video is supplemented, the insufficient resolution of the remote sensing image is compensated, the monitoring accuracy is improved, and the spatiotemporal consistency analysis algorithm is used for identification and verification, the false alarm rate is reduced, and real-time monitoring of the dynamic changes of Spartina alterniflora at a larger spatiotemporal scale is achieved.

[0018] In the process of processing video data, the present invention uses an improved first U-Net network to extract features and classify and identify Spartina alterniflora in video data. The present invention introduces a multi-scale attention module in the first U-Net network, and through a multi-branch parallel structure, while encoding channel information in two spatial directions, captures spatial multi-scale feature representation through convolution, encodes cross-channel information, retains accurate spatial structure information in the channel, and improves the detection accuracy; at the same time, the multi-scale attention module reconstructs part of the channel into a batch processing dimension, and groups the channel dimension into multiple sub-features, recalibrates the channel weight of each parallel branch through the encoded information, and reduces the computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 A flow chart of a method for detecting Spartina alterniflora invasion by integrating video and remote sensing provided in an embodiment of the present invention;

[0021] Figure 2 A structural diagram of an improved first U-Net network provided by an embodiment of the present invention;

[0022] Figure 3 A structural diagram of a multi-scale attention module provided by an embodiment of the present invention;

[0023] Figure 4 The actual distribution map and recognition effect map of Spartina alterniflora provided by the embodiment of the present invention;

[0024] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. It should be understood that the above general description and the detailed description that follows are merely exemplary and explanatory and cannot limit the present application.

[0026] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that, although the term first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination". Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process and method including a series of elements include not only those elements, but also include other elements not clearly listed, or also include elements inherent to such process and method. In the absence of more restrictions, the elements limited by the statement "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0028] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0029] See also Figure 1 The embodiment of the present invention provides a method for detecting invasion of Spartina alterniflora by integrating video and remote sensing, the method comprising:

[0030] Step S1, obtaining sea-facing video data and remote sensing data corresponding to the area to be detected.

[0031] Furthermore, in the process of obtaining the seaward video data corresponding to the area to be detected, it is necessary to ensure that the video data has an appropriate resolution and field of view, and can capture sufficient details and coverage to provide an information source for the monitoring of Spartina alterniflora invasion.

[0032] Furthermore, in the process of acquiring remote sensing data corresponding to the area to be detected, satellite remote sensing multispectral images are used to collect remote sensing data covering the same area, and the corresponding remote sensing data preprocessing algorithm is configured. The medium spatial resolution remote sensing data in the remote sensing data acquisition process are Landsat (US satellite program) series and GF-1WFV satellite images, and the coverage time period matches the video data. It is mainly used to monitor the invasion process of Spartina alterniflora in the study area in terms of time scale and spatial coverage. The high spatial resolution remote sensing data are GF-1PMS and GF-2PMS satellite images and drone data of the corresponding time interval, which are used to verify the monitoring results of Spartina alterniflora.

[0033] In this embodiment, multispectral remote sensing impact data are used, wherein the medium spatial resolution remote sensing data are Landsat (US satellite program) series and GF-1WFV satellite images, and the coverage time period matches the video data, which is mainly used to monitor the invasion process of Spartina alterniflora in the study area in terms of time scale and spatial coverage.

[0034] Table 1 Multispectral remote sensing image parameters

[0035]

[0036] Step S2, performing target detection on Spartina alterniflora in the video data facing the sea, and obtaining a first Spartina alterniflora feature extraction result.

[0037] Specifically, the step S2 includes:

[0038] Step S201, stabilizing, denoising, and color correcting the video data facing the sea;

[0039] Step S202, dividing the seaward side video data into image data, and converting the image data into orthophotos;

[0040] Step S203, inputting the orthophoto into a pre-selected and trained first U-Net network, segmenting the Spartina alterniflora from the background image, and obtaining a Spartina alterniflora recognition image;

[0041] Furthermore, if Figure 2 As shown, the first U-Net network includes 4 downsampling modules connected in sequence and 5 3×3 convolutional layers connected in sequence; the downsampling module includes a 3×3 convolutional layer, a pooling layer and a multi-scale attention module;

[0042] Among them, Figure 3As shown, the input feature map of the multi-scale attention module passes through three parallel paths constructed by two 1x1 average pooling layers and one 3x3 convolution layer respectively. The two 1x1 average pooling layers encode channel information in the two spatial directions of x and y. The features processed by the two 1x1 average pooling layers are concatenated and then convolution feature extraction is performed. After passing through an average pooling layer, the features are dot-product aggregated with the spatial multi-scale features captured by the 3x3 branch convolution to generate the first spatial attention map. At the same time, the features extracted by the 3x3 convolution are average-pooled and then dot-product aggregated with the channel features obtained by convolution processing with the two 1x1 branches to generate the second spatial attention map. Finally, the first spatial attention map and the second spatial attention map are aggregated to obtain the output map of the multi-scale attention module.

[0043] It should be noted that in this example, the image data is divided into a training set, a validation set, and a test set in a ratio of 6:2:2, and the training set is used to train the first U-Net network. It should be understood that the efficient multi-scale attention module reconstructs some channels into batch dimensions, groups the channel dimensions into multiple feature sub-features, and recalibrates the channel weights of each parallel branch through encoding information, thereby reducing the computational overhead while retaining the information of each channel.

[0044] Step S204, color marking the Spartina alterniflora recognition image to obtain a first Spartina alterniflora feature extraction result.

[0045] In this example, the first Spartina alterniflora feature extraction result is as follows Figure 4 As shown, Figure 4 (a) Figure 4 (b) in the figure shows the actual distribution of Spartina alterniflora and the target recognition result of Spartina alterniflora. The image pointed by the arrow below is the enlarged image of the selected area in the image above. Figure 4 Compared with Figure (a), Figure 4 The darkest area in Figure (b) is the labeled Spartina alterniflora.

[0046] Step S3, extracting and classifying vegetation from the remote sensing data, distinguishing Spartina alterniflora from other vegetation, and obtaining a second Spartina alterniflora feature extraction result.

[0047] Specifically, step S3 includes:

[0048] Step S301, performing radiation correction, atmospheric correction, and geometric correction on remote sensing data;

[0049] Step S302, fusing the multispectral band and the panchromatic band in the remote sensing data;

[0050] Step S303, excluding non-vegetation objects in the remote sensing data;

[0051] Step S304, performing image registration on the remote sensing data to eliminate position deviation;

[0052] Step S305, using GPS and camera to record the location and remote sensing interpretation information of the survey site, and establish remote sensing interpretation marks of typical land objects;

[0053] Specifically, in this embodiment, the automatic identification and classification of Spartina alterniflora is performed using Landsat 8 images as the data source. To ensure the reliability of the ground object classification results, the location and remote sensing interpretation information of the survey site are recorded using GPS and camera, respectively, and the remote sensing interpretation marks of 7 major typical ground objects, including Spartina alterniflora, reed marsh, Tamarix shrub, sparse vegetation on tidal flats, bare tidal flats, water bodies and bare land, are established in combination with the images.

[0054] Step S306, extracting grayscale features, RGB average pixel features and local texture features in the remote sensing data, and fusing them into shallow feature fusion bands;

[0055] Specifically, the remote sensing data image is converted into a grayscale image, and the grayscale value of each pixel is extracted as a grayscale feature;

[0056] The average pixel value of the RGB channels of the remote sensing data image is calculated respectively as the RGB average pixel feature;

[0057] The LBP operator is used to extract local texture features in remote sensing data images;

[0058] The frequency histograms of grayscale features, RGB average pixel features and local texture features belonging to different categories are calculated to obtain shallow feature fusion bands.

[0059] Step S307, inputting the shallow feature fusion band into a pre-trained second U-Net network for classification to obtain a second Spartina alterniflora feature extraction result.

[0060] Step S4, performing spatiotemporal consistency analysis on the seaward side video data and the remote sensing data; using an optical flow algorithm to estimate motion vectors in the seaward side video data; and based on the motion vectors, fusing the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result.

[0061] Specifically, the seaward side video data at the initial moment is set as the fusion starting point;

[0062] The motion vector in the seaward video data is estimated using the optical flow algorithm;

[0063] Based on the motion vector, the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result are fused through Kalman filtering:

[0064] During the data fusion process, time correction and space correction are performed to conduct temporal and spatial consistency analysis of the video data and remote sensing data on the sea side.

[0065] Among them, time correction calculates the data changes between adjacent time steps to ensure the smooth transition of the fusion result in time; spatial correction considers the spatial neighborhood relationship to ensure the continuity of the fusion result in space.

[0066] Accordingly, the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for detecting the invasion of Spartina alterniflora by integrating video and remote sensing. Figure 5 As shown, a hardware structure diagram of any device with data processing capability is provided for the method for detecting the invasion of Spartina alterniflora by integrating video and remote sensing provided by an embodiment of the present invention, except Figure 5 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0067] Correspondingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned method for detecting invasion of Spartina alterniflora by integrating video and remote sensing. The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting Spartina alterniflora invasion by integrating video and remote sensing, characterized in that: The method comprises: Obtain video data and remote sensing data of the seaward side of the area to be detected; Target detection is performed on Spartina alterniflora in the video data of the Chaohai side, and the first Spartina alterniflora feature extraction result is obtained; Extract and classify vegetation from remote sensing data, distinguish Spartina alterniflora from other vegetation, and obtain the second Spartina alterniflora feature extraction result; The temporal and spatial consistency of the video data and remote sensing data on the sea side was analyzed; the motion vector in the video data on the sea side was estimated using the optical flow algorithm; based on the motion vector, the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result were fused.

2. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 1, characterized in that: Acquisition of remote sensing data includes: The remote sensing data of the area to be detected is obtained by using satellite remote sensing multispectral images; the coverage time of the remote sensing data is the same as the time corresponding to the video data.

3. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 1, characterized in that: The process of performing target detection on Spartina alterniflora in the video data of the North Sea side and obtaining the first Spartina alterniflora feature extraction result includes: Stabilize, denoise, and color correct the video data from the North Sea side; Segment the seaward side video data into image data, and convert the image data into orthophotos; The orthophoto is input into the pre-selected and trained first U-Net network to segment the Spartina alterniflora from the background image to obtain the Spartina alterniflora recognition image; The Spartina alterniflora recognition image is color-marked to obtain the first Spartina alterniflora feature extraction result.

4. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 3, characterized in that: The first U-Net network includes four downsampling modules connected in sequence and five 3×3 convolutional layers connected in sequence; The downsampling module includes a 3×3 convolution layer, a pooling layer and a multi-scale attention module; Among them, the input feature map of the multi-scale attention module passes through three parallel paths constructed by two 1x1 average pooling layers and one 3x3 convolution layer respectively. The two 1x1 average pooling layers encode channel information in the two spatial directions of x and y. The features processed by the two 1x1 average pooling layers are concatenated and then convolution feature extraction is performed. After passing through an average pooling layer, the features are dot-product aggregated with the spatial multi-scale features captured by the 3x3 branch convolution to generate the first spatial attention map; at the same time, the features extracted by the 3x3 convolution are average-pooled and then dot-product aggregated with the channel features obtained by convolution processing with two 1x1 branches to generate the second spatial attention map. Finally, the first spatial attention map and the second spatial attention map are aggregated to obtain the output map of the multi-scale attention module.

5. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 1, characterized in that: The process of extracting and classifying vegetation from remote sensing data, distinguishing Spartina alterniflora from other vegetation, and obtaining the second Spartina alterniflora feature extraction result includes: Perform radiation correction, atmospheric correction, and geometric correction on remote sensing data; Fusion of multispectral bands and panchromatic bands in remote sensing data; Exclude non-vegetation objects from remote sensing data; Perform image registration on remote sensing data to eliminate position deviation; Establish typical landform remote sensing interpretation signs; Extract grayscale features, RGB average pixel features and local texture features from remote sensing data and fuse them into shallow feature fusion bands; The shallow feature fusion band is input into the pre-trained second U-Net network for classification to obtain the second Spartina alterniflora feature extraction result.

6. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 5, characterized in that: The process of extracting grayscale features, RGB average pixel features, and local texture features from remote sensing data and fusing them into shallow feature fusion bands includes: Convert the remote sensing data image into a grayscale image and extract the grayscale value of each pixel as a grayscale feature; The average pixel value of the RGB channels of the remote sensing data image is calculated respectively as the RGB average pixel feature; The LBP operator is used to extract local texture features in remote sensing data images; The frequency histograms of grayscale features, RGB average pixel features and local texture features belonging to different categories are calculated to obtain shallow feature fusion bands.

7. The method for detecting Spartina alterniflora invasion by integrating video and remote sensing according to claim 1, characterized in that: Conduct temporal and spatial consistency analysis on the video data and remote sensing data on the sea side; use the optical flow algorithm to estimate the motion vector in the video data on the sea side; Based on the motion vector, the process of fusing the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result includes: Set the seaward video data at the initial moment as the fusion starting point; The motion vector in the seaward video data is estimated using the optical flow algorithm; Based on the motion vector, the first Spartina alterniflora feature extraction result and the second Spartina alterniflora feature extraction result are fused through Kalman filtering: During the data fusion process, time correction and space correction are performed to conduct temporal and spatial consistency analysis of the video data and remote sensing data on the sea side.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the Spartina alterniflora invasion detection method based on fusion of video and remote sensing as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for detecting invasion of Spartina alterniflora by integrating video and remote sensing as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for detecting invasion of Spartina alterniflora by integrating video and remote sensing as described in any one of claims 1 to 7 is implemented.

Images