Straw coverage detection method and device based on cross-scale multi-source data fusion

Abstract

The invention provides a straw coverage detection method and device based on cross-scale multi-source data fusion, which are applied to the technical field of remote sensing image processing, and the method comprises the steps: inputting a ground shot picture into a visual large model for semantic segmentation, and obtaining the straw coverage of a corresponding ground scale range; based on the straw coverage of the ground scale range, carrying out scale range adaptation on the unmanned aerial vehicle image to obtain an unmanned aerial vehicle sample image; performing image similarity calculation on the basis of the ground shot pictures corresponding to the unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images to obtain the overall similarity of each unmanned aerial vehicle image block; selecting target unmanned aerial vehicle image blocks with a fixed proportion number to generate a new sample; constructing a straw coverage regression model based on the unmanned aerial vehicle image straw coverage sample set; inputting the satellite remote sensing image into the straw coverage regression model to obtain a straw coverage extraction result of the target area; according to the invention, accurate extraction of the straw coverage in a large-area range can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a straw coverage detection method and device based on cross-scale multi-source data fusion. BACKGROUND

[0002] Traditional methods for detecting straw coverage in farmland include visual judgment or measurement using a tape measure in the field. However, these direct measurement or photographic methods are laborious, time-consuming, and not conducive to large-scale implementation due to their inherent discontinuity.

[0003] Satellite remote sensing technology has become a popular method for estimating crop straw coverage due to its high spatial coverage and fast revisit time. The most widely used method for estimating crop straw coverage through satellite remote sensing is to establish a correlation model between field measurement data and straw coverage spectral indices using parametric and non-parametric methods. Common spectral indices include the normalized difference senescent vegetation index (NDSVI), the normalized difference residue index (NDRI), and the like. The accuracy of the correlation model established by field measurement data and satellite spectral indices largely depends on the number of field observation sample data, and field observation has a high cost.

[0004] Therefore, the straw coverage detection method in the related art has the technical problem of great limitations. SUMMARY

[0005] The present application provides a straw coverage detection method and device based on cross-scale multi-source data fusion, which solves the defect of great limitations of the straw coverage detection method in the prior art and realizes accurate extraction of straw coverage in a large area.

[0006] The application provides a straw coverage detection method based on cross-scale multi-source data fusion, comprising the following steps. Satellite remote sensing images, unmanned aerial vehicle images and ground shooting photos of a target area are obtained; the ground shooting photos are input into a visual large model for semantic segmentation to obtain ground scale range straw coverage corresponding to the ground shooting photos output by the visual large model; the unmanned aerial vehicle images are adapted in scale range based on the ground scale range straw coverage to obtain unmanned aerial vehicle sample images; the unmanned aerial vehicle sample images are divided according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; image similarity calculation is performed on the ground shooting photos corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images to obtain the overall similarity of each unmanned aerial vehicle image block; each unmanned aerial vehicle image block is sorted according to the overall similarity, and a fixed proportion of target unmanned aerial vehicle image blocks are selected for new sample generation to obtain a unmanned aerial vehicle image straw coverage sample set; a straw coverage regression model based on a Transformer model is constructed based on the unmanned aerial vehicle image straw coverage sample set; the satellite remote sensing images are input into the straw coverage regression model to obtain the straw coverage extraction result of the target area output by the straw coverage regression model.

[0007] According to the straw coverage detection method based on cross-scale multi-source data fusion provided by the application, the ground scale range side length corresponding to the ground shooting photos is determined, the ratio of the ground scale range side length to the ground resolution of the unmanned aerial vehicle image is taken as the scale size of the unmanned aerial vehicle image, and the unmanned aerial vehicle image is cropped according to the scale size to obtain a unmanned aerial vehicle sample image, wherein the straw coverage of the unmanned aerial vehicle sample image is associated with the straw coverage of the ground scale range.

[0008] According to the straw coverage detection method based on cross-scale multi-source data fusion provided by the application, the image similarity calculation is performed on the ground shooting photos corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images to obtain the overall similarity of each unmanned aerial vehicle image block, comprising: for each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks, the following processing is performed: a first structural similarity index is determined between the unmanned aerial vehicle image block and the ground shooting photo corresponding to the unmanned aerial vehicle image block; a second structural similarity index is determined between the unmanned aerial vehicle image block and the unmanned aerial vehicle sample image corresponding to the unmanned aerial vehicle image block; the first structural similarity index and the second structural similarity index are weighted and summed to obtain the overall similarity of the unmanned aerial vehicle image block.

[0009] According to the straw coverage detection method of cross-scale multi-source data fusion provided by the application, the fixed proportion of target unmanned aerial vehicle image blocks is selected to generate new samples, and a sample set of unmanned aerial vehicle image straw coverage is obtained, which comprises: determining the straw coverage of the unmanned sample image with the highest overall similarity to the target unmanned aerial vehicle image block; the product of the straw coverage and the overall similarity of the target unmanned aerial vehicle image block is taken as the straw coverage of the target unmanned aerial vehicle image block, and a sample set of unmanned aerial vehicle image straw coverage is obtained.

[0010] According to the straw coverage detection method of cross-scale multi-source data fusion provided by the application, the straw coverage of the unmanned sample image with the highest overall similarity to the target unmanned aerial vehicle image block is determined, which comprises: determining the first structural similarity index between the target unmanned aerial vehicle image block similarity and each ground photograph; determining the second structural similarity index between the target unmanned aerial vehicle image block similarity and the unmanned sample image corresponding to each ground photograph; performing weighted summation based on the first structural similarity index and the second structural similarity index to obtain the overall similarity of each unmanned sample image corresponding to the target unmanned aerial vehicle image block; and taking the straw coverage of the unmanned sample image with the highest overall similarity as the straw coverage of the target unmanned aerial vehicle image block.

[0011] According to the straw coverage detection method of cross-scale multi-source data fusion provided by the application, the straw coverage regression model based on the Transformer model is constructed based on the sample set of unmanned aerial vehicle image straw coverage, which comprises: based on the linear relationship between the spatial resolution of the satellite remote sensing image and the spatial resolution of the unmanned aerial vehicle image, the sample set of unmanned aerial vehicle image straw coverage is aggregated to obtain a sample set of satellite image scale straw coverage; based on the sample set of satellite image scale straw coverage, the satellite image reflectivity and the spatial feature are taken as the input, and the straw coverage is taken as the regression object to construct the straw coverage regression model based on the Transformer model.

[0012] The application further provides a straw coverage detection device based on cross-scale multi-source data fusion, comprising the following modules: an acquisition module, configured to acquire satellite remote sensing images, unmanned aerial vehicle images and ground shooting photos of a target area; a segmentation module, configured to input the ground shooting photos into a visual large model for semantic segmentation to obtain straw coverage of a ground scale range corresponding to the ground shooting photos output by the visual large model; an adaptation module, configured to perform scale range adaptation on the unmanned aerial vehicle images based on the straw coverage of the ground scale range to obtain unmanned aerial vehicle sample images; a division module, configured to divide the unmanned aerial vehicle sample images according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; a similarity module, configured to perform image similarity calculation on the ground shooting photos corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images to obtain overall similarity of the each unmanned aerial vehicle image block; a generation module, configured to sort the each unmanned aerial vehicle image block according to the overall similarity and select a fixed proportion of target unmanned aerial vehicle image blocks to generate new samples to obtain a straw coverage sample set of unmanned aerial vehicle images; a construction module, configured to construct a straw coverage regression model based on a Transformer model based on the straw coverage sample set of unmanned aerial vehicle images; and an extraction module, configured to input the satellite remote sensing images into the straw coverage regression model to obtain a straw coverage extraction result of the target area output by the straw coverage regression model.

[0013] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-scale multi-source data fusion-based straw coverage detection method according to any one of the above when executing the computer program.

[0014] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the cross-scale multi-source data fusion-based straw coverage detection method according to any one of the above.

[0015] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the cross-scale multi-source data fusion-based straw coverage detection method according to any one of the above.

[0016] The straw coverage detection method and device for cross-scale multi-source data fusion provided by the application first use a visual large model to accurately segment a ground photo to obtain a high-reliability ground-scale coverage true value; based on this, the unmanned aerial vehicle image is scaled and divided into grids, and then the similarity between the image blocks and the ground photo is calculated to screen out target image blocks with strong representation for automatically generating an unmanned aerial vehicle coverage sample set; finally, the high-quality samples are used to train a Transformer regression model, and the accurate features learned by small-scale (ground, unmanned aerial vehicle) are effectively transferred to large-scale satellite images, so that the final large-area high-precision straw coverage extraction based on satellite images is realized. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 is a flowchart of the straw coverage detection method for cross-scale multi-source data fusion provided by the application.

[0019] Figure 2 is a sample generation schematic diagram based on a ground photo provided by the application.

[0020] Figure 3 is a sample generation schematic diagram based on unmanned aerial vehicle image augmentation provided by the application.

[0021] Figure 4 is a flowchart of the straw coverage detection method for cross-scale multi-source data fusion provided by the application.

[0022] Figure 5 is a module schematic diagram of the straw coverage detection device for cross-scale multi-source data fusion provided by the application.

[0023] Figure 6 is an entity structure schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be clearly and completely described below in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0026] The present application relates to remote sensing image processing technology, specifically, to a cross-scale multi-source data fusion straw coverage detection method, which realizes the calculation and analysis of the straw coverage of farmland ground in a large area range by cooperatively shooting photos on the ground, unmanned aerial vehicle aerial images and satellite images, and then serves the application needs of farmland plot condition monitoring in agricultural remote sensing monitoring.

[0027] Remote sensing observation data has very wide application prospects and has been effectively applied in many application fields such as environmental monitoring and agricultural remote sensing monitoring. In the field of agriculture, remote sensing technology is widely used for crop growth monitoring and yield estimation, crop growth conditions are analyzed through vegetation index (such as NDVI, EVI), and growth retardation or vigorous growth areas are identified; historical data and real-time remote sensing images are combined to establish a yield model to estimate grain yield in advance; and the crop growth period (sowing, emergence, flowering, maturity, etc.) can be tracked to optimize the arrangement of agricultural activities. At the same time, remote sensing technology has also been widely used in precision agriculture and variable management, which can realize precision fertilization / irrigation by combining soil nutrients and crop demand remote sensing data; and can use unmanned aerial vehicle or satellite data to provide operation guidance for agricultural machinery to generate prescription maps and achieve variable rate fertilization.

[0028] In addition, remote sensing technology can also be used in land use and farmland management, which can identify farmland boundaries, abandoned land and reclamation land to support farmland protection; distinguish crop types (such as wheat / corn) through multi-temporal images to monitor crop rotation systems; evaluate the distribution and status of irrigation channels, roads and other facilities to monitor farmland infrastructure. In the field of agricultural remote sensing monitoring, one of the problems that needs to be paid attention to is the calculation of straw coverage in farmland. The proportion of straw coverage in a certain area of farmland to the ground area is called straw coverage.

[0029] Straw mulching after crop harvesting can reduce wind and water erosion, increase soil organic carbon content and microbial population, improve soil water retention capacity, and enhance soil physical and chemical properties, which is called conservation tillage. Experience has shown that conservation tillage is the most environmentally friendly tillage method, which can significantly reduce greenhouse gas emissions, increase crop yield in some cases, and increase soil microbial diversity and soil organic carbon. Crop straw return as an alternative to traditional straw disposal method provides an effective method to reduce air pollution and reduce harmful emissions. Therefore, how to quickly estimate the straw coverage of farmland on a large scale is crucial.

[0030] Currently, unmanned aerial vehicles are becoming an important data acquisition platform in the field of agricultural remote sensing. Because unmanned aerial vehicles can acquire data at a low flight altitude, the images acquired have great similarity with photos taken on the ground. Therefore, unmanned aerial vehicle images become an important tool for establishing the correlation between ground photos and satellite images.

[0031] In the embodiment of the present application, the correlation between observation data of multiple scales of ground (field ground photos), machine (unmanned aerial vehicle images), and star (satellite remote sensing images) is established by data fusion. Initial training samples are obtained from a small number of ground photos, a large number of samples are expanded by unmanned aerial vehicle images, and finally a straw coverage analysis model is established using satellite images to obtain straw coverage information of farmland in a large area.

[0032] Optionally, the straw coverage detection method of the cross-scale multi-source data fusion of the embodiment of the present application can be executed by a server, or can be executed by a terminal device, or can be executed by a server and a terminal device together. Taking the straw coverage detection method of the cross-scale multi-source data fusion of the present embodiment executed by the server as an example.

[0033] Figure 1 is a flowchart of the straw coverage detection method of the cross-scale multi-source data fusion provided by the present application, as shown in Figure 1 The method comprises the following steps.

[0034] Step 101, acquiring satellite remote sensing images, unmanned aerial vehicle images, and ground photos of a target area.

[0035] Acquiring satellite remote sensing images of a target area comprises: acquiring satellite remote sensing images covering the entire target agricultural area, the image space range should completely contain the farmland plot to be analyzed, and the satellite remote sensing images should contain visible light and near-infrared bands to support subsequent spectral analysis.

[0036] The unmanned aerial vehicle image of the target area is obtained, including: planning a flight route for the unmanned aerial vehicle in the target area, and ensuring that the unmanned aerial vehicle image obtained by aerial photography of the unmanned aerial vehicle covers all farmland plots where the ground photos are sampled. The unmanned aerial vehicle image needs to have an RGB or visible light band to support similarity comparison with the ground photos.

[0037] The ground photos of the target area are obtained, including: uniformly distributing and selecting sampling points in the target farmland area, each sampling point representing an independent observation unit, and a 1m x 1m ground area is photographed at each sampling point, and the camera is vertically downward to ensure that the image is free of distortion, shadow obstruction or glare interference. Clearly cover the straw and bare land, and facilitate visual large model (such as SAM) segmentation and identification.

[0038] It should be noted that the satellite remote sensing image, the unmanned aerial vehicle image and the ground photos need to be obtained within the same crop straw coverage period to avoid errors caused by environmental changes. All images need to be geometrically corrected and matched with geographic coordinates to ensure that the multi-source data at the same position are spatially aligned.

[0039] In step 102, the ground photos are input into a visual large model for semantic segmentation to obtain the straw coverage of the ground scale range corresponding to the ground photos output by the visual large model.

[0040] In the embodiments of the present application, the visual large model is used to automatically analyze the ground photos, identify the straw coverage in the ground photos, and calculate the straw coverage of the ground scale range represented by the ground photos.

[0041] In some embodiments, the visual large model refers to a general model capable of performing semantic segmentation on natural scene pictures, namely a Segment Anything Model (SAM model). The SAM model can automatically perform semantic segmentation on the input ground photos to distinguish between bare land and straw; The straw coverage of the ground scale range represented by the ground photos is that the ground photos are vertically downward, and if the ground range photographed by the photos is 1m x 1m, the straw coverage of the sample point (expressed in percentage) is obtained by dividing the number of pixels representing straw coverage in the ground photos by the total number of pixels in the photos. The ground photos and the obtained straw coverage data constitute a sample based on the ground photos.

[0042] In step 103, the unmanned aerial vehicle image is adapted to the scale range based on the straw coverage of the ground scale range to obtain an unmanned aerial vehicle sample image.

[0043] In the embodiment of the present application, according to the spatial resolution of the unmanned aerial vehicle image, the straw coverage degree in the ground scale range is adapted to the straw coverage degree in the unmanned aerial vehicle scale range to form a straw coverage degree sample based on the unmanned aerial vehicle image (i.e. the unmanned aerial vehicle sample image).

[0044] According to the straw coverage degree detection method for cross-scale multi-source data fusion provided by the present application, the unmanned aerial vehicle image is subjected to scale range adaptation based on the straw coverage degree in the ground scale range to obtain the unmanned aerial vehicle sample image, which comprises the following steps: determining the side length of the ground scale range corresponding to the ground photograph; taking the ratio of the side length of the ground scale range to the ground resolution of the unmanned aerial vehicle image as the scale size of the unmanned aerial vehicle image; cropping the unmanned aerial vehicle image according to the scale size to obtain the unmanned aerial vehicle sample image, wherein the straw coverage degree of the unmanned aerial vehicle sample image is associated with the straw coverage degree of the ground scale range.

[0045] In the embodiment of the present application, the size of the unmanned aerial vehicle image sample is obtained by dividing the side length of the ground range covered by the ground photograph by the ground resolution of the unmanned aerial vehicle image. The unmanned aerial vehicle image is cropped with the size of the obtained unmanned aerial vehicle image sample as a constraint, and the calculated straw coverage degree data are combined to form the unmanned aerial vehicle sample image.

[0046] In some embodiments, the ground side length is fixed at 1 meter (determined by the shooting specification). The ground resolution of the unmanned aerial vehicle image is calculated by the flight height and the camera focal length (for example: 0.02 meters / pixel). If the resolution of the unmanned aerial vehicle is 0.02 meters / pixel, then the sample size = 1 / 0.02 = 50 pixels x 50 pixels.

[0047] With each ground photograph sampling point as the center, the corresponding area is cropped from the unmanned aerial vehicle image according to the calculated size (such as 50x50 pixels) to obtain the unmanned aerial vehicle sample image block. The straw coverage degree of the unmanned aerial vehicle sample image block directly inherits (or corresponds according to a linear relationship) the calculation result (percentage value) of the corresponding ground photograph, without the need for re-calculation.

[0048] Through the embodiment of the present application, the size is calculated by the resolution ratio to ensure that the unmanned aerial vehicle sample strictly corresponds to the actual ground range and eliminate the scale error. The coverage true value of the ground photograph (verified by the visual large model segmentation) is directly inherited to avoid the deviation introduced by the recalculation of the unmanned aerial vehicle scale.

[0049] Step 104, according to the preset rule grid, the unmanned aerial vehicle sample image is divided to obtain a plurality of unmanned aerial vehicle image blocks.

[0050] In the embodiments of the present application, the regular grid is a uniform two-dimensional partition framework covering the entire UAV sample image. The grid size (i.e. the width and height of each cell) is predefined and can be adjusted according to actual application requirements, such as being divided into fixed-size squares, for example, 50x50 pixels or 100x100 pixels.

[0051] The purpose of grid partitioning is to decompose the UAV sample image into multiple independent image blocks for block-by-block processing. This avoids the subjectivity of manual random partitioning and ensures the systematicness and repeatability of the coverage sample generation.

[0052] In some embodiments, a computational geometry algorithm (such as the sliding window method) is used to superimpose the regular grid onto the UAV sample image: starting from the image (0, 0) coordinates, move the window in row-major order with the grid size as the window size. The area covered by each window is cropped into an independent UAV image block. For example, if the sample image size is 500x500 pixels and the grid cell is 50x50 pixels, 100 image blocks (10 rows x 10 columns) are obtained after partitioning. If the image size cannot be evenly divided by the grid size, the remaining boundary pixels are discarded.

[0053] Each output UAV image block is stored as an independent data unit with its grid coordinate index (such as row and column number) for subsequent similarity matching.

[0054] Step 105, based on the ground photograph corresponding to each of the plurality of UAV image blocks and the UAV sample image, image similarity calculation is performed to obtain the overall similarity of each UAV image block.

[0055] In the embodiments of the present application, image similarity calculation is performed on each UAV image block and the corresponding ground photograph and UAV sample image, and the overall similarity is obtained.

[0056] According to the straw coverage detection method provided by the present application, the ground photograph corresponding to each of the plurality of UAV image blocks and the UAV sample image are used to perform image similarity calculation to obtain the overall similarity of each UAV image block, which comprises: For each of the plurality of UAV image blocks, the following processing is performed: determine the first structural similarity index between the UAV image block and the ground photograph corresponding to the UAV image block; determine the second structural similarity index between the UAV image block and the UAV sample image corresponding to the UAV image block; perform weighted summation based on the first structural similarity index and the second structural similarity index to obtain the overall similarity of the UAV image block.

[0057] In the embodiment of the present application, the overall similarity refers to the similarity calculated respectively between the unmanned aerial vehicle image block and the unmanned aerial vehicle sample image and the ground shot photo, wherein the similarity is calculated by using the SSIM (Structural Similarity) index for measuring the structural similarity between two images, and then the two similarity values calculated are multiplied by 0.5 respectively to obtain the overall similarity SSIM_total.

[0058] Here, the first structural similarity index is used to measure the structural similarity between the unmanned aerial vehicle image block and the corresponding ground shot photo. The second structural similarity index is used to measure the structural similarity between the unmanned aerial vehicle image block and the corresponding unmanned aerial vehicle sample image.

[0059] In the embodiment of the present application, the structural similarity index (SSIM) formula is used to calculate the similarity of brightness, contrast and structure between two images.

[0060] Through the embodiment of the present application, the structural similarity between the unmanned aerial vehicle image block and the corresponding ground shot photo ensures the reliability of the true value, and the structural similarity between the unmanned aerial vehicle image block and the corresponding unmanned aerial vehicle sample image verifies the scale consistency.

[0061] Step 106, sort each unmanned aerial vehicle image block according to the overall similarity, and select a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain a set of unmanned aerial vehicle image straw coverage samples.

[0062] In the embodiment of the present application, selecting a fixed proportion refers to arranging the similarity calculated for all unmanned aerial vehicle image blocks in descending order from large to small, and then selecting the top 80% of the unmanned aerial vehicle image blocks in the order to generate samples.

[0063] According to the straw coverage detection method for cross-scale multi-source data fusion provided by the present application, a fixed proportion of target unmanned aerial vehicle image blocks are selected for new sample generation to obtain a set of unmanned aerial vehicle image straw coverage samples, which comprises: determining the straw coverage of the unmanned aerial vehicle sample image with the highest overall similarity to the target unmanned aerial vehicle image block; multiplying the straw coverage and the overall similarity of the target unmanned aerial vehicle image block to obtain the straw coverage of the target unmanned aerial vehicle image block, and obtaining a set of unmanned aerial vehicle image straw coverage samples.

[0064] In the embodiment of the present application, a fixed proportion of image blocks are selected from all unmanned aerial vehicle image blocks to generate new samples, and the sample straw coverage is calculated by normalizing and scaling the most similar sample to obtain a large number of unmanned aerial vehicle image straw coverage samples.

[0065] Here, the normalized scaling method calculation refers to the calculation of the straw coverage of the unmanned aerial vehicle image block to be processed, which is obtained by multiplying the straw coverage of the sample closest to the unmanned aerial vehicle image block by the overall similarity, that is, if the unmanned aerial vehicle image block to be processed is completely consistent with the sample image (SSIM_total=1), the straw coverage of the sample is directly assigned to the unmanned aerial vehicle image block to be processed (xSSIM_total=x1).

[0066] In some embodiments, for each target unmanned aerial vehicle image block, the overall similarity (SSIM_total) of the target unmanned aerial vehicle image block with all known samples (including ground photos and unmanned aerial vehicle image samples) is calculated.

[0067] The sample with the maximum overall similarity is selected as the closest sample, and the straw coverage of the closest sample is read; If the overall similarity of the target unmanned aerial vehicle image block and the closest sample is 1 (completely consistent), the straw coverage of the closest sample is directly inherited; if the target unmanned aerial vehicle image block and the closest sample are different, the straw coverage is attenuated according to the similarity ratio.

[0068] Reference Figure 2 , Figure 2 is a sample generation schematic diagram based on ground photos provided by the application, which comprises: an artificial photographing point / sample based on ground photos, an unmanned aerial vehicle photographing area, and an agricultural area.

[0069] Reference Figure 3 , Figure 3 is a sample generation schematic diagram based on unmanned aerial vehicle image expansion provided by the application, which comprises: a sample based on unmanned aerial vehicle image expansion, an unmanned aerial vehicle photographing area, and an agricultural area.

[0070] According to the embodiment of the application, the similarity weighting mechanism is adopted, so that the closer the image structure is to the reference sample, the more reliable the straw coverage assignment is.

[0071] According to the straw coverage detection method for cross-scale multi-source data fusion provided by the application, the straw coverage of the unmanned aerial vehicle sample image with the highest overall similarity to the target unmanned aerial vehicle image block is determined, comprising: determining the first structural similarity index between the target unmanned aerial vehicle image block similarity and each ground photograph; determining the second structural similarity index between the target unmanned aerial vehicle image block similarity and the unmanned aerial vehicle sample image corresponding to each ground photograph; weighting and summing based on the first structural similarity index and the second structural similarity index to obtain the overall similarity of each unmanned aerial vehicle sample image corresponding to the target unmanned aerial vehicle image block; The straw coverage of the sample image of the unmanned aerial vehicle with the highest overall similarity is taken as the straw coverage of the target unmanned aerial vehicle image block.

[0072] In the embodiment of the application, the most similar sample refers to selecting the sample with the highest overall similarity after calculating the overall similarity of the unmanned aerial vehicle image block and all ground shooting photos and corresponding unmanned aerial vehicle sample images, and taking the straw coverage of the most similar sample as the straw coverage of the target unmanned aerial vehicle image block.

[0073] Through the embodiment of the application, the ground shooting photos guarantee the accuracy of the true value, the unmanned aerial vehicle sample images ensure the consistency of the same scale, and the weighting mechanism balances the contributions of the two, thereby avoiding the deviation of a single data source.

[0074] In step 107, a straw coverage regression model based on a Transformer model is constructed based on the sample set of the straw coverage of the unmanned aerial vehicle image.

[0075] In the embodiment of the application, the obtained sample of the straw coverage of the unmanned aerial vehicle image is aggregated to form a sample set of the straw coverage of the satellite image scale, and a straw coverage regression model is constructed based on a Transformer model.

[0076] Here, the aggregation of the sample of the straw coverage of the unmanned aerial vehicle image to form a sample set of the straw coverage of the satellite image per pixel refers to aggregating the sample of the straw coverage of the unmanned aerial vehicle scale into a sample of the straw coverage of the corresponding satellite image scale according to the spatial resolution correspondence between the unmanned aerial vehicle image and the satellite image.

[0077] According to the straw coverage detection method for cross-scale multi-source data fusion provided by the application, a straw coverage regression model based on a Transformer model is constructed based on a sample set of the straw coverage of the unmanned aerial vehicle image, which comprises the following steps: The sample set of the straw coverage of the unmanned aerial vehicle image is aggregated based on the linear relationship between the spatial resolution of the satellite remote sensing image and the spatial resolution of the unmanned aerial vehicle image, to obtain a sample set of the straw coverage of the satellite image scale; Based on the sample set of the straw coverage of the satellite image scale, a straw coverage regression model based on a Transformer model is constructed by taking the reflectivity and spatial features of the satellite image as input and the straw coverage as the regression object.

[0078] In the embodiment of the present application, the straw coverage regression model is constructed based on the Transformer model, which means that the satellite image scale straw coverage sample data is taken as input, the satellite image reflectivity and spatial features are taken as input, the straw coverage is taken as the regression object, the Transformer model is selected, the model parameters are set, and then the model optimization is performed, and after the model optimization is stable, the straw coverage regression model for satellite image is obtained.

[0079] Through the embodiment of the present application, the cross-scale multi-source data fusion straw coverage detection method fully utilizes the correlation between different scale observation data (ground photos, unmanned aerial vehicle images, satellite images), and through layer-by-layer decoupling processing, the straw coverage detection method based on satellite images in a large area range can be effectively established.

[0080] In step 108, the satellite remote sensing image is input into the straw coverage regression model to obtain the straw coverage extraction result of the target area output by the straw coverage regression model.

[0081] In the embodiment of the present application, the satellite remote sensing image, the unmanned aerial vehicle image and the ground shooting photo obtained by comprehensively utilizing observation means such as ground (field ground shooting photo), machine (unmanned aerial vehicle shooting image) and star (satellite remote sensing image) are first subjected to semantic segmentation based on a visual large model to obtain the straw coverage in the photo, and then the straw coverage degree of the region is calculated, and then the straw coverage sample data based on the unmanned aerial vehicle image is obtained by contrast learning based on the ground shooting photo and the unmanned aerial vehicle image, and finally the straw coverage regression model based on the satellite remote sensing image is established based on the Transformer model to realize accurate extraction of the straw coverage in a large area range.

[0082] Reference Figure 4 , Figure 4 is the cross-scale multi-source data fusion straw coverage detection method flowchart provided by the present application.

[0083] As Figure 4 shown, the ground shooting photo is first input into the automatic analysis module based on the visual large model to generate the ground scale coverage information; and then the straw coverage sample in the range of the unmanned aerial vehicle scale is generated based on this, this step simultaneously receives the unmanned aerial vehicle image input, and is guided to the sample expansion branch based on the contrast enhancement of the unmanned aerial vehicle image and the ground photo; the other branch performs the straw coverage regression based on the Transformer model on the satellite image. Finally, the two branches are merged, and the straw coverage extraction result in a large area range is output.

[0084] By the embodiment of the present application, first, the ground photo is precisely segmented by using the visual large model to obtain the ground scale coverage true value with high reliability; on this basis, the unmanned aerial vehicle image is scaled and divided into grids, and then the target image block with strong representation is selected by calculating the similarity between the image block and the ground photo to automatically generate the unmanned aerial vehicle coverage sample set; finally, the high-quality samples are used to train the Transformer regression model, and the precise features learned in small scale (ground, unmanned aerial vehicle) are effectively transferred to large scale satellite image, so that the large-area high-precision straw coverage extraction based on satellite image is realized.

[0085] The cross-scale multi-source data fusion straw coverage detection device provided by the present application is described below, and the cross-scale multi-source data fusion straw coverage detection device described below can be correspondingly referred to the cross-scale multi-source data fusion straw coverage detection method described above.

[0086] Reference Figure 5 , Figure 5 is a module schematic diagram of the cross-scale multi-source data fusion straw coverage detection device provided by the present application.

[0087] The acquisition module 501 is configured to acquire satellite remote sensing images, unmanned aerial vehicle images and ground shooting photos of a target area. The segmentation module 502 is configured to input the ground shooting photo into a visual large model for semantic segmentation to obtain the straw coverage of the ground scale range corresponding to the ground shooting photo output by the visual large model. The adaptation module 503 is configured to perform scale range adaptation on the unmanned aerial vehicle image based on the straw coverage of the ground scale range to obtain an unmanned aerial vehicle sample image. The division module 504 is configured to divide the unmanned aerial vehicle sample image according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks. The similarity module 505 is configured to perform image similarity calculation on each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks based on the ground shooting photo corresponding to each unmanned aerial vehicle image block and the unmanned aerial vehicle sample image to obtain the overall similarity of each unmanned aerial vehicle image block. The generation module 506 is configured to sort each unmanned aerial vehicle image block according to the overall similarity, and select a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain a unmanned aerial vehicle image straw coverage sample set. The construction module 507 is configured to construct a straw coverage regression model based on the Transformer model based on the unmanned aerial vehicle image straw coverage sample set. The extraction module 508 is configured to input the satellite remote sensing image into the straw coverage regression model to obtain the straw coverage extraction result of the target area output by the straw coverage regression model.

[0088] Specifically, the straw coverage detection device for cross-scale multi-source data fusion provided by the present application can realize all the method steps implemented by the straw coverage detection method embodiment for cross-scale multi-source data fusion, and achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiment in this embodiment will not be described in detail here.

[0089] Figure 6 is the schematic diagram of the physical structure of the electronic device provided by the present application, as shown in Figure 6 The electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can invoke the logical instructions in the memory 630 to execute the straw coverage detection method for cross-scale multi-source data fusion, which includes: acquiring satellite remote sensing images, unmanned aerial vehicle images, and ground shooting photos of a target area; inputting the ground shooting photos into a visual large model for semantic segmentation to obtain the straw coverage of the ground scale range corresponding to the ground shooting photos output by the visual large model; performing scale range adaptation on the unmanned aerial vehicle images based on the straw coverage of the ground scale range to obtain unmanned aerial vehicle sample images; dividing the unmanned aerial vehicle sample images according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; performing image similarity calculation on the ground shooting photos corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images to obtain the overall similarity of each unmanned aerial vehicle image block; sorting each unmanned aerial vehicle image block according to the overall similarity and selecting a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain a unmanned aerial vehicle image straw coverage sample set; constructing a straw coverage regression model based on a Transformer model based on the unmanned aerial vehicle image straw coverage sample set; inputting the satellite remote sensing images into the straw coverage regression model to obtain the straw coverage extraction result of the target area output by the straw coverage regression model.

[0090] In addition, the logic instructions in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0091] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the cross-scale multi-source data fusion straw coverage detection method provided by the above-mentioned methods. The method comprises: acquiring satellite remote sensing images, unmanned aerial vehicle images and ground shooting photos of a target area; inputting the ground shooting photos into a visual large model for semantic segmentation to obtain the ground scale range straw coverage corresponding to the ground shooting photos output by the visual large model; based on the ground scale range straw coverage, performing scale range adaptation on the unmanned aerial vehicle images to obtain unmanned aerial vehicle sample images; dividing the unmanned aerial vehicle sample images according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; based on the image similarity calculation of the corresponding ground shooting photos and the unmanned aerial vehicle sample images of each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks, obtaining the overall similarity of each unmanned aerial vehicle image block; sorting each unmanned aerial vehicle image block according to the overall similarity, and selecting a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain an unmanned aerial vehicle image straw coverage sample set; based on the unmanned aerial vehicle image straw coverage sample set, constructing a straw coverage regression model based on a Transformer model; inputting the satellite remote sensing images into the straw coverage regression model to obtain the straw coverage extraction result of the target area output by the straw coverage regression model.

[0092] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements a method for detecting straw coverage by cross-scale multi-source data fusion provided by the methods described above. This method includes: acquiring satellite remote sensing images, UAV images, and ground-based photographs of a target area; inputting the ground-based photographs into a large-scale visual model for semantic segmentation to obtain the straw coverage at the ground scale corresponding to the ground-based photographs output by the large-scale visual model; performing scale range adaptation on the UAV images based on the straw coverage at the ground scale to obtain UAV sample images; and dividing the UAV sample images according to a preset rule grid. Multiple UAV image patches are obtained. Image similarity is calculated between the ground-captured photos corresponding to each UAV image patch and the UAV sample images to obtain the overall similarity of each UAV image patch. Each UAV image patch is sorted according to its overall similarity, and a fixed proportion of target UAV image patches are selected to generate new samples, resulting in a UAV image straw coverage sample set. Based on this sample set, a straw coverage regression model based on the Transformer model is constructed. Satellite remote sensing images are input into the straw coverage regression model to obtain the straw coverage extraction results for the target area output by the model.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0095] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting straw coverage degree by cross-scale multi-source data fusion, characterized in that, The method comprises the following steps: acquire satellite remote sensing images, unmanned aerial vehicle images and ground shooting photos of a target area; input the ground shooting photos into a visual large model for semantic segmentation to obtain ground scale range straw coverage degrees corresponding to the ground shooting photos output by the visual large model; based on the ground scale range straw coverage degrees, perform scale range adaptation on the unmanned aerial vehicle images to obtain unmanned aerial vehicle sample images; divide the unmanned aerial vehicle sample images according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; based on the ground shooting photos corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks and the unmanned aerial vehicle sample images, perform image similarity calculation to obtain an overall similarity of each unmanned aerial vehicle image block; sort each unmanned aerial vehicle image block according to the overall similarity and select a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain a unmanned aerial vehicle image straw coverage degree sample set; based on the unmanned aerial vehicle image straw coverage degree sample set, construct a straw coverage degree regression model based on a Transformer model; input the satellite remote sensing images into the straw coverage degree regression model to obtain a straw coverage degree extraction result of the target area output by the straw coverage degree regression model.

2. The method according to claim 1, wherein, The method comprises the following steps: determine the side length of the ground scale range corresponding to the ground shooting photos; take the ratio of the side length of the ground scale range to the ground resolution of the unmanned aerial vehicle images as the scale size of the unmanned aerial vehicle images; cut the unmanned aerial vehicle images according to the scale size to obtain unmanned aerial vehicle sample images, wherein the straw coverage degrees of the unmanned aerial vehicle sample images are associated with the straw coverage degrees of the ground scale range.

3. The method according to claim 1, wherein, The method comprises the following steps: for each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks, perform the following processing: determine a first structural similarity index between the unmanned aerial vehicle image block and the ground shooting photo corresponding to the unmanned aerial vehicle image block; determine a second structural similarity index between the unmanned aerial vehicle image block and the unmanned aerial vehicle sample image corresponding to the unmanned aerial vehicle image block; based on the first structural similarity index and the second structural similarity index, perform weighted summation to obtain the overall similarity of the unmanned aerial vehicle image block.

4. The method according to claim 1, wherein, The method comprises the following steps: determine the straw coverage degree of the unmanned aerial vehicle sample image with the highest overall similarity to the target unmanned aerial vehicle image block; take the product of the straw coverage degree and the overall similarity of the target unmanned aerial vehicle image block as the straw coverage degree of the target unmanned aerial vehicle image block to obtain a unmanned aerial vehicle image straw coverage degree sample set.

5. The method according to claim 4, wherein, The determination of the straw coverage degree of the unmanned aerial vehicle sample image with the highest overall similarity to the target unmanned aerial vehicle image block comprises: determining a first structural similarity index between the target unmanned aerial vehicle image block similarity and each ground photograph; determining a second structural similarity index between the target unmanned aerial vehicle image block similarity and the unmanned aerial vehicle sample image corresponding to each ground photograph; performing weighted summation based on the first structural similarity index and the second structural similarity index to obtain the overall similarity of each unmanned aerial vehicle sample image corresponding to the target unmanned aerial vehicle image block; taking the straw coverage degree of the unmanned aerial vehicle sample image with the highest overall similarity as the straw coverage degree of the target unmanned aerial vehicle image block.

6. The method according to claim 1, wherein, The construction of the straw coverage degree regression model based on the Transformer model based on the unmanned aerial vehicle image straw coverage degree sample set comprises: performing aggregation on the unmanned aerial vehicle image straw coverage degree sample set based on the linear relationship between the spatial resolution of the satellite remote sensing image and the spatial resolution of the unmanned aerial vehicle image to obtain a satellite image scale straw coverage degree sample set; constructing a straw coverage degree regression model based on the Transformer model based on the satellite image scale straw coverage degree sample set, taking satellite image reflectivity and spatial features as input and taking straw coverage degree as regression object.

7. A device for detecting the coverage of straw across scales and multi-source data fusion, characterized in that, comprise: an acquisition module configured to acquire satellite remote sensing images, unmanned aerial vehicle images, and ground photographs of a target region; a segmentation module configured to input the ground photographs into a visual large model for semantic segmentation to obtain ground scale range straw coverage degrees corresponding to the ground photographs output by the visual large model; an adaptation module configured to perform scale range adaptation on the unmanned aerial vehicle images based on the ground scale range straw coverage degrees to obtain unmanned aerial vehicle sample images; a division module configured to divide the unmanned aerial vehicle sample images according to a preset rule grid to obtain a plurality of unmanned aerial vehicle image blocks; a similarity module configured to perform image similarity calculation on the ground photographs and the unmanned aerial vehicle sample images corresponding to each unmanned aerial vehicle image block in the plurality of unmanned aerial vehicle image blocks to obtain an overall similarity of each unmanned aerial vehicle image block; a generation module configured to sort the each unmanned aerial vehicle image block according to the overall similarity and select a fixed proportion of target unmanned aerial vehicle image blocks for new sample generation to obtain an unmanned aerial vehicle image straw coverage degree sample set; a construction module configured to construct a straw coverage degree regression model based on the Transformer model based on the unmanned aerial vehicle image straw coverage degree sample set; an extraction module configured to input the satellite remote sensing images into the straw coverage degree regression model to obtain a straw coverage degree extraction result of the target region output by the straw coverage degree regression model.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the cross-scale multi-source data fusion straw coverage degree detection method according to any one of claims 1 to 6. The processor executes the computer program to implement the cross-scale multi-source data fusion straw coverage degree detection method according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the straw coverage detection method of cross-scale multi-source data fusion as claimed in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the straw coverage detection method of cross-scale multi-source data fusion as claimed in any one of claims 1 to 6.

Images