A Multi-Scale Spatiotemporal Fusion Method and System for Phenological Identification Based on Remote Sensing Images

Abstract

This application discloses a multi-scale spatiotemporal fusion method and system for phenological identification based on remote sensing images. It utilizes a multi-scale spatiotemporal fusion deep learning framework, constructing continuous time series based on high-frequency acquired baseline resolution images as time anchors. This is then spatially enhanced by combining these with sparse high-resolution images. By introducing a missing-aware gating fusion mechanism, multi-resolution features on unaligned time axes are dynamically fused. The resulting fused feature sequence is then input into a long short-term memory network to achieve phenological stage identification of crop germplasm resources. This application can stably identify the jointing-booting stage, heading-flowering stage, milk stage, waxy ripening stage, and maturity stage under conditions of sparse high-resolution images and asynchronous sampling times, balancing identification accuracy and operational efficiency. It is suitable for large-scale germplasm resource phenotypic monitoring, critical period early warning, and precision breeding applications, providing a flexible, scalable, and cost-effective solution for high-throughput phenotypic analysis.

Description

Technical Field

[0001] This application belongs to the fields of agricultural remote sensing, crop phenotyping and image processing technology, and in particular relates to a multi-scale spatiotemporal fusion method and system for phenological identification based on remote sensing images. Background Technology

[0002] Crop phenological stages are key determinants of yield prediction and germplasm resource evaluation. However, in large-scale rice breeding trials involving hundreds of varieties, achieving accurate phenological monitoring faces significant challenges due to the inherent asynchronicity of phenology among varieties and the trade-off between spatial resolution and temporal continuity in UAV remote sensing. Especially for major food crops like rice, key phenological stages such as tillering, jointing, heading, flowering, and maturity not only determine the optimal window for yield formation and field management but also constitute core phenotypic traits for germplasm evaluation and variety improvement. Accurately identifying these phenological stages can provide decision support for agricultural management practices such as fertilization, irrigation, and pest and disease control in the field, thus representing a fundamental problem urgently needing to be solved for precision rice management and regional breeding.

[0003] However, significant phenological asynchrony often exists among hundreds of varieties with different genetic backgrounds, making it difficult for traditional single-phase observation methods to comprehensively characterize phenological diversity at the population scale in large-scale crop breeding trials involving hundreds of varieties. To address this issue, existing technologies using manual observation are labor-intensive, inefficient, prone to accumulating errors due to human subjectivity, and often fail to meet the requirement of rapidly capturing the temporal continuity of phenological transitions. While some crop growth models based on meteorological drivers exist to simulate developmental processes, their inability to characterize the spatial heterogeneity of growth dynamics under real field conditions limits their usability in experiments with extremely high genetic diversity.

[0004] High-throughput phenotypic analysis techniques can detect plant growth status; however, existing phenotypic analysis methods heavily rely on shallow image features, such as vegetation indices and texture / color indicators, supplemented by empirical rules or shallow machine learning models. Therefore, these methods are extremely sensitive to feature selection, threshold setting, sampling frequency, and data gaps, resulting in insufficient robustness in complex backgrounds or across growing seasons. Furthermore, since phenology is essentially a continuous developmental process that evolves over time, the identification of key growth stages largely depends on the dynamic changes before and after key transition periods. Therefore, single-image datasets often fail to reliably distinguish adjacent growth stages, and accurately locate key transition nodes such as heading or flowering when different varieties develop asynchronously. In scenarios with complex backgrounds or multi-peak curves, the boundaries of growth stages identified by existing technologies are prone to shift, thus limiting their migration ability across different varieties and years. How to achieve accurate identification of phenological stages for large-scale germplasm resources while maintaining both spatial resolution and temporal continuity has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application addresses the difficulties faced by existing phenological detection technologies by providing a method and system based on a multi-scale temporal deep learning framework, capable of achieving multi-scale spatiotemporal fusion for phenological identification from remote sensing images. This application uses high-frequency acquired baseline resolution (MR) images as temporal anchors, combining them with high-resolution (HR) images obtained according to sparse acquisition frequencies for spatial enhancement. By introducing a Missing-Aware Gated Fusion (MAGF) mechanism, multi-resolution features on unaligned time axes are dynamically integrated, thereby achieving robust modeling under irregular sampling conditions.

[0006] To address the aforementioned technical problems, this application provides a multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images. The method comprises the following steps: First, for remote sensing images of different resolutions sampled at preset periods, after alignment, stitching, and correction, a base-resolution image acquired at a high frequency is used as a time anchor point. A time series is constructed according to the capture timestamp of the base-resolution image. The high-resolution images acquired at low frequencies are aligned to the corresponding time index positions in the time series, and a missing mask is used to indicate the availability status of the high-resolution images in the time series. Then, the base-resolution image and the high-resolution image are input into two parallel feature extraction branches, respectively. The image features corresponding to each of the images are extracted separately. Then, the extracted image features from images of different resolutions are mapped to a unified feature space. Multi-scale feature fusion is performed using a missing-aware gating fusion mechanism. When high-resolution images are unavailable, baseline features are generated according to the features of the base resolution image. When high-resolution images are available, the high-resolution image features and the base resolution image features are fused according to adaptive gating weights to generate a fused feature sequence. Finally, the fused feature sequence is input into a long short-term memory network. After processing by a classification layer, the predicted probability distribution of each phenological stage is output, and the crop phenological stage identification result is provided accordingly.

[0007] Optionally, in any of the above-described multi-scale spatiotemporal fusion methods for phenological identification based on remote sensing images, the step of sampling remote sensing images of different resolutions according to a preset period is as follows: specifically, remote sensing images of different resolutions are acquired by controlling the flight altitude of the UAV, wherein a basic resolution image is acquired based on the image taken by the UAV at a first flight altitude, and a high-resolution image is acquired based on the image taken by the UAV at a second flight altitude; the second flight altitude is lower than the first flight altitude, and the sampling frequency of the basic resolution image is higher than the sampling frequency of the high-resolution image.

[0008] Optionally, in any of the above-described multi-scale spatiotemporal fusion methods for phenological identification based on remote sensing images, when acquiring the base resolution image, the forward and lateral overlap during the shooting process is maintained at more than 60%. After orthorectification and stitching of the acquired images, the processed orthophoto mosaic is used as the base resolution image. During the high-resolution image sampling process, single-scene photography of the field is carried out, and the image of the field corresponding to the center position of the image is extracted from the high-resolution images of the same field as the high-resolution image.

[0009] Optionally, in any of the above-described multi-scale spatiotemporal fusion methods for phenological identification based on remote sensing images, the specific steps of using a missing mask to indicate the availability of high-resolution images in the time series include: in the time series constructed based on the capture timestamps of the base resolution images, matching the corresponding base resolution image for each time index position t. High-resolution images and the corresponding missing mask, wherein, when a high-resolution image with a usable state exists at any t-th time index position, based on the high-resolution image The corresponding shooting timestamp will be the missing mask corresponding to the time index position. Set as When no high-resolution image is available at the t-th time index position, set And the missing mask corresponding to the time index position. Set as .

[0010] Optionally, the multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images, as described above, includes the step of inputting the base resolution image and the high resolution image into two parallel feature extraction branches to extract their image features. Specifically, this step includes: using a backbone feature extraction network capable of outputting fixed-dimensional feature vectors to extract features from the base resolution image and the high resolution image respectively, and extracting the image features corresponding to each image at different resolutions after global average pooling processing; wherein the backbone feature extraction network is a convolutional neural network or a lightweight visual feature extraction network.

[0011] Optionally, in any of the above-described multi-scale spatiotemporal fusion methods for phenological identification based on remote sensing images, the two feature extraction branches input from the base resolution image and the high resolution image respectively adopt the same backbone network architecture but do not share weights. The two backbone feature extraction networks are trained independently to learn the image feature representations at their corresponding resolutions.

[0012] Optionally, in any of the above-described multi-scale spatiotemporal fusion methods for phenological identification based on remote sensing images, the specific steps for generating a fusion feature sequence using a missing-aware gating fusion mechanism based on base-resolution image features and high-resolution image features mapped to a unified feature space include: [The text abruptly ends here, so the translation stops.] and high-resolution image features obtained from high-resolution images Perform vector concatenation to form a joint context vector. ;Joint context vector Input a learnable gating network to generate adaptive gating weights. Based on the missing mask Gated residual fusion is performed on the base resolution image features and high resolution image features at each time index position t. The fusion result of each time index position t Arranged sequentially, a fused feature sequence is obtained; among which, This represents the activation function used to normalize the adaptive gate weights to... Within the interval (0,1); This represents the weight matrix in a gated network that projects the high-dimensional context as a scalar importance index. This represents the bias term used to adjust the activation threshold; This indicates element-wise multiplication.

[0013] To address the aforementioned technical problems, this application also provides a multi-scale spatiotemporal fusion system for phenological identification based on remote sensing images, comprising: an image acquisition unit for acquiring remote sensing images of different resolutions at a preset period; a data storage unit for backing up and storing the remote sensing images of different resolutions; and an image processing unit for acquiring the remote sensing images in the data storage unit and outputting crop phenological period identification results according to any of the methods described above.

[0014] Optionally, in the multi-scale spatiotemporal fusion system for phenological identification based on remote sensing images as described above, the image acquisition unit is mounted on a drone and is used to capture remote sensing images at different flight altitudes according to different sampling frequencies.

[0015] Simultaneously, this application also provides a computer storage medium storing a computer program that can run on a processor. When the program is executed, it implements the multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in any of the preceding claims.

[0016] Beneficial effects

[0017] The multi-scale spatiotemporal fusion method and system for phenological identification based on remote sensing images disclosed in this application utilizes a multi-scale temporal deep learning framework. It uses high-frequency base-resolution images as time anchors to construct continuous time series, which are then combined with high-resolution images acquired at sparse sampling frequencies for spatial enhancement. This approach simultaneously addresses the temporal continuity and spatial detail required for phenological identification, overcoming the shortcomings of existing technologies where single-phase images struggle to accurately distinguish adjacent phenological stages and single-resolution images cannot simultaneously achieve temporal and spatial resolution. Furthermore, the multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images provided in this application introduces a Missing-Aware Gated Fusion (MAGF) mechanism to adaptively fuse multi-resolution features on non-aligned time axes. This allows for effective utilization of continuous temporal information provided by the base-resolution image and fine phenotypic information provided by the high-resolution image, even when high-resolution images are missing or sampling is asynchronous. This improves the stability and robustness of phenological identification under irregular sampling conditions. By inputting the fused temporal features into a Long Short-Term Memory (LSTM) network... By using Short-Term Memory (LSTM) modeling, this application can fully utilize the dynamic characteristics of continuous changes in crop phenology over time, enhancing the ability to distinguish between adjacent growth stages such as jointing-booting stage, heading-flowering stage, milk stage, waxy ripening stage, and maturity stage, thereby improving the phenological stage identification effect of germplasm resources. In addition, this application adopts a combination of high-frequency acquisition of basic resolution images and low-frequency acquisition of high-resolution images, which can significantly reduce the operation time and cost required for high-resolution UAV sampling while ensuring the phenological identification effect, improving the efficiency of large-scale rice germplasm resource phenotypic monitoring, and providing a flexible, scalable, and cost-effective technical solution for high-throughput phenotypic analysis in precision breeding.

[0018] The phenological identification method provided in this application is particularly suitable for large-scale rice breeding trials involving hundreds of varieties. This is because the flowering times of hundreds of rice varieties in breeding fields typically vary. In such scenarios, to accurately identify the phenological stages of different varieties and provide decision support for agricultural management such as fertilization, irrigation, and pest and disease control, it is necessary to precisely identify the phenological stage of the varieties planted in each plot. Simply relying on daily manual field recording is very difficult. Completing a single patrol takes a long time, and it is difficult to guarantee that the patrol density can meet the needs of phenological identification. In some cases, some plots may require several days between manual patrols. Furthermore, existing phenological identification methods based on remote sensing images are insufficient in terms of accuracy and precision. In existing technologies, to achieve identification accuracy similar to that of this application, frequent low-altitude drone flights for high-resolution sampling are required, relying entirely on high-resolution images for model training. Under such sampling requirements, drones need to fly at low altitudes for approximately 4 hours to cover 1000 rice plots. This application utilizes base-resolution images to construct a complete time series for the model, achieving similar accuracy to existing technologies that rely entirely on high-resolution images, with only a smaller number of high-resolution images required. In this application, a single sampling flight of base-resolution images takes only about 20 minutes to cover 0.5 hectares of paddy fields corresponding to 1000 plots. In other words, the hybrid data acquisition strategy employed in this application—collecting base-resolution images daily or every other day to maintain temporal continuity, supplemented by high-resolution image acquisition flights every 5 to 7 days to optimize spatial details and perform periodic calibration—can significantly reduce the total weekly flight time from 28 hours of sampling time required for full coverage with high-resolution images daily to approximately 6 hours while maintaining high phenological recognition accuracy. Attached Figure Description

[0019] The embodiments of this application will be described in further detail below with reference to the accompanying drawings:

[0020] Figure 1 This is a schematic diagram of the multi-scale sample construction and preprocessing process used in this application.

[0021] Figure 2 This is a flowchart illustrating the overall process of the multi-scale spatiotemporal deep learning framework used in this application. Figure 2 Part (a) in the diagram is a schematic diagram of the construction and preprocessing process of the multi-scale time-series input dataset of this application. Figure 2 Part (b) is a schematic diagram of the dual-resolution feature extraction and missing-aware gating fusion method of this application. Figure 2 Part (c) is a schematic diagram of the temporal modeling and phenological classification process of the long short-term memory network with fused feature sequences provided in this application.

[0022] Figure 3 The graph shows a comparison of the recognition performance and resource overhead of the multi-scale spatiotemporal fusion method provided in this application when using different backbone network models under different resolution conditions.

[0023] Figure 4 Feature visualizations used to explain the high classification accuracy of the lightweight convolutional neural network and long short-term memory network employed in this application: Figure 4 Part (a) in the figure is a model response heatmap of typical samples at different phenological stages in this application. Figure 4 Part (b) in the figure is a schematic diagram of the feature distribution under different resolution inputs and fusion strategies in this application.

[0024] Figure 5 This is a graph showing the variation of phenological period recognition performance indicators under different high-resolution image sampling time intervals in this application.

[0025] Figure 6 The flowchart illustrates the steps of the multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images provided in this application. Detailed Implementation

[0026] To make the objectives and technical solutions of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0027] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0028] The term "connection" as used in this application can mean a direct connection between components or an indirect connection between components through other components.

[0029] Existing deep time-series models struggle to simultaneously achieve high spatial resolution and strong temporal continuity. While high-resolution images acquired through low-altitude aerial photography provide the fine-grained spatial information necessary for identifying subtle differences between varieties, they suffer from low operational efficiency, high shooting costs, and sparse and irregular sampling of observation data in the temporal dimension. This application addresses this issue by regularly acquiring baseline-resolution images through high-altitude aerial photography, combined with high-resolution images acquired at lower flight altitudes during certain periods. By integrating the image features extracted from both methods, it achieves high-frequency coverage of baseline-resolution images at a lower cost, thereby constructing densely sampled baseline-resolution time series. Simultaneously, by leveraging the complementarity between baseline-resolution and high-resolution images acquired at low frequencies, it captures fine-grained morphological details and micro-phenotypic differences during key developmental transitions, enabling the capture of continuous phenological trajectories in field scenarios.

[0030] To address the challenge of collaborative fusion between high-frequency baseline resolution imagery and low-frequency high-resolution imagery, particularly the temporal discontinuity caused by sparse sampling of high-resolution images, this application provides a robust and scalable multi-scale spatiotemporal fusion identification framework. This framework preserves fine-grained spatial discrimination capabilities while prioritizing temporal continuity through the time series of baseline resolution images. This adapts to the significant sample heterogeneity and inherent irregular sampling issues in large-scale rice germplasm trials, and captures subtle phenological differences in germplasm resources through temporal modeling. The multi-scale spatiotemporal fusion identification framework provided in this application optimizes the trade-off between monitoring costs and identification accuracy by constructing a robust multi-scale fusion model, achieving a balance between high throughput, high accuracy, and high timeliness.

[0031] In this application, the multi-scale spatiotemporal fusion model adopts an end-to-end deep learning architecture, which performs spatiotemporal collaborative fusion of high-frequency acquired basic resolution images and low-frequency acquired high-resolution images, and introduces a missing perception gating fusion mechanism to overcome the spatiotemporal misalignment problem caused by irregular sampling and asynchronous phenology among germplasms, adaptively aligning and fusing data sources of different resolutions. At the same time, this application also forms a flexible and cost-effective hybrid sampling strategy by quantifying the relationship between UAV sampling frequency and phenological identification accuracy. This strategy can provide quantitative basis and decision support for budget allocation and accuracy requirements in large-scale breeding practices, thereby achieving phenological monitoring with target accuracy at a lower flight cost.

[0032] This embodiment specifically uses field phenological identification of approximately 500 rice germplasm resources as an example to illustrate the multi-scale spatiotemporal fusion method and system for phenological identification based on remote sensing images provided in this application. This application uses high-frequency sampled base-resolution images as time anchors, combined with high-resolution images obtained from low-frequency sampling to provide spatial enhancement. A missing-aware gating fusion mechanism adaptively fuses features at different resolutions under different time index positions, and combines a long short-term memory network to perform temporal modeling of the continuous crop growth process, thereby achieving crop phenological stage identification.

[0033] See Figure 2 The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images provided in this embodiment relies on an image acquisition unit mounted on a UAV to acquire remote sensing images of different resolutions at preset intervals. Each acquired remote sensing image is backed up and stored in a data storage unit. Then, an image processing unit is triggered to train the model according to the following steps to obtain a model capable of accurately identifying crop phenological stages:

[0034] The first step is data preparation for the crop phenological model. Remote sensing images of different resolutions, sampled at preset intervals, are processed. Using the high-frequency base resolution images as time anchors, time series are constructed based on the capture timestamps of these base resolution images. The time series corresponding to different plots are randomly divided into training, test, and validation sets in an 8:1:1 ratio, using plots as units. During the division, time series corresponding to plots planted with different or the same rice varieties are randomly assigned to the training set; time series corresponding to other plots with different or the same rice varieties are assigned to the test set; and time series corresponding to other collected plots are assigned to the validation set. It is ensured that the plot numbers corresponding to the time series in the three datasets do not overlap. Different plot numbers can correspond to different rice varieties or the same variety of rice, and the specific planting arrangement can be flexibly adjusted according to experimental needs.

[0035] The second step is to establish a crop phenological stage model. The model first aligns the low-frequency high-resolution images to the corresponding time index positions in the time series, and uses a missing mask to indicate the availability of the high-resolution images in the time series. Then, the base resolution image and the high-resolution image are input into two parallel feature extraction branches, respectively, to extract their corresponding image features. Subsequently, the extracted image features from images of different resolutions are mapped to a unified feature space, and a missing-aware gating fusion mechanism is used for multi-scale feature fusion. Adaptively, when the high-resolution image is unavailable, baseline features are generated according to the base resolution image features; when the high-resolution image is available, the high-resolution image features and the base resolution image features are fused according to adaptive gating weights to generate a fused feature sequence. Finally, the fused feature sequence is input into a Long Short-Term Memory (LSTM) network, processed by a classification layer, and outputs the predicted probability distribution of each phenological stage. The identification result of the crop phenological stage is provided based on the phenological stage with the highest probability.

[0036] The third step is the crop phenological stage training phase. The crop phenological stage identification model is trained using the training set. After each epoch of training, the model parameters are adjusted and performance is evaluated using the validation set. Based on the validation results of each training epoch, the overall accuracy, precision, recall, F1 score, or Kappa coefficient of each model are considered to select the optimal model. The optimal model weights are then saved and used in subsequent application stages to directly perform phenological identification.

[0037] The fourth step is the crop phenological stage testing phase. The test set is input into the trained crop phenological stage recognition model to obtain crop phenological stage detection and classification results, and the model's recognition performance is evaluated based on these results.

[0038] The model obtained from the above steps can be used to construct a continuous time series based on the base resolution image to characterize the overall changes in the canopy by using the optimal model weights obtained through training. Furthermore, by combining the base resolution image with high-resolution images acquired at a lower sampling frequency, fine phenotypic information such as plant spikes, leaf postures, local textures, and canopy details can be supplemented in the continuous time series, thereby effectively improving the accuracy of phenological identification.

[0039] When performing the above training steps, the first step of preparing the crop phenological model data, in acquiring basic resolution images and high resolution images, and constructing multi-scale time-series inputs, can be found in [reference needed]. Figure 1 The method shown is to complete the acquisition, preprocessing, plot-level sample construction, and time series alignment of multi-resolution image data by following steps S1-1 to S1-4:

[0040] S1-1. Collect basic resolution images and high-resolution images according to the preset high-frequency sampling period requirements and low-frequency sampling period requirements, respectively. In this embodiment, the basic resolution images are preferably acquired by taking pictures of the planting area on the ground while maintaining the drone at a relatively high flight altitude to achieve high-frequency coverage of the entire test area; the high-resolution images are preferably acquired by taking pictures of the planting area while maintaining the drone at a relatively low flight altitude to retain more fine phenotypic information such as plant spikes, leaf posture, and canopy texture through a closer shooting distance. In practice, this embodiment preferably maintains the drone shooting altitude for basic resolution images between 15m and 25m above the ground, with the optimal shooting altitude for basic resolution images typically being 20m above the planting ground; correspondingly, the shooting altitude for high-resolution images can be set to 4m to 6m above the ground. The sampling period for basic resolution images is preferably 1 to 3 days, and the sampling period for high-resolution images is preferably 5 to 14 days, so that the sampling frequency of basic resolution images is higher than that of high-resolution images, and the ratio of the two sampling frequencies is controlled between 1.5 times and 14 times to obtain the best recognition effect with fewer high-resolution samplings. The drones used for remote sensing data acquisition are preferably equipped with color cameras and real-time dynamic differential positioning (RTK) modules. They should be able to complete data acquisition between 10:00 and 14:00 under clear or partly cloudy weather conditions to minimize the impact of light fluctuations. During the sampling of basic resolution images, the drones should plan their flight paths and shutter release times for each frame according to the requirements of 75%-85% forward overlap and 65%-75% lateral overlap, or at least 60% for both forward and lateral overlap. This will facilitate the subsequent alignment and stitching of the images to obtain the best recognition results.

[0041] S1-2. Align, stitch, correct, and orthorectify the base resolution image to generate an orthophoto mosaic. The specific processing steps for this step can be found in [reference needed]. Figure 1As shown on the left, after acquiring the basic resolution image, initial alignment is first performed based on the exchangeable image file information (i.e., EXIF ​​information) and the Global Navigation Satellite System (GNSS) coordinates contained therein during the UAV shooting process. Then, based on the measured coordinates of each control point fixed on the field ground of the experimental area before shooting and their correspondence with pixels in the acquired image, camera parameters are optimized and corrected. The acquired image is then finely aligned. Next, a dense point cloud is generated based on the aligned and corrected image. Interpolation processing is performed on the dense point cloud to generate a corresponding digital elevation model. Finally, based on the digital elevation model, orthorectification and stitching are used to generate an orthophoto mosaic covering the target crop area, thus obtaining a basic resolution orthophoto mosaic covering the entire experimental area with consistent spatial geometry. The above processing can be performed using photogrammetry software. The orthophoto mosaic serves as the basic resolution image, while the high-resolution image is the corresponding plot image extracted from low-altitude sampled images based on the same plot.

[0042] S1-3. Back-project the boundaries of the test plots onto the original image or the corresponding orthophoto mosaic to construct plot-level samples. For detailed processing procedures, please refer to [link to relevant documentation]. Figure 1 As shown on the left, after obtaining the basic resolution orthophoto mosaic through stitching and correction processing, the plot boundaries can be further back-projected onto the original image, or the orthophoto mosaic can be cropped according to the plot boundaries, thereby extracting basic resolution samples corresponding one-to-one with each experimental plot. Thus, taking plots as units, separate time series are constructed for plots planted with different rice varieties in the field. The images corresponding to different plots are sorted according to the shooting and sampling time sequence to facilitate the model's extraction of their temporal features. Simultaneously, each time series can also reduce the geometric deformation caused by large-scale orthophoto stitching on local plot samples through the aforementioned image cropping processing, thereby improving the spatial consistency of the samples.

[0043] S1-4. Plot-level sample extraction and phenological labeling are required for high-resolution images. For specific processing methods in this step, please refer to [link / reference needed]. Figure 1 As shown on the right, high-resolution images are mainly used to preserve fine-grained spatial information, and are manually labeled using time-series labeling tools. In this step, the plant phenological development stage coding system scale is preferred, combined with continuous temporal images, to label the samples and divide them into five phenological stages: jointing-booting stage, heading-flowering stage, milk stage, waxy stage, and maturity stage. The stage division can also be flexibly adjusted according to research needs, dividing the total growth period of the plant into 3 or 7 periods to train models with different phenological recognition accuracies. Figure 1The right side shows typical sample illustrations of the five phenological stages mentioned above. Here, considering that the differences between the jointing stage and the booting stage are small from a remote sensing perspective, this embodiment preferably combines them into "jointing-booting stage"; the heading stage and the flowering stage highly overlap in time and have similar canopy phenotypes, so this embodiment preferably combines them into "heading-flowering stage".

[0044] This embodiment addresses the establishment of the crop phenological model in the second step of the training process. For details of this embodiment, please refer to [link to embodiment]. Figure 2 As shown, the process proceeds in four main steps: 1. Constructing a multi-resolution time series input, aligning the high-resolution images acquired at low frequencies to the corresponding time index positions in the time series, and using a missing mask to indicate the availability of the high-resolution images in the time series; 2. The feature extraction module inputs the base resolution image and the high-resolution image into two parallel feature extraction branches, extracting the image features corresponding to each resolution image, and then mapping the extracted image features from different resolution images to a unified feature space; 3. Performing multi-modal feature fusion, using a missing-aware gating fusion mechanism for multi-scale feature fusion, adaptively generating baseline features based on the base resolution image features when the high-resolution image is unavailable, and fusing the high-resolution image features with the base resolution image features according to adaptive gating weights when the high-resolution image is available, generating a fused feature sequence; 4. Performing time series modeling and classification based on a Long Short-Term Memory (LSTM) network, inputting the fused feature sequence into the LSTM network, and outputting the predicted probability distribution of each phenological period after processing by the classification layer, and providing the identification result of the crop phenological period based on the phenological period with the highest probability. In practice, this embodiment can be implemented in the following manner, from S2-1 to S2-4: S2-1. Construction of Multi-Resolution Temporal Input. This application aligns high-resolution images acquired at low frequencies from the training, test, or validation sets used in the current training to the corresponding time index positions in the time series, and uses a missing mask to indicate the availability status of the high-resolution images in the time series. Specifically, in this embodiment, the time series length T can be preferably set to T=5, that is, each sample consists of 5 basic resolution time index positions arranged in the sampling time order under the same plot number. The multi-scale temporal input sample composed of 5 time index positions is treated as a whole and input into the recognition model of this application. Taking a time series with 7 time index positions as an example, the images at each time index position can be labeled as image 1, image 2, image 3, image 4, image 5, image 6, and image 7 according to the shooting order of the sampling process. In this application scenario, when the model of this application receives an input with a time series length T=5, it can construct the first input in the manner of [empty, empty, image 1, image 2, image 3], and then obtain the phenological period corresponding to image 1 through model recognition; then, it can construct the second input in the manner of [empty, image 1, image 2, image 3, image 4], and then obtain the phenological period corresponding to image 2 through model recognition; then, it can construct the third input in the manner of [image 1, image 2, image 3, image 4, image 5], and then obtain the phenological period corresponding to image 3 through model recognition, and so on. For each time index position t in the time series, a basic resolution image is set accordingly. Then, the high-resolution image corresponding to the sampling time closest to the current time index position is matched. The high-resolution image whose shooting time is closest to the sampling time corresponding to the current time index position is aligned to the numbered item of that time index position, denoted as... If a valid, usable high-resolution image exists at the current time index position, then after aligning the time index positions, the missing mask at that position is set to... =1; If there is no valid, usable high-resolution image at the current time index position, then fill it with a zero tensor of the same size as the high-resolution image, and set the missing mask corresponding to that time index position to 1. =0. This forms a multi-scale temporal input sample { , , }(t=1…T). The multi-scale time-series input sample is as follows: Figure 2As shown in part (a), the base resolution image sequence exists continuously across the entire time axis, while the high-resolution image sequence exists only at certain time index positions. The time index positions of missing high-resolution images are marked using a missing mask. The time series constructed in this way maintains a consistent sequence structure even if high-resolution images are missing at certain time index positions, thus providing continuous input for subsequent time series modeling. Through this construction method, the model can simultaneously utilize the continuous temporal information of the base resolution images and the local spatial detail information of the high-resolution images for feature extraction within a fixed time series length T. Furthermore, because the missing state of the high-resolution images is preserved through high-resolution validity marking in this step, the subsequent fusion module can adaptively fuse based on the availability of high-resolution images.

[0045] The S2-2 feature extraction module inputs the base-resolution image and the high-resolution image from the time series into two parallel feature extraction branches, extracting their respective image features. Then, the extracted image features from the different resolution images are mapped to a unified feature space. Specifically, in this embodiment, as... Figure 2 As shown in section (b), the base resolution image is... Input the first backbone feature extraction network corresponding to the base resolution branch, and process the high-resolution image. The second backbone feature extraction network corresponding to the high-resolution branch is input. These two backbone feature extraction networks can be selected as convolutional neural networks or lightweight visual feature extraction networks, respectively. Preferably, the two backbone feature extraction networks are as follows: Figure 3 As shown, based on a comparison of model performance and resource overhead, a mobile vision-efficient convolutional neural network (MobileNetV3) can be used to independently perform convolutional encoding on images of different resolutions to obtain basic resolution feature maps and high-resolution feature maps respectively. In other implementations, this application can also adjust the two backbone feature extraction networks mentioned above to two structurally identical but independently operating 16-layer visual geometry group convolutional neural networks (VGG16), or two structurally identical but independently operating convolutional neural networks (GoogLeNet), or two structurally identical but independently operating 50-layer residual networks (ResNet50), or other structurally identical but independently operating convolutional neural networks capable of outputting fixed-dimensional feature vectors.

[0046] The selection of the backbone feature extraction network in this application can be referenced. Figure 3 As shown, the optimal network structure is determined based on the performance and resource overhead of different backbone networks under different resolution conditions. Specifically, Figure 3This paper presents a comparison of the overall accuracy, precision, recall, F1 score, Kappa coefficient, parameter count, floating-point computation, and memory usage of different backbone networks for phenological identification under both basic and high-resolution image conditions. Figure 3 It is known that while the 16-layer Visual Geometry Group Convolutional Neural Network (VGG16) has high accuracy, it has a large number of parameters and floating-point operations; the 50-layer Residual Network (ResNet50) has relatively stable overall performance, but its resource consumption is still high; the Convolutional Neural Network (GoogLeNet) has a small number of parameters, but its accuracy is relatively weak; the Mobile Vision Efficient Convolutional Neural Network (MobileNetV3) has high overall accuracy and F1 score under the condition of basic resolution image, while its parameter count, computational cost, and memory usage are significantly lower, thus balancing recognition accuracy and computational efficiency. Therefore, in this embodiment, the Mobile Vision Efficient Convolutional Neural Network (MobileNetV3) is preferably used as the backbone structure of the two-branch feature extraction network. Of course, in other embodiments, the network structures used in the above comparison or other convolutional neural networks that can output fixed-dimensional feature vectors can also be used.

[0047] After the convolutional layers of the backbone feature extraction network described above, global average pooling (GAP) can be applied in the following manner to generate compact feature vectors for the extracted high-resolution features and basic resolution features, respectively:

[0048]

[0049]

[0050] Then, the global feature vectors of the two branches are transformed as follows: and Mapping to a unified feature space to eliminate dimensional differences between features of different resolutions, the linear projection expressions for this mapping process are as follows:

[0051]

[0052]

[0053] in, and These represent the linear projection matrices for the base resolution branch and the high resolution branch, respectively; and These represent the base resolution image features and high resolution image features, respectively, mapped to a unified feature space with d dimensions. Preferably, the... and The indexes are shared across different time points to ensure consistency in feature extraction throughout the growing season.

[0054] In this embodiment, the same network architecture is used for feature extraction and transformation for the two feature extraction branches corresponding to different resolutions, while ensuring that they do not share weights through independent execution. This provides the subsequent Long Short-Term Memory (LSTM) network with input sequences carrying feature information of different dimensions. Using the same network architecture ensures the consistency of features at different resolutions in terms of dimensions; keeping the two branches from sharing weights allows them to be trained independently to learn phenotypic discrimination patterns suitable for their respective resolutions. In the two feature extraction branches, the basic resolution branch is mainly used to extract the overall morphology, color changes, and population structure evolution features of the canopy; the high-resolution branch is mainly used to extract spikelet details, leaf texture, and local spatial structure features.

[0055] S2-3 Multimodal Feature Fusion. This step utilizes a missing-aware gating fusion mechanism for multi-scale feature fusion. It adaptively generates baseline features based on the base resolution image features when high-resolution images are unavailable, and fuses the high-resolution image features with the base resolution image features using adaptive gating weights when high-resolution images are available, generating a fused feature sequence. See details in [link to relevant documentation]. Figure 2 As shown in section (b), this embodiment specifically obtains the basic resolution features mapped to a unified feature space. and high-resolution features Subsequently, the collaborative fusion problem between features of different resolutions, especially the robust fusion problem when high-resolution images are missing, is addressed using the following missing-aware gated fusion mechanism (MAGF). The missing-aware gated fusion mechanism (MAGF) used in this step can generate a specific fusion feature sequence as follows:

[0056] For each time index position t, the base resolution image features that have been mapped to the unified feature space are... and high-resolution image features The vectors are concatenated according to the following formula to form a joint context vector:

[0057]

[0058] Wherein, Concat(·) represents the vector concatenation operation. This represents the joint context feature at the t-th time index position. This joint context vector simultaneously contains the continuous temporal semantic information provided by the base resolution image and the fine spatial semantic information provided by the high-resolution image.

[0059] Joint context vector Input the gating network and generate adaptive gating weights. For details on this step, please refer to [link / reference]. Figure 2 As shown in part (b), the joint context vector The input is fed into a gating network, which outputs gating weights based on the multi-scale feature state at the current time index position t. The gating weight Specifically, it can be obtained according to the following rules:

[0060]

[0061] in, For the weight matrix of the gated network, σ represents the bias term, and σ represents the activation function, which can be chosen as the logistic activation function (Sigmoid). The value ranges from 0 to 1. This gating weight It can be used to characterize the importance of high-resolution features for current phenological discrimination tasks.

[0062] At this point, we can further combine this with the missing mask. Complete the residual adaptive fusion. Specifically, in this step, it is necessary to obtain the gating weights. Then, compare it with the missing mask. Together they act on high-resolution features Complete residual fusion to obtain fusion features represented as follows. :

[0063]

[0064] The missing mask of the fused feature sequence at any time index t When =1, it means that there is a valid, usable high-resolution image at the current time index position. The high-resolution features at this time can be superimposed on the base resolution features under gated weight adjustment, thereby enhancing the ability to describe local details and key morphological features; while correspondingly, in the absence of a mask... When the value is 0, meaning there is no valid, usable high-resolution image at the current time index position, the feature enhancement term provided by the high-resolution image is directly adjusted to zero. This causes the fusion result to automatically degenerate into basic resolution features due to the constraint of the missing mask. Therefore, this application does not need to discard the missing temporal index positions of high-resolution images, and can still rely on the basic resolution temporal anchor points to maintain temporal continuity.

[0065] By arranging the fusion results of each time index position obtained from the above gated residual fusion processing in order of their time index positions, a fused feature sequence can be generated. This is for use in the temporal modeling of the long short-term memory network in S2-4.

[0066] In step S2-4, the temporal modeling and classification based on a long short-term memory network, the fused feature sequence needs to be input into the long short-term memory network. After processing by the classification layer, the predicted probability distribution of each phenological stage is output, and the identification result of the crop phenological stage is provided based on the phenological stage with the highest probability. The specific method is as follows:

[0067] S2-4-1. Input the fused feature sequence into the Long Short-Term Memory network for temporal modeling. In this embodiment, the fused feature sequence... The input Long Short-Term Memory (LSTM) network, whose backbone network can be specifically referred to as follows: Figure 2 The structure is constructed as shown in section (c). Since crop phenological changes exhibit significant temporal continuity and phase transitions, this step utilizes the input gate, forget gate, and output gate in the backbone network to selectively retain and update historical information. This allows for the accumulation of information representing the continuous developmental trajectory of the crop from jointing to heading to maturity, while suppressing short-term environmental fluctuations. Specifically, let the hidden state of the Long Short-Term Memory network at position t in the sequence be... The cell state is The cyclic update at time step t can be formally represented as:

[0068]

[0069] in, This represents the hidden state that encodes the current phenological state, while This serves as the cell state for storing long-term contextual information.

[0070] S2-4-2. Input the final hidden state into the classification layer and output the phenological recognition probability. This step is performed at the end of the sequence, taking the final hidden state. This comprehensive temporal representation of the current sample is input into a fully connected classification layer, and the predicted probability distribution for each phenological stage is output through a normalized exponential function (Softmax):

[0071]

[0072] In the above formula, and These represent the weights and biases of the classification layer, respectively. The predicted probabilities correspond to five stages: jointing-booting stage, heading-flowering stage, milk stage, waxy stage, and maturity stage. The model selects the stage with the highest probability from these five stages as the final phenological identification result for the current plot sample.

[0073] The third step is the crop phenological stage training phase. After constructing the complete crop phenological stage model as described in the second step, this step sequentially trains the crop phenological stage identification model using the training set, and then adjusts the model parameters and evaluates the performance using the validation set. Based on the validation results, the optimal model weights are saved to obtain the final model that can be used for field phenological identification. In this embodiment, the model can be implemented based on the deep learning framework (PyTorch 2.0) and uses a unified computing device architecture (CUDA 12.1) as the parallel computing backend. The adaptive moment estimation optimizer (AdamW) with weight decay used for training can be set with an initial learning rate of 1×10^(-4) and a weight decay of 1×10^(-4) during training. A cosine annealing learning rate scheduler is used to dynamically adjust the learning rate throughout the training process. In each training epoch, this embodiment sequentially inputs the entire training set into the model in batches of 64. After the model completes one epoch of training on the entire training set, the model parameters are adjusted and performance is evaluated using the validation set to obtain the overall precision, accuracy, recall, F1 score, or Kappa coefficient of the model trained in this epoch. After 100 epochs of training with the entire training set input into the model, the optimal model weights are selected and saved based on the overall precision, accuracy, recall, F1 score, or Kappa coefficient on the validation set, and this is used as the final model output.

[0074] To improve training efficiency and reduce memory usage, the above training process preferably employs an automatic mixed-precision training method. During training, if a batch contains fewer than 64 training samples, it still participates in training with the actual number of samples. For each multi-scale temporal input sample, this embodiment sets its corresponding time series length T to 5. Therefore, in each batch, for the 224*224 pixel color (RGB) sample images it processes, the model's base resolution input can be expressed as: ∈R^(64×5×3×224×224), the high-resolution input can be represented as ∈R^(64×5×3×224×224). The model inputs these features into a mobile vision efficient convolutional neural network (MobileNetV3) to extract the base resolution image features and high resolution image features for each time index position. Then, a linear projection layer unifies the two features into a 512-dimensional feature space, and a missing-aware gated fusion module is used to obtain the fused features for each time index position. In the process, for each sample, five time index positions correspond to five fused features, thus forming a fused feature sequence of length 5. For a batch, the feature tensor dimension of the input Long Short-Term Memory (LSTM) network is 64×5×512. The LSM network performs temporal modeling on this fused feature sequence and outputs the hidden state at each time index position. Subsequently, a classification layer is used to obtain the prediction result of the fertility period corresponding to each time index position. In this model, the output dimension of the LSM network is 64×5×C, where C is the number of fertility period categories, which is determined during the data labeling stage according to the required recognition accuracy. For this embodiment, we set C=5 to represent the five phenological stages: jointing-booting stage, heading-flowering stage, milk stage, waxy ripening stage, and maturity stage. For other application scenarios, C can be set to 7 to represent the seven phenological stages: jointing stage, booting stage, heading stage, flowering stage, milk stage, waxy ripening stage, and maturity stage, depending on actual needs. Alternatively, C can be set to 3 to reduce the division of phenological stages.

[0075] During training, the model output is compared with the actual reproductive period labels, the prediction error is calculated using the cross-entropy loss function, the gradient of the model parameters is calculated through backpropagation, and the learnable parameters in the dual-branch mobile vision efficient convolutional neural network (MobileNetV3), linear projection layer, gated fusion layer, long short-term memory network and classification layer are updated using the weighted adaptive moment estimation optimizer (AdamW).

[0076] After each epoch of training, the program calculates accuracy, F1 score, and other metrics on the validation set as follows. Therefore, if the macro-average F1 score on the validation set for a given epoch is higher than the historical best, the current model parameters are saved as the optimal weights (best.pt). This embodiment sets the model to train for a maximum of 100 epochs; if the validation set performance does not improve after several consecutive epochs, training can be stopped early. Finally, the best-performing model on the validation set (best.pt) is selected and evaluated on an independent test set to obtain the final recognition results and evaluation metrics.

[0077] In the above process, in order to evaluate the accuracy of the phenological identification results in this embodiment, the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) obtained by statistics are also used as the basic calculation elements for performance evaluation. The overall precision, accuracy, recall, F1 score, or Kappa coefficient of the model obtained in each round of training are calculated as follows. The specific calculation formula can be expressed as:

[0078] Overall accuracy

[0079] Accuracy

[0080] Recall rate

[0081]

[0082] Carpa coefficient

[0083] In the above formula, This represents the observed probability of consistency. This represents the probability of expected consistency due to chance factors. The Kappa coefficient ranges from -1 to 1; the closer the value is to 1, the stronger the consistency between the model's prediction and the true label.

[0084] The fourth step is the crop phenological stage testing phase. The test set is input into the trained crop phenological stage recognition model to obtain crop phenological stage detection and classification results, and the model's recognition performance is evaluated based on these results.

[0085] When identifying the phenological stages of plants in the field based on the optimal model obtained from the above training, this application can be operated in the manner shown in steps A1 to A9 below:

[0086] Step A1: After acquiring high-resolution and basic-resolution remote sensing images corresponding to several time-series nodes according to a preset sampling period, the remote sensing images are processed as follows: After acquiring the basic-resolution image, initial alignment is completed based on the exchangeable image file information (EXIF information) and the Global Navigation Satellite System (GNSS) coordinates contained therein during the UAV shooting process. Then, the camera parameters are optimized and corrected based on the measured coordinates of each control point that was fixed on the field ground of the test area before shooting and its correspondence with the pixels in the captured image. The acquired images are then finely aligned. Dense point clouds are generated based on the aligned and corrected images. Interpolation processing is performed on the dense point clouds to generate a corresponding digital elevation model. Then, based on the digital elevation model, orthorectification and stitching are used to generate an orthophoto mosaic that can cover the target crop area. Thus, the basic-resolution orthophoto mosaic that covers the entire test area and has consistent spatial geometry is used as the input sample for subsequent phenological identification. For high-resolution remote sensing images, this application obtains the original images of a single field scene by taking photographs. Then, it extracts the image of the field at the center of the image from the high-resolution images of the same field as a high-resolution image, providing input samples for subsequent phenological identification. Here, in the application stage of the model, we generally take 3 to 5 sets of remote sensing images, of which only at least one set of high-resolution remote sensing images is needed to effectively identify the current phenological stage of the crops in the field.

[0087] Step A2: Using the base resolution input sample obtained after processing in the above manner as a time anchor, construct a time series according to the shooting timestamp of the base resolution input sample, align the high-resolution images acquired at low frequencies to the corresponding time index positions in the time series, and set the missing mask corresponding to the time index position t. The availability status of high-resolution images in a time series is marked with 0 or 1.

[0088] Step A3: Input the base resolution image and high resolution image from the time series constructed according to the above steps into two parallel mobile vision efficient convolutional neural networks (MobileNetV3), respectively. The two independent mobile vision efficient convolutional neural networks (MobileNetV3) extract the image features corresponding to the high resolution image. Image features corresponding to the base resolution image .

[0089] Step A4: Extract feature vectors of high-resolution features using Global Average Pooling (GAP) layer. Feature vectors of basic resolution features Then, each image is mapped to a unified feature space according to its corresponding linear projection matrix at its respective resolution, resulting in the base resolution image features mapped to a unified feature space with d dimensions. and corresponding high-resolution image features .

[0090] Step A5: Map the base resolution image features that have been mapped to a unified feature space. and high-resolution image features Concatenate to form a joint context vector .

[0091] Step A6, combine the context vectors The input is fed into a missing-aware gating network to generate its corresponding adaptive gating weights.

[0092] Step A7: Perform multi-scale feature fusion on the vectors at each time index position according to the missing mask and adaptive gating weights. This adaptively generates baseline features based on the features of the base resolution image when the high-resolution image is unavailable, and fuses the high-resolution image features with the base resolution image features according to the corresponding adaptive gating weights when the high-resolution image is available, generating a fused feature sequence. .

[0093] Step A8: Load the optimal model parameters obtained during the training phase, and combine the fused feature sequences obtained in the previous steps. The input is fed into a Long Short-Term Memory (LSTM) network constructed using the optimal weights obtained in the aforementioned training phase for phenological identification. The temporal hidden features output by this LTM network are mapped through a classification layer to obtain the predicted probability distribution of phenological periods corresponding to each time index position. For each time index position, the phenological period category with the highest predicted probability is selected as the identification result for that time index position. After processing by the classification layer, the predicted probability distribution of each phenological period is output, and the crop phenological period with the highest probability is taken as the model's identification result.

[0094] To further illustrate the ability of the method of this invention to focus on key phenotypic regions at different phenological stages, we also... Figure 4 The paper presents a visual analysis of typical samples from different phenological stages and their corresponding model response heatmaps. For details on the model training strategy, please refer to [link / reference needed]. Figure 4 The image shows typical samples and their model response heatmaps at different phenological stages. Figure 4 As shown in Part (a), when using only single-resolution information, the Long Short-Term Memory (LSTM) network exhibits some overlap in its regions of interest across different phenological stages. However, by combining feature information from both basic and high-resolution images, the model can focus more intently on regions closely related to changes in phenological stages. For instance, during the jointing-booting stage, it primarily focuses on changes in overall plant posture and canopy density; during the heading-flowering stage, it focuses on the area where the panicle layer appears; and during the milk, waxy, and maturity stages, it focuses more on areas showing changes in panicle color and maturity status. Figure 4 Part (b) further illustrates the comparison results of feature distribution under different feature configurations. It can be seen that filtering based solely on the basic resolution features and filtering based solely on the high-resolution features yields worse dimensionality reduction distribution results compared to filtering based on a fusion feature sequence combining both basic and high-resolution features. The addition of fusion features enables the model to exhibit better inter-class separation and intra-class compactness across different phenological stages, and also improves the overall accuracy, precision, recall, F1 score, and Kappa coefficient. In other words, the multi-scale spatiotemporal fusion strategy adopted in this application can effectively enhance feature representation capabilities and improve phenological identification performance.

[0095] Because acquiring high-resolution remote sensing images requires a long shooting time each time, in order to optimize the balance between sampling efficiency and recognition accuracy, this embodiment further compares the changes in model recognition performance caused by different high-resolution image sampling intervals (Δ), while ensuring the continuity of the basic resolution. For details, please refer to... Figure 5 As shown, as the sampling interval Δ between two consecutive high-resolution images gradually increases from 1 day, the overall evaluation indicators of the model show a slow downward trend. However, when the sampling interval Δ is maintained within the range of 5 to 7 days, the overall accuracy, precision, recall, F1 score, and Kappa coefficient of the model can still remain at a high level. This indicates that appropriately reducing the sampling frequency of high-resolution images within this range does not significantly weaken the recognition effect of this model. Therefore, in practical applications, users can flexibly optimize the sampling interval of high-resolution images or the flight frequency of UAVs according to the scale of the monitoring task, cost constraints, and target recognition accuracy requirements.

[0096] In this embodiment, remote sensing data needs to be collected from a total of 1040 plots, encompassing nearly 500 crop varieties. Each basic resolution image allows the drone to fly across the entire field at a height of 20 meters, a process that takes approximately 20 minutes each time. A high-resolution image taken at a height of 4 meters takes approximately 4 hours. Based on the hybrid sampling strategy recommended in this study—daily basic resolution image acquisition combined with weekly high-resolution image acquisition—the estimated total drone flight time required per week is 7 × 20 minutes + 1 × 4 hours = 380 minutes ≈ 6.3 hours / week. In comparison, the traditional daily pure human resource strategy requires approximately 7 × 4 hours = 28 hours / week. Phenological identification technology relying on daily high-resolution image acquisition requires 7 × 4 hours = 28 hours / week. Although the accuracy of the phenological identification model provided in this application may vary depending on field size and local operating conditions, this application still provides a practical reference for large-scale breeding applications, achieving a significant overall efficiency advantage through the hybrid sampling strategy.

[0097] In summary, this application is able to pass. Figure 6 The four-step process shown, combined with Figure 1 The multi-scale sample construction and phenological annotation process shown Figure 2 The dual-branch feature extraction and missing-aware gating fusion mechanism shown Figure 3 The comparison results of performance and resource overhead of different backbone networks shown are as follows: Figure 4 The response heatmap and feature distribution shown indicate the ability to comprehensively select appropriate models and sampling periods. This application achieves high-precision identification of phenological stages in rice germplasm resources. Through comprehensive evaluation of overall accuracy, precision, recall, F1 score, and Kappa coefficient, it can be verified that this application maintains good classification performance and consistency even when high-resolution images are sparse or missing. Simultaneously, by employing a hybrid sampling method of high-frequency acquisition of basic resolution images and low-frequency supplementation of high-resolution images, this application significantly reduces the number of repeated flights by high-resolution UAVs and shortens the total operation time within the monitoring period, while ensuring temporal continuity and phenological identification accuracy, thus achieving an effective balance between identification accuracy and sampling efficiency. Compared with existing technologies, this application not only significantly improves the ability to distinguish adjacent phenological stages through information complementarity between basic and high-resolution images and temporal modeling using long short-term memory networks, but also meets the high-throughput monitoring needs of large-scale breeding scenarios, thus possessing good practicality, scalability, and application value.

Claims

1. A multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images, characterized by the following steps: include: First, for remote sensing images of different resolutions sampled according to a preset period, after they are aligned, stitched and corrected, the high-frequency base resolution image is used as the time anchor point, and a time series is constructed according to the shooting timestamp of the base resolution image. The low-frequency high-resolution images are aligned to the corresponding time index positions in the time series, and the availability status of the high-resolution images in the time series is indicated by a missing mask. Then, the base resolution image and the high resolution image are input into two parallel feature extraction branches respectively, and their respective image features are extracted. Next, the extracted image features from images of different resolutions are mapped to a unified feature space. Multi-scale feature fusion is performed using a missing-aware gating fusion mechanism. When high-resolution images are unavailable, baseline features are generated according to the features of the base resolution image. When high-resolution images are available, the high-resolution image features and the base resolution image features are fused according to adaptive gating weights to generate a fused feature sequence. Finally, the fused feature sequence is input into a long short-term memory network, and after being processed by a classification layer, the predicted probability distribution of each phenological stage is output, thereby providing crop phenological stage identification results. In the step of obtaining remote sensing images of different resolutions by sampling according to a preset period, remote sensing images of different resolutions are obtained by controlling the flight altitude of the UAV. Specifically, a basic resolution image is acquired based on the image taken by the UAV at a first flight altitude, and a high resolution image is acquired based on the image taken by the UAV at a second flight altitude. The second flight altitude is lower than the first flight altitude, and the sampling frequency of the basic resolution image is higher than the sampling frequency of the high resolution image.

2. The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in claim 1, characterized in that, When acquiring base resolution images, the forward and lateral overlap during the shooting process is maintained at more than 60%. After orthorectification and stitching of the captured images, the resulting orthophoto mosaic is used as the base resolution image. During the high-resolution image sampling process, single-view photos of the field are taken, and the image of the field corresponding to the center position of the image is extracted from the high-resolution photos of the same field as the high-resolution image.

3. The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in claim 1, characterized in that, The specific steps for indicating the availability status of high-resolution images in a time series using a missing mask include: In the time series constructed based on the capture timestamps of the base resolution images, the corresponding base resolution image is matched for each time index position t. High-resolution images and the corresponding missing mask, wherein, when a high-resolution image with a usable state exists at any t-th time index position, based on the high-resolution image The corresponding shooting timestamp will be the missing mask corresponding to the time index position. Set as If no high-resolution image is available at any time index position t, set the high-resolution image corresponding to that time index position. And the missing mask corresponding to the time index position. Set as .

4. The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in claim 3, characterized in that, The base resolution image and the high resolution image are respectively input into two parallel feature extraction branches. The specific steps for extracting their image features include: A backbone feature extraction network capable of outputting fixed-dimensional feature vectors is used to extract features from both basic resolution and high-resolution images. After global average pooling, the image features corresponding to each resolution image are extracted. The backbone feature extraction network is a convolutional neural network or a lightweight visual feature extraction network.

5. The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in claim 4, characterized in that, The two feature extraction branches input to the base resolution image and the high resolution image respectively adopt the same backbone network architecture, but do not share weights. The two backbone feature extraction networks are trained independently to learn the image feature representations at their respective resolutions.

6. The multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in claim 4, characterized in that, The specific steps for generating a fused feature sequence based on basic resolution image features mapped to a unified feature space and high-resolution image features using a missing-aware gated fusion mechanism include: The base resolution image features extracted from the base resolution image and high-resolution image features obtained from high-resolution images Perform vector concatenation to form a joint context vector. ; Joint context vector Input a learnable gating network to generate adaptive gating weights. ; Based on the missing mask Gated residual fusion is performed on the base resolution image features and high resolution image features at each time index position t. The fusion result of each time index position t Arrange them sequentially to obtain the fused feature sequence; in, This represents the activation function used to normalize the adaptive gate weights to... Within the interval (0,1); This represents the weight matrix in a gated network that projects the high-dimensional context as a scalar importance index. This represents the bias term used to adjust the activation threshold; This indicates element-wise multiplication.

7. A multi-scale spatiotemporal fusion system for phenological identification based on remote sensing images, characterized in that, include: The image acquisition unit is used to acquire remote sensing images of different resolutions according to a preset cycle; A data storage unit is used to back up and store the remote sensing images of different resolutions; An image processing unit is used to acquire remote sensing images from a data storage unit and output crop phenological period identification results according to any one of claims 1-6.

8. The multi-scale spatiotemporal fusion system for phenological identification based on remote sensing images as described in claim 7, characterized in that, The image acquisition unit is mounted on the UAV and is used to capture remote sensing images at different flight altitudes according to different sampling frequencies.

9. A computer-readable storage medium, characterized in that, The system stores a computer program that can run on a processor, which, when executed, implements the multi-scale spatiotemporal fusion method for phenological identification based on remote sensing images as described in any one of claims 1-6.

Images