Image rape growth period classification method and system based on tower video monitoring image

Abstract

This invention discloses a method and system for classifying the growth stages of rapeseed based on video surveillance images from iron towers. The method includes sequentially performing brightness homogenization, watermark removal, and registration processing on the rapeseed iron tower video surveillance images; constructing a forward sequence sorted by time ascending and a reverse sequence sorted by time descending from the processed rapeseed iron tower video images into datasets, which are then input into a dual-branch encoder for processing to extract spatial and bidirectional temporal features; and fusing bidirectional features with a decoder to extract higher-level features and capture dynamic changes in the rapeseed growth stages, classifying the rapeseed at different growth stages through a classification layer. This invention aims to combine deep learning and iron tower video data to overcome the limitations of insufficient data from traditional remote sensing weather monitoring and the high cost and limited battery life of drones, thereby improving the stability, economy, and timeliness of agricultural resource monitoring and achieving efficient, accurate identification and automated classification of rapeseed growth stages.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for classifying the growth stage of rapeseed based on video surveillance images from iron towers. Background Technology

[0002] With the rapid development of smart agriculture, accurate identification of rapeseed growth stages has become crucial for improving field management efficiency. Traditional manual observation methods are highly subjective and have limited coverage, making them unsuitable for large-scale planting. Satellite remote sensing is susceptible to weather interference and lacks sufficient spatiotemporal resolution, while drone remote sensing is costly and difficult to maintain continuously. In contrast, high-definition video monitoring technology offers advantages such as flexible deployment, low cost, and continuous data, providing a stable data source for dynamic monitoring of the growth stage. However, this technology still faces multiple challenges: there are scene differences between high-definition video monitoring technology based on field towers and high-definition video monitoring technology based on drones, and changes in field lighting, weather interference, and complex backgrounds affect image quality. The phenotypic features of rapeseed during the transition stages of the growth period are subtle, and differences between adjacent periods are difficult to capture. Traditional methods combining color and texture features with shallow machine learning have limitations in feature representation and generalization capabilities. In recent years, the development of deep learning technology, especially convolutional neural networks, has provided a new path for accurate classification of the growth stage. By automatically learning multi-level abstract features of images, key information about the growth stage can be effectively mined, significantly improving classification robustness and automation. Therefore, exploring a method for classifying rapeseed growth stages by integrating tower video images with deep learning has significant theoretical value and application prospects for promoting intelligent crop growth monitoring and smart agriculture. How to integrate tower video images with deep learning to achieve accurate classification of rapeseed growth stages has become a key technical issue in promoting intelligent crop growth monitoring and smart agriculture. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method and system for classifying the growth stage of rapeseed based on tower video monitoring images, addressing the aforementioned problems in the prior art. This invention aims to combine deep learning and tower video data to overcome the limitations of insufficient data due to weather constraints in traditional remote sensing and the high cost and limited battery life of drones, thereby improving the stability, economy, and timeliness of agricultural resource monitoring and achieving efficient and accurate identification and automated classification of the rapeseed growth stage.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for classifying the growth stage of rapeseed based on video surveillance images from iron towers includes the following steps: S1, perform brightness uniformization processing on the video monitoring image sequence of the rapeseed tower; S2, perform watermark removal processing on the video surveillance image sequence of the rapeseed tower after brightness uniformization processing; S3, specify a reference image for the watermark-free rapeseed tower video surveillance image sequence, and register other images other than the reference image with the reference image; S4. The registered rapeseed tower video surveillance image sequence is classified using a pre-trained rapeseed growth stage classification model to obtain the rapeseed growth stage.

[0005] Optionally, when performing brightness uniformization processing on the rapeseed tower video monitoring image sequence in step S1, the processing for each rapeseed tower video monitoring image in the sequence includes: S1.1, Convert the video surveillance image of the rapeseed tower from the BGR color space to the HSV color space; S1.2, Gaussian filtering is applied to the luminance channel of the rapeseed tower video surveillance image converted to the HSV color space; S1.3 is the gamma correction coefficient for the smoothed image obtained after Gaussian filtering; S1.4 For each pixel of the rapeseed tower video surveillance image converted to HSV color space, the exponential value of the gamma correction coefficient is calculated as the new brightness to achieve gamma correction of the rapeseed tower video surveillance image converted to HSV color space to adjust the brightness distribution. S1.5 converts the gamma-corrected video surveillance image of the rapeseed tower back to the BGR color space.

[0006] Optionally, the functional expression for calculating the gamma correction coefficient in step S1.3 is: ; in, Gamma correction factor Average brightness This represents the brightness of the original video surveillance image of the iron tower.

[0007] Optionally, in step S2, when performing watermark removal processing on the rapeseed tower video surveillance image sequence after brightness homogenization processing, the processing for each rapeseed tower video surveillance image in the sequence includes: S2.1, Obtain the watermark position mask in the input iron tower video. The watermark position mask is the same size as the rapeseed iron tower video monitoring image, and the pixel value is 255 and 0 to distinguish the watermark position and the non-watermark position. S2.2, Based on the watermark position mask, replace the pixel value of the watermark position in the rapeseed tower video monitoring image after brightness homogenization with a preset fixed value to prevent watermark interference in the watermark position mask area.

[0008] Optionally, registering images other than the reference image with the reference image in step S3 includes: S3.1, The watermark-removed rapeseed tower video surveillance image is used as the image to be registered, and Fourier transforms are performed on the image to be registered and the reference image respectively. S3.2 Calculate the phase difference between the image to be registered and the reference image based on the Fourier transform results; S3.3, Perform inverse Fourier transform on the phase difference to obtain the correlation map between the image to be registered and the reference image, find the peak position in the correlation map and construct the translation vector between the image to be registered and the reference image; S3.4, perform a translation operation on the image to be registered based on the translation vector to obtain the registered tower video surveillance image.

[0009] Optionally, the function expression for calculating the phase difference in step S3.2 is: ; in, For phase difference, The Fourier transform result of the reference image. The Fourier transform result of the image to be registered. For the conjugate of F2, for The model.

[0010] Optionally, the rapeseed growth period classification model used in step S4 consists of an encoder and a decoder. The decoder includes a forward decoding branch, a backward decoding branch, and a stitching module. The forward decoding branch processes forward time image sequences, and the backward decoding branch processes backward time image sequences. The forward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in forward time, and the backward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in reverse time. Both the forward and backward decoding branches consist of three sequentially connected 3D convolutional modules and two 3D dilated convolutional modules. The output features of the forward and backward decoding branches are stitched together by the stitching module and output as the output features of the encoder to the decoder. The decoder includes four cascaded decoding units and a... Each classification unit includes a decoding unit comprising a 3D upsampling layer, a stitching module, and a 3D convolutional module. The 3D upsampling layer is used to upsample the output features of the encoder or the previous decoding unit and feed them into the stitching module. The stitching module stitches the upsampled features with the output features of the corresponding scale 3D convolutional module or 3D dilated convolutional module in the encoder and outputs the result to the 3D convolutional module in the current decoding unit. After 3D convolution by the 3D convolutional module in the current decoding unit, the result is output to the next decoding unit or classification unit. The classification unit comprises a feature refinement module (FRB), a self-attention module, and a classification module (CLB) connected in sequence. The feature refinement module (FRB) is composed of three stacked 3D convolutional modules, and the classification module (CLB) is composed of a 3D convolutional module and a Softmax activation layer.

[0011] Optionally, the self-attention module is a temporal self-attention module (TSA). The processing of the input feature map by the temporal self-attention module (TSA) includes: using three feedforward network branches, each composed of fully connected Dense layers, to map the input feature map into a query matrix Q, a key matrix K, and a value matrix V, respectively; reshaping the query matrix Q and the key matrix K and then calculating the attention weights through similarity calculation; reshaping the value matrix V and then performing weighted fusion with the attention weights through matrix multiplication to obtain temporal features effective for identifying the rapeseed growth period; and reshaping the temporal features effective for identifying the rapeseed growth period and then performing feature fusion and stabilization through residual connections and layer normalization (LN) to obtain the enhanced feature map of the self-attention module.

[0012] Optionally, the training of the rapeseed growth period classification model includes: S101: Acquire video monitoring image samples of rapeseed towers at different growth stages in multiple regions; S102, perform watermark removal processing on the rapeseed tower video surveillance image samples after brightness uniformization processing; S103, Register the watermark-free rapeseed tower video surveillance image sample with the reference image; S104, the registered rapeseed tower video surveillance image samples are divided into image blocks of a specified size to construct a training dataset consisting of rapeseed tower video surveillance image sequence samples and their labels. S105, Construct a rapeseed growth period classification model. Supervised training of the rapeseed growth period classification model is performed using the training dataset and a pre-defined loss function to obtain a pre-trained rapeseed growth period classification model. The loss function is the background stability constraint cross-entropy loss, and its functional expression is as follows: ; in, This represents the background stability constraint cross-entropy loss. This represents the category-weighted cross-entropy loss. Indicates the penalty loss for soft background. Indicates hard background penalty loss, It is a loss of spatial stability; ; in, Indicates the number of samples. Indicates the number of categories. The sample representing the predicted output Category The original output score, Indicates category The weight, Indicates sample Category The true label; ; in, The background penalty weight coefficient, Indicates sample False rapeseed forecasts It is a numerical protection constant used to ensure that the gradient can still propagate normally under numerical limits, and the training process will not be interrupted. Represents pixels The actual background category label, 1 for background and 0 for others; Indicates sample The sum of probabilities that the background-free portion belongs to different rapeseed growth stage categories. Indicates sample Predicted as category The probability of; ; in, Regarding conditions Indicator function, condition A value of 1 indicates a valid condition, otherwise a value of 0. Indicates the hard penalty threshold; ; in, This represents the number of valid samples, which are samples excluding image boundaries. and These represent valid samples respectively. Spatial gradients of the real background labels in the vertical and horizontal directions. and Representing samples respectively The spatial gradient of the predicted probability in the vertical direction. and These represent the unit offset in the vertical and horizontal directions, respectively. Indicates sample The actual background category label, 1 for background and 0 for others; Indicates sample Realistic background category tags, Indicates sample Realistic background category tags; Indicates the location On the sample The sum of probabilities that a rapeseed is predicted to belong to different growth stage categories.

[0013] Furthermore, the present invention also provides an image rapeseed growth period classification system based on tower video surveillance images, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the image rapeseed growth period classification method based on tower video surveillance images.

[0014] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The rapeseed growth stage classification method based on tower video monitoring images of the present invention includes: performing brightness homogenization processing on the rapeseed tower video monitoring image sequence; removing watermarks from the brightness homogenized rapeseed tower video monitoring image sequence; assigning a reference image to the watermark-removed rapeseed tower video monitoring image sequence and registering other images other than the reference image with the reference image; and classifying the registered rapeseed tower video monitoring image sequence using a pre-trained rapeseed growth stage classification model to obtain the rapeseed growth stage. The rapeseed growth stage classification method based on tower video monitoring images of the present invention, by combining deep learning and tower video data, can effectively overcome the limitations of insufficient data due to weather constraints in traditional remote sensing and the high cost and limited endurance of UAVs, significantly improving the stability, economy, and timeliness of agricultural resource monitoring, thereby achieving efficient and accurate identification and automated classification of rapeseed growth stages. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a comparison of the original rapeseed tower video surveillance images and the watermark-removed images in an embodiment of the present invention. (a1), (b1), (c1), and (d1) are four original rapeseed tower video surveillance images, and (a2), (b2), (c2), and (d2) are four original rapeseed tower video surveillance images obtained after watermark removal.

[0017] Figure 3 This is a schematic diagram of the network structure of the rapeseed growth period classification model in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the network structure of the self-attention module in an embodiment of the present invention.

[0019] Figure 5 The above is a comparison of the actual label image and the predicted result image in the embodiments of the present invention. Among them, (a1) is the actual label image of the seedling stage in the field, (b1) is the actual label image of the budding and bolting stage, (c1) is the actual label image of the flowering stage, (d1) is the actual label image of the silique development stage, (e1) is the actual label image of the mature harvest stage, (f1) is the actual label image of the completed harvest, (a2) is the predicted result image of the seedling stage in the field, (b2) is the predicted result image of the budding and bolting stage, (c2) is the predicted result image of the flowering stage, (d2) is the predicted result image of the silique development stage, (e2) is the predicted result image of the mature harvest stage, and (f2) is the predicted result image of the completed harvest. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0021] like Figure 1 As shown, the rapeseed growth stage classification method based on tower video surveillance images in this embodiment includes the following steps: S1, perform brightness uniformization processing on the video monitoring image sequence of the rapeseed tower; S2, perform watermark removal processing on the video surveillance image sequence of the rapeseed tower after brightness uniformization processing; S3, specify a reference image for the watermark-free rapeseed tower video surveillance image sequence, and register other images other than the reference image with the reference image; S4. The registered rapeseed tower video monitoring image sequence is classified using a pre-trained rapeseed growth stage classification model to obtain the rapeseed growth stages, including the seedling stage, budding and bolting stage, flowering stage, pod development stage, maturity and harvest stage, and completion of harvest.

[0022] The varying lighting conditions in images collected by the tower video monitoring system at different dates and times result in significant differences in visual features such as color and texture among rapeseed plants at the same growth stage. This can lead to the model misinterpreting changes in lighting as changes in the growth stage. To address this technical problem, step S1 of this embodiment performs brightness homogenization processing on the rapeseed tower video monitoring image sequence. Brightness homogenization effectively eliminates lighting differences, ensuring that rapeseed at the same growth stage exhibits consistent visual characteristics under different lighting conditions. This guarantees similar brightness distribution in images of the same field at different times, guiding the model to focus on the actual growth dynamics rather than the interference caused by changes in lighting. Specifically, the brightness homogenization processing of the rapeseed tower video monitoring image sequence in step S1 of this embodiment includes the following processing for each rapeseed tower video monitoring image in the sequence: S1.1, Convert the video surveillance image of the rapeseed tower from the BGR color space to the HSV color space; S1.2, Gaussian filtering is applied to the luminance channel V of the rapeseed tower video surveillance image converted to the HSV color space; S1.3 is the gamma correction coefficient for the smoothed image obtained after Gaussian filtering; S1.4 For each pixel of the rapeseed tower video surveillance image converted to HSV color space, the exponential value of the gamma correction coefficient is calculated as the new brightness to achieve gamma correction of the rapeseed tower video surveillance image converted to HSV color space to adjust the brightness distribution. S1.5 converts the gamma-corrected video surveillance image of the rapeseed tower back to the BGR color space.

[0023] The functional expression for calculating the gamma correction coefficient in step S1.3 of this embodiment is as follows: ; in, Gamma correction factor Average brightness This represents the brightness of the original video surveillance image of the iron tower.

[0024] In step S2 of this embodiment, when performing watermark removal processing on the rapeseed tower video monitoring image sequence after brightness homogenization processing, the processing for each rapeseed tower video monitoring image in the sequence includes: S2.1, Obtain the watermark location mask in the input iron tower video. The watermark location mask is the same size as the rapeseed iron tower video monitoring image, and the pixel value is 255 and 0 to distinguish the watermark location and the non-watermark location. In this embodiment, the pixel value of the watermark area is set to 255, and the non-watermark area is set to 0. S2.2, based on the watermark location mask, the pixel values ​​of the watermark location in the rapeseed tower video monitoring image after brightness homogenization are replaced with preset fixed values ​​to prevent watermark interference in the watermark location mask area. In this embodiment, specifically, the sample image is input into the watermark removal module based on the LaMa (Large Mask Inpainting) model to generate a mask image. The global receptive field is obtained through fast Fourier convolution, and content-aware inpainting is performed on the watermark area to output a clean image without watermark interference. The LaMa model is a well-known existing model, built based on FFC (Fast Fourier Convolution). It generates the repair content through an encoder-decoder structure and uses frequency domain branches to establish a global receptive field in the shallow layer, enabling the model to perceive the overall layout of the rapeseed field (such as the direction of the ridges and the distribution of the canopy) during the repair process, avoiding splicing marks or texture breaks between the repair area and the surrounding background, and achieving a more global and coordinated repair result. Figure 3 This example shows a comparison between the original images and the watermark-removed images. (a1), (b1), (c1), and (d1) are four original rapeseed tower video surveillance images, while (a2), (b2), (c2), and (d2) are the images obtained after watermark removal from the four original rapeseed tower video surveillance images.

[0025] It is understandable that in the time-series image sequence of the tower video, the watermark, as a fixed pattern, is unrelated to rapeseed growth but occupies effective pixels in the image, introducing redundant information irrelevant to the task of identifying the growth stage. Its presence disrupts the feature consistency of the same pixel at different time points, causing the model to establish a false association between the watermark feature and a certain growth stage, thus interfering with accurate identification. By performing watermark removal based on the LaMa model, while maintaining the integrity of the field edge structure, content-aware restoration is performed on the watermark area, effectively eliminating the interference caused by the watermark and providing a cleaner and more comparable data foundation for subsequent time-series modeling. It should be noted that the watermark is widely distributed in the original image. Therefore, only the watermark appearing in the region of interest is removed. The sample image here refers to the image after brightness homogenization adjustment.

[0026] In step S3 of this embodiment, when registering the watermark-removed rapeseed tower video surveillance image sequence with a reference image, the reference image can be the first image in the rapeseed tower video surveillance image sequence, and the remaining images can be used as images to be registered. The phase difference between the reference image and each sample image can be calculated to obtain the translation vector between the images. Registration of each sample image is then performed based on the translation vector. Registering other images besides the reference image with the reference image includes: S3.1, The watermark-removed rapeseed tower video surveillance image is used as the image to be registered, and Fourier transforms are performed on the image to be registered and the reference image respectively. S3.2 Calculate the phase difference between the image to be registered and the reference image based on the Fourier transform results; S3.3, Perform inverse Fourier transform on the phase difference to obtain the correlation map between the image to be registered and the reference image, find the peak position in the correlation map and construct the translation vector between the image to be registered and the reference image; S3.4, perform a translation operation on the image to be registered based on the translation vector to obtain the registered tower video surveillance image.

[0027] The expression for calculating the phase difference in step S3.2 of this embodiment is as follows: ; in, For phase difference, The Fourier transform result of the reference image. The Fourier transform result of the image to be registered. For the conjugate of F2, for The model.

[0028] Understandably, although the images are all captured from the same shooting angle, slight geometric shifts may still exist between images due to unavoidable factors such as camera shake, wind, and animal activity. Without registration, these shifts will cause spurious pixel changes between adjacent frames, which the model may mistakenly interpret as crop growth features. Image registration ensures that the same spatial location corresponds to the same physical region at different time points, effectively eliminating noise interference introduced by geometric shifts. This allows the model to focus on the dynamic changes in crop growth itself, providing a reliable spatial alignment foundation for constructing clean and comparable time-series datasets.

[0029] like Figure 3As shown, the rapeseed growth period classification model (DSTA-3D FCN) used in step S4 of this embodiment consists of an encoder and a decoder. The decoder includes a forward decoding branch, a backward decoding branch, and a stitching module. The forward decoding branch processes forward time image sequences, and the backward decoding branch processes backward time image sequences. The forward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in forward time, and the backward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in reverse time. It should be noted that the forward time series images are image sequences arranged in ascending time order (from past to future), and the backward time series images are image sequences arranged in descending time order (from future to past). The total dimensions of the forward and reverse time-series image sequences are (N, H, W, T, C), where N represents the number of samples, H represents the image patch height, W represents the image patch width, T represents the length of the time series, and C represents the number of channels. For a single forward and reverse time-series image sequence, the dimensions are (H, W, T, C), for example, in this embodiment the dimensions are 128×128×T×3.

[0030] In this embodiment, by reversing the dataset in the time dimension, forward and reverse time-series images are generated and input into a dual-branch encoder network. Multi-scale feature extraction is achieved through 3D convolution and dilated convolution modules, introducing nonlinear transformations to capture broader spatiotemporal context information and effectively improve feature representation capabilities. The encoder output is further processed by the decoder, utilizing upsampling, skip connections, and attention mechanisms to enhance feature expression and establish global temporal dependencies, significantly strengthening the model's ability to model the continuity of crop growth and dynamic transitions in the reproductive cycle.

[0031] The forward decoding branch and the reverse decoding branch are each composed of three sequentially connected 3D convolutional modules Conv 3D and two 3D dilated convolutional modules Dil Conv. The output features of the forward decoding branch and the reverse decoding branch are concatenated by the stitching module C and output as the output features of the encoder to the decoder. The decoder includes four cascaded decoding units and a classification unit. The decoding unit includes a 3D upsampling layer U, the stitching module C, and a 3D convolutional module Conv 3D. The 3D upsampling layer U is used to upsample the output features of the encoder or the previous decoding unit and send them to the stitching module. The stitching module C is used to stitch the 3D upsampled features with the output features of the corresponding scale 3D convolutional module or 3D dilated convolutional module in the encoder and output them to the 3D convolutional module in this decoding unit. After 3D convolution by the 3D convolutional module in this decoding unit, the output is output to the next decoding unit or classification unit. Figure 3In the 3D convolutional module Conv3D and the 3D dilated convolutional module Dil Conv, F:3×3×5 indicates that the kernel size is 3×3×5, S:(1,1,1) indicates that the stride of the 3D convolution operation is 1, #32 indicates that the output feature size is 32×32, and D:(1,1,2) indicates that the porosity of the 3D dilated convolution is 1, 1, and 2, respectively. The meanings of the kernel size, stride, porosity, and output feature size of different 3D convolutional modules Conv3D and Dil Conv can be deduced similarly. In this embodiment, the 3D convolutional module is composed of a 3×3×5 3D convolutional layer, a layer normalization layer LN, and a Gelu activation function layer connected in sequence. The 3D dilated convolutional module is composed of a 2×2×1 3D max pooling layer, a 1×1×2 3D dilated convolutional layer, a layer normalization layer LN, and a Gelu activation function layer connected in sequence. The system comprises two 3D convolutional modules. The first module has a stride of 1×1×1 and primarily extracts preliminary feature information from the raw data. It encodes features by increasing the number of channels and introduces nonlinearity to mitigate the vanishing gradient problem and accelerate training. The second and third modules use 2×2×1 strides to downsample the data, further extracting higher-level features while reducing the spatial dimension of the feature maps. The final two dilated convolutional modules reduce the spatial dimension of the feature maps using 3D max-pooling layers. By using 1×1×2 dilated convolutional layers, the temporal receptive field is expanded without increasing computational complexity, enabling the capture of broader contextual information and enhancing the model's ability to perceive long-distance dependencies and global structures. The concatenation and fusion of positive and negative features integrates bidirectional time-series information, thereby more comprehensively capturing the spatiotemporal features of the data and enhancing the model's ability to perceive dynamic changes and the richness of feature representation.

[0032] The decoder first passes the output of the dual-branch encoder through four repeated passes. The operations of the 3D upsampling layer, concatenate layer feature stitching, and 3D convolution module progressively achieve scale restoration of feature maps, multi-channel feature fusion, and feature extraction, while introducing nonlinearity. Specifically, in the four repeated operations, the upsampling layer uses the input features for spatial dimensioning; the stitching module performs skip connections between the upsampling results and the forward and reverse features of the same layer as the encoder, fusing deep semantics with shallow details; the 3D convolution module further extracts higher-level features from the concatenated features through a 3×3×5 3D convolutional layer, a layer normalization layer (LN), and a Gelu activation function layer, enhancing the expressive power of the features and introducing nonlinearity to gradually generate higher resolution and richer feature representations. Then, the output features are input into the FRB (Feature Refinement Module), composed of three 3D convolutional modules, to refine the features, and a TSA (Temporal Self-Attention) mechanism is introduced to focus on the temporal feature associations of the same pixel and capture dynamically changing features.

[0033] In this embodiment, the classification unit includes a feature refinement module (FRB), a self-attention module, and a classification module (CLB) connected in sequence. The feature refinement module (FRB) is composed of three stacked 3D convolutional modules, and the classification module (CLB) is composed of... The system consists of a 3D convolutional module and a softmax activation layer. The classification block (CLB), composed of the 3D convolutional module and the softmax activation layer, performs the classification, achieving refined identification of the rapeseed growth stage. The 3D convolutional layer reduces the number of channels in the feature map to 6 for feature compression and integration. The softmax activation function layer transforms the feature map output by the 3D convolutional layer into a probability distribution, which is used as the final output for the classification task. The number of channels in the output feature map of the 3D convolutional layer is consistent with the number of classes required by the task.

[0034] like Figure 4As shown, the self-attention module in this embodiment is a temporal self-attention module (TSA). The processing of the input feature map by the TSA includes: using three feedforward network branches, each composed of fully connected Dense layers, to map the input feature map into a query matrix Q, a key matrix K, and a value matrix V, respectively. Q is used to locate the phenological change stages that require focus in the time series, K is used to measure the correlation between different time steps, and V is used to integrate temporal feature information. After reshaping the query matrix Q and the key matrix K, attention weights are obtained through similarity calculation. These attention weights can enhance the key attention. The model identifies the characteristics of the growth period while suppressing the influence of non-critical stages. After reshaping the value matrix V, it is weighted and fused with the attention weights through matrix multiplication (matmul) to obtain temporal features effective for identifying the growth period of rapeseed. The temporal features effective for identifying the crop growth period are selected and the attention-enhanced feature representation is generated. The temporal features effective for identifying the growth period of rapeseed are reshaped and then fused and stabilized through residual connections and layer normalization (LN) to obtain the enhanced feature map of the self-attention module. This process effectively improves the model's ability to model the continuity of crop growth and the turning point of the growth period.

[0035] Specifically, when reshaping the temporal features effective for identifying the rapeseed growth stage, and then fusing and stabilizing these features using residual connections and layer normalized LNs to obtain the enhanced feature map of the self-attention module, the reshaping is specifically a scaling operation, and the functional expression for the scaling operation is as follows: ; in, This represents the result of the dot product operation between Q and K. This represents the feature dimension of the key vector K. This indicates the scaled result.

[0036] In this embodiment, the training of the rapeseed growth period classification model includes: S101: Acquire rapeseed tower video monitoring image samples at different growth stages in multiple regions; In this embodiment, rapeseed sample images from the same region are taken from the same angle, and the sample images come from the high-tower video monitoring system deployed by China Tower Corporation in these regions; The rapeseed image acquisition time interval can be 1-5 days; Figure 2 These are video monitoring image samples of rapeseed at different growth stages collected in this embodiment, where (a) is the seedling stage in the field, (b) is the budding and bolting stage, (c) is the flowering stage, (d) is the silique development stage and (e) is the mature harvest stage.

[0037] S102, perform watermark removal processing on the rapeseed tower video surveillance image samples after brightness uniformization processing; S103, Register the watermark-free rapeseed tower video surveillance image sample with the reference image; S104, the registered rapeseed tower video surveillance image samples are segmented into image blocks of a specified size (e.g., 128×128 pixels) to construct a training dataset consisting of rapeseed tower video surveillance image sequence samples and their labels. S105, Construct a rapeseed growth period classification model. Supervised training of the rapeseed growth period classification model is performed using the training dataset and a pre-defined loss function to obtain a pre-trained rapeseed growth period classification model. The loss function is the background stability constraint cross-entropy loss, which is a composite loss function with the following expression: ; in, This represents the background stability constraint cross-entropy loss. This represents the category-weighted cross-entropy loss. Indicates the penalty loss for soft background. Indicates hard background penalty loss, It is a loss of spatial stability; ; in, Indicates the number of samples. Indicates the number of categories. The sample representing the predicted output Category The original output score, Indicates category The weight, Indicates sample Category The true label; ; in, This is the background penalty weighting coefficient (the desired value can be selected as needed, for example, the value is 10 in this embodiment). Indicates sample False rapeseed forecasts It is a numerical protection constant used to ensure that the gradient can still propagate normally under numerical limits, and the training process will not be interrupted. Represents pixels The actual background category label, 1 for background and 0 for others; Indicates sample The sum of probabilities that the background-free portion belongs to different rapeseed growth stage categories. Indicates sample Predicted as category The probability of; ; in, Regarding conditions Indicator function, condition A value of 1 indicates a valid condition, otherwise a value of 0. This represents the hard penalty threshold (the desired value can be selected as needed, for example, 0.1 in this embodiment). ; in, This represents the number of valid samples, which are samples excluding image boundaries. and These represent valid samples respectively. Spatial gradients of the real background labels in the vertical and horizontal directions. and Representing samples respectively The spatial gradient of the predicted probability in the vertical direction. and These represent the unit offset in the vertical and horizontal directions, respectively. Indicates sample The actual background category label, 1 for background and 0 for others; Indicates sample Realistic background category tags, Indicates sample Realistic background category tags; Indicates the location On the sample The background stability constraint cross-entropy loss in this embodiment is a composite loss function. First, it uses class-weighted numerically stable cross-entropy (CE) as the basic classification term. This alleviates class imbalance by assigning the background class five times the weight of each rapeseed growth stage category, and directly calculates the loss from logits (the original output score) using log-sum-exp (logarithm-sum-exponential) to avoid numerical instability. Second, a dual background constraint mechanism is introduced: a soft penalty based on the square root progressively penalizes small errors using the concave function characteristic, ensuring effective gradient propagation in the early training phase; a hard penalty based on a 0.1 threshold imposes a binarization constraint on severe violations. Both work together to prevent the background region from being misclassified as any rapeseed category. Finally, a spatial stability loss is added as a regularization term. By calculating the spatial gradient of the background region, it penalizes drastic fluctuations in the rapeseed prediction probability, thereby ensuring the spatial coherence and smoothness of the prediction results for the background region and avoiding noisy predictions. This design provides an effective optimization objective for multi-classification tasks, ensuring that background areas are not misclassified as rapeseed and that their spatial distribution is consistent. By employing the background stability constraint cross-entropy loss in this embodiment for supervised training, and selecting optimal weights based on early stopping on the validation set, the constructed neural network model is suitable for accurate classification of crop growth stages in complex scenarios, effectively improving prediction accuracy and classification robustness. Figure 5 This example compares the actual label image and the predicted result image. (a1) is the actual label image for the seedling stage in the field, (b1) is the actual label image for the budding and bolting stage, (c1) is the actual label image for the flowering stage, (d1) is the actual label image for the silique development stage, (e1) is the actual label image for the mature harvest stage, (f1) is the actual label image for the completed harvest stage, (a2) is the predicted result image for the seedling stage in the field, (b2) is the predicted result image for the budding and bolting stage, (c2) is the predicted result image for the flowering stage, (d2) is the predicted result image for the silique development stage, (e2) is the predicted result image for the mature harvest stage, and (f2) is the predicted result image for the completed harvest stage.

[0038] To verify the effectiveness and accuracy of the rapeseed growth period classification method based on tower video surveillance images in this embodiment, tower video surveillance images from Wuling Village in Changde City, Paishan Village in Chenzhou City, and Zhongzhou Village in Zhuzhou City were used as the training set to train the rapeseed growth period classification model (DSTA-3D FCN) in this embodiment. Tower video surveillance images from Longtang Village in Hengyang City, Dongfengqiao Village in Yiyang City, and Youdong Village in Yiyang City were used as the test set for testing. The average overall accuracy (Average OA) and average Kappa coefficient (Average Kappa) were used as performance indicators for evaluation. The final test results are shown in Table 1.

[0039] Table 1: Test results of the rapeseed growth period classification model (DSTA-3D FCN) in this embodiment.

[0040]

[0041] Experimental evaluation shows that the average overall accuracy of the rapeseed growth period classification model (DSTA-3D FCN) in this embodiment is 0.9678, 0.9597, and 0.9221 in Longtang Village, Hengyang City, Youdong Village, and Dongfengqiao Village, Yiyang City, respectively, with average Kappa coefficients of 0.885, 0.847, and 0.845, verifying that the rapeseed growth period classification model (DSTA-3D FCN) in this embodiment has good prediction accuracy. It can be seen that the method in this embodiment is based on video surveillance data from China Tower, and after preprocessing such as brightness homogenization, watermark removal, and phase correlation registration, a 128×128 pixel multidimensional dataset is constructed. By reversing the dataset in the time dimension, forward and reverse dual time series images are generated and input into a dual-branch encoder network: multi-scale feature extraction is achieved through three-dimensional convolution and dilated convolution modules, introducing nonlinear transformation to capture broader spatiotemporal context information, effectively improving feature representation capabilities. The encoder output is further processed by the decoder, which utilizes upsampling, skip connections, and attention mechanisms to enhance feature representation and establish global temporal dependencies, significantly strengthening the model's ability to model the continuity of crop growth and dynamic transitions in the growth stage. Finally, a multi-task loss function is used for supervised training, and optimal weights are selected based on early cessation on the validation set. The constructed neural network model is suitable for accurate classification of crop growth stages in complex scenarios, effectively improving prediction accuracy and classification robustness.

[0042] Those skilled in the art will understand that the technical solutions provided by this invention can take the form of a method, a system, or a computer program product. Therefore, this invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. For example, this invention can provide an image rapeseed growth period classification system based on tower video surveillance images, including an interconnected microprocessor and a memory, the microprocessor being programmed or configured to execute the image rapeseed growth period classification method based on tower video surveillance images. Furthermore, this invention can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, produce instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0043] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for classifying the growth stage of rapeseed based on video surveillance images from iron towers, characterized in that, Includes the following steps: S1, perform brightness uniformization processing on the video monitoring image sequence of the rapeseed tower; S2, perform watermark removal processing on the video surveillance image sequence of the rapeseed tower after brightness uniformization processing; S3, specify a reference image for the watermark-free rapeseed tower video surveillance image sequence, and register other images other than the reference image with the reference image; S4. The registered rapeseed tower video surveillance image sequence is classified using a pre-trained rapeseed growth stage classification model to obtain the rapeseed growth stage.

2. The method for classifying rapeseed growth stages based on tower video surveillance images according to claim 1, characterized in that, In step S1, when performing brightness uniformization processing on the rapeseed tower video surveillance image sequence, the processing for each rapeseed tower video surveillance image in the sequence includes: S1.1, Convert the video surveillance image of the rapeseed tower from the BGR color space to the HSV color space; S1.2, Gaussian filtering is applied to the luminance channel of the rapeseed tower video surveillance image converted to the HSV color space; S1.3 is the gamma correction coefficient for the smoothed image obtained after Gaussian filtering; S1.4 For each pixel of the rapeseed tower video surveillance image converted to HSV color space, the exponential value of the gamma correction coefficient is calculated as the new brightness to achieve gamma correction of the rapeseed tower video surveillance image converted to HSV color space to adjust the brightness distribution. S1.5 converts the gamma-corrected video surveillance image of the rapeseed tower back to the BGR color space.

3. The method for classifying the growth stage of rapeseed based on video surveillance images from iron towers according to claim 2, characterized in that, The functional expression for calculating the gamma correction coefficient in step S1.3 is as follows: ； in, Gamma correction factor Average brightness This represents the brightness of the original video surveillance image of the iron tower.

4. The method for classifying the growth stage of rapeseed based on video surveillance images from iron towers according to claim 1, characterized in that, In step S2, when performing watermark removal processing on the rapeseed tower video surveillance image sequence after brightness homogenization, the processing for each rapeseed tower video surveillance image in the sequence includes: S2.1, Obtain the watermark position mask in the input iron tower video. The watermark position mask is the same size as the rapeseed iron tower video monitoring image, and the pixel value is 255 and 0 to distinguish the watermark position and the non-watermark position. S2.2, Based on the watermark position mask, replace the pixel value of the watermark position in the rapeseed tower video monitoring image after brightness homogenization with a preset fixed value to prevent watermark interference in the watermark position mask area.

5. The method for classifying rapeseed growth stages based on tower video surveillance images according to claim 1, characterized in that, Step S3, which involves registering images other than the reference image with the reference image, includes: S3.1, The watermark-removed rapeseed tower video surveillance image is used as the image to be registered, and Fourier transforms are performed on the image to be registered and the reference image respectively. S3.2 Calculate the phase difference between the image to be registered and the reference image based on the Fourier transform results; S3.3, Perform inverse Fourier transform on the phase difference to obtain the correlation map between the image to be registered and the reference image, find the peak position in the correlation map and construct the translation vector between the image to be registered and the reference image; S3.4, perform a translation operation on the image to be registered based on the translation vector to obtain the registered tower video surveillance image.

6. The method for classifying the growth stage of rapeseed based on tower video surveillance images according to claim 5, characterized in that, The function expression for calculating the phase difference in step S3.2 is as follows: ； in, For phase difference, The Fourier transform result of the reference image. The Fourier transform result of the image to be registered. For the conjugate of F2, for The model.

7. The method for classifying rapeseed growth stages based on tower video surveillance images according to claim 1, characterized in that, The rapeseed growth period classification model used in step S4 consists of an encoder and a decoder. The decoder includes a forward decoding branch, a backward decoding branch, and a stitching module. The forward decoding branch processes forward time image sequences, and the backward decoding branch processes backward time image sequences. The forward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in forward time, and the backward time image sequences are obtained by arranging rapeseed tower video surveillance image sequences in reverse time. Both the forward and backward decoding branches consist of three sequentially connected 3D convolutional modules and two 3D dilated convolutional modules. The output features of the forward and backward decoding branches are stitched together by the stitching module and then output as the encoder's output features to the decoder. The decoder includes four cascaded decoding units and a sub-module. The decoding unit includes a 3D upsampling layer, a stitching module, and a 3D convolution module. The 3D upsampling layer is used to upsample the output features of the encoder or the previous decoding unit and feed them into the stitching module. The stitching module stitches the upsampled features with the output features of the corresponding scale 3D convolution module or 3D dilated convolution module in the encoder and outputs them to the 3D convolution module in this decoding unit. After 3D convolution by the 3D convolution module in this decoding unit, the output is sent to the next decoding unit or classification unit. The classification unit includes a feature refinement module (FRB), a self-attention module, and a classification module (CLB) connected in sequence. The feature refinement module (FRB) is composed of three stacked 3D convolution modules, and the classification module (CLB) is composed of a 3D convolution module and a Softmax activation layer.

8. The method for classifying the growth stage of rapeseed based on video surveillance images from iron towers according to claim 7, characterized in that, The self-attention module is a temporal self-attention module (TSA). The processing of the input feature map by the TSA includes: using three feedforward network branches, each composed of fully connected Dense layers, to map the input feature map into a query matrix Q, a key matrix K, and a value matrix V, respectively; reshaping the query matrix Q and the key matrix K and then calculating the attention weights through similarity calculation; reshaping the value matrix V and then performing weighted fusion with the attention weights through matrix multiplication to obtain temporal features effective for identifying the rapeseed growth period; reshaping the temporal features effective for identifying the rapeseed growth period and then performing feature fusion and stabilization through residual connections and layer normalization (LN) to obtain the enhanced feature map of the self-attention module.

9. The method for classifying the growth stage of rapeseed based on video surveillance images from iron towers according to claim 7, characterized in that, The training of the rapeseed growth period classification model includes: S101: Acquire video monitoring image samples of rapeseed towers at different growth stages in multiple regions; S102, perform watermark removal processing on the rapeseed tower video surveillance image samples after brightness uniformization processing; S103, Register the watermark-free rapeseed tower video surveillance image sample with the reference image; S104, the registered rapeseed tower video surveillance image samples are divided into image blocks of a specified size to construct a training dataset consisting of rapeseed tower video surveillance image sequence samples and their labels. S105, Construct a rapeseed growth period classification model. Supervised training of the rapeseed growth period classification model is performed using the training dataset and a pre-defined loss function to obtain a pre-trained rapeseed growth period classification model. The loss function is the background stability constraint cross-entropy loss, and its functional expression is as follows: ； in, This represents the background stability constraint cross-entropy loss. This represents the category-weighted cross-entropy loss. Indicates the penalty loss for soft background. Indicates hard background penalty loss, It is a loss of spatial stability; ； in, Indicates the number of samples. Indicates the number of categories. The sample representing the predicted output Category The original output score, Indicates category The weight, Indicates sample Category The true label; ； in, The background penalty weight coefficient, Indicates sample False rapeseed forecasts It is a numerical protection constant used to ensure that the gradient can still propagate normally under numerical limits, and the training process will not be interrupted. Represents pixels The actual background category label, 1 for background and 0 for others; Indicates sample The sum of probabilities that the background-free portion belongs to different rapeseed growth stage categories. Indicates sample Predicted as category The probability of; ； in, Regarding conditions Indicator function, condition A value of 1 indicates a valid condition, otherwise a value of 0. Indicates the hard penalty threshold; ； in, This represents the number of valid samples, which are samples excluding image boundaries. and These represent valid samples respectively. Spatial gradients of the real background labels in the vertical and horizontal directions. and Representing samples respectively The spatial gradient of the predicted probability in the vertical direction. and These represent the unit offset in the vertical and horizontal directions, respectively. Indicates sample The actual background category label, 1 for background and 0 for others; Indicates sample Realistic background category tags, Indicates sample Realistic background category tags; Indicates the location On the sample The sum of probabilities that a rapeseed is predicted to belong to different growth stage categories.

10. A rapeseed growth stage classification system based on video surveillance images from iron towers, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the image rapeseed growth period classification method based on tower video surveillance images as described in any one of claims 1 to 9.

Images