Multi-scale semantic enhanced point supervision algorithm for counting multiple types of crop plants in aerial image

Abstract

The invention discloses a multi-scale semantic enhanced point supervision algorithm for counting multiple types of crop plants in an aerial image, and the algorithm comprises the following steps: data preprocessing: carrying out the marking of a crop image through a point; the RcsNet50 backbone network is used as a feature extractor to encode the input image; a CSGEM module is used for enhancing crop plant semantics and background information semantics such as soil and shadow in a semantic weight adjustment mode; an RDDCM module is used to adapt to scale changes of different crop plants through a parallel nonlinear convolution mechanism; performing dual-path decoding by using an attention map path AMP and a density map path DMP, and introducing a CoT module in the decoding process; and applying element-by-element multiplication to AMP and DMP decoding results to obtain the predicted number of crop plants. The influence of complex background information and scale change of crops on counting precision can be reduced, the MCPCNet is greatly improved in the aspects of precision, calculation efficiency and generalization ability, and the MCPCNet has superiority and practicability in the aspect of coping with complex agricultural challenges.

Description

Technical Field

[0001] The present invention relates to the field of smart agricultural technology, and in particular to a multi-scale semantically enhanced point supervision algorithm for counting various types of crop plants in aerial images. Background Art

[0002] Crop phenotyping is an important part of promoting the development of smart agricultural technology, and crop counting is an important part of crop phenotyping. Crop counting helps calculate the emergence rate and thus provides a basis for agricultural production activities such as seedling selection and reseeding. It improves farmers' decision-making in yield estimation, stress detection, disease prevention, etc. In plant breeding, counting targets such as leaves, fruits or flowers can provide valuable reference for the automatic evaluation of specific hybrids or cultivars. However, the traditional method of counting plants by manual measurement is time-consuming, laborious and prone to errors. This method is not only inefficient and subjective, but also difficult to meet the development needs of modern large-scale agriculture.

[0003] The team from Sichuan Agricultural University proposed a method based on the YOLOX framework, called YOLOPOD. Based on YOLOX, this method added a module for predicting the number of soybean pods and a decoupling module for classification and positioning during the training process, normalized the network output using Softmax, modified the loss function, and thus constructed a multi-task model. It also introduced a convolutional block attention module (CBAM) to achieve accurate recognition and counting of soybean pods.

[0004] Under the CSRNet density map generation framework, the team from Wuhan University of Technology combined the global feature information of seedling images and the correlation between features to propose a cotton seedling counting algorithm based on feature fusion. The algorithm enhances the global feature information of cotton seedlings through an attention module, and uses the dropout method for the feature fusion process. Finally, a density map is generated through de-redundancy and normalization operations to obtain the counting results.

[0005] The soybean pod counting algorithm YOLOPOD only performs single-type target detection on soybean pods and cannot detect other crops at the same time. In addition, in terms of data labeling, YOLOPOD uses anchor frames to label targets, which increases the basic development costs due to the large number of targets.

[0006] Although the cotton seedling counting algorithm adopts the density map counting method, the cotton seedlings present various postures during the growth process. In the images collected at different times, the scale of the cotton seedlings is complex, which is specifically manifested in the density or sparseness of the cotton seedlings, the scale changes caused by morphological differences, etc. These factors lead to a large difference between the predicted number and the actual number of some test images, resulting in poor adaptability of such algorithms. The above problems need to be further improved.

[0007] Therefore, how to provide a multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images to solve the problems existing in the existing technology is of great significance for its application. Summary of the invention

[0008] In view of this, the purpose of this application is to provide a multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images to solve the problem.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A multi-scale semantically enhanced point supervision algorithm for counting multi-type crop plants in aerial images includes the following steps:

[0011] S1: Data preprocessing: use points to mark the crop image, and the marking points are placed in the center of the crop as much as possible. The data is cropped, flipped, and processed for data augmentation.

[0012] S2: The RcsNet50 backbone network is used as a feature extractor to encode the input image and obtain high-level semantic features in the image by downsampling the crop plant image;

[0013] S3: Use the CSGEM module to enhance the semantics of crop plants and suppress the semantics of background information such as soil and shadows by adjusting the semantic weights;

[0014] S4: Use the RDDCM module to adapt to the scale changes of different crop plants through parallel nonlinear convolution mechanism;

[0015] S5: Use the attention map path AMP and the density map path DMP for dual-path decoding respectively, and introduce the Contextual transformer (CoT) module in the decoding process, fully mine the context information through CoT and use the self-attention mechanism of CoT to enhance the algorithm's optimization ability;

[0016] S6: Apply element-wise multiplication to the AMP and DMP decoding results to generate a refined final crop plant count map pair F refine , and use the sum function to calculate F refineSum the predicted number of crop plants S counting .

[0017] Preferably, the S1 is divided into the following steps:

[0018] S1.1: Collect TIF format data of various types of crop plants at multiple scales;

[0019] S1.2: The image size is cropped to 640X640;

[0020] S1.3: Perform data annotation and data enhancement.

[0021] Preferably, the specific steps of S1 are as follows:

[0022] The crop plants used data of four kinds of crops, namely watermelon, cotton, corn and rice. The watermelon, cotton and corn plant data were obtained through independent field collection, and the collection equipment was a drone equipped with a visible light imaging sensor;

[0023] After screening, we finally collected 1246 crop plant images with good quality. Labelme was used to annotate the data points, and a multi-category crop plant count dataset MCPC-Dataset containing 71566 annotated points was obtained.

[0024] Data augmentation is performed during the training process. The data augmentation methods include: first, randomly resizing the short side of the image to 512 according to the ratio [0.8, 1.2], and cropping image blocks with a fixed size of 400×400 at random positions; then randomly flipping it horizontally with probability 0.5, using parameters [0.5, 1.5] with probability 0.3 to perform gamma contrast transformation on the image, and randomly changing the image to grayscale with probability 0.1.

[0025] Preferably, the specific steps of S1 are as follows:

[0026] First, ResNet50 with the fully connected layer removed is used as the feature encoder (FME) to encode the crop plant image. The output form is a feature tensor. For the input image In, the encoding process uses F i (·) indicates that conv2_3, con3_4, conv4_6, and conv5_3 output the original image size respectively. 1 / 2 , 1 / 4 , 1 / 8 and 1 / 16 The semantic feature tensor of the feature encoder is called Res out, the encoding process can be expressed as shown in formula 1:

[0027] Res out =F i (In) (1);

[0028] Preferably, the specific steps of S3 are as follows:

[0029] The CSGEM module is used to enhance the semantics of crop plants and suppress the semantics of background information such as soil and shadow by adjusting the semantic weights. The features input to CSGEM are Res out ;

[0030] For the concurrent spatial group enhancement module CSGEM, specifically: the encoding feature Res with size C×H×W is out Split into G groups along dimension C, and then use the spatial averaging function Get the global statistical feature g to approximate the semantic feature Res out , as shown in Formula 2:

[0031]

[0032] Global statistical features g and Res out Perform Position-wiseDotProduct and Normalization to obtain good response distribution characteristics a i , using feature a i And Sigmoid activation function for Res out Scaling is performed as shown in Formula 3:

[0033]

[0034] In addition, Res out Perform global pooling, then perform ReLu and Sigmoid activation to obtain calibration weights Res out Perform 1×1×1 convolution and directly use Sigmoid activation to get calibration weights So the final calibration weight is Using Calibration Weights right Weight calibration is performed to obtain semantically enhanced features of crop plants As shown in Formula 4:

[0035]

[0036] Preferably, the specific steps of S4 are as follows:

[0037] The residual dynamic dilated convolution module (RDDCM) can promote the algorithm’s perception of crop scale changes through parallel nonlinear convolutions. The features input to the RDDCM are In order to facilitate the expression of the formula, is x;

[0038] Global pooling is performed on x and normalized using a fully connected layer, ReLU function, and Softmax. This process generates four normalized weights {π 1 ,π 2 ,π 3 ,π 4}, these normalized weights π i It is not fixed, but dynamically adapts to changes in the input feature x; normalized weight π i Obey the normalization constraint and ensure that π i The sum is equal to 1, parallel convolution conv i The weights and biases are represented as {W i ,b i}, the nonlinear function can be expressed as Formula 5:

[0039]

[0040] Where g(·) represents the activation function, W(x) and b(x) are shown in Formula 6 and Formula 7 respectively:

[0041] W(x)=∑π i (x)W i (6);

[0042] b(x)=∑π i (x)b i (7);

[0043] Then, a 1×1 convolution is used to calibrate the crop plant feature x and combine it with the dynamic convolution feature Combined to derive the calibration characteristics Then, the attention mechanism (such as Figure 5 shown at the top) is applied to x as a weight and applied to the calibration feature Thus, the adjustment feature y is obtained which takes into account the change in crop plant scale.

[0044] Preferably, the specific steps of S4 are as follows: cascading and decoding the semantic features obtained from steps S2 to S4, upsampling and cascading the semantic features obtained from steps S2 to S4 in order of output for decoding, and fusing the features in order according to the semantic feature sizes; in the feature fusion process, introducing the CoT module, and using the self-attention mechanism in CoT to fully mine the contextual information of the fused semantic features, thereby improving the generalization of the algorithm.

[0045] Preferably, the specific steps of S6 are as follows: As shown in formula (8), the dual-path fusion semantic features of AMP and DMP in S5 are element-wise multiplied (Element-wise multiplication), and a 1×1×1 convolution is used to generate the final crop plant count density map F refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting ;

[0046] S counting =sum(F refine ) (8).

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention uses point-labeled data for training. During the training process, data enhancement methods such as cropping and flipping are used to enrich the amount of training data. Compared with anchor frame regression training, the basic development cost of data labeling and large-scale data preprocessing is reduced at the same time;

[0049] 2. The target objects of the method of the present invention are various crops, which solves the problem of studying a single type of crop in the current research field and proposes a universal counting algorithm for various types of crop targets;

[0050] 3. In order to reduce the impact of complex background information and scale changes of crops on counting accuracy, the present invention adopts three improvement measures. First, the present invention designs a concurrent spatial group enhancement module CSGEM, which enhances the semantics of crop plants by adjusting the weights of concurrent spatial groups, and suppresses the influence of background information such as soil and shadows; secondly, the present invention designs a residual dynamic hole convolution module RDDCM, which adapts to plant scale changes through parallel nonlinear convolution; finally, the present invention introduces the Contextualtransformer module to fuse the contextual features of crop plants, and combines the self-attention mechanism to further enhance the generalization ability of the model.

[0051] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.

[0052] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0054] Figure 1 It is a technical flow chart of the present invention;

[0055] Figure 2 This is an aerial image sample of the Changji Comprehensive Experimental Base of the present invention;

[0056] Figure 3 It is the MCPCNet framework diagram of the present invention;

[0057] Figure 4 It is the CSGEM structure diagram of the present invention;

[0058] Figure 5 It is the RDDCM framework diagram of the present invention;

[0059] Figure 6 is the CoT framework of the present invention;

[0060] Figure 7 A schematic diagram of crop labeling of the present invention;

[0061] Figure 8 The crop plant activation feature map of the present invention;

[0062] Fig. 9 The crop plant class activation feature map using CSGEM and RDDCM of the present invention;

[0063] Fig.10 It is the activation feature map of the crop plant class in the MCPCNet of the present invention;

[0064] Fig.11 This is an example diagram of prediction of various types of crop plants of the present invention;

[0065] Fig.12 It is the crop plant counting diagram of the comparison algorithm of the present invention;

[0066] Fig.13 Predicting the fruiting map for the watermelon plant orthophoto TIF map of the present invention;

[0067] Fig.14 It is a prediction result diagram of the rice plant orthophoto TIF map of the present invention;

[0068] Fig.15 It is a prediction result diagram of the corn plant orthophoto TIF diagram of the present invention;

[0069] Fig.16 This is a prediction result diagram of the cotton plant orthophoto TIF map of the present invention. DETAILED DESCRIPTION

[0070] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.

[0071] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0072] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0073] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.

[0074] See also Figure 1-16 The present invention provides a technical solution of a multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images, comprising the following steps:

[0075] S1: Data preprocessing: use points to mark the crop image, and the marking points are placed in the center of the crop as much as possible. The data is cropped, flipped, and processed for data augmentation.

[0076] S2: The RcsNet50 backbone network is used as a feature extractor to encode the input image and obtain high-level semantic features in the image by downsampling the crop plant image;

[0077] S3: Use the CSGEM module to enhance the semantics of crop plants and suppress the semantics of background information such as soil and shadows by adjusting the semantic weights;

[0078] S4: Use the RDDCM module to adapt to the scale changes of different crop plants through parallel nonlinear convolution mechanism;

[0079] S5: Use the attention map path AMP and the density map path DMP for dual-path decoding respectively, and introduce the Contextual transformer (CoT) module in the decoding process, fully mine the context information through CoT and use the self-attention mechanism of CoT to enhance the algorithm's optimization ability;

[0080] S6: Apply element-wise multiplication to the AMP and DMP decoding results to generate a refined final crop plant count map pair F refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting .

[0081] The specific steps of S1 are as follows:

[0082] (1) The experiment used plant data of four crops: watermelon, cotton, corn, and rice. The plant data of watermelon, cotton, and corn were obtained through independent field collection in May 2023 at the Changji Laolong River Comprehensive Experimental Base in Urumqi, Xinjiang, China (longitude 87.332E, latitude 44.263N). The collection equipment was a drone equipped with a visible light imaging sensor. The overall data collection process is as follows: Figure 2 The rice plant data is an open source dataset URCdatasets on the Internet. The tillering stage data was collected in August 2018, and the drone flight altitude was 20m.

[0083] (2) After screening, the algorithm finally collected 1246 images of crop plants with good quality. Labelme was used for data annotation. In order to solve the defect of poor model adaptability caused by the scale changes of various crop plants mentioned in the background technology, the present invention selected aerial images of crop plants with obvious differences in four scales. Finally, the algorithm obtained a multi-type crop plant counting dataset MCPC-Dataset containing 71566 annotated points.

[0084] (3) Data augmentation is performed during the training process. The data augmentation methods include: First, the algorithm randomly resizes the short side of the image to 512 according to the ratio [0.8, 1.2], and crops the image block with a fixed size of 400×400 at a random position. Then, it randomly flips it horizontally with a probability of 0.5, and uses the parameters [0.5, 1.5] with a probability of 0.3 to perform gamma contrast transformation on the image. The algorithm also randomly changes the image to grayscale with a probability of 0.1.

[0085] S2: Encode the original crop images in sequence to learn useful high-level semantic features. The encoding process uses ResNet50 with the fully connected layer removed to downsample the crop features in the image in sequence to obtain the crop feature information and use it as the feature input for the next step.

[0086] The MCPCNet framework of the multi-type crop plant counting algorithm proposed in this invention is as follows Figure 3 As shown in the figure, the residual network can solve the problems such as gradient explosion caused by the deepening of the algorithm layers, and also effectively reduce the number of network parameters. The present invention first uses ResNet50 with the fully connected layer removed as the feature encoder (FME) to encode the crop plant image, and the output form is a feature tensor. For the input image In, the encoding process uses F i (·) indicates that conv2_3, con3_4, conv4_6, and conv5_3 output the original image size respectively. 1 / 2 , 1 / 4 , 1 / 8 and 1 / 16 To facilitate subsequent description, this algorithm refers to the feature after the feature encoder is encoded as Res out , the encoding process can be expressed as shown in formula 1:

[0087] Res out =F i (In) (1)

[0088] S3. Figure 4 As shown in Figure 2, the concurrent spatial group enhancement module CSGEM can adjust the feature semantic weights to enhance the semantics of crop plants and suppress the background semantics such as soil and shadow. The features input to CSGEM are Res out .

[0089] For the concurrent space group enhancement module CSGEM, specifically, this algorithm converts the encoding feature Res with size C×H×W into out Split into G groups along dimension C, and then use the spatial averaging function Get the global statistical feature g to approximate the semantic feature Res out , as shown in Formula 2:

[0090]

[0091] Global statistical features g and Res out Perform Position-wiseDotProduct and Normalization to obtain good response distribution characteristics a i , using feature a i And Sigmoid activation function for Res out Scaling is performed as shown in Formula 3:

[0092]

[0093] In addition, this algorithm will Res out Perform global pooling, then perform ReLu and Sigmoid activation to obtain calibration weights Res out Perform 1×1×1 convolution and directly use Sigmoid activation to get calibration weights So the algorithm gets the final calibration weight Using Calibration Weights right Weight calibration is performed to obtain semantically enhanced features of crop plants As shown in Formula 4:

[0094]

[0095] S4. Figure 5 As shown in Figure 2, the residual dynamic dilated convolution module (RDDCM) can facilitate the algorithm’s perception of crop scale changes through parallel nonlinear convolution. The features input to the RDDCM are In order to facilitate the expression of formulas, this algorithm is x.

[0096] This algorithm performs global pooling on x and normalizes it using a fully connected layer, ReLU function, and Softmax. This process generates four normalized weights {π 1 ,π 2 ,π 3 ,π 4}, these normalized weights π i It is not fixed, but dynamically adapts to changes in the input feature x. Normalized weight π i Obey the normalization constraint and ensure that π i The sum of is equal to 1. Parallel convolution conv i The weights and biases are represented as {W i ,b i}, the nonlinear function can be expressed as Formula 5:

[0097]

[0098] Where g(·) represents the activation function, W(x) and b(x) are shown in Formula 6 and Formula 7 respectively:

[0099] W(x)=∑π i (x)W i (6);

[0100] b(x)=∑π i (x)b i (7);

[0101] The algorithm then uses a 1×1 convolution to calibrate the feature x of the crop plant and enter the formula here. It combines it with the dynamic convolution feature x to obtain the calibration feature y. The algorithm then uses the attention mechanism (such as Figure 5 top) is applied to x as a weight and applied to the calibration feature y to obtain the adjusted feature y that takes into account the variation in crop plant scale.

[0102] S5. Concatenate and decode the semantic features obtained from steps S2 to S4, such as Figure 3 As shown in Figure 2, the semantic features obtained from steps S2 to S4 are upsampled and cascaded in order to be decoded, and feature fusion is performed in order according to the semantic feature size. In the feature fusion process, this algorithm introduces Figure 6 The CoT module shown uses the self-attention mechanism in CoT to fully explore the contextual information of the fused semantic features, thereby improving the generalization of the algorithm.

[0103] S6. As shown in formula (8), the dual-path fusion semantic features of AMP and DMP in S5 are element-wise multiplied (Element-wise multiplication), and a 1×1×1 convolution is used to generate the final crop plant count density map F refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting .

[0104] S counting =sum(F refine ) (8).

[0105] The following is a specific example and combined with Figure 3 The present invention is further described in detail. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0106] (1) Figure 7 As shown, point marking is used to mark crop plants;

[0107] (2) Figure 8 As shown in FIG. 1 , the present invention first uses ResNet50 to extract the semantics of crop plants. Figure 8 In the class activation feature map, ResNet50 has semantics such as shadow and soil (the yellow box marks the part). In order to enhance the semantics of crop plants and suppress the semantics of shadow and soil, this paper develops CSGEM using the crop plant feature weight adjustment mechanism. This algorithm can see that the addition of CSGEM effectively suppresses the semantics of shadow and soil, and the crop plant features are more convergent.

[0108] Figure 8 (a) Watermelon plants; (b) Rice plants; (c) Corn plants; (d) Cotton plants.

[0109] (3) This algorithm found that after adding CSGEM to ResNet50, some soil semantics still existed in the algorithm class activation heat map. In order to guide the model to further focus on the characteristics of the crop plants themselves, this algorithm designed RDDCM through a parallel nonlinear convolution mechanism to promote the counting model to adapt to the scale changes of crop plants in the image, thereby promoting the convergence of crop plant features to the center of the plant. Fig. 9 As shown in the figure, by analyzing the class activation map after adding RDDCM, it is found that the class activation area is highly focused on the central area of ​​the crops, which greatly eliminates the focus on soil semantics. This shows that RDDCM can enable the model to more accurately perceive the central part characteristics of crops of different scales.

[0110] Fig. 9 (a) Watermelon plants; (b) Rice plants; (c) Corn plants; (d) Cotton plants.

[0111] (4) Perform dual-path feature fusion on the crop plants obtained in step (3). During the feature fusion process, this algorithm introduces the CoT module to form MCPCNet. This algorithm uses the self-attention mechanism in CoT to fully mine the contextual information of the fused semantic features, thereby improving the generalization of the algorithm. Fig.10 As shown in the figure, by analyzing the class activation graph after adding CoT to the algorithm, it is found that the central class activation area of ​​the four crop plants accounts for a larger proportion, indicating that CoT guides the model to pay more attention to the characteristic center of the crop plants, thereby improving the model's accuracy and generalization ability. This is mainly because the self-attention mechanism in the CoT module can fully mine the contextual information of the crop plants, thereby enhancing the model's attention to the central area of ​​the crop plants and realizing the model's generalization ability for counting different crop plants.

[0112] Fig.10 (a) Watermelon plants; (b) Rice plants; (c) Corn plants; (d) Cotton plants.

[0113] (5) Fig.11 As shown in the figure, the dual-path fusion semantic features of AMP and DMP in step (4) are element-wise multiplied, and a 1×1×1 convolution is used to generate the crop plant count map F. refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting .

[0114] Fig.11 (a) Watermelon plants; (b) Rice plants; (c) Corn plants; (d) Cotton plants.

[0115] The present invention aims at the defects in the current crop technology research field and has the following three key points:

[0116] (1) The requirements for point labeling of various types of crops are explained, and point label data is used for training. During the training process, data enhancement methods such as cropping and flipping are used to enrich the amount of training data. Compared with anchor frame regression training, the basic development cost of data labeling and large-scale data preprocessing is reduced;

[0117] (2) The target objects of the method of the present invention are various crops, which solves the problem of current research fields focusing on a single type of crop and proposes a universal counting algorithm for various types of crop targets.

[0118] (3) In order to reduce the impact of complex background information and scale changes of crops on counting accuracy, this algorithm adopts three improvement measures. First, this algorithm designs a concurrent spatial group enhancement module CSGEM, which enhances the semantics of crop plants by adjusting the weights of concurrent spatial groups and suppresses the influence of background information such as soil and shadows; second, this algorithm designs a residual dynamic dilated convolution module RDDCM, which adapts to plant scale changes through parallel nonlinear convolution; finally, this algorithm introduces the Contextualtransformer module to integrate the contextual features of crop plants, and combines the self-attention mechanism to further enhance the generalization ability of the model.

[0119] This paper selects four more advanced point supervision algorithms in recent years for comparison with MCPCNet, namely: M_SFANet (DOI: 10.1109 / ICPR48806.2021.9413286),

[0120] DSNet(DOI:10.1145 / 3460426.3463628),

[0121] P2PNet_Soy(Doi:10.34133 / plantphenomics.0026),

[0122] MLAENet(Doi:10.34133 / plantphenomics.0100)

[0123] The present invention uses the absolute mean error MAE, mean square error MSE, symmetric mean absolute percentage error SMAPE and correlation coefficient R 2 As the evaluation indicators of algorithm accuracy and robustness, the computational load FLOPs and FPS are used as the evaluation indicators of algorithm computational load and inference speed. The calculation of the accuracy evaluation index is shown in formulas (9) and (12), and the comparison results are shown in Table 1. This algorithm uses bold words to represent the optimal index.

[0124]

[0125] From formula (9) to (12), C i represents the predicted value of crop plants, represents the true value of the crop plant, represents the average value of the true value, N represents the number of images in the test set, and among the evaluation indicators, the smaller the values ​​of MAE, MSE and SMAPE, the better. 2 The closer it is to 1, the better, which proves that the model is more accurate and robust.

[0126]

[0127] Table 1. Algorithm indicator comparison chart

[0128] The comparative analysis in Table 1 shows that MCPCNet outperforms the other four point-supervised algorithms. MCPCNet has the lowest MAE of 2.577, with significantly reduced prediction errors compared to M_SFANet (3.002), DSNet (4.473), P2PNet_Soy (5.267), and MLAENet (3.866). This indicates that MCPCNet provides more accurate crop plant counting results. Similarly, MCPCNet has the lowest MSE of 14.289, which further highlights its accuracy in handling plant size and distribution variations. In addition, the determination coefficient R of MCPCNet is 2 The highest is 0.991. Compared with Baseline (0.985), DSNet (0.972), P2PNet_Soy (0.964) and MLAENet (0.977), MCPCNet has a higher fit with the actual data, indicating that its prediction results are highly reliable and accurately reflect the actual annotation situation.

[0129] In terms of computational efficiency, MCPCNet performs well, with the lowest FLOPs of 62.927G, which is more efficient than Baseline (232.461G), DSNet (197.473G), P2PNet_Soy (158.557G), and MLAENet (158.327G). In addition, MCPCNet has the highest FPS of 31.214, surpassing other algorithms in processing speed. This efficiency is attributed to MCPCNet's choice of ResNet50 as an effective feature extractor and the implementation of parallel nonlinear convolution using shared output channels in the RDDCM module. This design greatly reduces the computational load of the algorithm while meeting the feature extraction requirements of various crop types. In addition, MCPCNet's SMAPE remains at the lowest level, only 4.145%, which shows that it has the ability to be effectively promoted under different conditions and scales.

[0130] In order to analyze the differences between MCPCNet and other algorithms, this algorithm visualizes the crop plant counting results of all comparison algorithms by introducing class activation heat maps. Fig.12 As shown, the prediction results (Prediction) of each algorithm are compared with the actual value (Ground_truth), which clearly shows its accuracy in a visual numerical form. The light circles and dark circles in the figure represent missed detections and false detections, respectively. As shown in the prediction results, MCPCNet is very close to the actual values ​​in the prediction results of all crops. Specifically, compared with the actual values ​​of 17, 123, 52, and 89, MCPCNet predicted 16.6 watermelons, 123.7 rice plants, 52.0 corn plants, and 88.7 cotton plants. The accuracy of different crop types remains consistent, highlighting the robustness and reliability of MCPCNet.

[0131] The prediction results show the error sources of various algorithms, especially mispredictions and missed predictions. The baseline and DSNet show a large number of mispredictions, especially in rice fields ( Fig.12 (b)), they misidentified soil shadows (circled) as plants. Similarly, in cotton fields ( Fig.12 (d)), these algorithms mistakenly count weeds as cotton plants. This problem highlights the inability of these algorithms to balance the semantic weights of crop plants and soil, resulting in poor counting accuracy and robustness. MCPCNet effectively solves this problem by combining CSGEM, which enhances the semantic representation of crop plants while suppressing background semantics such as soil. The adjustment of semantic weights not only improves accuracy, but also provides inspiration for improving point supervision algorithms in other fields.

[0132] In addition, as shown in the prediction results of MLAENet and P2PNet_Soy, missed detection is another source of error.

[0133] These combined advantages highlight the potential of MCPCNet for monitoring and large-scale agricultural applications as an accurate and efficient tool for counting crop plants. The significant improvements in accuracy, computational efficiency, and generalization capabilities of MCPCNet highlight its superiority and practicality in addressing complex agricultural challenges.

[0134] The above description is only the preferred embodiment of the present invention, which does not limit the protection scope of the present invention. For those skilled in the art, the present invention can be modified and varied in various ways. Within the spirit and principle of the present invention, any change, modification, replacement, integration and parameter change of these embodiments by conventional substitution or capable of achieving the same function without departing from the principle and spirit of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images, characterized by: The following steps are involved: S1: Data preprocessing: use points to mark the crop image, and the marking points are placed in the center of the crop as much as possible. The data is cropped, flipped, and processed for data augmentation. S2: The RcsNet50 backbone network is used as a feature extractor to encode the input image and obtain high-level semantic features in the image by downsampling the crop plant image; S3: Use the CSGEM module to enhance the semantics of crop plants and suppress the semantics of background information such as soil and shadows by adjusting the semantic weights; S4: Use the RDDCM module to adapt to the scale changes of different crop plants through parallel nonlinear convolution mechanism; S5: Use the attention map path AMP and the density map path DMP for dual-path decoding respectively, and introduce the Contextual transformer (CoT) module in the decoding process, fully mine the context information through CoT and use the self-attention mechanism of CoT to enhance the algorithm's optimization ability; S6: Apply element-wise multiplication to the AMP and DMP decoding results to generate a refined final crop plant count map pair F refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting .

2. A multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 1, characterized in that: The S1 is divided into the following steps: S1.1: Collect TIF format data of various types of crop plants at multiple scales; S1.2: The image size is cropped to 640X640; S1.3: Perform data annotation and data enhancement.

3. The multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images according to claim 1, characterized in that: The specific steps of S1 are as follows: The crop plants used data of four kinds of crops, namely watermelon, cotton, corn and rice. The watermelon, cotton and corn plant data were obtained through independent field collection, and the collection equipment was a drone equipped with a visible light imaging sensor; After screening, we finally collected 1246 crop plant images with good quality. Labelme was used to annotate the data points, and a multi-category crop plant count dataset MCPC-Dataset containing 71566 annotated points was obtained. Data augmentation is performed during the training process. The data augmentation methods include: first, randomly resizing the short side of the image to 512 according to the ratio [0.8, 1.2], and cropping image blocks with a fixed size of 400×400 at random positions; then randomly flipping it horizontally with probability 0.5, using parameters [0.5, 1.5] with probability 0.3 to perform gamma contrast transformation on the image, and randomly changing the image to grayscale with probability 0.

1.

4. The multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 2, characterized in that: The specific steps of S1 are as follows: First, ResNet50 with the fully connected layer removed is used as the feature encoder (FME) to encode the crop plant image. The output form is a feature tensor. For the input image In, the encoding process uses F i (·) indicates that conv2_3, con3_4, conv4_6, and conv5_3 output semantic feature tensors of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image size, respectively. The features encoded by the feature encoder are called Res out , the encoding process can be expressed as shown in formula 1: Res out =F i (In)(1)。 5. The multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 3, characterized in that: The specific steps of S3 are as follows: The CSGEM module is used to enhance the semantics of crop plants and suppress the semantics of background information such as soil and shadow by adjusting the semantic weights. The features input to CSGEM are Res out ; For the concurrent spatial group enhancement module CSGEM, specifically: the encoding feature Res with size C×H×W is out Split into G groups along dimension C, and then use the spatial averaging function Get the global statistical feature g to approximate the semantic feature Res out , as shown in Formula 2: Global statistical features g and Res out Perform Position-wiseDotProduct and Normalization to obtain good response distribution characteristics a i , using feature a i And Sigmoid activation function for Res out Scaling is performed as shown in Formula 3: In addition, Res out Perform global pooling, then perform ReLu and Sigmoid activation to obtain calibration weights Res out Perform 1×1×1 convolution and directly use Sigmoid activation to get calibration weights So the final calibration weight is Using Calibration Weights right Weight calibration is performed to obtain semantically enhanced features of crop plants As shown in Formula 4:

6. A multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 5, characterized in that: The specific steps of S4 are as follows: The residual dynamic dilated convolution module (RDDCM) can promote the algorithm’s perception of crop scale changes through parallel nonlinear convolutions. The features input to the RDDCM are In order to facilitate the expression of the formula, is x; Perform global pooling on x and normalize it using the fully connected layer, ReLU function and Softmax. This process will produce 4 normalized weights {π1, π2, π3, π4} with different convolution kernels. These normalized weights π i It is not fixed, but dynamically adapts to changes in the input feature x; normalized weight π i Obey the normalization constraint and ensure that π i The sum is equal to 1, parallel convolution conv i The weights and biases are represented as {W i ,b i }, the nonlinear function can be expressed as Formula 5: Where g(·) represents the activation function, W(x) and b(x) are shown in Formula 6 and Formula 7 respectively: W(x)=∑π i (x)W i (6); b(x)=∑π i (x)b i (7); Then, a 1×1 convolution is used to calibrate the crop plant feature x and combine it with the dynamic convolution feature Combined to derive the calibration characteristics Subsequently, the attention mechanism (shown at the top of Figure 5) is applied to x as a weight and applied to the calibration features Thus, the adjustment feature y is obtained which takes into account the change in crop plant scale.

7. A multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 6, characterized in that: The specific steps of S4 are as follows: concatenating and decoding the semantic features obtained from steps S2 to S4, upsampling and cascading the semantic features obtained from steps S2 to S4 in order of output, and performing feature fusion in order according to the semantic feature size; in the feature fusion process, introducing the CoT module, and using the self-attention mechanism in CoT to fully mine the context information of the fused semantic features, thereby improving the generalization of the algorithm.

8. The multi-scale semantically enhanced point supervision algorithm for counting multiple types of crop plants in aerial images as claimed in claim 7, characterized in that: The specific steps of S6 are as follows: As shown in formula (8), the dual-path fusion semantic features of AMP and DMP in S5 are element-wise multiplied (Element-wise multiplication), and a 1×1×1 convolution is used to generate the final crop plant count density map F refine , and use the sum function to calculate F refine Sum the predicted number of crop plants S counting ; S counting =sum(F refine )(8)。

Images